For day-to-day work we have todos, milestones, and sprints, and those work fine when we're actively in that mode. But my open source projects with larger arcs, and ideas that need a lot of thought before I can build them (let alone anybody else), have always been missing something. It turns out I already knew what it was: a written proposal process, sometimes called ADRs. They've been very effective working with coding agents, but also, having coding agents has finally made the process practical for me.

The usual loop

Working with coding agents, I've settled into a plan-then-implement rhythm. I'll write a rough sketch prompt, then have the LLM turn it into a PLAN that's sort of like a spec. I review it, modify it, and then tell the agent to implement it.

From there it usually drifts out of sync, because there's no formal process. Sometimes when things are roughly done, I'll have the agent convert the PLAN into an updated DESIGN or SPEC that lives with the code, so there's some kind of design documentation and history. Other times I don't even keep the PLAN around. I just make commits and iterate forward.

For most projects so far that's been okay. Many open source projects, and some of my bigger projects usually ask you to "submit an issue before a PR" so there's a chance to talk about the design first. That's a hint at what's missing: a place for the design to be discussed and decided before anyone writes code. Not just initially, but throughout development.

How hyper9 started

hyper9 began the way most of my projects do lately: chatting with ChatGPT on my phone, notebook in hand. I put the idea together in my head, did research, and validated assumptions. Eventually I said "draft a design doc," and that became the working design document until implementation started.

The project was too big for a single plan, so the doc broke it into milestones. Each milestone was a collection of things. The early ones were small, but they got bigger and bigger. Though, you wouldn't notice it by looking at the words.

I should have seen the problem coming when I started doing "milestone prep." Before taking on M3, I'd ask what needed to happen first. That turned into a planning step of its own, full of unlisted work: set up a test harness, do some prerequisite work, answer some open questions. Once that was done, I could usually turn the milestone over to the agent to implement on its own.

Stuck in research mode

As the project grew, the milestones I'd defined upfront grew too. Sometimes after prep and planning, the agent would just read files and look things up for 30 minutes. One time I left it for over two hours, and it still hadn't written anything. It was all reading source and researching, which does go into its context and makes for a better implementation.

It was funny, it was like a new junior programmer too afraid to start because they don't feel prepared. Safer to just keep learning.

The real issue was that the milestones were too big. They were one-liners like "M4: this, this, this, and this. Should be able to X." Each of those "this" items probably deserved its own plan.

Patching the process

This time I did one thing right: I had the agent keep the overall plan, which had become DESIGN, up to date. Whenever planning or implementing a milestone surfaced something that would change or add to the design, the agent updated the doc. It also added a short entry to a Milestone Changelog section, recording what changed and why, grouped by milestone.

That worked, but then dependencies showed up. Dependencies were how the milestones got ordered in the first place, and new discoveries could change that order.

Then I started running work in parallel. A milestone plan would lay out steps, and if one could be done independently, I'd have another agent session take it. They usually touched different files, and occasional overlap was fine. But the DESIGN doc and its changelog became a shared hotspot that every session wanted to edit.

I originally had M0 through M10 milestones. I assumed the early ones would be the hardest, with all the setup and foundation work, and later ones would get easier as "just features." The opposite happened. The project got bigger and more complex, and the later milestones listed a bunch of pretty big things.

So I stopped at M4. It was a good point for validation and demos that showed, yes, this will work. And then I had the agent help me come up with a new process.

Enter proposals

I've always wanted to run a project on a written proposal system. The idea goes back to internet protocols and RFCs (Requests for Comments). Python has PEPs (Python Enhancement Proposals), and Go has its own proposal process. You get a written record of how an idea evolved: its design and its plan for implementation. You can collect comments and revise before approving and building it. And it lines up neatly with the plan-then-implement loop you already want with coding agents.

So I based hyper9's new process on Go's, with some modifications. Fittingly, the first proposal was for the proposal system itself: proposals.md explains the motivation, how it works, and how we migrated to it. The rules live in the design README.

The short version:

• No more milestones. Each piece of work is its own proposal, a Markdown file in design/ covering design and planning.
• Dependencies in frontmatter. Each proposal lists what it depends on, along with status and other bookkeeping. The order of work comes from the dependency graph, not a numbered sequence.
• A clear workflow. A sketch becomes an initial proposal with open questions. Then there's room for discussion, a final comment period, and formal approval.
• A tool that knows what's ready. It reports when a proposal is accepted and all its dependencies are implemented, so it's ready to build. It also validates the graph in CI and regenerates a status table in the project README.
• GitHub issues for discussion. Filing a proposal opens an issue for open discussion and tracking. Revisions happen as commits to the doc.
• DESIGN stays current. DESIGN.md now describes only what exists. The PR that finishes a proposal updates it.
• A human signs off. Agents can help write proposals, but a named person has to direct, review, and sign the work.

The old M0 through M4 became a single "baseline" proposal marked implemented. The remaining milestones were split by feature into separate drafts.

Why it works

This solves the pre-development design process. It documents the thinking and the original plan, and it keeps the main design doc up to date as a current spec rather than a pile of plans.

It also makes open contributions easier, both for design and for the question "what's ready that I can help with?" And that applies to agents too. I can point an agent at a proposal to flesh it out, challenge it, or at another to implement it. I can point several agents at different proposals without them getting in each other's way on design, prep, and planning. Actual implementation happens in branches when needed.

This setup makes agent assisted human contribution easy, too. You can do the same thing I do: point your coding agent at a proposal, keep a loose eye on what it does, and review what it completes. Though, if you didn't write the proposal, it's probably a good idea to talk through it with the agent to better understand it.

The best part is that the formalisms, the design and customization of this workflow, but more importantly the work of actually following it are suddenly not overwhelming. Agents carry most of that load. A process like this would have felt like bureaucracy for a small project before. Now it's mostly free.

A real roadmap

hyper9 now has a real roadmap: proposals at various stages, each with a GitHub issue. Some are ready to start, and some are longer-term ideas still being developed. There are also a few unnumbered sketches in the design/ directory that don't have issues yet. A table in the project README, kept up to date by a tool, makes it easy to see what's ready to work on.

I can think through a bunch of ideas, load them in as proposals, and let them simmer and develop until I think they're ready to be implemented. It's not quite a software factory, but it definitely feels like an augmented assembly line. One where feel in control of the design, and make better design decisions.

In this series of posts, I’ll show how I’ve been turning VS Code into a reusable module; something you can embed, extend, and ship as part of your own product.

Why VS Code as a Platform?

I never thought the developer ecosystem would embrace a single editor as much as it has VS Code. Its dominance as a general-purpose coding environment is undeniable. This popularity and familiarity is one reason to build on it, but there are other attributes that make it a solid building block:

• Generalized Design: It was designed not just as an editor, but as an integrated environment for various tools involved in the development workflow and beyond.
• Highly Extensible: Its extension API allows for deep customization, from minor modifications to total transformations, even into non-code oriented tooling.
• MIT License: The permissive MIT license means you can freely adapt and distribute it, provided you don't call it VS Code. The open source project is officially called "Code - OSS"

Given its intended purpose is for software development, you might think this limits it to being a framework for devtools and code adjacent domains. Really, though, you can disable and rip all of that out and still have a great shell for applications that use the same general shape of its user interface. It might not be a fit for all products, but any application built around tabbed views, user-customizable sidebars, and is advanced enough to need a command palette would be a good candidate.

Even then, having integrated code editing abilities might be more of a benefit for non-development applications than you might think. Any professional knowledge tool these days needs some level of integration and extensibility. Imagine giving power users the ability to build these in-app. An embedded VS Code could just be used for this mode of your application. Imagine a game that comes with its own modding environment. Development environments are incredibly intricate systems to build, so having the most familiar of them that you can embed in your application is a huge win.

However, there is another argument that some kind of development environment might be worth including in your application, if not building on top of one directly.

Embracing the Future of AI Agents

Rather obnoxiously, AI is everywhere. For the foreseeable future, we are going to be working with more and more AI. But also for the foreseeable future, code will remain significantly cheaper to run than AI.

Luckily, AI is pretty good at writing code. To me, this points to an eventuality where most AI agents will be automating with code rather than direct control. This is great for a lot of reasons, which I can get into another time if people are interested. For now I just want to point out that any application or system that involves agentic AI will likely need a development environment and editor as a sort of "admin backend" for a behind-the-scenes code generating agent.

As of right now, the leading AI co-development environment, regardless of model or coding agent technology, is VS Code. So if you're interested in building agentic AI products or involving code generating agentic AI in your product, it might be worth thinking about how you can embed an environment like VS Code in your product.

Maybe Avoid Forking VS Code

Let's say you're in on using the VS Code open source project in your product. Your first thought might be to fork it, essentially adopting it wholesale. After all, that's what Cursor and Windsurf did! Well here are some quick stats that might be good to know about the VS Code project:

• Massive Codebase: Currently it contains 1.3 million lines of TypeScript. This is comparable to the codebase of Grand Theft Auto III.
• Extensive Dependencies: Right now I count over 900 dependencies, which is a lot even for your average already-too-many-dependencies Node.js project.
• Bloated Project Size: After a full npm install, the project directory expands to a hefty 2.6GB. Not sure what all of that is, but almost 1 gig of that is node_modules.

This is what you'd be inheriting and you haven't even started your application. On a high-end machine like my M3 Max, it takes around 30 minutes just to build the frontend. It takes 50 minutes on GitHub Actions. Why? I don't know, but maybe this is partly why they're porting the TypeScript compiler to Go.

Maybe you're just one person, like me, and this already seems overwhelming. Even for a small team, such a large and complex codebase and all those dependencies is rather prohibitive. Or maybe your team isn't interested in working with TypeScript, let alone 1.3 million lines of it. Or maybe, like me, you'd like to avoid Node.js and all those dependencies like the plague.

So is it already game over or is there an alternative approach? Well, since I'm also on this path as one developer with just a handful of GitHub sponsors funding this work, I had to find one. Because building an alternative to VS Code from scratch is also not an option for me.

Towards Another Approach

Thanks to VS Code's extensibility, there's a good chance the majority of what we'd want to change or add, we could do through extensions. This means changes to the codebase could at least be minimized to a few patches, making a full fork potentially unnecessary. We'd still need a build of VS Code we can configure and ideally embed in our application. It turns out there is a little known official option for this, but I'll have to explain some background on how it works.

VS Code is an Electron app, which means it's mostly a web frontend that runs inside a webview in a native window. That's essentially a browser window without the chrome that just runs VS Code. What can't run in the webview is run in Node.js outside the webview. Electron provides a framework for this with cross-platform APIs to common native functionality like webview windows, menus, etc. The webview frontend talks to the Node.js backend via RPC and that's generally how most Electron apps work.

For VS Code, what runs outside the webview are subsystems for filesystem and shell access, the ability to run subprocesses for language servers, and extensions themselves that are typically run in a separate Node.js process. This will come up later.

Despite being made to allow an app to be built with web technologies, most Electron apps become quite coupled to Electron and Node.js. When people started building cloud IDEs where Electron didn't make sense and you could just view the frontend in your browser, they couldn't use VS Code because of this coupling. Eventually, thanks to projects like openvscode-server, which patched VS Code to run as a normal web app in a Docker container, VS Code took many of these changes upstream and began to decouple VS Code from Electron.

VS Code for the Web

This decoupling led to VS Code for the Web, a build target of VS Code that can run almost entirely in a webview or browser without the Node.js backend. Where openvscode-server decoupled it from Electron, VS Code for the Web even decoupled it from its own Node.js backend. In other words, it's a build of the VS Code editor that's just static files you can serve with any HTTP server.

When compressed into a zip file, the result is an 18MB artifact that could be deployed or embedded anywhere. In fact, there is a public deployment of VS Code for the Web at vscode.dev, and even a semi-secret version deployed for GitHub. While logged in to GitHub, if you hit period (.) while on any repository, you're taken to that repository opened in an instance of VS Code for the Web. This is different from a GitHub Codespace, which runs a full backend and development environment in the cloud, similar to what you can do with openvscode-server. The reason it's different gets at the core limitations of VS Code for the Web:

• No shell or direct filesystem access
• No extensions that use Node.js APIs

It might be obvious but they both have to do with not having the Node.js backend. One of the major changes they made to support this build target was to make an extension host that runs in-browser using web workers, so extensions can only do what can be done in a browser web worker. Luckily, most of the core extensions have been ported to work in web workers.

Unfortunately, most third-party extensions are likely to use at least one Node.js module or NPM package. This hurts most for language extensions that typically use Node.js to run an LSP server subprocess. So if you need to support specific third-party extensions, you'll have to fork them and find a way to make them work. Better than forking all of VS Code?

The lack of shell and filesystem are pretty easy to fix because both can be provided by extensions. We can just have a kind of "system extension" that gets a filesystem and shell over WebSocket.

Embed and Extend

With VS Code for the Web we get an 18MB editor "module" in a single file that we can embed in our apps and then extend with extensions. While it comes with some limitations, I believe it opens up even more possibilities. We could:

• serve it from any CDN or static host for the cloud.
• use it on the desktop through any webview wrapper like Electron, Tauri, etc.
• serve it in an application of any language to access via the browser.

It becomes trivial enough to embed as a built-in editor for whatever application you're building. And it's extensible and malleable enough to become any number of different non-code editors, like a DAW for audio, or a CAD tool, or a game engine editor.

In the following posts for this series, we'll build a cross-platform desktop code editor in Go using this approach. Effectively a clone of VS Code that you can use as a foundation for your own application, without all the baggage of building directly on a fork of the VS Code project.

With this last release of Wanix, I'm at it again. This time with the successor to Unix, a little known operating system called Plan 9 from Bell Labs.

Plan 9 has been on my mind for quite a while. In fact, around the time of that first talk on webhooks, the team behind Unix and Plan 9 was being re-assembled to create the Go programming language. I pretty instantly fell in love with the Go worldview, which turns out to be an outgrowth of the Unix and Plan 9 values of simplicity, pragmatism, economy, and ultimately compositionality.

Like Unix, the Plan 9 environment is really made for programmers and system operators. I'll leave a deeper explanation of what makes Plan 9 great for another post, but I do get into it a bit in the demo video for Wanix:

youtube · loads when you press play

While I wanted to incorporate Plan 9 ideas into Wanix from the beginning, it wasn't until we rebuilt it from scratch with that intention that the magic really starts to come through. That's what this preview release is about.

Wanix is a whole new beast now. It's no longer a singular computing environment that runs in the browser. It's now a primitive for building environments in general. The demo shows a shell environment, but this environment is not the point. It's just a way to bootstrap Wanix so you can use and explore it interactively.

The point of this preview release is to get this primitive out there. I have my uses for Wanix, and I plan to share them with the final 0.3 release, but until then I wanted to let it all percolate. Maybe inspire people to get creative with their own use cases.

Here's a quick rundown of Wanix features in this release:

Plan 9 inspired design

With the original intention to enable exploring Plan 9 ideas on modern platforms, we've ended up with a radically simple architecture around per-process namespaces composed of file service capabilities using similar design patterns to those found in Plan 9.

Single executable toolchain

The wanix executable includes everything needed to produce Wanix environments.

Filesystem is the only API

The Wanix microkernel is now simply a VFS module with several built-in file services exposed via a standard filesystem API. This ends up making the module itself a file service.

Built-in Linux shell

Using the built-in file service primitives, Wanix can bootstrap a Linux-compatible shell based on Busybox. It comes with several helper commands for working with built-in file services.

Tasks and namespaces

The Wanix unit of compute is a task, which is equivalent and compatible with POSIX processes but allows for different execution strategies. Each task has its own "namespace," which is the customizable filesystem exposed to the task.

Core file services

Wanix includes two singleton file services: one to manage tasks (similar to procfs), and one to manage "capabilities" which are user allocated file services. Built-in capabilities include: tarfs, tmpfs, and loopback.

Web file services

With future non-browser deployments in mind, all web related file services are packaged in a web module, which is also built-in but not considered core. This module includes these work-in-progress file services:

• opfs: For working with the OPFS browser storage API
• dom: For inspecting and manipulating the DOM
• worker: For managing web workers
• pickerfs: Capability wrapping the window.showDirectoryPicker() method (not available yet in Safari and Firefox)
• ws: Capability for working with WebSocket connections
• sw: For configuring the service worker, which is used by the system now to cache all resources needed to run Wanix allowing offline usage, as well as exposing virtual URLs to the root namespace.

Go programmers might also appreciate the filesystem toolkit we've been working on since before Wanix. It builds on the fs.FS abstraction in the standard library and gives you DSL-like utilities for defining virtual filesystems like the file services in Wanix. More on that in a dedicated post as well.

So far, the feedback has been really positive. I appreciate everybody taking the time to process it. There's still lots to do. Wanix is itself its own universe, but it's just one layer of the Tractor project. After a little vacation I'll be back to continue work on both fronts. As usual, I'd love help.

Speaking of help, shout-out to Joël Franusic for the help and support. And as usual, big thanks to my GitHub sponsors for making this possible.

]]>https://progrium.xyz/articles/2025/state-of-tractor-2025https://progrium.xyz/articles/2025/state-of-tractor-2025Fri, 28 Mar 2025 00:00:00 GMTHey everybody, welcome to 2025! Belated as it may be. For the past 5 years I've been heads down on an epic journey to find new and better (read: simpler) ways to build software. Three years ago I posted a kind of thesis video for this work and gave the journey a name: Tractor. Today I want to share a progress report.

Tractor is not a single project, but a growing toolkit and developing philosophy around building software. However, documenting the specifics of this has not been an immediate priority. The vision is so big and abstract that just writing about it, especially so early, seemed less efficient (and less fun) than just building. Plus, for me it feels impossible to talk about new ideas without being able to reference tangible examples and let people process them interactively.

This year seems to be a turning point. I have enough building blocks that they are starting to come together in interesting ways. I've also collected a bunch of bite sized developer tips along the way. For these reasons, I've decided to start posting publicly more. Hold me to it!

Process and Aims

Like all of my work, I'm driven by an interactive process of idealized design, speculative primitive disaggregation, and bottom-up exploration. There is an evolving vision of how making software could be, which is reified into an imaginary toolset and paradigm. This is then broken down into the fewest primitives needed to build that toolset and paradigm. Each primitive is imagined and validated against two criteria: how easily it enables the vision, and how many other possibilities it enables.

Put succinctly: I'm collecting and building powerful software primitives and design patterns that will enable the future of software and computing.

If people don't do this, nothing changes. In fact, by default, things get worse. In this case, the "future" is a stand-in for "something better", which I understand is quite relative and is a good reason I should write some kind of manifesto. In the meantime, here are a few ideas I'm aiming for:

• Minimize the number of steps to get to any software from 0
• Remain polyglot, the answer is not one langugage
• Align with emerging technologies that matter (ie CRDTs not blockchain)
• Be a platform for many visions, the answer is not one vision
• Close gap between builder and user of software with live malleability
• Explore malleability through virtualization of development tools
• Bridge worlds and simplify, the answer is not one perspective/paradigm
• Start with plumbing, build bottom-up (after idealizing top-down)

This is a long journey, but it should pay dividends for everyone.

2024 in Review

Last year we got a lot done. We had some important releases, solved some hard problems, and started the process of getting feedback from peers on key technologies now that they've been baking a few years.

Wanix

Wanix started around October 2023 as a standalone experiment that would be fun to talk about. The experiment showed promise, so I collaborated with a friend to explore further. About this time last year, we put out a 0.2 release and got to show it off at WASM I/O.

Wanix started as an homage to early Unix, seeing what fewest elements are needed to build a general purpose computing environment capable of self-hosting its own development. Except this time, starting in the browser and building around WebAssembly.

By the 0.2 release, Wanix was a good demo, but not terribly useful. It was also somewhat impractical being a 60MB asset for the web, and not easy to get running locally for development. However, a few things I think it validated:

• WebAssembly is exciting, but still quite early
• A CRDT-based multiuser filesystem is compelling, even if "slow"
• It's a lot of work recreating a Unixy userland

I also came away convinced Plan 9 was onto something, but disappointed this iteration of Wanix wasn't able to capture that magic. I was also a bit bummed I wasn't able to get VSCode to run in Wanix, as it would have greatly improved the UX of using Wanix as a development environment.

DarwinKit

DarwinKit is an ambitious yet boring project to make Apple APIs usable in Go. After deciding Electron was not going to be the answer in Tractor for cross-platform webview apps, I started a cross-platform library in Go focusing more on the desktop APIs exposed by Electron. This meant working with Apple APIs, and at the time there were no Go bindings.

The desktop API library only needed a few basic Apple APIs, but the challenge of making high-quality, idiomatic bindings for every Apple API (one of the largest API surface areas, rouhgly 1/4 the size of AWS) was compelling. It would have to be automated. If we could do it for Apple, we could do it for any API, which would come in handy in the future.

Back in 2023, I re-wrote the whole system, getting us an order of magnitude more coverage, from maybe 150 symbols represented to over 30k symbols. Still far from total coverage, but enough that it seemed possible someday. Not only was this a rewrite, but a rename, so it was initially released as a preview to give time for people to prepare for a new repository name and backwards incompatible API.

In July 2024, we finally released DarwinKit 0.5. Other than Wanix, DarwinKit has been the only project since 2020 to find a regular trickle of organic discovery. This is somewhat annoying because it's also at the bottom of one of the deepest Tractor yak shaves and does not contribute much to the development of the system. Though I have found it useful in developing techniques to generate these kinds of bindings, which I documented in this talk.

Manifold

Manifold is one of the key technologies of the Tractor system, providing a sort of universal structure to assemble software systems on. As such, it's been allowed the most time to cook. It's been re-written at least 5 times by now and I went into it knowing that would probably be necessary.

Last October it was finally demonstrated publicly at SPLASH (previously known as OOPSLA). Unfortunately, trying to pack so much into a 20 minute talk probably left most people confused. I did get kudos from a Smalltalker for making Go more like Smalltalk.

It was a one-day workshop packed with dense talks and demos. By my slot my brain was nearly fried, so it was also not one of my best performances. The win, though, was just getting it out there, and I've been able to have some good conversations about it since.

New Apptron

If Apptron is a familiar name, be prepared to unlearn what it is. Going back to the Electron-inspired desktop API, it was originally packaged up as a standalone primitive that could be used a bunch of ways, from any language, and could even be used in shell scripts. It originally had a name as boring as the technology was, Hostbridge, but that name turned out to be trademarked. Brainstorming led to Apptron, which was less descriptive, but definitely more cool, and even sounded like Electron.

With all these projects, I'm not one to push something more than there is signal for it. The response to Apptron was lackluster. Maybe because it was sponsor-walled, or maybe because people had an Electron API at home, but it doesn't matter that much. I made a great component for Tractor and I knew these early foundational projects would be hard sells, even just for attention.

Eventually the code for what was once called Apptron was rolled into a package simply called "desktop", freeing up Apptron to be used for something more deserving of a cool name.

After the Wanix 0.2 release, I went hard on "VSCode in the browser." After 9 months off and on, I finally had VSCode running in the browser with a full Linux development environment that was actually usable. I'm not sure how many people have achieved this. I might be one of the first. Yes, cloud IDEs are a thing, but they run the Linux environment, usually Docker, in the cloud. They usually even run the VSCode backend in the cloud.

I started to envision a possible product. Not quite "Tractor Studio" but definitely something resembling it that could iterate towards it. I'll save specifics for later but we started using the name Apptron for this. In fact, there is a "tech demo" version online today at apptron.dev. Check it out!

There's a pretty good story about this side quest that involves some of these other projects in the next section. Hopefully I'll get to sharing it soon.

Other projects

Most of these aren't documented yet and will probably come up in future posts, but here are some other project repositories and some forks that went up last year:

• https://github.com/progrium/env86
• https://github.com/progrium/httpfs
• https://github.com/progrium/vscode-web
• https://github.com/progrium/go-vscode
• https://github.com/progrium/go-netstack
• https://github.com/progrium/sys-wasm

2025 in Preview

This year I plan to write more and share more publicly here on my blog. This could even include some kind of manifesto or at least an updated thesis. However, what I have most in my writing queue are just quick helpful bits. Patterns and conventions I use in my projects. Simple DIY alternatives to using complex tools. Shorter, more useful posts.

I'm also trying to be more social. More importantly, I'm trying to get back into community building. I have my eye on doing in-person events again, but for now I'm warming up with online events in the Discord or on Twitch. And if you have any favorite conferences or in-person meetups, coworking groups, or anything like that, please let me know! I'm mostly back in the SF bay area these days.

In terms of Tractor projects, there are a few important releases coming up. After that my forward energy will probably go into the new Apptron. However, I'm very open to collaborations, especially projects relevant to my roadmap. As an example:

Taragen

Besides Wanix, over the last few months I've been working with a friend on a Revolutionary, Extraordinary ... static site generator. No really. Actually, I'm pretty thrilled about it. It's minimalistic, but extremely capable. It's written in Go but uses JSX for templates. I've already ported my website to it.

In fact, planning to blog more is what motivated me to finish it up. I'm that unsatisfied with existing options. Finally, though! Something I'm making that will finally be relevant to you. You have a static site, right?

Wanix

This is pretty big to me. Since the beginning of the year, we've been re-building Wanix up from scratch. This time, focusing on the Plan 9 architecture, making it more production ready, and all kinds of exciting things.

We've made a kind of stream series about it all called Wanix From Scratch. Every week we'd do 1 or 2 Twitch streams about 2 hours long. Usually the first hour going over everything that happened since the last one. Then the second hour we try to get some work done. There will be 20 days or 20 videos after the final DEMO DAY happening live on Twitch this Wednesday the 2nd at 11am pacific!

Then after that a preview release will be up. Loads to talk about, but I'll save it for later. I'm very excited, even though it's still just plumbing and most people won't see obvious use cases for it, I think something special is going on.

How to Help

You made it this far. Now you want to, what, stay informed? Join the community? Spread the word? First, thanks for reading. Second, any kind of participation would be huge.

More posts are coming, so get subscribed here if you want. As I mentioned, we're doing more events and coworking in the Discord, so feel free to join us there too.

You can also participate on GitHub with any of the fine projects mentioned. Finally, thanks to my GitHub Sponsors as usual, and a big thanks to my friends that have been helping out more recently.

]]>https://progrium.xyz/articles/2023/treehouse-influenceshttps://progrium.xyz/articles/2023/treehouse-influencesThu, 27 Apr 2023 00:00:00 GMTA quick tour of some of the influences for the Treehouse frontend and where it fits in the ecosystem.Over the past few months, we've built the Treehouse frontend framework into an elegant, quality outliner that's open source, extensible, and gives you control of your data. I’d like to share some of the design influences for the Treehouse frontend, which should give a sense of the unique direction Treehouse is going from here.

The biggest influences for Treehouse are Tana, Notion, and Obsidian. These three represent the state of the art of personal and collaborative information management, sometimes simplified as note-taking tools. However, "note-taking tools" sells them short as they go beyond note-taking and information management. For lack of a better descriptor, many consider them tools for thought.

From Notes to Tools for Thought

For most, note-taking brings to mind simple apps like Apple Notes and Google Keep, or even just a text file editor. These work well for people because they're already there and focus on quick and easy plain text capture. We could call this casual note-taking.

Back in the 2000s, hosted and self-hosted wikis became popular for easy, collaborative web publishing and knowledge management. Like Wikipedia, they could be used to build out hyperlinked knowledge repositories. Many wiki-based tools focused on their use as personal notebooks, one of the most influential examples being TiddlyWiki. The simple versatility of the wiki laid the groundwork for what we call "tools for thought" today.

When Notion appeared in the mid-2010s, it built on the idea of the wiki and introduced structured data management with flexible views that effectively gave you integrated, customizable versions of other productivity tools. Notion, Airtable, and others helped bring in the age of no-code and low-code tools, allowing knowledge workers and entrepreneurs to build their own "apps" or solutions to problems without traditionally building software. Notion brought it all together in a simple, user-friendly experience based around the core idea of wiki-like information management.

Meanwhile, a separate paradigm of note-taking tools emerged, focusing on the nested, tree-like structure of the outline. Perhaps inspired by tools like OmniOutliner and Org Mode for Emacs of the 2000s, Workflowy appeared in 2010 as a no-frills web-based outliner.

Obsidian arrived in 2020 and is a local app focusing on Markdown files stored on your filesystem. Obsidian has a large plugin ecosystem giving it a wide breadth of features, but it’s especially appealing to those that want to own their data. If you strip away the plugins, Obsidian is a pretty simple hyperlinked Markdown editor.

Most recently, a tool in early access called Tana caught my attention. Their key innovation is taking the linked outline model of Workflowy and introducing schemas for nodes, making them into structured data. This gives Tana the embedded database functionality of Notion and Airtable, a step towards bringing the two paradigms of note-taking software together towards powerful, malleable tools for thought.

How Treehouse Fits In

By now there's no shortage of options in this space, both as SaaS and open source. Take a look at this growing encyclopedia of note-taking tools. Like Notion and Tana, many of the apps listed are much more than note-taking tools. Some lean into the framing of "collaborative documents", and some are just categorized more generally as "productivity tools". Tana goes so far as to say "the everything OS".

Note-taking is just the beginning. It's a tangible gateway for something more powerful inherent to computing. Ever since Engelbart's mother of all demos, the computing revolution seems to start with powerful tools for thought, which are, at minimum, good note-taking tools.

Treehouse is a frontend and starter kit for anybody else that wants to explore this space with us. We will release a standalone product based on it soon, but most of the user-facing development will be done in the open source Treehouse project.

Today with Treehouse you can build your own Workflowy equivalent, but soon it will become more comparable to Tana and Notion with the open extensibility of Obsidian. That alone is pretty exciting to have in a minimal open source project, but I can't wait to show you what will come next.

Coming Soon

In the next post, I'll start getting technical and share details on the Treehouse project stack and architecture. If you can't wait, we do have documentation for you to check out.

Thanks for reading, and a big thanks to my sponsors for supporting this kind of open source work. Share your thoughts and favorite note-taking tools in the discussion thread for this post.

]]>https://progrium.xyz/articles/2023/welcome-to-treehousehttps://progrium.xyz/articles/2023/welcome-to-treehouseThu, 23 Feb 2023 00:00:00 GMTAnnouncing Treehouse, an open source note-taking frontend and will evolve into something much more.I'm excited to announce the start of a new major project that I'll be sharing the journey of on this fancy new blog. The project is called Treehouse, which is starting as an open source note-taking frontend and will evolve into something much more.

We're creating a simple, hackable kernel of a note-taking tool as a web frontend that can be extended and customized by developers. Then we'll use that frontend to build a new kind of note-taking product. Today we have an early preview of our first minimum viable prototype of the frontend, and we'll be finishing up this release in the open on GitHub.

The open source project is a functional app frontend that you can deploy and back in various ways. For developers, this "bring your own backend" approach makes Treehouse a great starting point to build your own note-taking tool, just as we intend to do. We also have pre-made backends for those who want to use it with little-to-no programming. Either way, you'll always own your data, and now the system presenting it to you as well.

In this post, I'm going to talk about our minimum viable prototype and the approach we're taking with Treehouse as an open source project.

Out of the Box

While inspired by powerful tools like Notion, Tana, and Obsidian, we wanted to boil our initial goal down to the essentials while still being usable enough to use ourselves. As we continue to plan our prototype release milestone, we'd love to hear what you think is essential for you to have in a tool and platform like this, but here is what to expect as a baseline.

Outliner with Markdown Pages

At the heart of Treehouse is a graph-like system I've been developing called Manifold. This will play a big part in the long-term extensibility of Treehouse, which I'll talk more about in the future. For now, this maps very cleanly to the outliner model popularized by Workflowy, Tana, and others.

However, I'm also a fan of Notion-style pages and Obsidian's commitment to working with plain Markdown files. The Manifold system allows us to make certain nodes into Markdown pages. This hybrid model gets us the best of both worlds with room for even more possibilities later on.

Quick Add and Daily Notes

Being able to quickly get thoughts and information into the system has been a key aspect of every good note-taking system. This can be exposed a number of ways from browser extensions to desktop shortcuts, but what makes any "quick add" functionality quick is not having to think about where it will be organized.

Luckily, "daily notes" has become such a common pattern that many recent tools have managed daily notes built-in. This typical gives you a "today" note that is automatically organized into a calendar-like structure. This conveniently becomes the perfect destination for "quick add" notes.

Full-text Search

Whether you are an obsessive organizer or are too busy to take the time, solid full-text search is basically a necessity. It's how we quickly get to our data.

Out of the box we have a full-text search that doesn't require a backend. However, our pluggable backend will allow you to power search in a number of ways, and also opens up the ability to search into external systems that are important to you.

Built-in Backends

"Bring your own backend" is a critical part of the design of this project, but Treehouse also needs to be usable by people who prefer not to build their own backend. Our prototype release is planned to include a handful of built-in backends to get started with.

Our preview demo is using a localStorage backend, which works for development and special use cases. Of course, we will have a simple local filesystem backend for desktop scenarios, similar to Obsidian.

However, we're most excited for the GitHub backend. This functions similarly to the local filesystem backend, but it would be versioned in a central repository on your GitHub account and be accessible on any online platform.

Backends will not only let you extend storage, search, and authentication, but other aspects in the future.

Project Overview

The Treehouse codebase and open source project is as much the product as its features. The entire user and developer experience is being designed around simplicity and human ergonomics.

The Treehouse frontend is built with web technologies intended for use across web, mobile, and desktop platforms. The project is written in TypeScript using Deno as its JavaScript toolchain.

The JavaScript ecosystem is a mess, but Deno provides a forward-looking, self-contained toolchain that's easy to love. We actually have a zero Node.js policy and avoid NPM modules as much as possible.

With minimal dependencies and an ongoing effort to keep the codebase small, it will always be easy to understand the entire Treehouse system. This will be further supported with a focus on documentation, both API docs and guides.

We're using a permissive open source license, allowing you to use or change our code as you see fit. Contribution and participation is welcome, but so is simply consuming downstream.

The project is actively being developed but we keep a preview demo running off the main branch that should always be in working order.

Coming Soon

I'm excited to give you a deeper look at the project stack in a future post. For now, keep an eye out for the next post digging into the influences for Treehouse. We'll also try to answer "why another note-taking tool", and tease some long-term ideas for the future.

Lastly, I want to mention the project is being developed in the open, not just with the code on GitHub, but most of my work is streamed and archived on Twitch. Feel free to come and co-work with me.

Thanks for reading, and a big thanks to my sponsors for supporting this kind of open source work.

Check out more at treehouse.sh or subscribe to our mailing list for updates!

]]>https://progrium.xyz/articles/2022/cross-platform-native-apis-made-accessible-with-apptronhttps://progrium.xyz/articles/2022/cross-platform-native-apis-made-accessible-with-apptronFri, 01 Jul 2022 00:00:00 GMTApptron is a software primitive that lets you drive native platform APIs using the technologies you’re already familiar with, enabling cross-platform webview apps and more.I’m happy to announce the first major building block of the Tractor System: Apptron.

Apptron is a software primitive that lets you drive native platform APIs using the technologies you’re already familiar with, enabling cross-platform webview apps and more.

Usable from any language, or even the command line, Apptron gives you webview windows and common platform APIs for your simple scripts, homebrew utilities, or full applications. Building cross-platform programs that leverage native functionality (menus, dialogs, notifications, global shortcuts, etc) has never been more accessible.

These APIs are packed into a single executable that can be used a number of different ways, making Apptron a simple but powerful software primitive.

Zero Config Native Apps from HTML

Turn an HTML file and any other web assets into a self-contained, cross-platform executable in a single command.

Language Agnostic Building Block

Use the Apptron APIs like a library from any language without C extensions or tricky event loops.

Shell Scriptable Native APIs

Use platform APIs from CLI commands to break shell scripting out of the terminal and into the desktop GUI.

These features and more will be showcased in a series of demos coming out over the next few weeks. Given all the different ways it can be used, Apptron is something you really have to see in action. Catch their premieres on Twitch, or subscribe on YouTube so you don't miss them.

Early Access

In the meantime, we're opening up Early Access via the Apptron website:

https://tractor.dev/apptron

By joining early access, you'll be invited to the currently private GitHub repository where you can try Apptron out, submit issues, and help bring Apptron to a stable 1.0 release and become public open source software.

Help Wanted

I've been using Apptron in some form with the rest of the Tractor System for over a year now, but to make this a solid standalone primitive I'll need your help. Please consider joining early access to play around with it and give feedback.

Anybody interested in contributing or helping drive this project should definitely request access and get involved. This is one of many projects I have in the pipeline and I can only do so much.

Thanks

A huge thanks to my sponsors, who I depend on to fund this early R&D work on the Tractor System. As long as it's sustainable, I look forward to releasing more powerful open source primitives.

Excited to tell you more next week with the first Apptron demo!

]]>https://progrium.xyz/articles/2022/progrium-technology-thesishttps://progrium.xyz/articles/2022/progrium-technology-thesisTue, 15 Mar 2022 00:00:00 GMTHow and why I'm building the "Photoshop of software" out of generative building blocks.Creating software is too complicated. Our options are overwhelming, our tooling is too bloated, and things rarely ever "just work".

8 years ago, the late Joe Armstrong, co-creator of Erlang, gave one of his last talks at the 2014 Strange Loop conference. It was titled "The Mess We're In" and talked about the increasing over-complexity of software. "Software entropy increases with time," he said, "we need to reverse entropy." He said in his conclusion:

"Computing was about controlling complexity, and we have failed miserably."

An echo of what Djikstra said in the 90s: "It is time to unmask the computing community as a Secret Society for the Creation and Preservation of Artificial Complexity."

Alan Kay, another computing pioneer, has a great word for this "artificial complexity": complication.

"The amount of complication [in software today] can be 100s, maybe 1000s of times more than the inherent complexity."

For 25 years, Kay has been giving talks saying, "The computer revolution hasn't happened yet!" I believe that this is true, in part, because complication has gotten in the way.

The risks with complexity, complication, and bloat are also only growing. It's 2022 and we’re still deploying software with logging libraries that let attackers run arbitrary code. We barely understand our own software systems, let alone the tools we use to make them. And yet, underneath this mess is probably the most powerful and constructive tool humanity has ever seen.
  Building software today could be as fast and intuitive as we made desktop publishing by the 1990s. Suddenly designers could sit with a client in front of the computer and just execute an idea, making iterations together in real-time. Watching the designer use these intuitive tools, the client might think, "Oh, I could do that".

To get there with building software we need to reduce the complexity, and especially the complications, of making modern software. It's not just about user friendliness or making the process more visual. And it’s not about adding more. If anything, it’s about revisiting the idea of doing more with less.

Recently, tools falling under the label of "nocode" have shown the demand for non-programmers to be able to just make what they need, with "no code". It does sort of hint at what a photoshop of software might be. But every nocode tool is a facade, another layer of complication. They exist for a particular market, offered as a service, with no path to a more general means of making software.

These tools have their place, the same way image macro generators have their place when Photoshop and Canva exist. I think if “nocode” tools could really save us, you'd be able to build a nocode tool with itself. They lack the true generativity of programming.

That said, I am all for anything that can be used as a building block in a software system. Whether it’s a library or a web API or a tool repurposed because you like its workflow. The less you have to build yourself, the better.

My experience has shown me time and again that there is a powerful strategy or guiding principle in what I call "generative building blocks."

This philosophy applies to more than libraries and web APIs. I design everything as a building block, and everything out of building blocks. This is about more than just modularity, it's about high-leverage generativity.
  I've worked on a number of projects and technologies trying to bring out this particular generative building block approach and there are a couple of standout hits that I'll use as examples.

In 2006 I was working on a hosted source control web app (like GitHub) trying to expose repository hooks. These are shell scripts triggered by your local source control tool that can be used to customize and extend your workflow, almost like registering a callback for an event. Running user code in a web app, especially shell scripts, was not safe or practical. However, imagining if somehow it was, a flood of possibilities hit me. If only web apps could have extension points! Without over formalizing it, I called this "webhooks" and started giving talks about what we could have if we added webhooks to our apps. Throwing a PHP script online was an easy and common activity for web developers then, so functioning as "hook" script with a URL, you just needed to have web apps let you register it and then hit it on certain events.

Webhooks did end up making web apps more extensible and integratable, increasing their value and potential as building blocks. It also allowed for new kinds of API building blocks to emerge like Twilio, where the initial core product was built around webhooks to integrate with telephony and SMS. Webhooks also started creating a demand for what would later manifest as AWS Lambda for event handling, which quickly inspired "serverless" architecture. The concept of functions as a service was something I was pioneering in 2009 because of webhooks.

Later I found myself helping to prototype and conceptualize Docker, which turned software into a kind of standard unit, hence the shipping container metaphor. Although many interpreted it as a lightweight VM to throw whole systems into, I was in love with microservices designed to be run in tiny containers, configured by APIs instead of files, acting more like a little appliance than traditionally managed software. More like a simple building block.

Docker itself became a building block, opening up a whole new generation of platforms and developer tooling. One of my initial use cases for it was as a primitive to create a simple, hackable version of Heroku. Sure enough, using Docker and a few other components I made, I was able to "throw together" a simple Docker-powered Heroku alternative in 90 lines of Bash. This was called Dokku.

Dokku was a bit of a revelation. An expensive, top-tier platform valued in the hundreds of millions of dollars recreated with the right primitives in a weekend as a free Bash script...
  I want to have the primitives to create more software like Dokku--extremely simple, hackable alternatives to the world's best software. The resulting software isn't really the important part, though. The resulting building blocks are. Why use Figma if you could somewhat easily throw together your ideal figma?

Building simpler alternatives by disaggregating into new primitives expands the adjacent possibility space of what can be thrown together, while also simplifying and commoditizing what it takes to build great software. It actually is a sort of reversing of entropy! Repacking down our abstractions, engineering for generativity.

What I love is how "building blocks" brings to mind a sort of ideal vision we have of building software. Like LEGO, we snap together little components to realize something new. But this metaphor also points out how far that ideal is from reality. LEGO is fun. Building software really isn't most of the time. And I think trying to express the reality through this metaphor is helpful:

While ALL our software is made of many small bricks or components or modules, most come to us fully assembled with the bricks glued in place and painted over. We can't change it or even get a sense of how it was composed. Most apps come as an opaque "turn-key solution" to a specific problem because... that’s how you sell things.

Perhaps there’s nothing wrong with that, but we can at least try to find ways to build more software like LEGO kits, where the focus is on good, re-usable, re-combinable building blocks and showcasing what you can do with them. In the long term, I believe this is the way that provides the most value.
  A few years ago I realized that I need to build a single cohesive system that lets me express all these ideas, and really show the potential I've been talking about. I need to create my own LEGO system for software that would be an umbrella for all my work and let me build the way I want to build, while being at the very least a useful example to others...

So I'm announcing the Tractor System. It's not one project, but a system of projects building towards a malleable software environment and ecosystem. It is my take on a "Photoshop of software".

Tractor is a system of building blocks and pragmatic ways of building centered around simplicity and generativity. It’s not a new language, it helps glue languages together. It’s not just an app builder, it’s a systems playground. Much of the tool is made with itself, out of the same primitives you use in it. It's designed to be as simple and understandable as it can be, top to bottom, inside and out.

The aim is to blur the line between user and developer. The computer is a computer because it can be programmed. The rushed, over-productization of shrink-wrapped software, pandering to enterprise organizations and the computer illiterate, has created so much complication it would seem we NEED big companies to build our software. But we don't. I'm constantly surprised how simple things really are or can be. With Tractor, eventually we'll just have operators: people that can use the computer in its full potential to solve problems, create systems, play, learn, work, and have fun.
  Some say we live in a post PC era, but the original idea of the PC, the personal computer, is and will always be significant because it puts a magical form of leverage into the hands of individuals for their own use. But instead of this becoming more obvious with time, this generative power has been centralized and sterilized into "pop computing". Today we consume and are defined by software that benefits engines of capitalism more than engines of equity. You could say, instead of being able to program our own leverage, we are being programmed by those that can.

Luckily, underneath all the apps, social media, and enterprise "solutions", there is still an imagination compiler. This is what I focus on. This is what I'd love to dust off, polish up, and see where that takes us in the proper hands.

Even if that doesn't happen, I'd love to just get some better building blocks. Perhaps just find simpler, more enjoyable ways to build software.
  As an independent creator, I depend on people like you to help fund this work, and I'd love you to come along and help make the Tractor System a reality by becoming a sponsor.

---

Originally posted on progrium.com and YouTube

]]>https://progrium.xyz/articles/2021/making-programs-interact-using-qtalkhttps://progrium.xyz/articles/2021/making-programs-interact-using-qtalkTue, 24 Aug 2021 00:00:00 GMTqtalk-go is a versatile IPC/RPC library for GoToday I'm releasing a beta of qtalk-go, a versatile IPC/RPC library for Go. I've been using and iterating on it for 5 years to get it as simple and clear as possible.

// client.go
package main

import (
	"context"
	"log"

	"github.com/progrium/qtalk-go/codec"
	"github.com/progrium/qtalk-go/fn"
	"github.com/progrium/qtalk-go/talk"
)

func main() {
	ctx := context.Background()

	// use talk.Dial to get a client
	client, err := talk.Dial("tcp", "localhost:9999", codec.JSONCodec{})
	if err != nil {
		log.Fatal(err)
	}
	defer client.Close()

	// call Upper and print the string return value
	var ret string
	_, err = client.Call(ctx, "Upper", fn.Args{"hello world"}, &ret)
	if err != nil {
		log.Fatal(err)
	}
	log.Println(ret)

	// call Error and expect err to be the returned error
	_, err = client.Call(ctx, "Error", fn.Args{"user error"}, nil)
	log.Println(err)

	// Output:
	// HELLO WORLD
	// remote: user error [/Error]
}

qtalk is based on over a decade of building and cataloging approaches, patterns, anti-patterns, and best practices in network programming. My interest goes all the way back to high school when I first started playing with Winsock attempting to make massively multiplayer games. Then as a young web developer, pushing the limits of HTTP, discovering how to stream real-time to the browser years before Websocket was dreamed up. Then further abusing HTTP to model other protocols like DNS and IMAP. I pioneered distributed callbacks with webhooks, which got me working at early Twilio where I started going deep on scalable, highly-available messaging architectures. This led me into distributed systems: discovery, coordination, scheduling, etc. I've seen a lot.

I originally wanted to release qtalk with a paper describing all the significant choices to consider when building a stack like this: message framing, data formats, transports, security mechanisms, protocol flows, queuing, multiplexing, batching, layering, schemas, IDLs, symmetrical vs asymmetrical, stateful vs stateless, TCP vs UDP, etc. It would be a sort of guide for building your own stack. I'd still like to write that at some point, but this post will have to suffice for now.

qtalk makes no significant claims other than being the most bang for buck in simplicity and versatility. I've made a full walkthrough of various ways it can be used on the wiki, but I'll share a taste here.

Here is the server to the client code from above for you to try. Together they show qtalk being used in the simplest case, traditional RPC:

// server.go
package main

import (
	"fmt"
	"log"
	"net"
	"strings"

	"github.com/progrium/qtalk-go/codec"
	"github.com/progrium/qtalk-go/fn"
	"github.com/progrium/qtalk-go/rpc"
)

type service struct{}

func (svc *service) Upper(s string) string {
	return strings.ToUpper(s)
}

// methods can opt-in to receive the call as last argument.
// also, errors can be returned to be received as remote errors.
func (svc *service) Error(s string, c *rpc.Call) error {
	return fmt.Errorf("%s [%s]", s, c.Selector)
}

func main() {
	// create a tcp listener
	l, err := net.Listen("tcp", "localhost:9999")
	if err != nil {
		log.Fatal(err)
	}

	// setup a server using fn.HandlerFrom to
	// handle methods from the service type
	srv := &rpc.Server{
		Codec:   codec.JSONCodec{},
		Handler: fn.HandlerFrom(new(service)),
	}

	// serve until the listener closes
	srv.Serve(l)
}

Features

Some basic features of qtalk-go are:

• heavily net/http inspired API
• pluggable format codecs
• optional reflection handlers for funcs and methods
• works over any io.ReadWriteCloser, including STDIO
• easily portable to other languages

The more unique features of qtalk-go I want to talk about are:

• connection multiplexing
• bidirectional calling

Multiplexing Layer

The connection multiplexing layer is based on qmux, a subset of SSH that I've written about previously. It was designed to optionally be swapped out with QUIC as needed. Either way, everything in qtalk happens over flow-controlled channels, which can be used like embedded TCP streams. Whatever you do with qtalk, you can also tunnel other connections and protocols on the same connection.

RPC is just a layer on top, where each call gets its own channel. This makes request/reply correlation simple, streaming call input/output easy, and lets you hijack the call channel to do something else without interrupting other calls. You can start with an RPC call and then let it become a full-duplex bytestream pipe. Imagine a call that provisions a database and then becomes a client connection to it.

Bidirectional Calling

Bidirectional calling allows both the client and server to make and respond to calls. Decoupling the caller and responder roles from the connection topology lets you implement patterns like the worker pattern, where a worker connects to a coordinator and responds to its calls.

This also allows for various forms of callbacks in either direction. Not only do callbacks let you build more extensible services, but generally open up more ways for processes to talk to each other. Especially when combined with the other aspects of qtalk.

Imagine a TCP proxy with an API letting services register a callback whenever a connection comes through, and the callback includes a tee of the client bytestream letting this external service monitor and maybe close the connection when it sees something it doesn't like.

State Synchronization

State synchronization isn't a feature but a common pattern you can easily implement in a number of ways with qtalk. While many people think about pubsub with messaging, which you can also implement with qtalk, I've learned you usually actually want state synchronization instead. Below is a simple example.

Our server will have a list of usernames connected, which is our state. When a client connects, it calls Join to add its username to the list. This also registers the client to receive a callback passing the list of usernames whenever it changes. The client can then call Leave, or if it disconnects abruptly it will be unregistered with the next update.

// server.go
package main

import (
	"context"
	"log"
	"net"
	"sync"

	"github.com/progrium/qtalk-go/codec"
	"github.com/progrium/qtalk-go/fn"
	"github.com/progrium/qtalk-go/rpc"
)

// State contains a map of usernames to callers,
// which are used as a callback client to that user
type State struct {
	users sync.Map
}

// Users gets a list of usernames from the keys of the sync.Map
func (s *State) Users() (users []string) {
	s.users.Range(func(k, v interface{}) bool {
		users = append(users, k.(string))
		return true
	})
	return
}

// Join adds a username and caller using the injected rpc.Call
// value, then broadcasts the change
func (s *State) Join(username string, c *rpc.Call) {
	s.users.Store(username, c.Caller)
	s.broadcast()
}

// Leave removes the user from the sync.Map and broadcasts
func (s *State) Leave(username string) {
	s.users.Delete(username)
	s.broadcast()
}

// broadcast uses the rpc.Caller values to perform a callback
// with the "state" selector, passing the current list of
// usernames. any callers that return an error are added to
// gone and then removed with Leave
func (s *State) broadcast() {
	users := s.Users()
	var gone []string
	s.users.Range(func(k, v interface{}) bool {
		_, err := v.(rpc.Caller).Call(context.Background(), "state", users, nil)
		if err != nil {
			log.Println(k.(string), err)
			gone = append(gone, k.(string))
		}
	})
	for _, u := range gone {
		s.Leave(u)
	}
}

func main() {
	// create a tcp listener
	l, err := net.Listen("tcp", "localhost:9999")
	if err != nil {
		log.Fatal(err)
	}

	// setup a server using fn.HandlerFrom to
	// handle methods from the state value
	srv := &rpc.Server{
		Codec:   codec.JSONCodec{},
		Handler: fn.HandlerFrom(new(State)),
	}

	// serve until the listener closes
	srv.Serve(l)
}

The Call pointer that handlers can receive has a reference to a Caller, which is a client to make calls back to the caller, allowing callbacks.

Our client is straightforward. After setting up a connection and a handler to receive and display an updated username listing, we call Join with a username, wait for SIGINT, and call Leave before exiting.

// client.go
package main

import (
	"context"
	"flag"
	"fmt"
	"log"
	"os"
	"os/signal"

	"github.com/progrium/qtalk-go/codec"
	"github.com/progrium/qtalk-go/fn"
	"github.com/progrium/qtalk-go/rpc"
	"github.com/progrium/qtalk-go/talk"
)

func fatal(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

func main() {
	flag.Parse()

	// establish connection to server
	client, err := talk.Dial("tcp", "localhost:9999", codec.JSONCodec{})
	fatal(err)

	// state callback handler that redraws the user list
	client.Handle("state", rpc.HandlerFunc(func(r rpc.Responder, c *rpc.Call) {
		var users interface{}
		if err := c.Receive(&users); err != nil {
			log.Println(err)
			return
		}
		// the nonsense are terminal escape codes
		// to return to the last line and clear it
		fmt.Println("\u001B[1A\u001B[K", users)
	}))

	// respond to incoming calls
	go client.Respond()

	// call Join passing a username from arguments
	_, err = client.Call(context.Background(), "Join", fn.Args{flag.Arg(0)}, nil)
	fatal(err)

	// wait until we get SIGINT
	ch := make(chan os.Signal)
	signal.Notify(ch, os.Interrupt)
	<-ch

	// call Leave before finishing
	_, err = client.Call(context.Background(), "Leave", fn.Args{flag.Arg(0)}, nil)
	fatal(err)
}

See the Examples wiki page for more code examples, including tunnels and proxies, selector routing, and streaming responses.

Roadmap

I'm trying to get to a 1.0 for qtalk-go, so I'd like more people to use and review its code. I also haven't actually gotten around to putting in QUIC as a usable base layer, which I think should be in a 1.0 release. It's in the name, qtalk was started with QUIC in mind. Not only will QUIC improve performance, resolve head of line blocking, and eventually be native to browsers, but being UDP-based means that hole punching can be used to establish peer-to-peer qtalk connections. I'd like to one day be able to use qtalk directly between machines behind NAT.

Meanwhile, I'm wrapping up a JavaScript implementation (in TypeScript) to officially release soon. I have the start of a Python implementation I could use help with, and I'd love to have a C# implementation.

That's it for now. A big thanks to my sponsors for making this happen and thanks to you for reading!

---

For more great posts like this sent directly to your inbox and to find out what all I'm up to, get on the list at progrium.com ✌️

]]>https://progrium.xyz/articles/2021/towards-the-personal-potential-of-softwarehttps://progrium.xyz/articles/2021/towards-the-personal-potential-of-softwareMon, 05 Jul 2021 00:00:00 GMTA midyear review and preview of what's coming.Since starting work under Progrium Technology Company, I've had a single objective: to build a system called Tractor. It's what I've been slowly approaching in my dense 22 year career as an independent programmer. The Tractor System is still hard for me to describe at this stage, but its goal is to make building production-grade personal software systems like building with Legos: fast and fun.

After a difficult year heads-down in 2020, I started posting here in January with the intention of sharing and officially releasing components I've been working on that are building up to Tractor. Besides warming up for when I'd eventually talk about Tractor, I wanted people to see how I make independently useful building blocks towards an ideal. That's a core part of the Tractor philosophy.

Let's review what I've shared so far, and then set up what's coming in the next 6 months.

The first big release of the year was the macdriver project, which got a significant response on Hacker News. It gives us Objective-C and Apple framework bindings for Go, letting you build simple Apple apps entirely from Go. That wasn't possible before, so it was a little exciting, but it was early. The native Go APIs included for commonly used Foundation and Cocoa classes were far from complete. They could still be used if you knew what you were doing, but with an ideal of total coverage of Apple frameworks, that wouldn't be enough. There were also unresolved issues just deciding how to best manage memory and pointers, which I knew would fall on me to figure out and take some time.

Luckily I've since gotten a colleague involved in the company, and he's been helping push macdriver towards a real beta. Part of this was made possible by a project I spun out of macdriver and posted about, though again in a very early state. That project was macschema.

With the macschema toolchain, you can generate API schemas about any Apple framework, class, function, etc based on their documentation and header declarations. This is useful for us in generating framework bindings in macdriver, but it would be useful for any project doing similar work, such as bindings for other languages. This sort of approach will play a big part in how Tractor will integrate with "whatever we want" down the line.

Along the way, I released a few demos for macdriver to make clear the possibilities and provide reference examples. One of them I spun off into a standalone project called topframe, which is also serving as a test bed for how we are doing cross-platform support. More on that in a moment.

Unrelated to macdriver, I released a protocol called qmux with a post explaining why this (but really any) muxing protocol, including and especially QUIC, is such a great primitive for network programming. The project came with two implementations, Go and TypeScript, with more on the way. The post about it was the first explainer article I'd done in a while and people liked it. This was important because the idea behind it was really more valuable than the protocol itself, but regardless we still need implementations of it to exist. To show the idea in action, I built a 130 line version of Ngrok, which turned into a great post walking through how it works.

That brings us to today, half way through 2021. Let's talk about what's coming.

More exciting than qmux is what we built on top of it, which is called qtalk. This is my meticulously designed, re-written-several-times network/IPC programming stack. While not that different from just JSON-RPC with pluggable codecs, its two unique features are callbacks and streams. Callbacks means its bi-directional, either side can expose or call methods. Just that alone is something rare in existing RPC stacks, but necessary for callbacks. And while streaming RPC is not new, our streams are full virtual connections, so you can stream more RPC results, arbitrary byte streams, or tunnel something else like a database connection. I'll share more about the possibilities when it's released.

So in the second half of 2021, we'll see qtalk, we'll see a major update to macdriver, and then with those two parts we can release shelldriver. This is a cross platform API to platform specific resources like windows, dialogs, menus, etc. It's about native GUI shell integration, it's not about having every native UI component. In fact, for app UI it really focuses on windows and webviews, a la Electron. Unlike Electron, it can be used from any language that has a qmux and qtalk implementation. The result should be a simple "Electron but as a library" that you can use from Go and other languages.

Once shelldriver is out I will be able to finally start talking about the first real piece of the Tractor system: Tractor Toolkit. I hope to have at least a public demo by the end of the year. The Toolkit is already being shown to friends and sponsors. The work I've shared so far is just the tip of the iceberg.

Which reminds me ...

I really, really have to thank my sponsors. As things are ramping up, I'm starting to work with more people and this is all self funded. Not only does sponsorship support all this open source work, but sponsors get early access to Tractor Toolkit and see more of what I'm up to.

Also to you, thanks for reading and following along. I'll be back soon with more releases.

For more posts like this sent directly to your inbox and to stay on top of what I'm up to, get on the list at progrium.com ✌️

]]>https://progrium.xyz/articles/2021/building-your-own-ngrok-in-130-lineshttps://progrium.xyz/articles/2021/building-your-own-ngrok-in-130-linesThu, 27 May 2021 00:00:00 GMTRemote port forwarding as a serviceRunning a local development server for your app is pretty common, but what if you wanted somebody else to access it? Maybe for a demo, or maybe to debug webhook integrations. If you've ever used Ngrok (or perhaps the original localtunnel), you know what I'm talking about.

Today we're going to make a system that emulates the core functionality of Ngrok, exposing a localhost HTTP server via a public URL.

Just to make sure we're clear on how this all works, let me explain the problem. Normally when you run a server on your computer it can only be accessed by other computers on the same local network. However, many development servers will bind to 127.0.0.1, or localhost, which means only a client or browser on your computer can access it. Either way, there are times when you might want another party to access it temporarily, but it's simply not addressable from the public Internet.

In the olden days, this could only be achieved if you had access to a server running SSH. You could configure SSH to have publicly addressable ports on that server be "forwarded" to a local port on your computer over an SSH connection. This is called remote port forwarding or sometimes SSH tunneling. Assuming SSH is configured properly, the command to set this up looks like this:

$ ssh -R 8080:localhost:3000 your-ssh-server.com

This would allow somebody to point their browser to your-ssh-server.com:8080 and get your localhost server running on port 3000 as long as this command was running. Around 2009, I noticed how useful this is (especially in the context of webhooks), but was annoyed you had to have an SSH server properly configured for it before you could even do it. Even then, I always found the SSH command difficult to remember how to use for forwarding. I envisioned a command as simple as:

$ localtunnel 3000

So I set about making this happen. My first attempt at building a system with Twisted Python failed because I didn't quite grok multiplexing. So I ended up wrapping SSH. I wrote a localtunnel client in Ruby that behind the scenes would just run the above SSH command using a server I had set up. The only difference being that I didn't give you a port like 8080 to connect to, I wanted you to get a generated subdomain. Instead of localtunnel.com:8080, you'd get wxyz.localtunnel.com, which was unique to your tunnel as it was intended as a public service.

To achieve that, I just ran a Twisted web server with the SSH server that would figure out which port to use based on the subdomain and proxy to it. This would only work with HTTP because the idea of virtual hosts was actually invented for HTTP. Normally when you get a TCP connection you have no idea what hostname is used.

So I had something like this:

And it worked! It became rather popular. However, due to the slight complexity (Ruby+Python+OpenSSH) and the fact I was too busy to maintain the service, people started cloning it. One jackass even called his clone "localtunnel".

Another jackass, who was less of a jackass and more my friend and coworker, decided to clone it as a way to learn Go. He called it Ngrok and decided to commercialize it. This bothered me for a number of reasons I won't get into, but eventually decided it was fine since he really improved on the idea and would at least listen to me if I had anything to say about it.

Anyway, today we're exploring a toy recreation of this system that we'll call groktunnel. It removes SSH from the setup and using two libraries lets us build the whole thing in about 130 lines of Go. You'd still need a server to run this on, so it's not as convenient as a hosted service, but this is more for educational purposes. After all, groktunnel is actually a demo for qmux, one of the libraries I mentioned. Let me explain these libraries.

What qmux does is give us a subset of the SSH protocol to multiplex many connections over a single connection. It was the missing piece of my original Twisted prototype. You can read more about how there aren't a lot of these protocols (but perhaps with QUIC are also the future of the Internet) in my previous post.

The other library is actually by the author of Ngrok. Called go-vhost, it helps with the problem of virtual hosts. Ideally when a connection comes in to be forwarded down the tunnel we can hand it off wholesale. This way the other end can just start reading off the connection as if received directly. But in order to figure out the hostname used, we have to start reading off the connection up to the Host header. Then we have to hand it off prepended with what was read. What Alan wrote is an abstraction that gives you a virtual listener that lets you accept new connections for a particular virtual host and get the connection as if it hadn't been read yet. A lot of this is due to the interface based approach of the Go standard library.

This brings us to the main utility function we'll be writing for this system. A TCP connection established between two peers cannot (easily) be handed off between programs, let alone hosts. We don't literally "forward" the connection, we sit in the middle and relay the bytes. This is how every proxy, load balancer, gateway, or tunnel works. It sort of welds two sockets together. I call this socket joining. And if you can avoid doing any protocol level stuff, building a proxy just becomes opening a connection to the backend when a connection comes in and then joining them.

Since Go models connections using the interfaces of the io package, we can use io.Copy to simplify reading off bytes from one connection into another until an EOF. A two-way copy like this that closes when finished looks like this:

func join(a io.ReadWriteCloser, b io.ReadWriteCloser) {
	go io.Copy(b, a)
	io.Copy(a, b)
	a.Close()
	b.Close()
}

This minimal approach benefits from setting a deadline on the connections in case one end doesn't hang up. You can see we copy bytes from a to b in the background, then wait for copying bytes from b to a before making sure both sides are closed once b gets an EOF.

We'll also define a helper to generate random subdomains. In a production service this is a big deal. You'd want an incredibly large space of possible subdomains, but here we generate 10 random alphanumeric characters using crypto/rand:

func newSubdomain() string {
	b := make([]byte, 10)
	if _, err := rand.Read(b); err != nil {
		panic(err)
	}
	letters := []rune("abcdefghijklmnopqrstuvwxyz1234567890")
	r := make([]rune, 10)
	for i := range r {
		r[i] = letters[int(b[i])*len(letters)/256]
	}
	return string(r) + "."
}

Another quick utility to define that's common in simple Go programs is a fatal function. This helps simplify all the error checking.

func fatal(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

Now we can get started in our main() which will contain both client and server modes. However, first we'll define some flags useful to both:

var port = flag.String("p", "9999", "server port to use")
var host = flag.String("h", "vcap.me", "server hostname to use")
var addr = flag.String("b", "127.0.0.1", "ip to bind [server only]")
flag.Parse()

The default hostname of vcap.me is a way to make the server easy to run locally. Since localhost doesn't allow subdomains, but any domain can resolve to 127.0.0.1, there are a few public domains that have wildcard subdomains all resolving to 127.0.0.1. If we ran this on a real server, we'd need a domain with wildcard subdomains resolving to the server IP, and we'd probably use port 80 instead of 9999.

Now we'll start with the client, which is quite simple. It's activated when there is an argument specifying the local port to expose publicly.

// client usage: groktunnel [-h=<server hostname>] <local port>
if flag.Arg(0) != "" {
	conn, err := net.Dial("tcp", net.JoinHostPort(*host, *port))
	fatal(err)
	client := httputil.NewClientConn(conn, bufio.NewReader(conn))
	req, err := http.NewRequest("GET", "/", nil)
	req.Host = net.JoinHostPort(*host, *port)
	fatal(err)
	client.Write(req)
	resp, _ := client.Read(req)
	fmt.Printf("port %s http available at:\n", flag.Arg(0))
	fmt.Printf("http://%s\n", resp.Header.Get("X-Public-Host"))
	c, _ := client.Hijack()
	sess := session.New(c)
	defer sess.Close()
	for {
		ch, err := sess.Accept()
		fatal(err)
		conn, err := net.Dial("tcp", "127.0.0.1:"+flag.Arg(0))
		fatal(err)
		go join(conn, ch)
	}
}

Here we set up the tunnel over HTTP since we'll be serving HTTP anyway, and we need a way to tell the client what the generated public host is to display a URL to the user. You can see we get the hostname from a response header.

We're manually creating an HTTP request with httputil.NewClientConn because it lets us hijack the connection. This simply means it stops processing as HTTP and treats it like a raw TCP connection. Alternatively we could upgrade to WebSocket for this, but this simply works just as well with fewer dependencies.

After hijacking we create a qmux session to start multiplexing virtual connections, or channels, over this connection. This is the tunnel. We don't open channels, we just sit in a loop waiting for incoming channels and assume they want to be joined with a connection to our localhost server. That's it.

Now the server. First we set everything up using our flags, including a TCP listener, which we wrap in a virtual host muxer. The meat of our server is in serve() but that runs asynchronously as we listen for vhost muxing errors until shutdown.

// server usage: groktunnel [-h=<hostname>] [-b=<bind ip>]
l, err := net.Listen("tcp", net.JoinHostPort(*addr, *port))
fatal(err)
defer l.Close()
vmux, err := vhost.NewHTTPMuxer(l, 1*time.Second)
fatal(err)

go serve(vmux, *host, *port)

log.Printf("groktunnel server [%s] ready!\n", *host)
for {
	conn, err := vmux.NextError()
	fmt.Println(err)
	if conn != nil {
		conn.Close()
	}
}

Now the heart of this system in serve():

func serve(vmux *vhost.HTTPMuxer, host, port string) {
	ml, err := vmux.Listen(net.JoinHostPort(host, port))
	fatal(err)
	srv := &http.Server{Handler: http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		publicHost := strings.TrimSuffix(net.JoinHostPort(newSubdomain()+host, port), ":80")
		pl, err := vmux.Listen(publicHost)
		fatal(err)
		w.Header().Add("X-Public-Host", publicHost)
		w.Header().Add("Connection", "close")
		w.WriteHeader(http.StatusOK)
		conn, _, _ := w.(http.Hijacker).Hijack()
		sess := session.New(conn)
		defer sess.Close()
		log.Printf("%s: start session", publicHost)
		go func() {
			for {
				conn, err := pl.Accept()
				if err != nil {
					log.Println(err)
					return
				}
				ch, err := sess.Open(context.Background())
				if err != nil {
					log.Println(err)
					return
				}
				go join(ch, conn)
			}
		}()
		sess.Wait()
		log.Printf("%s: end session", publicHost)
	})}
	srv.Serve(ml)
}

Here we create a new listener using the vhost muxer passed in to listen for connections on the naked domain (vcap.me). Then we make an HTTP handler that sets up a new tunnel. First it generates a new hostname and tells our vhost muxer to start listening on it. We'll hold on to that listener.

Then we start writing headers. Remember this is how we tell the client what our generated hostname is. We also use Connection: close to tell the other end we're going to be done and normally hang up, but after we flush the headers with a 200 OK status, we hijack the connection like we know is going to happen on the other end. We make a new qmux session and start a new goroutine.

This goroutine handles connections coming in from our new vhost listener. All this does is sit in a loop accepting new connections, then using our qmux session opens a channel, then joins the new connection with that channel.

Back in our HTTP handler, we call Wait() on our session, which will just block until the connection breaks. This handler has been put into an HTTP server instance set to serve on our original vhost listener for the naked vcap.me domain.

Now we can try it out. You can follow the instructions in the groktunnel README to see it in action.

This little program gives us a client and server that recreates the original localtunnel system. It showcases the use case of tunneling for connection multiplexing, but also how easily you can wire things up in Go using mostly the standard library.

If you wanted to tunnel normal TCP connections, not HTTP connections, this program gets even simpler. However, that's an exercise for the reader. You can see the full groktunnel source in the demos directory of the qmux project. Hope you were inspired to build something cool!

For more posts like this sent directly to your inbox and to find out what all I'm up to, get on the list at progrium.com ✌️

]]>https://progrium.xyz/articles/2021/the-history-and-future-of-socket-level-multiplexinghttps://progrium.xyz/articles/2021/the-history-and-future-of-socket-level-multiplexingThu, 06 May 2021 00:00:00 GMTTowards a socket API with builtin stream muxingMultiplexing (sometimes called muxing) has a long history of innovation in telecommunications. In 1875, Edison figured out how to transmit four separate telegraph lines over a single wire. In 1910, multiplexing technology was brought to telephony. Within 30 years the Bell System could multiplex 2,400 channels over a single cable. Then in the 1960s we got packet switching, a kind of unlimited multiplexing, which led to computer networking and Internet protocols.

I think of layered protocols and packet switching as "virtualized" multiplexing. Every layer provides a new medium or virtual transport for more protocols to operate, all over the same physical medium. Ethernet allows IPX, IP, and other packets. IP then allows UDP, TCP, and other packets. TCP allows HTTP, SSH, and so on.

The layers nearly end at TCP and UDP because these two protocols provide most of what any programmer would need to make two programs talk. This was more or less set in stone in the 1980s when Unix introduced the socket API, which has been the de facto standard interface for network programming ever since.

This API layer is where you get "application protocols" like FTP, Telnet, DNS, and so many others. TCP, UDP, and the layers below were typically concerned with just getting bytes from one point to another. Application protocols introduced more specific interactions like request-reply, such as with HTTP. Some, like SSH and DNS, became rather multifaceted protocols to serve their domain with many different flows and message types. Rarely would you find an application protocol multiplexed to provide a generic transport for another.

An early exception was SSH. One of the reasons SSH fascinates me is that it was designed for many uses and breaks down into its own set of layers and internal protocols, almost mirroring TLS, TCP, and UDP.

First, SSH establishes an authenticated and encrypted line, similar to TLS. Then you have two very generic ways to communicate over this line. You can send "request" messages that may or may not get a response, similar to use cases of UDP. Or you can establish "channels" that were bidirectional streams just like TCP.

From there, SSH defines how to use these requests and channels as primitives to set up remote shells. But besides that, you could also forward TCP connections, which worked by bridging a TCP socket with an SSH channel, "tunneling" the TCP connection through the SSH connection.

If you've ever used Ngrok to open a public endpoint to a localhost server, you may not know it was one of several clones of a tool I made in 2010 called localtunnel. The original localtunnel was just a wrapper around SSH, literally using OpenSSH on the server side.

So SSH was one of the earliest application level protocols designed to further multiplex its transport protocol. The next protocol to multiplex at the application/socket level was HTTP/2. It was introduced in 2015, which is still quite recent, but 20 years after SSH.

HTTP has become a new de facto API, like Unix sockets did. What was originally used to just download files into the browser has been co-opted and repurposed. As people started building more and more complex applications in the browser and with the emergence of web APIs, HTTP evolved into a sort of "do everything" protocol.

In 2012, Google started an effort to design a more efficient HTTP with what they called SPDY, which they were able to test end-to-end in Chrome without people even knowing. After proving its improved performance and cost saving benefits, SPDY basically became HTTP/2. They did a number of things to make it more efficient, but the core idea was multiplexing.

HTTP/2 was more or less HTTP, but over newly introduced multiplexed "streams" similar to normal TCP connections. This makes them roughly the equivalent of SSH "channels". At least at one point, Ngrok used a subset of HTTP/2 as their muxing protocol the same way localtunnel used SSH.

Both HTTP/2 and SSH do more than multiplexing, and usually aren't thought of specifically for muxing. Most library implementations don't expose the lower level multiplexing primitives. SSH at least has clear separation in its specification, where multiplexed channels are a specific and independent layer of the protocol architecture.

After spending so much time with these multiplexing protocols, I noticed something. Most other application protocols also did something similar to multiplexing without realizing.

For example, any request-reply protocol that allows more than one request over a connection is "multiplexing" each request and reply pair. A request and response will often get a shared ID so they can be correlated as other requests and responses can be sent in the meantime. This is the heart of any muxing protocol!

I've found implementing RPC or any request-reply over a muxing API not only simplifies the protocol, but opens up the ability to also stream or tunnel other protocols on the same connection. You could "attach" a tunneled database connection with an RPC response.

Working with muxed streams over a single connection gives me a similar feeling to discovering green threads for the first time. Suddenly, streams between the same two points are so cheap and lightweight you can think about using them in totally different ways.

I now believe a socket API with muxing builtin is a better base layer and API for designing application protocols. I was validated when Google pushed multiplexing into QUIC.

If Google could redesign HTTP, why not also TCP+TLS? Their experience with SPDY started them down a path to reduce the overhead of TLS and TCP session establishment even more by creating a more efficient kind of encrypted TCP over UDP. Built into this layer is stream muxing that would not suffer from head of line blocking.

It's taken nearly 10 years for QUIC to be refined and adopted in the wild and we're basically there. There's even a new browser API in the works called WebTransport.

In the meantime, I started designing protocols on the assumption of having a muxing layer directly beneath. Eventually, that would be QUIC, but as a stopgap, what would it be? Remember, there haven't been any standard TCP muxing protocols.

I considered what Ngrok did with a stripped down HTTP/2, but then I realized... SSH has been around a lot longer. SSH is known to work on nearly every platform and is trusted by basically everybody. Best of all, the SSH muxing layer is simpler and more straightforward than HTTP/2.

So I literally took the muxing chapter of the SSH spec, cut out a couple of SSH specific identifiers, stripped it down to just the channels, and I've been using it for the past 5 years. It's called qmux.

The great thing is that even with QUIC available, qmux can be used over any streaming transport and provide the same foundation for higher level protocols built on muxing. Whether the transport is WebSocket or stdio, as long as it can reliably transport ordered byte streams, qmux drops right in. Protocols built on it will translate easily to QUIC.

I'm only releasing this now because it might not have made sense without all this context or more awareness of QUIC. The banality of a brand new subset of a 25 year old protocol does not escape me, but it's the shift in perspective that brings potential. In that way it's similar to webhooks!

If nothing else, I hope people use this story as inspiration to think about using multiplexing to design better, simpler application protocols. Even if for some reason QUIC isn't the future, you now have qmux!

For more great posts like this sent directly to your inbox and to find out what all I'm up to, get on the list at progrium.com ✌️

]]>https://progrium.xyz/articles/2021/apple-api-schemas-for-code-generation-and-morehttps://progrium.xyz/articles/2021/apple-api-schemas-for-code-generation-and-moreFri, 16 Apr 2021 00:00:00 GMTWhy I love API schemas and built a tool to generate schemas from Apple documentation.Schemas are pretty dope. My appreciation for API schemas as a toolmaker comes from a desire to build tools that can integrate with many systems, and open new avenues for people to use APIs.

For most, the use case that comes to mind for API schemas is generating documentation. Translating a machine-readable schema into a human friendly representation is a scalable, automated way to maintain solid reference documentation. Although many systems exist to generate documentation directly from code, having an intermediary representation is useful for other use cases.

Code generation is the other big one. It's common for companies like Google and Amazon to generate client libraries across languages for their many web APIs. Even some libraries for GitHub's API are generated because their API has gotten so big it would be untenable otherwise. This is particularly important in the open source community where maintainer time is a scarce resource. Schemas save a lot of time, but it's still pretty uncommon to find a schema for any given web API.

Even less common are schemas for non-web APIs. In 2016, Electron started releasing a JSON schema of all their APIs that allowed me to build a bridge to use Electron APIs from Go. I had that prototype in mind when I started the macdriver project that was released a couple months ago. Right now we're manually wrapping Apple framework classes with Go types so you can write native Apple platform applications that look like this:

These kinds of bindings used to be more supported by Apple as evidenced by bridgesupport files, which were XML schemas of their APIs. However, they aren't very well supported any longer. Moreover, bridgesupport files were more like headers and didn't provide much in the form of documentation.

So if we wanted to generate the APIs in macdriver, we could parse Objective-C headers, or maybe try to use what's available in bridgesupport files. But we'd also like to generate human readable descriptions and ideally link to official docs.

If schemas are used to generate documentation, there's no reason you can't generate schemas from documentation.

So I built a tool that could parse Apple documentation into schemas. Since their docs also have the declarations for everything, we don't even need to use header files. The result is a nice JSON document describing their classes and other entities like this:

From this we can start to automate the generation of wrapper types in macdriver, but as a standalone toolchain, the schemas it produces can be used to write bindings in languages other than Go as well.

Schemas like this can also be used in developer tooling for all sorts of things like autocompletion, or help create visual programming interfaces.

Today most "no-code" tools need to support integrations to stay competitive, but they're all writing and maintaining their own integrations. In fact, every application with user programmable integrations can benefit from schemas like this. So I'd like to see more.

If this is interesting, take a look at the tool I built, macschema, which is open source on GitHub. Right now it's just the toolchain, but in theory we could generate schemas for all of Apple's APIs and put them in a repo in such a way that can be updated by a combination of macschema and user contributed patches. I'll let somebody else do that though.

Thanks for reading, think about schemas more, and consider sponsoring my work!

]]>https://progrium.xyz/articles/2021/customize-your-computer-screen-with-htmlhttps://progrium.xyz/articles/2021/customize-your-computer-screen-with-htmlTue, 30 Mar 2021 00:00:00 GMTA local webpage screen overlay for customizing your computing experienceAnybody that knows how to mess around with HTML can now mess around with their desktop computing experience. Topframe is an open source tool that lets you customize your desktop screen using HTML/CSS/JavaScript.

It started as a demo for macdriver, but over the last week or so I've been making a standalone version. 90% of that time was playing around with it while trying to make demos.

Could this bring back the computing aesthetic of personalized MySpace pages and Geocities sites all over again? No?? ...maybe? Well, either way, I'm happy to share a tool for people to experiment once again.

Topframe is a fullscreen overlay that displays a special webpage served from your home directory. Editing the page source and hitting save will magically update your screen with nearly anything you can do in the browser.

Yes, it has Tailwind built-in, as well as trusty old jQuery. However, you can drop anything you want in the topframe directory to use. Or hotlink, why not? This isn't WEBSCALE, it's just on your computer. Your personal computer.

Getting Started with Topframe

Topframe is still very early. The whole project is less than 400 lines of Go. You'll also need a Mac, but it'll be cross-platform by 1.0.

Refer to the README for more details, but installing with Homebrew is as easy as:

$ brew install progrium/taps/topframe

Then you can just run topframe and it'll run the overlay webpage with some crazy demo content. SOrry. The important thing to notice is the new menu bar icon 🔲 and menu:

This is where you actually control Topframe, but there's not much to it. You can enable/disable (hide), make it interactive (by default mouse events just pass through), and quickly get to the source directory.

Clicking Show Source and opening the index.html file with your favorite editor, you can just wipe out all that garbage I put on your screen. HOWEVER, you can also look through it and see what you can find before blowing it away. There might be some cool ideas or features tucked away in there.

What could this be used for?

As the demo attempts to communicate, there is a LOT you can do with Topframe. For those streaming on Twitch (like me), you can use this as an alternative to doing overlays in OBS. Or, with some resourcefulness, you could make skins for all your windows. If anybody remembers the OS X Dashboard that they removed, you could make your own (better) version of that.

Or you could make some kind of screen Tamagotchi like Dogz from the 90s:

The only real limit is ... yourself.

What Next

Again, this is an early project, but it's also a mostly contributor-driven project. I'm going to keep focusing on tools to make stuff like this (including what'll make it cross-platform), so it'll be up to you all to really make this a real thing. For example, it could probably really use a global keyboard shortcut to toggle hiding or interactivity.

If you have questions, I'm watching the GitHub discussions so feel free to post there. Otherwise, have fun and consider sponsoring my work!

]]>https://progrium.xyz/articles/2021/use-mac-apis-and-build-mac-apps-with-gohttps://progrium.xyz/articles/2021/use-mac-apis-and-build-mac-apps-with-goTue, 02 Feb 2021 00:00:00 GMTAn Objective-C bridge for Go with bindings to common Mac APIsIf you work with Apple devices and you're a Go programmer, or are thinking about learning Go, we just got some new powers that are pretty cool. Today I'm releasing an alpha of macdriver, an Objective-C bridge for Go with bindings to common Mac APIs.

Those that were paying attention to my last post about Large Type in 80 lines of Go might have already found macdriver. The largetype program is one of the example projects for macdriver.

Another example in the macdriver repo is this little menu bar extra (aka systray or status icon) that gives you a Pomodoro timer. This little program is also less than 80 lines of Go using macdriver.

Lastly, if you've been playing with the Go 1.16 beta, or can wait for its imminent release, there is also an example that gives you a full screen webview with a transparent background, letting you draw overlays and such on your screen using web tech. Twitch streamers might be into that one. Pretty cool, yea?

Years in the making

The motivation for macdriver was just making applications in Go that use Mac system APIs like Cocoa for making GUI apps. That said, most of my work now uses web tech for user interfaces, a la Electron.

Last year I started an alternative Electron stack in Go using the webview project, which uses your platform's native browser engine and makes a window with a webview. While absolutely simple and cross platform, the project has not been prioritizing any platform specific functionality.

On top of this, if you want to also make a little menu applet like the Pomodoro timer in the same program (not... uncommon), you can find another standalone cross platform Go library for systray stuff, but these one-off libraries often suffer from being incomplete APIs. More importantly they are not composable! You can't use the systray library with the webview library because they both presume to own the main Cocoa thread.

So as usual, I was frustrated there wasn't a broader thought out, more general solution. What do I do if I wanted to use some other Mac API? Wait for another incomplete, non-composable project? No, we build our own future here.

I've actually been working on this (and so many other things) for a few years. I even found an Objective-C bridge for Go by a Danish fellow named Mikkel from 2013, except... it didn't work.

This whole thing should be straightforward. The Objective-C runtime is actually a C library called libobjc. So in theory you should be able to use cgo to call libobjc and work with Objective-C objects from Go like any other C library.

Unfortunately, the method calling functions in libobjc, to be generic to any method call, are variadic (takes arbitrary number of arguments). Cgo doesn't support variadic function calls! So Mikkel did an assembly hack to make nearly any variadic C function callable. Amazing! But since 2013, changes to the Go runtime broke this and he had long since moved on to other projects.

With some determination I got the variadic hack working again. I even forgot I posted about this on dev.to in 2018. There were still more things to figure out, but I shelved it until early last year when I needed to solve this webview and systray composability problem. I brought in the abandoned objc bridge package, the variadic package I fixed, and started making native Go wrappers for some of the more common Mac APIs. The result was macdriver!

Benefits

Now you can make fast, machine native binaries in Go that do Mac things. And I don't just mean make MacOS apps. There are a lot of APIs that just open up new possibilities. For example, I'm excited to use the Core ML APIs for working with the Neural Engine on the new M1 devices.

Another benefit is just being able to quickly make Mac apps using the Go build toolchain. No more Xcode! Just go build.

What Next

I'm excited to see what people do with macdriver. There's lots more information on using it in the README, but I am still working on docs. At this point it should mostly be self explanatory, but feel free to ask questions in the forum.

If you're interested in exploring new possibilities with me, I've been meaning to try using macdriver for an Apple Watch app. There was an iOS proof of concept in the original bridge package by Mikkel, but I haven't tried it. Assuming it works (which may involve Xcode or other developer tools), it seems plausible it would also work for the Apple Watch or even... the Apple TV.

Thanks for reading, have fun, and consider sponsoring my work!

]]>https://progrium.xyz/articles/2021/large-type-legacy-celebrated-in-80-line-go-programhttps://progrium.xyz/articles/2021/large-type-legacy-celebrated-in-80-line-go-programFri, 15 Jan 2021 00:00:00 GMTA tiny, standalone largetype command-line utilityThere is a little known feature in the MacOS Contacts app where if you right-click a phone number, you get a Large Type option. Large Type fills your screen with the phone number in the largest font size possible, allowing you or anybody else to easily see the number from far away. Nifty!

As far as I know, this feature has been in every version of OS X since its release back in 2000. Contacts was known as Address Book back then. The builtin calculator also had a Large Type view, but it seems it has since been removed. The feature might even exist in NeXTSTEP apps.

Presumably inspired by OS X Address Book, Large Type was also added to the legendary Quicksilver app launcher from the mid-2000's. Now, using Quicksilver, you could quickly type or paste in any text and display it in Large Type. More nifty!

Since I was involved in the SHDH hacker parties, this feature got a lot of use as a way to share a piece of information across a noisy, crowded room. Many users of Quicksilver also had a shell script that would run AppleScript to get Quicksilver to display text from the command line. Now it could be scripted!

It became the most memorable feature of my Quicksilver experience.

Large Type has the same sort of charm as the say command. It's nifty, maybe unnecessary, but really simple, there when you need it, and is kind of just a cool thing to play around with and show off.

The feature almost died with Quicksilver. Unsurprisingly, the modern Quicksilver equivalent, Alfred, also has Large Type. Except. I don't really want Alfred or Quicksilver installed just for Large Type now. It would be nice to have a standalone utility!

Thanks to a project I'm sharing in a couple weeks, I basically achieved perfection for such a utility. A standalone largetype command, in a single native code binary smaller than 4MB. And in under 80 lines of Go code:

{% gist https://gist.github.com/progrium/cb967815b3ed21a77e65a4ad9b1dbdf6 file=largetype.go %}

Building

Note: For now, Apple Silicon users need to be set up for x86 mode

First, download Go or brew install go. Then, put largetype.go in a directory called largetype and from there run:

$ go mod init largetype
$ go build

This will make you a 4MB binary called largetype. Now you can run it:

$ ./largetype "Hello world"

]]>https://progrium.xyz/articles/2021/making-the-world-more-programmablehttps://progrium.xyz/articles/2021/making-the-world-more-programmableMon, 04 Jan 2021 00:00:00 GMTA draft talk I made 5 years ago trying to communicate my missionI'm starting this devlog up again to talk about some of the work I've been doing quietly for the past few years. It will mostly be project reveals, which should be exciting.

To kick things off I wanted to set the tone a little. I haven't given any serious talks in a while, and this is one of the last ones I put together. It's actually a rehearsal recording that's been private for 5 years UNTIL NOW of me framing my work around this mission: making the world more programmable.

This was one attempt at a clear narrative for what I'm trying to achieve, but I didn't feel it was good enough. Five years is a good while, though, and looking back it is a decent introduction for what's to come.

If I named this talk today I might instead call it "Making the future malleable" or something.

]]>