Writing
Writing
Notes from building Railyard, deploy software that runs your whole app on servers you own: how a platform works, the bugs that taught me something, and the design calls behind it.
Start here
Why I'm building deploy software instead of a host
What Railyard is and why it deploys to servers you own.
Read →Why my Rails dashboard took two seconds to load
A Rails performance postmortem, from 1.8 s to 15 ms.
Read →One door for every deploy
Moving deploy rules out of controllers and into the domain model.
Read →All posts
-
A deadlock Postgres can't see
A Rails migration waited forever on a lock held by its own transaction, through a second connection pool. Why Postgres' deadlock detector can't catch that, how to spot it in pg_stat_activity, and why the logs were empty.
-
Why parallel deploys waited on each other
Several Railyard deploys at once queued behind each other and froze the dashboard. The shared resources one kind of work could fill, how I split them, and the two bugs the fixes created.
-
One door for every deploy
Railyard had nineteen places that could start a deploy, each with its own copy of the rules. How I moved every one of them behind a single model method, and the table-driven test that keeps them there.
-
Why my Rails dashboard took two seconds to load
A performance postmortem on Railyard's Rails and Hotwire control plane: an image build on the live box, a hidden tab that streamed logs into Solid Cable, and a page that rendered fourteen hidden tabs to show one.
-
Deploys that survive a dropped stream or a control-plane restart
How I moved each Railyard build into an agent-side session that outlives the gRPC stream, so a ping timeout or a control-plane release no longer kills a customer's deploy, and the Go context bug that came with it.
-
Point-in-time recovery for Postgres on a server you own
How I added point-in-time recovery to Railyard's managed Postgres: WAL archiving, a daily base backup, and a restore that rebuilds the past in a throwaway Postgres so the live database is never touched until you say so.
-
Moving Railyard's control plane from Fly.io to one EC2 box
Why the Railyard control plane needed four Fly machines and a dedicated IPv4, how it runs now on one Graviton EC2 instance with RDS and Caddy, and the cache-table bug that returned 500 on every request after the cutover.
-
The deploy that hung for an hour
Two bugs in Railyard's Go agent about the end of a process's life: a build timeout that killed one process out of a tree, and a stale Puma pidfile that survived every restart.
-
Hardening a deploy platform against apps I didn't write
How I harden Railyard against open-source Rails apps: deploy each one unmodified, group the failures into classes, fix the class on its second occurrence, and only call something fixed when a fresh build reproduces it.
-
Four ways to build someone else's repo
The options a deploy platform has for turning a repository it didn't write into a running container, what each costs, and two bugs in Nixpacks and the buildpack launcher I hit deploying Discourse.
-
A cloud firewall broke every deploy, and the fix was to stop dialing Postgres
Railyard created each app's Postgres database over the server's public port 5432, so a cloud firewall made every deploy time out. I moved the SQL into the agent's existing gRPC channel and put app-to-database traffic on the Docker network.
-
Why Railyard is a Rails app and a Go agent joined by one proto file
Railyard splits into a Rails control plane that decides and a Go agent that does the work on each server, joined by a gRPC contract. How the contract is laid out, how build logs stream, and what the split costs.
-
What a platform does between git push and a working URL
The five jobs every PaaS does after you push: build an image, run a release step, start your processes, route traffic to them, and roll back when it goes wrong. Explained from first principles, with Railyard's two-week-old pipeline as the running example.
-
Why I'm building deploy software instead of a host
Railyard deploys your whole app onto servers in your own DigitalOcean, Hetzner or Vultr account. Why I chose that shape, where it sits next to Heroku, Kamal and Coolify, and what the first two days of code looked like.