PostgreSQL

A deadlock Postgres can't see

A Rails migration waited forever on a lock held by its own transaction, through a second connection pool. Why Postgres' deadlock detector can't catch that, how to spot it in pg_stat_activity, and why the logs were empty.

Why parallel deploys waited on each other

Several Railyard deploys at once queued behind each other and froze the dashboard. The shared resources one kind of work could fill, how I split them, and the two bugs the fixes created.

Point-in-time recovery for Postgres on a server you own

How I added point-in-time recovery to Railyard's managed Postgres: WAL archiving, a daily base backup, and a restore that rebuilds the past in a throwaway Postgres so the live database is never touched until you say so.

A cloud firewall broke every deploy, and the fix was to stop dialing Postgres

Railyard created each app's Postgres database over the server's public port 5432, so a cloud firewall made every deploy time out. I moved the SQL into the agent's existing gRPC channel and put app-to-database traffic on the Docker network.