Skip to content
A Bekkour
Writing
GoRailsgRPCDocker

Streaming a deploy from a Go agent to Rails over gRPC

8 min read

I’m building a small platform that deploys a Rails app from a git URL: point it at a repo, and it clones, builds a Docker image, swaps the running container, and shows you every log line as it happens. The product is unremarkable. The interesting engineering is the transport — how does a Go process running docker build on a build server stream that build, line by line, back to a Rails app (and on to your browser)?

The answer is a gRPC server-streaming RPC: Rails makes one call, the Go agent sends back a stream of typed events until the deploy finishes. Here’s the whole path, with the real code from both sides.

The contract: one .proto, two services

Everything starts from a single proto file that generates both the Ruby and the Go types, so the two languages literally cannot disagree about the wire format. There are two services — one each direction:

service ControlPlane {                 // implemented by Rails
  rpc Register(RegisterRequest) returns (RegisterResponse);
  rpc Heartbeat(HeartbeatRequest) returns (HeartbeatResponse);
}

service Agent {                        // implemented by the Go agent
  rpc Deploy(DeployRequest) returns (stream DeployEvent);
}

message DeployEvent {
  enum Kind { KIND_UNSPECIFIED = 0; KIND_STDOUT = 1; KIND_STDERR = 2; KIND_COMPLETED = 3; KIND_SYSTEM = 4; }
  Kind kind = 1;
  string line = 2;
  int32 exit_code = 3;
  google.protobuf.Timestamp emitted_at = 4;
}

Each build server runs an agent and registers itself with the Rails control plane (with heartbeats reporting CPU/memory/disk), so Rails knows which servers exist and how to reach them. Deploys go the other way: Rails calls the agent’s Deploy, and the stream DeployEvent return type is the whole point — one request, many responses. A deploy is exactly “one action that produces many log lines,” which is precisely what server-streaming is for. Each event is tagged: agent commentary (SYSTEM), a line of command output (STDOUT/STDERR), or the terminal COMPLETED with an exit code.

The Go agent: a pipeline of streamed steps

The agent’s Deploy is a straight-line pipeline, and every step reports as it runs. The shape:

func (s *Server) Deploy(req *pb.DeployRequest, stream pb.Agent_DeployServer) error {
    if err := s.authorize(stream.Context()); err != nil {
        return err
    }
    // ... clone the repo (shallow, at the requested ref) ...
    if code, err := s.runStep(stream, "git", "clone", "--depth", "1", "--branch", req.GitRef, req.GitUrl, workdir); err != nil {
        return err
    } else if code != 0 {
        return s.sendCompleted(stream, code)
    }
    // ... docker build with BuildKit ...
    if code, err := s.runStepWithEnv(stream, []string{"DOCKER_BUILDKIT=1"}, "docker", "build", "-t", tag, workdir); err != nil {
        return err
    } else if code != 0 {
        return s.sendCompleted(stream, code)
    }
    // ... replace the old container, docker run the new one, verify it's Up ...
    return s.sendCompleted(stream, 0)
}

The whole thing is legible top to bottom: clone → build → replace → run → verify, and any non-zero exit code short-circuits to a COMPLETED event carrying that code. Each of those runStep calls is where the streaming actually happens.

runStep: stream a command’s output, line by line

This is the reusable primitive and the most useful thing in the codebase. It runs a command, fans its stdout and stderr into a channel as DeployEvents, forwards each onto the gRPC stream, and returns the process’s exit code:

func (s *Server) runStepWithEnv(stream pb.Agent_DeployServer, extraEnv []string, name string, args ...string) (int, error) {
    cmd := exec.CommandContext(stream.Context(), name, args...)
    if len(extraEnv) > 0 {
        cmd.Env = append(os.Environ(), extraEnv...)
    }
    stdout, _ := cmd.StdoutPipe()
    stderr, _ := cmd.StderrPipe()

    if err := cmd.Start(); err != nil {
        _ = sendSystem(stream, fmt.Sprintf("failed to start %s: %v", name, err))
        return 1, nil
    }

    events := make(chan *pb.DeployEvent, 64)
    var wg sync.WaitGroup
    wg.Add(2)
    go scanLines(stdout, pb.DeployEvent_KIND_STDOUT, events, &wg)
    go scanLines(stderr, pb.DeployEvent_KIND_STDERR, events, &wg)
    go func() { wg.Wait(); close(events) }()

    for ev := range events {
        if err := stream.Send(ev); err != nil {
            _ = cmd.Process.Kill()          // client disconnected — don't orphan the build
            return 0, fmt.Errorf("stream send: %w", err)
        }
    }

    if err := cmd.Wait(); err != nil {
        var exitErr *exec.ExitError
        if errors.As(err, &exitErr) {
            return exitErr.ExitCode(), nil   // a failed build is a normal outcome, not an error
        }
        return 1, fmt.Errorf("wait: %w", err)
    }
    return 0, nil
}

Four details worth calling out:

  • exec.CommandContext(stream.Context()) ties the child process to the RPC, so a cancelled deploy or a dropped connection tears down the docker CLI process. Be precise about the limit, because an interviewer will: CommandContext kills the direct child — the docker client — not the process group, and the real build runs in the Docker daemon / BuildKit. In practice BuildKit cancels the build when its client connection drops, but a container that already started via docker run isn’t reaped this way. “No orphaned builds” is the goal, not a guarantee I’d make under questioning.
  • One goroutine does the sending. stdout and stderr are scanned by two goroutines that feed a single channel, and only the consumer loop calls stream.Send — which matters because gRPC’s Send isn’t safe to call concurrently. Funnelling both streams through one sender is what keeps that correct (ordering is then by channel arrival, which is why each event also carries its own emitted_at).
  • Kill on send failure. If stream.Send fails, the client is gone, so we kill the process rather than let a pointless build run on. One thing I’d fix: that path kills without a following cmd.Wait(), so it leaks a zombie child — easy to miss, cheap to correct.
  • A non-zero exit code is data, not an error. A failed docker build returns its exit code up the pipeline (a COMPLETED{exit_code} event); only a genuinely broken invocation returns a Go error. Keeping “the build failed” and “the agent broke” separate is what lets the UI show a red build instead of a 500.

The line scanner uses a big buffer, because Docker/BuildKit output has long lines that the default 64KB scanner would choke on:

func scanLines(r io.Reader, kind pb.DeployEvent_Kind, events chan<- *pb.DeployEvent, wg *sync.WaitGroup) {
    defer wg.Done()
    scanner := bufio.NewScanner(r)
    scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024)   // up to 1MB per line
    for scanner.Scan() {
        events <- &pb.DeployEvent{Kind: kind, Line: scanner.Text(), EmittedAt: timestamppb.Now()}
    }
}

The Rails side: consume the stream, persist each line

Rails dials the agent, sends a DeployRequest, and iterates the returned stream. Each event becomes a row in deploy_logs; the terminal event sets the deploy’s final status:

stub = Railyard::V1::Agent::Stub.new(agent_addr, :this_channel_is_insecure)

request = Railyard::V1::DeployRequest.new(
  application_id: @deploy.application_id,
  git_url:        @deploy.application.git_url,
  git_ref:        @deploy.application.git_ref,
  host_port:      @deploy.application.host_port,
  container_port: @deploy.application.container_port,
  env_vars:       (@deploy.application.env_vars || {}).transform_keys(&:to_s).transform_values(&:to_s),
)
auth = { "authorization" => "bearer #{ENV.fetch('RAILYARD_SECRET')}" }

stub.deploy(request, metadata: auth).each do |event|
  case event.kind
  when :KIND_STDOUT, :KIND_STDERR
    @deploy.deploy_logs.create!(stream: event.kind == :KIND_STDOUT ? "stdout" : "stderr",
                                line: event.line, emitted_at: timestamp_to_time(event.emitted_at))
  when :KIND_SYSTEM
    @deploy.deploy_logs.create!(stream: "system", line: event.line, emitted_at: timestamp_to_time(event.emitted_at))
  when :KIND_COMPLETED
    succeeded = event.exit_code.zero?
    @deploy.deploy_logs.create!(stream: "system", line: "Pipeline exited with code #{event.exit_code}",
                                emitted_at: timestamp_to_time(event.emitted_at))
  end
end

@deploy.update!(status: succeeded ? :succeeded : :failed, finished_at: Time.current)

Two practical notes. The whole thing runs in a background Thread with its own database connection (ActiveRecord::Base.connection_pool.with_connection), because a deploy stream is long-lived and has no business holding a web request. And this is where the browser finally comes in: browsers can’t speak gRPC, so Rails is the bridge — it translates the gRPC event stream into deploy_logs rows, and the browser tails those over plain HTTP. That translation layer isn’t overhead; it’s the reason the architecture works. gRPC for service-to-service, HTTP for the browser, and one table in between.

Auth (and an honest caveat)

Both directions authenticate with a shared bearer secret in the gRPC metadata. The Go agent checks it before doing anything:

func (s *Server) authorize(ctx context.Context) error {
    md, _ := metadata.FromIncomingContext(ctx)
    auths := md.Get("authorization")
    if len(auths) == 0 { return status.Error(codes.Unauthenticated, "no authorization") }
    if strings.TrimPrefix(auths[0], "bearer ") != s.secret {
        return status.Error(codes.Unauthenticated, "invalid secret")
    }
    return nil
}

The other direction — an agent registering with the Rails control plane — uses a constant-time compare:

unless ActiveSupport::SecurityUtils.secure_compare(expected, presented)
  raise GRPC::Unauthenticated.new("invalid secret")
end

Which surfaces an asymmetry I should own rather than let you catch: look again at the Go agent above — it compares the secret with a plain !=, not crypto/subtle. So the path that actually triggers a deploy (Rails → agent) is checked without timing protection, while the less sensitive registration path is. On a trusted network that timing channel barely matters, but it’s inconsistent to harden one side and not the other, and moving the agent’s check to a constant-time compare is the obvious fix.

The bigger caveat: the channel is :this_channel_is_insecure — a shared secret with no transport encryption. That’s fine when the agent and control plane sit on a trusted network (a private VPC or a WireGuard mesh), which is the deployment model here. Exposed to the open internet it would need TLS, and ideally mTLS so each side verifies the other’s certificate. A shared secret authenticates who; it doesn’t protect the bytes in transit.

Why gRPC here

Because the problem is a perfect fit for it: server-streaming gives you “one call, many typed events” natively (no long-polling, no framing your own protocol over a socket), and a single .proto generates matching types for Ruby and Go so the contract can’t silently drift. The cost is real — a codegen step, binary frames you can’t just curl, and the fact that browsers can’t speak it — but for a Rails-orchestrates-Go deploy engine, that last point is a feature in disguise: it forces the clean split where Go does the machine work, Rails owns the state and the UI, and the log stream lands in a database the browser can actually read.

This is the deploy engine behind Railyard, a one-click Rails-on-your-own-servers platform I’m still building. If you’re orchestrating Go from Rails, or just weighing gRPC against REST for a streaming job, get in touch. There’s a related Go-streaming piece too: uploads over WebSockets with io.Pipe backpressure.

Al Mokhtar Bekkour

Senior Rails & Go engineer in Quebec. I build multi-tenant SaaS and financial integrations, and ship my own products — Constat, Railyard, and Recall. Open to work.

About me Contact GitHub LinkedIn
← All writing