#Runner autoscaling

Runners are the compute that executes your Actions. On a dedicated pool, Fjord scales that compute to match demand and tears it back down to nothing when the queue is empty. This runbook explains how that works, the controls that keep it from running away, and how the scaler authenticates without ever holding a long-lived forge admin token.

#Demand-driven, not scheduled

The scaler does not run on a clock. It watches the work queue and reacts:

• The signal is queue pressure. Pending jobs against available runners. When jobs are waiting and there is capacity left, the scaler creates more runners.
• The resting state is zero. With no work in flight, idle runners are a defect, not a buffer. They are reaped promptly, so an idle instance costs nothing in runner compute.

Because the trigger is queue depth rather than a fixed pool size, a burst of CI gets capacity quickly and a quiet weekend scales to nothing without anyone touching a dial.

#One job per runner

Every runner is ephemeral and single-use. The lifecycle is roughly:

requested -> booting -> registered -> running -> draining -> destroyed

A fresh machine boots, registers itself, runs exactly one job, and is destroyed. Nothing is reused between jobs. That is what lets us promise a clean environment every time, and it removes a whole class of “the runner had stale state” bugs. It also bounds the blast radius of a malicious or broken job to the single machine it ran on.

#Cost controls

Autoscaling without limits is a way to spend money quickly. Demand is bounded by layered guards, evaluated before any machine is created. The exact numbers are deployment-specific; the shape is:

#Admin-endpoint authentication

The sharp edge of autoscaling is credentials. A scaler that can register and delete runners needs to talk to privileged forge endpoints, and a leaked long-lived admin token there is a serious incident. The design principle:

The autoscaler holds no standing forge admin credential.

Two mechanisms make that possible:

• Short-lived, single-use registration tokens. Each new runner gets a token that is scoped to that one registration, valid for minutes rather than hours, and delivered to the machine through boot-time user data, never baked into an image. After the runner registers, the token is spent.
• Privileged operations sit behind a control plane. Confirming, deregistering, and reaping a runner are capabilities the scaler requests from a separate control-plane service, which is the only component that knows the forge admin secret. The scaler asks; it never wields the credential itself.

That split means a compromise of the scaler does not hand an attacker the keys to the forge, and admin secrets can rotate without redeploying the scaler.

#Failure modes worth knowing

• The stuck boot. A machine that boots but never registers will silently consume a slot. The startup timeout exists precisely to reap it; without one, a bad image quietly caps your effective capacity.
• The runaway signal. A misread queue can ask for hundreds of machines at once. The create rate limit turns that into a controlled ramp instead of an instant bill.
• The orphan. If a create event is lost after the machine exists, you have a runner nobody is tracking. Tag machines with the intent that created them so a reconcile pass can find and recover orphans rather than leaking them.

#Verify it is healthy

Watch these signals:

• Queue pressure ratio (pending jobs over available runners) should spike and then recover, not climb steadily.
• Live runner count should track demand and return to zero when idle.
• Circuit breaker state should be closed in normal operation; an open breaker means spend tripped a threshold and needs a human.

To run jobs on these runners, target the Linux pool and reference Fjord Actions by full URL, as shown in Runners. For the workflow trigger model underneath, see the upstream

#Related

• Runners: the tier model and how to target the pool.
• Leader-only background workers: the scaler’s control loops are singletons elected with the advisory-lease pattern.
• Supply-chain CI gate: the runner binary itself is pinned and checksum-verified at boot.