Workers

A worker is where the actual work happens: it calls the LLM, runs the tools the model asks for, and streams the results back. It holds no durable session state, so any worker can serve any dispatch it is capable of, and a lost worker costs a re-dispatch rather than a lost session. This page covers how a worker joins the fleet, the loop it runs, the sandbox chain every tool passes through, and how it recovers from a disconnect. For how a dispatch is chosen and routed to a worker, see The orchestrator.

Dial-in registration

A worker dials outbound to the orchestrator on startup — the orchestrator never reaches into workers. The worker connects to the orchestrator’s address, retrying with bounded backoff until it is reachable, and its first frame is a capability advertisement: the tiers it supports, the tools and evaluators it accepts, its current load, and operator-set tags. All work then flows over that one long-lived bidirectional gRPC stream — dispatches down, lifecycle/cost/result events up — with a periodic heartbeat so the orchestrator can tell a live-but-idle worker from a dead one.

Rendering diagram…

A worker joins by dialing out and advertising capability; the same stream then carries every dispatch down and every event back up

Why outbound? A worker that initiates the connection needs no inbound port, which is exactly what lets a worker run behind NAT or a firewall — on an edge gateway, an on-prem GPU box, or a personal machine — over the same wire path as a cloud worker. There is no separate edge-only code path; the same dial-out shape keeps the edge-network direction open — a door the protocol holds without a second wire path.

The tool-use loop

Once a worker receives a dispatch, it runs the agent’s tool-use loop. It first checks that the dispatch is for a tier a worker may run — a planner tier is rejected, because planning runs in-process in the orchestrator, not on a worker — then emits the agent’s Created and Running lifecycle events and enters the loop:

1. Call the model with the conversation so far and the agent’s tools.
2. If the model returned no tool calls, that text is the answer — the loop emits a goal-met terminal outcome and the dispatch ends.
3. Otherwise, run each requested tool (through the sandbox chain below), feed the results back into the conversation as tool results, checkpoint working memory, and repeat.

The loop is bounded by a hard iteration cap, so a model that never stops calling tools terminates with a quality failure rather than running forever. The provider is routed once per dispatch, not per turn. When the provider supports streaming, each chunk is emitted as a streaming event as it arrives and aggregated back into a normal response for the loop’s own logic — so a live tail sees tokens in real time while the loop still reasons over a complete turn. The conversation is compacted to a token runway as it grows, keeping a long dispatch inside the model’s context window; the richer prompt-composition story is the context plane.

The isolation chain

Every tool call runs in an OS-level sandbox — there is no trusted-tool escape hatch. The chain is worker → per-dispatch sandbox → runner subprocess:

• The worker spawns one throwaway subprocess per tool call (Bubblewrap with seccomp on Linux, sandbox-exec with an SBPL profile on macOS — selected at build time for the platform). The tool’s arguments and its resolved credentials are serialized and written to the subprocess over stdin; nothing crosses through the process environment, which is cleared to a fixed pass-through set.
• A side-effecting tool is journaled first: its call is keyed by a content hash of its arguments, and a prior recorded result is replayed instead of running the tool again. This is what makes a re-dispatch after a crash safe — a call that already ran once is not run twice.

The credentials a tool needs never become readable by the agent — the orchestrator resolves them per tenant and injects them at this boundary, over the transient channel that never touches the event log. That contract is the credential plane; the sandbox and runner mechanics live in their reference crate pages. This page only names the chain — it does not re-explain either.

The second execution tier: workload runtimes

The chain above confines a tool — a short-lived request/response command. A worker also hosts the other unit of execution: an OCI image, run as a long-lived confined workload with a framed bidirectional channel and a resource envelope. That is how a programmable agent runs.

The two tiers are distinguished by what is being executed, never by the confinement technology — a new isolation backend joins a tier as an adapter rather than becoming a third tier. They share their confinement primitives instead of each growing a half-correct copy, which is why one crate owns both.

For the workload tier the worker is also the server: it answers the function families whose execution needs a credential or a sandbox decision — a model call, a tool call — so the runtime receives outputs and never a key. It starts, evicts, and restarts the workload, and owns the result journal that lets a replayed step be served rather than re-paid.

What a worker holds versus persists

A worker holds only transient, in-memory state: per-session registries (tools, providers) it builds when a session is initialized and drops when the session ends, and the in-flight conversation history for the dispatch it is running. None of it is backed by a store. Everything durable is streamed back to the orchestrator as events — the worker is a producer of the event log, never its writer:

• Lifecycle transitions (Created → Running → a terminal outcome).
• Working-memory checkpoints at each node boundary, which the orchestrator persists so a resumed dispatch can pick up where it left off.
• Cost events per model call and per priced tool, streaming chunks, tool results, and compaction records.

Because the worker keeps nothing durable, its correctness never depends on its own survival — the event log is the memory.

Reconnect and resume

The disconnect story has two independent halves that meet at the wire.

Worker side. If the stream drops — a clean close or a transport reset — the worker backs off and redials, re-sending its capability advertisement to re-register. Its in-memory per-session state is left intact across a reconnect, so a brief blip doesn’t discard warm registries.

Orchestrator side. Liveness is tracked by heartbeat, not RPC errors — a worker can be alive but partitioned. When a worker misses its heartbeats past a threshold, the orchestrator unregisters it first (so routing won’t re-pick it), cancels the in-flight dispatch, and re-acquires a fresh worker: it loads the agent’s latest checkpoint, emits an AgentResuming event, re-dispatches, and emits AgentResumed on completion. The re-dispatched agent keeps its logical identity, so its journaled side-effecting tool calls hit the replay cache instead of firing twice. The end-to-end recovery invariant — including recovery across a full daemon restart — is on Reconstruction & recovery.

Where this shows up

• Operate: The operator console lists live workers and in-flight dispatches; The local stack runs a worker in-process or as a compose service.
• Build: Deep Research is a worked domain whose worker, validator, and synthesizer tiers all run on the execution plane.
• Reference: the worker crate, the sandbox crate, and the tool-runner binary under Reference.