The orchestrator

The orchestrator is the single control-plane daemon. One runs per deployment, and it owns everything durable: the sessions in flight, the routing of work to workers, supervision of failures, human-in-the-loop adjudication, cost and budgets, and every write to the state plane. This page walks each of those responsibilities at mechanism level. For how a session moves through them front to back, see The dispatch lifecycle.

The per-session actor model

Each live session is an owned actor, not a row a pool of threads mutate under locks. When a session starts, the orchestrator spawns one asynchronous task that owns that session’s state exclusively — its agent tree, working memory, HITL waiters, cost aggregator, and dispatch bookkeeping — and the only way to touch that state is to send the actor a message.

• The actor processes a closed message vocabulary: run the session, cancel it, deliver a HITL response, bind a project, snapshot state, terminate. Every mutation arrives as one of these messages over a bounded mailbox, so there is exactly one writer per session and no shared-state races.
• The actor is registered before it is spawned, under one critical section — so a cancel that races the very first dispatch can never miss its target.
• Its run loop drives the coordinator and watches the mailbox at the same time, so a cancel or a HITL response is honored while the loop is mid-flight, not after it returns.

This owned-actor shape is what makes a session’s state consistent without global locks, and it is the seam the durability story rests on — the actor’s state is a fold of the same event log everything else reads. See Sessions, events & durability.

The coordinator seam

The actor owns state; a coordinator owns the session’s loop shape. The two are deliberately separate types. The daemon holds one coordinator implementation per SessionKind behind a small strategy-selection contract, and picks the right one when a session starts. The chosen coordinator then drives the owned actor through a dispatch-handle interface — spawn a child, dispatch an agent, ask whether the parent has budget left — never by reaching into the actor’s state directly.

That separation is the whole reason the coordination shape is a first-class, swappable choice rather than something baked into the agent. The loop shapes the platform ships — plan-execute-verify-replan, evaluate-and-refine, fan-out-and-reduce, multi-round debate — are all coordinators over this same seam. The Coordination loops group is their canonical home.

Routing work to workers

The orchestrator decides which worker receives each dispatch. Routing runs in stages, short-circuiting whenever a cheaper answer is available:

Rendering diagram…

Worker selection: an affinity ticket short-circuits to the same worker; otherwise an eligibility filter narrows the fleet and a weighted score picks among the survivors

Eligibility is a hard filter. A worker is a candidate only if it supports the agent’s capability tier, advertises every capability and evaluator the agent requires, accepts the agent’s tools, and is permitted to serve the agent’s tenant. This stage is pass/fail, not a preference.

Scoring picks among the eligible. Each survivor gets a weighted score over four criteria — cost, latency, load, and warmth — combined by operator-tunable weights. Today load is the criterion with a live signal (least-loaded wins); the others are wired into the model as reserved weights, so the effective default is “least-loaded among the eligible.” Ties break deterministically.

Affinity keeps a resumed agent on its worker. When a stateful agent should return to the same worker, the orchestrator mints a server-side affinity ticket carrying the worker’s identity and a generation stamp. A later dispatch with a live ticket skips scoring entirely and goes warm; if that worker has since restarted, the generation no longer matches and routing falls through to a fresh selection.

Routing is a substrate component the orchestrator calls at dispatch time — the fleet routes, the orchestrator owns the worker-registry lifecycle.

Supervision: typed failures, named strategies

Failures are typed, and each type is handled by a named strategy — there are no ad-hoc retry loops scattered through the code. A failure carries a class (a quality miss, a tool failure, a budget or timeout exhaustion, a preflight escalation, a worker process loss, a quota breach), and the agent’s supervisor strategy decides how that failure travels. The strategy vocabulary is Erlang/OTP’s supervision tree, named directly: OneForOne (act on just the failed child), Escalate (climb the parent chain — the default), and SelectiveReplan (the richest — choose an action per failure class); OneForAll and RestForOne are reserved in the vocabulary for sibling-group restarts.

A strategy resolves a failure to a bounded action: escalate to the parent, retry with exponential backoff, retry with revision feedback, re-acquire a fresh worker, or pause for a human. Every retry path passes universal gates first — it degrades to escalation if the budget can’t cover a retry or the retry cap is hit — so a failure can never spin. Unhandled failures climb the parent chain; a failure that reaches the root with no strategy claiming it terminates the session with a durable final snapshot.

HITL adjudication

Some steps must pause for a human — a preflight gate before an irreversible action, a plan awaiting approval. The orchestrator adjudicates these human-in-the-loop pauses. It installs the waiter into the session actor before it appends the HitlPaused event, so a response can never arrive before there is something to resolve. An operator’s answer — approve (optionally with a grant), reject (the reason becomes revision feedback), redirect, or accept-partial — is first appended to the event log as the durable audit moment, then used to resolve the paused agent and let the loop continue. The responder’s identity is stamped server-side from the authenticated call, never taken from the wire payload. Operators drive these from the console; the pause survives a restart because it is reconstructed from the log, not held only as a live in-memory wait.

Cost and budget enforcement

Two mechanisms enforce spend, and they are distinct from the read-side rollups:

• Per-agent budgets are carved at spawn. A parent’s allocator reserves a child’s budget from its own when the child is created, so a subtree can never outspend its parent. Exceeding a declared dimension raises a budget failure into supervision.
• Per-tenant quotas are checked before a worker is ever selected. A ceiling breach (concurrency, rate, or cumulative cost/token limits over a rolling window) short-circuits the dispatch — no worker runs, no budget is consumed — and raises a quota-exceeded failure.

The per-session and per-tenant cost aggregators are read-side consumers of the cost event stream — they answer “what did this cost,” they do not gate spend. The enforcement points are the budget carve and the quota check. See Budgets & cost.

The event bus is the one write path

Everything above becomes durable the same way: through the event bus, the single append API to the state plane. Bypassing it is a contract violation, and workers have no path to it at all — a worker streams envelopes back, and the orchestrator is the one that appends them. Every consumer (a live tail, the audit view, cost rollups, session reconstruction) reads that one log. This is what makes the control plane the sole writer and the event log the single source of truth.

Where this shows up

• Operate: The operator console shows live sessions, workers, pending HITL, and in-flight dispatches; Monitoring consumes the same event stream.
• Build: The Python SDK and REST & SSE start sessions and respond to HITL over the control plane.
• Reference: the orchestrator crate and the substrate routing, supervisor, and event-bus modules under Reference.