Skip to content
apomesh

Control & execution

apomesh separates deciding from doing. One orchestrator — the control plane — owns every durable decision and all persistent state. A fleet of workers — the distributed execution plane — does the actual LLM and tool work and holds no state of its own. This group explains that split: what each plane owns, the wire that joins them, and how a single session flows across the boundary. For where sessions sit in the whole system, start at the architecture overview.

Why the split

Section titled “Why the split”

A multi-agent run has two kinds of work with opposite requirements. Coordination — deciding what to run next, recording what happened, recovering from failure — must be consistent and durable: one authority, one ordered history, survivable across a process restart. Execution — calling a model, running a tool — is stateless and elastic: it should scale out, run anywhere (including untrusted hardware), and fail without taking state down with it.

apomesh gives each its own home. The orchestrator is a singleton that concentrates all consistency; workers are fungible and carry nothing, so a lost worker costs a re-dispatch, never a lost session. This is the control plane + distributed execution plane shape — the same family as Temporal and its workers, a Kubernetes scheduler and its kubelets, or Erlang/OTP supervision across nodes. It is neither microservices (there are no per-worker databases) nor a plain monolith (execution genuinely distributes).

The two planes and the trust boundary between them: the orchestrator owns all state, workers execute and stream events back over one wire

What each plane owns

Section titled “What each plane owns”
Control plane (orchestrator)Execution plane (workers)
Countone singleton per deploymentmany, scaled horizontally
Statethe event log + all persistence (sole writer)none durable — streams events back
Decidesrouting, supervision, HITL, budgets, the loopnothing — executes what it’s handed
Doesno LLM or tool work directlythe LLM tool-use loop + sandboxed tools
Trustholds credentials, resolves per tenantreceives secrets only over a proven-secure hop

The trust boundary

Section titled “The trust boundary”

The two planes meet at one wire: a worker dials outbound to the orchestrator, advertises its capabilities, and then work flows over that single long-lived gRPC stream. That boundary is a real network protocol, so it is also a trust boundary. The orchestrator holds every secret and resolves each credential server-side per tenant; it forwards a resolved credential to a worker only when the hop is proven secure (mutual TLS, or an in-process channel in single-binary mode). A plaintext hop forwards nothing — the tool surfaces a typed config error instead of leaking. The full contract lives on the credential plane.

The four sub-pages

Section titled “The four sub-pages”
  • The orchestrator — the control plane in depth: the per-session actor model, the coordinator seam, worker routing, supervision, HITL adjudication, and budget enforcement.
  • Workers — the execution plane in depth: dial-in registration, the tool-use loop, the sandbox isolation chain, and reconnect-and-resume.
  • The dispatch lifecycle — one session walked end-to-end at mechanism level, from StartSession to terminal synthesis.
  • Deployment & isolation — the single-binary, compose, and cluster shapes, multi-tenant isolation, and what scales versus what is singular.

Guarantees

Section titled “Guarantees”
  • Single writer, one truth. Only the orchestrator writes the state plane, and everything observable flows through the event log — no side channels. See Sessions, events & durability.
  • Workers are fungible and stateless. They carry no session state, so a lost worker costs a re-dispatch, never a lost session.
  • Tenant isolation is enforced at the boundary. Every persistence call, dispatch, and event carries a tenant context; a cross-tenant access fails loudly, never reads silently. See Identity & authorization.
  • One coordinator per session. A session runs under exactly one coordinator; strategies plug into fixed phases rather than reorder them. See Coordination loops.

Where this shows up

Section titled “Where this shows up”