Deployment & isolation
The control/execution split is one design that runs in several shapes. The same code runs everything in one process for local development and fans out across a worker fleet in production — configuration chooses the shape, not a code fork. This page covers those shapes, the tenant isolation that holds across all of them, and the direction the topology is built to grow toward. For the planes themselves, start at the group overview.
Single-binary: everything in one process
Section titled “Single-binary: everything in one process”The development shape runs the orchestrator and a worker as tasks in one process, with no external services. The worker is dispatched over an in-process channel rather than gRPC, backed by an in-memory or embedded store instead of Postgres and Redis. It is the same executor code the cluster runs — only the wiring differs.
This is also why the in-process hop is trusted to carry a resolved credential: nothing crosses a network, so the hop is secure even though it is not (and cannot be) mutual TLS. The credential plane forwards secrets only over a hop it can prove secure, and the in-process channel qualifies by construction — which is what makes the zero-dependency local stack able to run real tool credentials.
Compose and cluster: the planes split apart
Section titled “Compose and cluster: the planes split apart”The compose stack is the middle shape: one orchestrator container and one worker container over real gRPC, backed by Postgres (with pgvector) and Redis. It exercises the genuine network boundary between the planes while staying a single-machine setup — the canonical way to run apomesh locally with the production wire path. It is a local evaluation stack, not a production template.
The cluster shape splits the planes across the network for real. One orchestrator daemon fronts the durable state plane — Postgres for durability (the event log, snapshots, memory tiers, the side-effect journal) and Redis for hot paths (the working-memory cache and event-stream replay). Workers dial in over mutual-TLS gRPC and scale out independently. Drivers reach the orchestrator three ways: SDK clients over gRPC, browser dashboards over REST and server-sent events, and the desktop console over its IPC bridge (which proxies to the same gRPC surface).
Worker authentication is its own boundary
Section titled “Worker authentication is its own boundary”In the cluster shape, worker-to-orchestrator authentication is a separate concern from how drivers authenticate. Driver identity is the identity & authorization plane; worker identity is a distinct, transport-layer backend. When mutual TLS is enabled for workers, the TLS layer validates the worker’s certificate chain before any request handler runs, and the worker’s identity is then extracted from the validated certificate (its common name or a subject alternative name). That validated-mTLS hop is exactly the “proven-secure” condition the credential plane requires before it will forward a tool secret to a worker — a plaintext worker hop forwards nothing. The local stack ships with worker auth off; enabling mTLS is an operator step.
Multi-tenant isolation, end to end
Section titled “Multi-tenant isolation, end to end”Every persistence call, dispatch, and event carries a tenant context, and it is
threaded through the whole system from one place: the tenant is recovered from an
agent’s opaque reference — a (tenant, agent) pair a caller cannot decode or forge — so
tenancy travels with the work rather than being re-asserted at each layer. That single
context gates the state store, the event tail, and worker routing.
Isolation fails loudly. The storage boundary surfaces a cross-tenant access as a typed error that the orchestrator turns into session-terminating escalation; internal detection paths panic at the point of detection rather than returning a recoverable error, because a cross-tenant read is a bug, not a metric. The Postgres adapter also scopes every query by tenant id as defense in depth — application-level scoping, not database row-level security, with the substrate’s tenant context remaining authoritative. Per-tenant credentials, quotas, and secrets all key off the same context. This is an Invariant-tier contract; see Identity & authorization.
What scales versus what is singular
Section titled “What scales versus what is singular”| Component | Shape |
|---|---|
| Orchestrator | one singleton per deployment; high availability via standard control-plane techniques, never sharded by tenant |
| Workers | horizontal — add capacity by adding workers, each fungible |
| State plane | one shared store per deployment (Postgres + Redis), the sole home of durable state |
| Tool sandbox | one throwaway subprocess per tool call — the finest-grained, most ephemeral unit |
The orchestrator concentrates consistency so workers don’t have to; workers absorb load so the orchestrator doesn’t have to. That division is what lets the control plane stay a single authority while execution scales without bound.
The edge direction
Section titled “The edge direction”Because a worker dials outbound, a worker that accepts no inbound connections — one behind NAT, on a personal machine, or on an on-prem box — already shares the same wire path as a cloud worker. The protocol reserves space for treating such edge peers as first-class members of the fleet: cost arbitrage (rent orchestration, push inference to a user’s own GPU), data residency (sensitive data never leaves the device), and air-gapped deployments. The full edge-fleet model — placement-aware routing, edge-peer trust — is a directional north-star vector, not shipped surface today; what is shipped is the outbound dial-in that keeps that door open without a second code path.
Where this shows up
Section titled “Where this shows up”- Operate: The local stack runs the single-binary and compose shapes; Monitoring covers the cluster’s observability.
- Build: The Python SDK and REST & SSE reach any shape identically — the wire is the same.
- Reference: the state-store adapters, the auth-provider crates, and the config crate under Reference.