Skip to content
apomesh

apomesh-worker

apomesh-worker is the worker layer: the execution endpoint that runs the substrate’s work. It dials out to an orchestrator, advertises what it can do, and then serves the dispatches it is sent — running each one through the LLM provider tool-use loop, executing tools inside the OS sandbox, evaluating predicates for Validator-tier verification, and streaming generated media into the asset store. It owns no session state; the orchestrator decides, the worker executes. The crate is a library split from its binary so the executor and dispatch loop can be embedded in-process for tests and single-process deployments. Touch it for execution behaviour: the dispatch loop, the reconnect posture, tool/media execution, and the worker’s mTLS handshake.

Position in the workspace

Section titled “Position in the workspace”
  • Layer: worker / tool execution. See Workers — the execution plane.
  • Depends on: apomesh-protocol (the wire it dials), apomesh-substrate (the executor, sandbox, provider router, and predicate evaluator it composes), plus apomesh-config, apomesh-substrate-catalog, apomesh-state-store-postgres (the durable asset-store adapter), and apomesh-vertical-deep-research-projection (the verification units it registers at boot).
  • Consumed by: as a binary, nothing — it is a leaf. As a library it is a dev-dependency of apomesh-orchestrator and both verticals’ integration tests, which embed the executor in-process through the Direct dispatch path rather than spawning a subprocess.

What it owns

Section titled “What it owns”
  • The dial-in registration model — the worker connects out to the orchestrator and sends a CapabilityAdvert as its first envelope (run_worker, spine_default_capability); the orchestrator routes against the advertised tiers, tools, tags, and evaluators.
  • The dispatch execution loop — WorkerExecutor runs each Dispatch through the provider tool-use loop, the sandbox, the Validator-tier predicate evaluator, the intra-reflection preflight gate, and the media route.
  • The runtime host — the worker starts, serves, evicts, and restarts a programmable agent’s confined workload, and serves the function families whose execution needs a credential or a sandbox decision (Generation, ToolInvoke, ObservedValue) rather than relaying them. It also owns the function-result journal, because it is the layer that reads it.
  • A resilient transport — bounded exponential-backoff connect and an outer reconnect loop that survives an orchestrator restart and re-advertises on its own (connect_with_retry, serve_with_reconnect).
  • In-process Direct embedding — the lib-vs-bin split so a WorkerExecutor drives dispatch with no subprocess, wired through the substrate’s DirectDispatch (direct_worker_from_executor).
  • MCP delegation over the wire — the worker-side sink that round-trips a tool’s tools/call back to the orchestrator by call_id (WireOrchestratorToolCallSink).

Public surface

Section titled “Public surface”
  • WorkerExecutor — the process-scoped execution engine; builder-constructed (with_router / with_predicate_evaluator / with_media_dispatch / …), holding the provider router, sandbox, base tool registry, and the per-session registry map.
  • run_worker — opens the bidi stream, sends the CapabilityAdvert, then serves inbound envelopes, routing each to a per-dispatch handler task.
  • serve_with_reconnect — wraps connect + register + serve in the outer reconnect loop; a stream end backs off and re-enters rather than exiting.
  • connect_with_retry — the initial gRPC connect with bounded exponential backoff (100 ms → 5 s), retried indefinitely so a down orchestrator parks the worker.
  • spine_default_capability — the default CapabilityAdvert (Worker / Validator / Synthesizer tiers, stateless lifecycle) used when no routing flags are set.
  • direct_worker_from_executor — wraps a WorkerExecutor as an in-process WorkerDispatch with both the per-envelope and session-envelope handlers wired.
  • direct_worker_from_executor_with_jobs — the same, plus the async-job seam bound to a (TenantContext, session_id).
  • WireOrchestratorToolCallSink + new_correlator — the MCP-delegation correlator that fulfils each pending call by call_id.

Usage

Section titled “Usage”

Run the binary in dispatch mode against a local orchestrator. The worker reads its orchestrator endpoint, tenant, and TLS material from flags or the matching environment variables; with no TLS flags it connects plaintext (the single-binary / dev path).

Terminal window
# Connect to a local orchestrator and serve dispatches.
apomesh-worker --orchestrator http://127.0.0.1:50051 dispatch


# Register as a Validator-only worker with mTLS to a remote orchestrator.
apomesh-worker \
  --orchestrator https://orchestrator.internal:50051 \
  --tier validator \
  --ca-cert /etc/apomesh/ca.pem \
  --client-cert /etc/apomesh/worker.pem \
  --client-key /etc/apomesh/worker.key \
  dispatch

The dispatch subcommand runs the registration + serve loop; the heartbeat subcommand (--count, --interval-ms) sends heartbeats only, for connectivity smoke-tests independent of the dispatch path. The three mTLS flags are all-or-none — a partial set is rejected at startup, and the --orchestrator URL must use https:// when they are present.

Section titled “Related”