Reconstruction & recovery

Because session state is the fold of the event log, apomesh can rebuild a session at any time from its events alone — and that same property is what lets a session survive a lost worker or a full daemon restart. This page covers the fold that reconstructs state, the checkpoints that give a resume its starting point, the two recovery paths, and the pinning that makes a resumed session run the same agent it started with.

Reconstruction: the fold

Reconstruction is a pure fold over a session’s envelopes: walk them once and accumulate a session-state view. The view carries the spawn tree of agents, each agent’s latest phase and its full resolved config (folded from the birth-time snapshot on its lifecycle event), the latest coordinator checkpoint, any in-flight async jobs, unmatched HITL pauses, whether the session is terminal, and the root’s terminal output. A narrower bootstrap view carries just the identity a resume needs — the goal, the root agent, and the pinned shape (below).

The load-bearing distinction is derived vs. durable. A lifecycle phase, a terminal reason, a checkpoint blob, the SessionStarted bootstrap fields — those are read straight off an event. The spawn-tree edges, whether the session is terminal, and the set of still-in-flight sub-tasks are computed by the fold; they are never themselves events. This is why a coordinator checkpoint carries loop position, not work product: the sub-task outputs it would otherwise duplicate are re-derived from the per-agent terminal lifecycle events already on the log.

Checkpoints

“Checkpoint” names two structurally distinct mechanisms — keeping them apart is the key to reading the recovery paths.

Per-agent snapshots. As an agent runs its tool-use loop, the orchestrator writes a snapshot at each node boundary. The snapshot holds the agent’s definition, its budget runtime state, and its working-memory conversation. The event on the log is only a pointer — the bytes live in the state store, never duplicated onto the log. What triggers a snapshot is a small, closed set of sources — node completion (the automatic per-iteration case), a pre-supervisor reset point, an agent’s explicit checkpoint.now, and an affinity release. There is no HITL checkpoint source: a pause is recovered organically by re-running the loop to the same stall, not by a special checkpoint.

Coordinator checkpoints. Separately, the coordinator writes a CoordinatorCheckpoint event as its loop advances — after planning, each iteration — carrying its opaque loop state (iteration counters, cap ceilings, the serialized plan and feedback). The substrate ferries this blob without interpreting it; only the coordinator that wrote it reads it back. The latest one is the resume point for the loop itself.

The two answer different questions: the per-agent snapshot resumes a worker-disconnected agent mid-dispatch; the coordinator checkpoint resumes the loop after a restart.

Resume after a worker disconnect

Workers are fungible and hold no durable state, so losing one costs a re-dispatch, never a session. Liveness is tracked by heartbeat, not RPC errors — a worker can be alive but partitioned, so an error is not proof of death and a silence is. When a worker misses its heartbeats past a threshold, the orchestrator unregisters it first (so routing won’t re-pick it), cancels the in-flight dispatch, loads the agent’s latest snapshot, emits AgentResuming, re-dispatches to a fresh worker under the same AgentRef, and emits AgentResumed on completion.

Rendering diagram…

Worker disconnect to resume: heartbeat silence is the trigger; the agent keeps its identity, and its journaled side effects are replayed rather than re-run

Keeping the same AgentRef is what makes re-dispatch safe. A side-effecting tool call is journaled — keyed by a content hash of its arguments under the agent’s identity — so a call that already ran is served from the journal instead of firing twice. Because the resumed agent keeps its identity, a call it already made hits that journal, and a re-dispatch after a crash never double-invokes a side effect.

Resume after a daemon restart

A session’s life can outlast the orchestrator process. On boot the orchestrator scans the log and rehydrates recoverable sessions:

1. List every session with log entries.
2. Skip any that already has a completed-session record — it is terminal and indexed.
3. Fold each remaining session’s log into a state view; skip terminal ones — a session whose root reached a terminal state is never brought back as live. A session whose log cannot be read whole is skipped too, loudly: the fold’s answer is only as good as the envelopes it saw, so silently dropping an undecodable row makes a finished session read as resumable and the daemon re-drives work that already ran. Skipping is the conservative direction — the session stays visible on the recoverable list for an operator, rather than being classified from a partial picture.
4. Classify the rest. A session with a durable SessionStarted bootstrap and no logged side effect is auto-resumed at the boot scan. A session that carries side effects, or predates the durable bootstrap, is not auto-resumed — it is surfaced on a recoverable-sessions list for an operator to resume deliberately, rather than being silently replayed or silently dropped.

Auto-resume rebuilds the session from the bootstrap and resumes its loop at the latest coordinator checkpoint. Unlike a worker-disconnect resume, a restart resume mints fresh child AgentRefs — which is safe precisely because the boot guard refuses to auto-resume any session that has already produced a side effect, so re-running only re-does read-only work. Recovery is a read of the log, not a second state mechanism — the same invariant that makes reconstruction possible makes restart-recovery possible.

Determinism on resume

A resumed session must compose the same agent it started with — the same skills, the same strategy — not whatever the registries happen to hold now. Live catalog state can drift between a session’s start and its resume; a resume that re-resolved against the current catalog could silently pick up an edited or deleted skill mid-flight. Closing that gap is the point of pinning: what a session was built from is frozen at start and re-read on resume. The durable SessionStarted event carries the resolved shape — the coordination kind and folded parameters, the root agent’s budget and tools, the per-role configuration, and the skill resolution — so a resume rebuilds from the record, never from a live lookup. apomesh guarantees this two ways, by how the session started:

• Published-agent sessions compose from a sealed bundle captured when the agent was published — the skill bodies and the strategy configuration, embedded in the immutable, version-pinned record. Composition never reads the live, mutable skill catalog; because a published (name, version) is immutable, re-reading it always yields byte-identical content. A skill the manifest references but the bundle doesn’t carry is a hard, typed error, never a fall-through to current catalog state. See the agent-configuration plane.
• Catalog-started sessions capture the exact skill bodies they resolved at first start into a pin on the SessionStarted event. On resume the session re-folds those pinned bodies byte-for-byte, with zero catalog reads — immune to any edit or delete since the session started.

In both cases a corrupt or undecodable pin fails the resume loudly with a typed error rather than silently substituting whatever the catalog now returns. Fail-loud is the correct posture here: falling through to live resolution would reintroduce the exact mid-flight-edit hazard the pin exists to prevent. The same discipline governs the coordination loop: the resolved kind and folded parameters are pinned at start and replayed without re-resolving, so a recovered session runs the loop it started under.

Where this shows up

• Operate: The operator console shows AgentResuming/AgentResumed in a live session and browses reconstructed historical ones.
• Build: The Python SDK and REST & SSE tail a session across a resume transparently — the same stream continues.
• Reference: the event-bus reconstruction module, the state-store snapshot layer, and the orchestrator recovery workflows under Reference.