Skip to content
apomesh

Durability & determinism

A programmable agent is a long-lived process the platform may stop at any time: an operator pauses it for a confirmation, it goes idle and is evicted to free a container, or the host restarts. When it comes back, the run has to continue — not restart, and not silently diverge.

Two mechanisms carry that. Durability decides how much work survives a stop. Determinism decides whether the work that is re-run observes the same world it observed the first time.

Two durability levels

Section titled “Two durability levels”

DurabilityLevel is declared on the published shape and is load-bearing, not a label.

Checkpointed is the floor every deployment provides. The runtime’s position rides the coordinator checkpoint, and work since the last checkpoint is re-executed on restore. An agent that reported its position at step 6 and stopped at step 9 resumes at step 6 and does 7, 8, 9 again.

Replayable additionally reconstructs post-checkpoint progress without re-executing it. Concretely: generations. That is the family whose re-execution costs real money, and the one the result journal serves.

The other families reconstruct as they do at the floor — a tool call through the side-effect log, the orchestrator-owned mutating ones through the applied-set fold. Replayable is not “everything is replayed”; it is “the expensive step is not re-paid.”

The result journal

Section titled “The result journal”

Replay is served from the function-result journal — a narrow durable store keyed by (session, era, call_seq).

It exists as a separate store because the event log cannot do this job: the log is body-free by design, so it structurally cannot carry a step’s returned value. The journal can, and only the journal does.

Two properties are worth knowing because they are what make replay trustworthy rather than merely fast:

A read compares a request witness, and refuses a disagreement. The key is positional, so the family alone proves nothing. Each entry carries a hash of the decoded request body, and a replay whose request does not match the record is refused rather than answered. Serving a recorded value back to a call that did not make it would be the one outcome worse than a refusal — a swallowed call reported as success.

The reader owns the store. The journal is read worker-side, so the worker also writes it. “Durable, therefore the orchestrator writes it” is the general default and it deliberately does not reach here: the two layers are the same rows only when both happen to bind one durable adapter. On the in-memory and sqlite backends they are independent instances, and the value would be recorded where the read cannot see it — silently.

Idle eviction, and why it is gated

Section titled “Idle eviction, and why it is gated”

A Replayable runtime that has been silent for the worker’s idle grace with no call outstanding on its behalf is evicted, freeing its container. Both halves matter: a runtime parked on a generation the orchestrator has not answered yet is not idle, and evicting it would destroy work the substrate itself was still holding.

Idle eviction never runs for a Checkpointed session. Ungated, it would stop and restart a session that must re-execute and re-bill every step since its last position report — unbounded cost, on a stop nobody asked for. The named trade is the other direction: a quiet Checkpointed runtime holds its container until the session ends.

Determinism: three tiers, no approximations

Section titled “Determinism: three tiers, no approximations”

If a step is replayed, everything it observed must be what it observed the first time. A value whose replay must be identical is a journal entry — there is no cheaper truth. Each tier below either derives its value from durable state or journals it; none approximates, because an approximate mechanism that reads as complete leaves the author an unenforceable “and also do not read X” clause.

Tier 1 — randomness. Derived per step from (session_seed, call_seq). The seed is derived, never minted: a domain-separated SHA-256 over the session id, so it survives an eviction with no durable write and no restore path. Deriving per step rather than running a stream PRNG means the cursor is implied by the call sequence, so no position has to survive a restart. It rides Init as hex text, because a 64-bit value above 2⁵³ silently rounds in a double-backed JSON parser and the peer would never know.

Tier 2 — the coarse logical clock. Derived from the journal: each entry records the platform timestamp its call was served at, and logical time is that stamp. The wall clock is read only for a fresh era’s first observation, so a restored start carries a clock derived from durable state rather than from the restore moment.

Logical time is a property of the step, not a session scalar. A replayed step reports the stamp its own record carries; a fresh journaled serve reports the stamp it minted; only a call that journals nothing reads the session scalar. One scalar cannot serve all three — it would tell a replayed step the era’s maximum instead of its own.

The clock advances only on observation, and that is documented as exactly what it is rather than as an approximation of a real clock: two reads with no intervening call return the same value, which is a true statement about a logical clock.

Tier 3 — ObservedValue. The general mechanism the other two do not cover: a precise timestamp, a uuid, a hostname, an env read. The runtime computes locally, the platform records only, and a replay serves the record. No callback in either direction — the platform never runs the author’s computation and never asks for it again.

Its cost is one durable write per value, which is why it is the author’s opt-in rather than the default path, and why tier 2 must be correct rather than convenient: an approximate clock would push every timestamp onto this tier.

What is not promised

Section titled “What is not promised”

The runtime is a sandboxed process, not a restricted interpreter. Nothing prevents an author reading the OS clock directly, and apomesh does not pretend otherwise. What the three tiers change is that a complete set of correct alternatives now exists, so reaching past them is a choice rather than a gap — and divergence detection makes the consequence a failed session rather than a corrupted one.

The OS clock stays perfectly legitimate for measurements that are not control flow.

Section titled “Related”