Monitoring

You observe apomesh two ways, and only two: you subscribe to the event log, or you read the daemons’ process logs. There is no hidden metrics plane — no built-in /metrics endpoint on the daemon, no statsd push, no separate telemetry channel. What a session did derives from the one per-session event log; how the daemons are running comes from their tracing output. This page shows how to stand both up: watch a session live, forward the log into a metrics stack, grep the daemon logs, and check liveness.

The event log itself — the envelope, the closed Payload vocabulary, who may emit, and what deliberately never rides it — is documented once, on The event log; this page is the operator how-to on top of it and does not re-explain the vocabulary.

The observation model

Rendering diagram…

Two observation sources — the durable event log (via the tail surfaces) and the daemons' process logs

A tail consumer is only ever as complete as the events the substrate emits — which is the point of the log-as-single-source-of-truth: every dashboard, audit trail, or billing rollup is a projection of it, never a side channel.

Watch a session live

Three surfaces tail the same log over one subscription primitive. They differ in who consumes them and in framing — not in what they show or what they guarantee.

• The operator console — the desktop control plane consumes the gRPC tail and renders the live agent tree, spend, and event stream interactively. Reach for it when a human is watching a session unfold. See The operator console.
• gRPC TailEvents — the server-streaming RPC on the daemon’s programmatic wire (50051 in the local stack). This is what a Rust or SDK consumer uses. It streams the durable event log from the state store: it backfills every event already appended for the session, then stays live. Because it reads the persisted log rather than a live actor, the tail survives a session that has already terminated.
• REST SSE — GET /api/v1/events?session=<id> on the REST listener (50052), a text/event-stream tail for browsers and any HTTP client. Each frame carries event: (the payload variant, snake_case), data: (the JSON projection), and id: (the position token). See REST + SSE.

Both wire tails are session-scoped and tenant-isolated: a subscriber sees only sessions its tenant owns, and an unknown or cross-tenant session_id yields an empty stream rather than an error — the same opacity everywhere. There is no cross-session or cross-tenant subscription primitive; a per-tenant rollup discovers sessions with ListSessions (gRPC) / GET /api/v1/sessions (REST) and subscribes to each.

Both wire paths are thin adapters over one subscription primitive, so they carry identical guarantees — the properties that decide whether your monitoring data can be trusted:

• No dropped events. The primitive advances a subscriber’s position only over entries it has read back from the durable log, so a consumer that falls behind gets a slower tail, never a gappy one. There is no dropped-event case to detect, and no reason for a metrics consumer to reconcile against a second source.
• Exact resume. Every event carries its position: a gRPC consumer reads it off the streamed item, an SSE consumer off the frame’s id:. Reconnect with the last one you handled — from_position on gRPC, the Last-Event-ID header (or the from= query parameter) on SSE — and you receive exactly the remainder. No gap, no double-counting, which is what makes a restarted counter-based consumer correct. A browser EventSource does this automatically.
• Correct late start. A tail opened against a session already in flight backfills what it missed before going live, so a consumer started mid-session still sees the whole history.

Build a metrics consumer

To get metrics, you run a tail consumer: a process that subscribes to the log and maps the variants it cares about into your monitoring stack. The substrate ships a worked reference for Prometheus at samples/tail-consumer-prometheus/. It connects over gRPC, subscribes to one session, and re-exposes the operator-relevant variants — cost, dispatch latency, lifecycle, backpressure, HITL, job progress, and more — as counters and histograms on a local /metrics endpoint, labelled by tenant (and, where relevant, provider and affinity).

Start a session first (via the console, an SDK, or POST /api/v1/sessions), then point the sample at its id — the tail is session-scoped:

cargo run -p apomesh-sample-tail-consumer-prometheus -- \
    --daemon-addr http://127.0.0.1:50051 \
    --session-id <session-id> \
    --metrics-listen 127.0.0.1:9090

Confirm the endpoint is serving:

curl -s http://127.0.0.1:9090/metrics | head -40

Point Prometheus at it with a static scrape target:

scrape_configs:
  - job_name: apomesh
    static_configs:
      - targets: ["127.0.0.1:9090"]

In Grafana, add the Prometheus datasource and query the metric names directly — for example, p99 dispatch overhead per tenant:

histogram_quantile(0.99, sum(rate(apomesh_dispatch_latency_total_overhead_us_bucket[5m])) by (tenant, le))

The sample’s README lists every metric it emits and more starter queries, and its src/main.rs is the pattern for any consumer: a single wildcard-free match over the closed Payload enum, so a new wire variant breaks compilation until you decide how to forward it. The same shape composes against OTLP, Kafka, or a direct database sink — the substrate emits, your consumer subscribes.

Read the daemon logs

Both the orchestrator and worker log to stdout/stderr, filtered by RUST_LOG (an EnvFilter string, default info). The local stack sets a per-crate filter in docker-compose.yml; tail it with docker compose logs -f orchestrator (or worker).

At boot, a healthy orchestrator emits a recognisable sequence — grep these to confirm the daemon came up configured:

• orchestrator starting — the process is up; the line carries the gRPC listen address.
• llm config store initialised (in-memory, shipped seed) — the model-config plane loaded (or (postgres, shipped seed persisted) with a database).
• LLM provider router configured from the store (config-import) — providers resolved; the line names the seeded providers and the chosen smart / fast model pins. An empty provider list here is why dispatches fail with no model.
• agent catalog initialised (in-memory, substrate-shipped seed) — the launch catalog seeded, with seed_count.
• starting both listeners — the gRPC and REST listeners are binding; the line carries both grpc and rest addresses. Under TLS you also see loading gRPC TLS server config just before it.

During an incident, the worker-facing lines matter most. heartbeat received (orchestrator side) confirms a worker is live; worker disconnected and worker stream error mark a lost worker — the trigger for the substrate’s heartbeat-based resume path. To trace a specific session more deeply, raise its level, e.g. RUST_LOG=info,apomesh_orchestrator::session=debug.

Note

Process logs are for human triage, not substrate state. Cost, lifecycle, and every other operationally-significant fact rides the event log, never a log line — so a metrics consumer reads the log, never scrapes stdout.

Liveness and readiness

The daemon exposes one health surface: GET /api/v1/health on the REST listener, a public route (no auth, even under the oauth profile) that returns 200 with {"status":"ok"} when the daemon is servicing requests.

curl -sSf http://localhost:50052/api/v1/health

Be honest about what it is: a shallow liveness probe proving the process is up and the REST listener is accepting requests. It is not a deep readiness check — it does not verify the state store is reachable, providers are credentialled, or a worker has registered; those richer states are a planned additive extension, not shipped today. There is also no gRPC health service and no daemon-side metrics endpoint — metrics exist only in a tail consumer you run (above). Point a load balancer’s liveness probe at /api/v1/health, but do not treat a 200 as proof the daemon can dispatch.

Latency and mTLS references

Two operator concerns have their own in-repo methodology references — read them there rather than re-deriving:

• docs/operational/latency-baselines.md — how to measure and interpret the per-dispatch overhead the DispatchLatency event carries. The substrate ships a cassette-replay regression-detector test that establishes a substrate-overhead floor; you layer a real-API measurement on top with your own provider keys. Treat that test as a drift monitor — refreshing the cassettes and re-running it (alongside the live-provider tests) catches an overhead regression before it reaches a deployment. The deterministic-replay posture those cassettes deliver is covered on Replay & history.
• docs/operational/mtls-rotation.md — how the daemon consumes mTLS material and the operator procedure for rotating it. Note the current constraint: the daemon reads its cert/key/CA once at startup and does not hot-reload, so a rotation is a restart, not a signal.

Where this shows up

• Concepts: The event log — the canonical home for the emission surface and the Payload vocabulary every consumer here projects.
• Operate: Budgets & cost reads the same CostEvent stream this page forwards; the console is the interactive tail consumer.
• Build: REST + SSE is the HTTP tail; the Python SDK wraps the gRPC tail.