Skip to content
apomesh

Credential plane

Agents call providers and tools that need secrets — a provider API key, a web-search service token, a signed cloud credential. The danger is always the same: a secret leaking somewhere it can be read — into an agent’s prompt, onto the event log, into a tool’s process environment. The credential plane exists so that never happens. Every secret in apomesh follows one pattern: it lives in one store, it is never readable by an agent, it is resolved server-side, and it reaches a tool only over a channel proven secure. That one pattern, applied without exception, is what keeps every new secret automatically correct.

What it governs

Section titled “What it governs”
  • Every secret, in one store. Provider keys, tool credentials, and service tokens are all entries in the per-tenant CredentialStore — sealed at rest, tagged with their provenance. There is no second secret home.
  • The agent-readability boundary. A secret value never enters anything the agent’s model can read — not the prompt, not tool-call arguments, not conversation history, working memory, or a checkpoint.
  • Server-side resolution. The orchestrator resolves a credential per tenant and injects it at the sandbox boundary — the agent’s workload never fetches it.
  • Fail-closed forwarding. A secret moves only over a worker hop proven secure; an unproven hop forwards nothing.
  • Typing and redaction. The credential’s kind rides the channel faithfully (never guessed from the secret’s bytes) and every wrapper redacts in Debug.

What it does not govern: who the caller is and what they may do — that is the identity & authorization plane, a separate concern. This plane governs the secret’s lifecycle, not the principal’s.

One store, no parallel home

Section titled “One store, no parallel home”

The load-bearing rule is that there is exactly one secret store: the per-tenant CredentialStore. A Serper key, an Anthropic key, and a service token are the same shape as far as storage is concerned — a ProviderCredential entry keyed by name. The tool plane reaches it through the injected TenantSecrets port; the LLM plane and the console’s credential CRUD reach the same store. One home, one credential type.

Proliferation is the failure mode this forbids. The moment a second secret store exists — an in-memory tool_secrets map beside the store, a separate credential type for one feature — the next secret has to ask which store?, and the weaker one spreads. A single ubiquitous pattern is what makes correctness the default instead of a per-feature decision. New credential adapters (a Vault backend, AWS Secrets Manager) implement the same port; they are never a second store.

The credential lifecycle

Section titled “The credential lifecycle”

A credential is authored, sealed, then resolved and injected — and at no point does its cleartext touch an observable path.

Authored, with provenance. A credential enters the store two ways: an operator writes it on the console’s Credentials surface (OperatorAuthored), or a deployment imports it from config (ConfigImport). Each entry carries a ConfigSource tag, and provenance sets precedence: an operator-authored value outranks an imported one, so a re-import never clobbers a secret an operator set by hand. The substrate ships no credential as a default — there is no seeded secret — so a credential’s provenance is only ever authored or imported.

Sealed at rest. The Postgres adapter seals every secret-bearing field before it is written, using ChaCha20-Poly1305 authenticated encryption with a fresh random nonce per value, so the JSONB row never carries cleartext and two equal secrets seal to distinct ciphertexts. The 32-byte key is operator-provided out of band (never stored beside the ciphertext); a missing or malformed key is a hard boot error — a sealing store refuses to start rather than persist plaintext — and a current-plus-accepted key window makes rotation possible without orphaning existing rows. The apomesh-state-store-postgres crate owns the sealer.

A credential's life: authored with provenance, sealed at rest, resolved per tenant, forwarded only over a secure hop, injected at the sandbox — never on an observable path.

Resolution and injection

Section titled “Resolution and injection”

The orchestrator owns the CredentialStore, so it is the orchestrator that resolves a secret — never the agent, never the worker reaching into the store. When a dispatch carries a tool that declares it needs a secret, the orchestrator resolves every declared name for the session’s tenant, carries each as a typed ResolvedToolSecret (its name, its kind, and the value) on the transient Dispatch envelope, and the worker decodes it into a ForwardedCredential and threads it into the tool’s ToolContext.secrets. The tool reads the credential from that injected channel — never from process environment.

The secret is resolved server-side and injected at the sandbox boundary; it never touches the event log, a checkpoint, or the SSE stream.

The sandbox reinforces the boundary from the other side. Each tool call runs in a per-dispatch subprocess that clears its environment and re-adds only a narrow pass-through allowlist, so an ambient container secret can never leak into a tool — and a tool that once read a vendor key from env must now receive it over ToolContext. The sandbox chain itself is the workers page; this plane only names where the secret enters it.

Fail-closed forwarding

Section titled “Fail-closed forwarding”

Forwarding is fail-closed. A secret is stamped onto the dispatch only when the orchestrator → worker hop is proven secure — a mutual-TLS gRPC connection, or an in-process direct call where there is no wire to intercept. If the hop is not proven secure, the orchestrator forwards nothing: the tool later surfaces a typed configuration error rather than running without its credential. A non-leak is never a crash and never a silent success — it is a typed runtime condition the tool reports.

The one relaxation is a secure-by-default policy for local development, SecretForwardingHopPolicy. SecureOnly is the production default; the AllowInsecureLocalDev opt-in exists only for the plaintext local stack and is loudly warned at boot. The default is absolute in the safe direction: an absent or unknown policy resolves to SecureOnly or a boot error, so forgetting the flag fails closed, never open. This is the sole bypass of the secure-hop requirement, and it never belongs anywhere but deploy/local.

Typed kinds and redaction

Section titled “Typed kinds and redaction”

A credential carries its kind — ApiKey, OAuthBearer, or AwsSigned — as a typed discriminator the whole way through. A consumer that needs the typed credential (reconstructing an auth shape) rebuilds it losslessly from that kind; it is never recovered by sniffing the secret’s bytes (sk-ant-… → OAuth is exactly the guess this forbids). The kind is a non-secret discriminator, safe to show for triage; the value is not.

Redaction is end to end. The in-memory credential type is deliberately non-serializable, so no accidental serde can put a cleartext secret on the wire or a log; the only path to the bytes is an explicit, audited accessor. Every wrapper — the stored credential, the forwarded credential, the tool secret — redacts its value in Debug, so a stray tracing::debug! on the forwarding path prints <redacted>, not a key. The console’s read projection shows only a last-4 fingerprint, never the secret. Signed cloud credentials show only their non-secret access-key id (which already appears in cloud audit logs); the secret key and session token are redacted unconditionally.

The read path redacts too, and that is the half worth checking. Sealing at rest and redacting Debug are theatre if a configuration read hands the value back. Every surface that returns a configured credential — including MCP server transports, whose stdio env values and HTTP auth headers are credentials in every sense — returns a redacted projection, never the value. The reason is a privilege boundary rather than tidiness: listing configured servers needs only a read scope, so a read-scoped caller who got cleartext back could harvest every credential the tenant had configured. That is read-config escalating to credential-harvest, not an over-share.

What an agent can never see

Section titled “What an agent can never see”

Stated crisply, because it is the contract the whole plane exists to keep: a secret value never enters the agent’s model-readable content, and never lands on an observable path. Concretely, it is absent from the prompt, tool-call arguments, conversation history, working memory, and every checkpoint; and it is absent from the event log, the state store, the SSE stream, and any session reconstruction. A secret rides exactly one channel — the transient dispatch-to-runner path — and that channel is the one thing in the substrate deliberately kept off the event log.

The deliberate carve-outs

Section titled “The deliberate carve-outs”

Three narrow paths read a secret outside the tool-forwarding contract, by design, and are honest about it:

  • The at-rest sealing key is read once at boot from the environment. It is deployment infrastructure — the key that seals every row — not an agent-facing tool credential.
  • The LLM/embedding provider plane’s own boot config may read a provider key from the environment for the substrate’s own model calls. These are provider-plane credentials for the substrate’s inference, not tool credentials forwarded to a sandbox — though a tenant’s provider keys still live in the same CredentialStore.
  • A privileged in-process operator tool (an operator-chat fetch running un-sandboxed in the orchestrator) is a single, non-tenant-scoped operator surface, deliberately environment-backed and carved out.

None of these is a second store, and none reaches an agent’s readable content.

Guarantees

Section titled “Guarantees”

These are the contracts you can rely on. This section is the heart of the plane.

  • A secret is never readable by an agent. It never enters model-readable content — prompt, tool-call arguments, conversation history, working memory, or a checkpoint.
  • A secret never lands on an observable path. It rides only the transient dispatch/stdin channel — never the event log, the state store, a checkpoint, the SSE stream, or a reconstruction. Secrets do not project onto any observable mirror, by design.
  • Forwarding is fail-closed. An unproven hop forwards nothing; a missing or unknown forwarding policy resolves to SecureOnly or a boot error. It fails safe, never open, and a non-forward surfaces as a typed config error, never a leak or a crash.
  • One store, per tenant, sealed. Every secret lives in the single CredentialStore, sealed at rest with authenticated encryption and resolved per tenant — no parallel store, no per-feature secret home.
  • Typed and redacted end to end. The credential kind is carried faithfully and reconstructed from the type, never sniffed from the secret’s bytes; every wrapper redacts in Debug, so no cleartext reaches logs or error text.
  • Provenance is respected. An operator-authored credential is never clobbered by a config re-import — provenance sets precedence.

Where this shows up

Section titled “Where this shows up”