Daemon protocol

Two phases: a one-shot HTTP enrollment (and decommission), then a long-lived WebSocket session the daemon dials out on (hosts need no inbound ports). Every WS message is a JSON Envelope; the payload and its security properties depend on the Type.

Endpoints & transport

Method	Path	Purpose
`POST`	`/enroll`	one-shot registration; exchanges an admin token for a host id
`POST`	`/hosts/{id}/decommission`	one-shot signed goodbye; revokes the host
`WS`	`/daemon`	the long-lived session (daemon upgrades its URL: `https→wss`, `http→ws`)

Timings that keep the link honest:

CP pings every 30 s (TypePing); the daemon answers TypePong.
CP read timeout 90 s; daemon read timeout 60 s — a silent link (no message, not even a ping) is treated as dead and torn down.
Daemon read cap 1 MiB (snapshots can be large).
Daemon reconnect: exponential backoff, 2 s doubling, capped at 30 s, with jitter so a CP restart doesn’t thundering-herd the fleet. Reset on a successful auth.

The envelope

type Envelope struct {
    Type         MessageType       // which message; selects how Data is read
    Data         json.RawMessage   // the payload, always the JSON of that type's struct
    Seq          uint64            // monotonic per-host; anti-replay on signed types
    Sig          []byte            // Ed25519 signature, present on signed types
    TraceContext map[string]string // W3C traceparent/tracestate; omitted when untraced
}

TraceContext is diagnostic metadata, not authorization or application data. Custos propagates only W3C traceparent and tracestate—never OpenTelemetry baggage. It is not part of snapshot or secret-set signing input, so it carries no security meaning; the receiver still independently verifies the signed Data and Seq fields.

Two trust layers

WSS / TLS — protects every message hop-by-hop and authenticates the server. May terminate at a proxy in front of the control plane.
App-level sign / seal — end-to-end CP ↔ host, survives that proxy. Only some messages carry it, by need (below).

So app-level crypto exists exactly where an end-to-end guarantee is required: snapshot/secret_sets are signed so authenticity + seq hold even past a TLS-terminating proxy; secret_sets is also sealed so the secrets stay confidential end-to-end (a proxy sees only the blob).

Message table

Message	Dir	Encrypted	Signed	What secures it
`ping` / `pong`	both	no	no	nothing — keepalive
`challenge`	CP → d	no	no	a random nonce; nothing to forge
`auth`	d → CP	no	signed by the daemon (host identity)	host proves who it is
`snapshot`	CP → d	no	signed by CP	authenticity + replay (public keys, no need to hide)
`upgrade`	CP → d	no	no	WSS + the sha256 digest in the payload anchors the download
`secret_sets`	CP → d	yes (to host X25519)	signed by CP	confidential and authentic end-to-end
`access_log`	d → CP	no	no	the authenticated session
`secret_read`	d → CP	no	no	the authenticated session

grant / revoke types exist in the protocol but aren’t in the live path — bulk revoke re-sends a full snapshot.

Enrollment (HTTP, one-shot)

The daemon’s Enroll generates a fresh Ed25519 identity keypair and an X25519 encryption keypair (the private halves never leave the machine), then POSTs them to /enroll:

EnrollRequest { token, hostname, public_key, encryption_key, machine_id, prior_host_id? }

token — admin-issued, single-use (consumed in the same transaction as host creation).
machine_id — a sha256 hash of /etc/machine-id (or fallbacks); empty for app containers.
prior_host_id — the previous host id on a re-enroll; dedup fallback when machine_id is empty.

The server runs token validation + host creation in one transaction. It enforces one active host per machine: if machine_id (or an active prior_host_id) already maps to an active host, the enroll is rejected with 409 — a revoked host frees the machine to re-enroll. Response:

EnrollResponse { host_id, signing_public_key }   // CP's snapshot-signing Ed25519 pubkey, pinned by the daemon

The daemon persists keys before config, so a rejected enroll leaves the working identity intact; the CP is the authority on duplicates.

The handshake (challenge / auth)

Authenticates the host to the CP (the CP is trusted via TLS — the daemon dialed a known URL):

CP → daemon: challenge = { nonce } (32 random bytes, unsigned).
daemon → CP: auth = { host_id, signature, version } — the daemon signs the nonce with its host identity Ed25519 key under the custos-host-auth:v1: domain prefix. The CP verifies against the identity public key registered at enroll and rejects hosts whose status isn’t active.

version is the daemon’s build; the CP persists it as hosts.agent_version and re-pushes an upgrade to hosts that are still behind desired_version.

Keys

Three keypairs, registered (public halves) at enroll:

Keypair	Alg	Private held by	Public held by	Used for
host identity	Ed25519	daemon	CP (`hosts.identity_key`)	daemon signs the `challenge` nonce in `auth`; decommission
CP signing	Ed25519	CP	daemon (`SigningPublicKey`)	CP signs `snapshot` and `secret_sets`
host encryption	X25519	daemon (`encryption.key`)	CP (`hosts.encryption_key`)	CP seals `secret_sets` to the host; daemon opens

Signing input & sequence

Signed messages sign tag ‖ len16(hostID) ‖ hostID ‖ seq(8B) ‖ Data, each type under its own domain-separation tag so a signature for one can’t be replayed as another:

snapshot → custos-snapshot:v1, sequenced by hosts.last_access_seq; applied/known seq tracked in hosts.acked_access_seq.
secret_sets → custos-sets:v1, sequenced by hosts.last_set_seq (separate counter); applied/known seq tracked in hosts.acked_set_seq.

The daemon verifies the signature before it decrypts, and drops any seq ≤ the last it applied.

`secret_sets` wire shape

The whole bundle is sealed once — set names and as_user live inside the ciphertext, so nothing leaks:

Envelope { type: "secret_sets", seq, sig, data }

  sig  = Ed25519( "custos-sets:v1" ‖ len16(hostID) ‖ hostID ‖ seq(8B) ‖ data )
  data = { "sealed": <blob, base64> }
  blob = eph_pub(32) ‖ nonce(12) ‖ AES-256-GCM( JSON(SecretSets) ) ‖ tag(16)

  SecretSets = { sets: [ { name, as_user, version, values:{KEY:val,…} }, … ] }

The seal is X25519 ECDH (ephemeral CP key × host ENC_pub) → HKDF-SHA256 (both public keys as salt) → AES-256-GCM. One ephemeral key per push; its public half is the first 32 bytes of the blob. The GCM tag proves integrity but not sender — anyone with the host’s public key could forge a valid-looking sealed blob — which is why the Ed25519 signature (authenticity) and seq (replay) are also required. Verify-sig-then-decrypt.

Connection lifecycle

Connect. The daemon dials /daemon, runs the challenge/auth handshake, then the CP registers the connection in the hub and immediately queues, in order:

a signed snapshot — the full authoritative key set; the daemon replaces its cache with it;
a sealed secret_sets bundle (only if the host registered an encryption key) — reconciled on every connect;
an upgrade if desired_version ≠ agent_version — so a host that was offline for its upgrade gets it on reconnect.

Live. The hub holds one send channel per connected host. Every message goes through a single writer goroutine (CP pings + pushed envelopes); daemon-side, pongs and queued access logs funnel through one writer too. The hub push is non-blocking: a full buffer is dropped, because the daemon reconciles from the snapshot on its next (re)connect anyway. Disconnect. The CP writer flushes any still-queued envelopes (e.g. a revoke’s purge) before the close completes. The daemon reconnects with jittered exponential backoff; on success the backoff resets and the snapshot reconciliation happens again. Rejection (revoke). Revoking a host sends an empty snapshot (the daemon’s whole authorized set becomes nothing) then force-disconnects it. The daemon sees the StatusPolicyViolation close — a deliberate rejection, not a transient drop — and purges its local access cache, then backs off at the maximum so it doesn’t hammer a CP that has disowned it. A decommissioned host re-enrolls fresh and must re-earn its grants.

Getting started

Concepts

Control plane

Hosts

Web app

Guides

Operations

Reference

Internals

Daemon protocol

Endpoints & transport

The envelope

Two trust layers

Message table

Enrollment (HTTP, one-shot)

The handshake (challenge / auth)

Keys

Signing input & sequence

`secret_sets` wire shape

Connection lifecycle

​Endpoints & transport

​The envelope

​Two trust layers

​Message table

​Enrollment (HTTP, one-shot)

​The handshake (challenge / auth)

​Keys

​Signing input & sequence

​secret_sets wire shape

​Connection lifecycle

Endpoints & transport

The envelope

Two trust layers

Message table

Enrollment (HTTP, one-shot)

The handshake (challenge / auth)

Keys

Signing input & sequence

`secret_sets` wire shape

Connection lifecycle