> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tofunmiadewuyi.com/custos/llms.txt
> Use this file to discover all available pages before exploring further.

# Daemon protocol

> The enrollment handshake, the WebSocket session, and what secures each message.

Two phases: a **one-shot HTTP** enrollment (and decommission), then a **long-lived
WebSocket** session the daemon dials out on (hosts need no inbound ports). Every WS message is a
JSON **`Envelope`**; the payload and its security properties depend on the `Type`.

## Endpoints & transport

| Method | Path | Purpose |
| - | - | - |
| `POST` | `/enroll` | one-shot registration; exchanges an admin token for a host id |
| `POST` | `/hosts/{id}/decommission` | one-shot signed goodbye; revokes the host |
| `WS` | `/daemon` | the long-lived session (daemon upgrades its URL: `https→wss`, `http→ws`) |

Timings that keep the link honest:

* CP pings every **30 s** (`TypePing`); the daemon answers `TypePong`.
* CP read timeout **90 s**; daemon read timeout **60 s** — a silent link (no message, not even a
  ping) is treated as dead and torn down.
* Daemon read cap **1 MiB** (snapshots can be large).
* Daemon reconnect: exponential backoff, 2 s doubling, **capped at 30 s**, with jitter so a CP
  restart doesn't thundering-herd the fleet. Reset on a successful auth.

## The envelope

```go theme={null}
type Envelope struct {
    Type         MessageType       // which message; selects how Data is read
    Data         json.RawMessage   // the payload, always the JSON of that type's struct
    Seq          uint64            // monotonic per-host; anti-replay on signed types
    Sig          []byte            // Ed25519 signature, present on signed types
    TraceContext map[string]string // W3C traceparent/tracestate; omitted when untraced
}
```

`TraceContext` is diagnostic metadata, not authorization or application data. Custos propagates
only W3C `traceparent` and `tracestate`—never OpenTelemetry baggage. It is not part of snapshot or
secret-set signing input, so it carries no security meaning; the receiver still independently
verifies the signed `Data` and `Seq` fields.

## Two trust layers

1. **WSS / TLS** — protects *every* message hop-by-hop and authenticates the *server*. May terminate
   at a proxy in front of the control plane.
2. **App-level sign / seal** — end-to-end **CP ↔ host**, survives that proxy. Only some messages carry
   it, by need (below).

So app-level crypto exists exactly where an end-to-end guarantee is required: `snapshot`/`secret_sets`
are **signed** so authenticity + `seq` hold even past a TLS-terminating proxy; `secret_sets` is also
**sealed** so the secrets stay confidential end-to-end (a proxy sees only the blob).

## Message table

| Message | Dir | Encrypted | Signed | What secures it |
| - | - | - | - | - |
| `ping` / `pong` | both | no | no | nothing — keepalive |
| `challenge` | CP → d | no | no | a random nonce; nothing to forge |
| `auth` | d → CP | no | **signed by the daemon** (host identity) | host proves who it is |
| `snapshot` | CP → d | no | **signed by CP** | authenticity + replay (public keys, no need to hide) |
| `upgrade` | CP → d | no | no | WSS + the sha256 digest in the payload anchors the download |
| `secret_sets` | CP → d | **yes** (to host X25519) | **signed by CP** | confidential *and* authentic end-to-end |
| `access_log` | d → CP | no | no | the authenticated session |
| `secret_read` | d → CP | no | no | the authenticated session |

`grant` / `revoke` types exist in the protocol but aren't in the live path — bulk revoke re-sends a
full `snapshot`.

## Enrollment (HTTP, one-shot)

The daemon's `Enroll` generates a **fresh** Ed25519 identity keypair and an X25519 encryption keypair
(the private halves never leave the machine), then `POST`s them to `/enroll`:

```
EnrollRequest { token, hostname, public_key, encryption_key, machine_id, prior_host_id? }
```

* `token` — admin-issued, **single-use** (consumed in the same transaction as host creation).
* `machine_id` — a sha256 hash of `/etc/machine-id` (or fallbacks); empty for app containers.
* `prior_host_id` — the previous host id on a re-enroll; **dedup fallback** when `machine_id` is empty.

The server runs token validation + host creation in one transaction. It enforces **one active host per
machine**: if `machine_id` (or an active `prior_host_id`) already maps to an active host, the enroll is
rejected with `409` — a revoked host frees the machine to re-enroll. Response:

```
EnrollResponse { host_id, signing_public_key }   // CP's snapshot-signing Ed25519 pubkey, pinned by the daemon
```

The daemon persists keys **before** config, so a rejected enroll leaves the working identity intact; the
CP is the authority on duplicates.

## The handshake (challenge / auth)

Authenticates the *host* to the CP (the CP is trusted via TLS — the daemon dialed a known URL):

* CP → daemon: `challenge` = `{ nonce }` (32 random bytes, unsigned).
* daemon → CP: `auth` = `{ host_id, signature, version }` — the daemon signs the nonce with its
  **host identity Ed25519 key** under the `custos-host-auth:v1:` domain prefix. The CP verifies against
  the identity public key registered at enroll and rejects hosts whose status isn't `active`.

`version` is the daemon's build; the CP persists it as `hosts.agent_version` and re-pushes an `upgrade`
to hosts that are still behind `desired_version`.

## Keys

Three keypairs, registered (public halves) at enroll:

| Keypair | Alg | Private held by | Public held by | Used for |
| - | - | - | - | - |
| host identity | Ed25519 | daemon | CP (`hosts.identity_key`) | daemon signs the `challenge` nonce in `auth`; decommission |
| CP signing | Ed25519 | CP | daemon (`SigningPublicKey`) | CP signs `snapshot` and `secret_sets` |
| host encryption | X25519 | daemon (`encryption.key`) | CP (`hosts.encryption_key`) | CP seals `secret_sets` to the host; daemon opens |

## Signing input & sequence

Signed messages sign `tag ‖ len16(hostID) ‖ hostID ‖ seq(8B) ‖ Data`, each type under its own
domain-separation tag so a signature for one can't be replayed as another:

* `snapshot` → `custos-snapshot:v1`, sequenced by `hosts.last_access_seq`; applied/known seq tracked in `hosts.acked_access_seq`.
* `secret_sets` → `custos-sets:v1`, sequenced by `hosts.last_set_seq` (separate counter); applied/known seq tracked in `hosts.acked_set_seq`.

The daemon **verifies the signature before it decrypts**, and drops any `seq ≤` the last it applied.

## `secret_sets` wire shape

The whole bundle is sealed **once** — set names and `as_user` live inside the ciphertext, so nothing
leaks:

```
Envelope { type: "secret_sets", seq, sig, data }

  sig  = Ed25519( "custos-sets:v1" ‖ len16(hostID) ‖ hostID ‖ seq(8B) ‖ data )
  data = { "sealed": <blob, base64> }
  blob = eph_pub(32) ‖ nonce(12) ‖ AES-256-GCM( JSON(SecretSets) ) ‖ tag(16)

  SecretSets = { sets: [ { name, as_user, version, values:{KEY:val,…} }, … ] }
```

The seal is X25519 ECDH (ephemeral CP key × host `ENC_pub`) → HKDF-SHA256 (both public keys as salt) →
AES-256-GCM. One ephemeral key per push; its public half is the first 32 bytes of the blob. The GCM tag
proves integrity but not *sender* — anyone with the host's public key could forge a valid-looking sealed
blob — which is why the Ed25519 signature (authenticity) and `seq` (replay) are also required.
Verify-sig-then-decrypt.

## Connection lifecycle

**Connect.** The daemon dials `/daemon`, runs the challenge/auth handshake, then the CP registers the
connection in the hub and immediately queues, in order:

1. a signed `snapshot` — the full authoritative key set; the daemon **replaces** its cache with it;
2. a sealed `secret_sets` bundle (only if the host registered an encryption key) — reconciled on every
   connect;
3. an `upgrade` if `desired_version ≠ agent_version` — so a host that was offline for its upgrade gets
   it on reconnect.

**Live.** The hub holds one `send` channel per connected host. Every message goes through a single
writer goroutine (CP pings + pushed envelopes); daemon-side, `pong`s and queued access logs funnel
through one writer too. The hub `push` is **non-blocking**: a full buffer is dropped, because the daemon
reconciles from the snapshot on its next (re)connect anyway.

**Disconnect.** The CP writer flushes any still-queued envelopes (e.g. a revoke's purge) before the
close completes. The daemon reconnects with jittered exponential backoff; on success the backoff resets
and the snapshot reconciliation happens again.

**Rejection (revoke).** Revoking a host sends an **empty snapshot** (the daemon's whole authorized set
becomes nothing) then force-disconnects it. The daemon sees the `StatusPolicyViolation` close — a
deliberate rejection, not a transient drop — and **purges its local access cache**, then backs off at
the maximum so it doesn't hammer a CP that has disowned it. A decommissioned host re-enrolls fresh and
must re-earn its grants.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.