SSH
A connected daemon reportscached ssh keys: 0. Force a resync and read key_count:
0 means the control plane itself has no active host.access grant plus matching public key — the
problem is the grant or the key, not delivery. Non-zero means delivery; check the daemon is actually
connected.
sshd is not consulting Custos at all. Three things have to be true:
Include /etc/ssh/sshd_config.d/*.conf line is the one Custos will not add for you — editing
the main sshd config is how you lose access to your own server. Without it the drop-in sits there
being ignored, which looks exactly like Custos silently not working.
Every login is denied, including ones that should work. Check AuthorizedKeysCommandUser is a
real user that can read the daemon’s socket and cache. The cache is 0600 and owned by custos, so
a different command user cannot read it, authkeys prints nothing, and nothing is ever authorized.
Logins work but nothing appears in the SSH audit. The daemon is being bypassed and authkeys is
falling back to its on-disk cache. Usually the socket path in the drop-in disagrees with the one the
daemon is listening on — see custosd CLI. Access still works, which is why
this one is quiet.
Revoked access still works. Custos sets AuthorizedKeysCommand, not AuthorizedKeysFile. sshd
authorizes if either source matches, so a leftover ~/.ssh/authorized_keys on the host is still
live and Custos cannot revoke it. Remove those files, or set AuthorizedKeysFile none.
Enrollment
409 on enroll. That machine already has an active host. Custos allows one per machine, keyed on
a hash of /etc/machine-id. Revoke the existing host first; revoking frees the machine.
State ends up unreadable by the service. Enroll as custos, not with plain sudo, and always
pass an absolute --dir. Under sudo, ~ expands to root’s home and the daemon cannot read what
it wrote.
API
Every request returns400. Almost always an encryption mismatch. CUSTOS_ENCRYPTION on the
control plane and on the client must agree — a plain-JSON body sent to a server expecting sealed
bytes fails decryption, and sealed bytes sent to a server expecting JSON fail to parse. Both produce
an indistinguishable 400. See Encrypted transport.
Login works, then everything returns 401. POST /refresh rotates the refresh token and
invalidates the one you sent. A client that stores only the new access token will fail on its next
refresh.
429 out of nowhere. Unauthenticated endpoints are rate-limited per IP at roughly 10/minute.
Login additionally locks an account for 15 minutes after 5 failures in 15 minutes. The lock clears
on its own — it is a window, not a hard lockout, because a hard lockout is a denial-of-service tool
pointed at your own users.
503 on anything touching a secret. The vault key wrapper is not configured — check
CUSTOS_MASTER_KEY. Custos returns 503 rather than storing plaintext.
503 on invitations or password reset. CUSTOS_APP_URL is unset, so no link can be built.
Secret sets
403 revealing a set. It is private. Private sets are deployable and never readable by a person;
only public sets can be revealed. See Machine secrets.
409 making a set public. It is bound to at least one host. Unbind them all first.
409 deleting a set, an entry, or a binding. A backup job depends on it.
Upgrades
422 on an upgrade. The named release does not exist, or is incomplete, in the release catalog.
502 on an upgrade. The control plane could not reach the release catalog at all. That is
CUSTOS_RELEASE_BASE or the artifact host, not your request.
Control plane service
status=217/USER. The service user does not exist. systemd aborts before ExecStart, so the
logs show nothing useful about Custos itself.
The service will not start after a config change. If you use the encrypted credential store,
LoadCredentialEncrypted needs the credential ID, the filename, and the name baked into the
ciphertext by systemd-creds encrypt --name= to agree. Renaming the file alone is not enough.
/readyz returns 503. Postgres is unreachable. /livez still answering means the process is
fine; the dependency is not.