> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tofunmiadewuyi.com/custos/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

> The failures that look like something other than what they are.

## SSH

**A connected daemon reports `cached ssh keys: 0`.** Force a resync and read `key_count`:

```bash theme={null}
curl -sS -X POST "$CUSTOS_URL/hosts/$HOST_ID/refresh" -H "Authorization: Bearer $TOKEN"
```

`0` means the control plane itself has no active `host.access` grant plus matching public key — the
problem is the grant or the key, not delivery. Non-zero means delivery; check the daemon is actually
connected.

**sshd is not consulting Custos at all.** Three things have to be true:

```bash theme={null}
cat /etc/ssh/sshd_config.d/70-custos.conf          # the drop-in exists
grep -i '^include' /etc/ssh/sshd_config            # the main config pulls the directory in
sudo sshd -t                                        # the config parses
```

The `Include /etc/ssh/sshd_config.d/*.conf` line is the one Custos will not add for you — editing
the main sshd config is how you lose access to your own server. Without it the drop-in sits there
being ignored, which looks exactly like Custos silently not working.

**Every login is denied, including ones that should work.** Check `AuthorizedKeysCommandUser` is a
real user that can read the daemon's socket and cache. The cache is `0600` and owned by `custos`, so
a different command user cannot read it, `authkeys` prints nothing, and nothing is ever authorized.

**Logins work but nothing appears in the SSH audit.** The daemon is being bypassed and `authkeys` is
falling back to its on-disk cache. Usually the socket path in the drop-in disagrees with the one the
daemon is listening on — see [custosd CLI](/custos/custos/reference/custosd). Access still works, which is why
this one is quiet.

**Revoked access still works.** Custos sets `AuthorizedKeysCommand`, not `AuthorizedKeysFile`. sshd
authorizes if *either* source matches, so a leftover `~/.ssh/authorized_keys` on the host is still
live and Custos cannot revoke it. Remove those files, or set `AuthorizedKeysFile none`.

## Enrollment

**`409` on enroll.** That machine already has an active host. Custos allows one per machine, keyed on
a hash of `/etc/machine-id`. Revoke the existing host first; revoking frees the machine.

**State ends up unreadable by the service.** Enroll as `custos`, not with plain `sudo`, and always
pass an absolute `--dir`. Under `sudo`, `~` expands to root's home and the daemon cannot read what
it wrote.

## API

**Every request returns `400`.** Almost always an encryption mismatch. `CUSTOS_ENCRYPTION` on the
control plane and on the client must agree — a plain-JSON body sent to a server expecting sealed
bytes fails decryption, and sealed bytes sent to a server expecting JSON fail to parse. Both produce
an indistinguishable `400`. See [Encrypted transport](/custos/custos/api-reference/encryption).

**Login works, then everything returns `401`.** `POST /refresh` rotates the refresh token and
invalidates the one you sent. A client that stores only the new access token will fail on its next
refresh.

**`429` out of nowhere.** Unauthenticated endpoints are rate-limited per IP at roughly 10/minute.
Login additionally locks an account for 15 minutes after 5 failures in 15 minutes. The lock clears
on its own — it is a window, not a hard lockout, because a hard lockout is a denial-of-service tool
pointed at your own users.

**`503` on anything touching a secret.** The vault key wrapper is not configured — check
`CUSTOS_MASTER_KEY`. Custos returns `503` rather than storing plaintext.

**`503` on invitations or password reset.** `CUSTOS_APP_URL` is unset, so no link can be built.

## Secret sets

**`403` revealing a set.** It is private. Private sets are deployable and never readable by a person;
only public sets can be revealed. See [Machine secrets](/custos/custos/guides/machine-secrets).

**`409` making a set public.** It is bound to at least one host. Unbind them all first.

**`409` deleting a set, an entry, or a binding.** A backup job depends on it.

## Upgrades

**`422` on an upgrade.** The named release does not exist, or is incomplete, in the release catalog.

**`502` on an upgrade.** The control plane could not reach the release catalog at all. That is
`CUSTOS_RELEASE_BASE` or the artifact host, not your request.

## Control plane service

**`status=217/USER`.** The service user does not exist. systemd aborts before `ExecStart`, so the
logs show nothing useful about Custos itself.

**The service will not start after a config change.** If you use the encrypted credential store,
`LoadCredentialEncrypted` needs the credential ID, the filename, and the name baked into the
ciphertext by `systemd-creds encrypt --name=` to agree. Renaming the file alone is not enough.

**`/readyz` returns `503`.** Postgres is unreachable. `/livez` still answering means the process is
fine; the dependency is not.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.