> ## Documentation Index
> Fetch the complete documentation index at: https://docs.tofunmiadewuyi.com/custos/llms.txt
> Use this file to discover all available pages before exploring further.

# Running under systemd

> Run custoscp as a systemd service with secrets in the encrypted credential store.

Run `custoscp serve` as a systemd service, with secrets held in systemd's
encrypted credential store instead of a plaintext env file. The unit template is
[the unit file below](#the-unit-file).

The control plane reads all config from **environment variables** (see [Configuration](/custos/custos/control-plane/configuration) for the full contract). Non-secret values go in the unit
as `Environment=`; secrets are encrypted at rest and decrypted into a per-service
tmpfs at runtime.

## 1. Service user

The unit runs as a dedicated locked system user. Missing it is the `status=217/USER`
failure — systemd aborts before `ExecStart`.

```bash theme={null}
sudo useradd --system --no-create-home --shell /usr/sbin/nologin custoscp
```

## 2. Generate keys

`gen-keys` prints an env-format bundle. Redirect to a root-only file so it never
hits terminal scrollback or shell history:

```bash theme={null}
custoscp gen-keys > /root/custos-keys && chmod 600 /root/custos-keys
```

The bundle contains:

```text theme={null}
CUSTOS_MASTER_KEY=...
CUSTOS_SIGNING_PRIVATE_KEY=...
CUSTOS_SERVER_TRANSPORT_PRIVATE_KEY=...
CUSTOS_SERVER_TRANSPORT_PUBLIC_KEY=...    # set on clients, not read by the CP
```

Do everything below as root (the bundle and credstore are root-only).

## 3. Encrypt secrets into the credential store

The idea is the same for every secret: **put it in a file, encrypt the file,
delete the file.** `systemd-creds` encrypts with a host/TPM key only root can
access. Each output file in `/etc/credstore.encrypted/` is found by its filename,
which must match the credential ID in the unit.

First make the store folder:

```bash theme={null}
sudo install -d -m 0700 /etc/credstore.encrypted   # root-only directory
```

The three CP keys are already in the bundle as `NAME=value` lines. This loop
grabs each value (the part after `=`) and encrypts it:

```bash theme={null}
for k in CUSTOS_MASTER_KEY CUSTOS_SIGNING_PRIVATE_KEY CUSTOS_SERVER_TRANSPORT_PRIVATE_KEY; do
  grep "^$k=" /root/custos-keys | cut -d= -f2- \
    | sudo systemd-creds encrypt --name="$k" - /etc/credstore.encrypted/"$k"
done
```

The DB URL and Resend key aren't in the bundle — put each in its own file with
an editor, then encrypt that file:

```bash theme={null}
nano /root/dburl     # paste the Supabase session-pooler URI, Ctrl-O to save, Ctrl-X to exit
sudo systemd-creds encrypt --name=CUSTOS_DATABASE_URL /root/dburl /etc/credstore.encrypted/CUSTOS_DATABASE_URL

nano /root/resend    # paste the Resend API key, save, exit
sudo systemd-creds encrypt --name=RESEND_API_KEY /root/resend /etc/credstore.encrypted/RESEND_API_KEY
```

Then delete every plaintext file:

```bash theme={null}
shred -u /root/custos-keys /root/dburl /root/resend
```

Notes:

* Secrets go into **files**, never onto a command line, so nothing lands in
  shell history or `ps`. After `shred`, the plaintext exists only in the running
  service's memory.
* A trailing newline from the editor is harmless: the service reads each value
  with `$(cat ...)`, which strips it.

Gotchas that cost time:

* **`--name` = credential ID = filename.** Mismatch fails at service start.
* **Run `systemd-creds` as root** (`sudo`). Without it you get
  `InteractiveAuthenticationRequired`; if you instead put `sudo` on the wrong
  command in a pipe (e.g. `sudo echo ... | systemd-creds`), only `echo` is root
  and `systemd-creds` still fails.

The `.cred` files are ciphertext, tied to this host's key; they do not decrypt
elsewhere.

## 4. Database migrations

`migrate` reads `CUSTOS_DATABASE_URL`. Decrypt it from the store straight into the
command's environment — never onto the command line (which would hit history and
`/proc/<pid>/cmdline`):

```bash theme={null}
CUSTOS_DATABASE_URL="$(sudo systemd-creds decrypt /etc/credstore.encrypted/CUSTOS_DATABASE_URL -)" \
  custoscp migrate up
```

History records the `decrypt` call, not the URL; the value enters only
`custoscp`'s environment. Run this once before first start and after upgrades
that ship migrations. (Supabase on an IPv4-only host: use the session-pooler URI;
the direct endpoint is IPv6-only.)

## 5. First admin

Same pattern:

```bash theme={null}
CUSTOS_DATABASE_URL="$(sudo systemd-creds decrypt /etc/credstore.encrypted/CUSTOS_DATABASE_URL -)" \
  custoscp create-admin --email you@example.com
```

## 6. Install and start

```bash theme={null}
# write the unit from "The unit file" below to /etc/systemd/system/custoscp.service,
# editing the domain values in the [Service] Environment= lines, then:
sudo systemctl daemon-reload
sudo systemctl enable --now custoscp
systemctl status custoscp
journalctl -u custoscp -f
```

Control-plane upgrades are documented separately in
[Upgrading the control plane](/custos/custos/control-plane/upgrading).

## Per-deployment values

Edit these in the unit before installing:

* `CUSTOS_APP_URL`, `CUSTOS_CORS_ORIGINS` — the public frontend origin.
* `CUSTOS_EMAIL_FROM` — a Resend-verified sender.
* `CUSTOS_LISTEN_ADDR` — loopback; host Nginx terminates TLS and proxies here.
* `OTEL_EXPORTER_OTLP_ENDPOINT` — Alloy's loopback OTLP receiver
  (`http://127.0.0.1:4318`). Because the CP runs on the host, `127.0.0.1`
  reaches Alloy's published port. A containerized CP would use `http://alloy:4318`
  on the telemetry network instead. See [the observability directory](https://github.com/tofunmiadewuyi/custos/tree/main/observability).

## Rotating a secret

Put the new value in a file, re-encrypt over the same credential, then restart:

```bash theme={null}
nano /root/newsecret     # paste the new value, save, exit
sudo systemd-creds encrypt --name=RESEND_API_KEY /root/newsecret /etc/credstore.encrypted/RESEND_API_KEY
shred -u /root/newsecret
sudo systemctl restart custoscp
```

If a key was ever exposed on a terminal or command line, rotate it rather than
trusting scrollback/history cleanup.

## The unit file

Install this to `/etc/systemd/system/custoscp.service` and edit the domain values in the
`Environment=` lines.

```ini theme={null}
# Custos control plane. Install to /etc/systemd/system/custoscp.service.
# Secrets come from systemd's encrypted credential store, never the unit file.
# See the Custos docs for the full setup flow.

[Unit]
Description=Custos control plane
After=network-online.target
Wants=network-online.target

[Service]
User=custoscp

# Non-secret config. Edit the domain values per deployment.
Environment=CUSTOS_LISTEN_ADDR=127.0.0.1:8123
Environment=CUSTOS_ENCRYPTION=true
Environment=CUSTOS_EMAIL_FROM='Custos <custos@custos.tofunmiadewuyi.com>'
Environment=CUSTOS_APP_URL=https://custos.tofunmiadewuyi.com
Environment=CUSTOS_CORS_ORIGINS=https://custos.tofunmiadewuyi.com
Environment=OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318

# Secrets, decrypted into a per-service tmpfs at $CREDENTIALS_DIRECTORY.
LoadCredentialEncrypted=CUSTOS_MASTER_KEY:/etc/credstore.encrypted/CUSTOS_MASTER_KEY
LoadCredentialEncrypted=CUSTOS_SIGNING_PRIVATE_KEY:/etc/credstore.encrypted/CUSTOS_SIGNING_PRIVATE_KEY
LoadCredentialEncrypted=CUSTOS_SERVER_TRANSPORT_PRIVATE_KEY:/etc/credstore.encrypted/CUSTOS_SERVER_TRANSPORT_PRIVATE_KEY
LoadCredentialEncrypted=CUSTOS_DATABASE_URL:/etc/credstore.encrypted/CUSTOS_DATABASE_URL
LoadCredentialEncrypted=RESEND_API_KEY:/etc/credstore.encrypted/RESEND_API_KEY

# The CP reads secrets from env vars, so read each credential file into the
# environment, then exec the binary so systemd tracks the right PID.
ExecStart=/bin/sh -c 'export \
  CUSTOS_MASTER_KEY="$(cat "$CREDENTIALS_DIRECTORY/CUSTOS_MASTER_KEY")" \
  CUSTOS_SIGNING_PRIVATE_KEY="$(cat "$CREDENTIALS_DIRECTORY/CUSTOS_SIGNING_PRIVATE_KEY")" \
  CUSTOS_SERVER_TRANSPORT_PRIVATE_KEY="$(cat "$CREDENTIALS_DIRECTORY/CUSTOS_SERVER_TRANSPORT_PRIVATE_KEY")" \
  CUSTOS_DATABASE_URL="$(cat "$CREDENTIALS_DIRECTORY/CUSTOS_DATABASE_URL")" \
  RESEND_API_KEY="$(cat "$CREDENTIALS_DIRECTORY/RESEND_API_KEY")"; \
  exec /usr/local/bin/custoscp serve'
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.