Prd/docs/operations.md

# Operations

The production instance runs on `moje.al.army`. Here's how to deploy it, roll it back, read its logs,
back it up, and what CI does.

## Where things are on the server

- Code: `/srv/helpdesk` (a checkout; the deploy script syncs into it).
- Service: `helpdesk.service` (systemd). Runs as the `helpdesk` user, binds `127.0.0.1:4000`, reads
  `/etc/helpdesk/env`.
- Data: `/var/lib/helpdesk` (recordings, and the wiki index). Deploys never touch it.
- Web: nginx serves the vmin mTLS vhost for `moje.al.army` (console, 443) and the certless device door
  on 8443 (`sites-available/helpdesk-device.conf` - the phone's HMAC plane). Both proxy to
  `127.0.0.1:4000`. Reference copies of all of it are in `deploy/`.
- Secrets: `/etc/helpdesk/env` (root-only). The full variable list is below.
- Firewall: `ufw`, default deny inbound, allowing only 22, 80, 443 and 8443 (v4 and v6), enabled at boot.
  Port 80 is not optional - it carries the ACME challenge that renews the server certificate. The backend
  on 4000 and PostgreSQL are loopback-only and were never exposed; the firewall is the layer that keeps a
  future service from binding publicly by accident.

## Deploy

Deploys go through CI. Push to `master`, let the pipeline pass, then click the manual
`deploy:production` job. It runs `deploy/helpdesk-deploy.sh`, which syncs the checkout into
`/srv/helpdesk`, runs the migration, restarts the service, and polls `/healthz`. If health doesn't come
back, it rolls back to the previous release by itself.

To deploy by hand on the box:

```
sudo /usr/local/sbin/helpdesk-deploy /path/to/checkout
```

## Roll back

The deploy keeps the previous release at `/srv/helpdesk.prev` and reverts on a failed health check. To
roll back manually:

```
sudo systemctl stop helpdesk
sudo rsync -a --delete /srv/helpdesk.prev/ /srv/helpdesk/
sudo systemctl start helpdesk
```

## Logs and health

```
journalctl -u helpdesk -f              # service logs
curl -s http://127.0.0.1:4000/healthz  # {"ok":true,...}
```

From outside the box, health needs a client cert: `curl --cert your.pem https://moje.al.army/healthz`.
Without one you get a 400, because mTLS is required.

## Backups

`helpdesk-pg-backup.timer` dumps the database nightly at 03:30 to `/var/backups/helpdesk` and keeps 14
days. To prove a dump restores, load it into a scratch database and check it:

```
sudo -u postgres createdb helpdesk_restore_test
sudo -u postgres pg_restore --no-owner -d helpdesk_restore_test /var/backups/helpdesk/helpdesk-<ts>.dump
sudo -u postgres psql -c '\dt' helpdesk_restore_test
sudo -u postgres dropdb helpdesk_restore_test
```

## Secrets and configuration

Everything lives in `/etc/helpdesk/env` (root-only; the systemd unit loads it). Names only - the values
stay on the server and in the password manager.

| Variable | What it does |
|---|---|
| `HELPDESK_DEV=0` | production posture: operator auth on, no dev endpoints, no demo seed |
| `HELPDESK_OP_AUTH=1`, `HELPDESK_PROXY_SECRET` | operator plane: require the mTLS proxy's injected headers |
| `HELPDESK_OPERATOR_CNS` | comma-separated cert CNs allowed on the console, e.g. `lucy,petr`. Unset = accept any cert the vmin CA verified. Also the revocation mechanism: no CRL is configured, so removing a CN here is how a lost operator certificate is turned off |
| `HELPDESK_AUTH=1`, `HELPDESK_DEVICE_TOKEN`, `HELPDESK_DEVICE_SECRET` | device plane: the phone's HMAC credentials |
| `HELPDESK_DATABASE_URL` | PostgreSQL as the `helpdesk` role (never `postgres`) |
| `HELPDESK_BIND=127.0.0.1` | never listen on a public interface; nginx is the only entry |
| `HELPDESK_SEED=0` | boot empty instead of seeding demo contacts |
| `HELPDESK_REC_DIR` | where uploaded recordings land (under `/var/lib/helpdesk`) |
| `WHISPER_URL` | hosted Whisper (`https://whisper.cajk.org/app`); selects the remote transcriber |
| `WHISPER_MTLS_P12`, `WHISPER_MTLS_PASS` | the vmin client cert the backend presents to Whisper (p12 at `/etc/helpdesk/`, pass from the password manager). Leave `WHISPER_CA` unset - the server cert is public. |
| `OPENROUTER_API_KEY` | all AI calls (summaries, dossier, embeddings) |
| `SUMMARISER_MODEL`, `MAINTAINER_MODEL`, `MAINTAINER_REBUILD_MODEL` | optional model overrides; the code defaults to `google/gemini-3.6-flash` for all three |
| `WIKI_INDEX_PATH` | the wiki RAG index (`/var/lib/helpdesk/wiki_index.json`); unset or missing file just means summaries run ungrounded |
| `HELPDESK_GLOSSARY_PATH` | override the domain vocabulary applied to transcripts; defaults to `lib/helpdesk/glossary_terms.txt` in the checkout. Only distinctive terms belong in it - see `docs/architecture.md` |
| `HELPDESK_CERT_WATCH_TLS` | `host:port` to read the served certificate from, e.g. `moje.al.army:8443`. Use the device door, not 443: it serves the same certificate without demanding a client cert. Unset = not checked |
| `HELPDESK_CERT_WATCH_CA` | the operator CA bundle to watch (`/etc/ssl/certs/ca-combined.crt`). Unset = not checked |
| `HELPDESK_CERT_WARN_DAYS` | when the console starts warning; default 30. The Whisper cert is picked up automatically from `WHISPER_MTLS_P12` |

## Certificates

Three certificates matter, and only one of them renews itself.

| Certificate | Where | Expires | Renewal |
|---|---|---|---|
| Server cert for `moje.al.army` | `/etc/ssl/moje.al.army/{fullchain,privkey}.pem`, shared by the console on :443 and the device door on :8443 | 2026-10-21 | automatic |
| vmin CA (`ca.vmin.cz`) | `/etc/ssl/certs/ca-combined.crt` | 2030-09-09 | manual, but see below before assuming a reissue |
| Operator + Whisper client certs | p12s with the operators; `/etc/helpdesk/vmin-whisper.p12` | leaf 2036-05-02, bound by the CA above | manual |

The server cert renews through **acme.sh** (not certbot), installed at `/root/.acme.sh` and run from
root's crontab four times a day. It was set up by `/root/07-acme-cert.sh`. It writes straight to the
paths nginx reads and then runs `systemctl reload nginx.service`, so a renewal needs no hand-holding;
next renewal is 2026-09-21, thirty days before expiry. Two things it depends on: port 80 must stay open
so the HTTP-01 challenge at `/.well-known/acme-challenge/` is reachable, and that location must keep its
own block in the vhost rather than being swept into the https redirect. Note that cron sends its output
to `/dev/null` and no log file is kept, so a renewal that fails repeatedly is invisible until the cert
actually expires - if you touch the firewall or DNS, check the cert date afterwards.

**Before treating a vmin expiry as a reissue job, check whether the server's CA copy is simply stale.**
Virtualmaster re-signs `ca.vmin.cz` keeping the same key, and the newer copy ships inside the operator
p12s - so the fix is often just installing a file you already have, with every existing client
certificate still working. This is not hypothetical: on 2026-07-27 the server trusted a copy expiring
2026-09-30 while the p12 already held one expiring 2030-09-09, same key and serial. Swapping that one
file moved the deadline out by four years and no certificate had to be reissued. `docs/runbooks/cert-renewal.md`
has the commands to tell the two situations apart, and the verification order that catches the
easily-forgotten half (transcription fails silently hours later, not at renewal time).

If a genuine reissue is ever needed, it means refreshing `/etc/ssl/certs/ca-combined.crt`, reissuing the
operator p12s and replacing the Whisper p12 - otherwise the console stops authenticating operators and
transcription stops at the same moment. Start about three weeks out, since it depends on Virtualmaster
replying.

The console warns about all of this by itself: a badge appears in the navbar when any certificate is
inside `HELPDESK_CERT_WARN_DAYS` (default 30) **and within 30 days**, and stays hidden otherwise. The
console applies that 30-day cap itself, so setting the variable higher makes the backend report earlier
without the badge shouting about certificates months away; setting it lower still tightens the warning
as you would expect. That includes **each
operator's own certificate**, which the server holds no copy of - nginx reports the expiry of whichever
one was presented, so people are warned about theirs individually. It needs these two headers on the
console vhost, and warns about nothing personal without them:

```
proxy_set_header X-Operator-Cert-Days    $ssl_client_v_remain;
proxy_set_header X-Operator-Cert-Expires $ssl_client_v_end;
```

Checking any of this without logging in:

```
# server cert dates, either port
openssl s_client -4 -connect moje.al.army:443 -servername moje.al.army </dev/null 2>/dev/null \
  | openssl x509 -noout -dates
# which CA the console will accept client certs from
openssl s_client -4 -connect moje.al.army:443 </dev/null 2>&1 | grep -A2 "Acceptable client"
```

On the dev box, the same keys live outside the repo: the OpenRouter key in
`~/Downloads/openrouterApiKlic2` (the `sk-or-v1-…` line), the wiki JWT in
`~/Downloads/apikeyvodafonewiki` (the `eyJ…` line), local env in `.claude/env.local` (gitignored).

## The wiki index

Summaries are grounded in the Vodafone wiki through a small cosine index. It is built offline and is
not in git (32 MB of embeddings), so a fresh server needs it placed once:

```
cd components/backend/wiki
WIKI_JWT=… OPENROUTER_API_KEY=… ruby build_index.rb        # fetches ~170 pages, embeds, writes wiki_index.json
scp wiki_index.json skyyf@moje.al.army:/tmp/ && ssh skyyf@moje.al.army \
  'sudo install -o helpdesk -g helpdesk /tmp/wiki_index.json /var/lib/helpdesk/wiki_index.json'
```

The service picks it up on restart via `WIKI_INDEX_PATH`. Rebuild it the same way when the wiki content
changes enough to matter; nothing breaks in between, retrieval just returns older text.

## The CI pipeline

`.gitlab-ci.yml`, running on the project's own runner on `moje.al.army` (jobs are tagged `helpdesk` so
they land there, not on the shared vmin runners). Four stages:

- lint - `ruby -c` on all Ruby, shellcheck on the scripts, a parse check on the patch files.
- test - the unit suites plus the JSON-vs-PostgreSQL parity harness, against a throwaway
  `helpdesk_test` database (the CI variable `HELPDESK_TEST_DATABASE_URL`, a low-privilege `helpdesk_ci`
  role).
- e2e - boots the backend on the throwaway database and an alternate port, then drives it with
  `sim/simulator.rb` over HTTP.
- deploy - manual, `master` only. Runs the deploy script above.

CI runs on the system Ruby (`/usr/bin/ruby`), not the Passenger Ruby that's also installed; the
`before_script` forces `PATH` and clears `GEM_HOME`/`GEM_PATH` to make that happen. If a job dies at
"prepare environment: exit status 1", that's the `gitlab-runner` user's shell profile, not your code.

To reproduce a failing job, run the same command from the job's `script` inside `components/backend`.