Prd/docs/operations.md

Operations

The production instance runs on moje.al.army. Here's how to deploy it, roll it back, read its logs, back it up, and what CI does.

Where things are on the server

• Code: /srv/helpdesk (a checkout; the deploy script syncs into it).
• Service: helpdesk.service (systemd). Runs as the helpdesk user, binds 127.0.0.1:4000, reads /etc/helpdesk/env.
• Data: /var/lib/helpdesk (recordings, and the wiki index). Deploys never touch it.
• Web: nginx serves the vmin mTLS vhost for moje.al.army (console, 443) and the certless device door on 8443 (sites-available/helpdesk-device.conf - the phone's HMAC plane). Both proxy to 127.0.0.1:4000. Reference copies of all of it are in deploy/.
• Secrets: /etc/helpdesk/env (root-only). The full variable list is below.
• Firewall: ufw, default deny inbound, allowing only 22, 80, 443 and 8443 (v4 and v6), enabled at boot. Port 80 is not optional - it carries the ACME challenge that renews the server certificate. The backend on 4000 and PostgreSQL are loopback-only and were never exposed; the firewall is the layer that keeps a future service from binding publicly by accident.

Deploy

Deploys go through CI. Push to master, let the pipeline pass, then click the manual deploy:production job. It runs deploy/helpdesk-deploy.sh, which syncs the checkout into /srv/helpdesk, runs the migration, restarts the service, and polls /healthz. If health doesn't come back, it rolls back to the previous release by itself.

To deploy by hand on the box:

sudo /usr/local/sbin/helpdesk-deploy /path/to/checkout

Roll back

The deploy keeps the previous release at /srv/helpdesk.prev and reverts on a failed health check. To roll back manually:

sudo systemctl stop helpdesk
sudo rsync -a --delete /srv/helpdesk.prev/ /srv/helpdesk/
sudo systemctl start helpdesk

Logs and health

journalctl -u helpdesk -f              # service logs
curl -s http://127.0.0.1:4000/healthz  # {"ok":true,...}

From outside the box, health needs a client cert: curl --cert your.pem https://moje.al.army/healthz. Without one you get a 400, because mTLS is required.

Backups

helpdesk-pg-backup.timer dumps the database nightly at 03:30 to /var/backups/helpdesk and keeps 14 days. To prove a dump restores, load it into a scratch database and check it:

sudo -u postgres createdb helpdesk_restore_test
sudo -u postgres pg_restore --no-owner -d helpdesk_restore_test /var/backups/helpdesk/helpdesk-<ts>.dump
sudo -u postgres psql -c '\dt' helpdesk_restore_test
sudo -u postgres dropdb helpdesk_restore_test

Secrets and configuration

Everything lives in /etc/helpdesk/env (root-only; the systemd unit loads it). Names only - the values stay on the server and in the password manager.

Variable	What it does
HELPDESK_DEV=0	production posture: operator auth on, no dev endpoints, no demo seed
HELPDESK_OP_AUTH=1, HELPDESK_PROXY_SECRET	operator plane: require the mTLS proxy's injected headers
HELPDESK_OPERATOR_CNS	comma-separated cert CNs allowed on the console, e.g. lucy,petr. Unset = accept any cert the vmin CA verified. Also the revocation mechanism: no CRL is configured, so removing a CN here is how a lost operator certificate is turned off
HELPDESK_AUTH=1, HELPDESK_DEVICE_TOKEN, HELPDESK_DEVICE_SECRET	device plane: the phone's HMAC credentials
HELPDESK_DATABASE_URL	PostgreSQL as the helpdesk role (never postgres)
HELPDESK_BIND=127.0.0.1	never listen on a public interface; nginx is the only entry
HELPDESK_SEED=0	boot empty instead of seeding demo contacts
HELPDESK_REC_DIR	where uploaded recordings land (under /var/lib/helpdesk)
WHISPER_URL	hosted Whisper (https://whisper.cajk.org/app); selects the remote transcriber
WHISPER_MTLS_P12, WHISPER_MTLS_PASS	the vmin client cert the backend presents to Whisper (p12 at /etc/helpdesk/, pass from the password manager). Leave WHISPER_CA unset - the server cert is public.
OPENROUTER_API_KEY	all AI calls (summaries, dossier, embeddings)
SUMMARISER_MODEL, MAINTAINER_MODEL, MAINTAINER_REBUILD_MODEL	optional model overrides; the code defaults to google/gemini-3.6-flash for all three
WIKI_INDEX_PATH	the wiki RAG index (/var/lib/helpdesk/wiki_index.json); unset or missing file just means summaries run ungrounded
HELPDESK_GLOSSARY_PATH	override the domain vocabulary applied to transcripts; defaults to lib/helpdesk/glossary_terms.txt in the checkout. Only distinctive terms belong in it - see docs/architecture.md
HELPDESK_CERT_WATCH_TLS	host:port to read the served certificate from, e.g. moje.al.army:8443. Use the device door, not 443: it serves the same certificate without demanding a client cert. Unset = not checked
HELPDESK_CERT_WATCH_CA	the operator CA bundle to watch (/etc/ssl/certs/ca-combined.crt). Unset = not checked
HELPDESK_CERT_WARN_DAYS	when the console starts warning; default 30. The Whisper cert is picked up automatically from WHISPER_MTLS_P12

Certificates

Three certificates matter, and only one of them renews itself.

Certificate	Where	Expires	Renewal
Server cert for moje.al.army	/etc/ssl/moje.al.army/{fullchain,privkey}.pem, shared by the console on :443 and the device door on :8443	2026-10-21	automatic
vmin CA (ca.vmin.cz)	/etc/ssl/certs/ca-combined.crt	2030-09-09	manual, but see below before assuming a reissue
Operator + Whisper client certs	p12s with the operators; /etc/helpdesk/vmin-whisper.p12	leaf 2036-05-02, bound by the CA above	manual

The server cert renews through acme.sh (not certbot), installed at /root/.acme.sh and run from root's crontab four times a day. It was set up by /root/07-acme-cert.sh. It writes straight to the paths nginx reads and then runs systemctl reload nginx.service, so a renewal needs no hand-holding; next renewal is 2026-09-21, thirty days before expiry. Two things it depends on: port 80 must stay open so the HTTP-01 challenge at /.well-known/acme-challenge/ is reachable, and that location must keep its own block in the vhost rather than being swept into the https redirect. Note that cron sends its output to /dev/null and no log file is kept, so a renewal that fails repeatedly is invisible until the cert actually expires - if you touch the firewall or DNS, check the cert date afterwards.

Before treating a vmin expiry as a reissue job, check whether the server's CA copy is simply stale. Virtualmaster re-signs ca.vmin.cz keeping the same key, and the newer copy ships inside the operator p12s - so the fix is often just installing a file you already have, with every existing client certificate still working. This is not hypothetical: on 2026-07-27 the server trusted a copy expiring 2026-09-30 while the p12 already held one expiring 2030-09-09, same key and serial. Swapping that one file moved the deadline out by four years and no certificate had to be reissued. docs/runbooks/cert-renewal.md has the commands to tell the two situations apart, and the verification order that catches the easily-forgotten half (transcription fails silently hours later, not at renewal time).

If a genuine reissue is ever needed, it means refreshing /etc/ssl/certs/ca-combined.crt, reissuing the operator p12s and replacing the Whisper p12 - otherwise the console stops authenticating operators and transcription stops at the same moment. Start about three weeks out, since it depends on Virtualmaster replying.

The console warns about all of this by itself: a badge appears in the navbar when any certificate is inside HELPDESK_CERT_WARN_DAYS (default 30) and within 30 days, and stays hidden otherwise. The console applies that 30-day cap itself, so setting the variable higher makes the backend report earlier without the badge shouting about certificates months away; setting it lower still tightens the warning as you would expect. That includes each operator's own certificate, which the server holds no copy of - nginx reports the expiry of whichever one was presented, so people are warned about theirs individually. It needs these two headers on the console vhost, and warns about nothing personal without them:

proxy_set_header X-Operator-Cert-Days    $ssl_client_v_remain;
proxy_set_header X-Operator-Cert-Expires $ssl_client_v_end;

Checking any of this without logging in:

# server cert dates, either port
openssl s_client -4 -connect moje.al.army:443 -servername moje.al.army </dev/null 2>/dev/null \
  | openssl x509 -noout -dates
# which CA the console will accept client certs from
openssl s_client -4 -connect moje.al.army:443 </dev/null 2>&1 | grep -A2 "Acceptable client"

On the dev box, the same keys live outside the repo: the OpenRouter key in ~/Downloads/openrouterApiKlic2 (the sk-or-v1-… line), the wiki JWT in ~/Downloads/apikeyvodafonewiki (the eyJ… line), local env in .claude/env.local (gitignored).

The wiki index

Summaries are grounded in the Vodafone wiki through a small cosine index. It is built offline and is not in git (32 MB of embeddings), so a fresh server needs it placed once:

cd components/backend/wiki
WIKI_JWT=… OPENROUTER_API_KEY=… ruby build_index.rb        # fetches ~170 pages, embeds, writes wiki_index.json
scp wiki_index.json skyyf@moje.al.army:/tmp/ && ssh skyyf@moje.al.army \
  'sudo install -o helpdesk -g helpdesk /tmp/wiki_index.json /var/lib/helpdesk/wiki_index.json'

The service picks it up on restart via WIKI_INDEX_PATH. Rebuild it the same way when the wiki content changes enough to matter; nothing breaks in between, retrieval just returns older text.

The CI pipeline

.gitlab-ci.yml, running on the project's own runner on moje.al.army (jobs are tagged helpdesk so they land there, not on the shared vmin runners). Four stages:

• lint - ruby -c on all Ruby, shellcheck on the scripts, a parse check on the patch files.
• test - the unit suites plus the JSON-vs-PostgreSQL parity harness, against a throwaway helpdesk_test database (the CI variable HELPDESK_TEST_DATABASE_URL, a low-privilege helpdesk_ci role).
• e2e - boots the backend on the throwaway database and an alternate port, then drives it with sim/simulator.rb over HTTP.
• deploy - manual, master only. Runs the deploy script above.

CI runs on the system Ruby (/usr/bin/ruby), not the Passenger Ruby that's also installed; the before_script forces PATH and clears GEM_HOME/GEM_PATH to make that happen. If a job dies at "prepare environment: exit status 1", that's the gitlab-runner user's shell profile, not your code.

To reproduce a failing job, run the same command from the job's script inside components/backend.