be9f14ce34
A small helpdesk system: an office Pixel running a patched GrapheneOS Dialer answers technician calls, records both call legs as separate channels, and a Ruby backend transcribes them through Whisper and files an AI summary against the caller. Squashed to a single commit for sharing. No credentials are included; secrets live outside the repo in /etc/helpdesk/env on the server or a gitignored .claude/env.local locally. See .claude/env.local.example for the shape. Start at README.md, then docs/architecture.md.
139 lines
6.8 KiB
Markdown
139 lines
6.8 KiB
Markdown
# Runbook - certificate renewal (vmin CA, operator p12s, Whisper cert)
|
|
|
|
Three certificates keep the Helpdesk running. One renews itself and needs nothing from you. The other two
|
|
need a human.
|
|
|
|
**Read the CA section first.** The usual cause of a looming vmin expiry is not that certificates need
|
|
reissuing, but that the copy of the CA on the server is older than one you already hold. That is a
|
|
five-minute fix, not a procurement exercise.
|
|
|
|
| Certificate | Expires | Renewal |
|
|
|---|---|---|
|
|
| Server cert for `moje.al.army` (`/etc/ssl/moje.al.army/`) | 2026-10-21 | automatic, acme.sh |
|
|
| vmin CA (`/etc/ssl/certs/ca-combined.crt`) | 2030-09-09 | manual; check for a stale copy FIRST |
|
|
| Operator + Whisper client p12s | leaf 2036-05-02, bound by the CA above | manual |
|
|
|
|
## What breaks, and when
|
|
|
|
The vmin CA is the trust anchor for the operator console *and* the issuer of the certificate the backend
|
|
presents to whisper.cajk.org. When it expires, **operators stop being able to open the console and
|
|
transcription stops, at the same moment**. Calls still arrive and record, because the phone plane uses
|
|
HMAC and does not touch this CA - so the failure looks like "the console is dead and summaries stopped",
|
|
not "the phone is broken".
|
|
|
|
How much lead time you need depends entirely on which of the two situations below you are in: a stale
|
|
CA copy is a five-minute fix, a genuine reissue depends on Virtualmaster replying and wants three weeks.
|
|
|
|
## The server certificate: nothing to do
|
|
|
|
It renews through acme.sh (not certbot), installed at `/root/.acme.sh` and run from **root's crontab**
|
|
four times a day. It writes to the paths nginx reads and then runs `systemctl reload nginx.service`.
|
|
Next renewal 2026-09-21, thirty days before expiry.
|
|
|
|
Two dependencies worth not breaking: port 80 must stay open so the HTTP-01 challenge is reachable, and
|
|
`/.well-known/acme-challenge/` must keep its own location block rather than being swallowed by the https
|
|
redirect. `ufw` allows port 80 explicitly for this reason.
|
|
|
|
Cron discards acme.sh's output and no log is kept, so a repeatedly failing renewal is silent. After any
|
|
firewall or DNS change, confirm the expiry moved:
|
|
|
|
```bash
|
|
openssl s_client -4 -connect moje.al.army:443 -servername moje.al.army </dev/null 2>/dev/null \
|
|
| openssl x509 -noout -dates
|
|
```
|
|
|
|
## The vmin CA: check this BEFORE asking Virtualmaster for anything
|
|
|
|
**The CA installed on the server may simply be an out-of-date copy of a CA that is still valid.**
|
|
Virtualmaster has re-signed `ca.vmin.cz` with a later expiry at least once, keeping the same key, and the
|
|
newer copy ships inside the operator p12s. When that is the case there is nothing to reissue: install the
|
|
newer CA file and the deadline moves years out, with every existing client certificate still working.
|
|
|
|
This was exactly the situation on 2026-07-27, and swapping the file is what moved the current expiry
|
|
to 2030. The server held a copy expiring 2026-09-30 while the p12 already contained one expiring
|
|
2030-09-09 - same public key, same Subject Key Identifier, same serial.
|
|
|
|
So check first:
|
|
|
|
```bash
|
|
# what the server currently trusts
|
|
openssl x509 -in /etc/ssl/certs/ca-combined.crt -noout -enddate -serial
|
|
|
|
# what is bundled in an operator p12 (needs its password)
|
|
openssl pkcs12 -in operator.p12 -cacerts -nokeys -passin pass:PASSWORD \
|
|
| openssl x509 -noout -enddate -serial
|
|
```
|
|
|
|
If the p12's copy expires later, extract it and install it (below). Confirm it is the same CA and not a
|
|
different one by comparing the public keys - identical output means every existing client certificate
|
|
keeps working:
|
|
|
|
```bash
|
|
openssl x509 -in /etc/ssl/certs/ca-combined.crt -noout -pubkey | openssl sha256
|
|
openssl pkcs12 -in operator.p12 -cacerts -nokeys -passin pass:PASSWORD | openssl x509 -noout -pubkey | openssl sha256
|
|
```
|
|
|
|
### Installing a newer CA copy (the common case)
|
|
|
|
Extract it from a p12 and install it. The public-key check is the important step: matching keys mean the
|
|
same issuer, so every certificate already in use keeps working and nobody needs a new p12.
|
|
|
|
```bash
|
|
# extract
|
|
openssl pkcs12 -in operator.p12 -cacerts -nokeys -passin pass:PASSWORD > /tmp/ca-new.crt
|
|
|
|
# REFUSE to install unless the key matches - a different key would invalidate every operator cert
|
|
diff <(openssl x509 -in /etc/ssl/certs/ca-combined.crt -noout -pubkey) \
|
|
<(openssl x509 -in /tmp/ca-new.crt -noout -pubkey) && echo "same issuer, safe to install"
|
|
|
|
sudo cp -a /etc/ssl/certs/ca-combined.crt /etc/ssl/certs/ca-combined.crt.bak-$(date +%F)
|
|
sudo install -m 644 -o root -g root /tmp/ca-new.crt /etc/ssl/certs/ca-combined.crt
|
|
sudo /opt/nginx/sbin/nginx -t && sudo systemctl reload nginx.service
|
|
```
|
|
|
|
Operators do not need to do anything and existing browser sessions keep working.
|
|
|
|
### If you really do need new certificates
|
|
|
|
Only when no newer CA copy exists. You need a new CA certificate, a client p12 per operator, and one for
|
|
the Whisper connection. These were issued to your operators specifically, not generic per-account
|
|
certificates, so they have to be reissued the same way. Each p12 has its own password; those belong in
|
|
the password manager, never in the repo. Then:
|
|
|
|
1. **Install the new CA** exactly as above, except the key will NOT match - that is expected here, and it
|
|
is why every operator needs a new p12 at the same time.
|
|
2. **Replace the Whisper certificate**: `/etc/helpdesk/vmin-whisper.p12`, kept `0640 root:helpdesk`.
|
|
Update `WHISPER_MTLS_PASS` in `/etc/helpdesk/env` if the password changed, then
|
|
`sudo systemctl restart helpdesk`.
|
|
3. **Distribute operator p12s** before switching, not after. Old certificates stop working the moment the
|
|
old CA is removed.
|
|
|
|
`nginx -t` only catches a malformed file, never an expired or wrong CA. Always verify by connecting.
|
|
|
|
## Verify, in this order
|
|
|
|
```bash
|
|
# 1. the console still demands a certificate (400 = nginx serving, cert required)
|
|
curl -4 -sk -o /dev/null -w '%{http_code}\n' https://moje.al.army/
|
|
|
|
# 2. the advertised client CA is the NEW one - check the expiry date shown
|
|
openssl s_client -4 -connect moje.al.army:443 </dev/null 2>&1 | grep -A2 "Acceptable client"
|
|
|
|
# 3. the phone plane is untouched by any of this (401 expected)
|
|
curl -4 -sk -o /dev/null -w '%{http_code}\n' \
|
|
'https://moje.al.army:8443/api/v1/device/heartbeat?device_id=probe'
|
|
```
|
|
|
|
Then load the console in a browser with a new p12, and put a real call through so a recording completes
|
|
transcription. Transcription is the half most easily forgotten, because it fails silently hours later
|
|
rather than at renewal time.
|
|
|
|
## Rollback
|
|
|
|
Restore the backed-up `ca-combined.crt` and reload nginx. This works only while the old CA is still
|
|
valid, so it is a fix for "the new CA was installed wrong", not for "we ran past the expiry date".
|
|
|
|
## Related
|
|
|
|
Expiry table and the env variables: `docs/operations.md`. The two auth planes and why the console's
|
|
trust anchor matters: `docs/architecture.md`.
|