Prd/docs/runbooks/cert-renewal.md

Runbook - certificate renewal (vmin CA, operator p12s, Whisper cert)

Three certificates keep the Helpdesk running. One renews itself and needs nothing from you. The other two need a human.

Read the CA section first. The usual cause of a looming vmin expiry is not that certificates need reissuing, but that the copy of the CA on the server is older than one you already hold. That is a five-minute fix, not a procurement exercise.

Certificate	Expires	Renewal
Server cert for moje.al.army (/etc/ssl/moje.al.army/)	2026-10-21	automatic, acme.sh
vmin CA (/etc/ssl/certs/ca-combined.crt)	2030-09-09	manual; check for a stale copy FIRST
Operator + Whisper client p12s	leaf 2036-05-02, bound by the CA above	manual

What breaks, and when

The vmin CA is the trust anchor for the operator console and the issuer of the certificate the backend presents to whisper.cajk.org. When it expires, operators stop being able to open the console and transcription stops, at the same moment. Calls still arrive and record, because the phone plane uses HMAC and does not touch this CA - so the failure looks like "the console is dead and summaries stopped", not "the phone is broken".

How much lead time you need depends entirely on which of the two situations below you are in: a stale CA copy is a five-minute fix, a genuine reissue depends on Virtualmaster replying and wants three weeks.

The server certificate: nothing to do

It renews through acme.sh (not certbot), installed at /root/.acme.sh and run from root's crontab four times a day. It writes to the paths nginx reads and then runs systemctl reload nginx.service. Next renewal 2026-09-21, thirty days before expiry.

Two dependencies worth not breaking: port 80 must stay open so the HTTP-01 challenge is reachable, and /.well-known/acme-challenge/ must keep its own location block rather than being swallowed by the https redirect. ufw allows port 80 explicitly for this reason.

Cron discards acme.sh's output and no log is kept, so a repeatedly failing renewal is silent. After any firewall or DNS change, confirm the expiry moved:

openssl s_client -4 -connect moje.al.army:443 -servername moje.al.army </dev/null 2>/dev/null \
  | openssl x509 -noout -dates

The vmin CA: check this BEFORE asking Virtualmaster for anything

The CA installed on the server may simply be an out-of-date copy of a CA that is still valid. Virtualmaster has re-signed ca.vmin.cz with a later expiry at least once, keeping the same key, and the newer copy ships inside the operator p12s. When that is the case there is nothing to reissue: install the newer CA file and the deadline moves years out, with every existing client certificate still working.

This was exactly the situation on 2026-07-27, and swapping the file is what moved the current expiry to 2030. The server held a copy expiring 2026-09-30 while the p12 already contained one expiring 2030-09-09 - same public key, same Subject Key Identifier, same serial.

So check first:

# what the server currently trusts
openssl x509 -in /etc/ssl/certs/ca-combined.crt -noout -enddate -serial

# what is bundled in an operator p12 (needs its password)
openssl pkcs12 -in operator.p12 -cacerts -nokeys -passin pass:PASSWORD \
  | openssl x509 -noout -enddate -serial

If the p12's copy expires later, extract it and install it (below). Confirm it is the same CA and not a different one by comparing the public keys - identical output means every existing client certificate keeps working:

openssl x509 -in /etc/ssl/certs/ca-combined.crt -noout -pubkey | openssl sha256
openssl pkcs12 -in operator.p12 -cacerts -nokeys -passin pass:PASSWORD | openssl x509 -noout -pubkey | openssl sha256

Installing a newer CA copy (the common case)

Extract it from a p12 and install it. The public-key check is the important step: matching keys mean the same issuer, so every certificate already in use keeps working and nobody needs a new p12.

# extract
openssl pkcs12 -in operator.p12 -cacerts -nokeys -passin pass:PASSWORD > /tmp/ca-new.crt

# REFUSE to install unless the key matches - a different key would invalidate every operator cert
diff <(openssl x509 -in /etc/ssl/certs/ca-combined.crt -noout -pubkey) \
     <(openssl x509 -in /tmp/ca-new.crt -noout -pubkey) && echo "same issuer, safe to install"

sudo cp -a /etc/ssl/certs/ca-combined.crt /etc/ssl/certs/ca-combined.crt.bak-$(date +%F)
sudo install -m 644 -o root -g root /tmp/ca-new.crt /etc/ssl/certs/ca-combined.crt
sudo /opt/nginx/sbin/nginx -t && sudo systemctl reload nginx.service

Operators do not need to do anything and existing browser sessions keep working.

If you really do need new certificates

Only when no newer CA copy exists. You need a new CA certificate, a client p12 per operator, and one for the Whisper connection. These were issued to your operators specifically, not generic per-account certificates, so they have to be reissued the same way. Each p12 has its own password; those belong in the password manager, never in the repo. Then:

1. Install the new CA exactly as above, except the key will NOT match - that is expected here, and it is why every operator needs a new p12 at the same time.
2. Replace the Whisper certificate: /etc/helpdesk/vmin-whisper.p12, kept 0640 root:helpdesk. Update WHISPER_MTLS_PASS in /etc/helpdesk/env if the password changed, then sudo systemctl restart helpdesk.
3. Distribute operator p12s before switching, not after. Old certificates stop working the moment the old CA is removed.

nginx -t only catches a malformed file, never an expired or wrong CA. Always verify by connecting.

Verify, in this order

# 1. the console still demands a certificate (400 = nginx serving, cert required)
curl -4 -sk -o /dev/null -w '%{http_code}\n' https://moje.al.army/

# 2. the advertised client CA is the NEW one - check the expiry date shown
openssl s_client -4 -connect moje.al.army:443 </dev/null 2>&1 | grep -A2 "Acceptable client"

# 3. the phone plane is untouched by any of this (401 expected)
curl -4 -sk -o /dev/null -w '%{http_code}\n' \
  'https://moje.al.army:8443/api/v1/device/heartbeat?device_id=probe'

Then load the console in a browser with a new p12, and put a real call through so a recording completes transcription. Transcription is the half most easily forgotten, because it fails silently hours later rather than at renewal time.

Rollback

Restore the backed-up ca-combined.crt and reload nginx. This works only while the old CA is still valid, so it is a fix for "the new CA was installed wrong", not for "we ran past the expiry date".

Related

Expiry table and the env variables: docs/operations.md. The two auth planes and why the console's trust anchor matters: docs/architecture.md.