Status as of 2026-10-23. Nothing has been changed in the cluster yet.
| What | Where |
|---|---|
| Update branch | mailcow-update (worktree ~/Arbeitskasten/Kubeia-mailcow-update), based on repo-cleanup |
| New image | docker.benkard.de/mulk/mailcow:9q2d15n90wjlwqph7rgf6967ir9bw9aj (pushed, built with make mailcow.load) |
| Old image (rollback) | docker.benkard.de/mulk/mailcow:zxabqwczc1nb589mz98ikmn25d6rrwd8 |
| Backups and config changes | ~/Arbeitskasten/mailcow-conf-2026-10/ (mode 700, contains secrets, never put in git) |
| Config update script | bin/apply-mailcow-conf (runs inside the helper pod, see step 4) |
| Live pod | namespace mulk, node ifirn; config PVC mailcow-config-v2 |
Contents of ~/Arbeitskasten/mailcow-conf-2026-10/:
conf-live-backup.tar.gz: the live data/conf (183 files), verified identical to the PVC.live-ownership.txt: owner and mode of every entry (uid:gid mode path); the PVC has mixed owners.live-snapshot.md5: checksums of the live config, used to check that nothing changed before step 4.conf-changes.tar.gz: the 45 files from upstream 2026-10 that are new or changed compared to live.changed-files.txt, deleted-files.txt: what step 4 updates and deletes (33 stale stock files; the old nginx .conf files must go because the new nginx includes conf.d/*.conf).mailcow-db-before-2026-10.sql.gz: full dump of the mailcow database (64 tables, no routines or events, see "MariaDB system tables" in TODO.md).55982ac7 (tag 2026-10), local patches re-applied (no Unbound DNS entries, IPv6 off via ENABLE_IPV6, DockerApi MySQL upgrade stubs, Postfix overrides).version: and Solr removed) in mailcow/default.nix.REDISPASS and SOGO_URL_ENCRYPTION_KEY added to mailcow.conf in mailcow-secrets (use bin/edit-mailcow-conf); dbpass key present.mailcow/mailcow.yaml on mailcow-update set to the new image tag: edited, not committed yet. kubectl diff shows only the image and the DBPASS → secretKeyRef change.Run everything from ~/Arbeitskasten/mailcow-conf-2026-10. Mail is down from step 3 until step 6 finishes.
cd ~/Arbeitskasten/mailcow-conf-2026-10 KUBEIA=~/Arbeitskasten/Kubeia
kubectl -n mulk scale deploy/mailcow --replicas=0 kubectl -n mulk wait --for=delete pod -l name=mailcow --timeout=300s
Start a helper pod on ifirn with the config PVC mounted at /conf:
kubectl -n mulk run mailcow-conf-sync --image=busybox:1.36 --restart=Never --overrides='{"apiVersion":"v1","spec":{"nodeName":"ifirn","containers":[{"name":"mailcow-conf-sync","image":"busybox:1.36","command":["sleep","3600"],"volumeMounts":[{"name":"conf","mountPath":"/conf"}]}],"volumes":[{"name":"conf","persistentVolumeClaim":{"claimName":"mailcow-config-v2"}}]}}' kubectl -n mulk wait --for=condition=Ready pod/mailcow-conf-sync
Check that the config hasn't changed since the backup. No output means OK. If anything is printed, stop here, delete the helper pod and take a fresh backup first.
kubectl -n mulk exec mailcow-conf-sync -- sh -c 'cd /conf && find . -type f -exec md5sum {} +' | sort -k2 | diff - live-snapshot.md5
Copy the changes in and apply them:
kubectl -n mulk cp conf-changes.tar.gz mailcow-conf-sync:/tmp/ kubectl -n mulk cp deleted-files.txt mailcow-conf-sync:/tmp/ kubectl -n mulk cp $KUBEIA/bin/apply-mailcow-conf mailcow-conf-sync:/tmp/apply-conf.sh kubectl -n mulk exec mailcow-conf-sync -- sh -c 'mkdir /tmp/changes && tar -C /tmp/changes -xzf /tmp/conf-changes.tar.gz && sh /tmp/apply-conf.sh'
The script prints one line per file (updated, created, deleted); expect 30 / 15 / 33. Then remove the helper pod:
kubectl -n mulk delete pod mailcow-conf-sync
kubectl apply -f ~/Arbeitskasten/Kubeia-mailcow-update/mailcow/mailcow.yaml
Then commit the image tag on mailcow-update (mailcow: Update image to 2026-10.).
The first start pulls about 20 images (now from ghcr.io) and builds 3 (netfilter, dockerapi, watchdog), so it takes a while.
kubectl -n mulk logs -f deploy/mailcow kubectl -n mulk exec deploy/mailcow -- docker ps # repeat until everything is healthy
Check:
REDISPASS) in the logscd ~/Arbeitskasten/mailcow-conf-2026-10 kubectl -n mulk scale deploy/mailcow --replicas=0 kubectl -n mulk wait --for=delete pod -l name=mailcow --timeout=300s
Restore the config: start the helper pod as in step 4, then
kubectl -n mulk cp conf-live-backup.tar.gz mailcow-conf-sync:/tmp/ kubectl -n mulk cp live-ownership.txt mailcow-conf-sync:/tmp/ kubectl -n mulk exec mailcow-conf-sync -- sh -c ' find /conf -mindepth 1 -delete && tar -C /conf -xzf /tmp/conf-live-backup.tar.gz && while read -r owner mode path; do chown "$owner" "/conf/$path"; chmod "$mode" "/conf/$path"; done </tmp/live-ownership.txt' kubectl -n mulk exec mailcow-conf-sync -- sh -c 'cd /conf && find . -type f -exec md5sum {} +' | sort -k2 | diff - live-snapshot.md5 kubectl -n mulk delete pod mailcow-conf-sync
The ownership loop is needed because the archive stores owner names from the local machine. The diff must print nothing.
Restore the database only if the new version has already migrated it (it does so on first start):
DBPASS=$(kubectl -n mulk get secret mailcow-secrets -o jsonpath='{.data.dbpass}' | base64 -d) zcat mailcow-db-before-2026-10.sql.gz | kubectl -n system exec -i deploy/mysql -- mariadb -u mailcow -p"$DBPASS" mailcow
Then go back to the old image (the old manifest on repo-cleanup already has it, and keeps the dbpass secret reference):
kubectl apply -f ~/Arbeitskasten/Kubeia/mailcow/mailcow.yaml
mailcow-update; merge into master when you're happy (not done yet).mailcow-solr PVC and the /vol/solr-data mount/volume (mailcow.yaml, and solr-data in mailcow/default.nix).mailcow.conf keys: SKIP_FTS, FTS_HEAP, FTS_PROCS, SKIP_OLEFY, AUTODISCOVER_SAN, HTTP_REDIRECT, ACME_ACCOUNT_EMAIL (replaces ACME_CONTACT), ACME_DNS_CHALLENGE, ACME_DNS_PROVIDER. Obsolete: SKIP_SOLR, SOLR_HEAP, SOLR_PORT.TODO.md) so mailcow's cleanup events get created.~/Arbeitskasten/mailcow-conf-2026-10/ until you're sure you won't need to roll back, then delete it (it contains secrets and a full mail database dump).