The two recent outages had two distinct root causes, both stemming from
`--restart=unless-stopped` on the three containers:
1. rm/run race: container metadata under /var/lib/docker/containers
persists across reboot. The daemon restores it asynchronously after
docker.service reports active, so the service's ExecStartPre rm -f
ran in a window where the containers were invisible to the daemon
(no-op), then `docker run --name caddy` later collided with the
just-restored container.
2. Bind-mount-before-mount race: docker's auto-restart of the previously
running containers happened *before* the mount unit activated
/mnt/disks/forgejo-data. The forgejo container ended up with `/data`
bind-mounted to the empty underlying mount-point directory on the
root fs, shadowed once the disk finally mounted -- presenting the
fresh-install setup screen despite the real DB sitting untouched on
/dev/sdb.
Drop --restart=unless-stopped and let systemd be the single owner of the
container lifecycle. Add --rm so containers don't persist on disk after
exit. Replace docker's mid-day auto-restart with a 1-minute watchdog
timer that restarts forgejo-stack.service if any container goes missing.
Add OnFailure=forgejo-stack-recover.service so a boot-time failure
self-heals after a 60s delay instead of waiting for me to notice.
Note: /etc on COS is overlayfs with upperdir on tmpfs, so the May 9
patch never took effect on reboot -- /etc/systemd/system/* is wiped
each boot and cloud-init re-renders from the metadata user-data. The
only durable path to changing units on this VM is terraform apply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Jason Hall
0890a62b46