The nightly reboot at 04:30 UTC takes ~7 min of downtime (cloud-init,
image pulls, container startup) and was tripping the 5-min alert. Jason
is fine with longer downtime overnight and prefers no page for it.
15 min still catches a stuck reboot or a real outage with reasonable
latency.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The two recent outages had two distinct root causes, both stemming from
`--restart=unless-stopped` on the three containers:
1. rm/run race: container metadata under /var/lib/docker/containers
persists across reboot. The daemon restores it asynchronously after
docker.service reports active, so the service's ExecStartPre rm -f
ran in a window where the containers were invisible to the daemon
(no-op), then `docker run --name caddy` later collided with the
just-restored container.
2. Bind-mount-before-mount race: docker's auto-restart of the previously
running containers happened *before* the mount unit activated
/mnt/disks/forgejo-data. The forgejo container ended up with `/data`
bind-mounted to the empty underlying mount-point directory on the
root fs, shadowed once the disk finally mounted -- presenting the
fresh-install setup screen despite the real DB sitting untouched on
/dev/sdb.
Drop --restart=unless-stopped and let systemd be the single owner of the
container lifecycle. Add --rm so containers don't persist on disk after
exit. Replace docker's mid-day auto-restart with a 1-minute watchdog
timer that restarts forgejo-stack.service if any container goes missing.
Add OnFailure=forgejo-stack-recover.service so a boot-time failure
self-heals after a 60s delay instead of waiting for me to notice.
Note: /etc on COS is overlayfs with upperdir on tmpfs, so the May 9
patch never took effect on reboot -- /etc/systemd/system/* is wiped
each boot and cloud-init re-renders from the metadata user-data. The
only durable path to changing units on this VM is terraform apply.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Last night's reboot still failed because commit 71c056f's rm -f changes
were authored but never deployed to the running VM (terraform was not
re-applied after the commit). Even with that fix in place, there is a
narrow race on COS: docker.service can report active before the daemon
finishes restoring persisted containers from disk, in which case the
ExecStartPre rm -f is a no-op and the subsequent docker run collides
with the just-restored container.
Add a wait-loop ExecStartPre that polls docker ps -a for up to 30s and
breaks as soon as any of caddy/forgejo/watchtower appears. Once one is
visible, the daemon has finished restoration and rm -f reliably wins.
On a clean first boot it spins the full 30s once, then proceeds.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The original ExecStop=docker stop left containers in Exited state, so the
next boot's ExecStart=docker run -d --name caddy collided with the still-
existing container and the unit failed at status=125. Site stayed down
across the first nightly reboot until manual cleanup.
- ExecStartPre=docker rm -f (idempotent; leading - ignores not-found)
- ExecStop=docker rm -f (actually removes on shutdown, not just stops)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- uptime check probes https://forge.kontain.me/api/v1/version every 60s
- alert: Forgejo is down (uptime fails 5+ min)
- alert: TLS cert expiring (<7 days remaining)
- alert: data disk >80% full (hourly disk-check.timer writes DISK_HIGH to journal; log-based metric)
- alert: container logs match panic/FATAL (cos_containers log-based metric)
- all alerts email admin_email via the existing notification channel
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
