imjasonh/forge
1
0
Fork 0
Code Activity
No description
6 commits 1 branch 0 tags 69 KiB
Find a file
Jason Hall 0890a62b46 forgejo-stack: stop fighting docker's restart policy; add watchdog 
The two recent outages had two distinct root causes, both stemming from
`--restart=unless-stopped` on the three containers:

1. rm/run race: container metadata under /var/lib/docker/containers
   persists across reboot. The daemon restores it asynchronously after
   docker.service reports active, so the service's ExecStartPre rm -f
   ran in a window where the containers were invisible to the daemon
   (no-op), then `docker run --name caddy` later collided with the
   just-restored container.
2. Bind-mount-before-mount race: docker's auto-restart of the previously
   running containers happened *before* the mount unit activated
   /mnt/disks/forgejo-data. The forgejo container ended up with `/data`
   bind-mounted to the empty underlying mount-point directory on the
   root fs, shadowed once the disk finally mounted -- presenting the
   fresh-install setup screen despite the real DB sitting untouched on
   /dev/sdb.

Drop --restart=unless-stopped and let systemd be the single owner of the
container lifecycle. Add --rm so containers don't persist on disk after
exit. Replace docker's mid-day auto-restart with a 1-minute watchdog
timer that restarts forgejo-stack.service if any container goes missing.
Add OnFailure=forgejo-stack-recover.service so a boot-time failure
self-heals after a 60s delay instead of waiting for me to notice.

Note: /etc on COS is overlayfs with upperdir on tmpfs, so the May 9
patch never took effect on reboot -- /etc/systemd/system/* is wiped
each boot and cloud-init re-renders from the metadata user-data. The
only durable path to changing units on this VM is terraform apply.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-11 10:55:37 -04:00
cloud-init forgejo-stack: stop fighting docker's restart policy; add watchdog 2026-05-11 10:55:37 -04:00
config initial commit 2026-05-07 20:02:59 -04:00
docs add budget alert and nightly OS-update reboot 2026-05-07 20:35:58 -04:00
scripts initial commit 2026-05-07 20:02:59 -04:00
terraform add uptime check and four alert policies 2026-05-07 21:35:43 -04:00
.gitignore initial commit 2026-05-07 20:02:59 -04:00
plan.md initial commit 2026-05-07 20:02:59 -04:00
README.md initial commit 2026-05-07 20:02:59 -04:00

README.md

forge

Self-hosted Forgejo on GCP. e2-micro VM on Container-Optimized OS, Caddy for HTTPS, IAP for admin SSH, nightly backups to GCS. Targets ~$3–5/month.

See plan.md for the full design rationale.

Quick start

# 1. Set the active project, enable APIs
gcloud config set project YOUR_PROJECT
gcloud services enable compute.googleapis.com secretmanager.googleapis.com \
  iap.googleapis.com storage.googleapis.com

# 2. Generate the Forgejo secrets in Secret Manager (one-time)
./scripts/bootstrap-secrets.sh

# 3. Configure Terraform (terraform.tfvars is gitignored)
$EDITOR terraform/terraform.tfvars

# 4. Apply
cd terraform
terraform init
terraform apply

Point your domain's A record at the static_ip output, then visit https://<your-domain> to run the Forgejo installer.

Day two

Layout

terraform/    GCP infrastructure (VM, network, IAM, GCS, optional DNS)
cloud-init/   user-data.yaml.tpl — systemd units that boot Forgejo + Caddy + Watchtower
config/       Caddyfile template (reference; the live copy is embedded in cloud-init)
scripts/      bootstrap-secrets.sh, backup.sh, restore.sh, test-restore.sh
docs/         runbook + disaster recovery