initial commit

Signed-off-by: Jason Hall <imjasonh@gmail.com>
2026-05-07 20:02:59 -04:00 · 2026-05-07 20:02:59 -04:00 · 4dc1b58f2f
commit 4dc1b58f2f
20 changed files with 1398 additions and 0 deletions
--- a/docs/runbook.md
+++ b/docs/runbook.md
@ -0,0 +1,110 @@
+# Runbook
+
+Common operations against the running Forgejo VM.
+
+## Admin SSH
+
+Public port 22 is closed. Use IAP tunneling:
+
+```bash
+gcloud compute ssh forgejo --zone=us-east1-b --tunnel-through-iap
+```
+
+Your Google account needs:
+- `roles/iap.tunnelResourceAccessor` on the instance (granted by Terraform via `var.admin_email`)
+- `roles/compute.osLogin` on the project (same)
+- 2FA on the Google account (manual, but strongly recommended — IAP is only as strong as your login)
+
+## Inspect the stack
+
+```bash
+docker ps                                  # caddy, forgejo, watchtower expected
+docker logs --tail 200 forgejo
+docker logs --tail 200 caddy
+docker logs --tail 200 watchtower
+journalctl -u forgejo-stack.service -n 200
+journalctl -u forgejo-backup.service -n 50
+systemctl list-timers forgejo-backup.timer
+```
+
+## Restart the stack
+
+```bash
+sudo systemctl restart forgejo-stack.service
+```
+
+Single container only:
+
+```bash
+docker restart forgejo
+```
+
+## Update containers immediately
+
+Watchtower pulls new images at 04:00 UTC by default. To force now:
+
+```bash
+docker exec watchtower kill -s SIGHUP 1
+# or, manually:
+docker pull codeberg.org/forgejo/forgejo:11
+sudo systemctl restart forgejo-stack.service
+```
+
+## Run a backup on demand
+
+```bash
+sudo /var/lib/google/forgejo/backup.sh
+gsutil ls gs://YOUR_PROJECT-forgejo-backups/
+```
+
+## Restore from a backup
+
+`scripts/restore.sh` is in the repo, not on the VM. Copy it over and run:
+
+```bash
+gcloud compute scp scripts/restore.sh forgejo:/tmp/restore.sh \
+  --zone=us-east1-b --tunnel-through-iap
+gcloud compute ssh forgejo --zone=us-east1-b --tunnel-through-iap \
+  --command='sudo bash /tmp/restore.sh forgejo-20260507T033000Z.tar.gz'
+```
+
+For a clean-environment dry run, use `scripts/test-restore.sh` from your workstation — it pulls the latest backup, boots Forgejo against it in a throwaway container, and probes the API.
+
+## Forgejo major version upgrade
+
+1. Read the [release notes](https://codeberg.org/forgejo/forgejo/releases) for breaking changes.
+2. Take a manual backup (`sudo /var/lib/google/forgejo/backup.sh`).
+3. Bump `forgejo_image` in `terraform.tfvars` (e.g. `codeberg.org/forgejo/forgejo:12`).
+4. `terraform apply` — replaces the VM. The data disk persists; first boot runs DB migrations.
+5. Watch `docker logs forgejo` to confirm migrations and startup.
+
+## Resize the data disk
+
+GCP supports online disk growth:
+
+```bash
+gcloud compute disks resize forgejo-data --zone=us-east1-b --size=40
+```
+
+Then on the VM:
+
+```bash
+sudo resize2fs /dev/disk/by-id/google-forgejo-data
+```
+
+Update `size = 40` in `terraform/main.tf` afterward to keep state in sync.
+
+## Rotate secrets
+
+```bash
+# Add a new version (the latest is read at boot):
+openssl rand -hex 32 | gcloud secrets versions add forgejo-secret-key --data-file=-
+sudo systemctl restart forgejo-stack.service
+```
+
+Rotating `SECRET_KEY` invalidates 2FA and some encrypted DB fields. Read the Forgejo docs before rotating.
+
+## Cost / billing watch
+
+- Set a project budget alert at $10/month in Cloud Billing (manual; not in Terraform by design — the budget API requires the billing-account-admin role).
+- Skim the billing report monthly. Egress is the most likely surprise.