# OTA updates over GHCR with cosign verification This document describes how OTA updates work end-to-end: how firmware is built and signed in CI, how it's distributed via an OCI registry, how the device discovers and applies updates, how per-device config gets onto the device in the first place, and how the device protects against bad updates and unauthorized signers. ## Architecture ``` GitHub Container Registry (GHCR) developer push │ ──────────────▶ ┌──────────────────┐ push ▼ poll │ GHA workflow │ ─────▶ ┌────────┐ ◀──────────── │ publish.yml │ sign │ OCI │ pull bundle │ • cargo build │ ─────▶ │artifact│ verify sig │ • espflash │ │ + sig │ stream blob │ • OCI push │ └────────┘ write OTA │ • cosign sign │ reboot │ (keyless, │ │ │ OIDC) │ ▼ └──────────────────┘ ┌────────┐ │ ESP32 │ └────────┘ ``` The published OCI artifact contains **only firmware code** — no Wi-Fi creds, no trust roots, no per-device config. Same bytes can run on any device. Per-device data lives in the device's NVS partition and is written via USB at provisioning time (see [Provisioning](#provisioning)). Manual `make publish` from a developer machine works the same way as the GHA workflow — just cosigns with a different OIDC identity (a developer's email instead of the workflow's URI). ## Components | Component | Lives at | Role | |---|---|---| | **Publisher** | `tools/publisher/` (Rust, host build) | Wraps the firmware `.bin` as an OCI artifact and pushes both `:latest` and `:sha-` tags to GHCR. Prints the manifest digest on stdout for the next step. | | **Provisioner** | `tools/provision/` (Rust, host build) | Reads `provisioning.toml`, builds an NVS partition image via ESP-IDF's `nvs_partition_gen.py`, flashes it to the device over USB. | | **Cosign** | system binary, version 3 | Signs the just-pushed digest using keyless OIDC. Stores the signature as a sibling OCI artifact (Sigstore Bundle v0.3 format). | | **GHA publish workflow** | `.github/workflows/publish.yml` | Runs `make publish` on every push to main. Uses ambient OIDC token for cosign — no secrets needed for signing. Only the auto-injected `GITHUB_TOKEN` is required. | | **CI workflow** | `.github/workflows/ci.yml` | Runs on PRs. Builds firmware + publisher + provisioner to verify they compile. No publishing or signing. | | **OTA loop** | `src/ota.rs` | Background pthread, polls GHCR every 60 s with backoff + jitter, fetches manifest, compares to last applied. | | **Sig verifier** | `src/sig.rs` | Parses Sigstore Bundle v0.3, verifies cert chain + DSSE signature + in-toto subject digest. | | **Trust loader** | `src/trust.rs` | Reads the allowlist of `(identity, issuer)` pairs and bundled Sigstore Fulcio root + intermediate CAs from NVS at boot. | ## Partition layout (4 MB flash) ``` 0x001000 bootloader (~28 KB) 0x008000 partition table 4 KB 0x009000 nvs 24 KB OTA state + provisioned per-device config 0x00f000 otadata 8 KB which slot to boot 0x011000 phy_init 4 KB 0x020000 ota_0 1.94 MB app slot 0 0x210000 ota_1 1.94 MB app slot 1 ``` The two app slots ping-pong: a new image always writes to the *inactive* slot, then `otadata` is updated to point at it on next boot. `partitions.csv` is the source of truth. The Makefile substitutes the absolute project path into `sdkconfig.defaults.in` because IDF resolves `CONFIG_PARTITION_TABLE_CUSTOM_FILENAME` relative to embuild's synthetic project under `target/`, not the repo root. ## NVS schema Per-device config and runtime OTA state live in NVS, organized by namespace. `tools/provision/` writes the config; the firmware reads at boot. ``` ns key type notes ───────────────────────────────────────────────────────────────────── wifi ssid str network SSID (provisioned) wifi pass str PSK (provisioned) ota last_digest str last successfully-applied layer digest (runtime) ota pending_digest str set during update, promoted on mark-valid (runtime) ota repo str optional override of compile-time default ota tag str optional override (default "latest") ota poll_secs u32 optional override (default 60) trust identities blob JSON: [{"identity":"...","issuer":"..."}, ...] trust fulcio_root blob PEM bytes (Sigstore root CA) trust fulcio_inter blob PEM bytes (Sigstore intermediate CA) gcp project_id str optional; cloud-logging GCP project gcp sa_email str optional; logging service-account email gcp sa_key_id str optional; key id for the JWT `kid` header gcp sa_key_pem blob optional; RSA private key PKCS#8 PEM gcp min_severity u8 optional; 0=TRACE..4=ERROR (default 2=INFO) ``` The `gcp` namespace is opt-in. If any required key is missing, the device boots with serial-only logging and never talks to GCP. See [`observability.md`](observability.md). The `wifi` and `trust` namespaces are written by `make provision` and never touched by OTA. The `ota` namespace is written at runtime by the firmware's OTA loop. ## Provisioning ### `provisioning.toml` The single source of truth for per-device config. Gitignored. Copy from `provisioning.toml.example`, fill in: ```toml [wifi] ssid = "..." pass = "..." [trust] fulcio_root_pem = "trust/fulcio_root.pem" fulcio_intermediate_pem = "trust/fulcio_intermediate.pem" [[trust.identities]] identity = "imjasonh@gmail.com" issuer = "https://accounts.google.com" [[trust.identities]] identity = "https://github.com/imjasonh/esp32/.github/workflows/publish.yml@refs/heads/main" issuer = "https://token.actions.githubusercontent.com" ``` Lose this file = lose your secrets. Keep a copy in a password manager. ### `tools/provision/` flow 1. Reads `provisioning.toml`. 2. Generates an intermediate CSV in the format ESP-IDF's `nvs_partition_gen.py` expects (one row per key, `binary` rows reference files on disk). 3. Shells out to `nvs_partition_gen.py` (in `.embuild/`, present after `make build` has run once) to produce a 24 KB NVS partition image. 4. Optionally `espflash write-bin --address 0x9000 target/nvs.bin` to write it to the device. ### Bootstrapping a new device ``` make provisioning.toml # creates from template (one-time) $EDITOR provisioning.toml # fill in wifi creds + trust identities make bootstrap # build, flash everything, write NVS make monitor # watch it boot and connect ``` `make bootstrap` is `flash-all` + `provision`. After it completes the device boots, reads NVS for wifi + trust, connects to Wi-Fi, starts polling GHCR. From then on updates flow over OTA. ### Strict-inert when not provisioned If a device boots and any of the required NVS keys (`wifi/ssid`, `wifi/pass`, `trust/identities`, `trust/fulcio_root`, `trust/fulcio_inter`) is missing, the firmware logs `NOT PROVISIONED — run \`make provision\`` to serial every 30 seconds and refuses to start Wi-Fi or OTA. No surprise boots, no fallback creds. Run `make provision` to fix. ## Update mechanism ### Polling loop (firmware, `src/ota.rs::run`) 1. Sleep `poll_interval ± 10%` jitter (default 60 s, NVS-configurable via `ota/poll_secs`). 2. Hit GHCR's anonymous token endpoint with scope `repository::pull`, get a Bearer token. 3. `GET /v2//manifests/` with that token. Compute SHA-256 of the response body — that's the manifest digest cosign signed. 4. Parse manifest. Check that `layers[0].mediaType` is `application/vnd.esp32.firmware.bin`. Compare layer digest to `last_digest` from NVS. - Match → `NoChange`, return to (1). - Different → continue. 5. **Verify the signature** (see [Device verification](#device-verification-srcsigrs) below). On failure → log, increment `consecutive_failures`, return to (1) with exponential backoff (`60 s × 2^failures`, capped at 1 h). 6. `GET /v2//blobs/`. Stream the body chunk by chunk into the inactive OTA partition via `EspOta::write()`, updating an SHA-256 hasher in parallel. 7. After last byte: confirm `(actual size, actual SHA)` matches the manifest descriptor. On any mismatch → `EspOta::abort()`, fail loudly. On match → `EspOta::complete()` (sets boot partition). 8. Persist `pending_digest = layer.digest` to NVS. 9. `esp_restart()`. ### Post-reboot validation (firmware, `src/main.rs`) 1. Early in `main()`, call `is_pending_verify()` → reads `esp_ota_get_state_partition()`. 2. If `ESP_OTA_IMG_PENDING_VERIFY`, run the bringup checks: Wi-Fi connect, then HTTPS GETs to ipify and wttr. 3. On bringup success, call `esp_ota_mark_app_valid_cancel_rollback()` and promote `pending_digest → last_digest` in NVS. 4. On failure, `esp_restart()` — the bootloader rolls back to the previous slot on the next boot because the new image was never marked valid. 5. Spawn the OTA polling thread with a 48 KB stack (HTTPS + JSON + SHA + cert parsing + ECDSA needs the headroom). ## Anti-bricking + USB recovery Two layers of protection: ### Layer 1 — bootloader rollback (always armed in firmware) `CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y` in `sdkconfig.defaults.in` turns on ESP-IDF's pending-verify state. After an OTA reboot, the new app runs in `PENDING_VERIFY`. If `mark_app_valid` isn't called before the next reboot, the bootloader reverts. We gate `mark_app_valid` on networking actually working — so an OTA that breaks Wi-Fi or DNS or TLS auto-rolls back. ### Layer 2 — USB recovery (when both slots are bad) If both OTA slots end up unbootable, or the bootloader / partition table itself is corrupted, recover over USB: 1. Put the chip in serial download mode. The Inland board has auto-reset; espflash usually drives DTR/RTS automatically. If not: hold **BOOT** (IO0), tap **EN**, release BOOT. 2. From the project root: ``` make bootstrap # full flash + re-provision # or: make flash-all # full flash without re-provisioning (NVS preserved # only if not previously erased) ``` `make bootstrap` wipes flash entirely and writes everything back including NVS, so it's the most reliable recovery path. Brief downtime; no risk of brick. ## CI signing (publisher → GHCR) ### Publisher tool `tools/publisher/` is a separate cargo project (its own `rust-toolchain.toml` pinning `stable`, its own `.cargo/config.toml` clearing the parent's xtensa target). Built for the host. CLI: ``` publisher push --bin --repo ghcr.io// --git-sha ``` Produces an OCI manifest with: - **Config**: `application/vnd.esp32.firmware.v1+json` — small JSON with `target_chip`, `idf_version`, `git_sha`, `built_at`, `bin_size`, `bin_sha256`. - **Layer**: `application/vnd.esp32.firmware.bin` — the raw firmware bytes (no compression). - **Annotations**: `org.opencontainers.image.source` so GHCR auto-links the package to the source repo on first push; `org.opencontainers.image.revision`, `created`. Pushes the same artifact under both `:latest` and `:sha-`. Both tags resolve to the same content-addressed digest, so one cosign sign covers both. The publisher prints `digest: sha256:...` on stdout. The Makefile captures it and passes it to `cosign sign` as `@` rather than a tag — eliminates the race window between publisher push and cosign sign. ### Cosign signing `make publish` runs `cosign sign --yes @` after the publisher push. Cosign: 1. Authenticates to its OIDC issuer (Sigstore's broker for interactive browser flow; or the workflow's ambient ID token in GHA). 2. Fetches a short-lived (10 min) X.509 cert from Fulcio with the OIDC identity baked in as a SAN. 3. Hashes the manifest payload, signs it with the cert's private key. 4. Bundles cert + signature + (optional) Rekor entry into a Sigstore Bundle v0.3 (Protobuf-as-JSON). 5. Pushes the bundle as a sibling OCI artifact at tag `sha256-` (no `.sig` suffix) using the OCI 1.1 referrers layout (image index → inner manifest → bundle blob). ### Identities used | Source | OIDC issuer | SAN form | |---|---|---| | Manual `make publish` | `https://accounts.google.com` | `rfc822Name` (developer's email) | | GHA workflow on push to main | `https://token.actions.githubusercontent.com` | `URI` (`https://github.com///.github/workflows/publish.yml@refs/heads/main`) | Both must be present in `trust/identities` (NVS) for the device to accept their signatures. Updating the allowlist requires editing `provisioning.toml` and re-running `make provision`; OTA cannot grant new signing identities. ### GHA workflow specifics - Trigger: `push` to `main`, plus `workflow_dispatch` for manual reruns. - Concurrency: `cancel-in-progress: true` with a single group, so a rapid sequence of pushes only publishes the latest commit. - Permissions: `packages: write` (push to GHCR), `id-token: write` (cosign keyless OIDC), `contents: read`. - Uses `astral-sh/setup-uv@v7` to provide Python 3.12 (the Makefile's `ensure-python-shim` symlinks `python3` to it), `esp-rs/xtensa-toolchain@v1.7` for the Xtensa Rust toolchain, `sigstore/cosign-installer@v4.1.1` for cosign 3. - Only secret needed: the auto-injected `GITHUB_TOKEN` (gets written to `gh.env` for cosign's registry auth). Wi-Fi creds and trust config are not in the firmware image and therefore not in the build. ## Device verification (`src/sig.rs`) For each candidate update, before downloading the firmware blob: 1. **Fetch the bundle.** Walk the OCI 1.1 referrers layout: image index at tag `sha256-` → inner manifest → `application/vnd.dev.sigstore.bundle.v0.3+json` layer blob. 2. **Parse the leaf cert.** Decode `verificationMaterial.certificate. rawBytes` from base64, parse as DER X.509 with the `x509-cert` crate. 3. **Identity check.** Read SAN extension. Accept either `Rfc822Name` (email) or `UniformResourceIdentifier` (workflow URI). Read OIDC issuer from extension OID `1.3.6.1.4.1.57264.1.1`. Reject if `(identity, issuer)` isn't in `trust/identities` (loaded from NVS at boot). 4. **Cert chain.** Verify leaf was signed by the bundled Sigstore intermediate (P-384 ECDSA-SHA384). Verify intermediate was signed by the bundled Sigstore root (also P-384 ECDSA-SHA384). Both are loaded from NVS (`trust/fulcio_inter`, `trust/fulcio_root`). 5. **DSSE signature.** Decode `dsseEnvelope.payload` (base64) and `signatures[0].sig` (base64). Compute the DSSE PAE (`"DSSEv1 "`). Verify the ECDSA-P256 signature using the leaf cert's public key. 6. **In-toto binding.** Parse the DSSE payload as an in-toto Statement. Confirm `subject[0].digest.sha256` exactly matches the manifest digest we're about to install. This is the cryptographic binding from "what the signer attested" to "what we're about to apply". If any step fails, the verifier returns an error. The OTA loop logs it (with anyhow chain), bumps `consecutive_failures` (driving backoff), and never touches the OTA partition. ## Trust separation: soft vs hard Storing `TRUSTED_IDENTITIES` and the Sigstore CAs in NVS gives a **clean architectural separation**: the OTA-distributed firmware contains no policy data, and the same image runs on every device. But it's a **soft guarantee**. The OTA-distributed firmware code is what reads NVS and runs the verifier; a malicious image could ignore NVS and hardcode "trust everyone". To prevent that cryptographically, we'd need **Secure Boot v2** (RSA-3072 keypair, public-key digest burned into eFuses, bootloader cryptographically verifies the app before running) plus **Flash Encryption** (also eFuse-rooted) so an attacker with physical access can't dump flash to extract the secrets that NVS holds. Both are irreversible eFuse burns; significant effort and risk; massive security upgrade. See [Future work](#future-work). In practice the soft guarantee is enough here: - The OCI image is no longer a per-device artifact — same bytes everywhere, no embedded secrets. - Trust changes happen via a deliberate USB re-provision, not as a side effect of regular OTA. - The convention is enforced by code review of the firmware that gets signed and shipped. ## Future work - **Rekor SET verification.** The Sigstore Bundle includes `tlogEntries[]` with Rekor's Signed Entry Timestamp. Verifying it on-device would give us (a) detection of a Fulcio key compromise that doesn't appear in the public log and (b) a trusted timestamp to enforce the cert validity window (Fulcio certs are 10 min, so without trusted time we currently skip the validity check). Bundle Rekor's public key, parse the SET, verify the inclusion proof. - **GitHub Actions OIDC trust scoping by `job_workflow_ref`.** The GHA identity in `trust/identities` pins to `publish.yml@refs/heads/main`. We could also enforce `1.3.6.1.4.1.57264.1.x` Fulcio extensions like `Run Invocation URI` for stricter trust. - **Force-update GPIO button.** Wire a button to a GPIO; on short press, skip the poll wait and trigger an immediate poll. - **Auto-reboot on bringup failure.** If `connect_wifi` errors during PENDING_VERIFY, currently `main` exits with the error and IDF doesn't auto-reboot — the user has to power-cycle to trigger the bootloader rollback. Cleaner: explicit `esp_restart()` so rollback happens automatically. - **`If-None-Match` on manifest fetches.** Marginal bandwidth savings given the manifest is ~500 bytes and we already digest-compare locally. Worth it only if poll interval drops well below 60 s. - **Secure Boot v2 + Flash Encryption.** Hardware-rooted trust and confidentiality. Irreversible eFuse burns. Closes the soft-trust gap above. - **NVS encryption** without Secure Boot — limited value (any flasher can disable it without Secure Boot to root the chain), but trivial to enable. Probably not worth it alone. - **Provisioning over BLE or Wi-Fi AP** — for at-scale fleet provisioning. Standard ESP-IDF `wifi_provisioning` component handles this. Adds firmware size; only worth it if we ever ship more than one device. - **Remote re-provisioning of non-secret keys** (poll interval, log severity, etc.) via the OTA channel — push small "config artifacts" alongside firmware artifacts in OCI. Combined with the next bullet, enables seamless SA-key rotation. - **Rotating the SA key** — currently means re-provisioning every device over USB. With remote re-provisioning above, becomes a push button.