esp32/ota.md

OTA updates over GHCR with cosign verification

This document describes how OTA updates work end-to-end: how firmware is built and signed in CI, how it's distributed via an OCI registry, how the device discovers and applies updates, how per-device config gets onto the device in the first place, and how the device protects against bad updates and unauthorized signers.

Architecture

                                                  GitHub
                                                Container
                                                Registry
                                                  (GHCR)
   developer push                                    │
   ──────────────▶  ┌──────────────────┐ push       ▼            poll
                    │  GHA workflow    │ ─────▶ ┌────────┐ ◀────────────
                    │  publish.yml     │ sign   │  OCI   │   pull bundle
                    │   • cargo build  │ ─────▶ │artifact│   verify sig
                    │   • espflash     │        │ + sig  │   stream blob
                    │   • OCI push     │        └────────┘   write OTA
                    │   • cosign sign  │                     reboot
                    │     (keyless,    │                       │
                    │      OIDC)       │                       ▼
                    └──────────────────┘                   ┌────────┐
                                                           │ ESP32  │
                                                           └────────┘

The published OCI artifact contains only firmware code — no Wi-Fi creds, no trust roots, no per-device config. Same bytes can run on any device. Per-device data lives in the device's NVS partition and is written via USB at provisioning time (see Provisioning).

Manual make publish from a developer machine works the same way as the GHA workflow — just cosigns with a different OIDC identity (a developer's email instead of the workflow's URI).

Components

Component	Lives at	Role
Publisher	tools/publisher/ (Rust, host build)	Wraps the firmware .bin as an OCI artifact and pushes both :latest and :sha-<short> tags to GHCR. Prints the manifest digest on stdout for the next step.
Provisioner	tools/provision/ (Rust, host build)	Reads provisioning.toml, builds an NVS partition image via ESP-IDF's nvs_partition_gen.py, flashes it to the device over USB.
Cosign	system binary, version 3	Signs the just-pushed digest using keyless OIDC. Stores the signature as a sibling OCI artifact (Sigstore Bundle v0.3 format).
GHA publish workflow	.github/workflows/publish.yml	Runs make publish on every push to main. Uses ambient OIDC token for cosign — no secrets needed for signing. Only the auto-injected GITHUB_TOKEN is required.
CI workflow	.github/workflows/ci.yml	Runs on PRs. Builds firmware + publisher + provisioner to verify they compile. No publishing or signing.
OTA loop	src/ota.rs	Background pthread, polls GHCR every 60 s with backoff + jitter, fetches manifest, compares to last applied.
Sig verifier	src/sig.rs	Parses Sigstore Bundle v0.3, verifies cert chain + DSSE signature + in-toto subject digest.
Trust loader	src/trust.rs	Reads the allowlist of (identity, issuer) pairs and bundled Sigstore Fulcio root + intermediate CAs from NVS at boot.

Partition layout (4 MB flash)

0x001000  bootloader             (~28 KB)
0x008000  partition table          4 KB
0x009000  nvs                     24 KB    OTA state + provisioned per-device config
0x00f000  otadata                  8 KB    which slot to boot
0x011000  phy_init                 4 KB
0x020000  ota_0                  1.94 MB   app slot 0
0x210000  ota_1                  1.94 MB   app slot 1

The two app slots ping-pong: a new image always writes to the inactive slot, then otadata is updated to point at it on next boot.

partitions.csv is the source of truth. The Makefile substitutes the absolute project path into sdkconfig.defaults.in because IDF resolves CONFIG_PARTITION_TABLE_CUSTOM_FILENAME relative to embuild's synthetic project under target/, not the repo root.

NVS schema

Per-device config and runtime OTA state live in NVS, organized by namespace. tools/provision/ writes the config; the firmware reads at boot.

ns      key             type   notes
─────────────────────────────────────────────────────────────────────
wifi    ssid            str    network SSID (provisioned)
wifi    pass            str    PSK (provisioned)

ota     last_digest     str    last successfully-applied layer digest (runtime)
ota     pending_digest  str    set during update, promoted on mark-valid (runtime)
ota     repo            str    optional override of compile-time default
ota     tag             str    optional override (default "latest")
ota     poll_secs       u32    optional override (default 60)

trust   identities      blob   JSON: [{"identity":"...","issuer":"..."}, ...]
trust   fulcio_root     blob   PEM bytes (Sigstore root CA)
trust   fulcio_inter    blob   PEM bytes (Sigstore intermediate CA)

gcp     project_id      str    optional; cloud-logging GCP project
gcp     sa_email        str    optional; logging service-account email
gcp     sa_key_id       str    optional; key id for the JWT `kid` header
gcp     sa_key_pem      blob   optional; RSA private key PKCS#8 PEM
gcp     min_severity    u8     optional; 0=TRACE..4=ERROR (default 2=INFO)

The gcp namespace is opt-in. If any required key is missing, the device boots with serial-only logging and never talks to GCP. See logs-plan.md.

The wifi and trust namespaces are written by make provision and never touched by OTA. The ota namespace is written at runtime by the firmware's OTA loop.

Provisioning

provisioning.toml

The single source of truth for per-device config. Gitignored. Copy from provisioning.toml.example, fill in:

[wifi]
ssid = "..."
pass = "..."

[trust]
fulcio_root_pem = "trust/fulcio_root.pem"
fulcio_intermediate_pem = "trust/fulcio_intermediate.pem"

[[trust.identities]]
identity = "imjasonh@gmail.com"
issuer = "https://accounts.google.com"

[[trust.identities]]
identity = "https://github.com/imjasonh/esp32/.github/workflows/publish.yml@refs/heads/main"
issuer = "https://token.actions.githubusercontent.com"

Lose this file = lose your secrets. Keep a copy in a password manager.

tools/provision/ flow

1. Reads provisioning.toml.
2. Generates an intermediate CSV in the format ESP-IDF's nvs_partition_gen.py expects (one row per key, binary rows reference files on disk).
3. Shells out to nvs_partition_gen.py (in .embuild/, present after make build has run once) to produce a 24 KB NVS partition image.
4. Optionally espflash write-bin --address 0x9000 target/nvs.bin to write it to the device.

Bootstrapping a new device

make provisioning.toml             # creates from template (one-time)
$EDITOR provisioning.toml          # fill in wifi creds + trust identities
make bootstrap                     # build, flash everything, write NVS
make monitor                       # watch it boot and connect

make bootstrap is flash-all + provision. After it completes the device boots, reads NVS for wifi + trust, connects to Wi-Fi, starts polling GHCR. From then on updates flow over OTA.

Strict-inert when not provisioned

If a device boots and any of the required NVS keys (wifi/ssid, wifi/pass, trust/identities, trust/fulcio_root, trust/fulcio_inter) is missing, the firmware logs NOT PROVISIONED — run \make provision`to serial every 30 seconds and refuses to start Wi-Fi or OTA. No surprise boots, no fallback creds. Runmake provision` to fix.

Update mechanism

Polling loop (firmware, src/ota.rs::run)

1. Sleep poll_interval ± 10% jitter (default 60 s, NVS-configurable via ota/poll_secs).
2. Hit GHCR's anonymous token endpoint with scope repository:<repo>:pull, get a Bearer token.
3. GET /v2/<repo>/manifests/<tag> with that token. Compute SHA-256 of the response body — that's the manifest digest cosign signed.
4. Parse manifest. Check that layers[0].mediaType is application/vnd.esp32.firmware.bin. Compare layer digest to last_digest from NVS.
   • Match → NoChange, return to (1).
   • Different → continue.
5. Verify the signature (see Device verification below). On failure → log, increment consecutive_failures, return to (1) with exponential backoff (60 s × 2^failures, capped at 1 h).
6. GET /v2/<repo>/blobs/<layer-digest>. Stream the body chunk by chunk into the inactive OTA partition via EspOta::write(), updating an SHA-256 hasher in parallel.
7. After last byte: confirm (actual size, actual SHA) matches the manifest descriptor. On any mismatch → EspOta::abort(), fail loudly. On match → EspOta::complete() (sets boot partition).
8. Persist pending_digest = layer.digest to NVS.
9. esp_restart().

Post-reboot validation (firmware, src/main.rs)

1. Early in main(), call is_pending_verify() → reads esp_ota_get_state_partition().
2. If ESP_OTA_IMG_PENDING_VERIFY, run the bringup checks: Wi-Fi connect, then HTTPS GETs to ipify and wttr.
3. On bringup success, call esp_ota_mark_app_valid_cancel_rollback() and promote pending_digest → last_digest in NVS.
4. On failure, esp_restart() — the bootloader rolls back to the previous slot on the next boot because the new image was never marked valid.
5. Spawn the OTA polling thread with a 48 KB stack (HTTPS + JSON + SHA + cert parsing + ECDSA needs the headroom).

Anti-bricking + USB recovery

Two layers of protection:

Layer 1 — bootloader rollback (always armed in firmware)

CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE=y in sdkconfig.defaults.in turns on ESP-IDF's pending-verify state. After an OTA reboot, the new app runs in PENDING_VERIFY. If mark_app_valid isn't called before the next reboot, the bootloader reverts. We gate mark_app_valid on networking actually working — so an OTA that breaks Wi-Fi or DNS or TLS auto-rolls back.

Layer 2 — USB recovery (when both slots are bad)

If both OTA slots end up unbootable, or the bootloader / partition table itself is corrupted, recover over USB:

1. Put the chip in serial download mode. The Inland board has auto-reset; espflash usually drives DTR/RTS automatically. If not: hold BOOT (IO0), tap EN, release BOOT.
2. From the project root:

   make bootstrap   # full flash + re-provision
   # or:
   make flash-all   # full flash without re-provisioning (NVS preserved
                    # only if not previously erased)
   

make bootstrap wipes flash entirely and writes everything back including NVS, so it's the most reliable recovery path. Brief downtime; no risk of brick.

CI signing (publisher → GHCR)

Publisher tool

tools/publisher/ is a separate cargo project (its own rust-toolchain.toml pinning stable, its own .cargo/config.toml clearing the parent's xtensa target). Built for the host.

CLI:

publisher push --bin <fw.bin> --repo ghcr.io/<owner>/<name> --git-sha <short>

Produces an OCI manifest with:

• Config: application/vnd.esp32.firmware.v1+json — small JSON with target_chip, idf_version, git_sha, built_at, bin_size, bin_sha256.
• Layer: application/vnd.esp32.firmware.bin — the raw firmware bytes (no compression).
• Annotations: org.opencontainers.image.source so GHCR auto-links the package to the source repo on first push; org.opencontainers.image.revision, created.

Pushes the same artifact under both :latest and :sha-<short>. Both tags resolve to the same content-addressed digest, so one cosign sign covers both.

The publisher prints digest: sha256:... on stdout. The Makefile captures it and passes it to cosign sign as <repo>@<digest> rather than a tag — eliminates the race window between publisher push and cosign sign.

Cosign signing

make publish runs cosign sign --yes <repo>@<digest> after the publisher push. Cosign:

1. Authenticates to its OIDC issuer (Sigstore's broker for interactive browser flow; or the workflow's ambient ID token in GHA).
2. Fetches a short-lived (10 min) X.509 cert from Fulcio with the OIDC identity baked in as a SAN.
3. Hashes the manifest payload, signs it with the cert's private key.
4. Bundles cert + signature + (optional) Rekor entry into a Sigstore Bundle v0.3 (Protobuf-as-JSON).
5. Pushes the bundle as a sibling OCI artifact at tag sha256-<hex> (no .sig suffix) using the OCI 1.1 referrers layout (image index → inner manifest → bundle blob).

Identities used

Source	OIDC issuer	SAN form
Manual make publish	https://accounts.google.com	rfc822Name (developer's email)
GHA workflow on push to main	https://token.actions.githubusercontent.com	URI (https://github.com/<owner>/<repo>/.github/workflows/publish.yml@refs/heads/main)

Both must be present in trust/identities (NVS) for the device to accept their signatures. Updating the allowlist requires editing provisioning.toml and re-running make provision; OTA cannot grant new signing identities.

GHA workflow specifics

• Trigger: push to main, plus workflow_dispatch for manual reruns.
• Concurrency: cancel-in-progress: true with a single group, so a rapid sequence of pushes only publishes the latest commit.
• Permissions: packages: write (push to GHCR), id-token: write (cosign keyless OIDC), contents: read.
• Uses astral-sh/setup-uv@v7 to provide Python 3.12 (the Makefile's ensure-python-shim symlinks python3 to it), esp-rs/xtensa-toolchain@v1.7 for the Xtensa Rust toolchain, sigstore/cosign-installer@v4.1.1 for cosign 3.
• Only secret needed: the auto-injected GITHUB_TOKEN (gets written to gh.env for cosign's registry auth). Wi-Fi creds and trust config are not in the firmware image and therefore not in the build.

Device verification (src/sig.rs)

For each candidate update, before downloading the firmware blob:

1. Fetch the bundle. Walk the OCI 1.1 referrers layout: image index at tag sha256-<manifest-digest> → inner manifest → application/vnd.dev.sigstore.bundle.v0.3+json layer blob.
2. Parse the leaf cert. Decode verificationMaterial.certificate. rawBytes from base64, parse as DER X.509 with the x509-cert crate.
3. Identity check. Read SAN extension. Accept either Rfc822Name (email) or UniformResourceIdentifier (workflow URI). Read OIDC issuer from extension OID 1.3.6.1.4.1.57264.1.1. Reject if (identity, issuer) isn't in trust/identities (loaded from NVS at boot).
4. Cert chain. Verify leaf was signed by the bundled Sigstore intermediate (P-384 ECDSA-SHA384). Verify intermediate was signed by the bundled Sigstore root (also P-384 ECDSA-SHA384). Both are loaded from NVS (trust/fulcio_inter, trust/fulcio_root).
5. DSSE signature. Decode dsseEnvelope.payload (base64) and signatures[0].sig (base64). Compute the DSSE PAE ("DSSEv1 <len> <payloadType> <len> <payload>"). Verify the ECDSA-P256 signature using the leaf cert's public key.
6. In-toto binding. Parse the DSSE payload as an in-toto Statement. Confirm subject[0].digest.sha256 exactly matches the manifest digest we're about to install. This is the cryptographic binding from "what the signer attested" to "what we're about to apply".

If any step fails, the verifier returns an error. The OTA loop logs it (with anyhow chain), bumps consecutive_failures (driving backoff), and never touches the OTA partition.

Trust separation: soft vs hard

Storing TRUSTED_IDENTITIES and the Sigstore CAs in NVS gives a clean architectural separation: the OTA-distributed firmware contains no policy data, and the same image runs on every device.

But it's a soft guarantee. The OTA-distributed firmware code is what reads NVS and runs the verifier; a malicious image could ignore NVS and hardcode "trust everyone". To prevent that cryptographically, we'd need Secure Boot v2 (RSA-3072 keypair, public-key digest burned into eFuses, bootloader cryptographically verifies the app before running) plus Flash Encryption (also eFuse-rooted) so an attacker with physical access can't dump flash to extract the secrets that NVS holds. Both are irreversible eFuse burns; significant effort and risk; massive security upgrade. See Future work.

In practice the soft guarantee is enough here:

• The OCI image is no longer a per-device artifact — same bytes everywhere, no embedded secrets.
• Trust changes happen via a deliberate USB re-provision, not as a side effect of regular OTA.
• The convention is enforced by code review of the firmware that gets signed and shipped.

Future work

• Rekor SET verification. The Sigstore Bundle includes tlogEntries[] with Rekor's Signed Entry Timestamp. Verifying it on-device would give us (a) detection of a Fulcio key compromise that doesn't appear in the public log and (b) a trusted timestamp to enforce the cert validity window (Fulcio certs are 10 min, so without trusted time we currently skip the validity check). Bundle Rekor's public key, parse the SET, verify the inclusion proof.
• GitHub Actions OIDC trust scoping by job_workflow_ref. The GHA identity in trust/identities pins to publish.yml@refs/heads/main. We could also enforce 1.3.6.1.4.1.57264.1.x Fulcio extensions like Run Invocation URI for stricter trust.
• Force-update GPIO button. Wire a button to a GPIO; on short press, skip the poll wait and trigger an immediate poll.
• Auto-reboot on bringup failure. If connect_wifi errors during PENDING_VERIFY, currently main exits with the error and IDF doesn't auto-reboot — the user has to power-cycle to trigger the bootloader rollback. Cleaner: explicit esp_restart() so rollback happens automatically.
• If-None-Match on manifest fetches. Marginal bandwidth savings given the manifest is ~500 bytes and we already digest-compare locally. Worth it only if poll interval drops well below 60 s.
• Secure Boot v2 + Flash Encryption. Hardware-rooted trust and confidentiality. Irreversible eFuse burns. Closes the soft-trust gap above.
• NVS encryption without Secure Boot — limited value (any flasher can disable it without Secure Boot to root the chain), but trivial to enable. Probably not worth it alone.
• Provisioning over BLE or Wi-Fi AP — for at-scale fleet provisioning. Standard ESP-IDF wifi_provisioning component handles this. Adds firmware size; only worth it if we ever ship more than one device.
• Remote re-provisioning of non-secret keys (poll interval, log severity, etc.) via the OTA channel — push small "config artifacts" alongside firmware artifacts in OCI. Combined with the next bullet, enables seamless SA-key rotation.
• Rotating the SA key — currently means re-provisioning every device over USB. With remote re-provisioning above, becomes a push button.