From 7e32f6decb409bef59676c5b240f153bc5fabf74 Mon Sep 17 00:00:00 2001 From: Jason Hall Date: Sun, 3 May 2026 08:30:43 -0400 Subject: [PATCH] docs: reorganise into docs/, fold logs-plan + monitoring-plan into observability.md MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - New `docs/observability.md`: descriptive (present-tense) write-up of the Cloud Logging + Cloud Monitoring pipelines as currently shipped. Replaces the historical `logs-plan.md` + `monitoring-plan.md` planning docs. - New `docs/setup.md`: prerequisites, first flash, day-to-day Make targets, and the optional GCP setup (lifted out of the README so the README can stay terse). Includes the Python 3.12-shim explanation from the old `notes.txt`. - Move `ota.md` → `docs/ota.md`. - Move `eink-plan.md` → `docs/eink-plan.md`. Per-feature plans still use the `-plan.md` name; once shipped they get rewritten in present tense alongside the other docs. - Delete `logs-plan.md`, `monitoring-plan.md`, `notes.txt`. Their user-facing content is now in `docs/setup.md`; their LLM-relevant bits (architectural rationale, partition-table CMake quirk, Python shim, no-LED, `make` conventions, NVS key length cap) are in `CLAUDE.md`. - Trim `README.md` to a top-level overview + links into `docs/`. - Update internal cross-references (Makefile, ota.md, eink-plan.md, tools/provision/src/main.rs doc-comment). Co-Authored-By: Claude Opus 4.7 (1M context) --- CLAUDE.md | 104 +++++++-- Makefile | 4 +- README.md | 151 +++---------- eink-plan.md => docs/eink-plan.md | 7 +- docs/observability.md | 348 ++++++++++++++++++++++++++++++ ota.md => docs/ota.md | 2 +- docs/setup.md | 156 ++++++++++++++ logs-plan.md | 241 --------------------- monitoring-plan.md | 276 ------------------------ notes.txt | 119 ---------- tools/provision/src/main.rs | 3 +- 11 files changed, 621 insertions(+), 790 deletions(-) rename eink-plan.md => docs/eink-plan.md (94%) create mode 100644 docs/observability.md rename ota.md => docs/ota.md (99%) create mode 100644 docs/setup.md delete mode 100644 logs-plan.md delete mode 100644 monitoring-plan.md delete mode 100644 notes.txt diff --git a/CLAUDE.md b/CLAUDE.md index 972f869..99cb35d 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -3,30 +3,37 @@ ## Project layout ``` -src/main.rs firmware entrypoint, Wi-Fi + HTTPS + OTA loop -src/ota.rs OTA polling, manifest fetch, blob streaming -src/sig.rs cosign Sigstore Bundle verification -src/trust.rs NVS-loaded trust config (identities + Sigstore CAs) -trust/ Sigstore Fulcio root + intermediate certs (provisioned) -tools/publisher/ host-side tool that pushes signed OCI artifacts -tools/provision/ host-side tool that builds + flashes NVS partition -.github/workflows/ ci.yml (PRs) + publish.yml (push to main) -Cargo.toml firmware deps -partitions.csv OTA-capable partition table (1.94 MB app slots) -sdkconfig.defaults.in ESP-IDF kconfig (Makefile substitutes paths) -Makefile build / flash / monitor / provision / publish entrypoints -provisioning.toml.example template for per-device NVS values -ota.md full OTA + provisioning + signing system documentation -eink-plan.md planned e-ink display work -logs-plan.md planned GCP Cloud Logging integration -notes.txt internal design notes + setup gotchas (Python 3.12 - shim, partition-table CMake quirk, etc.) +src/main.rs firmware entrypoint, Wi-Fi + boot orchestration +src/ota.rs OTA poll loop, manifest fetch, blob streaming +src/sig.rs cosign Sigstore Bundle verification +src/trust.rs NVS-loaded trust config (signer identities + Sigstore CAs) +src/cloud_log.rs tracing → Cloud Logging (queue + sender thread) +src/metrics.rs chip-health snapshots → Cloud Monitoring time series +src/gcp_auth.rs multi-scope JWT mint, shared TokenProvider, ShortHttpsLock, + OtaDownloadGuard, HTTPS helpers +trust/ Sigstore Fulcio root + intermediate certs (provisioned via NVS) +tools/publisher/ host-side tool that pushes signed OCI artifacts to GHCR +tools/provision/ host-side tool that builds + flashes NVS partition +.github/workflows/ ci.yml (PRs) + publish.yml (push to main) +Cargo.toml firmware deps +partitions.csv OTA-capable partition table (1.94 MB app slots) +sdkconfig.defaults.in ESP-IDF kconfig template (Makefile substitutes paths) +Makefile build / flash / monitor / provision / publish entrypoints +provisioning.toml.example template for per-device NVS values +docs/setup.md first-flash + day-to-day workflow + GCP setup +docs/ota.md full OTA + provisioning + signing system documentation +docs/observability.md Cloud Logging + Monitoring design (current state) +docs/eink-plan.md planned e-ink display work +README.md short top-level overview, links into docs/ ``` ## Conventions established in this repo -- **Plans live in the repo** as `*-plan.md` (per-feature) and `ota.md` - (descriptive doc once a system is operational). Not in Claude memory. +- **`docs/` for descriptive docs and per-feature plans.** Forward- + looking work uses `-plan.md`; once shipped, the plan is + rewritten in present tense and lives next to the other reference + docs (e.g. `logs-plan.md` + `monitoring-plan.md` collapsed into + `docs/observability.md`). - **Cargo.lock IS tracked** for all three crates here — they're all binaries. - **Don't use "and" in Make target names** — chain existing targets @@ -35,3 +42,60 @@ notes.txt internal design notes + setup gotchas (Python 3.12 project dir, not `~`. - **GHCR auth** uses classic PATs with `write:packages` (fine-grained PATs don't expose a Packages permission for user-owned packages). +- **Use `make` targets**, not raw `espflash`/`cargo` commands. The + Makefile sets toolchain env, the Python 3.12 shim, and (critically) + always passes `--partition-table partitions.csv` to `espflash flash` + — without that, espflash 4.x silently writes a default + factory-only partition table over our OTA layout. + +## Architectural decisions worth knowing + +- **std Rust on top of ESP-IDF**, not no_std with `esp-hal`. We need + easy Wi-Fi + HTTP + TLS via `esp-idf-svc`, plus standard + threading/heap. The cost is ESP-IDF's C SDK + CMake/Ninja/Python + toolchain; the lighter no_std path is worth knowing for future + projects. +- **Pure-Rust `rsa` crate for JWT signing**, not mbedtls FFI. Adds + ~50 KB but keeps the code in safe Rust. +- **mbedtls heap is tight.** With cloud_log + metrics + OTA each + potentially holding a TLS session, three concurrent handshakes + (~25–35 KB each) blow our heap. Mitigations stacked in + `sdkconfig.defaults.in` (`MBEDTLS_DYNAMIC_BUFFER`, free + config/CA-cert, `KEEP_PEER_CERTIFICATE=n`) plus runtime serialization + via `gcp_auth::ShortHttpsLock` and an `OTA_DOWNLOAD_IN_PROGRESS` + gate. See `docs/observability.md` for the full chain. +- **OTA download is *not* serialized.** Multi-second blob downloads + must not block per-5 s log flushes. cloud_log + metrics check + `OTA_DOWNLOAD_IN_PROGRESS` and skip-but-don't-drain while a + download is in flight; the queue absorbs and flushes after. + +## Gotchas to remember + +- **Custom partition table path resolution.** ESP-IDF resolves + `CONFIG_PARTITION_TABLE_CUSTOM_FILENAME` relative to its top-level + CMake project, which for embuild is `target/.../esp-idf-sys-/out/`, + not the repo root. Workaround: `sdkconfig.defaults.in` is the + template (committed), with the placeholder + `@PROJECT_DIR@/partitions.csv`; the Makefile substitutes the + absolute path at build time and writes the resolved + `sdkconfig.defaults` (gitignored). Don't "fix" the placeholder. +- **Python 3.12 shim.** embuild grabs whatever `python3` is first on + PATH. macOS's default is 3.9.6, which silently breaks ESP-IDF + dependency resolution. The Makefile creates + `.embuild/python-shim/python3 → uv-managed 3.12` and prepends it to + PATH. If a dep-check error appears after a fresh clone: + `rm -rf .embuild && make build`. +- **No user LED on this board.** Inland ESP-WROOM-32 has only a + power LED; the DOIT/DEVKITC blue LED on GPIO 2 is *not* populated. + Don't suggest GPIO blink demos — use serial logging via `make + monitor` instead. +- **NVS key names cap at 15 characters.** When adding new keys to + `tools/provision/`, abbreviate (`metric_intvl`, not + `metrics_interval`). The `.toml` field can be long and readable; + the NVS key is the constrained one. +- **`make flash` after a fresh `make bootstrap`** can boot the wrong + partition: bootstrap clears otadata so the bootloader picks ota_0, + but a subsequent OTA download installs to ota_1 and otadata starts + pointing there. A bare `make flash` writes to ota_0 (the first app + partition in the table) but the device will keep booting ota_1 + until you `make bootstrap` again or wait for OTA promotion. diff --git a/Makefile b/Makefile index 3268236..0d407a2 100644 --- a/Makefile +++ b/Makefile @@ -1,5 +1,5 @@ # ESP32 Rust — runnable documentation for build / flash / monitor. -# See notes.txt for one-time setup (espup, brew deps, Python 3.12 shim). +# See docs/setup.md for one-time setup (espup, brew deps, Python 3.12 shim). # # Quick start (new device): # make provisioning.toml # one-time: copy template @@ -94,7 +94,7 @@ sdkconfig.defaults: sdkconfig.defaults.in partitions.csv # back to the shim itself (self-reference) producing ELOOP. Recreating # from scratch each run avoids both. ensure-python-shim: - @command -v uv >/dev/null || { echo "ERROR: uv not installed. See notes.txt step 4."; exit 1; } + @command -v uv >/dev/null || { echo "ERROR: uv not installed. See docs/setup.md."; exit 1; } @mkdir -p $(PYTHON_SHIM) @rm -f $(PYTHON_SHIM)/python $(PYTHON_SHIM)/python3 @uv python find 3.12 >/dev/null 2>&1 || uv python install 3.12 diff --git a/README.md b/README.md index c715fcb..2afd244 100644 --- a/README.md +++ b/README.md @@ -4,152 +4,49 @@ [![Publish OTA](https://github.com/imjasonh/esp32/actions/workflows/publish.yml/badge.svg)](https://github.com/imjasonh/esp32/actions/workflows/publish.yml) Std Rust on an Inland ESP-WROOM-32 dev board with end-to-end OTA over -GHCR + cosign keyless signing. E-ink display work coming next (see -[`eink-plan.md`](eink-plan.md)). +GHCR + cosign keyless signing. Optional Cloud Logging + Cloud +Monitoring shipped from the device. E-ink display work coming next. ## What it does today -- Connects to Wi-Fi, fetches public IP and weather over HTTPS. -- Polls `ghcr.io/imjasonh/esp32:latest` every ~60s for new firmware. -- For each new digest, fetches the cosign Sigstore Bundle, verifies the - signature, the cert chain to the Sigstore root, and that the signer's +- Polls `ghcr.io/imjasonh/esp32:latest` every ~60 s for new firmware. +- For each new digest: fetches the cosign Sigstore Bundle, verifies + the signature + cert chain to the Sigstore root + that the signer's identity matches the allowlist provisioned into the device's NVS. - On verify-pass, streams the layer to the inactive OTA partition, - reboots, and only marks the new image valid after Wi-Fi + registry + reboots, marks the new image valid only after Wi-Fi + registry bringup checks pass. Any post-OTA failure auto-rolls back via the bootloader. +- Optionally ships structured `tracing` events to **Cloud Logging** + and chip-health metrics (heap, stack, wifi, cpu, …) to **Cloud + Monitoring**. Opt-in per device via the `[gcp]` block in + `provisioning.toml`. -Push to `main` → CI builds → publish workflow pushes a signed image to -GHCR → device picks it up on its next poll. See -[`ota.md`](ota.md) for the full design. +Push to `main` → CI builds → publish workflow pushes a signed image +to GHCR → device picks it up on its next poll. -## Hardware +## Docs -- **Board**: Inland ESP-WROOM-32 (Micro Center SKU 027466). Connects as - `/dev/cu.usbserial-0001` on macOS via the onboard CP210x USB-UART. -- **No user LED** on this variant — only a power LED. Watch logs via - `make monitor`. +- [`docs/setup.md`](docs/setup.md) — prerequisites, first flash, + day-to-day commands, GCP setup +- [`docs/ota.md`](docs/ota.md) — OTA + provisioning + signing design +- [`docs/observability.md`](docs/observability.md) — Cloud Logging + + Cloud Monitoring design (current state) +- [`docs/eink-plan.md`](docs/eink-plan.md) — planned e-ink display work -## Prerequisites (one-time, macOS) +## Quick start ```bash cargo install espup espflash ldproxy -brew install cmake ninja dfu-util cosign +brew install cmake ninja dfu-util cosign jq espup install --targets esp32 -curl -LsSf https://astral.sh/uv/install.sh | sh # if you don't have uv -``` +curl -LsSf https://astral.sh/uv/install.sh | sh -## First flash (USB) - -```bash make provisioning.toml # creates from template $EDITOR provisioning.toml # fill in wifi creds + trust identities make bootstrap # build, flash everything, write NVS make monitor # watch it boot and connect ``` -The first build clones ESP-IDF v5.2.2 into `.embuild/` (5–10 min). -Subsequent builds are fast. - -The OTA-distributed firmware **contains no secrets** — Wi-Fi creds and -trust roots live in NVS, written via USB by `make provision`. See -[`ota.md`](ota.md) for the full design. - -## Day-to-day - -``` -make build Compile firmware -make flash Build + flash app (use flash-all after partitions change) -make flash-all Erase + write bootloader, partition table, app -make provision Write NVS partition from provisioning.toml over USB -make bootstrap flash-all + provision (new device setup) -make monitor Open serial monitor; Ctrl+C to exit -make run Build + flash + monitor -make publish Build, push OCI artifact to ghcr.io/imjasonh/esp32, cosign sign -make clean cargo clean -``` - -`make publish` requires `gh.env` (see [`ota.md`](ota.md) for -PAT setup) and a real cosign OIDC flow the first time per ~10min window -— a browser pops to authenticate. CI does this automatically via the -GitHub Actions workflow's ambient OIDC token. - -## Optional: GCP Cloud Logging + Monitoring - -The firmware can ship structured `tracing` events (both app and OTA) to -Google Cloud Logging *and* periodic chip-health metrics (heap, stack, -wifi, cpu, uptime, …) to Cloud Monitoring. Both are **opt-in per -device** — without the `[gcp]` block in `provisioning.toml` the device -boots normally and just emits to serial. Full design in -[`logs-plan.md`](logs-plan.md) and [`monitoring-plan.md`](monitoring-plan.md). - -One service account and one key cover both APIs (the JWT requests -both scopes; one cached access token is shared by the cloud_log and -metrics threads). - -One-time GCP setup, using `gcloud`: - -```bash -PROJECT_ID= -SA_NAME= -SA_EMAIL=$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com - -# Create the service account. -gcloud iam service-accounts create $SA_NAME \ - --display-name="ESP32 device logger" \ - --project=$PROJECT_ID - -# Grant only logging.logWriter + monitoring.metricWriter — least -# privilege. The device can write log entries and metric points; -# it cannot read, delete, or do anything else. -gcloud projects add-iam-policy-binding $PROJECT_ID \ - --member="serviceAccount:$SA_EMAIL" \ - --role="roles/logging.logWriter" -gcloud projects add-iam-policy-binding $PROJECT_ID \ - --member="serviceAccount:$SA_EMAIL" \ - --role="roles/monitoring.metricWriter" - -# Create + download a JSON key. Keep this file safe — anyone with it -# can write logs + metrics as this SA. -gcloud iam service-accounts keys create gcp-sa-key.json \ - --iam-account=$SA_EMAIL \ - --project=$PROJECT_ID - -# Extract the RSA private key PEM and the key id into the forms -# `tools/provision/` wants. -jq -j .private_key gcp-sa-key.json > gcp-sa-key.pem # -j: no trailing newline -KEY_ID=$(jq -r .private_key_id gcp-sa-key.json) -echo "sa_key_id = $KEY_ID" -``` - -Then add a `[gcp]` block to `provisioning.toml` (template in -`provisioning.toml.example`) using `$PROJECT_ID`, `$SA_EMAIL`, the -printed `KEY_ID`, and the path `gcp-sa-key.pem`. Re-run `make provision` -and reboot the device. - -Logs land in Cloud Logging under -`projects/kontaindotme/logs/esp32-firmware`. The device's MAC is in -`resource.labels.node_id` for fleet filtering; the originating tracing -target is in `jsonPayload.module` so you can split app vs OTA in the -Cloud Logging UI (e.g. -`jsonPayload.module = "esp32_blinky::ota"` or -`= "esp32_blinky::sig"`). - -Metrics land under `custom.googleapis.com/esp32/` (e.g. -`free_heap`, `wifi_rssi`, `stack_hwm` with a `task` label). Snapshot -cadence is `metrics_interval_secs` in the `[gcp]` block (default 300s, -0 disables — cloud_log keeps running). Inspect with: - -```bash -gcloud monitoring time-series list \ - --filter='metric.type=starts_with("custom.googleapis.com/esp32/")' \ - --interval-end-time=$(date -u +%Y-%m-%dT%H:%M:%SZ) \ - --interval-start-time=$(date -u -v-30M +%Y-%m-%dT%H:%M:%SZ) \ - --project=$PROJECT_ID -``` - -**Threat model**: the SA private key sits in NVS unencrypted. Anyone -with physical access to the chip can extract it. Mitigation is -strict scoping (only `roles/logging.logWriter`, only on a -logs-tolerant project). Real hardening = Flash Encryption + Secure -Boot v2 (deferred; see [`ota.md`](ota.md) Future work). +See [`docs/setup.md`](docs/setup.md) for full details, including +optional GCP Logging + Monitoring setup. diff --git a/eink-plan.md b/docs/eink-plan.md similarity index 94% rename from eink-plan.md rename to docs/eink-plan.md index 5aacdb3..493bb8c 100644 --- a/eink-plan.md +++ b/docs/eink-plan.md @@ -1,9 +1,10 @@ # E-ink display — plan The Inland ESP-WROOM-32 dev board doesn't have a software-controllable LED -(see `notes.txt`), so the visible-output story for this project is an e-ink -display. This file captures the hardware choice, the Rust graphics stack -we'll use, and a shortlist of projects to build on top. +(see [`setup.md`](setup.md)), so the visible-output story for this +project is an e-ink display. This file captures the hardware choice, +the Rust graphics stack we'll use, and a shortlist of projects to +build on top. ## Hardware diff --git a/docs/observability.md b/docs/observability.md new file mode 100644 index 0000000..146fa4f --- /dev/null +++ b/docs/observability.md @@ -0,0 +1,348 @@ +# Observability — current state + +The firmware ships structured `tracing` events to **Google Cloud +Logging** and periodic chip-health snapshots to **Google Cloud +Monitoring**. Both are opt-in per device: without a `[gcp]` block in +`provisioning.toml`, the firmware boots normally and emits to serial +only. + +One service account, one private key, one cached access token covers +both APIs (the JWT requests `logging.write` + `monitoring.write` +scopes; the cached bearer is shared between the two sender threads). + +## Architecture + +``` +┌──────────────────── ESP32 ────────────────────┐ ┌──── GCP ─────┐ +│ │ │ │ +│ tracing event ──┐ │ │ │ +│ (any module) ├─▶ ring buffer ──▶ cloud_log │ HTTPS │ Logging │ +│ tracing event ──┘ (256 entries) thread │────────▶│ /v2/ │ +│ drop-oldest │ │ entries: │ +│ │ │ write │ +│ │ │ │ +│ metrics thread ─ snapshot heap/stack/wifi/... │ HTTPS │ Monitoring │ +│ (every N s) build CreateTimeSeries │────────▶│ /v3/.../ │ +│ │ │ timeSeries │ +│ │ │ │ +│ gcp_auth: shared TokenProvider │ HTTPS │ oauth2 │ +│ - mints multi-scope JWT │────────▶│ /token │ +│ - caches bearer for ~1h │ │ │ +│ │ │ │ +└────────────────────────────────────────────────┘ └──────────────┘ +``` + +## Modules + +| File | What it owns | +|------|--------------| +| `src/gcp_auth.rs` | `TokenProvider` (multi-scope JWT mint + bearer cache), `ShortHttpsLock`, `OtaDownloadGuard`, shared HTTPS / time / base64url helpers, `device_mac()`. | +| `src/cloud_log.rs` | `GcpConfig` NVS load, `LogEntry`, `LogQueue` (bounded ring buffer), `CloudLogLayer` (tracing subscriber), sender thread that POSTs to `logging.googleapis.com`. | +| `src/metrics.rs` | Per-task `TaskHandle_t` registry, snapshot collector (FFI calls into ESP-IDF), sender thread that POSTs to `monitoring.googleapis.com`. | + +## Auth + +The `TokenProvider` mints a single OAuth2 access token covering both +APIs: + +``` +JWT claims.scope = "https://www.googleapis.com/auth/logging.write + https://www.googleapis.com/auth/monitoring.write" +``` + +Standard service-account flow: build header + claims, base64url-encode, +sign with the SA's RSA private key (RS256), POST to +`oauth2.googleapis.com/token`, cache the returned `access_token` for +its `expires_in` minus a 5-minute safety margin. + +The cache lives in a `Mutex>` inside the provider; +`get_or_refresh()` returns the bearer directly on hit, mints under the +lock on miss. cloud_log + metrics share an `Arc` so +neither pays for its own RSA sign. + +## Cloud Logging + +### Capture + +`CloudLogLayer` is a `tracing_subscriber::Layer`. Every emitted event: + +- Severity mapped: `TRACE`/`DEBUG → DEBUG`, `INFO → INFO`, `WARN → + WARNING`, `ERROR → ERROR`. +- Wall-clock timestamp from `SystemTime::now()` if NTP has synced + (anything before 2020-01-01 is treated as not-yet-synced → entry + goes without a timestamp; Cloud Logging assigns server-side time). +- All structured fields captured into `serde_json::Map`. +- Pushed onto a 256-entry bounded queue. When full, oldest is dropped + and a counter is bumped; the next entry pushed carries + `dropped_before` so readers can spot lossy windows in the cloud. +- The layer's own module path (`esp32_blinky::cloud_log`), + `esp32_blinky::metrics`, and `esp32_blinky::gcp_auth` are + **excluded** from capture — otherwise the senders' own tracing + calls would land on the queue they're draining and create a tight + feedback loop. + +The layer composes with `EspLogger`, so events still print to serial +in addition to being queued. + +### Sender thread + +Background pthread, ~32 KB stack. Wakes every 5 s (or backoff if the +last POST failed): + +1. Skip the cycle if `OTA_DOWNLOAD_IN_PROGRESS` is set (see "Heap + budget" below). +2. Drain up to 50 entries from the queue. +3. Acquire `ShortHttpsLock` (serialises against metrics + OTA short + fetches). +4. `auth.get_or_refresh()` — may transparently mint a token. +5. POST to `https://logging.googleapis.com/v2/entries:write`. +6. Drop the lock. + +On 4xx/5xx the batch is dropped (rather than re-queued) — long outages +otherwise grow the queue unboundedly. Drop count is surfaced on the +next entry pushed. + +### Schema + +```json +{ + "logName": "projects//logs/esp32-firmware", + "resource": { + "type": "generic_node", + "labels": { + "project_id": "", + "location": "global", + "namespace": "esp32", + "node_id": "" + } + }, + "entries": [ + { + "severity": "INFO", + "timestamp": "2026-05-02T19:45:00Z", + "jsonPayload": { + "message": "ota: boot summary", + "module": "esp32_blinky::ota", + "fw_version": "abc1234", + "fw": "abc1234", + "repo": "ghcr.io/imjasonh/esp32", + "tag": "latest", + "poll_secs": 60, + "last_digest":"sha256:..." + } + } + ] +} +``` + +Single `logName` for both app + OTA; `jsonPayload.module` lets +queries split by component: + +``` +jsonPayload.module="esp32_blinky::ota" AND jsonPayload.fw_version="abc1234" +``` + +## Cloud Monitoring + +### Snapshot + +Every `metrics_interval_secs` (default 300, NVS-tunable, 0 disables +the thread entirely), `metrics::collect()` reads: + +| Metric type (`custom.googleapis.com/esp32/…`) | Source | Unit | Kind | +|-----------------------------------------------|---------------------------------------------------------|------|-------| +| `free_heap` | `esp_get_free_heap_size()` | By | GAUGE | +| `free_heap_internal` | `heap_caps_get_free_size(MALLOC_CAP_INTERNAL)` | By | GAUGE | +| `min_free_heap` | `esp_get_minimum_free_heap_size()` (water-mark / boot) | By | GAUGE | +| `largest_free_block` | `heap_caps_get_largest_free_block(MALLOC_CAP_DEFAULT)` | By | GAUGE | +| `stack_hwm` *(label `task`)* | `uxTaskGetStackHighWaterMark` per published task | By | GAUGE | +| `wifi_rssi` | `esp_wifi_sta_get_ap_info().rssi` | dBm | GAUGE | +| `wifi_channel` | …`.primary` | 1 | GAUGE | +| `cpu_freq_mhz` | `CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ` | MHz | GAUGE | +| `uptime_secs` | `esp_timer_get_time() / 1_000_000` | s | GAUGE | +| `nvs_used_entries` | `nvs_get_stats().used_entries` | 1 | GAUGE | +| `nvs_free_entries` | …`.free_entries` | 1 | GAUGE | +| `cloud_log_queue_depth` | `LogQueue::stats().0` | 1 | GAUGE | +| `cloud_log_dropped_total` | lifetime drop counter on `LogQueue` | 1 | GAUGE | + +`stack_hwm` carries a `task` label (`main` / `ota` / `cloud_log` / +`metrics`) so all four series chart on one graph. Each thread +publishes its own `TaskHandle_t` via +`metrics::publish_self(&handles::)` at the top of its run loop; +`metrics::collect()` reads the FFI handle out of an `AtomicUsize` (0 = +unpublished, field omitted from the snapshot). + +Note: ESP-IDF Xtensa `StackType_t = uint8_t`, so +`uxTaskGetStackHighWaterMark` returns **bytes** (not "words" as +upstream FreeRTOS docs say). No multiplication. + +### Sender thread + +Background pthread, ~16 KB stack. Wakes every `metrics_interval_secs`: + +1. Skip the cycle if `OTA_DOWNLOAD_IN_PROGRESS`. +2. `collect()` — pure FFI, no allocations beyond the snapshot struct. +3. Acquire `ShortHttpsLock`. +4. `auth.get_or_refresh()`. +5. POST one `CreateTimeSeries` request containing all ~16 series in a + single body to `monitoring.googleapis.com/v3/projects//timeSeries`. + +Cloud Monitoring auto-creates `MetricDescriptor`s on first write; +no separate provisioning step on the GCP side. + +### Schema + +```json +{ + "timeSeries": [ + { + "metric": { + "type": "custom.googleapis.com/esp32/free_heap", + "labels": { "fw_version": "abc1234" } + }, + "resource": { + "type": "generic_node", + "labels": { + "project_id": "", + "location": "global", + "namespace": "esp32", + "node_id": "" + } + }, + "metricKind": "GAUGE", + "valueType": "INT64", + "points": [ + { "interval": { "endTime": "2026-05-02T22:30:00Z" }, + "value": { "int64Value": "182456" } } + ] + } + ] +} +``` + +`fw_version` rides on every metric as a **metric label** (resource +labels can't carry it — `generic_node` has a fixed schema). Adding +new label keys to an existing `MetricDescriptor` is forbidden by Cloud +Monitoring; if the fleet's running fw doesn't match the descriptor +schema you'll see HTTP 500 on the POST. Migration in that case is +manual: + +```bash +TOKEN=$(gcloud auth print-access-token) +URL='https://monitoring.googleapis.com/v3/projects//metricDescriptors' +for m in $(curl -sG "$URL" -H "Authorization: Bearer $TOKEN" \ + --data-urlencode 'filter=metric.type=starts_with("custom.googleapis.com/esp32/")' \ + | jq -r '.metricDescriptors[].type'); do + curl -s -X DELETE "$URL/$m" -H "Authorization: Bearer $TOKEN" +done +``` + +This deletes all historical points for those metrics. There's no +in-place schema migration in Cloud Monitoring. + +## Heap budget + serialization + +Each TLS handshake on this chip allocates ~25–35 KB of mbedtls +context. With cloud_log + metrics + OTA each potentially holding a +TLS session, the worst case is three concurrent handshakes pinning +~90 KB — past the heap budget, producing +`MBEDTLS_ERR_SSL_ALLOC_FAILED` (-0x7F00). + +Three knobs work together to keep this in budget: + +1. **mbedtls config** (`sdkconfig.defaults.in`): + - `CONFIG_MBEDTLS_DYNAMIC_BUFFER` — I/O buffers grow as needed, + freed between handshakes (vs. pinned 16+16 KB). + - `CONFIG_MBEDTLS_DYNAMIC_FREE_CONFIG_DATA` — frees handshake + config struct after handshake. + - `CONFIG_MBEDTLS_DYNAMIC_FREE_CA_CERT` — frees the CA bundle + after handshake; re-attached on the next handshake. + - `CONFIG_MBEDTLS_SSL_KEEP_PEER_CERTIFICATE=n` — drops the parsed + peer cert after handshake (~3–5 KB/session). We don't reuse + sessions or do mutual TLS. + +2. **Short HTTPS serialised** (`gcp_auth::ShortHttpsLock`): cloud_log + POST + metrics POST + OTA's manifest fetch + sig-bundle fetch all + take this `Mutex<()>` at their call sites. Held *at the call site* + (not inside `http_post`) so token mints inside `get_or_refresh()` + are covered by the caller's lock — std `Mutex` isn't reentrant. + OTA's poll_once releases between phases (token+manifest, then + verify which is CPU-only, then sig bundle) so cloud_log/metrics + aren't blocked behind ~2 s of pure-Rust X.509/ECDSA work. + +3. **OTA download pause** (`gcp_auth::OTA_DOWNLOAD_IN_PROGRESS`): + `ota::download_and_apply` flips this `AtomicBool` true via an RAII + guard for the duration of the multi-second blob download (which + intentionally **does not** take `ShortHttpsLock` — it'd otherwise + block per-5-s log flushes for tens of seconds). cloud_log + metrics + check the flag at the top of each cycle and skip — but don't + drain — when set. Entries accumulate in the bounded queue and + flush in one batched POST after the download finishes. + +Even with all three, peak heap during a 2-way concurrent TLS +(OTA + cloud_log handshake on top of OTA's held-open download) was +historically tight — the gate in #3 is what made downloads reliably +complete. + +## Provisioning (NVS) + +Optional `[gcp]` block in `provisioning.toml` (see +`provisioning.toml.example`): + +```toml +[gcp] +project_id = "..." +sa_email = "esp32-logger@my-project.iam.gserviceaccount.com" +sa_key_id = "abc123..." # `private_key_id` from the SA JSON key +sa_key_pem = "gcp-sa-key.pem" # path; tool reads + embeds the bytes +min_severity = "info" # debug / info / warn / error; default info +metrics_interval_secs = 300 # 0 disables metrics; cloud_log keeps running +``` + +NVS keys (15-char limit): + +| ns | key | type | notes | +|-----|----------------|------|----------------------------------------------------| +| gcp | project_id | str | GCP project (logs + metrics land in) | +| gcp | sa_email | str | service-account email | +| gcp | sa_key_id | str | private-key id (JWT `kid` header) | +| gcp | sa_key_pem | blob | RSA private key, PKCS#8 PEM (~1.7 KB) | +| gcp | min_severity | u8 | 0=TRACE..4=ERROR; default 2=INFO | +| gcp | metric_intvl | u32 | metrics snapshot interval in seconds; default 300 | + +Missing the namespace entirely or any of the four required string/blob +fields disables both pipelines. Missing optional fields fall back to +defaults. + +The OTA-distributed firmware contains no GCP secrets — the SA key +lives in NVS and is written via USB by `make provision`. + +## Concerns / known limitations + +- **NVS unencrypted**: anyone with physical access can dump the SA + key. Mitigation is strict scoping (only `logging.logWriter` + + `monitoring.metricWriter`, on a logs-tolerant project). Real fix is + Flash Encryption + Secure Boot v2; deferred (see [`ota.md`](ota.md) + Future work). +- **Wi-Fi outages**: the cloud_log queue is bounded at 256 entries. + Long offline periods are lossy; drops are surfaced via + `dropped_before` on subsequent entries. +- **Metric descriptor sticky labels**: see "Schema" above. Adding a + new metric label key requires a one-time descriptor delete. +- **CPU freq metric is build-time**: `esp_clk_cpu_freq` isn't in the + esp-idf-svc bindings, so we report the configured default. Becomes + inaccurate if/when `CONFIG_PM_ENABLE` is turned on. + +## Future work + +- **Per-task CPU runtime stats** via `vTaskGetRunTimeStats` — needs + `CONFIG_FREERTOS_USE_TRACE_FACILITY=y` + `CONFIG_FREERTOS_GENERATE_RUN_TIME_STATS=y`. +- **Light-sleep stats** once `esp_pm_*` is adopted — pairs with + modem-sleep work for power telemetry. +- **External power telemetry** (INA219/INA226 over I2C) — would warrant its own plan. +- **Pre-declared MetricDescriptors** with units + descriptions for nicer Cloud Monitoring UI. +- **Alerting policies** as Terraform / `gcloud alpha monitoring` resources, checked in. +- **Cumulative metrics with `startTime`** for proper rate aggregation + (drops/sec, polls/sec) instead of GAUGE snapshots of running totals. +- **OIDC instead of SA key** — would remove the key-leak concern, but + ESP32 has no source of a usable OIDC token today. diff --git a/ota.md b/docs/ota.md similarity index 99% rename from ota.md rename to docs/ota.md index 0a4e050..1fda4cd 100644 --- a/ota.md +++ b/docs/ota.md @@ -101,7 +101,7 @@ gcp min_severity u8 optional; 0=TRACE..4=ERROR (default 2=INFO) The `gcp` namespace is opt-in. If any required key is missing, the device boots with serial-only logging and never talks to GCP. See -[`logs-plan.md`](logs-plan.md). +[`observability.md`](observability.md). The `wifi` and `trust` namespaces are written by `make provision` and never touched by OTA. The `ota` namespace is written at runtime by the diff --git a/docs/setup.md b/docs/setup.md new file mode 100644 index 0000000..37e1ba1 --- /dev/null +++ b/docs/setup.md @@ -0,0 +1,156 @@ +# Setup + +End-to-end first-time setup for an Inland ESP-WROOM-32 dev board on +macOS. + +## Hardware + +- **Board**: Inland ESP-WROOM-32 (Micro Center SKU 027466, rebadged + Keyestudio KS0413, "mini" form factor). Connects as + `/dev/cu.usbserial-0001` over the onboard CP210x USB-UART. +- **No user LED** on this variant — only a power LED ("D1", hardwired + to 3V3, always on). The DOIT/DEVKITC-style blue LED on GPIO 2 is + *not* populated. Watch boot via `make monitor`. + +## One-time host install + +```bash +cargo install espup espflash ldproxy +brew install cmake ninja dfu-util cosign jq +espup install --targets esp32 +curl -LsSf https://astral.sh/uv/install.sh | sh # if you don't have uv +``` + +`uv` provides Python 3.12 for ESP-IDF's tooling — see [why](#python-shim) below. + +## Provisioning + +```bash +make provisioning.toml # creates from template +$EDITOR provisioning.toml # fill in wifi creds + trust identities +make bootstrap # build, flash everything, write NVS +make monitor # watch it boot and connect +``` + +The first build clones ESP-IDF v5.2.2 into `.embuild/` (5–10 min). +Subsequent builds are fast. + +The OTA-distributed firmware contains **no secrets** — Wi-Fi creds, +Sigstore trust roots, and (optionally) GCP service-account keys all +live in NVS, written via USB by `make provision`. See +[`ota.md`](ota.md) for the full design. + +## Day-to-day Make targets + +``` +make build Compile firmware +make flash Build + flash app (use flash-all after partitions change) +make flash-all Erase + write bootloader, partition table, app +make provision Write NVS partition from provisioning.toml over USB +make bootstrap flash-all + provision (new device setup) +make monitor Open serial monitor; Ctrl+C to exit +make run Build + flash + monitor +make publish Build, push OCI artifact to ghcr.io/imjasonh/esp32, cosign sign +make clean cargo clean +``` + +`make publish` requires `gh.env` (see [`ota.md`](ota.md) for PAT setup) +and a real cosign OIDC flow the first time per ~10 min window — a +browser pops to authenticate. CI does this automatically via the +GitHub Actions workflow's ambient OIDC token. + +## Optional: GCP Cloud Logging + Monitoring + +The firmware can ship structured `tracing` events to **Cloud +Logging** and chip-health metrics (heap, stack, wifi, cpu, …) to +**Cloud Monitoring**. Both are opt-in per device — without a `[gcp]` +block in `provisioning.toml`, the device boots normally and emits to +serial only. Full design in [`observability.md`](observability.md). + +One service account and one key cover both APIs. One-time GCP setup: + +```bash +PROJECT_ID= +SA_NAME= +SA_EMAIL=$SA_NAME@$PROJECT_ID.iam.gserviceaccount.com + +# Create the service account. +gcloud iam service-accounts create $SA_NAME \ + --display-name="ESP32 device logger" \ + --project=$PROJECT_ID + +# Grant only logging.logWriter + monitoring.metricWriter — least +# privilege. The device can write log entries and metric points; +# nothing else. +gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:$SA_EMAIL" \ + --role="roles/logging.logWriter" +gcloud projects add-iam-policy-binding $PROJECT_ID \ + --member="serviceAccount:$SA_EMAIL" \ + --role="roles/monitoring.metricWriter" + +# Create + download a JSON key. Keep this file safe — anyone with it +# can write logs + metrics as this SA. +gcloud iam service-accounts keys create gcp-sa-key.json \ + --iam-account=$SA_EMAIL \ + --project=$PROJECT_ID + +# Extract the RSA private key PEM and the key id into the forms +# `tools/provision/` wants. Use `jq -j` (no trailing newline) — the +# device-side PEM parser is strict. +jq -j .private_key gcp-sa-key.json > gcp-sa-key.pem +KEY_ID=$(jq -r .private_key_id gcp-sa-key.json) +echo "sa_key_id = $KEY_ID" +``` + +Then add a `[gcp]` block to `provisioning.toml` (template in +`provisioning.toml.example`) using `$PROJECT_ID`, `$SA_EMAIL`, the +printed `KEY_ID`, and the path `gcp-sa-key.pem`. Re-run `make +provision` and reboot the device. + +Inspect what's flowing in: + +```bash +# Logs (Cloud Logging) +gcloud logging read \ + 'logName="projects/'"$PROJECT_ID"'/logs/esp32-firmware"' \ + --limit=20 --project=$PROJECT_ID + +# Metrics (Cloud Monitoring) — one metric type at a time +TOKEN=$(gcloud auth print-access-token) +START=$(date -u -v-30M +%Y-%m-%dT%H:%M:%SZ) +END=$(date -u +%Y-%m-%dT%H:%M:%SZ) +curl -sG "https://monitoring.googleapis.com/v3/projects/$PROJECT_ID/timeSeries" \ + -H "Authorization: Bearer $TOKEN" \ + --data-urlencode 'filter=metric.type="custom.googleapis.com/esp32/free_heap"' \ + --data-urlencode "interval.startTime=$START" \ + --data-urlencode "interval.endTime=$END" | jq +``` + +**Threat model**: the SA private key sits in NVS unencrypted. Anyone +with physical access to the chip can extract it. Mitigation is strict +SA scoping (see roles above) and a logs/metrics-only project. Real +hardening = Flash Encryption + Secure Boot v2 (deferred; see +[`ota.md`](ota.md) Future work). + +## Python 3.12 shim + +embuild bootstraps the ESP-IDF venv using whatever `python3` is first +on `PATH`. On macOS that's Apple's `/usr/bin/python3` (3.9.6). ESP-IDF +v5.2 nominally supports 3.8–3.12, but pip dependency resolution on +3.9 silently drops some transitive deps (notably `ruamel.yaml` and its +dependents) and IDF's check then fails with cryptic "Failed to run +Python dependency check ... Error: 255". + +Fix: the Makefile creates `.embuild/python-shim/python3` as a symlink +to a uv-managed Python 3.12 and prepends that directory to `PATH` for +every recipe. The shim isn't checked in — it's regenerated by the +`ensure-python-shim` Make target, a prerequisite of `build`. + +If you hit the dependency-check error after upgrading or after a fresh +clone: + +```bash +rm -rf .embuild +make build +``` diff --git a/logs-plan.md b/logs-plan.md deleted file mode 100644 index 189dc08..0000000 --- a/logs-plan.md +++ /dev/null @@ -1,241 +0,0 @@ -# Cloud Logging — plan (issue #2) - -Ship structured logs from the ESP32 to Google Cloud Logging so we can -monitor + diagnose remotely without a serial cable. No prebuilt -GCP/auth libraries on-device — talk directly to the REST APIs. - -GCP credentials follow the same model the rest of the project now uses: -**NVS-provisioned via USB**, not embedded into the OTA-distributed -firmware. The published OCI image stays device-agnostic. - -## Architecture - -``` -┌─────────── ESP32 ───────────┐ ┌─────────── GCP ───────────┐ -│ │ │ │ -│ tracing event ──┐ │ │ │ -│ tracing event ──┼─▶ buffer │ POST logs: │ logging.googleapis.com │ -│ tracing event ──┘ (ring) │ ──────────────▶│ /v2/entries:write │ -│ │ │ Bearer │ │ -│ ▼ │ │ │ -│ background │ POST oauth2: │ oauth2.googleapis.com │ -│ sender task │ ──────signed─▶│ /token (grant_type= │ -│ │ │ JWT (RS256) │ jwt-bearer) │ -│ ▼ │ ◀──────────────│ │ -│ HTTPS POST │ access_token │ │ -│ │ │ │ -└──────────────────────────────┘ └────────────────────────────┘ -``` - -A new tracing **layer** captures events into a bounded ring buffer in -RAM — both `src/main.rs` (wifi connect, HTTPS GETs, app behavior) and -`src/ota.rs` / `src/sig.rs` (OTA polling, verification, apply). All -events flow through tracing already, so a single layer gets both -"app" and "OTA" logs for free. - -A background pthread drains the buffer, batches entries, and POSTs -them to Cloud Logging using a cached OAuth2 access token. Tokens are -obtained via the standard service-account JWT bearer flow and cached -for their ~1-hour lifetime so we only do RSA signing once per refresh. - -## Auth flow (no library) - -Service-account JWT → access-token → API call. Standard Google flow. - -1. **JWT.** Header `{"alg":"RS256","typ":"JWT","kid":""}`, - claims `{"iss":"","scope":"https://www.googleapis.com/auth/logging.write","aud":"https://oauth2.googleapis.com/token","iat":,"exp":}`. - Base64url-encode each, concatenate with `.`, sign with the SA's - RSA private key (RS256 = RSA-PKCS#1 v1.5 over SHA-256), append - `.`. -2. **Token exchange.** POST to `https://oauth2.googleapis.com/token` - with form body - `grant_type=urn:ietf:params:oauth:grant-type:jwt-bearer&assertion=`. - Response: `{"access_token":"...","expires_in":3599,"token_type":"Bearer"}`. -3. **Cache.** Hold the access token in RAM with its expiry. Re-mint - only when expired (or close to). RSA signing is the expensive step - we want to avoid doing per-log. - -## Logging API call - -`POST https://logging.googleapis.com/v2/entries:write` with -`Authorization: Bearer ` and JSON body: - -```json -{ - "logName": "projects//logs/esp32-firmware", - "resource": { - "type": "generic_node", - "labels": { - "project_id": "", - "location": "global", - "namespace": "esp32", - "node_id": "" - } - }, - "entries": [ - { - "severity": "INFO", - "timestamp": "2026-05-02T19:45:00Z", - "jsonPayload": { - "message": "ota: boot summary", - "module": "esp32_blinky::ota", - "fw": "979c862", - "repo": "ghcr.io/imjasonh/esp32", - "tag": "latest", - "poll_secs": 60, - "last_digest": "sha256:..." - } - }, - ... - ] -} -``` - -Single `logName` for both app + OTA logs; the tracing target's -`module` (e.g. `esp32_blinky::ota`, `esp32_blinky::sig`, -`esp32_blinky` for main) is folded into `jsonPayload` so Cloud -Logging filters can split by component (`jsonPayload.module = -"esp32_blinky::ota"`). - -Resource labels carry device identity (MAC). `jsonPayload` carries -every tracing structured field as a top-level JSON key. - -## Provisioning (NVS) - -Add to the `gcp` NVS namespace, written by `tools/provision/`: - -``` -ns key type notes -───────────────────────────────────────────────────────────────────── -gcp project_id str GCP project (the logs land in) -gcp sa_email str service-account email -gcp sa_key_id str private-key id (for JWT `kid` header) -gcp sa_key_pem blob RSA private key, PKCS#8 PEM (~1.7 KB) -gcp min_severity u8 0=DEBUG..4=ERROR; default 1=INFO -``` - -Updates to `provisioning.toml`: - -```toml -[gcp] -project_id = "..." -sa_email = "esp32-logger@my-project.iam.gserviceaccount.com" -sa_key_id = "abc123..." # from the SA JSON key file -sa_key_pem = "gcp-sa-key.pem" # path; tool reads + embeds the bytes -min_severity = "info" # debug / info / warn / error -``` - -If the entire `[gcp]` section is missing from `provisioning.toml`, -cloud logging is **disabled** — the firmware boots normally and just -emits to serial. Cloud logging is opt-in per device, like everything -else in NVS. - -## On-device implementation - -### Tracing capture (`src/cloud_log.rs`, new) - -A `tracing_subscriber::Layer` that intercepts events. For each event: -- Capture severity (`INFO`/`WARN`/`ERROR`/`DEBUG`/`TRACE` mapped from tracing levels). -- Capture all structured fields into a `serde_json::Map` (string repr - for non-string types). -- Capture the tracing target (module path). -- Take a wall-clock timestamp (requires NTP — see below). -- Push onto a bounded queue. If full, drop oldest and bump a - dropped-counter; we log the drop count in the next batch so it - shows up in Cloud Logging. - -This layer composes with the existing log bridge (tracing → -EspLogger → serial) — events go to both serial and the cloud queue. - -### Sender task - -Background pthread, ~32 KB stack: - -1. Wait for Wi-Fi up. -2. Wait for NTP sync (or proceed without timestamps). -3. Loop: - - Sleep `flush_interval` (default 5s). - - Drain queue (up to N entries or M bytes per batch). - - If empty, continue. - - Ensure access token is valid; refresh via JWT exchange if needed. - - POST batch to `logging.googleapis.com`. On 4xx → log to serial, - drop the batch. On 5xx / network error → keep batch, retry with - backoff (similar to the OTA loop). - -### Time sync (NTP) - -Cloud Logging entries need real timestamps; without NTP our system -clock starts at 1970. Add an SNTP sync at boot via -`esp-idf-svc::sntp::EspSntp` (synchronous) before the sender task -starts. If sync fails, sender task falls back to omitting `timestamp` -so Cloud Logging assigns server-side time (less accurate but valid). - -NTP sync is done by `main.rs` after Wi-Fi connect; it's a precondition -for both cloud logging *and* (eventually) Rekor SET timestamp validity -(see ota.md Future Work). - -### Crates and footprint - -- `rsa` — RSA-PKCS#1 v1.5 signing for JWT. ~50 KB. -- `base64` — already have. -- `sha2` — already have (used by RSA signing too). -- `serde_json` — already have. -- `time` — for RFC 3339 timestamp formatting + UNIX time. ~40 KB. -- Logging code itself: ~10 KB. - -Total budget estimate: ~100 KB additional firmware. We have -plenty of headroom in the 1.94 MB OTA slot. - -RAM: ring buffer (default ~4 KB), token cache (~2 KB), JWT scratch -(~1 KB) → ~10 KB total during steady state. - -## Concerns - -1. **Log volume + cost.** If we ship every tracing event we could - easily hit Cloud Logging quotas or bill. Default to filtering at - `INFO` for cloud, leave `DEBUG` for serial only. The threshold - is NVS-configurable via `gcp/min_severity`. - -2. **Wi-Fi outages / device offline.** Sender task buffers with a - bounded queue, drops oldest when full, reports drop count in next - batch. Long offline period = lossy. Acceptable for a monitoring - channel; not a record of truth. - -3. **NTP dependency.** We don't currently sync time, so adding SNTP - is part of this work. SNTP fail → use server-side timestamps. - -4. **NVS partition is unencrypted.** The SA private key sits in flash - bytes. Anyone with physical access to the chip can dump it. Same - threat model as the wifi password and the trust roots. Mitigation - (real fix) is Flash Encryption + Secure Boot v2; deferred. For - now we accept the risk and scope the SA tightly: role - `roles/logging.logWriter` only, on a dedicated logs-only project. - -5. **RSA-2048 signing on ESP32 is slow** — ~hundreds of ms per JWT. - Cached for 1h, so amortized cost is negligible. But the first-time - sign blocks the sender task for noticeable time. Acceptable. - -## Settled - -- **Provisioning**: NVS via USB (matches wifi + trust). The SA key - never enters the OTA-distributed firmware image. -- **Severity threshold default**: `INFO`+ for cloud, configurable via - `gcp/min_severity` in NVS. -- **Single log name** with `module` field for component filtering, - rather than per-component `logName`s. -- **Pure-Rust `rsa`** crate, not mbedtls FFI. Headroom is plenty - for the ~50 KB. -- **Opt-in per device**: missing `[gcp]` block → cloud logging - disabled, serial-only. No GCP setup required to provision a device. - -## Future work (not in v1) - -- **Trace export** (full structured spans, not just events) — would - let us see request flows and timing. -- **Metrics export** to Cloud Monitoring — counters for OTA verifies, - poll cycles, Wi-Fi reconnects. -- **Alerting** based on log-based metrics (e.g., page on >5 OTA - verify failures in 1h). -- **OIDC instead of SA key** — would eliminate the key-leak concern - (paired with Secure Boot v2), but ESP32 doesn't have a way to get - a usable OIDC token from anywhere right now. diff --git a/monitoring-plan.md b/monitoring-plan.md deleted file mode 100644 index 7a26a7e..0000000 --- a/monitoring-plan.md +++ /dev/null @@ -1,276 +0,0 @@ -# Device monitoring — plan - -Periodically collect what the chip already knows about itself (heap, -stack, wifi, cpu, uptime) and ship it as **Cloud Monitoring time series** -via `monitoring.googleapis.com`. Same SA + key as cloud_log, broader -scope on the access token, separate POST path. Real metrics — chartable, -alertable — not log-based metrics. - -External sensors (power, current, temperature) are out of scope for v1. - -## What we collect - -All from ESP-IDF APIs already linked in. No sdkconfig changes for v1. -Numeric only; categorical things (`reset_reason`, `wifi_ps_type`) stay -in `cloud_log` events where they belong. - -| Metric type (custom.googleapis.com/esp32/…) | Source | Unit | Kind | -|---------------------------------------------|---------------------------------------------------------|------|-------| -| `free_heap` | `esp_get_free_heap_size()` | By | GAUGE | -| `free_heap_internal` | `heap_caps_get_free_size(MALLOC_CAP_INTERNAL)` | By | GAUGE | -| `min_free_heap` | `esp_get_minimum_free_heap_size()` (water-mark / boot) | By | GAUGE | -| `largest_free_block` | `heap_caps_get_largest_free_block(MALLOC_CAP_DEFAULT)` | By | GAUGE | -| `stack_hwm` *(label `task`)* | `uxTaskGetStackHighWaterMark` per known task | By | GAUGE | -| `wifi_rssi` | `esp_wifi_sta_get_ap_info().rssi` | dBm | GAUGE | -| `cpu_freq_mhz` | `esp_clk_cpu_freq()` / 1e6 | MHz | GAUGE | -| `uptime_secs` | `esp_timer_get_time() / 1_000_000` | s | GAUGE | -| `boot_count` | NVS-persisted u32, ++'d at boot | 1 | GAUGE | -| `nvs_free_entries` | `nvs_get_stats().free_entries` | 1 | GAUGE | -| `cloud_log_queue_depth` | `LogQueue::len()` (new accessor) | 1 | GAUGE | -| `cloud_log_dropped_total` | running drop counter on `LogQueue` | 1 | GAUGE | - -`stack_hwm` is one metric type with a `task` label (e.g. -`task=ota`, `task=cloud_log`, `task=metrics`, `task=main`) — keeps the -metric catalogue small and lets Cloud Monitoring chart all tasks on one -graph. - -OTA-loop counters (`consecutive_failures`, `total_polls`, -`updates_applied`) belong to `src/ota.rs` and should be emitted from -there into the same metrics path — not reached into from `metrics.rs`. -Same goes for any future module-owned counters. `metrics.rs` only owns -chip-wide stats. - -## Architecture - -``` -┌─────────────── ESP32 ───────────────┐ ┌──── GCP ────┐ -│ │ │ │ -│ cloud_log (existing) ──HTTPS──────────────────────▶│ Logging API │ -│ ▲ │ │ -│ │ shares │ │ -│ CachedToken │ │ -│ (logging+monitoring scopes) │ │ -│ │ │ │ -│ metrics (new) ──HTTPS───────────────────────────────│ Monitoring │ -│ - sample every 5 min │ │ API │ -│ - one CreateTimeSeries POST │ │ │ -└──────────────────────────────┴─────┘ └──────────────┘ -``` - -The new component is one thread that wakes every N seconds, snapshots -~12 metrics, and POSTs a single `CreateTimeSeries` request. It piggy- -backs on cloud_log's access-token cache via a shared -`Arc>` — no second JWT mint, no second RSA sign. - -## Auth - -Same SA, same key as cloud_log. The JWT mint requests **both scopes**: - -``` -scope = "https://www.googleapis.com/auth/logging.write \ - https://www.googleapis.com/auth/monitoring.write" -``` - -(space-separated; Google accepts multi-scope JWTs and the resulting -access token works against both APIs.) - -New IAM binding for the SA — least-privilege, only metric writes: - -```bash -gcloud projects add-iam-policy-binding $PROJECT_ID \ - --member="serviceAccount:$SA_EMAIL" \ - --role="roles/monitoring.metricWriter" -``` - -`metricWriter` grants `monitoring.timeSeries.create` and metric- -descriptor create — nothing else. README "Optional: GCP Cloud Logging" -section grows to "Cloud Logging + Monitoring" with this extra binding. - -## API call - -`POST https://monitoring.googleapis.com/v3/projects//timeSeries` -with `Authorization: Bearer ` and: - -```json -{ - "timeSeries": [ - { - "metric": { - "type": "custom.googleapis.com/esp32/free_heap" - }, - "resource": { - "type": "generic_node", - "labels": { - "project_id": "", - "location": "global", - "namespace": "esp32", - "node_id": "" - } - }, - "metricKind": "GAUGE", - "valueType": "INT64", - "points": [ - { "interval": { "endTime": "2026-05-02T22:30:00Z" }, - "value": { "int64Value": "182456" } } - ] - }, - { - "metric": { - "type": "custom.googleapis.com/esp32/stack_hwm", - "labels": { "task": "ota" } - }, - "resource": { "type": "generic_node", "labels": { ... } }, - "metricKind": "GAUGE", - "valueType": "INT64", - "points": [ ... ] - } - // ... one entry per metric per task-label combination - ] -} -``` - -Cloud Monitoring auto-creates `MetricDescriptor`s on first write — no -separate provisioning step. Up to 200 series per request; we have ~15 -per snapshot, so one POST per cycle. - -## On-device implementation - -### Shared `gcp_auth` (refactor) - -Move JWT mint + `CachedToken` out of `src/cloud_log.rs` into a small -`src/gcp_auth.rs`. Owns the SA key, mints a multi-scope token, -exposes `Arc>` to both `cloud_log` and `metrics`. -First caller through the door does the RSA sign; the other gets the -cached token for free. - -### `src/metrics.rs` (new) - -```rust -pub fn run( - cfg: GcpConfig, - auth: gcp_auth::TokenHandle, - queue_stats: cloud_log::QueueStatsHandle, -) -> ! { - let mac = device_mac(); - let resource = generic_node_resource(&cfg.project_id, &mac); - let nvs_handle = open_nvs_for_stats(); - let task_handles = TaskHandles::collect(); // ota, cloud_log, metrics, main - loop { - std::thread::sleep(cfg.metrics_interval); - let snapshot = collect(&task_handles, &nvs_handle, &queue_stats); - let body = build_create_time_series(&resource, &snapshot, now_rfc3339()); - let bearer = auth.get_or_refresh()?; - post_create_time_series(&cfg.project_id, &bearer, &body); - } -} -``` - -Stack budget ~16 KB (HTTPS + JSON, no RSA on the hot path since auth -is cached). Spawned from `main.rs` only if `gcp.is_some()`. - -### Boot count - -Persisted in the existing `ota` NVS namespace as `boot_count` (u32); -incremented once at startup before any thread spawns. Doubles as a -useful number for OTA debugging too ("how many times has this device -rebooted in this slot?"). - -### Task handles for stack hwm - -Each spawned thread publishes its `TaskHandle_t` into a `OnceLock` in -its own module: - -```rust -pub static TASK_HANDLE: OnceLock<*mut c_void> = OnceLock::new(); -fn run(...) -> ! { - let _ = TASK_HANDLE.set(unsafe { xTaskGetCurrentTaskHandle() } as _); - ... -} -``` - -`metrics::collect()` reads each handle (skipping any unset), calls -`uxTaskGetStackHighWaterMark` per. Missing → field omitted, not zeroed. - -## Provisioning (NVS) - -One new optional key in the existing `gcp` namespace: - -``` -ns key type notes -───────────────────────────────────────────────────────────────── -gcp metrics_interval_secs u32 default 300; 0 disables -``` - -`provisioning.toml`: - -```toml -[gcp] -# ... existing fields ... -metrics_interval_secs = 300 # optional; default 300, 0 = off -``` - -Metrics thread only spawns if `gcp.is_some()`. If -`metrics_interval_secs = 0` the module is loaded but the thread -isn't spawned. - -## Concerns - -1. **Cloud Monitoring quotas + cost.** Custom metrics are billed at - $0.30/MiB/month after the free tier (150 MiB/month). 15 series × - 1 sample / 5 min × 30 days ≈ 130 K samples/device/month, well - under the per-project series cap. Tunable via - `metrics_interval_secs`. - -2. **Reset detection on `boot_count`.** With GAUGE we just snapshot - the current value; Cloud Monitoring doesn't try to interpret it as - a rate. If we ever want "boots per hour" we'd switch to CUMULATIVE - and provide `startTime`. Same for `dropped_total`. - -3. **Auto-created descriptors are sticky.** Once Cloud Monitoring - sees a custom metric, the descriptor sticks (with whatever label - keys we sent first). Renaming `task=ota` → `task=ota_loop` later - would create a new descriptor. Keep label vocab stable. - -4. **`metrics_interval_secs = 0` should mean off, not zero-sleep.** - Defensive check at thread spawn — don't enter a spinloop. - -5. **Time-series clock skew.** `endTime` must be ≥ chip's NTP-synced - wall clock. We already wait for NTP before spawning cloud_log; - metrics gates on the same precondition. - -## Settled - -- **Cloud Monitoring API**, not log-based metrics. -- **Same SA, multi-scope JWT** — single access token covers logging + - monitoring; cached and shared between threads. -- **GAUGE for all v1 metrics**, including the running totals — keeps - the descriptor catalogue simple. Promote to CUMULATIVE only if - needed for rate-style alerting. -- **Custom metric prefix `custom.googleapis.com/esp32/`** with - the `task` label for per-task fanout. -- **Auto-create descriptors** — no separate provisioning step on the - GCP side; first POST creates the schema. -- **`generic_node` resource** with the same labels as cloud_log - (consistent fleet identity). -- **Module owns its counters.** `metrics.rs` collects chip-wide - stats only. OTA / cloud_log emit their own module-specific metrics - via the same path. No cross-module reach-arounds. - -## Future work (not in v1) - -- **Per-task CPU runtime stats** via `vTaskGetRunTimeStats`. Needs - `CONFIG_FREERTOS_USE_TRACE_FACILITY=y` + - `CONFIG_FREERTOS_GENERATE_RUN_TIME_STATS=y`. Adds CPU% per task as - another `stack_hwm`-style multi-task metric. -- **Light-sleep stats.** Once `esp_pm_*` is enabled, expose - time-in-light-sleep as a CUMULATIVE metric. Pairs with the - modem-sleep / power-saving work. -- **External power telemetry.** INA219/INA226 over I2C → bus voltage, - current, instantaneous power as proper metrics. Worth its own plan. -- **Pre-declared MetricDescriptors with units + descriptions** — - improves the Cloud Monitoring UI but not necessary for charting. -- **Alerting policies** as Terraform / `gcloud alpha monitoring` - resources, checked into the repo so they version with the metrics - themselves. -- **Cumulative metrics with `startTime`** for per-second rate - aggregation in Cloud Monitoring (drops/sec, polls/sec, …). diff --git a/notes.txt b/notes.txt deleted file mode 100644 index 0358717..0000000 --- a/notes.txt +++ /dev/null @@ -1,119 +0,0 @@ -esp32 std Rust — internal design notes -====================================== - -This file is internal: design decisions, issues we hit during setup, and -gotchas to remember. For prerequisites and day-to-day workflow, see -README.md. - -Why std instead of no_std -------------------------- -Std Rust on ESP32 is built on top of `esp-idf-sys`, which builds and -links the ESP-IDF C SDK. That means we need the IDF C/C++ toolchain, -CMake/Ninja, and ESP-IDF's Python orchestration (kconfig, partition -tables, esptool). - -The lighter alternative is no_std bare-metal with `esp-hal`. No Python, -no CMake, no C SDK — just LLVM. Worth knowing for future projects where -std isn't required. We picked std for this project because we want easy -Wi-Fi + HTTP + TLS via esp-idf-svc, plus standard threading/heap. - -Why we pin Python 3.12 via a generated shim -------------------------------------------- -embuild bootstraps the IDF venv using whatever `python3` is first on PATH. -On macOS that's Apple's /usr/bin/python3 (3.9.6). ESP-IDF v5.2 nominally -supports 3.8–3.12, but pip dependency resolution on 3.9 silently drops -some transitive deps that the IDF check then complains about (see issue B -below). - -Fix: the Makefile creates `.embuild/python-shim/python3` as a symlink to -a uv-managed Python 3.12, and prepends that directory to PATH for every -recipe. The shim isn't checked in — it's regenerated by the -`ensure-python-shim` Make target, which is a prerequisite of `build`. - -Issues hit during initial setup -------------------------------- - -A. CMake "Failed to run Python dependency check ... Error: 255", - complaining that `esp-idf-kconfig` requires `ruamel.yaml`. - Cause: the IDF python venv was missing `ruamel.yaml` even though it - was declared as a transitive dep. Pip's resolution on Python 3.9 - silently dropped it. - First-pass fix that *seemed* to work but didn't fully: - uv pip install \ - --python ./.embuild/espressif/python_env/idf5.2_py3.9_env/bin/python \ - ruamel.yaml - Made the import work, but the IDF check still failed for the next dep - down (see B). - -B. After installing ruamel.yaml, IDF's check still reported it missing - for the next dep (`esp-idf-panic-decoder` -> ruamel.yaml). The real - issue was the venv being on Python 3.9.6. Several of these resolution - glitches go away on 3.12, which is also what newer IDF versions - recommend. - Real fix: - 1. install uv (one-time) - 2. The Makefile's `ensure-python-shim` target installs 3.12 via uv, - creates `.embuild/python-shim/python3` -> uv 3.12, and the Makefile - prepends that dir to PATH for every recipe. - 3. rm -rf .embuild (force IDF venv recreate) - 4. make build (embuild bootstraps a fresh 3.12 venv; - ensure-python-shim re-creates the shim too) - -Why WIFI_SSID/WIFI_PASS are compile-time env vars -------------------------------------------------- -src/main.rs reads them via `env!()`, baked into the firmware binary. -Tradeoff: simple, no provisioning UI needed, works the moment the chip -boots. Downside: changing networks means rebuilding and reflashing. -For a single-developer playground this is the right tradeoff. If we -ever want to make this distributable, switch to NVS-stored creds with -either an initial AP-mode provisioning portal or BLE provisioning. - -The build.rs has cargo:rerun-if-env-changed lines for both vars so -editing wifi.env triggers a rebuild without `cargo clean`. - -Why export-esp.sh is gone, replaced by Makefile-internal env ------------------------------------------------------------- -espup drops `~/export-esp.sh` with hardcoded versioned paths to the -Xtensa toolchain. Sourcing it before each cargo command was awkward. -The Makefile sets LIBCLANG_PATH and prepends to PATH itself, using -wildcards over the date-stamped subdirectories so espup version bumps -don't require editing the file. Keeps the env close to the build steps -that need it. - -Custom partition table path resolution --------------------------------------- -ESP-IDF resolves `CONFIG_PARTITION_TABLE_CUSTOM_FILENAME` relative to -its top-level CMake project's source dir. For embuild's synthetic IDF -project, that's `target/.../esp-idf-sys-/out/`, not our repo root. -A bare `"partitions.csv"` in sdkconfig therefore fails: - ninja: error: '.../out/partitions.csv', needed by 'partition-table.bin', missing - -Workaround we settled on: keep `sdkconfig.defaults.in` checked in with -the placeholder `@PROJECT_DIR@/partitions.csv`, and have the Makefile -substitute the absolute path at build time, writing the real -`sdkconfig.defaults` (gitignored). The .in template is committed so -anyone cloning the repo gets the right shape; the resolved sdkconfig -is per-machine. - -Other things tried that didn't pan out: -- Putting partitions.csv at the repo root and hoping IDF would walk up - to find it — it doesn't. -- `[package.metadata.esp-idf-sys] esp_idf_sdkconfig_defaults = [...]` - with a generated absolute-path file — chicken-and-egg with build - ordering (esp-idf-sys's build.rs runs before any file we'd write). -- Pre-copying partitions.csv into the synthetic project's `out/` from - the Makefile — works, but only after the synthetic project has been - created at least once, requiring an awkward two-build first run. - -Onboard LED — there isn't one on this board -------------------------------------------- -The Inland ESP-WROOM-32 we have (Micro Center SKU 027466, rebadged -Keyestudio KS0413, "mini" form factor) has only a power LED ("D1", -hardwired to 3V3, always on). The DOIT/DEVKITC-style blue LED on -GPIO 2 is NOT populated on this variant. - -Confirmed by flashing a GPIO 2 toggle and watching the serial monitor: -the firmware boots, prints `blinky starting`, and the IDF gpio driver -logs reconfiguring GPIO 2 — but no LED visibly changes. To see anything -blink on this board you need either an external LED + ~330Ω resistor, -or the e-ink display we're adding (see eink-plan.md). diff --git a/tools/provision/src/main.rs b/tools/provision/src/main.rs index 7b26413..5e23aab 100644 --- a/tools/provision/src/main.rs +++ b/tools/provision/src/main.rs @@ -1,5 +1,6 @@ //! Build a binary NVS partition image from a `provisioning.toml` and -//! (optionally) flash it to the device. See `provisioning-plan.md`. +//! (optionally) flash it to the device. See `docs/ota.md` for the +//! provisioning + NVS-namespace design. //! //! The actual NVS binary format is generated by ESP-IDF's //! `nvs_partition_gen.py` (already on disk inside `.embuild/`). We just