diff --git a/py-image-builder/DESIGN.md b/py-image-builder/DESIGN.md new file mode 100644 index 0000000..3cb8a98 --- /dev/null +++ b/py-image-builder/DESIGN.md @@ -0,0 +1,289 @@ +# `py-image-builder` — a docker-less, layer-aware Python image builder + +> Status: **Design / plan** (no code yet). This document proposes the architecture +> and an incremental implementation plan. Comments and pushback welcome. + +## 1. Goal + +Build OCI container images for Python applications **without a Docker daemon**, in a +way that is fast and cheap because it exploits content-addressed layering: + +- **Wheel dependencies are split into reusable layers.** Once a layer for a given + wheel exists, builds reuse it **without downloading the wheel or uploading the + layer bytes** again. +- **Only genuinely new dependencies cause layer changes.** Bumping app code, or + rebuilding an unchanged dependency set, changes *no* dependency layers. +- **Application code is a thin layer on top** of the dependency layers, so the + common edit-rebuild loop touches only one small layer. +- Builds are **reproducible**: same lock + same source + same base ⇒ same image + digest, which lets us skip work entirely when nothing changed. + +### Non-goals (initially) + +- Compiling C extensions / building wheels from sdists. We consume **pre-built + wheels** only. (sdist→wheel is a future extension; it would run a build, then + the resulting wheel re-enters the normal layer path.) +- Replacing dependency *resolution*. We delegate resolution to existing, + correct tools (`uv` / `pip`) and consume their lockfile output. +- A general-purpose Dockerfile interpreter. This is a focused Python app builder + (think `ko`, but for Python wheels). + +## 2. Why this works (the core idea) + +Three well-understood registry/OCI facts combine to make "no-bytes" rebuilds possible: + +1. **Layers are content-addressed.** A layer blob is referenced by the digest of + its (gzip-compressed) tar. Two builds that produce byte-identical layers + produce the same digest, and a registry stores it once. +2. **Registries support blob existence checks and cross-repo mounts.** Before + uploading, we `HEAD /v2//blobs/`. If the blob already exists in + the target repo, we upload nothing. If it exists elsewhere in the same + registry, we `POST .../blobs/uploads/?mount=&from=` to mount + it — **zero bytes transferred**. +3. **A manifest is just JSON referencing blob digests.** If every layer + config + blob already exists, a "build" is reduced to `PUT manifest` (a few KB). + +So if we make each wheel produce a **deterministic** layer and we **cache the +mapping `wheel-sha256 → {diffID, blob digest, size}`**, then for a dependency we +have seen before we never touch the wheel bytes: we already know its layer digest, +we confirm the blob is in the registry, and we reference it. Adding a brand-new +dependency is the only thing that forces a download + layer build + one upload. + +This is the same family of tricks behind `crane`/`go-containerregistry` blob +mounting, `nixery`'s per-package layering, and Bazel `rules_oci` `py_image`'s +deps-vs-app split — specialized here for the Python wheel ecosystem. + +## 3. Layering strategy + +### 3.1 One deterministic layer per wheel (default) + +Each resolved distribution (`name==version`, for a specific +platform/abi/python tag) becomes **its own layer** whose tar contains exactly the +files that installing that wheel writes into the environment's `site-packages` +(plus its `*.dist-info` and any `bin/` console scripts). + +Properties this gives us, matching the requirements: + +- Adding a dependency ⇒ exactly one new layer; all existing layers keep the same + digest ⇒ already present in registry ⇒ mounted/skipped. +- Removing a dependency ⇒ that layer is dropped from the manifest; no other layer + changes. +- Bumping a dependency version ⇒ old layer dropped, one new layer added. + +Layer **order** in the manifest is sorted deterministically (by distribution name, +then version) so the assembled image/config is itself reproducible and cacheable. +The app code layer is always last (top). + +### 3.2 Determinism requirements (critical) + +A wheel must always produce a byte-identical layer, or the cache and reuse break. +"Installing" a wheel is essentially: unzip it, lay files under a fixed +`site-packages` prefix, generate `RECORD`, and synthesize `console_scripts`. We do +this ourselves (in Go) to control every byte: + +- **tar normalization:** fixed `mtime` (epoch 0 or a fixed `SOURCE_DATE_EPOCH`), + `uid=gid=0`, fixed mode bits (`0644` files / `0755` dirs / preserve exec bit on + scripts), no device/xattr/PAX-extra records, **entries emitted in sorted order**, + consistent name prefixes, no `__pycache__`/`.pyc` (compiled at runtime or in a + separate optional step). +- **deterministic gzip:** pin compression level and strip the gzip header mtime. + Better still, **cache the compressed blob** keyed by content so we never + recompress (avoids depending on the compressor being bit-stable across versions). +- **fixed install layout:** a single prefix shared by all wheels, e.g. + `/app/.venv/lib/python/site-packages` with scripts in `/app/.venv/bin`. The + base image sets `PATH`/`VIRTUAL_ENV` (or we drop a `.pth`) so the interpreter + finds them. +- **disjoint file sets:** distributions own mostly-disjoint files; tar handles + repeated *directory* headers fine. Namespace packages (pkgutil/PEP-420) share a + dir but distinct files. We generate each package's `RECORD` deterministically and + set `INSTALLER` to a constant. Edge cases (data files, `*.pth` from a package) + are captured inside that package's own layer. + +The cache key for a layer is: +`(wheel-sha256, target-os/arch, python-tag/abi-tag, install-layout-version)`. +Pure-python wheels (`py3-none-any`) are platform-independent ⇒ one layer shared +across all arches. + +### 3.3 Layer-count trade-off (configurable) + +Per-wheel layering maximizes reuse but a large dependency tree can exceed practical +limits (registries/runtimes tolerate many layers, but ~100+ tiny blobs slow pulls +and some tooling caps near 127). Strategies, selectable via flag: + +- `per-wheel` (default): best reuse, one layer per dist. +- `hybrid`: large/heavily-shared wheels get their own layer; the long tail of tiny + wheels is **bin-packed by a stable partition** (e.g. hash of name into K buckets) + so adding a dep usually perturbs only one bucket layer. Bounded layer count, + slightly weaker "only-new-dep" guarantee for bucketed deps. +- `single-deps-layer`: all deps in one layer + app layer. Simplest, weakest reuse + (any dep change rebuilds the whole deps layer). Useful for tiny apps. + +## 4. The "no-bytes" build flow + +``` +lockfile + source + base-ref + │ + ▼ +1. Ingest lock ─► [{name, version, wheel-url, sha256, tags}, ...] + │ + ▼ +2. For each dep: + key = sha256 (+ platform/py tags) + ├─ layer meta in cache? ── no ─► download wheel ─► build layer + │ │ yes │ (deterministic) + │ ▼ ▼ + │ know {blob digest, diffID} cache meta + blob + ▼ +3. For each layer blob: + HEAD /v2//blobs/ + ├─ exists in target repo ─► reference only (0 bytes) + ├─ exists elsewhere in reg. ─► cross-repo mount (0 bytes) + └─ missing ─► upload once + │ + ▼ +4. App layer: deterministic tar of source (respect ignore file, drop __pycache__), + content-addressed → same HEAD/mount/upload logic. + │ + ▼ +5. Assemble: base (by digest) + sorted dep layers + app layer. + Build config (entrypoint, env: PATH/VIRTUAL_ENV/PYTHONPATH, workdir, user, + labels), compute manifest digest. + │ + ▼ +6. If multi-arch: repeat per arch, assemble an image index. + │ + ▼ +7. If manifest digest already tagged ─► done (no upload at all). + Else PUT config (if missing) + PUT manifest/index. ← only small JSON moves +``` + +For an unchanged dependency set, steps 2–3 transfer **zero** dependency bytes; only +the app layer (if source changed) and the manifest move. + +### 4.1 Caches + +- **Local cache** (always): content-addressed dirs for wheels and built layer + blobs + a small metadata DB (`wheel-sha256 → layer meta`). Lets repeated local + builds skip downloads and recompression. +- **Remote shared cache** (optional): the **target registry itself** is the cache + for blobs — `HEAD` is the lookup. Optionally a side bucket (GCS/S3) for layer + metadata so CI runners share the `sha256 → digest` map without each re-deriving + it. This is what enables a cold runner to build with no wheel downloads. + +## 5. Dependency resolution / lock ingestion + +We do **not** resolve; we ingest a fully-pinned, hashed lock so every wheel is +identified by URL + sha256. Supported inputs (pluggable parsers): + +- `requirements.txt` with `--hash=sha256:...` (from `pip-compile`/`uv pip compile`). +- `uv.lock`. +- `poetry.lock` / PDM lock. +- `pip install --report ` output (rich, includes resolved wheel URLs). + +If only loose requirements are provided, we can **shell out** to `uv pip compile` +to produce a hashed lock first (clearly an online step, separate from the +no-bytes build path). The lock is the contract that makes builds reproducible. + +## 6. Base image handling + +- Reference a base by **digest** (e.g. a distroless or Chainguard `python` image). + We fetch only its *manifest + config* (small) — base layer blobs are referenced + by digest and **never pulled**. +- We append our layers on top and rewrite the config: set + `Env` (`PATH`, `VIRTUAL_ENV`, optional `PYTHONPATH`), `WorkingDir`, `User` + (non-root by default), `Entrypoint`/`Cmd`, and `Labels`/annotations. +- The base must provide a compatible CPython (matching the wheels' python/abi + tags). We validate the base's interpreter version against the lock's tags and + fail fast on mismatch. + +## 7. Reproducibility, SBOM, provenance + +- Deterministic everything ⇒ identical inputs yield an identical image digest. + We expose `--dry-run`/`--print-digest` to compute the would-be digest offline. +- **SBOM**: we already hold every wheel's name/version/sha256, so we emit an + SBOM (SPDX/CycloneDX) and attach it. (Natural fit for this repo's Chainguard + leanings.) +- Optional signing/attestation (`cosign`-style) and SLSA provenance as a later + add-on. + +## 8. Multi-arch + +Wheels are tagged by platform/abi (manylinux, macОS, etc.). For multi-arch images +we resolve a wheel set **per target platform**, build per-arch images (sharing all +pure-python layers), and publish an **image index** referencing each arch manifest. + +## 9. Surface area & tech choices + +- **Language: Go**, consistent with the rest of this monorepo, using + [`go-containerregistry`](https://github.com/google/go-containerregistry) (ggcr) + for all registry I/O, layer/image types, blob mounting, and auth (keychain). No + Docker daemon, no shelling out to `crane`. +- Wheel "install" implemented in Go: parse the wheel zip (it is a zip with + `*.dist-info/{RECORD,WHEEL,METADATA,entry_points.txt}`), lay files into the + staging prefix, synthesize console scripts, write deterministic tar. +- **CLI** first: + ``` + py-image-builder build \ + --base cgr.dev/chainguard/python@sha256:... \ + --lock requirements.txt \ + --source ./ \ + --entrypoint "python -m myapp" \ + --platform linux/amd64,linux/arm64 \ + -t registry.example.com/me/myapp:latest + ``` +- Later, a thin **server** (Cloud Run, matching the pattern in `image-workflow`, + `iap`, `litestream`) that builds on push/dispatch, plus its own `main.tf`. +- New self-contained module dir `py-image-builder/` with its own `go.mod`, + `README.md`, and (eventually) `main.tf`, mirroring repo conventions. + +## 10. Incremental implementation plan + +Each phase is independently useful and reviewable. + +1. **Skeleton + deterministic tar.** New `py-image-builder/` Go module; a function + that writes a normalized, reproducible tar and a round-trip determinism test + (same input ⇒ same bytes ⇒ same digest). +2. **Wheel → layer.** Parse a wheel zip, lay out files into the fixed prefix, + generate `RECORD`/console scripts, produce a ggcr `v1.Layer`. Golden-file tests + asserting stable diffIDs for fixture wheels (one pure-python, one manylinux). +3. **Lock ingestion.** Parser(s) for hashed `requirements.txt` (first), normalized + to `{name, version, url, sha256, tags}`. Add `uv.lock` next. +4. **Local layer cache.** Content-addressed wheel + blob store and the + `sha256 → layer meta` DB. Prove second build does zero downloads/recompression. +5. **Image assembly + push (ggcr).** Base-by-digest, sorted dep layers, app layer, + config rewrite, manifest. Wire `HEAD` existence checks + cross-repo blob mount. + Prove an unchanged-deps rebuild pushes only the manifest (and app layer if + source changed). +6. **App layer + ignore file.** Deterministic source tar honoring an ignore file, + excluding `__pycache__`/VCS dirs. +7. **Multi-arch + image index.** +8. **SBOM emission**, then optional signing/provenance. +9. **Cloud Run server + Terraform** to trigger builds (optional, repo-pattern). + +## 11. Risks & open questions + +- **Determinism is load-bearing.** Any non-reproducible byte (mtime, gzip header, + entry ordering, locale-dependent sorting) silently breaks reuse. Mitigation: + cache the *compressed blob* by content, plus determinism tests in CI. +- **Layer-count vs. reuse.** Per-wheel is ideal for reuse but can produce many + layers; the `hybrid` strategy bounds it at some reuse cost. Pick defaults and + document the trade-off (§3.3). +- **File collisions** between distributions (shared namespace dirs, data files, + duplicate scripts). Need a conflict policy (last-writer / fail / dedicated + layer). Mostly rare in practice; must be handled deterministically. +- **`__pycache__`/`.pyc`.** Excluded for determinism; optionally pre-compile in a + separate, clearly-marked optional layer (bytecode is hash-stable given fixed + inputs + `PYTHONHASHSEED`). +- **Cross-repo mount auth.** The push credential must also have *pull* scope on the + source repo for `mount=...&from=...` to succeed; otherwise we fall back to + upload. Handle the 202-without-mount case gracefully. +- **Base/interpreter mismatch.** Wheel tags must match the base CPython; validate + and fail fast. +- **sdist-only dependencies.** Out of scope initially; later, build the wheel once + then feed it into the normal layer path. + +## 12. Prior art (for reference) + +`ko`, `apko`/`melange`, `crane`/`go-containerregistry` (blob mounts), Bazel +`rules_oci` / `rules_docker` `py3_image` (deps-vs-app split), `nixery` (per-package +layers), Cloud Native Buildpacks (layer reuse + rebase), `distroless`.