diff --git a/pymage/README.md b/pymage/README.md index aebf70f..05ba658 100644 --- a/pymage/README.md +++ b/pymage/README.md @@ -73,6 +73,8 @@ the config value, which overrides the built-in default. | `max-wheel-layers` | `--max-wheel-layers` | *(derived from `max-layers`)* | | `push-concurrency` | `--push-concurrency` | auto (≥ 4, scales with CPUs) | | `no-cache` | `--no-cache` | `false` (caching is on by default) | +| `extras` | `--extra` (repeatable) | — (enables uv project optional-dependency groups) | +| `package` | `--package` | — (build a single uv workspace member) | | `python` | `--python` | auto-detected from the base | | `prefix` | `--prefix` | `/app/.venv` | | `workdir` | `--workdir` | `/app` | @@ -147,6 +149,57 @@ pymage build \ --repo registry.example.com/me/myapp -t latest ``` +### Optional dependencies, workspaces, and markers (uv.lock) + +pymage installs the project's **runtime closure** from `uv.lock` (the deps you'd +get from `uv sync --no-dev`), not every package in the lock: + +- `--extra ` enables one of the project's own + `[project.optional-dependencies]` groups (repeatable). Extras requested *by* + your dependencies (e.g. `fastapi[standard]`) are always followed. +- `--package ` roots the closure at a single uv **workspace member** + instead of the union of all members — useful for monorepos that build several + images from one lock. +- **Environment markers are evaluated for the target.** A dependency gated on + `sys_platform == 'win32'` or `python_version < '3.11'` is included only when it + applies to the platform/interpreter being built, so Linux images don't carry + Windows-only or stale-Python-only packages. Markers are evaluated per platform, + so each arch of a multi-arch build gets the correct set. + +### Source distributions (sdists) + +When the lock pins a package that has **no compatible wheel** (an sdist-only +release, or a compiled package for a platform without a published wheel), pymage +builds a wheel from the sdist using the host's `pip` (`pip wheel --no-deps`). +This requires a Python toolchain (`python3`/`python` with `pip`) on the build +host. + +- **Pure-python sdists** build to a `py3-none-any` wheel and work for any target. +- **Compiled sdists** can only be built for the **host platform**; building such + a package for a different architecture fails with a clear compatibility error. + Prefer a base/lock that provides pre-built wheels for those. + +Built wheels are cached (keyed by the sdist hash and target) so they aren't +rebuilt on subsequent builds. + +### Base image requirements (OS / system libraries) + +pymage installs Python wheels on top of the base image; it does **not** install +OS packages. The base image must already provide everything your dependencies +need at runtime beyond the interpreter and pure-Python code, including: + +- the Python interpreter and standard library (matching the lock's `cp` tags); +- shared system libraries that compiled wheels link against (e.g. `libffi`, + `libssl`, `libstdc++`, `libgomp` for some ML wheels); and +- non-Python runtime tools your app shells out to (e.g. ImageMagick for Wand, + `ffmpeg`, `git`). + +Choose (or build) a base that bundles these. For Debian-style bases that means a +variant with the libraries preinstalled; for Chainguard/Wolfi, compose a base +with the needed `apk` packages. If a dependency needs a system library the base +lacks, the image builds fine but fails at runtime — pymage can't add `apt`/`apk` +packages for you. + ### Choosing a base image The base is an input to the build, so it affects reproducibility just like the @@ -184,6 +237,8 @@ base that advertises its version. | `--push-concurrency` | Max concurrent layer uploads when pushing (0 = auto). | | `--platform` | Target platform(s); selects compatible wheels and base. Repeatable / comma-separated (e.g. `linux/amd64,linux/arm64`) builds a multi-arch image index. Defaults to the platforms the base image supports. | | `--python` | Interpreter version, e.g. `python3.12`. Optional — **auto-detected from the base** when omitted; if set, must match the base. Drives wheel selection and the site-packages layout. | +| `--extra` | Enable a uv project optional-dependency group (repeatable). | +| `--package` | Build a single uv workspace member by name (default: union of all members). | | `--cache-dir` | Cache root (default: `$PYMAGE_CACHE_DIR` or the per-user cache dir). Caches compressed layers, downloaded wheels, and base interpreter detection. | | `--no-cache` | Disable all caching (layers, downloaded wheels, interpreter detection). | | `--prefix` | install prefix / venv root (default `/app/.venv`). | diff --git a/pymage/docs/real-world-comparison.md b/pymage/docs/real-world-comparison.md index b0eaa43..f9e231d 100644 --- a/pymage/docs/real-world-comparison.md +++ b/pymage/docs/real-world-comparison.md @@ -2,9 +2,10 @@ This is a hands-on study of migrating real, uv-based Python projects that ship a `Dockerfile` to `pymage`. The goals were to (a) see how the resulting images -differ and (b) surface gaps/bugs real projects hit. It directly drove a -significant fix (runtime-only dependency resolution) and a ranked list of -migration blockers. +differ and (b) surface gaps/bugs real projects hit. It directly drove several +fixes — runtime-only dependency resolution, sdist→wheel building, `--extra` / +`--package` selection, and environment-marker evaluation — plus a ranked list of +the remaining migration blockers. ## Method & caveats @@ -75,47 +76,94 @@ Dockerfile: contains. - **Fast, docker-less builds** from any OS. +## Incremental rebuilds: where layering pays off + +The study above compares *first* builds. The bigger day-to-day win is what +happens on the **second** build, after a small change. Because pymage puts each +wheel (or a stable hash-bucket of wheels) in its own content-addressed layer, +only the layers whose bytes actually changed need to be pushed; every other +layer is deduplicated by the registry (a `HEAD` that returns "already present"). + +Measured against `uv-docker-example` (44 layers: base + 40 wheels + app), pushing +both the original and modified image to the same registry and diffing the +resulting manifests: + +| Change | Layers changed | Bytes re-uploaded | +| --- | --- | --- | +| Edit one app source file | **1 / 44** (the app layer) | **10.2 KB** | +| Bump one dependency (`six` 1.16.0 → 1.15.0) | **1 / N** (just that wheel's layer) | **11.3 KB** | + +Contrast with the typical uv Dockerfile, which installs the whole environment in +one step: + +```dockerfile +RUN --mount=... uv sync --frozen --no-install-project # one big "deps" layer +COPY . /app # app layer +``` + +- **Editing app code** invalidates the `COPY` layer in *both* approaches, so + here they're comparable (both re-push only the small app layer). +- **Bumping a single dependency** re-runs `uv sync`, which rewrites the entire + virtualenv — a *single* image layer containing **all** dependencies. The whole + deps layer is a new blob and must be re-pushed and re-pulled: tens to hundreds + of MB (≈15 MB for `uv-docker-example`, ≈35 MB for the FastAPI backend, ≈190 MB + for `imgpush`). pymage re-uploads only the one changed wheel (≈11 KB here). + +So the same one-line dependency bump costs the Dockerfile its full dependency +layer, while pymage uploads kilobytes. This mirrors the unit/e2e guarantees +(`TestAutoAddingDepChangesOneLayer`, `TestAutoVersionBumpChangesOneLayer`, and +the e2e no-bytes rebuild test): adding or bumping one dependency changes at most +one layer, and an unchanged input re-pushes nothing. + +(The `auto` layer budget bin-packs wheels into ≤127 layers when there are more +wheels than the budget; a changed wheel then re-pushes only its bucket-mates, so +the worst case is one bucket rather than the whole environment.) + ## Gaps & migration blockers (ranked) 1. **(FIXED) Installed the whole `uv.lock`, including dev groups.** Caused 50–63% dependency bloat above. Now resolves the runtime closure. -2. **sdist-only dependencies are unsupported.** `imgpush` fails on - `timeout-decorator==0.5.0`, which publishes no wheel: - `uv.lock: timeout-decorator==0.5.0 has no wheels (sdist-only deps are not - supported)`. uv/pip build the sdist into a wheel transparently; pymage - consumes pre-built wheels only. This is a real blocker for projects with any - wheelless dependency. *Possible fix: build sdists to wheels once (shell out to - `uv`/`pip wheel`) and feed them into the existing layer path.* -3. **No system/OS packages.** pymage installs Python wheels, not apt/apk - packages. `imgpush` needs `libmagickwand` (for `Wand`) and `nginx`; those must - come from the base image. Projects with system-library dependencies must pick - a base that already includes them — pymage can't `apt-get install`. -4. **No extras selection.** `imgpush`'s Dockerfile uses `uv sync --extra rembg`; - pymage installs only the default runtime closure (no opt-in extras), so it - can't reproduce that dependency set. *Possible fix: a `--extra` flag feeding - the closure's root extras.* +2. **(FIXED) sdist-only dependencies.** `imgpush` previously failed on + `timeout-decorator==0.5.0`, which publishes no wheel. pymage now builds a + wheel from the sdist (`pip wheel --no-deps`) and feeds it into the existing + layer path; the build host needs `python`/`pip`. Pure-python sdists build to + `py3-none-any` (any target); compiled sdists build for the host platform only + (cross-arch builds of those still error clearly). `imgpush` now builds with 52 + runtime wheels including the sdist-built `timeout-decorator`. +3. **No system/OS packages (by design — documented).** pymage installs Python + wheels, not apt/apk packages. `imgpush` needs `libmagickwand` (for `Wand`) and + `nginx`; those must come from the **base image**. Projects with system-library + dependencies must pick/compose a base that already includes them — pymage + can't `apt-get install`. See the README "Base image requirements" section. +4. **(FIXED) Extras selection.** `--extra ` enables a project's own + `[project.optional-dependencies]` group (e.g. `imgpush`'s `rembg`), feeding the + closure's roots. Extras requested by dependencies (`fastapi[standard]`) were + already followed. 5. **Defaulting to the base's platforms explodes on Docker Hub `python`.** That image advertises ~8 platforms (incl. `linux/arm/v7`, `linux/386`, `s390x`…), and pymage tried to build all of them, failing on `httptools` (no `cp312` `armv7l` wheel). Chainguard's base (amd64+arm64 only) is fine. *Possible fix: default the auto platform set to a curated common subset (amd64/arm64) or the host arch, and require explicit `--platform` - otherwise.* -6. **No workspace / `--package` model.** `full-stack-fastapi-template` is a uv - workspace; the Dockerfile builds one member (`--package app`). pymage treats - all local packages as roots and unions their runtime closures — which happens - to be correct here, but there's no way to target a single member or pass the - lock at a different path cleanly. -7. **Environment markers aren't evaluated** in the closure, so a few - marker-gated deps may be over-included. Bounded (wheel platform-filtering - catches incompatibilities), but a runtime dep gated to a non-target platform - with no compatible wheel would error. + otherwise.* (Marker evaluation, item 7, mitigates the *marker-gated* slice of + this; wheel availability for the arch is still required.) +6. **(FIXED) Workspace / `--package` model.** `full-stack-fastapi-template` is a + uv workspace; the Dockerfile builds one member (`uv sync --package app`). + `--package ` now roots the closure at a single workspace member; + omitting it unions all members (the prior behavior). +7. **(FIXED) Environment markers are now evaluated** per target. Dependency + markers (`sys_platform`, `platform_machine`, `python_version`, `os_name`, …, + with `and`/`or`/`in`/comparisons) are evaluated against the platform and + interpreter being built, so marker-gated deps are included only when they + apply (and per-arch for multi-arch builds). Unparseable markers conservatively + include the dependency rather than drop it. ## Verdict -After the runtime-closure fix, **pymage is a smaller, faster, reproducible -alternative to a uv Dockerfile for pure-wheel applications** (the FastAPI cases), -with no daemon and no build tooling in the image. It is **not yet a drop-in -replacement** for projects that need system libraries, depend on sdist-only -packages, or rely on optional extras — those are the highest-value items to -close next (sdist→wheel building and `--extra` support chief among them). +After the runtime-closure fix and this round of work, **pymage is a smaller, +faster, reproducible alternative to a uv Dockerfile** for the projects studied: +all three now build (including `imgpush`, via sdist building), with no daemon and +no build tooling in the image, and incremental rebuilds re-upload only changed +layers. The main remaining caveat is **runtime system libraries**, which must be +provided by the base image (documented), plus the multi-platform default for +bases that advertise many architectures (item 5).