Drops the no-execution sdist repacker and --repack-sdists flag. Building/repacking
sdists adds too much surface (correctness risk and complexity) for little gain
(~99.7% of real deps already have wheels). sdist-only dependencies continue to
fail fast with guidance to pre-build a wheel and pass --find-links. Can revisit
later if needed.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
Reconstructs a py3-none-any wheel from an sdist's STATIC contents only
(PKG-INFO -> METADATA; egg-info/top_level.txt or src/ layout for the file set;
egg-info or declarative pyproject for entry points) without running setup.py or
any PEP 517 hook -- no RCE vector. Conservative: rejects native sources, a
compiling build backend, declared data_files, version mismatch, or an ambiguous
layout, directing those to --find-links. Opt-in via --repack-sdists; one build
serves all arches (pure wheel). Validated against the real timeout-decorator
and iniconfig sdists.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
Removes the built-in closure walker (resolveUV/runtimeClosure/ParseUVLockFile/
ParseAny) entirely so there is no second, potentially divergent resolver. uv.lock
resolution now goes only through `uv export`; if uv is not on PATH the build
fails with an actionable error pointing at requirements.txt + --find-links.
uv.lock parsing is kept solely to attach wheel/sdist URLs.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
- sbom: normalize component names (PEP 503) for canonical pkg:pypi PURLs and stable ordering
- wheel: reject any member with a '..' path segment (catches escapes that resolve back under the prefix but out of site-packages)
- cache/wheelhouse: replace file atomically across platforms (os.Rename fails on existing dest on Windows)
- project: include requirements.lock in the no-lock-found error
- tests for each
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
Building sdists runs dependency build code on the host (RCE surface, no
sandbox), is non-hermetic/non-reproducible, needs a toolchain, and is
host-arch-only for compiled packages -- all at odds with pymage's guarantees.
Remove the builder and --build-sdists opt-in; keep sdist parsing only so the
error can explain the package is sdist-only and point at the --find-links
pre-build workflow.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
Previously pymage installed every package in uv.lock, including dev-dependency
groups and the whole resolution universe. On real projects this bloated images
badly: ruff alone was 50% of deps in astral-sh/uv-docker-example, and dev/CI
tools (ruff, mypy, ty, zizmor, ...) were 63% of deps in
fastapi/full-stack-fastapi-template.
ParseUVLockFile now computes the runtime closure from the local/project
packages' dependencies, transitively following dependencies and expanding
requested extras via optional-dependencies, never following dev-dependencies
(matching 'uv sync --no-dev'). Locks with no project package still install
everything. sdist-only errors now only trigger for packages actually in the
closure. Cuts deps 50-63% on the studied projects.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
Each dependency layer descriptor now carries a dev.pymage.wheels annotation
listing the wheel(s) it installs (comma-separated name==version), mirroring the
config-history description and correctly listing all wheels in packed
multi-wheel layers. Adds a test covering per-wheel and packed layers.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
Caching (layer blobs, downloaded wheels, base interpreter detection) is now
enabled by default at the per-user cache dir (or --cache-dir / $PYMAGE_CACHE_DIR).
--no-cache disables it: layer/meta caches off and wheels downloaded into an
ephemeral temp dir that's cleaned up after the build.
- setupCaches centralizes cache wiring (one cache.Cache serves layers + meta);
--no-cache uses a temp wheel dir with cleanup
- DefaultCacheDir honors $PYMAGE_CACHE_DIR; cli tests point it at a temp dir so
the on-by-default cache doesn't touch the user's cache
- tests: setupCaches (default + --no-cache cleanup); README/config updated
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
A no-op build's ~2-3s was almost all registry I/O:
- the base index was fetched 3x (BasePlatforms + remote.Image per platform)
- the base's top layer was downloaded once per platform every build to read
/etc/apko.json for interpreter detection (Chainguard has no PYTHON_VERSION)
Changes:
- build.BaseSet resolves the base reference once and serves per-platform child
images memoized (index fetched once; verified by TestBaseSet)
- cache the detected interpreter version by base digest in a default per-user
metadata cache (cache.GetText/PutText), so repeat builds skip the apko layer
download
- lazy wheel-cache dir creation; wheelhouse.Resolve defaults to the per-user
cache dir
Tests: TestBaseSet (index fetched once + per-platform memoized), TestCachedInterpreter
(write + cache-hit short-circuit), cache text round-trip.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
- src-layout PYTHONPATH is now derived from --workdir (project exposes SrcLayout;
CLI computes <workdir>/src) instead of hard-coding /app/src
- wheelhouse.Resolve defaults to the per-user wheel cache dir so lock-based
downloads work; cache dir creation is lazy so local-only resolution is hermetic
- --max-layers error message matches the accepted range (>= 0)
- DiscoverLock error lists requirements.lock too
- tests for the workdir-derived PYTHONPATH and python-prefix
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
- parsePythonTag now rejects tags without the 'python' prefix (e.g. '3.12'),
which would otherwise produce a bogus .../lib/3.12/site-packages layout
- parseConsoleScripts iterates with strings.Split instead of bufio.Scanner, so
a long line/large entry_points.txt can't silently truncate the launcher set
- tests for both
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
Replace per-tag remote.Write/WriteIndex with a single remote.NewPusher reused
for every tag. The pusher caches per-repo upload state, so identical blobs are
checked/uploaded only once even when pushing the same digest to multiple tags
(previously each tag re-HEADed every blob). Still uploads layers/index children
concurrently via WithJobs.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
go-containerregistry already uploads layers (and index children) concurrently,
but only at its default of 4. Set remote.WithJobs explicitly with an auto
default that scales with CPU count (floor 4), plus a --push-concurrency flag and
[tool.pymage] push-concurrency config to tune it.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
Default --layer-strategy is now 'auto'. Adds max-layers (default 127) and
max-wheel-layers config keys/flags with validation. Documents the layer budget
and reuse-stable bin packing in README and DESIGN.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
The new default 'auto' strategy keeps one layer per wheel while the total image
layer count (base + deps + app) fits a budget (default 127), and otherwise
bin-packs wheels into the budget by hashing each distribution's normalized name
to a stable bucket. A wheel's bucket depends only on its name, so adding,
removing, or version-bumping one dependency changes only that bucket's layer —
every other layer keeps its digest and is reused.
- MaxLayers / MaxWheelLayers options; base layer count + app layer reserved
- group layers built in parallel and content-addressed in the cache
- tests: packing, stable bucketing, budget from max-layers (base/app reserved),
clamp when base exceeds cap, and the reuse guarantees (add dep / version bump
change <= 1 layer), reproducibility
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
/abs/wheels isn't absolute on Windows (no drive letter), so it was joined to
the root and the assertion failed. Use a real absolute temp dir and a TOML
literal string so backslashes survive.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
- project: a [project.scripts] entry becomes a direct 'python -c "from mod
import fn; fn()"' (or 'python -m') invocation instead of the bare script
name, which pymage never installs as a launcher (fixes 'exec: "example":
executable file not found')
- build: PYTHONPATH is now additive (site-packages + base + app /app/src) so an
app PYTHONPATH no longer clobbers the venv site-packages (deps stayed
unimportable)
- tests for script-target translation and PYTHONPATH accumulation
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
When no platform is given, inspect the base reference: a multi-arch base index
yields a multi-arch build (filtering out attestation/unknown entries), a
single-arch base yields one image. Explicit --platform still overrides. Adds
build.BasePlatforms with tests + a CLI default-platforms test; documents it.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
- output() writes the single by-digest reference (repo@sha256:...) to stdout so
'docker run "$(pymage build)"' works; per-tag pointers and ggcr's per-blob
progress (pushed/existing/mounted blob) go to stderr via logs.Progress
- adds TestBuildStdoutIsImageRef asserting the stdout contract and that blob
logs land on stderr; documents it in the README
- completes and removes todo.txt
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
If --python is given it must match the base's detected Python version; if
omitted, the version is auto-detected (PYTHON_VERSION env, else apko.json) and
used for both wheel selection and the site-packages layout. Errors clearly when
the base version can't be determined and --python is unset. CI now relies on
auto-detection (drops --python). Adds resolveInterpreter tests and asserts the
auto-detected version flows into the image.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
When the base doesn't set PYTHON_VERSION (Chainguard/Wolfi images), fall back to
reading /etc/apko.json from the top-most layer and parsing the python-X.Y
package, so --python validation works for those bases too. Env still takes
precedence. Adds tests for the apko fallback and env precedence.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
Reads the base image's advertised PYTHON_VERSION (config env, no layer
download) and fails the build when it doesn't match --python, so a base tag
that slides to a new Python version can't silently produce a broken image.
Documents base-pinning tradeoffs in the README. Adds unit + CLI tests.
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
- --platform is now repeatable/comma-separated; >1 platform builds one image
per platform and assembles them into an OCI image index (remote.WriteIndex /
OCI layout). SBOM aggregates wheels across platforms.
- adds TestBuildMultiArchIndex (2-platform index, reproducible) against an
in-process registry; documents multi-arch in the README
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
- console-script targets like 'pkg.mod:main [extra]' no longer leak the
'[extra]' suffix into the generated launcher; module-less entry points are
skipped instead of producing invalid Python
- stripComment now strips from the first '#' (not only ' #') and trims trailing
whitespace, matching its docstring
- adds tests demonstrating both behaviors
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
- add internal/cache content-addressed layer store; build per-wheel layers in
parallel and reuse cached compressed blobs across rebuilds (--cache-dir)
- thread platform/python target into wheel resolution; validate --env
- add common secret patterns to the default source ignore list
- README: copy/paste-safe usage example + new flags; DESIGN status updated
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
- wheelhouse now filters candidates by target platform/ABI/python tag and
prefers the most specific compatible wheel (fixes wrong-arch selection); also
streams sha256 and uses an index map instead of an O(n*m) scan
- wheel: add tag parser + compatibility, cap per-member size (zip-bomb guard),
reject members that escape the install prefix, stable sort
- lock: error on non-pinned/URL/editable lines instead of silently dropping
Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
