mirror of
https://github.com/imjasonh/terraform-playground
synced 2026-07-18 23:20:18 +00:00
wheelhouse: don't build sdists; fail fast and direct to --find-links pre-built wheels
Building sdists runs dependency build code on the host (RCE surface, no sandbox), is non-hermetic/non-reproducible, needs a toolchain, and is host-arch-only for compiled packages -- all at odds with pymage's guarantees. Remove the builder and --build-sdists opt-in; keep sdist parsing only so the error can explain the package is sdist-only and point at the --find-links pre-build workflow. Co-authored-by: Jason Hall <imjasonh@users.noreply.github.com>
This commit is contained in:
parent
bf565be0ce
commit
971e955c35
9 changed files with 88 additions and 371 deletions
|
|
@ -25,8 +25,11 @@ because it exploits content-addressed layering:
|
|||
### Non-goals (initially)
|
||||
|
||||
- Compiling C extensions / building wheels from sdists. We consume **pre-built
|
||||
wheels** only. (sdist→wheel is a future extension; it would run a build, then
|
||||
the resulting wheel re-enters the normal layer path.)
|
||||
wheels** only. Building an sdist would run the dependency's own build code on
|
||||
the host — non-hermetic, non-reproducible, an RCE surface with no sandbox, and
|
||||
host-arch-only for compiled packages — so it is intentionally out of scope.
|
||||
When a lock has no compatible wheel, pymage fails fast and directs the user to
|
||||
pre-build a wheel out-of-band and supply it via `--find-links`.
|
||||
- Replacing dependency *resolution*. We delegate resolution to existing,
|
||||
correct tools (`uv` / `pip`) and consume their lockfile output.
|
||||
- A general-purpose Dockerfile interpreter. This is a focused Python app builder
|
||||
|
|
|
|||
|
|
@ -73,7 +73,6 @@ the config value, which overrides the built-in default.
|
|||
| `max-wheel-layers` | `--max-wheel-layers` | *(derived from `max-layers`)* |
|
||||
| `push-concurrency` | `--push-concurrency` | auto (≥ 4, scales with CPUs) |
|
||||
| `no-cache` | `--no-cache` | `false` (caching is on by default) |
|
||||
| `build-sdists` | `--build-sdists` | `false` (sdists are opt-in; see warning below) |
|
||||
| `extras` | `--extra` (repeatable) | — (enables uv project optional-dependency groups) |
|
||||
| `package` | `--package` | — (build a single uv workspace member) |
|
||||
| `python` | `--python` | auto-detected from the base |
|
||||
|
|
@ -167,34 +166,35 @@ get from `uv sync --no-dev`), not every package in the lock:
|
|||
Windows-only or stale-Python-only packages. Markers are evaluated per platform,
|
||||
so each arch of a multi-arch build gets the correct set.
|
||||
|
||||
### Source distributions (sdists) — opt-in
|
||||
### Source distributions (sdists)
|
||||
|
||||
By default pymage installs **only pre-built wheels**. If the lock pins a package
|
||||
with no compatible wheel, the build fails fast and tells you to either supply a
|
||||
wheel via `--find-links` or opt into sdist building.
|
||||
pymage installs **pre-built wheels only — it does not build sdists.** This is a
|
||||
deliberate choice: building a source distribution runs the dependency's own build
|
||||
code (`setup.py` / build backend) on the build host, which would break pymage's
|
||||
core guarantees — it's not hermetic or byte-reproducible, it's a remote-code-
|
||||
execution surface with no container to sandbox it, it needs a build toolchain,
|
||||
and compiled packages can only be built for the host architecture (no multi-arch).
|
||||
|
||||
Building from an sdist is **off by default** and must be enabled with
|
||||
`--build-sdists` (or `build-sdists = true` in `[tool.pymage]`). When enabled,
|
||||
pymage downloads the hash-verified sdist and builds a wheel with the host's `pip`
|
||||
(`pip wheel --no-deps`), which requires a Python toolchain (`python3`/`python`
|
||||
with `pip`) on the build host. It is a power-user feature with two important
|
||||
caveats:
|
||||
In practice this is rarely an issue: the modern ecosystem is wheel-first, so
|
||||
mainstream dependencies on common targets all publish wheels. You only hit an
|
||||
sdist for (a) older, low-maintenance pure-python packages that never uploaded a
|
||||
wheel, (b) compiled packages on a brand-new Python or uncommon platform before
|
||||
wheels are published, or (c) the occasional source-only package.
|
||||
|
||||
- **Security: building an sdist runs arbitrary code from the dependency** on the
|
||||
build host. An sdist's `setup.py` / build backend executes during the build,
|
||||
and — unlike a `docker build` — pymage does **not** sandbox it. A malicious or
|
||||
compromised dependency can therefore achieve code execution on your build
|
||||
machine. Only enable this for locks you trust, ideally in an ephemeral/CI
|
||||
environment, and prefer pre-built wheels (`--find-links`, or a registry that
|
||||
publishes wheels) whenever possible.
|
||||
- **Architecture: compiled sdists are single-arch.** A pure-python sdist builds
|
||||
to a `py3-none-any` wheel that works on any target. A *compiled* sdist can only
|
||||
be built for the **host** platform, so a multi-arch build that needs to build
|
||||
such a package will fail the compatibility check for the non-host arch. Use a
|
||||
base/lock that provides pre-built wheels for every target arch instead.
|
||||
If the lock pins a package with no compatible wheel, the build fails fast and
|
||||
tells you exactly how to proceed: **pre-build the wheel out-of-band with your own
|
||||
(trusted, ideally isolated) tooling and point `--find-links` at it.**
|
||||
|
||||
Built wheels are cached (keyed by the sdist hash and target) so they aren't
|
||||
rebuilt on subsequent builds.
|
||||
```
|
||||
# Build wheels once, with isolation/tooling you control:
|
||||
uv pip wheel -r requirements.txt -w ./wheelhouse # or: pip wheel ...
|
||||
|
||||
# Then build the image from the local wheelhouse:
|
||||
pymage build --find-links ./wheelhouse -t latest
|
||||
```
|
||||
|
||||
This keeps the escape hatch for the rare cases while keeping pymage's builds
|
||||
hermetic, reproducible, multi-arch, and free of arbitrary build-time code.
|
||||
|
||||
### Base image requirements (OS / system libraries)
|
||||
|
||||
|
|
@ -255,7 +255,6 @@ base that advertises its version.
|
|||
| `--package` | Build a single uv workspace member by name (default: union of all members). |
|
||||
| `--cache-dir` | Cache root (default: `$PYMAGE_CACHE_DIR` or the per-user cache dir). Caches compressed layers, downloaded wheels, and base interpreter detection. |
|
||||
| `--no-cache` | Disable all caching (layers, downloaded wheels, interpreter detection). |
|
||||
| `--build-sdists` | Allow building wheels from sdists when no compatible wheel exists. **Runs the dependency's build code on the host (no sandbox); single-arch for compiled packages.** Off by default. |
|
||||
| `--prefix` | install prefix / venv root (default `/app/.venv`). |
|
||||
| `--workdir` | image working dir and source destination (default `/app`). |
|
||||
| `--user` | image user, e.g. `65532`. |
|
||||
|
|
|
|||
|
|
@ -3,9 +3,9 @@
|
|||
This is a hands-on study of migrating real, uv-based Python projects that ship a
|
||||
`Dockerfile` to `pymage`. The goals were to (a) see how the resulting images
|
||||
differ and (b) surface gaps/bugs real projects hit. It directly drove several
|
||||
fixes — runtime-only dependency resolution, sdist→wheel building, `--extra` /
|
||||
`--package` selection, and environment-marker evaluation — plus a ranked list of
|
||||
the remaining migration blockers.
|
||||
fixes — runtime-only dependency resolution, `--extra` / `--package` selection,
|
||||
and environment-marker evaluation — plus a ranked list of the remaining
|
||||
migration blockers.
|
||||
|
||||
## Method & caveats
|
||||
|
||||
|
|
@ -123,21 +123,18 @@ the worst case is one bucket rather than the whole environment.)
|
|||
|
||||
1. **(FIXED) Installed the whole `uv.lock`, including dev groups.** Caused
|
||||
50–63% dependency bloat above. Now resolves the runtime closure.
|
||||
2. **(FIXED, opt-in) sdist-only dependencies.** `imgpush` previously failed on
|
||||
`timeout-decorator==0.5.0`, which publishes no wheel. With `--build-sdists`
|
||||
(or `build-sdists` in `[tool.pymage]`) pymage builds a wheel from the sdist
|
||||
(`pip wheel --no-deps`, host `python`/`pip` required) and feeds it into the
|
||||
existing layer path; `imgpush` then builds with 52 runtime wheels including
|
||||
the sdist-built `timeout-decorator`. It is **off by default and intentionally
|
||||
a power-user feature** for two reasons:
|
||||
- **Security:** building an sdist runs the dependency's own build code on the
|
||||
host with no container isolation (pymage has no daemon to sandbox it), so a
|
||||
malicious dependency could get code execution on the build machine. Without
|
||||
the flag, a wheelless package fails fast with guidance to supply a wheel via
|
||||
`--find-links` or opt in.
|
||||
- **Single-arch:** compiled sdists can only be built for the host platform, so
|
||||
enabling them doesn't make a compiled package multi-arch — pure-python
|
||||
sdists build to `py3-none-any` and remain portable.
|
||||
2. **sdist-only dependencies (by design — not built).** `imgpush` pins
|
||||
`timeout-decorator==0.5.0`, which publishes no wheel. pymage installs
|
||||
pre-built wheels only and **does not build sdists**: doing so would run the
|
||||
dependency's build code on the host (RCE surface, no sandbox), be
|
||||
non-reproducible and non-hermetic, need a build toolchain, and produce
|
||||
host-arch-only output for compiled packages — all at odds with pymage's
|
||||
guarantees. Instead the build fails fast and directs you to pre-build the
|
||||
wheel out-of-band with tooling you control and pass `--find-links`:
|
||||
`uv pip wheel timeout-decorator==0.5.0 -w ./wheelhouse` (or `pip wheel`), then
|
||||
`pymage build --find-links ./wheelhouse …`. This is rare in practice (the
|
||||
ecosystem is wheel-first); `timeout-decorator` was the only sdist across all
|
||||
four projects studied.
|
||||
3. **No system/OS packages (by design — documented).** pymage installs Python
|
||||
wheels, not apt/apk packages. `imgpush` needs `libmagickwand` (for `Wand`) and
|
||||
`nginx`; those must come from the **base image**. Projects with system-library
|
||||
|
|
@ -169,9 +166,12 @@ the worst case is one bucket rather than the whole environment.)
|
|||
## Verdict
|
||||
|
||||
After the runtime-closure fix and this round of work, **pymage is a smaller,
|
||||
faster, reproducible alternative to a uv Dockerfile** for the projects studied:
|
||||
all three now build (including `imgpush`, via opt-in sdist building), with no daemon and
|
||||
no build tooling in the image, and incremental rebuilds re-upload only changed
|
||||
layers. The main remaining caveat is **runtime system libraries**, which must be
|
||||
provided by the base image (documented), plus the multi-platform default for
|
||||
faster, reproducible alternative to a uv Dockerfile** for wheel-based apps, with
|
||||
no daemon and no build tooling in the image, and incremental rebuilds that
|
||||
re-upload only changed layers. The FastAPI projects build directly; `imgpush`
|
||||
builds once its one sdist-only dependency (`timeout-decorator`) is pre-built into
|
||||
a `--find-links` wheelhouse, and still needs a base that ships ImageMagick at
|
||||
runtime. The remaining caveats are **runtime system libraries** (must come from
|
||||
the base image — documented), sdist-only dependencies (pre-build wheels
|
||||
out-of-band; pymage won't run build code), and the multi-platform default for
|
||||
bases that advertise many architectures (item 5).
|
||||
|
|
|
|||
|
|
@ -72,7 +72,6 @@ type buildFlags struct {
|
|||
insecure bool
|
||||
cacheDir string
|
||||
noCache bool
|
||||
buildSdists bool
|
||||
}
|
||||
|
||||
func buildCmd() *cobra.Command {
|
||||
|
|
@ -120,7 +119,6 @@ func buildCmd() *cobra.Command {
|
|||
fs.BoolVar(&f.insecure, "insecure", false, "use plain HTTP for the registry")
|
||||
fs.StringVar(&f.cacheDir, "cache-dir", "", "cache root directory (default: per-user cache dir)")
|
||||
fs.BoolVar(&f.noCache, "no-cache", false, "disable all build caches (layers, downloaded wheels, interpreter detection)")
|
||||
fs.BoolVar(&f.buildSdists, "build-sdists", false, "allow building wheels from source distributions when no compatible wheel exists (runs the dependency's build code on this host; single-arch for compiled packages)")
|
||||
return cmd
|
||||
}
|
||||
|
||||
|
|
@ -258,7 +256,7 @@ func buildOne(ctx context.Context, f *buildFlags, lk *lock.Lock, platform *v1.Pl
|
|||
return nil, nil, err
|
||||
}
|
||||
|
||||
wheels, err := wheelhouse.ResolveContext(ctx, reqs, f.findLinks, target, wheelCache, f.buildSdists)
|
||||
wheels, err := wheelhouse.ResolveContext(ctx, reqs, f.findLinks, target, wheelCache)
|
||||
if err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
|
|
|
|||
|
|
@ -75,9 +75,6 @@ func applyDefaults(cmd *cobra.Command, f *buildFlags) error {
|
|||
if !changed("package") && cfg.Package != "" {
|
||||
f.pkg = cfg.Package
|
||||
}
|
||||
if !changed("build-sdists") && cfg.BuildSdists {
|
||||
f.buildSdists = true
|
||||
}
|
||||
if !changed("python") && cfg.Python != "" {
|
||||
f.pythonTag = cfg.Python
|
||||
}
|
||||
|
|
|
|||
|
|
@ -48,7 +48,6 @@ type Config struct {
|
|||
NoCache bool `toml:"no-cache"`
|
||||
Extras []string `toml:"extras"`
|
||||
Package string `toml:"package"`
|
||||
BuildSdists bool `toml:"build-sdists"`
|
||||
}
|
||||
|
||||
const defaultBase = "cgr.dev/chainguard/python:latest"
|
||||
|
|
|
|||
|
|
@ -1,222 +0,0 @@
|
|||
package wheelhouse
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"fmt"
|
||||
"io"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/imjasonh/terraform-playground/pymage/internal/lock"
|
||||
"github.com/imjasonh/terraform-playground/pymage/internal/wheel"
|
||||
)
|
||||
|
||||
// buildFromSdist builds a wheel from a requirement's source distribution and
|
||||
// returns it as a ResolvedWheel. The built wheel is cached (keyed by the sdist
|
||||
// hash and target) so it isn't rebuilt on subsequent builds.
|
||||
//
|
||||
// This is only reached when the caller opted in (allowSdist), because building
|
||||
// runs the dependency's own build code on the host with no isolation. Building
|
||||
// runs the host's `pip wheel`, so a Python toolchain must be present.
|
||||
// A pure-python sdist yields a `py3-none-any` wheel usable on any target; a
|
||||
// compiled sdist is built for the host platform only, so cross-arch builds of
|
||||
// such packages will fail the compatibility check with a clear error.
|
||||
func buildFromSdist(ctx context.Context, req lock.Requirement, target wheel.Target, cache *wheelCache) (ResolvedWheel, error) {
|
||||
if cache == nil {
|
||||
return ResolvedWheel{}, fmt.Errorf("wheelhouse: building %s==%s from sdist requires a cache dir (or use --no-cache)", req.Name, req.Version)
|
||||
}
|
||||
key := req.Sdist.SHA256 + "-" + targetKey(target)
|
||||
if p, ok := cache.getBuilt(key); ok {
|
||||
sum, err := fileSHA256(p)
|
||||
if err != nil {
|
||||
return ResolvedWheel{}, err
|
||||
}
|
||||
return ResolvedWheel{Name: req.Name, Version: req.Version, Path: p, SHA256: sum}, nil
|
||||
}
|
||||
|
||||
work, err := os.MkdirTemp("", "pymage-sdist-")
|
||||
if err != nil {
|
||||
return ResolvedWheel{}, err
|
||||
}
|
||||
defer func() { _ = os.RemoveAll(work) }()
|
||||
|
||||
sdistPath := filepath.Join(work, req.Sdist.Filename)
|
||||
if err := downloadVerify(ctx, req.Sdist.URL, req.Sdist.SHA256, sdistPath); err != nil {
|
||||
return ResolvedWheel{}, fmt.Errorf("wheelhouse: %s==%s sdist: %w", req.Name, req.Version, err)
|
||||
}
|
||||
|
||||
out := filepath.Join(work, "out")
|
||||
if err := os.MkdirAll(out, 0o755); err != nil {
|
||||
return ResolvedWheel{}, err
|
||||
}
|
||||
if err := buildWheel(ctx, sdistPath, out); err != nil {
|
||||
return ResolvedWheel{}, fmt.Errorf("wheelhouse: build %s==%s from sdist: %w", req.Name, req.Version, err)
|
||||
}
|
||||
|
||||
built, err := selectBuiltWheel(out, req, target)
|
||||
if err != nil {
|
||||
return ResolvedWheel{}, err
|
||||
}
|
||||
|
||||
dst := cache.builtPath(key)
|
||||
if err := os.MkdirAll(filepath.Dir(dst), 0o755); err != nil {
|
||||
return ResolvedWheel{}, err
|
||||
}
|
||||
if err := os.Rename(built, dst); err != nil {
|
||||
if err := copyFile(built, dst); err != nil {
|
||||
return ResolvedWheel{}, err
|
||||
}
|
||||
}
|
||||
sum, err := fileSHA256(dst)
|
||||
if err != nil {
|
||||
return ResolvedWheel{}, err
|
||||
}
|
||||
return ResolvedWheel{Name: req.Name, Version: req.Version, Path: dst, SHA256: sum}, nil
|
||||
}
|
||||
|
||||
func targetKey(t wheel.Target) string {
|
||||
return fmt.Sprintf("%s-%s-cp%d%d", t.OS, t.Arch, t.PyMajor, t.PyMinor)
|
||||
}
|
||||
|
||||
func (c *wheelCache) builtPath(key string) string {
|
||||
return filepath.Join(c.dir, "built", key+".whl")
|
||||
}
|
||||
|
||||
func (c *wheelCache) getBuilt(key string) (string, bool) {
|
||||
if c == nil {
|
||||
return "", false
|
||||
}
|
||||
if _, err := os.Stat(c.builtPath(key)); err == nil {
|
||||
return c.builtPath(key), true
|
||||
}
|
||||
return "", false
|
||||
}
|
||||
|
||||
// buildWheel builds a wheel from an sdist into outDir using `pip wheel`.
|
||||
func buildWheel(ctx context.Context, sdist, outDir string) error {
|
||||
py := findPython()
|
||||
if py == "" {
|
||||
return fmt.Errorf("no python3/python on PATH to build the sdist (install Python+pip, supply a pre-built wheel via --find-links, or use a lock with wheels)")
|
||||
}
|
||||
cmd := exec.CommandContext(ctx, py, "-m", "pip", "wheel", "--no-deps", "--no-cache-dir", "--wheel-dir", outDir, sdist)
|
||||
if out, err := cmd.CombinedOutput(); err != nil {
|
||||
return fmt.Errorf("pip wheel: %w\n%s", err, tail(string(out), 20))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func findPython() string {
|
||||
for _, name := range []string{"python3", "python"} {
|
||||
if p, err := exec.LookPath(name); err == nil {
|
||||
return p
|
||||
}
|
||||
}
|
||||
return ""
|
||||
}
|
||||
|
||||
// selectBuiltWheel returns the built wheel in dir matching the requirement and
|
||||
// compatible with the target.
|
||||
func selectBuiltWheel(dir string, req lock.Requirement, target wheel.Target) (string, error) {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
var compatible []candidate
|
||||
for _, e := range entries {
|
||||
if e.IsDir() || !strings.HasSuffix(e.Name(), ".whl") {
|
||||
continue
|
||||
}
|
||||
tags, err := wheel.ParseTags(e.Name())
|
||||
if err != nil {
|
||||
continue
|
||||
}
|
||||
if lock.NormalizeName(tags.Name) != lock.NormalizeName(req.Name) || tags.Version != req.Version {
|
||||
continue
|
||||
}
|
||||
if !tags.CompatibleWith(target) {
|
||||
continue
|
||||
}
|
||||
compatible = append(compatible, candidate{path: filepath.Join(dir, e.Name()), tags: tags})
|
||||
}
|
||||
if len(compatible) == 0 {
|
||||
return "", fmt.Errorf("wheelhouse: built wheel for %s==%s is not compatible with %s/%s python%d.%d (a compiled package can only be built for the host platform)",
|
||||
req.Name, req.Version, target.OS, target.Arch, target.PyMajor, target.PyMinor)
|
||||
}
|
||||
return pickBest(compatible).path, nil
|
||||
}
|
||||
|
||||
func downloadVerify(ctx context.Context, url, wantSum, dest string) error {
|
||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, url, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
client := &http.Client{Timeout: 5 * time.Minute}
|
||||
resp, err := client.Do(req)
|
||||
if err != nil {
|
||||
return fmt.Errorf("download %s: %w", url, err)
|
||||
}
|
||||
defer func() { _ = resp.Body.Close() }()
|
||||
if resp.StatusCode != http.StatusOK {
|
||||
return fmt.Errorf("download %s: HTTP %s", url, resp.Status)
|
||||
}
|
||||
f, err := os.Create(dest)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
h := sha256.New()
|
||||
if _, err := io.Copy(io.MultiWriter(f, h), resp.Body); err != nil {
|
||||
_ = f.Close()
|
||||
return err
|
||||
}
|
||||
if err := f.Close(); err != nil {
|
||||
return err
|
||||
}
|
||||
if got := hex.EncodeToString(h.Sum(nil)); !strings.EqualFold(got, wantSum) {
|
||||
return fmt.Errorf("sdist hash %s != expected %s", got, wantSum)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func fileSHA256(path string) (string, error) {
|
||||
f, err := os.Open(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer func() { _ = f.Close() }()
|
||||
h := sha256.New()
|
||||
if _, err := io.Copy(h, f); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return hex.EncodeToString(h.Sum(nil)), nil
|
||||
}
|
||||
|
||||
func copyFile(src, dst string) error {
|
||||
in, err := os.Open(src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer func() { _ = in.Close() }()
|
||||
out, err := os.Create(dst)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := io.Copy(out, in); err != nil {
|
||||
_ = out.Close()
|
||||
return err
|
||||
}
|
||||
return out.Close()
|
||||
}
|
||||
|
||||
func tail(s string, n int) string {
|
||||
lines := strings.Split(strings.TrimRight(s, "\n"), "\n")
|
||||
if len(lines) > n {
|
||||
lines = lines[len(lines)-n:]
|
||||
}
|
||||
return strings.Join(lines, "\n")
|
||||
}
|
||||
|
|
@ -2,8 +2,6 @@ package wheelhouse
|
|||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
|
|
@ -11,56 +9,10 @@ import (
|
|||
"github.com/imjasonh/terraform-playground/pymage/internal/wheel"
|
||||
)
|
||||
|
||||
func writeFiles(t *testing.T, dir string, names ...string) {
|
||||
t.Helper()
|
||||
for _, n := range names {
|
||||
if err := os.WriteFile(filepath.Join(dir, n), []byte("x"), 0o644); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestSelectBuiltWheelPurePython(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeFiles(t, dir, "timeout_decorator-0.5.0-py3-none-any.whl", "notes.txt")
|
||||
req := lock.Requirement{Name: "timeout-decorator", Version: "0.5.0"}
|
||||
target := wheel.Target{OS: "linux", Arch: "arm64", PyMajor: 3, PyMinor: 12}
|
||||
|
||||
got, err := selectBuiltWheel(dir, req, target)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if filepath.Base(got) != "timeout_decorator-0.5.0-py3-none-any.whl" {
|
||||
t.Fatalf("got %q", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSelectBuiltWheelIncompatibleCompiled(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
// A compiled wheel built for the host (linux x86_64) when targeting arm64.
|
||||
writeFiles(t, dir, "ujson-5.0.0-cp312-cp312-manylinux_2_17_x86_64.whl")
|
||||
req := lock.Requirement{Name: "ujson", Version: "5.0.0"}
|
||||
target := wheel.Target{OS: "linux", Arch: "arm64", PyMajor: 3, PyMinor: 12}
|
||||
|
||||
if _, err := selectBuiltWheel(dir, req, target); err == nil {
|
||||
t.Fatal("expected incompatibility error for cross-arch compiled wheel")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSelectBuiltWheelWrongVersion(t *testing.T) {
|
||||
dir := t.TempDir()
|
||||
writeFiles(t, dir, "timeout_decorator-0.4.0-py3-none-any.whl")
|
||||
req := lock.Requirement{Name: "timeout-decorator", Version: "0.5.0"}
|
||||
target := wheel.Target{OS: "linux", Arch: "amd64", PyMajor: 3, PyMinor: 12}
|
||||
|
||||
if _, err := selectBuiltWheel(dir, req, target); err == nil {
|
||||
t.Fatal("expected error: built wheel version mismatch")
|
||||
}
|
||||
}
|
||||
|
||||
func TestSdistRequiresOptIn(t *testing.T) {
|
||||
// An sdist-only requirement (no wheels) must error unless opted in, rather
|
||||
// than silently building (which runs the dependency's build code).
|
||||
// pymage does not build sdists: a requirement available only as a source
|
||||
// distribution must error with guidance to pre-build a wheel and use
|
||||
// --find-links, rather than executing the dependency's build code.
|
||||
func TestSdistOnlyErrors(t *testing.T) {
|
||||
req := lock.Requirement{
|
||||
Name: "timeout-decorator",
|
||||
Version: "0.5.0",
|
||||
|
|
@ -68,21 +20,13 @@ func TestSdistRequiresOptIn(t *testing.T) {
|
|||
}
|
||||
target := wheel.Target{OS: "linux", Arch: "amd64", PyMajor: 3, PyMinor: 12}
|
||||
|
||||
_, err := ResolveContext(context.Background(), []lock.Requirement{req}, nil, target, t.TempDir(), false)
|
||||
_, err := ResolveContext(context.Background(), []lock.Requirement{req}, nil, target, t.TempDir())
|
||||
if err == nil {
|
||||
t.Fatal("expected error when sdist build is not opted in")
|
||||
t.Fatal("expected an error for an sdist-only requirement")
|
||||
}
|
||||
if !strings.Contains(err.Error(), "--build-sdists") {
|
||||
t.Fatalf("error should point at the opt-in flag, got: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestBuiltPathKey(t *testing.T) {
|
||||
c := &wheelCache{dir: "/tmp/c"}
|
||||
target := wheel.Target{OS: "linux", Arch: "amd64", PyMajor: 3, PyMinor: 12}
|
||||
key := "abc123" + "-" + targetKey(target)
|
||||
want := filepath.Join("/tmp/c", "built", key+".whl")
|
||||
if got := c.builtPath(key); got != want {
|
||||
t.Fatalf("builtPath = %q, want %q", got, want)
|
||||
for _, want := range []string{"source distribution", "--find-links", "timeout-decorator"} {
|
||||
if !strings.Contains(err.Error(), want) {
|
||||
t.Errorf("error %q should mention %q", err.Error(), want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -47,7 +47,7 @@ type candidate struct {
|
|||
// per-user cache directory. Results are sorted by (normalized name, version)
|
||||
// for deterministic ordering.
|
||||
func Resolve(reqs []lock.Requirement, dirs []string, target wheel.Target) ([]ResolvedWheel, error) {
|
||||
return ResolveContext(context.Background(), reqs, dirs, target, defaultWheelCacheDir(), false)
|
||||
return ResolveContext(context.Background(), reqs, dirs, target, defaultWheelCacheDir())
|
||||
}
|
||||
|
||||
// defaultWheelCacheDir returns the per-user wheel download cache, or "" if the
|
||||
|
|
@ -62,10 +62,8 @@ func defaultWheelCacheDir() string {
|
|||
}
|
||||
|
||||
// ResolveContext is Resolve with an explicit context and on-disk wheel cache
|
||||
// directory for lock-file downloads. When allowSdist is true, packages with no
|
||||
// compatible wheel may be built from their source distribution (which executes
|
||||
// the dependency's build code on this host); otherwise such packages error.
|
||||
func ResolveContext(ctx context.Context, reqs []lock.Requirement, dirs []string, target wheel.Target, wheelCacheDir string, allowSdist bool) ([]ResolvedWheel, error) {
|
||||
// directory for lock-file downloads.
|
||||
func ResolveContext(ctx context.Context, reqs []lock.Requirement, dirs []string, target wheel.Target, wheelCacheDir string) ([]ResolvedWheel, error) {
|
||||
var index map[string][]candidate
|
||||
if len(dirs) > 0 {
|
||||
var err error
|
||||
|
|
@ -82,7 +80,7 @@ func ResolveContext(ctx context.Context, reqs []lock.Requirement, dirs []string,
|
|||
|
||||
out := make([]ResolvedWheel, 0, len(reqs))
|
||||
for _, req := range reqs {
|
||||
rw, err := resolveOne(ctx, req, index, target, cache, allowSdist)
|
||||
rw, err := resolveOne(ctx, req, index, target, cache)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
|
@ -99,7 +97,7 @@ func ResolveContext(ctx context.Context, reqs []lock.Requirement, dirs []string,
|
|||
return out, nil
|
||||
}
|
||||
|
||||
func resolveOne(ctx context.Context, req lock.Requirement, index map[string][]candidate, target wheel.Target, cache *wheelCache, allowSdist bool) (ResolvedWheel, error) {
|
||||
func resolveOne(ctx context.Context, req lock.Requirement, index map[string][]candidate, target wheel.Target, cache *wheelCache) (ResolvedWheel, error) {
|
||||
key := lock.NormalizeName(req.Name) + "\x00" + req.Version
|
||||
if index != nil {
|
||||
if cands := index[key]; len(cands) > 0 {
|
||||
|
|
@ -127,32 +125,33 @@ func resolveOne(ctx context.Context, req lock.Requirement, index map[string][]ca
|
|||
}
|
||||
}
|
||||
|
||||
if len(req.Wheels) == 0 && req.Sdist == nil {
|
||||
if len(req.Wheels) == 0 {
|
||||
if req.Sdist != nil {
|
||||
// Distributed only as an sdist (no wheel anywhere). pymage installs
|
||||
// pre-built wheels only; see errSdistOnly.
|
||||
return ResolvedWheel{}, errSdistOnly(req, target)
|
||||
}
|
||||
return ResolvedWheel{}, fmt.Errorf("wheelhouse: no wheel found for %s==%s (pass --find-links or use a uv.lock with wheel URLs)", req.Name, req.Version)
|
||||
}
|
||||
if cache == nil {
|
||||
return ResolvedWheel{}, fmt.Errorf("wheelhouse: %s==%s not in local wheelhouse and no wheel cache dir configured", req.Name, req.Version)
|
||||
}
|
||||
|
||||
// Prefer a pre-built lock wheel; fall back to building from the sdist.
|
||||
if len(req.Wheels) > 0 {
|
||||
rw, err := fetchRequirement(ctx, req, target, cache)
|
||||
if err == nil {
|
||||
return rw, nil
|
||||
}
|
||||
if req.Sdist == nil {
|
||||
return ResolvedWheel{}, err
|
||||
}
|
||||
// No compatible lock wheel; consider building from the sdist below.
|
||||
rw, err := fetchRequirement(ctx, req, target, cache)
|
||||
if err == nil {
|
||||
return rw, nil
|
||||
}
|
||||
if req.Sdist == nil {
|
||||
return ResolvedWheel{}, err
|
||||
}
|
||||
// The lock has wheels but none compatible with this target, and the only
|
||||
// other artifact is an sdist, which pymage does not build.
|
||||
return ResolvedWheel{}, errSdistOnly(req, target)
|
||||
}
|
||||
|
||||
// Building from an sdist runs the dependency's own build code (setup.py /
|
||||
// build backend) on this host with no container isolation, so it is gated
|
||||
// behind explicit opt-in.
|
||||
if !allowSdist {
|
||||
return ResolvedWheel{}, fmt.Errorf("wheelhouse: %s==%s has no compatible pre-built wheel and would have to be built from its source distribution, which runs arbitrary build code from the dependency on this machine; pass --build-sdists (or set build-sdists in [tool.pymage]) to allow it, or supply a pre-built wheel via --find-links", req.Name, req.Version)
|
||||
}
|
||||
return buildFromSdist(ctx, req, target, cache)
|
||||
func errSdistOnly(req lock.Requirement, target wheel.Target) error {
|
||||
return fmt.Errorf("wheelhouse: %s==%s has no compatible pre-built wheel for %s/%s python%d.%d; it is only available as a source distribution (sdist), which pymage does not build. Pre-build a wheel out-of-band (e.g. `uv pip wheel %s==%s -w ./wheelhouse` or `pip wheel %s==%s -w ./wheelhouse`) and pass --find-links ./wheelhouse",
|
||||
req.Name, req.Version, target.OS, target.Arch, target.PyMajor, target.PyMinor, req.Name, req.Version, req.Name, req.Version)
|
||||
}
|
||||
|
||||
// pickBest chooses the most specific compatible wheel: platform-specific and
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue