krust/CLAUDE.md

krust Development Notes

This document captures key learnings and decisions made during the development of krust with Claude.

Project Overview

krust is a container image build tool for Rust applications, inspired by ko.build for Go. It builds static binaries and packages them into minimal OCI container images without requiring Docker.

Key Design Decisions

1. Static Binaries with musl

We chose musl libc over glibc for static linking because:

• True static linking: glibc uses dynamic loading internally (NSS) which breaks in static binaries
• Smaller binaries: musl static binaries are 5-10x smaller than glibc
• No runtime surprises: glibc static binaries often fail with DNS resolution, user lookups, or locale issues
• Container-optimized: Perfect for minimal container images

2. Default Push Behavior

krust pushes images by default (use --no-push to skip) because:

• Aligns with the common workflow of building and immediately using images
• Enables the docker run $(krust build) pattern
• Reduces friction for the most common use case

3. Output Design

• stdout: Only the pushed image reference by digest (e.g., ttl.sh/user/app@sha256:...)
• stderr: All logging and progress information
• This enables composability with other tools

4. Image Naming Strategy

• Uses KRUST_REPO environment variable for repository prefix
• Automatically appends project name from Cargo.toml
• Can be overridden with --image flag
• Default tag is latest

Technical Learnings

OCI Image Building

1. Layer Digest vs Diff ID:

   • Layer digest: SHA256 of the compressed (gzip) layer
   • Diff ID: SHA256 of the uncompressed tar (goes in image config)
   • Docker validates these match during pull

2. Image Structure:

   Manifest -> Config + Layers
   Config contains: architecture, OS, environment, command, diff_ids
   Layers contain: compressed tar.gz files
   

3. Registry API:

   • Push blobs (config and layers) first
   • Then push manifest referencing those blobs
   • Manifest URL contains the final digest

Google Artifact Registry (GAR) Blob Uploads

GAR has special handling for blob uploads that differs from the standard OCI spec:

1. Location Header Format:

   • POST to /v2/.../blobs/uploads/ returns location: /artifacts-uploads/...
   • The location is a relative path starting with /artifacts-uploads/, NOT /v2/
   • Must build absolute URL as https://{registry}{location} for ANY path starting with /

2. Upload Flow (Resumable):

   • Monolithic upload not supported: PUT with body to location?digest= returns 301 Moved Permanently
   • Must use resumable upload flow instead:
     1. POST to /v2/.../blobs/uploads/ → get upload location
     2. PATCH to upload location with blob data → returns 202 Accepted
     3. PUT to finalize location with ?digest= and empty body → returns 201 Created

3. Critical Implementation Details:

   • GAR returns 301 redirect on monolithic PUT attempts (not 307)
   • HTTP spec says 301 means don't resend body, so automatic redirect following fails
   • Must explicitly handle 301 as a signal to switch to resumable upload
   • PATCH response may include a new location header for the finalize PUT
   • Use reqwest with redirect::Policy::none() to handle redirects manually

4. Why reqwest over hyper:

   • Initially used raw hyper but it doesn't auto-follow redirects with request bodies
   • Switched to reqwest for cleaner API and better redirect handling
   • Disabled automatic redirects to manually handle GAR's upload flow

5. Memory Efficiency:

   • Blob downloads return bytes::Bytes instead of Vec<u8> to avoid unnecessary copies
   • Blob uploads require .to_vec() due to reqwest's 'static requirement for request bodies
   • This is acceptable as reqwest streams the data internally

Cross-Compilation

krust requires cargo-zigbuild for cross-compilation. This eliminates the need for per-target system linkers and .cargo/config.toml linker configuration. If zigbuild is not available, krust fails with install instructions.

Required targets are auto-installed via rustup target add when needed.

Rust Static Linking

• Use RUSTFLAGS="-C target-feature=+crt-static" for static linking
• musl targets default to static, but explicit is better
• The resulting binary has no runtime dependencies

Architecture Decisions

Module Structure

src/
├── main.rs          # CLI entry point and orchestration
├── lib.rs           # Public API exports
├── cli/             # Command-line interface definitions
├── builder/         # Rust compilation logic
├── image/           # OCI image construction
├── registry/        # Registry push operations
└── config/          # Configuration management

Error Handling

• Used anyhow for error propagation with context
• Errors include contextual information for debugging
• All errors go to stderr, preserving stdout for output

Dependencies

Key crates chosen:

• clap - CLI parsing with derive macros
• tokio - Async runtime for registry operations
• reqwest - HTTP client with automatic redirect handling
• tar + flate2 - Layer creation
• sha256 - Digest calculation
• tracing - Structured logging
• cargo-zigbuild - Cross-compilation backend (external tool, not a crate dep)

Testing Strategy

1. Unit tests for each module
2. Integration tests for CLI commands
3. E2E tests that actually run the built binary
4. Used assert_cmd for testing CLI behavior

Development Workflow

The iterative development process:

1. Start with basic CLI structure
2. Implement core functionality (build, image, push)
3. Test with real registries (ttl.sh for anonymous push)
4. Fix issues discovered during real usage
5. Refine UX based on actual workflows

Pre-commit Checks

Before committing changes, always run:

make check-fmt  # Check code formatting
make lint       # Run clippy linter
make test       # Run all tests

Or run all checks at once:

make check  # Runs check-fmt, lint, and test

Features Implemented

YAML Resolution (krust resolve)

Inspired by ko's Kubernetes integration, krust can resolve krust:// references in YAML files:

1. Reference Syntax: Use krust://path/to/project in YAML (e.g., image: krust://./example/hello-krust)
2. Deduplication: Multiple references to the same path are deduplicated - builds only once
3. Multi-document support: Handles YAML files with multiple --- separated documents
4. Directory support: Can process entire directories of YAML files with -f ./k8s/
5. Output: Resolved YAML to stdout with all krust:// replaced by digests

Implementation details:

• Uses serde_yml (maintained fork of deprecated serde_yaml) for parsing and serialization
• Recursively walks YAML tree to find all string values with krust:// prefix
• Builds each unique path once, stores digest mapping
• Second pass replaces all references with digests
• Preserves YAML structure and formatting

Usage:

• krust resolve -f deployment.yaml | kubectl apply -f -
• Or use the convenience command: krust apply -f deployment.yaml

Apply Command (krust apply)

Convenience wrapper that combines resolve with kubectl apply:

• Resolves krust:// references
• Pipes resolved YAML directly to kubectl apply -f -
• Exits with kubectl's exit code
• Requires kubectl to be installed and configured

Usage: krust apply -f deployment.yaml

Future Improvements

Potential enhancements identified:

1. Registry authentication support ✓ Implemented (supports Docker credential helpers)
2. YAML resolution for Kubernetes deployments ✓ Implemented (krust resolve)
3. Multi-platform image manifests
4. Build caching ✓ Implemented (persistent target/krust/ directory)
5. Image layer optimization
6. Support for custom Dockerfile-like configs
7. SBOM (Software Bill of Materials) generation
8. Optimize blob uploads (check if blob exists before uploading) ✓ Implemented
9. Stream uploads from disk instead of buffering in memory
   • Currently buffers tar and compressed layer in memory
   • Could write to temp file, calculate diff_id, then stream upload
   • Would reduce memory usage for large binaries

Useful Commands

# Test with anonymous registry (ttl.sh)
export KRUST_REPO=ttl.sh/test
docker run $(krust build example/hello-krust)

# Test with Google Artifact Registry
export KRUST_REPO=us-central1-docker.pkg.dev/project-id/repo-name
krust build example/hello-krust

# Debug output
krust build -v 2>&1 | less

# Check static linking
file target/krust/x86_64-unknown-linux-musl/release/binary
ldd target/krust/x86_64-unknown-linux-musl/release/binary  # should say "not a dynamic executable"

# Verify pushed image
crane manifest $(krust build --no-push example/hello-krust)

Resources

• OCI Image Spec
• OCI Distribution Spec
• ko.build - Inspiration for this project
• ttl.sh - Anonymous ephemeral registry for testing