mirror of https://github.com/imjasonh/terraform-playground synced 2026-07-13 18:06:46 +00:00

History

…
..
crates
scripts
.gitignore
Cargo.lock
Cargo.toml
README.md

README.md

firecracker-container-vm

Example Firecracker VM runner that uses a container image reference as the guest root filesystem, loading files on demand at runtime over HTTP range requests into gzip-compressed OCI layers.

This implements the lazy-pull approach described in dagdotdev registry explorer:

Build a gzip zran-style index (indexed_deflate) over each tar+gzip layer.
Record a tar table of contents mapping paths to uncompressed byte offsets.
On file read, fetch only the compressed byte range needed from the registry, seek in the gzip stream, and return file bytes.
Serve the merged overlay via virtio-fs using vhost-user-backend + fuse-backend-rs.

Architecture

┌─────────────────────┐      vhost-user UDS      ┌──────────────────────┐
│ Firecracker (guest) │ ◄──────────────────────► │ fc-vhostfsd          │
│  virtio-fs driver   │                        │  vhost-user-backend  │
└─────────────────────┘                        │  fuse-backend-rs     │
                                               │  OverlayFs (lazy)    │
                                               └──────────┬───────────┘
                                                          │ HTTP Range
                                                          ▼
                                               ┌──────────────────────┐
                                               │ OCI registry blobs   │
                                               │ (tar+gzip layers)    │
                                               └──────────────────────┘

Crates

Crate	Role
`fc-oci-fs`	Registry client, gzip index, tar TOC, overlay `FileSystem`
`fc-vhostfsd`	vhost-user virtio-fs daemon
`fc-runner`	Spawns `fc-vhostfsd` and configures Firecracker

Prerequisites

Requirement	Notes
Rust 1.96+	`source /usr/local/cargo/env` in the devcontainer
`/dev/kvm` read/write	`[ -w /dev/kvm ] && echo OK` — add yourself to the `kvm` group if needed
Docker config (optional)	`~/.docker/config.json` for private registries
Host tools for VM path	`git`, `curl`, `docker`, `sudo` (kernel + Firecracker builds use Firecracker's `devtool`)

Important: virtio-fs needs two things that official Firecracker release artifacts do not provide out of the box:

A guest vmlinux with CONFIG_VIRTIO_FS=y — the CI kernels from S3 do not enable virtio-fs.
A Firecracker binary with generic vhost-user (PR #5773) — release builds only expose vhost-user block, not virtio-fs.

The scripts under scripts/ automate both builds into .deps/.

Build this project

source /usr/local/cargo/env
cd firecracker-container-vm
cargo build --release

Quickstart (full path)

1. Host dependencies (Firecracker + vmlinux)

One-shot (builds from source; needs Docker and sudo):

cd firecracker-container-vm
./scripts/setup-host.sh

Or step by step:

export DEPS_DIR="$PWD/.deps"

# Firecracker with PUT /vhost-user-devices/{id} (PR #5773 branch)
./scripts/build-firecracker-virtiofs.sh

# Guest kernel with virtio-fs enabled (patches Firecracker's 6.1 CI config)
./scripts/build-vmlinux-virtiofs.sh

To download a stock Firecracker release only (no virtio-fs frontend):

./scripts/download-firecracker.sh   # installs .deps/firecracker

2. Verify KVM

[ -r /dev/kvm ] && [ -w /dev/kvm ] && echo "KVM OK" || echo "Fix KVM access first"

3. Run the lazy rootfs daemon

./target/release/fc-vhostfsd \
  --image docker.io/library/alpine:3.20 \
  --socket /tmp/fc-vhostfs.sock \
  --cache-dir /tmp/fc-oci-cache \
  --tag rootfs \
  --metrics-addr 127.0.0.1:9100

In another terminal, confirm metrics and that the socket exists:

curl -s localhost:9100/metrics | rg '^fc_startup_ready_milliseconds'
ls -l /tmp/fc-vhostfs.sock

4. Boot Firecracker with the container image as rootfs

./target/release/fc-runner \
  --image docker.io/library/alpine:3.20 \
  --firecracker "$PWD/.deps/firecracker-virtiofs" \
  --kernel "$PWD/.deps/vmlinux-virtiofs" \
  --cache-dir /tmp/fc-oci-cache \
  --vhost-socket /tmp/fc-vhostfs.sock \
  --tag rootfs \
  --memory-mib 512

Use --dry-run to print the Firecracker API payload without starting the VM (still starts fc-vhostfsd).

Guest cmdline (set by fc-runner):

console=ttyS0 reboot=k panic=1 pci=off init=/bin/sh rootfstype=virtiofs root=/ root=rootfs

Alpine provides /bin/sh on the merged root. You should get a shell on the serial console when the VM starts.

5. Smoke test without a VM (daemon only)

If you only want to validate lazy pulls + vhost-user handshake:

cargo test
# vhost-user protocol test against a local fixture layer (no KVM/registry)
cargo test -p fc-runner --test vhost_e2e

Run the vhost-fs daemon (reference)

./target/release/fc-vhostfsd \
  --image docker.io/library/alpine:3.20 \
  --socket /tmp/fc-vhostfs.sock \
  --cache-dir /tmp/fc-oci-cache \
  --tag rootfs

On first access to a layer, the daemon downloads the blob (if not cached), builds a gzip index + tar TOC, then serves files via range reads.

Registry authentication

fc-vhostfsd / fc-oci-fs read credentials the same way the Docker CLI does:

~/.docker/config.json (or $DOCKER_CONFIG/config.json)
auths entries (inline base64 user/pass)
per-registry credHelpers
global credsStore (e.g. docker-credential-gcr, osxkeychain, pass)

Run docker login (or your cloud provider's helper) once; private images should work without extra flags.

Metrics

fc-vhostfsd exposes Prometheus text on http://127.0.0.1:9100/metrics by default (--metrics-addr).

Metric	Meaning
`fc_bytes_fetched_from_registry`	Compressed bytes actually pulled from the registry
`fc_bytes_saved_vs_full_pull`	`sum(layer sizes) - fetched` (data you did not download vs a full `docker pull`)
`fc_layer_compressed_bytes_total`	Total compressed layer bytes in the resolved image
`fc_registry_range_requests_total`	HTTP range requests to blob URLs
`fc_full_blob_downloads_total`	Layers fully copied to local cache (first-time index build)
`fc_gzip_index_builds_total`	Gzip index builds
`fc_fuse_requests_total` / `fc_fuse_reads_total`	virtio-fs / FUSE traffic
`fc_startup_ready_milliseconds`	Image open → vhost socket listening
`fc_process_rss_bytes`	Daemon resident memory
`fc_cache_dir_bytes_on_disk`	Local cache size (blobs + indexes)

curl -s localhost:9100/metrics | rg '^fc_'

Which VMM can use this? (virtio-fs frontends)

fc-vhostfsd is a vhost-user backend. Something in the VMM must act as the vhost-user frontend and connect to --socket. Your options:

Frontend	Status	Notes
Firecracker (generic vhost-user)	Needs recent build	PR #5773 adds `PUT /vhost-user-devices/{id}` so virtio-fs works without native Firecracker virtio-fs code. `fc-runner` targets this API.
Cloud Hypervisor	Works today	First-class virtio-fs + vhost-user; point `--socket` at the same path. No Firecracker-specific API.
QEMU	Works today	`virtiofsd` / custom daemon via `-chardev socket` + `vhost-user-fs-pci` device.
crosvm	Works today	Can run vhost-user fs backends against a virtio-fs device.
Stock Firecracker (released)	No virtio-fs	Only block/net/vsock unless you build from the generic vhost-user branch.

You do not need to change fc-vhostfsd between these — only the VMM configuration differs. fc-runner is Firecracker-specific; for Cloud Hypervisor or QEMU, run fc-vhostfsd manually and wire the socket in that VMM's config.

Guest kernel (`vmlinux`)

You need a Linux guest kernel with virtio-fs over virtio-mmio (Firecracker's bus).

Do not use the stock CI vmlinux for this example

Firecracker's getting-started guide downloads kernels from S3:

ARCH="$(uname -m)"
release_url="https://github.com/firecracker-microvm/firecracker/releases"
latest_version=$(basename "$(curl -fsSLI -o /dev/null -w '%{url_effective}' "${release_url}/latest")")
CI_VERSION=${latest_version%.*}
latest_kernel_key=$(curl "http://spec.ccfc.min.s3.amazonaws.com/?prefix=firecracker-ci/$CI_VERSION/$ARCH/vmlinux-&list-type=2" \
  | grep -oP "(?<=<Key>)(firecracker-ci/$CI_VERSION/$ARCH/vmlinux-[0-9]+\.[0-9]+\.[0-9]{1,3})(?=</Key>)" \
  | sort -V | tail -1)
wget -O vmlinux-ci "https://s3.amazonaws.com/spec.ccfc.min/${latest_kernel_key}"

That kernel is fine for block-device rootfs smoke tests, but its config has # CONFIG_VIRTIO_FS is not set. Use ./scripts/build-vmlinux-virtiofs.sh instead.

Recommended: build virtio-fs-enabled vmlinux

./scripts/build-vmlinux-virtiofs.sh
# output: .deps/vmlinux-virtiofs

This clones Firecracker, enables CONFIG_VIRTIO_FS=y in the 6.1 CI guest config, and runs ./tools/devtool build_ci_artifacts kernels 6.1.

Minimum kernel options (already present in Firecracker CI configs except virtio-fs):

CONFIG_VIRTIO_MMIO=y
CONFIG_VIRTIO_FS=y
CONFIG_FUSE_FS=y
CONFIG_SERIAL_8250_CONSOLE=y for console=ttyS0

Manual build details: Firecracker rootfs and kernel setup.

Firecracker binary

For virtio-fs (`fc-runner`)

Released binaries from GitHub releases do not include PUT /vhost-user-devices/{id} yet. Build from PR #5773:

./scripts/build-firecracker-virtiofs.sh
# output: .deps/firecracker-virtiofs

Download latest official release (block/net only)

./scripts/download-firecracker.sh
# output: .deps/firecracker
curl -fsSL https://github.com/firecracker-microvm/firecracker/releases/latest/download/firecracker-$(curl -fsSLI -o /dev/null -w '%{url_effective}' https://github.com/firecracker-microvm/firecracker/releases/latest | xargs basename)-$(uname -m).tgz | tar -xz

Or manually (from upstream docs):

ARCH="$(uname -m)"
release_url="https://github.com/firecracker-microvm/firecracker/releases"
latest=$(basename "$(curl -fsSLI -o /dev/null -w '%{url_effective}' "${release_url}/latest")")
curl -fsSL "${release_url}/download/${latest}/firecracker-${latest}-${ARCH}.tgz" | tar -xz
mv "release-${latest}-${ARCH}/firecracker-${latest}-${ARCH}" firecracker

Run the Firecracker example (reference)

./target/release/fc-runner \
  --image docker.io/library/alpine:3.20 \
  --firecracker "$PWD/.deps/firecracker-virtiofs" \
  --kernel "$PWD/.deps/vmlinux-virtiofs" \
  --cache-dir /tmp/fc-oci-cache \
  --dry-run

Without --dry-run, fc-runner configures Firecracker via its HTTP API:

PUT /boot-source
PUT /machine-config
PUT /vhost-user-devices/rootfs (virtio-fs via generic vhost-user frontend)
PUT /actions (InstanceStart)

Guest kernel cmdline includes rootfstype=virtiofs root=/ root=rootfs.

Tests

cargo test

Integration test vhost_user_virtiofs_daemon_handshake verifies the vhost-user protocol end-to-end against a local fixture layer (no registry/KVM required).

Benchmarks

cargo bench -p fc-oci-fs

Benchmarks build a synthetic tar.gz layer with hundreds of files and measure:

read_last_file_1kb — random read of the last tar entry (exercises gzip seek + range-style decompression)
lookup_opt_data_file — FUSE lookup through the overlay

Design notes & limitations (example code)

This is an example implementation, not production hardened:

Read-only rootfs; no whiteout/opaque-dir completeness guarantees beyond basic .wh.* handling
Single-platform manifests only (no OCI index platform selection)
Layer blobs are cached to disk on first full fetch; subsequent reads use the on-disk gzip index
fc-runner is Firecracker-specific; other VMMs run fc-vhostfsd directly (see table above)

References

dagdotdev explore README — gzip zran / range request strategy
indexed_deflate — gzip random access indexes
vhost-user-backend — Rust vhost-user daemon framework
Firecracker generic vhost-user