Keep the CLI tool and internal packages (npm, symbols, diff) for
local use. The Cloud Run event-driven approach isn't cost-effective
for this workload; see cost.md for analysis.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add tracing spans, httpmetrics, and clog context propagation
- Tarball size limit (20MB), per-file limit (1MB), file count limit (2000)
- Concurrent old/new version processing via errgroup
- Request deadline from env var, aligned with Cloud Run timeout
- Skip events recorded to BQ with skipped_reason for coverage tracking
- Accept *http.Client in npm package for metrics transport
- Disable ast-diff service in production (commented out in ast-diff.tf)
pending cost optimization; see cost.md for analysis
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add ExtractFromTarGz to stream tar entries and parse only JS/TS
files one at a time, avoiding writing the full tarball to Cloud
Run's memory-backed filesystem. The service now pipes the HTTP
response body directly into the streaming extractor.
Also bump instance to 4 vCPU / 16Gi with scale-to-zero and
concurrency=20, and use clog/gcp/init for structured logging.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Add PermanentError type to the npm package for 404/403 responses
that will never succeed on retry. The service now ACKs permanent
failures (private packages, deleted tarballs) so PubSub doesn't
waste retry budget, while returning errors for transient failures
(timeouts, 5xx) so PubSub retries with backoff.
Also adds ast-diff service dashboard, cron dashboard fix, and
min_instances=1 with concurrency=10 for better burst handling.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The service was using ALL_TRAFFIC egress through VPC, which routed
npm registry requests through VPC where they couldn't reach the
internet. PRIVATE_RANGES_ONLY routes only broker traffic through
VPC while npm traffic goes direct, matching the poller's config.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The CGO-based tree-sitter bindings can't cross-compile with ko for
Linux containers. gotreesitter is a pure-Go reimplementation with
embedded grammars, so it builds cleanly with ko for Cloud Run.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cloud Run service subscribes to dev.npm.package.released.v1 events,
downloads old+new tarballs, diffs symbols with tree-sitter, and emits
dev.npm.package.diff.v1 events back to the broker for BQ recording.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Uses tree-sitter to parse JS/TS files and extract top-level symbols
(functions, classes, variables, methods), then diffs them by content
hash to identify added/removed/modified symbols between two versions.
Intended to help correlate CVE fixes with specific changed functions.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Emit one release event per version published (not per package), using
the metadata time map to find versions published since last poll
- Add modification events for metadata-only changes (dist-tag moves,
maintainer updates, etc.) with full raw JSON body for diffing
- Include raw package metadata (versions key stripped) in release and
modification events for later analysis
- Store poll state as JSON (seq + timestamp) instead of plain seq
- Switch recorder to method=gcs (Pub/Sub writes directly to GCS,
eliminating the recorder Cloud Run service)
- Add queries.md with example BQ queries including a window-function
diff query for tracking metadata changes over time
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Go cron job that tails the npm CouchDB _changes feed, fetches package
metadata, and emits CloudEvents for releases and deletions. Events flow
through a Chainguard cloudevent-broker to a cloudevent-recorder that
streams them into BigQuery.
Infrastructure is managed with Terraform using chainguard-dev/terraform-
infra-common modules for networking, broker, recorder, and cron.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
