npm-snoop/CLAUDE.md

npm-snoop

Go cron job that tails the npm registry changes feed and records releases, metadata modifications, and deletions to BigQuery via CloudEvents. Includes an AST diff service that compares symbols between consecutive npm package versions.

Structure

• cmd/npm-cron/ — the poller cron job
• cmd/ast-diff/ — CLI tool for diffing symbols between two npm package versions
• internal/npm/ — npm registry client (metadata, tarball download, semver logic)
• internal/symbols/ — tree-sitter JS/TS parser for extracting top-level symbols
• internal/diff/ — symbol map diffing logic
• terraform/ — all infrastructure (networking, broker, recorder, cron job, ast-diff service, dashboards)
• terraform/schema.json — BigQuery schema for release events
• terraform/schema-modified.json — BigQuery schema for modification events
• terraform/schema-deleted.json — BigQuery schema for deletion events

Build and deploy

The cron Terraform module builds the Go binary with ko automatically. To redeploy after code changes:

cd terraform && terraform apply -var=project_id=jason-491415

To just rebuild the cron job image without touching other infra, use -target=module.cron. To rebuild only the ast-diff service, use -target=module.ast-diff.

Key design decisions

• The poller sends CloudEvents to the broker using an ID-token-authenticated HTTP client (google.golang.org/api/idtoken), because the broker is a private Cloud Run service with ingress: internal.
• The cron job uses vpc_access with PRIVATE_RANGES_ONLY egress so broker traffic routes through the VPC (via private DNS for *.run.app) while npm registry traffic goes directly over the internet.
• Event IDs replace / with _ because the recorder uses the event ID as a filename, and scoped npm packages contain slashes (e.g., @scope/name).
• The poller loops through batches until fully caught up, saving progress to GCS after each batch. This prevents missed events when more changes occur in a 15-minute window than a single batch can handle.
• On first run (no state file), the poller bootstraps from the registry's current update_seq rather than processing the entire history.
• CloudEvent sends are checked with IsACK (not IsUndelivered) to catch non-2xx HTTP responses that the SDK would otherwise treat as delivered.
• State is stored as JSON (state.json) with both a sequence number and a timestamp. The timestamp is used to find new versions in each package's time map.
• Release events are emitted per version, not per package. A single changes feed entry can produce multiple release events if several versions were published between polls.
• Modified events include the full raw package metadata (with versions stripped) so changes can be diffed later.
• Package-level deletions (entire package removed) are recorded as deletion events. Version unpublishes show up as modification events where the version disappears from the metadata.
• The ast-diff service subscribes to dev.npm.package.released.v1 events, downloads both the new and previous version tarballs, parses all JS/TS files with tree-sitter, and emits a dev.npm.package.diff.v1 event listing added/removed/modified symbols.
• Symbol extraction only covers top-level declarations (functions, classes, variables, methods), not local variables inside function bodies. TypeScript interface/type declarations are not yet extracted.
• Symbols are identified by file:name:kind; modification is detected by SHA-256 hash of source text.
• .min.js files and node_modules/ directories are skipped during parsing.
• Destructured variable declarations (const { a, b: c } = ...) are expanded into individual symbol entries.

Gotchas

• The npm license field can be a string or an object; extractLicense handles both.
• Packages that return 404 from the registry (unpublished between the changes feed and the metadata fetch) are emitted as deletion events.
• The versions key is stripped from raw metadata in event bodies — it contains full metadata for every version ever published and can be megabytes.
• Terraform state is local. Back it up or move to a GCS backend.
• The recorder uses event IDs as filenames, so any / in IDs will cause write failures.