npm-snoop

Polls the npm registry's CouchDB changes feed and records package activity to BigQuery via CloudEvents.

Architecture

A Cloud Run cron job runs every 15 minutes:

1. Reads state (sequence number + last poll timestamp) from a GCS bucket
2. Polls https://replicate.npmjs.com/registry/_changes for new changes
3. Fetches metadata for each changed package from https://registry.npmjs.org/<name>
4. Classifies each change and emits CloudEvents to a broker (Chainguard cloudevent-broker)
5. The recorder (Chainguard cloudevent-recorder) writes events to BigQuery via GCS + DTS

The poller loops through batches until fully caught up, saving progress after each batch.

Event types

dev.npm.package.released.v1

One event per version published. The poller compares the package's time map against the last poll timestamp to find newly published versions. A single package change can produce multiple release events if several versions were published between polls.

Fields: name, version, description, license, homepage, published_at, body (raw package metadata JSON, versions key stripped).

dev.npm.package.modified.v1

A package was changed but no new versions were published. This covers metadata-only changes like description edits, maintainer updates, dist-tag moves, ownership transfers, or npm security actions.

Fields: name, modified_at, body (raw package metadata JSON, versions key stripped). To determine what changed, diff the body against a previous modified or released event for the same package.

dev.npm.package.deleted.v1

An entire package was removed from npm. This is a package-level deletion, not a version unpublish — version removals appear as modified events (the version disappears from the metadata).

Fields: name, deleted_at.

Deploy

Requires gcloud auth, terraform, and ko.

cd terraform
terraform init
terraform apply -var=project_id=YOUR_PROJECT_ID

Key variables (see terraform/variables.tf):

Variable	Default	Description
project_id	(required)	GCP project
region	us-east4	GCP region
schedule	*/15 * * * *	Cron schedule
bq_retention_days	365	BigQuery retention
deletion_protection	false	Protect resources from deletion

Manual run

gcloud run jobs execute npm-snoop-cron --project YOUR_PROJECT_ID --region us-east4 --wait

Queries

See queries.md for example queries, including how to diff consecutive metadata modifications using window functions.