npm-snoop/ast-diff-plan.md

AST Diff Service Plan

Goal

Build a service that subscribes to dev.npm.package.released.v1 events, downloads the new and previous version tarballs, parses all JS/TS files with tree-sitter, diffs the symbols (functions, classes, variables), and emits a dev.npm.package.diff.v1 event describing what changed. This enables correlating CVE reports with specific changed symbols.

Architecture

[broker] ---> [ast-diff service] ---> [broker] ---> [recorder/BQ]
               (subscribes to            (emits
                released.v1)              diff.v1)

• A new Cloud Run service, deployed alongside the existing poller.
• Subscribes to dev.npm.package.released.v1 via the existing broker.
• Emits dev.npm.package.diff.v1 back to the broker.
• Recorded to BigQuery via the existing recorder module.

Steps

1. Scaffold the new service (cmd/ast-diff/main.go)

• Cloud Run HTTP service that receives CloudEvents.
• Receives dev.npm.package.released.v1 events containing name, version, etc.
• Env config: K_SINK (broker ingress URL for sending diff events).

2. Find the previous version

• Fetch package metadata from https://registry.npmjs.org/<name>.
• Extract all version strings from the versions object keys.
• Parse as semver, sort, find the version immediately before the new one.
• Skip if this is the first version (nothing to diff against).
• Use github.com/Masterminds/semver/v3 for parsing/sorting.

3. Download and extract tarballs

• npm tarballs are at https://registry.npmjs.org/<name>/-/<basename>-<version>.tgz.
• For scoped packages: https://registry.npmjs.org/<@scope/name>/-/<name>-<version>.tgz.
• Download both versions, decompress (gzip), untar to temp dirs.
• All files are under a package/ prefix in the tarball.

4. Parse JS/TS files with tree-sitter

• Use github.com/tree-sitter/go-tree-sitter with JavaScript and TypeScript grammars.
• Walk both extracted dirs, parse every .js, .mjs, .cjs, .ts, .mts, .cts file.
• For each file, extract symbols: function declarations, class declarations, variable declarations, method definitions. Record name + kind + file path + line range.
• Build a map of file:symbol_name:kind -> source text (or hash of source text) for each version.

5. Diff the symbol maps

• Compare old vs new symbol maps:
  • Added: symbols in new but not old
  • Removed: symbols in old but not new
  • Modified: symbols in both but with different source text/hash
• For modified symbols, the source text hash is enough — we don't need a fine-grained AST diff, just "this function changed".

6. Emit diff event

Event type: dev.npm.package.diff.v1

{
  "name": "@scope/package",
  "old_version": "1.2.3",
  "new_version": "1.2.4",
  "symbols_added": [
    {"name": "bar", "kind": "function", "file": "lib/utils.js"}
  ],
  "symbols_removed": [...],
  "symbols_modified": [
    {"name": "foo", "kind": "function", "file": "lib/handler.js"}
  ],
  "files_added": ["lib/new.js"],
  "files_removed": ["lib/old.js"]
}

7. BigQuery schema (terraform/schema-diff.json)

Fields: name, old_version, new_version, symbols_added (JSON string), symbols_removed (JSON string), symbols_modified (JSON string), files_added (JSON string), files_removed (JSON string).

8. Terraform changes

• New regional-go-service module for the ast-diff service.
• Subscribe it to dev.npm.package.released.v1 from the broker.
• Authorize it to publish back to the broker.
• Add dev.npm.package.diff.v1 to the recorder's types.
• Service account with minimal permissions.

Future work

• TypeScript type/interface extraction: .d.ts files contain interface and type declarations that we don't currently extract. Less relevant for CVE analysis (no runtime behavior) but could matter for API surface diffing.

Key considerations

• Tarball size: Some packages are huge. We should set a size limit (say 50MB) and skip packages over that threshold.
• Timeout: Parsing large packages will take time. Cloud Run allows up to 60 min; we should set a reasonable timeout (5 min?) and skip if exceeded.
• Scoped packages: Tarball URL format differs for @scope/name packages — the tarball name is just name-version.tgz without the scope.
• Pre-releases: Semver sorting handles pre-releases naturally. We should diff pre-releases against their base version's predecessor.
• Binary/minified files: tree-sitter will parse minified JS fine but the symbols will be mangled. We could skip .min.js files. Detecting bundled/minified code more broadly is a stretch goal.

Implementation order

1. Scaffold service + tarball download/extraction
2. tree-sitter parsing + symbol extraction
3. Diffing logic
4. CloudEvent emission
5. Terraform infra (service, subscription, BQ schema)
6. Testing with real packages