1
0
Fork 0
mirror of https://github.com/imjasonh/snoop synced 2026-07-19 07:04:01 +00:00

Add memory-bounded deduplication with LRU cache

Implements configurable memory limits for the file deduplication cache
to prevent unbounded memory growth in long-running production deployments.

- Add LRU cache implementation with configurable max size
- Add -max-unique-files flag (default: 0 = unbounded)
- Add snoop_events_evicted_total metric for monitoring
- Track and log cache evictions with delta warnings
- Add comprehensive tests for bounded and unbounded behavior

When max-unique-files is set, the cache evicts least-recently-used
entries to stay within the limit. Evicted files may be re-recorded
if accessed again, making this suitable for workloads with temporal
locality in file access patterns.

Closes milestone 3 task: Memory-bounded deduplication

Signed-off-by: Jason Hall <jason@chainguard.dev>
This commit is contained in:
Jason Hall 2026-01-14 10:44:04 -05:00
parent 87dab9839a
commit b3f7c96343
8 changed files with 458 additions and 231 deletions

View file

@ -8,8 +8,9 @@ Milestone 3 progress:
- ✅ Prometheus metrics endpoint (`/metrics` on port 9090)
- ✅ Structured logging with clog
- ✅ Ring buffer overflow handling and metrics
- ✅ Memory-bounded deduplication with LRU cache
**Next**: Memory-bounded deduplication
**Next**: Health check endpoint
See [Milestone 3](#milestone-3-production-hardening) for remaining tasks.
@ -364,7 +365,7 @@ Volume mounts:
- [x] Prometheus metrics endpoint
- [x] Structured logging with levels (clog)
- [x] Ring buffer overflow handling and metrics
- [ ] Memory-bounded deduplication (LRU or bloom filter for extreme cases)
- [x] Memory-bounded deduplication (LRU cache with configurable max size)
- [ ] Health check endpoint
- [ ] Configuration validation
- [ ] Resource limit recommendations documented