snoop/RESOURCE_LIMITS.md

Resource Limits and Recommendations

This document provides guidance on configuring resource limits for the snoop sidecar in production environments.

Overview

Snoop is designed to be lightweight and have minimal impact on application performance. However, proper resource limits ensure predictable behavior and protect against resource exhaustion under unusual conditions.

Memory Usage

Components

Snoop's memory usage consists of several components:

1. eBPF Maps (Kernel Memory)

   • Ring buffer: 256 KB (fixed)
   • Per-CPU heap: 1 entry × ~300 bytes × CPU count (negligible)
   • Traced cgroups map: 64 entries × ~16 bytes = ~1 KB
   • Dropped events counter: negligible
   • Total kernel memory: ~300 KB (plus per-CPU overhead)

2. Userspace Deduplication Cache

   • Each file path entry: ~256 bytes (path string + map/list overhead)
   • Default (unbounded): memory grows with unique files seen
   • With max-unique-files=N: capped at approximately N × 256 bytes

   Examples:

   • 10,000 files: ~2.5 MB
   • 100,000 files: ~25 MB
   • 1,000,000 files: ~250 MB

3. Go Runtime Overhead

   • Base runtime: ~5-10 MB
   • Goroutines: minimal (4-5 goroutines)
   • HTTP server (metrics/health): ~1-2 MB

4. Report Buffer

   • JSON serialization buffer: proportional to unique files
   • Temporary, released after each report write
   • Peak usage: ~2× the deduplication cache size during report generation

Recommendations

Conservative (Most Applications)

For applications with typical file access patterns (thousands of unique files):

resources:
  requests:
    memory: 32Mi
  limits:
    memory: 128Mi

Configuration:

-max-unique-files=50000  # Cap at 50K unique files (~12 MB)

Moderate (Large Applications)

For applications that access many files (e.g., monorepos, data processing):

resources:
  requests:
    memory: 64Mi
  limits:
    memory: 256Mi

Configuration:

-max-unique-files=200000  # Cap at 200K unique files (~50 MB)

Unbounded (Long-Running Observability)

For long-term observation where you want complete data:

resources:
  requests:
    memory: 128Mi
  limits:
    memory: 512Mi

Configuration:

-max-unique-files=0  # Unbounded, monitor snoop_unique_files metric

Important: When using unbounded mode, actively monitor the snoop_unique_files metric to detect unexpected growth.

CPU Usage

Characteristics

Snoop's CPU usage is primarily driven by:

1. Event Processing: Proportional to syscall frequency

   • Each event: path normalization, deduplication lookup, metric updates
   • Typical overhead: <50 µs per event

2. Report Writing: Periodic JSON serialization

   • Frequency: configurable via -interval (default: 30s)
   • Duration: 1-10ms for typical workloads

3. eBPF Overhead: Kernel-side filtering and event emission

   • Per-syscall overhead: <1 µs
   • Negligible for most workloads

Expected CPU Usage

Workload	File Accesses/sec	Expected CPU	Notes
Idle	<10	<0.1%	Background syscalls only
Light	10-100	0.1-0.5%	Typical web services
Moderate	100-1000	0.5-2%	Active applications
Heavy	1000-10000	2-5%	High I/O workloads
Extreme	>10000	5-10%	May hit ring buffer limits

Recommendations

Conservative (Most Applications)

resources:
  requests:
    cpu: 10m      # 10 millicores (1% of 1 CPU)
  limits:
    cpu: 100m     # 100 millicores (10% of 1 CPU)

Moderate (High I/O Applications)

resources:
  requests:
    cpu: 20m      # 20 millicores (2% of 1 CPU)
  limits:
    cpu: 200m     # 200 millicores (20% of 1 CPU)

High-Throughput (Data Processing)

resources:
  requests:
    cpu: 50m      # 50 millicores (5% of 1 CPU)
  limits:
    cpu: 500m     # 500 millicores (50% of 1 CPU)

Note: CPU limits should be generous to avoid throttling during burst activity (e.g., application startup).

Disk I/O

Snoop writes reports to disk periodically. The I/O pattern is:

• Frequency: Every -interval seconds (default: 30s)
• Write size: Proportional to unique files (typical: 10-500 KB per report)
• Method: Atomic write via temp file + rename
• Peak I/O: 2× the report size (temp file + rename)

Recommendations

1. Report Interval:

   • Default (30s) is appropriate for most workloads
   • Increase for large file counts to reduce I/O frequency:

     -interval=60s   # For 100K+ unique files
     -interval=120s  # For 500K+ unique files
     

2. Volume Type:

   • Any volume type is suitable (even NFS)
   • Atomic rename is used for crash safety
   • No special I/O performance requirements

3. Volume Size:

   • Minimum: 100 MB
   • Recommended: 1 GB (allows for log rotation, multiple reports)
   • Report size: typically 10-500 KB, max ~5 MB for 1M files

Complete Examples

Example 1: Small Web Service

Application characteristics:

• 2-3 containers per pod
• ~5,000 unique files accessed
• Light file I/O (<100 accesses/sec)

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  containers:
  - name: app
    image: myapp:latest
    # ... app config ...
    
  - name: snoop
    image: snoop:latest
    args:
    - -cgroup=/sys/fs/cgroup/kubepods/pod$(POD_UID)/$(CONTAINER_ID)
    - -report=/data/snoop-report.json
    - -interval=30s
    - -max-unique-files=10000
    - -log-level=info
    securityContext:
      capabilities:
        add: [SYS_ADMIN, BPF, PERFMON]
      readOnlyRootFilesystem: true
    resources:
      requests:
        cpu: 10m
        memory: 32Mi
      limits:
        cpu: 100m
        memory: 128Mi
    volumeMounts:
    - name: snoop-data
      mountPath: /data
    - name: cgroup
      mountPath: /sys/fs/cgroup
      readOnly: true
      
  volumes:
  - name: snoop-data
    emptyDir:
      sizeLimit: 100Mi
  - name: cgroup
    hostPath:
      path: /sys/fs/cgroup

Example 2: Data Processing Application

Application characteristics:

• Single container, processing large datasets
• ~100,000 unique files accessed
• Heavy file I/O (1,000+ accesses/sec)

apiVersion: v1
kind: Pod
metadata:
  name: data-processor
spec:
  containers:
  - name: app
    image: data-processor:latest
    # ... app config ...
    
  - name: snoop
    image: snoop:latest
    args:
    - -cgroup=/sys/fs/cgroup/kubepods/pod$(POD_UID)/$(CONTAINER_ID)
    - -report=/data/snoop-report.json
    - -interval=60s
    - -max-unique-files=200000
    - -log-level=info
    securityContext:
      capabilities:
        add: [SYS_ADMIN, BPF, PERFMON]
      readOnlyRootFilesystem: true
    resources:
      requests:
        cpu: 50m
        memory: 128Mi
      limits:
        cpu: 500m
        memory: 512Mi
    volumeMounts:
    - name: snoop-data
      mountPath: /data
    - name: cgroup
      mountPath: /sys/fs/cgroup
      readOnly: true
      
  volumes:
  - name: snoop-data
    emptyDir:
      sizeLimit: 500Mi
  - name: cgroup
    hostPath:
      path: /sys/fs/cgroup

Example 3: Long-Running Observability

Application characteristics:

• Production service, running for weeks
• Unknown file access patterns (new deployment)
• Want complete observability without data loss

apiVersion: v1
kind: Pod
metadata:
  name: production-service
spec:
  containers:
  - name: app
    image: production-service:latest
    # ... app config ...
    
  - name: snoop
    image: snoop:latest
    args:
    - -cgroup=/sys/fs/cgroup/kubepods/pod$(POD_UID)/$(CONTAINER_ID)
    - -report=/data/snoop-report.json
    - -interval=60s
    - -max-unique-files=0  # Unbounded - monitor snoop_unique_files
    - -log-level=warn      # Reduce log noise in production
    - -metrics-addr=:9090
    securityContext:
      capabilities:
        add: [SYS_ADMIN, BPF, PERFMON]
      readOnlyRootFilesystem: true
    resources:
      requests:
        cpu: 20m
        memory: 128Mi
      limits:
        cpu: 200m
        memory: 1Gi  # Generous limit for unbounded mode
    volumeMounts:
    - name: snoop-data
      mountPath: /data
    - name: cgroup
      mountPath: /sys/fs/cgroup
      readOnly: true
    ports:
    - name: metrics
      containerPort: 9090
      
  volumes:
  - name: snoop-data
    persistentVolumeClaim:
      claimName: snoop-data-pvc  # Persistent storage for long-term observation
  - name: cgroup
    hostPath:
      path: /sys/fs/cgroup

Monitoring and Alerts

Use Prometheus metrics to monitor snoop resource usage:

Memory Monitoring

# Current unique files being tracked
snoop_unique_files

# Memory estimate (bytes): unique_files × 256
snoop_unique_files * 256

# Alert when approaching memory limits
snoop_unique_files * 256 > 100000000  # 100 MB

CPU Monitoring

# Event processing rate
rate(snoop_events_received_total[5m])

# Events dropped (ring buffer overflow)
rate(snoop_events_dropped_total[5m])

# Cache evictions (memory pressure)
rate(snoop_events_evicted_total[5m])

Recommended Alerts

# Alert when ring buffer is dropping events
- alert: SnoopRingBufferOverflow
  expr: rate(snoop_events_dropped_total[5m]) > 10
  for: 5m
  annotations:
    summary: "Snoop is dropping events due to ring buffer overflow"
    description: "Consider increasing CPU limits or reducing file access rate"

# Alert when cache eviction is occurring
- alert: SnoopCacheEviction
  expr: rate(snoop_events_evicted_total[5m]) > 0
  for: 5m
  annotations:
    summary: "Snoop is evicting cached file paths"
    description: "Consider increasing max-unique-files or memory limits"

# Alert when memory usage is high
- alert: SnoopHighMemoryUsage
  expr: snoop_unique_files > 300000
  for: 10m
  annotations:
    summary: "Snoop is tracking a large number of unique files"
    description: "Consider setting max-unique-files limit or investigating unusual file access patterns"

Tuning Guidelines

Reducing Memory Usage

If memory usage is higher than expected:

1. Set a limit: Use -max-unique-files to cap memory growth

   -max-unique-files=50000  # Limit to ~12 MB
   

2. Increase exclusions: Filter out unnecessary paths

   -exclude=/proc/,/sys/,/dev/,/tmp/  # Add /tmp/ if temp files aren't relevant
   

3. Monitor evictions: Check if LRU evictions are affecting data completeness

   rate(snoop_events_evicted_total[1h])
   

Reducing CPU Usage

If CPU usage is higher than expected:

1. Increase report interval: Reduce JSON serialization frequency

   -interval=60s  # or 120s
   

2. Check event rate: Verify application file access patterns

   rate(snoop_events_received_total[5m])
   

3. Verify filtering: Ensure cgroup filtering is working correctly

   # Check that only target container events are being processed
   # Log level debug will show which cgroup IDs are traced
   -log-level=debug
   

Handling Ring Buffer Overflow

If snoop_events_dropped_total is increasing:

1. This is expected under extreme load (>10,000 file accesses/sec)
2. Options:
   • Increase CPU limits to process events faster
   • Accept data loss during burst periods (best-effort design)
   • For critical observability, consider increasing ring buffer size (requires recompilation)

Performance Impact on Application

Snoop's design minimizes impact on the traced application:

• Syscall overhead: <1 µs per syscall (eBPF filtering is extremely fast)
• No application changes: Zero code changes required
• Kernel-side filtering: Only relevant cgroups emit events
• Ring buffer: Asynchronous event delivery, no blocking

Expected application performance impact: <0.1% for typical workloads.

Verification

To verify snoop is not impacting your application:

1. Before/after benchmarks: Run application benchmarks with and without snoop
2. Monitor application metrics: Watch application-specific performance metrics
3. Syscall latency: Use perf to measure syscall latency changes

# Measure syscall latency without snoop
sudo perf stat -e 'syscalls:sys_enter_openat' -p $(pidof myapp)

# Compare with snoop enabled

Troubleshooting

High Memory Usage (Unbounded Mode)

Symptom: Memory grows continuously beyond expected levels

Diagnosis:

snoop_unique_files  # Check current file count

Solutions:

1. Verify application isn't accessing an unusual number of files
2. Check for symlink loops or recursive directory traversal
3. Add exclusions for problematic paths
4. Set -max-unique-files limit

OOMKilled

Symptom: Snoop container is killed by OOM

Diagnosis: Check kubectl describe pod or container logs

Solutions:

1. Set or increase -max-unique-files
2. Increase memory limits
3. Reduce -interval so reports are written more frequently (releases memory during GC)
4. Add more path exclusions

High CPU Usage

Symptom: Snoop using more CPU than expected

Diagnosis:

rate(snoop_events_received_total[5m])  # Check event rate

Solutions:

1. Verify event rate is reasonable for application
2. Check if filtering is correct (only target cgroup should be traced)
3. Increase CPU limits if processing legitimate high event rate
4. Reduce log level from debug to info or warn

Events Dropped

Symptom: snoop_events_dropped_total is increasing

Diagnosis:

rate(snoop_events_dropped_total[5m])  # Check drop rate

Solutions:

1. This is expected under extreme load (>10K events/sec)
2. Increase CPU limits to process events faster
3. Accept data loss (snoop is designed as best-effort)
4. For critical observability, consider tuning ring buffer size (code change required)

Summary

Default Configuration (suitable for 80% of workloads):

resources:
  requests:
    cpu: 10m
    memory: 32Mi
  limits:
    cpu: 100m
    memory: 128Mi

args:
- -max-unique-files=50000
- -interval=30s
- -log-level=info

Key Principles:

1. Start conservative, scale up based on metrics
2. Always monitor snoop_unique_files in unbounded mode
3. CPU limits should be generous to handle burst activity
4. Ring buffer drops are acceptable under extreme load
5. Snoop is designed for best-effort observability, not guaranteed delivery