imjasonh/snoop
1
0
Fork 0
mirror of https://github.com/imjasonh/snoop synced 2026-07-18 22:55:39 +00:00
Code Activity

Add multi-container pod support to snoop

Browse source 
Implement support for tracing multiple containers within a single
Kubernetes pod. This completes Milestone 4.

Changes:
- Add -cgroups flag for comma-separated cgroup paths
- Maintain backwards compatibility with -cgroup flag
- Add CgroupPaths []string field to Config
- Create helper utilities for container discovery
- Add comprehensive tests (all passing)
- Document multi-container usage patterns
- Include complete working example manifest

The implementation supports up to 64 containers per pod (eBPF map limit)
and allows selective tracing of specific containers while excluding
others (e.g., snoop's own container).

Files:
- pkg/config/config.go: Add CgroupPaths field and parsing
- cmd/snoop/main.go: Add -cgroups flag and multi-cgroup initialization
- pkg/cgroup/multi_container.go: Container discovery utilities
- pkg/config/config_test.go: Comprehensive test coverage
- deploy/kubernetes/README.md: Multi-container documentation
- deploy/kubernetes/multi-container-example.yaml: Working example
- MULTI_CONTAINER_SUPPORT.md: Implementation guide
- plan.md: Mark Milestone 4 complete

Signed-off-by: Jason Hall <jason@chainguard.dev>
This commit is contained in:
Jason Hall 2026-01-14 16:22:51 -05:00
parent 358cc7d795
commit a7f8a9a8fe
9 changed files with 868 additions and 243 deletions
Download patch file Download diff file Expand all files Collapse all files

128
deploy/kubernetes/README.md
View file

@ -7,6 +7,7 @@ This directory contains Kubernetes manifests for deploying snoop as a sidecar co
- `rbac.yaml` - RBAC resources (ServiceAccount, ClusterRole, ClusterRoleBinding)
- `deployment.yaml` - Example deployment with snoop sidecar and test application
- `example-app.yaml` - Example showing how to add snoop to an nginx deployment
- `multi-container-example.yaml` - Example showing snoop in a multi-container pod (tracing specific containers)
## Prerequisites
@ -171,7 +172,8 @@ The snoop sidecar accepts the following command-line arguments:
| Argument | Default | Description |
|----------|---------|-------------|
| `-cgroup` | (required) | Cgroup path to trace |
| `-cgroup` | (required*) | Single cgroup path to trace |
| `-cgroups` | (required*) | Comma-separated list of cgroup paths (for multi-container pods) |
| `-report` | `/data/snoop-report.json` | Path to write JSON reports |
| `-interval` | `30s` | Interval between report writes |
| `-exclude` | `/proc/,/sys/,/dev/` | Comma-separated path prefixes to exclude |
@ -181,6 +183,130 @@ The snoop sidecar accepts the following command-line arguments:
| `-container-id` | (optional) | Container ID for report metadata |
| `-image` | (optional) | Image reference for report metadata |
*Either `-cgroup` or `-cgroups` must be specified.
## Multi-Container Pod Support
If your pod has multiple containers and you want to trace specific containers (not all), you can use the `-cgroups` flag with multiple paths:
### Method 1: Trace all containers in the pod
Modify the init container to discover all container cgroups:
```yaml
initContainers:
- name: cgroup-finder
image: busybox:latest
command:
- sh
- -c
- |
# Get the pod cgroup (parent of our container)
SELF_CGROUP=$(cat /proc/self/cgroup | cut -d: -f3)
POD_CGROUP=$(dirname "$SELF_CGROUP")
# List all container cgroups in the pod
cd "/sys/fs/cgroup$POD_CGROUP"
CGROUPS=""
for dir in */; do
if [ -d "$dir" ]; then
CGROUP_PATH="$POD_CGROUP/${dir%/}"
if [ -z "$CGROUPS" ]; then
CGROUPS="$CGROUP_PATH"
else
CGROUPS="$CGROUPS,$CGROUP_PATH"
fi
fi
done
echo "$CGROUPS" > /snoop-data/cgroup-paths
echo "Found cgroups: $CGROUPS"
volumeMounts:
- name: snoop-data
mountPath: /snoop-data
- name: cgroup
mountPath: /sys/fs/cgroup
readOnly: true
```
Then update the snoop args to use `-cgroups`:
```yaml
args:
- -cgroups=$(cat /data/cgroup-paths)
- -report=/data/snoop-report.json
# ... other args
```
### Method 2: Trace specific containers by name pattern
If you know the container names or IDs, you can manually specify them:
```yaml
args:
- -cgroups=/sys/fs/cgroup/kubepods/burstable/pod<uid>/<container1-id>,/sys/fs/cgroup/kubepods/burstable/pod<uid>/<container2-id>
- -report=/data/snoop-report.json
# ... other args
```
### Method 3: Exclude snoop's own container
To trace all containers except snoop itself:
```yaml
initContainers:
- name: cgroup-finder
image: busybox:latest
command:
- sh
- -c
- |
# Get pod cgroup
SELF_CGROUP=$(cat /proc/self/cgroup | cut -d: -f3)
POD_CGROUP=$(dirname "$SELF_CGROUP")
# Mark snoop's container ID to exclude it later
# Snoop will be the last container started, we'll filter in snoop container
echo "$POD_CGROUP" > /snoop-data/pod-cgroup
volumeMounts:
- name: snoop-data
mountPath: /snoop-data
```
Then in the snoop container, use a wrapper script to discover and filter:
```yaml
containers:
- name: snoop
# ... other config
command:
- sh
- -c
- |
# Discover all containers except self
POD_CGROUP=$(cat /data/pod-cgroup)
SELF_CGROUP=$(cat /proc/self/cgroup | cut -d: -f3)
CGROUPS=""
cd "/sys/fs/cgroup$POD_CGROUP"
for dir in */; do
CGROUP_PATH="$POD_CGROUP/${dir%/}"
# Skip our own cgroup
if [ "/sys/fs/cgroup$CGROUP_PATH" != "/sys/fs/cgroup$SELF_CGROUP" ]; then
if [ -z "$CGROUPS" ]; then
CGROUPS="/sys/fs/cgroup$CGROUP_PATH"
else
CGROUPS="$CGROUPS,/sys/fs/cgroup$CGROUP_PATH"
fi
fi
done
echo "Tracing cgroups: $CGROUPS"
exec /usr/local/bin/snoop -cgroups="$CGROUPS" -report=/data/snoop-report.json # ... other args
```
**Note**: The third method (excluding snoop) is more complex but ensures snoop doesn't trace its own file access, which keeps reports cleaner.
## Security Considerations
The snoop sidecar requires elevated capabilities to load eBPF programs:

267
deploy/kubernetes/multi-container-example.yaml Normal file
View file

@ -0,0 +1,267 @@
# Example of using snoop in a multi-container pod
#
# This example shows a pod with:
# - An nginx web server (main app)
# - A log shipper sidecar (simulated with busybox)
# - Snoop sidecar that traces ONLY the nginx container
#
# This demonstrates how to selectively trace specific containers in a pod.
apiVersion: v1
kind: Namespace
metadata:
name: multi-container-example
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: multi-container-app
namespace: multi-container-example
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: multi-container-app
namespace: multi-container-example
labels:
app: multi-container-demo
spec:
replicas: 1
selector:
matchLabels:
app: multi-container-demo
template:
metadata:
labels:
app: multi-container-demo
annotations:
prometheus.io/scrape: "true"
prometheus.io/port: "9090"
prometheus.io/path: "/metrics"
spec:
serviceAccountName: multi-container-app
volumes:
- name: snoop-data
emptyDir: {}
- name: cgroup
hostPath:
path: /sys/fs/cgroup
type: Directory
- name: debugfs
hostPath:
path: /sys/kernel/debug
type: Directory
- name: shared-logs
emptyDir: {}
initContainers:
# This init container discovers the pod cgroup path
# The snoop container will later filter out its own cgroup
- name: cgroup-finder
image: busybox:latest
command:
- sh
- -c
- |
# Get pod cgroup (parent directory of our cgroup)
SELF_CGROUP=$(cat /proc/self/cgroup | cut -d: -f3)
POD_CGROUP=$(dirname "$SELF_CGROUP")
echo "Pod cgroup: $POD_CGROUP"
echo "$POD_CGROUP" > /snoop-data/pod-cgroup
volumeMounts:
- name: snoop-data
mountPath: /snoop-data
containers:
# Main application: nginx web server
- name: nginx
image: nginx:1.25-alpine
ports:
- name: http
containerPort: 80
volumeMounts:
- name: shared-logs
mountPath: /var/log/nginx
resources:
requests:
cpu: 100m
memory: 64Mi
limits:
cpu: 500m
memory: 128Mi
# Log shipper sidecar (simulated - in practice this might be fluentd, filebeat, etc.)
- name: log-shipper
image: busybox:latest
command:
- sh
- -c
- |
echo "Log shipper starting..."
while true; do
# Simulate reading logs
if [ -f /logs/access.log ]; then
tail -n 1 /logs/access.log 2>/dev/null || true
fi
sleep 5
done
volumeMounts:
- name: shared-logs
mountPath: /logs
readOnly: true
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
cpu: 50m
memory: 32Mi
# Snoop sidecar - configured to trace ALL containers EXCEPT itself
- name: snoop
image: ghcr.io/imjasonh/snoop:latest
imagePullPolicy: Always
securityContext:
privileged: false
capabilities:
add:
- SYS_ADMIN
- BPF
- PERFMON
readOnlyRootFilesystem: true
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
# Use a shell wrapper to discover containers and exclude self
command:
- sh
- -c
- |
set -e
echo "Discovering container cgroups..."
POD_CGROUP=$(cat /data/pod-cgroup)
SELF_CGROUP=$(cat /proc/self/cgroup | cut -d: -f3)
echo "Pod cgroup: $POD_CGROUP"
echo "Self cgroup: $SELF_CGROUP"
# Build list of all container cgroups except our own
CGROUPS=""
cd "/sys/fs/cgroup$POD_CGROUP"
for dir in */; do
if [ ! -d "$dir" ]; then
continue
fi
# Remove trailing slash
dir_name="${dir%/}"
CGROUP_PATH="$POD_CGROUP/$dir_name"
FULL_PATH="/sys/fs/cgroup$CGROUP_PATH"
# Skip our own cgroup
if [ "$FULL_PATH" = "/sys/fs/cgroup$SELF_CGROUP" ]; then
echo "Skipping self: $CGROUP_PATH"
continue
fi
# Add to list
if [ -z "$CGROUPS" ]; then
CGROUPS="$FULL_PATH"
else
CGROUPS="$CGROUPS,$FULL_PATH"
fi
echo "Will trace: $CGROUP_PATH"
done
if [ -z "$CGROUPS" ]; then
echo "ERROR: No cgroups found to trace"
exit 1
fi
echo "Starting snoop with cgroups: $CGROUPS"
# Start snoop with discovered cgroups
exec /usr/local/bin/snoop \
-cgroups="$CGROUPS" \
-report=/data/snoop-report.json \
-interval=30s \
-exclude=/proc/,/sys/,/dev/,/var/log/nginx \
-metrics-addr=:9090 \
-log-level=info \
-max-unique-files=50000 \
-container-id="$POD_NAME" \
-image=nginx:1.25-alpine
volumeMounts:
- name: snoop-data
mountPath: /data
- name: cgroup
mountPath: /sys/fs/cgroup
readOnly: true
- name: debugfs
mountPath: /sys/kernel/debug
readOnly: true
ports:
- name: metrics
containerPort: 9090
protocol: TCP
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 200m
memory: 128Mi
livenessProbe:
httpGet:
path: /healthz
port: 9090
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 3
readinessProbe:
httpGet:
path: /healthz
port: 9090
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 3
---
apiVersion: v1
kind: Service
metadata:
name: multi-container-app
namespace: multi-container-example
spec:
type: ClusterIP
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: metrics
port: 9090
targetPort: 9090
protocol: TCP
selector:
app: multi-container-demo