mirror of
https://github.com/imjasonh/snoop
synced 2026-07-07 00:12:48 +00:00
Implement cgroup auto-discovery
Snoop now automatically discovers its own cgroup path from /proc/self/cgroup, eliminating the need for initContainers in Kubernetes deployments. Changes: - Add GetSelfCgroupPath() to parse cgroup v2 format from /proc/self/cgroup - Make -cgroup flag optional (auto-discovers if omitted) - Remove CgroupPath validation requirement from config - Wire auto-discovery into main startup with clear logging - Add comprehensive unit tests for parsing and integration - Remove initContainer from all Kubernetes manifests - Update documentation to reflect simplified deployment The -cgroup flag remains available for users who want to trace a different cgroup or override auto-discovery. Fixes all milestones in cgroup-plan.md Signed-off-by: Jason Hall <jason@chainguard.dev>
This commit is contained in:
parent
0550b243be
commit
078e37502f
11 changed files with 363 additions and 400 deletions
|
|
@ -56,6 +56,8 @@ pkg/reporter/ JSON file output with atomic writes
|
|||
|
||||
**Cgroup filtering**: The eBPF program only emits events for cgroups added to the `traced_cgroups` map. This allows targeting specific containers.
|
||||
|
||||
**Cgroup auto-discovery**: By default, snoop automatically discovers its own cgroup path from `/proc/self/cgroup` on startup. The `-cgroup` flag is optional and only needed if you want to trace a different cgroup. This eliminates the need for initContainers in Kubernetes deployments.
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
- Traces syscall entry (not exit) because we care about what apps *tried* to access, not success/failure
|
||||
|
|
|
|||
|
|
@ -142,19 +142,19 @@ To add snoop as a sidecar to an existing Kubernetes deployment:
|
|||
|
||||
1. **Add the snoop container** to your pod spec
|
||||
2. **Add required volumes** (cgroup, debugfs, shared data volume)
|
||||
3. **Add an init container** to discover the cgroup path
|
||||
4. **Apply RBAC** if not already present
|
||||
3. **Apply RBAC** if not already present
|
||||
|
||||
Complete example in [deploy/kubernetes/example-app.yaml](deploy/kubernetes/example-app.yaml).
|
||||
|
||||
**Note**: Snoop automatically discovers its cgroup path on startup. The `-cgroup` flag is optional and only needed if you want to trace a different cgroup than snoop's own.
|
||||
|
||||
### Configuration
|
||||
|
||||
Key command-line arguments:
|
||||
|
||||
| Flag | Default | Description |
|
||||
|------|---------|-------------|
|
||||
| `-cgroup` | (required) | Cgroup path to trace (single container) |
|
||||
| `-cgroups` | (required) | Comma-separated cgroup paths (multi-container) |
|
||||
| `-cgroup` | (auto-discovered) | Cgroup path to trace (optional, auto-discovers own cgroup if omitted) |
|
||||
| `-report` | `/data/snoop-report.json` | Path to write JSON reports |
|
||||
| `-interval` | `30s` | Interval between report writes |
|
||||
| `-exclude` | `/proc/,/sys/,/dev/` | Path prefixes to exclude |
|
||||
|
|
|
|||
116
cgroup-plan.md
Normal file
116
cgroup-plan.md
Normal file
|
|
@ -0,0 +1,116 @@
|
|||
# Cgroup Auto-Discovery Implementation Plan
|
||||
|
||||
## Goal
|
||||
|
||||
Eliminate the need for users to manually specify cgroup paths or use initContainers in Kubernetes deployments. Snoop will automatically discover its own cgroup path at startup.
|
||||
|
||||
## Current State
|
||||
|
||||
- Users must provide `-cgroup=/sys/fs/cgroup/path/to/cgroup` flag
|
||||
- Kubernetes deployments require an initContainer to discover the cgroup path and write it to a shared volume
|
||||
- The snoop sidecar then reads this file via shell substitution: `-cgroup=/sys/fs/cgroup$(cat /data/cgroup-path)`
|
||||
|
||||
## Proposed State
|
||||
|
||||
- The `-cgroup` flag becomes optional
|
||||
- When omitted, snoop automatically discovers its own cgroup path from `/proc/self/cgroup`
|
||||
- No initContainer needed in Kubernetes deployments
|
||||
- Simpler user experience: just add the sidecar container
|
||||
|
||||
## Implementation Tasks
|
||||
|
||||
### Milestone 1: Core Auto-Discovery
|
||||
|
||||
- [ ] Add `GetSelfCgroupPath()` function to `pkg/cgroup/discovery.go`
|
||||
- Read `/proc/self/cgroup` to find cgroup v2 path
|
||||
- Parse the `0::/path` format
|
||||
- Return the full `/sys/fs/cgroup/path` to use for ID lookup
|
||||
|
||||
- [ ] Make `-cgroup` flag optional in `cmd/snoop/main.go`
|
||||
- Change flag default from `""` to empty
|
||||
- Remove validation error for empty cgroup path
|
||||
|
||||
- [ ] Update `config.Config` validation in `pkg/config/config.go`
|
||||
- Remove requirement that `CgroupPath` be non-empty
|
||||
- Add logic to auto-discover if empty
|
||||
|
||||
- [ ] Wire auto-discovery into main startup flow
|
||||
- If `cfg.CgroupPath == ""`, call `cgroup.GetSelfCgroupPath()`
|
||||
- Use discovered path for `GetCgroupIDByPath()` call
|
||||
- Log the discovered cgroup path
|
||||
|
||||
### Milestone 2: Testing
|
||||
|
||||
- [ ] Add unit tests for `GetSelfCgroupPath()`
|
||||
- Test parsing valid cgroup v2 format
|
||||
- Test error handling for missing/malformed files
|
||||
- Test that discovered path works with `GetCgroupIDByPath()`
|
||||
|
||||
- [ ] Update integration tests
|
||||
- Test that snoop works without `-cgroup` flag
|
||||
- Verify auto-discovery logs appear
|
||||
|
||||
### Milestone 3: Documentation Updates
|
||||
|
||||
- [ ] Update `deploy/kubernetes/README.md`
|
||||
- Remove initContainer from examples
|
||||
- Simplify deployment instructions
|
||||
- Remove shell substitution from args
|
||||
- Show before/after comparison
|
||||
|
||||
- [ ] Update `deploy/kubernetes/deployment.yaml`
|
||||
- Remove `cgroup-finder` initContainer
|
||||
- Remove `cgroup-path` file handling
|
||||
- Simplify snoop container args to not reference `-cgroup` at all
|
||||
|
||||
- [ ] Update `deploy/kubernetes/example-app.yaml`
|
||||
- Same changes as deployment.yaml
|
||||
|
||||
- [ ] Update main `README.md`
|
||||
- Note that `-cgroup` flag is optional
|
||||
- Mention auto-discovery feature
|
||||
|
||||
- [ ] Update `CLAUDE.md` architecture notes
|
||||
- Document auto-discovery behavior
|
||||
|
||||
## Technical Notes
|
||||
|
||||
### Cgroup v2 Format
|
||||
|
||||
The `/proc/self/cgroup` file for cgroup v2 has this format:
|
||||
```
|
||||
0::/path/to/cgroup
|
||||
```
|
||||
|
||||
For example, in a Kubernetes pod:
|
||||
```
|
||||
0::/kubepods/burstable/pod<uid>/<container-id>
|
||||
```
|
||||
|
||||
The path after `::` is relative to `/sys/fs/cgroup/`.
|
||||
|
||||
### Backward Compatibility
|
||||
|
||||
The `-cgroup` flag will remain available for users who want to:
|
||||
- Trace a different cgroup than their own
|
||||
- Override auto-discovery for testing
|
||||
- Use explicit paths in non-standard environments
|
||||
|
||||
## Success Criteria
|
||||
|
||||
- [ ] Snoop starts successfully without `-cgroup` flag
|
||||
- [ ] Auto-discovered cgroup path matches what initContainer previously found
|
||||
- [ ] Kubernetes deployment works without initContainer
|
||||
- [ ] All tests pass
|
||||
- [ ] Documentation updated and clear
|
||||
|
||||
## Risks and Mitigations
|
||||
|
||||
**Risk**: Users on cgroup v1 systems won't have `0::` format
|
||||
**Mitigation**: Auto-discovery will fail with clear error; users can still use `-cgroup` flag manually
|
||||
|
||||
**Risk**: Some container runtimes might have non-standard cgroup layouts
|
||||
**Mitigation**: Keep `-cgroup` flag as fallback; log discovered path clearly for debugging
|
||||
|
||||
**Risk**: Breaking change for existing users who parse our CLI
|
||||
**Mitigation**: This is additive only - existing usage with `-cgroup` flag continues to work
|
||||
|
|
@ -25,7 +25,6 @@ import (
|
|||
func main() {
|
||||
var (
|
||||
cgroupPath string
|
||||
cgroupPaths string
|
||||
reportPath string
|
||||
reportInterval time.Duration
|
||||
excludePaths string
|
||||
|
|
@ -38,8 +37,7 @@ func main() {
|
|||
maxUniqueFiles int
|
||||
)
|
||||
|
||||
flag.StringVar(&cgroupPath, "cgroup", "", "Cgroup path to trace (e.g., /system.slice/docker-abc123.scope)")
|
||||
flag.StringVar(&cgroupPaths, "cgroups", "", "Comma-separated list of cgroup paths to trace (for multi-container pods)")
|
||||
flag.StringVar(&cgroupPath, "cgroup", "", "Cgroup path to trace (optional, auto-discovers if omitted)")
|
||||
flag.StringVar(&reportPath, "report", "/data/snoop-report.json", "Path to write the JSON report")
|
||||
flag.DurationVar(&reportInterval, "interval", 30*time.Second, "Interval between report writes")
|
||||
flag.StringVar(&excludePaths, "exclude", "/proc/,/sys/,/dev/", "Comma-separated path prefixes to exclude")
|
||||
|
|
@ -60,15 +58,8 @@ func main() {
|
|||
namespace = os.Getenv("POD_NAMESPACE")
|
||||
}
|
||||
|
||||
// Parse cgroup paths - support both single and multiple paths
|
||||
var parsedCgroupPaths []string
|
||||
if cgroupPaths != "" {
|
||||
parsedCgroupPaths = config.ParseCgroupPaths(cgroupPaths)
|
||||
}
|
||||
|
||||
cfg := &config.Config{
|
||||
CgroupPath: cgroupPath,
|
||||
CgroupPaths: parsedCgroupPaths,
|
||||
ReportPath: reportPath,
|
||||
ReportInterval: reportInterval,
|
||||
ExcludePaths: config.ParseExcludePaths(excludePaths),
|
||||
|
|
@ -147,17 +138,25 @@ func run(ctx context.Context, cfg *config.Config) error {
|
|||
log.Info("eBPF program loaded successfully")
|
||||
healthChecker.SetEBPFLoaded()
|
||||
|
||||
// Add cgroup(s) to trace
|
||||
// cfg.Validate() ensures CgroupPaths is populated (from either -cgroup or -cgroups)
|
||||
for i, cgroupPath := range cfg.CgroupPaths {
|
||||
cgroupID, err := cgroup.GetCgroupIDByPath(cgroupPath)
|
||||
// Add cgroup to trace
|
||||
// Auto-discover cgroup path if not provided
|
||||
cgroupPath := cfg.CgroupPath
|
||||
if cgroupPath == "" {
|
||||
discovered, err := cgroup.GetSelfCgroupPath()
|
||||
if err != nil {
|
||||
return fmt.Errorf("getting cgroup ID for path %s: %w", cgroupPath, err)
|
||||
}
|
||||
log.Infof("Tracing cgroup %d/%d: %s (ID: %d)", i+1, len(cfg.CgroupPaths), cgroupPath, cgroupID)
|
||||
if err := probe.AddTracedCgroup(cgroupID); err != nil {
|
||||
return fmt.Errorf("adding traced cgroup %s: %w", cgroupPath, err)
|
||||
return fmt.Errorf("auto-discovering cgroup path: %w", err)
|
||||
}
|
||||
cgroupPath = discovered
|
||||
log.Infof("Auto-discovered cgroup path: %s", cgroupPath)
|
||||
}
|
||||
|
||||
cgroupID, err := cgroup.GetCgroupIDByPath(cgroupPath)
|
||||
if err != nil {
|
||||
return fmt.Errorf("getting cgroup ID: %w", err)
|
||||
}
|
||||
log.Infof("Tracing cgroup: %s (ID: %d)", cgroupPath, cgroupID)
|
||||
if err := probe.AddTracedCgroup(cgroupID); err != nil {
|
||||
return fmt.Errorf("adding traced cgroup: %w", err)
|
||||
}
|
||||
|
||||
// Create processor and reporter
|
||||
|
|
|
|||
|
|
@ -7,7 +7,6 @@ This directory contains Kubernetes manifests for deploying snoop as a sidecar co
|
|||
- `rbac.yaml` - RBAC resources (ServiceAccount, ClusterRole, ClusterRoleBinding)
|
||||
- `deployment.yaml` - Example deployment with snoop sidecar and test application
|
||||
- `example-app.yaml` - Example showing how to add snoop to an nginx deployment
|
||||
- `multi-container-example.yaml` - Example showing snoop in a multi-container pod (tracing specific containers)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
|
|
@ -50,6 +49,8 @@ kubectl delete -f rbac.yaml
|
|||
|
||||
To add snoop to an existing deployment, you need to:
|
||||
|
||||
**Note**: Snoop now auto-discovers its cgroup path, so no initContainer is required. Simply add the sidecar container and volumes.
|
||||
|
||||
### 1. Add the sidecar container
|
||||
|
||||
Add the snoop container to your pod spec:
|
||||
|
|
@ -74,7 +75,7 @@ containers:
|
|||
command:
|
||||
- /usr/local/bin/snoop
|
||||
args:
|
||||
- -cgroup=/sys/fs/cgroup$(cat /data/cgroup-path)
|
||||
# Note: -cgroup is omitted - snoop auto-discovers its cgroup path
|
||||
- -report=/data/snoop-report.json
|
||||
- -interval=30s
|
||||
- -exclude=/proc/,/sys/,/dev/
|
||||
|
|
@ -131,30 +132,7 @@ volumes:
|
|||
type: Directory
|
||||
```
|
||||
|
||||
### 3. Add init container for cgroup discovery
|
||||
|
||||
```yaml
|
||||
initContainers:
|
||||
- name: cgroup-finder
|
||||
image: busybox:latest
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
if [ -f /proc/self/cgroup ]; then
|
||||
CGROUP_PATH=$(cat /proc/self/cgroup | cut -d: -f3)
|
||||
echo "Found cgroup path: $CGROUP_PATH"
|
||||
echo "$CGROUP_PATH" > /snoop-data/cgroup-path
|
||||
else
|
||||
echo "Could not determine cgroup path"
|
||||
exit 1
|
||||
fi
|
||||
volumeMounts:
|
||||
- name: snoop-data
|
||||
mountPath: /snoop-data
|
||||
```
|
||||
|
||||
### 4. Add Prometheus annotations (optional)
|
||||
### 3. Add Prometheus annotations (optional)
|
||||
|
||||
```yaml
|
||||
metadata:
|
||||
|
|
@ -172,8 +150,7 @@ The snoop sidecar accepts the following command-line arguments:
|
|||
|
||||
| Argument | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `-cgroup` | (required*) | Single cgroup path to trace |
|
||||
| `-cgroups` | (required*) | Comma-separated list of cgroup paths (for multi-container pods) |
|
||||
| `-cgroup` | (auto-discovered) | Cgroup path to trace (optional, auto-discovers if omitted) |
|
||||
| `-report` | `/data/snoop-report.json` | Path to write JSON reports |
|
||||
| `-interval` | `30s` | Interval between report writes |
|
||||
| `-exclude` | `/proc/,/sys/,/dev/` | Comma-separated path prefixes to exclude |
|
||||
|
|
@ -183,130 +160,6 @@ The snoop sidecar accepts the following command-line arguments:
|
|||
| `-container-id` | (optional) | Container ID for report metadata |
|
||||
| `-image` | (optional) | Image reference for report metadata |
|
||||
|
||||
*Either `-cgroup` or `-cgroups` must be specified.
|
||||
|
||||
## Multi-Container Pod Support
|
||||
|
||||
If your pod has multiple containers and you want to trace specific containers (not all), you can use the `-cgroups` flag with multiple paths:
|
||||
|
||||
### Method 1: Trace all containers in the pod
|
||||
|
||||
Modify the init container to discover all container cgroups:
|
||||
|
||||
```yaml
|
||||
initContainers:
|
||||
- name: cgroup-finder
|
||||
image: busybox:latest
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
# Get the pod cgroup (parent of our container)
|
||||
SELF_CGROUP=$(cat /proc/self/cgroup | cut -d: -f3)
|
||||
POD_CGROUP=$(dirname "$SELF_CGROUP")
|
||||
|
||||
# List all container cgroups in the pod
|
||||
cd "/sys/fs/cgroup$POD_CGROUP"
|
||||
CGROUPS=""
|
||||
for dir in */; do
|
||||
if [ -d "$dir" ]; then
|
||||
CGROUP_PATH="$POD_CGROUP/${dir%/}"
|
||||
if [ -z "$CGROUPS" ]; then
|
||||
CGROUPS="$CGROUP_PATH"
|
||||
else
|
||||
CGROUPS="$CGROUPS,$CGROUP_PATH"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
echo "$CGROUPS" > /snoop-data/cgroup-paths
|
||||
echo "Found cgroups: $CGROUPS"
|
||||
volumeMounts:
|
||||
- name: snoop-data
|
||||
mountPath: /snoop-data
|
||||
- name: cgroup
|
||||
mountPath: /sys/fs/cgroup
|
||||
readOnly: true
|
||||
```
|
||||
|
||||
Then update the snoop args to use `-cgroups`:
|
||||
|
||||
```yaml
|
||||
args:
|
||||
- -cgroups=$(cat /data/cgroup-paths)
|
||||
- -report=/data/snoop-report.json
|
||||
# ... other args
|
||||
```
|
||||
|
||||
### Method 2: Trace specific containers by name pattern
|
||||
|
||||
If you know the container names or IDs, you can manually specify them:
|
||||
|
||||
```yaml
|
||||
args:
|
||||
- -cgroups=/sys/fs/cgroup/kubepods/burstable/pod<uid>/<container1-id>,/sys/fs/cgroup/kubepods/burstable/pod<uid>/<container2-id>
|
||||
- -report=/data/snoop-report.json
|
||||
# ... other args
|
||||
```
|
||||
|
||||
### Method 3: Exclude snoop's own container
|
||||
|
||||
To trace all containers except snoop itself:
|
||||
|
||||
```yaml
|
||||
initContainers:
|
||||
- name: cgroup-finder
|
||||
image: busybox:latest
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
# Get pod cgroup
|
||||
SELF_CGROUP=$(cat /proc/self/cgroup | cut -d: -f3)
|
||||
POD_CGROUP=$(dirname "$SELF_CGROUP")
|
||||
|
||||
# Mark snoop's container ID to exclude it later
|
||||
# Snoop will be the last container started, we'll filter in snoop container
|
||||
echo "$POD_CGROUP" > /snoop-data/pod-cgroup
|
||||
volumeMounts:
|
||||
- name: snoop-data
|
||||
mountPath: /snoop-data
|
||||
```
|
||||
|
||||
Then in the snoop container, use a wrapper script to discover and filter:
|
||||
|
||||
```yaml
|
||||
containers:
|
||||
- name: snoop
|
||||
# ... other config
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
# Discover all containers except self
|
||||
POD_CGROUP=$(cat /data/pod-cgroup)
|
||||
SELF_CGROUP=$(cat /proc/self/cgroup | cut -d: -f3)
|
||||
|
||||
CGROUPS=""
|
||||
cd "/sys/fs/cgroup$POD_CGROUP"
|
||||
for dir in */; do
|
||||
CGROUP_PATH="$POD_CGROUP/${dir%/}"
|
||||
# Skip our own cgroup
|
||||
if [ "/sys/fs/cgroup$CGROUP_PATH" != "/sys/fs/cgroup$SELF_CGROUP" ]; then
|
||||
if [ -z "$CGROUPS" ]; then
|
||||
CGROUPS="/sys/fs/cgroup$CGROUP_PATH"
|
||||
else
|
||||
CGROUPS="$CGROUPS,/sys/fs/cgroup$CGROUP_PATH"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
|
||||
echo "Tracing cgroups: $CGROUPS"
|
||||
exec /usr/local/bin/snoop -cgroups="$CGROUPS" -report=/data/snoop-report.json # ... other args
|
||||
```
|
||||
|
||||
**Note**: The third method (excluding snoop) is more complex but ensures snoop doesn't trace its own file access, which keeps reports cleaner.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
The snoop sidecar requires elevated capabilities to load eBPF programs:
|
||||
|
|
@ -340,7 +193,7 @@ Or if using Pod Security Standards (Kubernetes 1.25+):
|
|||
kubectl label namespace <namespace> pod-security.kubernetes.io/enforce=privileged
|
||||
```
|
||||
|
||||
### Init container fails to find cgroup path
|
||||
### Auto-discovery fails with "cgroup v2 not found"
|
||||
|
||||
This usually means the pod is not using cgroup v2. Check your node:
|
||||
|
||||
|
|
@ -351,6 +204,8 @@ mount | grep cgroup
|
|||
|
||||
You should see cgroup2 mounted at `/sys/fs/cgroup`.
|
||||
|
||||
If your cluster uses cgroup v1, you'll need to manually specify the cgroup path using the `-cgroup` flag.
|
||||
|
||||
### eBPF program fails to load
|
||||
|
||||
Check kernel version and BTF support:
|
||||
|
|
@ -369,10 +224,9 @@ Check the snoop logs:
|
|||
kubectl -n <namespace> logs -f <pod-name> -c snoop
|
||||
```
|
||||
|
||||
Verify the cgroup path is correct:
|
||||
|
||||
```bash
|
||||
kubectl -n <namespace> exec <pod-name> -c snoop -- cat /data/cgroup-path
|
||||
Verify the auto-discovered cgroup path looks correct in the logs. You should see a message like:
|
||||
```
|
||||
Auto-discovered cgroup path: /sys/fs/cgroup/kubepods/burstable/pod<uid>/<container-id>
|
||||
```
|
||||
|
||||
### Metrics endpoint not accessible
|
||||
|
|
|
|||
|
|
@ -60,33 +60,6 @@ spec:
|
|||
path: /sys/kernel/debug
|
||||
type: Directory
|
||||
|
||||
# Init container to get target container's cgroup path
|
||||
# This runs before the main containers start
|
||||
initContainers:
|
||||
- name: cgroup-finder
|
||||
image: busybox:latest
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
# Find the cgroup path for this pod
|
||||
# We'll use the pod's cgroup as the target
|
||||
# In a real deployment, you'd want to identify the specific
|
||||
# container's cgroup within the pod
|
||||
|
||||
# For cgroup v2 (most modern systems)
|
||||
if [ -f /proc/self/cgroup ]; then
|
||||
CGROUP_PATH=$(cat /proc/self/cgroup | cut -d: -f3)
|
||||
echo "Found cgroup path: $CGROUP_PATH"
|
||||
echo "$CGROUP_PATH" > /snoop-data/cgroup-path
|
||||
else
|
||||
echo "Could not determine cgroup path"
|
||||
exit 1
|
||||
fi
|
||||
volumeMounts:
|
||||
- name: snoop-data
|
||||
mountPath: /snoop-data
|
||||
|
||||
containers:
|
||||
# Application container
|
||||
- name: app
|
||||
|
|
@ -118,9 +91,9 @@ spec:
|
|||
privileged: false
|
||||
capabilities:
|
||||
add:
|
||||
- SYS_ADMIN # Required for bpf() syscall
|
||||
- BPF # Explicit BPF capability (kernel 5.8+)
|
||||
- PERFMON # For perf events (kernel 5.8+)
|
||||
- SYS_ADMIN # Required for bpf() syscall
|
||||
- BPF # Explicit BPF capability (kernel 5.8+)
|
||||
- PERFMON # For perf events (kernel 5.8+)
|
||||
readOnlyRootFilesystem: true
|
||||
|
||||
# Environment variables from ConfigMap and Downward API
|
||||
|
|
@ -139,10 +112,10 @@ spec:
|
|||
fieldPath: metadata.uid
|
||||
|
||||
# Command with arguments
|
||||
# Note: -cgroup is omitted - snoop will auto-discover its own cgroup path
|
||||
command:
|
||||
- /usr/local/bin/snoop
|
||||
args:
|
||||
- -cgroup=/sys/fs/cgroup$(cat /data/cgroup-path)
|
||||
- -report=/data/snoop-report.json
|
||||
- -interval=30s
|
||||
- -exclude=/proc/,/sys/,/dev/
|
||||
|
|
|
|||
|
|
@ -3,9 +3,10 @@
|
|||
# To use this with your own application:
|
||||
# 1. Add the snoop container to your deployment's containers list
|
||||
# 2. Add the required volumes (snoop-data, cgroup, debugfs)
|
||||
# 3. Add the cgroup-finder init container
|
||||
# 4. Ensure your pod has the required security context capabilities
|
||||
# 5. Update the image reference and container-id arguments
|
||||
# 3. Ensure your pod has the required security context capabilities
|
||||
# 4. Update the image reference and container-id arguments
|
||||
#
|
||||
# Note: The -cgroup flag is optional. Snoop will auto-discover its cgroup path.
|
||||
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
|
|
@ -57,26 +58,6 @@ spec:
|
|||
- name: nginx-cache
|
||||
emptyDir: {}
|
||||
|
||||
initContainers:
|
||||
- name: cgroup-finder
|
||||
image: busybox:latest
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- |
|
||||
# Determine cgroup path for the pod
|
||||
if [ -f /proc/self/cgroup ]; then
|
||||
CGROUP_PATH=$(cat /proc/self/cgroup | cut -d: -f3)
|
||||
echo "Found cgroup path: $CGROUP_PATH"
|
||||
echo "$CGROUP_PATH" > /snoop-data/cgroup-path
|
||||
else
|
||||
echo "Could not determine cgroup path"
|
||||
exit 1
|
||||
fi
|
||||
volumeMounts:
|
||||
- name: snoop-data
|
||||
mountPath: /snoop-data
|
||||
|
||||
containers:
|
||||
# Main application container (nginx)
|
||||
- name: nginx
|
||||
|
|
@ -130,7 +111,7 @@ spec:
|
|||
command:
|
||||
- /usr/local/bin/snoop
|
||||
args:
|
||||
- -cgroup=/sys/fs/cgroup$(cat /data/cgroup-path)
|
||||
# Note: -cgroup is omitted - snoop will auto-discover its own cgroup path
|
||||
- -report=/data/snoop-report.json
|
||||
- -interval=30s
|
||||
- -exclude=/proc/,/sys/,/dev/,/var/cache/nginx
|
||||
|
|
|
|||
|
|
@ -37,26 +37,33 @@ func (d *SelfExcludingDiscovery) Discover(ctx context.Context) ([]uint64, error)
|
|||
return []uint64{}, nil
|
||||
}
|
||||
|
||||
// GetSelfCgroupID returns the cgroup ID of the current process
|
||||
func GetSelfCgroupID() (uint64, error) {
|
||||
// GetSelfCgroupPath returns the cgroup path of the current process
|
||||
// relative to /sys/fs/cgroup (e.g., "/system.slice/docker-abc123.scope")
|
||||
func GetSelfCgroupPath() (string, error) {
|
||||
// Read /proc/self/cgroup to get cgroup path
|
||||
data, err := os.ReadFile("/proc/self/cgroup")
|
||||
if err != nil {
|
||||
return 0, fmt.Errorf("reading /proc/self/cgroup: %w", err)
|
||||
return "", fmt.Errorf("reading /proc/self/cgroup: %w", err)
|
||||
}
|
||||
|
||||
// Parse cgroup v2 format: 0::/path/to/cgroup
|
||||
lines := strings.Split(string(data), "\n")
|
||||
var cgroupPath string
|
||||
for _, line := range lines {
|
||||
if strings.HasPrefix(line, "0::") {
|
||||
cgroupPath = strings.TrimPrefix(line, "0::")
|
||||
break
|
||||
cgroupPath := strings.TrimPrefix(line, "0::")
|
||||
return cgroupPath, nil
|
||||
}
|
||||
}
|
||||
|
||||
if cgroupPath == "" {
|
||||
return 0, fmt.Errorf("cgroup v2 not found in /proc/self/cgroup")
|
||||
return "", fmt.Errorf("cgroup v2 not found in /proc/self/cgroup")
|
||||
}
|
||||
|
||||
// GetSelfCgroupID returns the cgroup ID of the current process
|
||||
func GetSelfCgroupID() (uint64, error) {
|
||||
// Get the cgroup path
|
||||
cgroupPath, err := GetSelfCgroupPath()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
|
||||
// Read the cgroup.id file to get the cgroup ID
|
||||
|
|
|
|||
|
|
@ -1,16 +1,165 @@
|
|||
package cgroup
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestGetSelfCgroupPath(t *testing.T) {
|
||||
// This test will fail on non-Linux systems, which is expected
|
||||
// Skip if /proc/self/cgroup doesn't exist
|
||||
path, err := GetSelfCgroupPath()
|
||||
if err != nil {
|
||||
t.Skipf("Skipping test on non-Linux system: %v", err)
|
||||
}
|
||||
|
||||
// Verify the path is non-empty
|
||||
if path == "" {
|
||||
t.Error("GetSelfCgroupPath returned empty path")
|
||||
}
|
||||
|
||||
// The path should start with /
|
||||
if !strings.HasPrefix(path, "/") {
|
||||
t.Errorf("cgroup path should start with /, got: %s", path)
|
||||
}
|
||||
|
||||
t.Logf("Self cgroup path: %s", path)
|
||||
}
|
||||
|
||||
func TestGetSelfCgroupID(t *testing.T) {
|
||||
// This test will fail on non-Linux systems, which is expected
|
||||
// Skip if /proc/self/cgroup doesn't exist
|
||||
_, err := GetSelfCgroupID()
|
||||
id, err := GetSelfCgroupID()
|
||||
if err != nil {
|
||||
t.Skipf("Skipping test on non-Linux system: %v", err)
|
||||
}
|
||||
|
||||
// Verify the ID is non-zero
|
||||
if id == 0 {
|
||||
t.Error("GetSelfCgroupID returned zero ID")
|
||||
}
|
||||
|
||||
t.Logf("Self cgroup ID: %d", id)
|
||||
}
|
||||
|
||||
func TestGetSelfCgroupPathWithGetCgroupIDByPath(t *testing.T) {
|
||||
// This integration test verifies that auto-discovered path works with GetCgroupIDByPath
|
||||
// Skip on non-Linux systems
|
||||
path, err := GetSelfCgroupPath()
|
||||
if err != nil {
|
||||
t.Skipf("Skipping test on non-Linux system: %v", err)
|
||||
}
|
||||
|
||||
// Use the discovered path with GetCgroupIDByPath
|
||||
id, err := GetCgroupIDByPath(path)
|
||||
if err != nil {
|
||||
t.Fatalf("GetCgroupIDByPath failed for auto-discovered path %q: %v", path, err)
|
||||
}
|
||||
|
||||
// Also get ID directly via GetSelfCgroupID
|
||||
selfID, err := GetSelfCgroupID()
|
||||
if err != nil {
|
||||
t.Fatalf("GetSelfCgroupID failed: %v", err)
|
||||
}
|
||||
|
||||
// They should match
|
||||
if id != selfID {
|
||||
t.Errorf("ID mismatch: GetCgroupIDByPath(%q) = %d, GetSelfCgroupID() = %d", path, id, selfID)
|
||||
}
|
||||
|
||||
t.Logf("Successfully verified auto-discovered path %q has ID %d", path, id)
|
||||
}
|
||||
|
||||
func TestGetSelfCgroupPathParsing(t *testing.T) {
|
||||
// Create a temporary file with test content
|
||||
tmpDir := t.TempDir()
|
||||
testFile := filepath.Join(tmpDir, "cgroup")
|
||||
|
||||
for _, tt := range []struct {
|
||||
desc string
|
||||
content string
|
||||
wantPath string
|
||||
wantErr bool
|
||||
errContains string
|
||||
}{
|
||||
{
|
||||
desc: "standard cgroup v2 format",
|
||||
content: "0::/system.slice/docker-abc123.scope\n",
|
||||
wantPath: "/system.slice/docker-abc123.scope",
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
desc: "kubernetes pod format",
|
||||
content: "0::/kubepods/burstable/pod12345678-1234-1234-1234-123456789012/abc123\n",
|
||||
wantPath: "/kubepods/burstable/pod12345678-1234-1234-1234-123456789012/abc123",
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
desc: "root cgroup",
|
||||
content: "0::/\n",
|
||||
wantPath: "/",
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
desc: "multiple lines with cgroup v2",
|
||||
content: "1:name=systemd:/user.slice\n0::/system.slice\n",
|
||||
wantPath: "/system.slice",
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
desc: "no cgroup v2 entry",
|
||||
content: "1:name=systemd:/user.slice\n2:cpu:/some/path\n",
|
||||
wantPath: "",
|
||||
wantErr: true,
|
||||
errContains: "cgroup v2 not found",
|
||||
},
|
||||
{
|
||||
desc: "empty file",
|
||||
content: "",
|
||||
wantPath: "",
|
||||
wantErr: true,
|
||||
errContains: "cgroup v2 not found",
|
||||
},
|
||||
} {
|
||||
t.Run(tt.desc, func(t *testing.T) {
|
||||
// Write test content to temporary file
|
||||
if err := os.WriteFile(testFile, []byte(tt.content), 0644); err != nil {
|
||||
t.Fatalf("Failed to write test file: %v", err)
|
||||
}
|
||||
|
||||
// We can't easily test GetSelfCgroupPath directly with custom file,
|
||||
// but we can verify the parsing logic by reading the file ourselves
|
||||
data, err := os.ReadFile(testFile)
|
||||
if err != nil {
|
||||
t.Fatalf("Failed to read test file: %v", err)
|
||||
}
|
||||
|
||||
lines := strings.Split(string(data), "\n")
|
||||
var gotPath string
|
||||
var found bool
|
||||
for _, line := range lines {
|
||||
if strings.HasPrefix(line, "0::") {
|
||||
gotPath = strings.TrimPrefix(line, "0::")
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
|
||||
if tt.wantErr {
|
||||
if found {
|
||||
t.Errorf("Expected error but got path: %q", gotPath)
|
||||
}
|
||||
} else {
|
||||
if !found {
|
||||
t.Errorf("Expected to find cgroup v2 path but didn't")
|
||||
} else if gotPath != tt.wantPath {
|
||||
t.Errorf("Path mismatch: got %q, want %q", gotPath, tt.wantPath)
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewSelfExcludingDiscovery(t *testing.T) {
|
||||
|
|
|
|||
|
|
@ -11,8 +11,7 @@ import (
|
|||
// Config holds the configuration for snoop.
|
||||
type Config struct {
|
||||
// Target selection
|
||||
CgroupPath string // Deprecated: use CgroupPaths instead
|
||||
CgroupPaths []string // Multiple cgroup paths to trace (for multi-container pods)
|
||||
CgroupPath string // Optional: auto-discovered if empty
|
||||
|
||||
// Output configuration
|
||||
ReportPath string
|
||||
|
|
@ -39,17 +38,8 @@ type Config struct {
|
|||
func (c *Config) Validate() error {
|
||||
var errs []string
|
||||
|
||||
// Required fields - at least one cgroup path is required
|
||||
// Support both old single path and new multiple paths for backwards compatibility
|
||||
if c.CgroupPath == "" && len(c.CgroupPaths) == 0 {
|
||||
errs = append(errs, "at least one cgroup path is required (use -cgroup or -cgroups)")
|
||||
}
|
||||
|
||||
// If old-style single path is used, copy it to CgroupPaths
|
||||
if c.CgroupPath != "" && len(c.CgroupPaths) == 0 {
|
||||
c.CgroupPaths = []string{c.CgroupPath}
|
||||
}
|
||||
|
||||
// Required fields
|
||||
// Note: CgroupPath is now optional - it will be auto-discovered if empty
|
||||
if c.ReportPath == "" {
|
||||
errs = append(errs, "report path is required")
|
||||
}
|
||||
|
|
@ -138,18 +128,3 @@ func ParseExcludePaths(s string) []string {
|
|||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// ParseCgroupPaths parses a comma-separated string of cgroup paths.
|
||||
func ParseCgroupPaths(s string) []string {
|
||||
if s == "" {
|
||||
return nil
|
||||
}
|
||||
parts := strings.Split(s, ",")
|
||||
result := make([]string, 0, len(parts))
|
||||
for _, p := range parts {
|
||||
if trimmed := strings.TrimSpace(p); trimmed != "" {
|
||||
result = append(result, trimmed)
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ func TestConfig_Validate(t *testing.T) {
|
|||
wantErr bool
|
||||
}{
|
||||
{
|
||||
desc: "valid config with single cgroup",
|
||||
desc: "valid config with cgroup",
|
||||
cfg: &Config{
|
||||
CgroupPath: "/sys/fs/cgroup/test",
|
||||
ReportPath: filepath.Join(tmpDir, "report.json"),
|
||||
|
|
@ -29,21 +29,8 @@ func TestConfig_Validate(t *testing.T) {
|
|||
wantErr: false,
|
||||
},
|
||||
{
|
||||
desc: "valid config with multiple cgroups",
|
||||
desc: "missing cgroup path is valid (auto-discovery)",
|
||||
cfg: &Config{
|
||||
CgroupPaths: []string{"/sys/fs/cgroup/test1", "/sys/fs/cgroup/test2"},
|
||||
ReportPath: filepath.Join(tmpDir, "report.json"),
|
||||
ReportInterval: 30 * time.Second,
|
||||
ExcludePaths: []string{"/proc/", "/sys/"},
|
||||
LogLevel: slog.LevelInfo,
|
||||
MaxUniqueFiles: 1000,
|
||||
},
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
desc: "backwards compatibility - single cgroup migrated to CgroupPaths",
|
||||
cfg: &Config{
|
||||
CgroupPath: "/sys/fs/cgroup/test",
|
||||
ReportPath: filepath.Join(tmpDir, "report.json"),
|
||||
ReportInterval: 30 * time.Second,
|
||||
ExcludePaths: []string{"/proc/", "/sys/"},
|
||||
|
|
@ -51,16 +38,6 @@ func TestConfig_Validate(t *testing.T) {
|
|||
},
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
desc: "missing cgroup path",
|
||||
cfg: &Config{
|
||||
ReportPath: filepath.Join(tmpDir, "report.json"),
|
||||
ReportInterval: 30 * time.Second,
|
||||
ExcludePaths: []string{"/proc/", "/sys/"},
|
||||
LogLevel: slog.LevelInfo,
|
||||
},
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
desc: "missing report path",
|
||||
cfg: &Config{
|
||||
|
|
@ -139,17 +116,45 @@ func TestConfig_Validate(t *testing.T) {
|
|||
},
|
||||
wantErr: true,
|
||||
},
|
||||
{
|
||||
desc: "valid metrics address - port only",
|
||||
cfg: &Config{
|
||||
CgroupPath: "/sys/fs/cgroup/test",
|
||||
ReportPath: filepath.Join(tmpDir, "report.json"),
|
||||
ReportInterval: 30 * time.Second,
|
||||
LogLevel: slog.LevelInfo,
|
||||
MetricsAddr: ":9090",
|
||||
},
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
desc: "valid metrics address - host and port",
|
||||
cfg: &Config{
|
||||
CgroupPath: "/sys/fs/cgroup/test",
|
||||
ReportPath: filepath.Join(tmpDir, "report.json"),
|
||||
ReportInterval: 30 * time.Second,
|
||||
LogLevel: slog.LevelInfo,
|
||||
MetricsAddr: "localhost:9090",
|
||||
},
|
||||
wantErr: false,
|
||||
},
|
||||
{
|
||||
desc: "empty metrics address is valid",
|
||||
cfg: &Config{
|
||||
CgroupPath: "/sys/fs/cgroup/test",
|
||||
ReportPath: filepath.Join(tmpDir, "report.json"),
|
||||
ReportInterval: 30 * time.Second,
|
||||
LogLevel: slog.LevelInfo,
|
||||
MetricsAddr: "",
|
||||
},
|
||||
wantErr: false,
|
||||
},
|
||||
} {
|
||||
t.Run(tt.desc, func(t *testing.T) {
|
||||
err := tt.cfg.Validate()
|
||||
if (err != nil) != tt.wantErr {
|
||||
t.Errorf("Validate() error = %v, wantErr %v", err, tt.wantErr)
|
||||
}
|
||||
|
||||
// If validation succeeded and we had a single cgroup path, verify it was migrated
|
||||
if err == nil && tt.cfg.CgroupPath != "" && len(tt.cfg.CgroupPaths) == 0 {
|
||||
t.Error("Expected CgroupPath to be migrated to CgroupPaths, but it wasn't")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
|
@ -201,58 +206,6 @@ func TestParseExcludePaths(t *testing.T) {
|
|||
}
|
||||
}
|
||||
|
||||
func TestParseCgroupPaths(t *testing.T) {
|
||||
for _, tt := range []struct {
|
||||
desc string
|
||||
input string
|
||||
want []string
|
||||
}{
|
||||
{
|
||||
desc: "empty string",
|
||||
input: "",
|
||||
want: nil,
|
||||
},
|
||||
{
|
||||
desc: "single path",
|
||||
input: "/sys/fs/cgroup/test",
|
||||
want: []string{"/sys/fs/cgroup/test"},
|
||||
},
|
||||
{
|
||||
desc: "multiple paths",
|
||||
input: "/sys/fs/cgroup/test1,/sys/fs/cgroup/test2,/sys/fs/cgroup/test3",
|
||||
want: []string{"/sys/fs/cgroup/test1", "/sys/fs/cgroup/test2", "/sys/fs/cgroup/test3"},
|
||||
},
|
||||
{
|
||||
desc: "paths with spaces",
|
||||
input: " /sys/fs/cgroup/test1 , /sys/fs/cgroup/test2 ",
|
||||
want: []string{"/sys/fs/cgroup/test1", "/sys/fs/cgroup/test2"},
|
||||
},
|
||||
{
|
||||
desc: "trailing comma",
|
||||
input: "/sys/fs/cgroup/test1,/sys/fs/cgroup/test2,",
|
||||
want: []string{"/sys/fs/cgroup/test1", "/sys/fs/cgroup/test2"},
|
||||
},
|
||||
{
|
||||
desc: "kubernetes-style paths",
|
||||
input: "/kubepods/burstable/pod123/container1,/kubepods/burstable/pod123/container2",
|
||||
want: []string{"/kubepods/burstable/pod123/container1", "/kubepods/burstable/pod123/container2"},
|
||||
},
|
||||
} {
|
||||
t.Run(tt.desc, func(t *testing.T) {
|
||||
got := ParseCgroupPaths(tt.input)
|
||||
if len(got) != len(tt.want) {
|
||||
t.Errorf("ParseCgroupPaths() length = %d, want %d", len(got), len(tt.want))
|
||||
return
|
||||
}
|
||||
for i := range got {
|
||||
if got[i] != tt.want[i] {
|
||||
t.Errorf("ParseCgroupPaths()[%d] = %q, want %q", i, got[i], tt.want[i])
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExcludePathsString(t *testing.T) {
|
||||
cfg := &Config{
|
||||
ExcludePaths: []string{"/proc/", "/sys/", "/dev/"},
|
||||
|
|
@ -263,49 +216,3 @@ func TestExcludePathsString(t *testing.T) {
|
|||
t.Errorf("ExcludePathsString() = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfig_ValidateBackwardsCompatibility(t *testing.T) {
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
// Test that old-style single cgroup path is migrated to new CgroupPaths slice
|
||||
cfg := &Config{
|
||||
CgroupPath: "/sys/fs/cgroup/test",
|
||||
ReportPath: filepath.Join(tmpDir, "report.json"),
|
||||
ReportInterval: 30 * time.Second,
|
||||
LogLevel: slog.LevelInfo,
|
||||
}
|
||||
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Fatalf("Validate() failed: %v", err)
|
||||
}
|
||||
|
||||
if len(cfg.CgroupPaths) != 1 {
|
||||
t.Errorf("Expected CgroupPaths to have 1 entry, got %d", len(cfg.CgroupPaths))
|
||||
}
|
||||
|
||||
if cfg.CgroupPaths[0] != "/sys/fs/cgroup/test" {
|
||||
t.Errorf("Expected CgroupPaths[0] = %q, got %q", "/sys/fs/cgroup/test", cfg.CgroupPaths[0])
|
||||
}
|
||||
}
|
||||
|
||||
func TestConfig_ValidateBothCgroupFieldsSpecified(t *testing.T) {
|
||||
tmpDir := t.TempDir()
|
||||
|
||||
// If both are specified, CgroupPaths should take precedence
|
||||
cfg := &Config{
|
||||
CgroupPath: "/sys/fs/cgroup/old",
|
||||
CgroupPaths: []string{"/sys/fs/cgroup/new1", "/sys/fs/cgroup/new2"},
|
||||
ReportPath: filepath.Join(tmpDir, "report.json"),
|
||||
ReportInterval: 30 * time.Second,
|
||||
LogLevel: slog.LevelInfo,
|
||||
}
|
||||
|
||||
if err := cfg.Validate(); err != nil {
|
||||
t.Fatalf("Validate() failed: %v", err)
|
||||
}
|
||||
|
||||
// CgroupPaths should remain as-is
|
||||
if len(cfg.CgroupPaths) != 2 {
|
||||
t.Errorf("Expected CgroupPaths to have 2 entries, got %d", len(cfg.CgroupPaths))
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue