mirror of
https://github.com/imjasonh/snoop
synced 2026-07-07 00:12:48 +00:00
Merge pull request #7 from imjasonh/apk
document failed apk mapping experiment
This commit is contained in:
Jason Hall
GitHub
commit
faa8756293
1 changed files with 240 additions and 0 deletions
240
apk-limitation.md
Normal file
240
apk-limitation.md
Normal file
|
|
@ -0,0 +1,240 @@
|
|||
# APK Package Attribution - Technical Limitation
|
||||
|
||||
## The Idea
|
||||
|
||||
Extend snoop to attribute file accesses to APK packages in Alpine/Wolfi containers. This would provide actionable insights for image slimming by showing:
|
||||
- Which packages are actively used vs installed but unused
|
||||
- Per-package file access counts
|
||||
- Package utilization rates
|
||||
|
||||
This would help answer: "Can I remove this package?" or "Which packages are essential for my workload?"
|
||||
|
||||
## How We Would Have Done It
|
||||
|
||||
### Implementation Overview
|
||||
|
||||
1. **Parse APK Database** (`/lib/apk/db/installed`)
|
||||
- Read the installed package database
|
||||
- Build maps: package → files, file → package
|
||||
- Store package metadata (name, version, file count)
|
||||
|
||||
2. **Detect APK Databases**
|
||||
- During container discovery, find PIDs in each container's cgroup
|
||||
- Access the container's filesystem via `/proc/{pid}/root`
|
||||
- Read `/proc/{pid}/root/lib/apk/db/installed`
|
||||
- Parse and cache for the container
|
||||
|
||||
3. **Track Package Access**
|
||||
- When processing file access events, look up owning package
|
||||
- Increment per-package access counters (thread-safe)
|
||||
- Track which files in each package were accessed
|
||||
|
||||
4. **Report Package Statistics**
|
||||
- Include APK section in JSON reports per container:
|
||||
```json
|
||||
"apk_packages": [
|
||||
{
|
||||
"name": "curl",
|
||||
"version": "8.5.0-r0",
|
||||
"total_files": 45,
|
||||
"accessed_files": 12,
|
||||
"access_count": 234
|
||||
},
|
||||
{
|
||||
"name": "ca-certificates",
|
||||
"version": "20230506-r0",
|
||||
"total_files": 10,
|
||||
"accessed_files": 0,
|
||||
"access_count": 0
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
### Code Architecture
|
||||
|
||||
- `pkg/apk/parser.go` - Parse APK database format
|
||||
- `pkg/apk/mapper.go` - Thread-safe package access tracking
|
||||
- `pkg/cgroup/discovery.go` - Detect and read APK databases from containers
|
||||
- `pkg/reporter/reporter.go` - Include APK stats in reports
|
||||
- Integration in main event loop to record accesses
|
||||
|
||||
## Why It Doesn't Work
|
||||
|
||||
### The Fundamental Problem: Container Filesystem Isolation
|
||||
|
||||
In Kubernetes with containerd, **sidecar containers cannot access other containers' filesystems**, even with full capabilities. This is by design for security.
|
||||
|
||||
### What We Tried
|
||||
|
||||
1. **Direct `/proc/{pid}/root` Access**
|
||||
```go
|
||||
// Attempt to read: /proc/{pid}/root/lib/apk/db/installed
|
||||
data, err := os.ReadFile(filepath.Join("/proc", pid, "root", "lib/apk/db/installed"))
|
||||
```
|
||||
**Result**: `open /proc/{pid}/root/lib/apk/db/installed: no such file or directory`
|
||||
|
||||
Even though the file exists in the target container, the sidecar cannot see it through `/proc/{pid}/root`.
|
||||
|
||||
2. **Namespace Switching with `setns()`**
|
||||
```go
|
||||
unix.Setns(int(nsFile.Fd()), unix.CLONE_NEWNS)
|
||||
```
|
||||
**Result**: `entering mount namespace: invalid argument`
|
||||
|
||||
The `setns()` syscall fails because:
|
||||
- Requires same user namespace
|
||||
- Containers are in different user namespaces
|
||||
- Cannot cross namespace boundaries even with CAP_SYS_ADMIN
|
||||
|
||||
3. **Multiple Retry Attempts**
|
||||
- Waited for PIDs to appear (they do)
|
||||
- Tried multiple PIDs per container
|
||||
- Added delays for filesystem propagation
|
||||
|
||||
**Result**: PIDs exist, but filesystem still inaccessible
|
||||
|
||||
### Why This Happens
|
||||
|
||||
1. **Mount Namespace Isolation**
|
||||
- Each container has its own mount namespace
|
||||
- Snoop sees its own mount namespace, not other containers'
|
||||
- `/proc/{pid}/root` is a symlink that only works within the same namespace context
|
||||
|
||||
2. **User Namespace Boundaries**
|
||||
- Containers run in separate user namespaces
|
||||
- `setns(CLONE_NEWNS)` requires being in the same user namespace
|
||||
- Cannot cross user namespace boundaries from within a container
|
||||
|
||||
3. **Containerd Filesystem Structure**
|
||||
- Container filesystems use overlay mounts
|
||||
- Mount points are not visible across namespace boundaries
|
||||
- `/proc/mounts` shows different views per namespace
|
||||
|
||||
### Evidence from Testing
|
||||
|
||||
```
|
||||
DEBUG: APK detection - found 1 PIDs on attempt 2
|
||||
DEBUG: APK detection - found 1 PIDs on attempt 3
|
||||
DEBUG: APK detection - cannot read from PID 1764 namespace: entering mount namespace: invalid argument
|
||||
DEBUG: APK detection - no APK database found after 5 attempts
|
||||
```
|
||||
|
||||
**Translation**: We find the PIDs successfully, but cannot access their filesystems.
|
||||
|
||||
## Where It WOULD Work
|
||||
|
||||
### Environments Where This Works
|
||||
|
||||
1. **Docker / Docker Compose**
|
||||
- Containers share more of the host namespace
|
||||
- `/proc/{pid}/root` typically accessible with proper capabilities
|
||||
- Less strict namespace isolation
|
||||
|
||||
2. **Kubernetes DaemonSet with `hostPID: true`**
|
||||
```yaml
|
||||
spec:
|
||||
hostPID: true
|
||||
containers:
|
||||
- name: snoop
|
||||
securityContext:
|
||||
privileged: true
|
||||
```
|
||||
- Runs in host PID namespace
|
||||
- Can access all container filesystems via `/proc`
|
||||
- Requires elevated privileges
|
||||
|
||||
3. **Host-Level Deployment**
|
||||
- Running directly on the node (not in a container)
|
||||
- Full access to all container filesystems
|
||||
- Not practical for per-pod monitoring
|
||||
|
||||
### Why Sidecar Pattern Fails
|
||||
|
||||
The sidecar pattern (running snoop alongside app containers in the same pod) is incompatible with this approach because:
|
||||
- Sidecars are isolated from other containers for security
|
||||
- This is a **feature, not a bug** - prevents malicious sidecars from accessing sensitive data
|
||||
- No amount of capabilities or permissions changes this
|
||||
|
||||
## Alternative Approaches (Not Implemented)
|
||||
|
||||
### 1. Init Container Pattern
|
||||
Copy APK database to shared volume during init:
|
||||
```yaml
|
||||
initContainers:
|
||||
- name: copy-apk-db
|
||||
image: alpine
|
||||
command: ['cp', '/lib/apk/db/installed', '/shared/apk-db']
|
||||
volumeMounts:
|
||||
- name: shared
|
||||
mountPath: /shared
|
||||
```
|
||||
**Downsides**:
|
||||
- Requires modifying app deployments
|
||||
- Snapshot at init time, doesn't reflect runtime changes
|
||||
- Extra complexity
|
||||
|
||||
### 2. Admission Webhook
|
||||
Inject APK database path via annotations:
|
||||
```yaml
|
||||
annotations:
|
||||
snoop.io/apk-database-path: "/data/apk-db/installed"
|
||||
```
|
||||
**Downsides**:
|
||||
- Requires cluster-level configuration
|
||||
- Users must manually set up volume sharing
|
||||
- Not transparent
|
||||
|
||||
### 3. eBPF Filesystem Tracing
|
||||
Trace `openat()` syscalls to detect when app opens `/lib/apk/db/installed`, then extract data:
|
||||
**Downsides**:
|
||||
- App might never read its own APK database
|
||||
- Complex, fragile
|
||||
- Still faces same filesystem access issues
|
||||
|
||||
### 4. CRI/Container Runtime Integration
|
||||
Use containerd CRI API to query container root filesystems:
|
||||
**Downsides**:
|
||||
- Requires access to container runtime socket
|
||||
- Not available in standard pod deployments
|
||||
- Would need DaemonSet architecture anyway
|
||||
|
||||
## Conclusion
|
||||
|
||||
### What We Learned
|
||||
|
||||
APK package attribution is **architecturally incompatible with Kubernetes sidecar deployments** due to fundamental container isolation. This is not a bug or missing capability - it's a core security feature.
|
||||
|
||||
### Implementation Status
|
||||
|
||||
- ✅ Full implementation completed (parser, mapper, integration)
|
||||
- ✅ All unit tests pass (99/99)
|
||||
- ✅ Code is production-ready
|
||||
- ❌ Feature cannot function in target environment (Kubernetes sidecars)
|
||||
|
||||
### Recommendation
|
||||
|
||||
**Do not pursue this feature for Kubernetes sidecar deployments.** The workarounds all require:
|
||||
- Elevated privileges (hostPID, privileged containers)
|
||||
- DaemonSet architecture instead of sidecar
|
||||
- User-visible changes to their deployments
|
||||
|
||||
The value proposition (identifying unused packages) doesn't justify:
|
||||
- The complexity of alternative approaches
|
||||
- The security implications of required workarounds
|
||||
- The operational burden on users
|
||||
|
||||
### Alternative: File-Level Analysis
|
||||
|
||||
Instead of package attribution, focus on file-level insights:
|
||||
- Which specific files are accessed (already implemented)
|
||||
- File access frequency and patterns
|
||||
- Directories that are never touched
|
||||
|
||||
Users can correlate this with their package managers manually if needed. This provides actionable data without requiring filesystem access to other containers.
|
||||
|
||||
## References
|
||||
|
||||
- Kubernetes container isolation: https://kubernetes.io/docs/concepts/security/pod-security-standards/
|
||||
- Linux namespaces: https://man7.org/linux/man-pages/man7/namespaces.7.html
|
||||
- APK database format: https://wiki.alpinelinux.org/wiki/Apk_spec
|
||||
- Snoop APK implementation: `pkg/apk/` (to be reverted)
|
||||
Loading…
Add table
Add a link
Reference in a new issue