git-k8s/ASSESSMENT.md

git-k8s Codebase Assessment

Overall Verdict: Well-Architected Beta — Not Yet Production-Hardened

The codebase has excellent architecture (generic reconciler pattern, stateless design, clean CRD state machines) and strong security fundamentals (distroless images, non-root containers, least-privilege RBAC). However, it lacks observability, health checks, and adequate unit test coverage for production use.

---

Strengths

• Clean reconciler pattern — Generic KindReconciler[T] adapter eliminates boilerplate; each controller only implements ReconcileKind
• Stateless design — All Git ops use go-git with memory.NewStorage(), no PVCs needed, trivial horizontal scaling
• Strong container security — Distroless base, readOnlyRootFilesystem: true, runAsNonRoot: true, all capabilities dropped
• Proper RBAC — ClusterRole scoped to git-k8s.imjasonh.com API group, Secrets access is read-only
• Comprehensive E2E tests — Full KinD + Gitea setup covering push, sync, resolver, and repo watcher workflows
• Fresh dependencies — go-git v5.13.2, client-go v0.32.2, recent Knative build
• Clear state machines — Pending→InProgress→Succeeded/Failed for transactions, InSync/Syncing/Conflicted/RequiresManualIntervention for syncs

---

Critical Gaps

1. No Metrics (Production Blocker)

No Prometheus /metrics endpoint. Zero visibility into reconciliation latency, error rates, work queue depth, or Git operation duration. Operators cannot monitor controller health.

2. No Health Checks (Production Blocker)

No readiness or liveness probes on any controller deployment. Kubernetes cannot detect hung controllers or restart failed ones.

3. Unit Test Coverage ~10% (Quality Risk)

Core reconciler logic (push/push.go, sync/sync.go, resolver/resolver.go) has near-zero unit test coverage. E2E tests exist but can't replace targeted unit tests for edge cases, error paths, and race conditions.

4. In-Memory Clones Without Bounds

Every reconciliation clones the full repository into memory. No depth limits, no timeout on Git operations, no protection against OOM from large repos. Memory limits (128-256Mi) are too low for real-world repositories.

---

High-Priority Issues

Issue	Impact	Location
No SSH key support	Blocks enterprise adoption	pkg/reconciler/push/push.go:139-172
No retry/backoff for Git ops	Transient failures cause permanent Failed state	All reconcilers
Stuck InProgress transactions	No timeout; transaction stays InProgress forever if push succeeds but status update fails	pkg/reconciler/push/push.go:43-49
Unbounded list operations	GitBranches().List() fetches all branches across all repos, filters in-memory	sync/sync.go:97, repowatcher/reconciler.go:87
Branch creation race condition	Multiple reconcilers can attempt to create the same GitBranch simultaneously	repowatcher/reconciler.go:106-141
No NetworkPolicy	Controllers can reach arbitrary network endpoints	config/
Hardcoded merge author	git-k8s-resolver <resolver@git-k8s.imjasonh.com> not configurable	resolver/resolver.go:165-166
No input validation	Git URLs not validated (SSRF risk); branch names used directly in refs	API types
HA not configured	All deployments run 1 replica; leader election exists but untested with >1	config/deployments/

---

Security Assessment

Area	Status	Notes
Container image	Pass	Distroless, non-root, read-only FS, caps dropped
RBAC	Pass	Least privilege, status subresource separated
Secrets handling	Adequate	Not logged, loaded per-reconciliation; no SSH, no rotation, no audit trail
Input validation	Missing	No URL validation, no branch name sanitization
Network isolation	Missing	No NetworkPolicy manifests

---

Performance Concerns

• Memory: Every reconciliation does a full in-memory clone. A 500MB repo exceeds the 256Mi limit.
• List operations: Branch lookups are O(all branches in namespace), not filtered server-side.
• Poll thundering herd: No jitter on DefaultPollInterval = 30s. Many repos polling simultaneously can spike API server load.
• No concurrent reconciliation limits: Knative's workqueue helps, but nothing prevents multiple large clones from running simultaneously.

---

What to Add Next (Prioritized)

Tier 1 — Production Readiness

1. Prometheus metrics — Expose /metrics with reconciliation latency histograms, error counters, work queue depth, and Git operation duration. Use Knative's built-in metrics support or prometheus/client_golang.

2. Health check endpoints — Add /healthz (liveness) and /readyz (readiness) to all controllers. Check informer sync status for readiness; check goroutine health for liveness. Add probes to all deployments.

3. Unit tests for reconciler logic — Target 80%+ coverage on push/push.go, sync/sync.go, resolver/resolver.go, and repowatcher/reconciler.go. Mock the Git client and Kubernetes client. Cover error paths, phase transitions, and edge cases.

4. Git operation timeouts — Add context deadlines to all git.CloneContext calls. Make timeout configurable per-repo or globally. Prevents indefinite blocking on network issues or huge repos.

Tier 2 — Reliability

5. Exponential backoff for transient failures — Categorize errors as transient vs. permanent. Retry transient failures (network errors, API conflicts) with backoff. Keep permanent failures terminal.

6. SSH key authentication — Extend GitAuth to support ssh-privatekey Secret field. Use go-git's SSH transport. Required for most enterprise Git hosting.

7. Transaction timeout/GC — Add a deadline to GitPushTransaction. If InProgress exceeds deadline, mark as Failed. Add a controller or CronJob to clean up old completed transactions.

8. Label selectors for list operations — Use label selectors when listing GitBranch resources instead of listing all and filtering in-memory. Add git-k8s.imjasonh.com/repository label to branches.

Tier 3 — Operations

9. Structured JSON logging — Configure Knative's zap logger for JSON output. Add correlation IDs linking related reconciliations.

10. NetworkPolicy manifests — Restrict controller egress to the Kubernetes API server and known Git remotes.

11. CRD validation — Add CEL validation rules or a validating webhook. Validate Git URLs (scheme, hostname), branch names (no path traversal), and cross-field constraints.

12. Configurable merge author — Allow setting merge commit author/email via ConfigMap or GitRepoSync spec.

Tier 4 — Scale

13. Shallow clones / clone depth — Add Spec.CloneDepth to reduce memory usage for large repos. Most sync operations only need recent history.

14. HorizontalPodAutoscaler — Add HPA manifests based on work queue depth or reconciliation latency.

15. Poll jitter — Add random jitter (±20%) to poll intervals to prevent thundering herd.

16. Webhook-driven sync — Support GitHub/Gitea webhooks as an alternative to polling, for near-instant sync triggers and reduced API load.

---

Production Readiness Checklist

Aspect	Status
RBAC	Ready
Container security	Ready
CRD design	Ready
E2E tests	Ready
CI/CD pipeline	Ready
Metrics	Missing
Health probes	Missing
Unit tests	Inadequate (~10%)
Git operation timeouts	Missing
Retry/backoff	Missing
SSH auth	Missing
Network isolation	Missing
Input validation	Missing
HA (multi-replica)	Untested