Performance audit: systematic review of bottlenecks across HTTP, CPU, memory, and concurrency

[dangrondahl]

Context

Triggered by #822 / commit 251f3bc, where json.MarshalIndent was inflating multipart payloads by ~2x. A full codebase audit was performed to find similar and other performance issues.

Four parallel investigations covered HTTP/network, CPU/algorithms, memory/allocations, and concurrency/I/O. 30+ raw findings were deduplicated into 18 unique issues across 3 priority tiers.

Tier 1 — High Impact, Fix Soon

#	Issue	Location	Category	Ticket
1	context.TODO() across all AWS API calls — no timeout, no cancellation. ECS/Lambda/S3 calls can hang indefinitely. ~15 call sites.	internal/aws/aws.go	Concurrency
2	Bare &http.Client{} without timeouts in 4 locations. A slow server hangs the CLI forever.	sonar.go:162, azure_apps.go:167,201, update_check.go:84	HTTP
3	json.MarshalIndent still used for non-multipart request bodies (line 122). Same class of bug as #822 — all regular JSON POST/PUT payloads inflated with whitespace.	internal/requests/requests.go:122	HTTP/Payload	#825
4	Sequential S3 file downloads — files downloaded one-by-one. Wall-clock time scales linearly with object count.	internal/aws/aws.go:373–422	Concurrency
5	regexp.Compile() called per-item in K8s namespace filtering, git commit matching, and digest validation — 3 hot paths.	resourceFilter.go:29,45, gitView.go:288, digest.go:359	CPU	#826
6	No connection pooling configured on Kosli HTTP client — no MaxIdleConns, IdleConnTimeout, or keep-alive tuning.	internal/requests/requests.go:60–71	HTTP

Tier 2 — Medium Impact

#	Issue	Location	Category	Ticket
7	Unbounded io.ReadAll on HTTP responses — no size limit. Large API responses or Azure logs could OOM.	requests.go:259, azure_apps.go:179,519,543	Memory
8	Directory fingerprinting: sequential hashing + temp file created per filename to hash a string.	internal/digest/digest.go:119–225	CPU/I/O
9	K8s processPods(): unbounded goroutines (1 per pod, no semaphore) + corev1.Pod passed by value (full struct copy).	internal/kube/kube.go:178–229	Concurrency/Memory
10	GitHub PR evidence: sequential queries with hard-coded 10/20/30s retry sleeps.	internal/github/github.go:281–282	Concurrency
11	No jitter in retry backoff — thundering herd risk with multiple CLI instances.	internal/requests/requests.go:327–342	HTTP
12	PayloadOutput double-buffers entire request body for debug logging.	internal/requests/requests.go:316–321	Memory

Tier 3 — Low Impact / Localized

#	Issue	Location	Category	Ticket
13	Sequential SonarQube API calls (4 serial round-trips)	internal/sonar/sonar.go:219–243	Concurrency
14	Bitbucket PR pagination is sequential	internal/bitbucket/bitbucket.go:200–266	Concurrency
15	Azure zip fingerprinting doesn't reuse bearer tokens	internal/azure/azure_apps.go:237–241	HTTP
16	String += in loops	cli_utils.go:314,320, multiHost.go:60,76	CPU
17	Slice/map pre-allocation missing	internal/github/github.go:114,408	Memory
18	Unnecessary string↔[]byte conversions in HTTPResponse	internal/requests/requests.go:270,295	Memory

Bonus: Correctness Bug

os.WriteFile("zipFileName", ...) at azure_apps.go:526 uses a string literal instead of the variable zipFileName. The surrounding code is unfinished (has TODOs, returns nil), but will be a bug when completed.

Recommended Phases

1. Quick wins — items 3 (MarshalIndent) and 5 (regex compilation). Same patterns as the triggering fix.
2. Resilience — items 1, 2, 6. Prevent hangs in CI/CD.
3. Throughput — items 4, 8, 9, 10. Biggest latency improvements.
4. Hardening — items 7, 11, 12. Defensive limits.