Skip to content

fix(server): stop reporting fabricated success on three operator paths - #1264

Merged
IAmJSD merged 2 commits into
LeanerCloud:mainfrom
IAmJSD:fix/issue-326
Oct 25, 2026
Merged

IAmJSD merged 2 commits into
LeanerCloud:mainfrom
IAmJSD:fix/issue-326

Conversation

@IAmJSD

@IAmJSD IAmJSD commented Oct 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes the three internal/server silent-fallback gaps from #326. The fourth finding, in the ECR cleanup script, was already fixed on main. Each finding was checked again against current main (a14dc8f2) before any change was made:

Finding Status on main a14dc8f2 Action in this PR
A06-018 analytics_refresh reports success with zero counters when the store is absent; sessions_deleted is never assigned Reproduces. handler.go still starts with status: success and partitions_created / partitions_dropped = 0, and the nil-store branch only logs. sessions_deleted is still set to 0, never assigned, and logged as a count. Fixed
A06-024 body read errors discarded, oversize bodies truncated Partly reproduces. The read error is still dropped: a mid-upload disconnect reaches the router with an empty body (the pre-fix test gets 401/404 instead of 400). The oversize half is mostly stale: since the router rejects bodies over 1 MB (api.MaxRequestBodySize) with a JSON 413, /api/* and /version already returned 413 for a truncated 10 MB body. Only /oidc/* routed it (404 instead of 413). The transport still buffered up to 10 MB that the router then threw away. Fixed
A06-025 STS failure gives an "unknown" account ID that is written onto exchange audit rows Reproduces. resolveAccountID still returns "unknown" on an STS error or a nil Account. Fixed
A14-042 ECR guard is missing the empty-string check Already fixed on main by #489 (9b50029c, "test(ci): tighten destroy-script guards and fixtures"). It added the case "$OWNED_REPO" in '' | *[![:graph:]]*) guard before the first AWS call, plus stubbed tests in scripts/test-ecr-delete-selection.sh. No change. Re-verified offline (see below).

Root cause and fix

  • A06-018 (internal/server/handler.go): handleRefreshAnalytics now reports status: "skipped" when app.Analytics is nil, the same status handleCollectAnalytics uses for that case. It no longer returns partitions_created / partitions_dropped, because this task never creates or drops partitions (collect_analytics does). views_refreshed stays, since it is populated. handleCleanupExpiredRecords no longer returns or logs sessions_deleted, because CleanupExpiredSessions reports no count.
  • A06-024 (internal/server/http.go, new internal/server/http_body.go):
    • httpToLambdaRequest(w, r) now returns an error.
    • The body is read through http.MaxBytesReader, capped at api.MaxRequestBodySize (the router's own 1 MB limit, so the transport and the router can't disagree).
    • All three callers (handleHTTPRequest, handleVersion, handleOIDCHTTP) answer a failed read with writeBodyReadError:
      • an oversize body gets 413 with the router's own JSON body, {"error":"Request body too large"}, so the status and shape seen by /api/* clients are unchanged;
      • any other read error gets 400 {"error":"Failed to read request body"}.
    • The new helpers live in a new file so http.go (already over the 500-line guideline) grows only by its call sites.
  • A06-025 (internal/server/handler_ri_exchange.go):
    • resolveAccountID takes a narrow callerIdentityGetter interface (satisfied by *sts.Client). It returns an error on an STS failure, a nil output, or a nil or empty Account, as resolveLadderIdentity already does.
    • handleRIExchangeReshape resolves the account before it builds any EC2 client and aborts the run if that fails.
    • As defense in depth, executeRIExchangeReshape refuses an empty accountID before it lists RIs or touches the store. A caller that drops the error still can't write a blank account onto audit rows.
    • The ladder resolver's comment, which described the old "unknown" behavior, is updated.

Closes #326

Verification

All of the evidence below is unit / mock / fixture evidence. No real AWS, STS or Postgres calls were made. Environment: AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null AWS_EC2_METADATA_DISABLED=true, Go 1.26.9.

Fail before. The new and changed tests were copied onto a git archive of main a14dc8f2. Tests that call the new signatures (httpToLambdaRequest(w, r) and resolveAccountID(ctx, client)) don't compile against main, so they were left out of this run.

go test ./internal/server/ -count=1 -v -run 'TestHTTPHandlers_Reject|TestExecuteRIExchangeReshape_RefusesEmptyAccountID|TestHandleRefreshAnalytics|TestHandleCleanupExpiredRecords'
--- FAIL: TestHandleCleanupExpiredRecords_WithAuthAndConfig   sessions_deleted is never populated and must not be reported
--- FAIL: TestHandleRefreshAnalytics_Success                  partitions_created is never populated and must not be reported
--- FAIL: TestHandleRefreshAnalytics_NilAnalytics             Expected skipped, got success
--- FAIL: TestExecuteRIExchangeReshape_RefusesEmptyAccountID   Expected an error, got nil
--- FAIL: TestHTTPHandlers_RejectOversizeBodyWith413/handleOIDCHTTP      Expected 413, got 404
--- PASS: TestHTTPHandlers_RejectOversizeBodyWith413/handleHTTPRequest   (router's 1 MB cap already 413s; see table)
--- PASS: TestHTTPHandlers_RejectOversizeBodyWith413/handleVersion
--- FAIL: TestHTTPHandlers_RejectBodyReadErrorWith400/handleHTTPRequest  Expected 400, got 401
--- FAIL: TestHTTPHandlers_RejectBodyReadErrorWith400/handleVersion      Expected 400, got 401
--- FAIL: TestHTTPHandlers_RejectBodyReadErrorWith400/handleOIDCHTTP     Expected 400, got 404

Pass after, on this branch:

go build ./... && go vet ./... && go vet -tags integration ./...            # OK
go test ./internal/server/ -count=1                                        # ok
go test ./internal/mocks/ -count=1                                         # ok
golangci-lint (v2.10.1) run --allow-parallel-runners ./internal/server/    # 0 issues
golangci-lint run --allow-parallel-runners --build-tags integration ./internal/server/   # 0 issues
gocyclo -over 10 -ignore "_test\.go" internal/server/{http,http_body,handler,handler_ri_exchange,handler_ladder}.go   # none

All of the tests above pass, plus:

  • TestHttpToLambdaRequest_BodyAtLimitIsKeptWhole: a body of exactly 1 MB is accepted whole.
  • TestHttpToLambdaRequest_OversizeBodyIsAnError: 1 MB + 1 gives *http.MaxBytesError. This test also catches a mutation that raises the cap back to 10 MB, which the handler-level test alone would miss because the router's own 413 hides it.
  • TestHttpToLambdaRequest_ReadErrorIsReturned
  • TestResolveAccountID_RefusesToFabricateAnAccount (STS error, nil output, nil account, empty account)
  • TestResolveAccountID_ReturnsTheCallerAccount

A14-042 (already fixed), checked offline only. The script was never run against real AWS. scripts/test-ecr-delete-selection.sh runs it end to end against stubbed terraform and aws binaries on PATH:

bash scripts/test-ecr-delete-selection.sh
PASS: behaviour: repository name "" exits 1 with its own error and no aws call
PASS: behaviour: repository name "a b" exits 1 with its own error and no aws call
passed: 59, failed: 0

Mutation probe on a throwaway copy, with the case "$OWNED_REPO" guard deleted: both cases above FAIL (57 passed, 2 failed). The existing test therefore pins the guard.

shellcheck 0.11.0 on scripts/force-delete-owned-ecr-repo.sh is clean. scripts/test-ecr-delete-selection.sh (unchanged) shows only pre-existing info-level SC1091 and SC2016 notes.

No DB or migration changes and no new integration tests: none of the fixes touches a query. The RI-exchange path change happens before any store call.

Combined-tree verification. This branch was tested together with the other queued branches on a local, never-pushed tree: queue/platform @ 936996ad = origin/main a14dc8f2 plus these heads, merged in this order:

Branch Tested head
fix/issue-337 ff3420c
fix/issue-169 7d75eb5
fix/issue-342 66b221f
fix/issue-310 5e11c21
fix/issue-326 5f0ea59
fix/issue-243 365e959
fix/issue-343 6c94036
fix/issue-312 d2848a8
fix/issue-316 acd60dd
fix/issue-339 75ea9b9
fix/issue-339-commitment-type 73bdb19
fix/issue-353 5d25fcd
fix/issue-313 3156bc4
fix/issue-357 53d9e58
fix/issue-368 0ba727d
fix/issue-419 61648c9
fix/issue-111 b04c372
fix/issue-686 0ad90b7
fix/issue-674 793095d
fix/issue-1055 4ec96f1
fix/issue-559 dfa9525
fix/issue-558 034eed4
fix/issue-874 1720539
fix/issue-666 1e120da
fix/issue-668 fb26345
fix/issue-568 ce04173
fix/issue-311 e54f250
fix/issue-177 0cbb88a
fix/issue-151 c21377c
fix/issue-166 41fb77b
fix/issue-27 2d67622
fix/issue-323 69288ae
fix/issue-59 36463e8
fix/issue-43 35e4069
fix/issue-158 6f76f8a
fix/issue-104 bb00491
fix/issue-55 971ef1b
fix/issue-361 89ac220
fix/issue-120 b458719
fix/issue-36 42bac6e
fix/issue-525 57c97dd
fix/issue-712 e5bcda0
fix/issue-72 6cc63ed

fix/issue-72 and fix/issue-312 conflict on one line in internal/api/handler_dashboard.go (both change the calculateCommitmentMetrics call). The tree used #72's side with #312's validated provider variable; whichever PR merges second takes that same resolution.

Linux, Go 1.26.9, AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null AWS_EC2_METADATA_DISABLED=true; Docker Postgres via testcontainers:

  • Passed at 936996ad: go build ./..., go vet ./..., go vet -tags integration ./... and the full go test -race -short -count=1 ./....
  • Static checks at 936996ad: golangci-lint v2.10.1 found 0 issues on every changed package, with and without --build-tags integration. tests/e2e is a separate module, so it was vetted under its own e2e tag. gocyclo -over 10 is silent on the changed non-test files.
  • Integration tests, every changed package, one package at a time (-tags integration -count=1, 60-minute local timeout because the VM was shared):
    • At 936996ad, these pass: internal/server, internal/credentials, internal/scheduler, internal/database/postgres/migrations, internal/purchase and internal/api.
    • internal/config passes except TestPostgresStoreDB_ResolveStaleRIExchange_ValidatesCloudCheck, which fails the same way on unmodified main (fixed by fix(config): compare resolve checked_at at the database's microsecond precision #1252).
    • These were run on the first build of this tree (26795376, same branches at earlier heads): internal/analytics, internal/auth, internal/commitmentopts, internal/database and cmd/lambda/clear-rate-limit all pass. The later fixes don't touch those packages.
  • What the combined run caught, each fixed on its own branch before this PR opened:
    • a compile break between fix/issue-104's test and fix/issue-326's httpToLambdaRequest signature change;
    • fix/issue-316 rejecting legacy-keyed plans;
    • fix/issue-120 versus main's global-filter test.
  • Secrets: git-secrets, using the repo's scripts/setup-git-secrets.sh patterns in a fresh repo with an isolated HOME, found nothing in the changed files, and a planted fake key was flagged. scripts/test-git-secrets-allowlist.sh: 137 PASS, 0 FAIL.
  • Nothing was applied to any cloud.

Behavior changes worth knowing

  • analytics_refresh with no analytics store:
    • The scheduled-task result now has "status":"skipped". The outer HTTP envelope from handleScheduledHTTP still says "status":"success" (the task ran without error), exactly as for collect_analytics's existing "skipped".
    • partitions_created, partitions_dropped and sessions_deleted are gone from the result maps. I found no consumer of them in the repo, including frontend/, Terraform and docs (other than the audit report).
  • HTTP transport:
    • Bodies are capped at 1 MB at the transport instead of 10 MB. Status codes don't change for /api/* and /version, because the router already returned 413 above 1 MB.
    • /oidc/* (GET-only endpoints) now answers 413 for a body over 1 MB and 400 for a read error instead of routing it.
    • The transport's 413/400 carries Content-Type: application/json and the security headers from the securityHeaders middleware, but not the router's CORS headers. The SPA is same-origin, so this shouldn't matter.
  • RI exchange reshape: an STS failure now fails the scheduled run with an error instead of writing account_id = "unknown". The stale-exchange alarm sweep still runs first, because it comes before the AWS config and STS step.

Out of scope

  • handleScheduledHTTP's outer "status":"success" envelope wraps any inner status (skipped, partial, disabled). That is the shared contract for every scheduled task, so I left it alone.
  • The Lambda Function URL path receives its body from the event and doesn't go through httpToLambdaRequest, so it is unaffected.

Maintainer: please mirror the issue's triage labels onto this PR: impact/internal, triaged, priority/p3, severity/low, urgency/eventually, effort/s, type/bug.

🤖 Generated with Claude Code

- analytics_refresh reports status "skipped" when the analytics store is
  absent and no longer returns partition counters it never writes;
  cleanup_expired_records drops the always-0 sessions_deleted counter.
- The HTTP transport reads request bodies through http.MaxBytesReader at
  the API router's own 1 MB cap: an oversize body is a 413 and a body
  read error (client disconnect mid-upload) is a 400, instead of a
  truncated or empty body being routed.
- resolveAccountID returns an error instead of "unknown", and the RI
  exchange reshape run aborts before touching AWS or the store when no
  account ID is resolved, so no audit row carries a fabricated account.

The fourth finding (empty-name guard in force-delete-owned-ecr-repo.sh)
was already fixed on main by LeanerCloud#489.

Closes LeanerCloud#326

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@IAmJSD IAmJSD added impact/internal Team-internal only triaged Item has been triaged priority/p3 Polish / idea / may never ship severity/low Minor harm urgency/eventually No deadline effort/s Hours type/bug Defect labels Oct 23, 2026
@coderabbitai

coderabbitai Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Caution

Review failed

The pull request is closed.

Finishing Touches
🧪 Generate unit tests (beta)
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

Brings in main's CI fixes (LeanerCloud#1236, LeanerCloud#1252) and CI changes since, so this PR gets a fresh CI run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@IAmJSD

IAmJSD commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE 311ca1670b89066f43f27d6886ae182e2172ca69

Blocking:

  • none

Non-blocking:

  • internal/server/http_body.go:35-46: writeBodyReadError sets only Content-Type. The router's own 413 (internal/api/handler.go:789) goes out with corsHeaders, which include the security headers and Access-Control-Allow-Origin. A cross-origin browser client on a container deployment that sends a body over 1 MB will now see an opaque CORS failure instead of a readable 413, and the response loses HSTS/nosniff. Reusing the API handler's header set, or at least the CORS origin, would keep the status and shape identical, as the PR body promises.
  • Behavior change worth knowing: an STS failure now aborts the scheduled RI exchange run (after the enabled check and the stale-exchange alarm) instead of running with account_id = "unknown". This is the safe direction for a money path and matches the ladder resolver, but the issue's wording ("fabricated success") doesn't spell it out, so operators should expect a failed task rather than a run with a bad audit row.

Attacked:

  • Size cap consistency: every /api/* request passes validateRequestBodySize (1 MB) in validateRequestContext before routing, and OPTIONS carries no body. Capping the transport at api.MaxRequestBodySize therefore can't reject a body the router would have accepted, and no route had a larger allowance. The only callers of httpToLambdaRequest are the three updated handlers. The new STS call in handleRIExchangeReshape sits after the RIExchangeEnabled check, so disabled installs never call STS. The executeRIExchangeReshape empty-ID guard runs before any AWS list call or store write. partitions_*/sessions_deleted have no consumers outside internal/server, and "skipped" is already a status that handleCollectAnalytics emits.
  • Fail-before: in a scratch tree I put back main's behavior behind the new signatures: a 10 MB LimitReader that swallows errors, "unknown" from resolveAccountID, no empty-ID guard, the old analytics/cleanup maps and success on a nil store. 10 tests failed: both HTTP handler tests (all subtests that main also failed), OversizeBodyIsAnError, ReadErrorIsReturned, RefusesToFabricateAnAccount (4/4), RefusesEmptyAccountID, both RefreshAnalytics tests and the Cleanup test. After restoring, all pass.
  • Ran go build ./..., go vet ./internal/server/ with and without -tags integration, go test -race -short -count=1 ./internal/server/, and go test -race -tags integration -count=1 ./internal/server/ (includes the RI exchange reservation integration tests). All ok. golangci-lint v2.10.1 found 0 issues with and without --build-tags integration, and gocyclo -over 10 is silent on the changed files.

Body-stale:

  • "an oversize body gets 413 with the router's own JSON body ... so the status and shape seen by /api/* clients are unchanged": the status and JSON body match, but the headers don't (see the first non-blocking item).

Reviewer: rB10. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths.

@IAmJSD

IAmJSD commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Independent review (IAmJSD's agent, fresh context, given only the diff, the issue and the PR text): VERDICT: APPROVE 311ca1670b89066f43f27d6886ae182e2172ca69

Blocking:

  • none

Non-blocking:

  • internal/server/http_body.go:35-46: the transport-level 413/400 goes out without the CORS headers the router's own 413 carried (buildResponse(413, corsHeaders, …)). In an HTTP-server deployment called cross-origin, an over-1 MB body used to reach the router and come back as a readable 413. Now the browser sees an opaque CORS failure instead. That matches how the existing 503/500 transport errors behave, and same-origin deployments are unaffected, but it's a small change to the "status and shape seen by /api/* clients are unchanged" claim (see Body-stale).
  • internal/server/handler_ri_exchange.go:55-58: a transient STS failure now fails the whole scheduled reshape run, where it used to proceed with "unknown". That's the issue's intent, and it aborts before any EC2/store call, so a retry or the next schedule can't double-act. Operators should know an STS blip now appears as a failed RI-exchange task rather than a silent run.
  • Changing analytics_refresh to status: "skipped" when the analytics store is nil, and dropping partitions_*/sessions_deleted, changes a scheduled-task response shape. No consumer in this repo (TS/TF/YAML/Go) reads those keys or tests for "success" there. External monitors keyed on them would need updating.

Attacked:

  • Money path (A06-025): resolveAccountID now errors on an STS error, a nil output, or a nil or empty Account. handleRIExchangeReshape resolves it after the RIExchangeEnabled check (a disabled config still makes no STS call) and before building clients. executeRIExchangeReshape independently refuses accountID == "" before listConvertibleRIs, the store or RunAutoExchange. Its only callers are handleRIExchangeReshape and tests. The ladder resolver was already strict.
  • Body (A06-024): the cap equals api.MaxRequestBodySize, the same limit the router enforces on every non-OPTIONS request, so no route that worked before is newly rejected. All three httpToLambdaRequest callers handle the error. handleScheduledHTTP doesn't use it.
  • Fail-before: in a scratch tree I reverted handler.go to the merge-base, restored the "unknown" fallback and removed the empty-account guard in handler_ri_exchange.go, and put back the 10 MB LimitReader with the error swallowed in readRequestBody. Every new or changed test failed: the refresh/cleanup tests, all 4 ResolveAccountID_RefusesToFabricate subtests, RefusesEmptyAccountID, 413/OIDC, the three 400 read-error cases, and OversizeBodyIsAnError/ReadErrorIsReturned. Restored afterwards.
  • Ran go build ./...; go vet ./internal/server/ with and without -tags integration; tests/e2e go vet -tags e2e; go test -race -short -count=1 ./internal/server/ (ok); go test -race -tags integration -count=1 ./internal/server/ (ok); golangci-lint with and without --build-tags integration (0 issues); gocyclo -over 10 on the changed files (silent). A scratch git merge --no-commit origin/main (73 commits ahead) builds, vets and passes go test -short ./internal/server/; aborted afterwards. A14-042: scripts/test-ecr-delete-selection.sh gives 59 passed, 0 failed, and the case "$OWNED_REPO" guard is on main.

Body-stale:

  • "an oversize body gets 413 with the router's own JSON body … so the status and shape seen by /api/* clients are unchanged": the status and JSON body are unchanged, but the router's CORS headers are no longer on that response (see first non-blocking point).

Reviewer: rB13. Bar while the maintainers are away (set by IAmJSD as acting owner): one approving independent review on the head, two for money, auth and security paths.

@IAmJSD

IAmJSD commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Local verification at 311ca1670b89066f43f27d6886ae182e2172ca69 merged onto main (5f717aa4), together with #1260, #1261, #1263, #1264, #1269, #1271, #1273, #1308, #1416 in that order (combined tree 62127c84, Linux, Go 1.26.10): go build ./..., go vet with and without -tags integration, the full go test -race -short -count=1 ./..., and go test -race -tags integration -count=1 on every changed package, against Docker Postgres. All pass. CI Success and pre-commit are green on this head, with 2 approving independent review(s) on it. Nothing was run against any cloud.

@IAmJSD
IAmJSD marked this pull request as ready for review October 25, 2026 21:54
@IAmJSD
IAmJSD merged commit f05b0e4 into LeanerCloud:main Oct 25, 2026
34 checks passed
@IAmJSD

IAmJSD commented Oct 25, 2026

Copy link
Copy Markdown
Collaborator Author

Merged by IAmJSD's agent at IAmJSD's request (IAmJSD is acting owner while the maintainers are away), under the bar IAmJSD set: 2 approving independent review(s) on 311ca167 (two required: money, auth or security path), green CI, and the combined-tree verification above. Squash-merged at exactly 311ca1670b89066f43f27d6886ae182e2172ca69.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

effort/s Hours impact/internal Team-internal only priority/p3 Polish / idea / may never ship severity/low Minor harm triaged Item has been triaged type/bug Defect urgency/eventually No deadline

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(server): low silent-fallback gaps in internal/server and the ECR cleanup script

1 participant