Repository navigation
fix(scheduler): scope the collection marker release to the invocation, not the marker - #1099
Conversation
…, not the marker The owner token MarkCollectionStarted hands the refresh handler is shared by every Lambda delivery of the async collect event, and async delivery is at-least-once. When a duplicate delivery was skipped by the task advisory lock, releaseSkippedCollectionMarker cleared the marker with that shared token while the lock winner was still collecting, so the in-progress banner dropped early and a refresh in the remaining window returned 202 without collecting. The lock winner now calls ClaimCollectionInvocation right after taking the lock. It re-stamps last_collection_owner_id with a fresh invocation token when the marker is still owned by the marker token, or free or stale (the latter covers a duplicate that cleared in the window between the winner's lock and its re-stamp, and a late redelivery). The winner collects and clears under the new token, so the duplicate's skip-path clear no longer matches. A run skipped behind a tokenless cron run still releases its own marker. No schema change: the owner column already exists. Closes LeanerCloud#84 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: APPROVE 52f4044 Reviewed at head 52f4044 (1 commit on base 8caed39). It merges cleanly onto current origin/main b6a0649 ( Findings (none blocking):
Does it fix #84? Yes. The lock winner re-stamps Attacked:
Local results (merged tree, macOS,
|
|
Independent review 2 (claude-fleet, cold): APPROVE 52f4044 Findings (none blocking):
Against #84: this matches the issue's proposed design and fixes the root cause.
Attacked:
Verified locally, on
Reviewed by an independent agent with no author context (second reviewer). Review only: nothing was pushed, and the GitHub review state is unchanged. |
…tgres container After merging main, TestIntegrationTestsShareOnePostgresContainer (LeanerCloud#1165) flagged the two new ClaimCollectionInvocation integration tests for starting a container each. They now take a migrated database from testhelpers.RequireMigratedDatabase, like the rest of internal/config. Also from review and the merge: - give sharedTaskLocker AdvisoryLockHeldFor, which main added to TaskLocker - claimCollectionInvocation clears OwnerToken on a copy of params instead of returning a fresh struct, so a field added later is not dropped - point the markerStore doc at store_postgres_collection_invocation_test.go and the claim doc at runScheduledTask - use mocks.MockInvocationToken in the analytics and server test fakes Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Addressed the VERDICT and Independent review 2 findings on 52f4044. I merged origin/main (a14dc8f) into the branch. Nothing was rebased. Merge with main
Findings
New head: d0f151e Verification on the merged tree (Linux, Go 1.26.9,
|
|
VERDICT: APPROVE d0f151e Re-review of #1099 (closes #84). Old head 52f4044, new head d0f151e. origin/main is 00a3f40, merge-tree c80afe0b, clean. Delta (PR-only diff, old base 8caed39 vs new base a14dc8f):
Findings: none blocking.
Concurrency, re-checked against the claim UPDATE (row-lock serialized, READ COMMITTED re-check):
Results on the merged tree:
Mutation probes (all reverted):
CI on d0f151e (run 39894096060, 2026-10-23, base before the Go bump):
@IAmJSD CI on this head is red only from the old merge base (govulncheck on go1.26.9, fixed on main by #1236; a Docker image advisory; and |
Brings in main's CI fixes (LeanerCloud#1236, LeanerCloud#1252) and CI changes since, so this PR gets a fresh CI run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
VERDICT: APPROVE dff10b0 Delta re-review after the refresh onto main (previously approved at d0f151e).
|
|
Independent review 2 (claude-fleet, cold), re-review: APPROVE dff10b0 Brief correction: #1099 is Delta (first-parent): merge 1520270 (main a14dc8f), d0f151e "test(config): move the #84 store tests onto the shared Postgres container", merge dff10b0 (main 503eb2e). Both merges have an EMPTY remerge-diff (no hand resolution). dff10b0 + current main (2ded648) merges cleanly. d0f151e: the two ClaimCollectionInvocation integration tests use Findings: none. Attacked:
Verified locally (go1.26.10, export of merge-tree origin/main + dff10b0, under the heavy lock):
Reviewed by an independent agent with no author context (second reviewer). Review only: nothing was pushed, and the GitHub review state is unchanged. |
|
Local verification at |
|
Merged by IAmJSD's agent at IAmJSD's request (IAmJSD is acting owner while the maintainers are away), under the bar IAmJSD set: 1 approving independent review(s) on |
Bring the branch up to date with main. Conflicted files: - internal/analytics/collector_test.go: main kept adding stubs to the hand-rolled analytics mockConfigStore (HistorySummaryAggregate from LeanerCloud#1200, ClaimCollectionInvocation from LeanerCloud#1099, the ReplaceRecommendations removal from LeanerCloud#1209, and the pgx import they need). This branch replaces that hand-rolled mock with one that embeds mocks.MockConfigStore, which already carries all of those methods on main. Kept the branch's version; main's stub edits are covered by the shared mock and drop out. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Root cause
MarkCollectionStartedhands the refresh handler an owner token that identifies the marker, and that token rides in the async self-invoke payload. Lambda async delivery is at-least-once, so a duplicate delivery carries the same token. The delivery that wins the task advisory lock collects; the duplicate is lock-skipped andreleaseSkippedCollectionMarkercallsClearCollectionStarted(token). The token still matches, so the marker is cleared while the winner is still collecting. The banner drops early, and a refresh during the rest of the run wins a new marker, gets lock-skipped, releases it and returns 202 without collecting.Fix
This is the invocation-scoped identity described in the issue:
ClaimCollectionInvocation(ctx, markerToken)(internal/config/store_postgres_recommendations.go, added toStoreInterface). It is a singleUPDATEonrecommendations_staterow 1 that setslast_collection_owner_idto a fresh UUID when the marker is:markerToken.started_atis left as the refresh recorded it.MarkCollectionStarted.started_atis set toNOW().It returns
("", false, nil)when a different run holds a fresh marker, and an error for an empty token.runScheduledTask(whichHandleScheduledTaskwraps) calls it throughclaimCollectionInvocation(newinternal/server/collection_invocation.go) right after it acquires the advisory lock. This only happens forcollect_recommendationsruns that carry a token. The run then collects, and its deferred clear runs, under the new token. After that, a lock-skipped duplicate'sClearCollectionStarted(markerToken)no longer matches and does nothing.releaseSkippedCollectionMarkerexists for. Paths with no locker (tests, the non-Lambda HTTP mode) do not claim.last_collection_owner_idalready exists (000093).Concurrency: production only claims while holding the advisory lock, so the race that matters is the winner's claim against the duplicate's clear. Both are single-row UPDATEs on row 1. They serialize on the row lock, and under READ COMMITTED the second re-checks its WHERE against the first's committed row. Either order leaves the winner owning a set marker: if the claim lands first the clear does nothing, and if the clear lands first the claim re-establishes the marker. The race test below runs this on real Postgres.
Verification
All commands ran from the repo root with
AWS_CONFIG_FILE=/dev/null AWS_SHARED_CREDENTIALS_FILE=/dev/null, because this machine's~/.aws/configdefault profile is invalid.Fail before / pass after (unit level, in-memory marker store plus a shared advisory-lock fake). New tests in
internal/server/collection_invocation_test.go:TestHandleScheduledTask_DuplicateDeliveryKeepsWinnersMarker: the winner blocks mid-collect, the duplicate is delivered and lock-skipped, and the test checks that the marker is still set.TestHandleScheduledTask_DuplicateClearsBeforeWinnerClaims: the duplicate is interleaved between the winner's lock and its claim.TestHandleScheduledTask_SkippedBehindCronStillReleases: the cron-overlap release still works.TestHandleScheduledTask_ClaimInvocationOutcomes: claimed, held by another run, and store error.TestHandleScheduledTask_ClaimOnlyForTokenCarryingCollect.Before the fix (with the store method and mock already present, without the handler change):
After:
ok github.com/LeanerCloud/cloud-commitments-platform/internal/server(also passed with-count=3 -race).Real Postgres (testcontainers, integration tag). New file
internal/config/store_postgres_collection_invocation_test.go. Its tests take their database fromtesthelpers.RequireMigratedDatabase(the package's shared container, #1165), soTestIntegrationTestsShareOnePostgresContainerpasses:Mutation probe: dropping the free/stale branch from the claim's WHERE makes
re-established,stale ... taken overand the claim-vs-clear race test fail.Other checks
go build ./..., andgo veton server/config/mocks/scheduler/analytics, with and without-tags integration.go test -count=1for./internal/server/(also with-raceand-tags integration),./internal/mocks/(repo-wide mock guards),./internal/scheduler/,./internal/analytics/,./internal/config/and./internal/api/: all ok.go test -tags integration ./internal/config/ -run 'Collection|Recommendations': ok.--build-tags integration: 0 issues.gocyclo -over 10 -ignore "_test\.go"on internal/server, config, mocks and scheduler: silent.scripts/gosec-hook.shon the changed files: clean.scripts/test-git-secrets-allowlist.sh: 42 PASS, 0 FAIL.Coverage gap: no real Lambda duplicate-delivery test. The duplicate is simulated by calling
HandleScheduledTasktwice with the same token against a shared lock fake. The SQL is exercised on real Postgres.Out of scope
AssertNotCalledshape ininternal/scheduler/scheduler_test.go, and the wider audit ofAssertNotCalledcall sites. Not touched here.internal/config/store_postgres_recommendations.gowas already over the 500-line guideline before this change (603 lines) and grows by this method. Splitting it would be a separate refactor.Maintainer: please mirror the issue's triage labels onto this PR:
priority/p2,severity/medium,urgency/this-quarter,impact/few,effort/m,type/bug,triaged.Closes #84
🤖 Generated with Claude Code