fix: Optimize cleanup jobs, eliminate raw ES client usage, add test coverage

[niemyjski]

Summary

Comprehensive optimization of cleanup jobs: eliminate direct Elasticsearch client usage from jobs, fix critical bugs, preserve cleanup correctness under concurrent ingestion, and add full integration test coverage.

Critical Bug Fixes

@min_count:2 → @min:2 in GetDuplicateSignaturesAsync

The @min_count syntax is silently ignored by Foundatio Parsers, causing the aggregation to return ALL signature hashes (not just duplicates). This would have caused FixDuplicateStacksAsync to treat every stack as a duplicate on the next run.

FixDuplicateStacksAsync only ran one batch (regression from refactor)

The original code looped until GetDuplicateSignaturesAsync returned empty. The refactor broke this. Restored the loop with:

• Batch tracking and per-batch logging
• Infinite-loop guard: exits if an entire batch fails to process
• ImmediateConsistency() on CountAsync inside GetDuplicateSignaturesAsync (one refresh per batch, matching the original Indices.RefreshAsync call pattern — NOT per item)

ReassignStackAsync data-loss hazard on empty sequence

If sourceStackIds was empty, PatchAllAsync would have no stack filter and would reassign ALL events to the target stack. Added materialization + early return guard.

FixDuplicateStacksAsync event-first ordering

Event reassignment completes before a duplicate stack is finalized as a redirect tombstone. If event reassignment fails, duplicate stacks remain visible and no data is lost.

GetDistinctFieldValuesAsync cursor correctness

Distinct event identifiers use composite aggregation pagination with an explicit next-value cursor so callers reliably detect the end of pagination.

Architecture (eliminate raw ES client from jobs)

• Refactored CleanupOrphanedDataJob to use repository methods exclusively
• Added GetDistinctFieldValuesAsync using composite aggregation (encapsulated in the repository — composite aggregation is not in Foundatio's DSL, so raw client use is justified and documented)
• Added RemoveAllByProjectIdsAsync/RemoveAllByOrganizationIdsAsync and RemoveAllByStackIdsAsync to IEventRepository
• Added ReassignStackAsync using a parameterized Painless script and strict post-write verification
• Added GetDuplicateSignaturesAsync to IStackRepository

Concurrency and lossless reconciliation

• Added durable duplicate-stack redirects with canonical target resolution, cycle detection, and late-event reconciliation
• Added an idempotent contribution ledger so retrying duplicate merges cannot double-count occurrences, including transitive redirect chains
• Added optimistic concurrency to Stack persistence so stale full-document saves cannot overwrite cleanup merges
• Replaced full-document writes in hot update paths with retrying repository patches for tags, status, regression state, and event counters
• Kept internal redirect/version fields out of the public JSON contract while persisting them in Elasticsearch
• Treat missing event indexes as empty only when Elasticsearch explicitly reports index_not_found_exception; other shard/index verification failures fail closed

Other Fixes

• Added OperationCanceledException handling before the intentionally broad per-signature catch
• Added lock renewal at page boundaries and while awaiting long-running reassignment operations
• Removed redundant is_deleted:false filters where repository soft-delete filtering applies
• Kept page size at 5 for organization/project cleanup (2.5s sleep/item makes larger pages impractical)
• Aligned async method and test names with repository conventions
• Added :{Message} to error log format strings

Merged onto current main

Merged the latest origin/main with a normal merge commit and kept the final diff limited to the intended cleanup, repository, serialization, and test files.

Test Coverage

Coverage includes:

• Cleanup pagination, retention, usage accounting, and tenant isolation
• Orphaned stack/project/organization events, including large and multi-page datasets
• Duplicate signature lookup, soft-delete filtering, empty-source reassignment, and closed/missing event indexes
• Partial failure and retry idempotency, redirect chains/cycles, late events/counters, and stale saves
• Concurrent tag, status, regression, and event-counter updates
• Public serialization and API compatibility for internal persistence fields

[Copilot]

Pull request overview

This PR refactors cleanup jobs to rely on repository abstractions, improves duplicate stack cleanup behavior, and adds integration coverage for cleanup/repository operations.

Changes:

• Reworked orphaned-data and cleanup jobs with lock renewal, cancellation checks, and repository-based deletes/updates.
• Added event repository helpers for bulk deletion, stack reassignment, and distinct-id aggregation.
• Added integration tests for cleanup pagination, retention, duplicate signatures, and event repository operations.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/Exceptionless.Core/Jobs/CleanupDataJob.cs	Extends lock duration and renews locks during paged cleanup/retention loops.
src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs	Replaces direct Elasticsearch calls with repository methods for orphan cleanup and duplicate-stack fixing.
src/Exceptionless.Core/Repositories/EventRepository.cs	Adds bulk delete, stack reassignment, and composite aggregation helpers.
src/Exceptionless.Core/Repositories/Interfaces/IEventRepository.cs	Exposes new event repository cleanup/query APIs and composite cursor type.
src/Exceptionless.Core/Repositories/StackRepository.cs	Adds duplicate signature aggregation lookup.
src/Exceptionless.Core/Repositories/Interfaces/IStackRepository.cs	Exposes duplicate signature lookup API.
tests/Exceptionless.Tests/Jobs/CleanupDataJobTests.cs	Adds cleanup pagination and retention integration coverage.
tests/Exceptionless.Tests/Jobs/CleanupOrphanedDataJobTests.cs	Adds integration coverage for orphan cleanup and duplicate stack merging.
tests/Exceptionless.Tests/Repositories/EventRepositoryTests.cs	Adds coverage for distinct ids, stack reassignment, and bulk delete helpers.
tests/Exceptionless.Tests/Repositories/StackRepositoryTests.cs	Adds duplicate signature repository coverage.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

[niemyjski]

Merged origin/main and addressed the outstanding review feedback in 92fa434.

Summary:

• Removed the accidental packages/ diff from the PR; the final diff against origin/main is back to the intended cleanup/repository/test files.
• Added/kept integration coverage for the new repository methods, duplicate signature lookup, soft-delete filtering, empty ReassignStackAsync safety, orphan cleanup, and cleanup usage accounting.
• Updated duplicate-stack cleanup to loop until no duplicate signatures remain and to renew locks during long-running cleanup paths.
• Documented the composite aggregation cursor path and aligned it with the current Elastic client API.
• Treated missing event indexes as zero matching events during event cleanup, which fixes empty soft-deleted project cleanup.

Verification:

• dotnet test -- --filter-class Exceptionless.Tests.Jobs.CleanupDataJobTests: 26 passed
• dotnet test -- --filter-class Exceptionless.Tests.Jobs.CleanupOrphanedDataJobTests: 24 passed
• dotnet test -- --filter-class Exceptionless.Tests.Repositories.EventRepositoryTests: 17 passed, 1 skipped performance test
• dotnet test -- --filter-class Exceptionless.Tests.Repositories.StackRepositoryTests: 13 passed

All review threads are resolved.

[niemyjski]

Production-readiness pass complete on f6fcc4b.

• Merged current origin/main with normal merge commit 65fe08e.
• Added lossless concurrent cleanup hardening, exact event-index verification/fail-closed behavior, durable/transitive duplicate-stack redirects, bounded late-event reconciliation, optimistic concurrency, and patch-based hot update paths.
• Added integration coverage for closed/missing indexes, partial failure/idempotency, stale saves, chain/cycle redirects, late events and counters, tenant deletion, serialization boundaries, and concurrent tag/status/regression updates.
• Local verification: dotnet build --no-restore -m:1 passed with 0 warnings/errors; dotnet test --no-build passed 2336 tests with 2 intentional performance-test skips and 0 failures.
• Current Build workflow is green: version, test-api with coverage, test-client, test-e2e with localhost AppHost, and docker-build all succeeded. Non-PR deploy/publish/preview jobs were skipped as designed.
• Final GraphQL audit: all 21 review threads are resolved; 0 unresolved actionable threads. PR merge state is CLEAN.

Production-readiness verdict: READY.

[niemyjski]

Production-readiness refresh complete on bd560b0.

• Merged current origin/main (87af2a0) with normal merge commit cfc28b7; no rebase or force-push.
• Re-reviewed the full 18-file PR diff for correctness, compatibility, security, test coverage, and maintainability.
• Fixed the remaining documented convention issue by renaming FixDuplicateStacksAsync and aligning new test names with MethodUnderTest_Scenario_ExpectedBehavior.
• Updated the PR description to document the durable redirect/reconciliation, optimistic-concurrency, and patch-based hot-path hardening already present in the final diff.
• Local build: dotnet build --disable-build-servers --no-restore -m:1 passed with 0 warnings and 0 errors.
• Local service-free contract proof: PublicApiCompatibilityTests passed 4/4. The service-backed local suite was contaminated by concurrent shared Elasticsearch state (alias/index collisions, duplicate fixed test ids, missing shards); isolated CI is authoritative.
• GitHub Actions run 29467194402 is green: version, test-api with the full .NET suite and coverage, test-client, localhost Aspire/Playwright test-e2e, and docker-build all succeeded. Deploy/publish/preview-only jobs skipped as designed.
• Final GraphQL audit: 21/21 review threads resolved, 0 unresolved; merge state CLEAN; no incoming main/head/review changes.

Production-readiness verdict: READY. PR remains open and unmerged.

[github-actions]

Package	Line Rate	Branch Rate	Complexity	Health
Exceptionless.Core	73%	65%	9579	➖
Exceptionless.AppHost	38%	40%	139	❌
Exceptionless.Insulation	23%	23%	205	❌
Exceptionless.Web	84%	67%	6935	✔
Summary	77% (22772 / 29740)	65% (10419 / 15926)	16858	✔

[niemyjski]

Follow-up reviewer audit complete on c2bcdc6.

Feedback ledger

• 21 review-thread roots and 38 inline comments were re-read against the current 18-file diff. All 21 threads were already resolved and outdated; no open thread required a new reply or resolution.
• Already fixed/outdated in current code and focused coverage: #3321326717, #3327871089, #3327871203, #3327871916, #3327872423, #3327872834, #3327873570, #3327875527, #3327876881, #3327877605, #3327877922, #3327878657, #3327879396, #3327880341, #3327881304, #3327881626, #3327927526, and #3327927540. These cover the obsolete cursor null-dereference shape, test naming/AAA, repository integration coverage, soft-delete filtering, pattern style, composite pagination documentation and record shape, structured logging, batch save/lock renewal, empty-source safety, and duplicate aggregation pagination.
• Stale/incorrect for the current contract: #3321326725 and #3327911191 duplicate generic-catch findings. Cooperative cancellation is explicitly rethrown; the remaining per-signature catch intentionally isolates corrupt, repository, transport, and transient failures so one duplicate group does not abort the whole cleanup job. Restricting it to TimeoutException and InvalidOperationException would regress that contract.
• Superseded by diff cleanup: #3329055683 targeted generated packages/ content that is no longer in the PR.
• Submitted reviews: Copilot review #4394198870 is an informational overview whose two findings are fixed above; #4397767971 was unable to review temporary 300-plus-file generated churn and is superseded by the current 18-file diff. The remaining submitted reviews have empty bodies and only contain the inline comments classified above.
• Top-level comments #4742267061, #4952560180, and #4987690483 are historical readiness summaries; #5081447901 is the current informational 77 percent coverage report. None contains an unresolved action.

RCA and current design proof

• Empty source ids previously omitted the stack filter and could match every event; ReassignStackAsync now materializes and rejects an empty source set, with regression coverage.
• A single duplicate-signature terms page could strand groups above the cap; cleanup now refreshes and loops until no duplicate buckets remain.
• Concurrent and in-flight updates could make duplicate cleanup lossy; durable redirects, strict reassignment verification, an idempotent occurrence ledger, reconciliation markers, optimistic concurrency, and patch-based hot updates preserve late events and metadata.
• Null duplicate metadata could be encoded as DateTime.MinValue and persisted as fake dates; the current head uses explicit presence flags and regression assertions.

Thermo-nuclear review

• Reviewed the complete branch-vs-main diff for code-judo opportunities, branching growth, boundaries, canonical helpers, sequential orchestration, and partial updates. No production file crosses 1,000 lines; the largest is 568 lines.
• Raw Elasticsearch operations are confined to the EventRepository boundary for composite pagination and strict closed-index verification. Cleanup jobs use repository abstractions.
• The serialized tag-patch loop is intentional cluster backpressure for batches of up to 1,000 distinct stacks; unbounded parallelization would create a new Elasticsearch burst rather than a safe simplification.
• No proven structural or correctness defect remains, so this audit made no speculative code change.

Verification

• dotnet build -m:1: passed, 0 warnings, 0 errors.
• PublicApiCompatibilityTests: 4/4 passed.
• StackRepositoryTests local attempt: 24 fixture-initialization errors before assertions because concurrent local hosts deleted and recreated the shared test-* indices, yielding index_not_found_exception and invalid_alias_name_exception. This is isolated as shared infrastructure state, not a branch failure.
• Authoritative clean run 30183197271 for this exact SHA: version, test-api with full .NET tests and coverage, test-client, localhost Aspire/Playwright test-e2e, and docker-build all passed.
• git diff --check passed; live PR state was OPEN, MERGEABLE, and CLEAN.

Verdict: READY. No code changes were required in this follow-up audit; the PR remains open and unmerged.