Skip to content

Scope orphaned-event cleanup to recent ingest window to eliminate 8-hour ES CPU spikes#2279

Open
niemyjski with Copilot wants to merge 13 commits into
mainfrom
copilot/investigate-cluster-cpu-spikes
Open

Scope orphaned-event cleanup to recent ingest window to eliminate 8-hour ES CPU spikes#2279
niemyjski with Copilot wants to merge 13 commits into
mainfrom
copilot/investigate-cluster-cpu-spikes

Conversation

Copilot AI commented May 31, 2026

Copy link
Copy Markdown
Contributor

What changed

  • Limits orphaned-event stack, project, and organization scans to events created within the last three days.
  • Applies the same cutoff to cardinality queries, partitioned terms queries, and delete-by-query operations.
  • Captures one cutoff per job run so all three cleanup phases use an atomic boundary.
  • Uses the current Elastic.Clients.Elasticsearch and repository architecture from main.
  • Adds regression coverage for stack, project, and organization orphans at and immediately before the inclusive cutoff.

Why

The cleanup job runs every eight hours. Scanning all event history caused unnecessary Elasticsearch CPU spikes even though actionable orphan cleanup is limited to the practical recent-ingest window.

Verification

  • dotnet build tests/Exceptionless.Tests/Exceptionless.Tests.csproj --no-restore --nologo --verbosity:minimal
  • Exceptionless.Tests --filter-class Exceptionless.Tests.Jobs.CleanupDataJobTests: 28 passed
  • Exact lookback-boundary regression: passed
  • Local Aspire dogfood with healthy Elasticsearch 8.19.15, Redis, API, and Jobs resources

Compatibility

No public API, WebSocket, configuration, or frontend behavior changes.

@CLAassistant

CLAassistant commented May 31, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ niemyjski
❌ Copilot
You have signed the CLA already but the status is still pending? Let us recheck it.

Copilot AI linked an issue Jun 1, 2026 that may be closed by this pull request
Copilot AI and others added 2 commits June 1, 2026 00:22
Co-authored-by: niemyjski <1020579+niemyjski@users.noreply.github.com>
Co-authored-by: niemyjski <1020579+niemyjski@users.noreply.github.com>
Copilot AI changed the title [WIP] Investigate cluster CPU spikes during orphan data cleanup Scope orphaned-event cleanup to recent ingest window to eliminate 8-hour ES CPU spikes Jun 1, 2026
Copilot AI requested a review from niemyjski June 1, 2026 00:28
…luster-cpu-spikes

# Conflicts:
#	src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs
#	tests/Exceptionless.Tests/Jobs/CleanupDataJobTests.cs
@niemyjski
niemyjski marked this pull request as ready for review July 10, 2026 19:34

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9531879f6

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b0910ef8ad

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c247cbf45c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs Outdated
@niemyjski

niemyjski commented Jul 26, 2026

Copy link
Copy Markdown
Member

Follow-up feedback and thermo-nuclear audit complete.

Feedback classification:

  • Ingest-time/index-scope thread: valid, fixed by b0910ef, now outdated/resolved. Current code queries the full versioned event pattern and filters by CreatedUtc; regression coverage includes a 30-day-old occurrence ingested now.
  • Empty/no-recent-event health thread: valid, fixed by d11819a and resolved. Empty and historical-only cases are covered.
  • Eight-hour schedule versus 65-minute health thread: valid, fixed by c9ebc8c and resolved. The health window is 9 hours, with exact-boundary coverage.
  • Health updated before successful cleanup thread: valid, fixed by d0bcc2b and resolved. RCA was both premature _lastRun mutation and unchecked invalid Elasticsearch responses. Health is now stamped only after completion; cleanup search/delete/refresh responses are validated; delete races explicitly proceed through version conflicts. A connection-failure regression proves stale health remains unhealthy.
  • Submitted-review bodies contained only review boilerplate; CLA and coverage comments are informational, not code findings.

Thermo-nuclear full-diff result: no remaining structural blocker. Production is 481 lines and tests 771; neither crosses 1k. The cleanup phases remain intentionally sequential to avoid multiplying Elasticsearch load, public wrappers preserve the existing API, and response validation is centralized in one canonical helper.

Verification:

  • dotnet build tests/Exceptionless.Tests/Exceptionless.Tests.csproj --no-restore --nologo --verbosity:minimal -m:1: passed, 0 warnings/errors.
  • CleanupOrphanedDataJobTests: 20/20 passed.
  • Focused coverage: 66/66 changed executable production lines covered.
  • git diff --check: passed.
  • Fresh CI on d0bcc2b: version, test-api, test-client, test-e2e, and docker-build all passed.

All four review threads are addressed and resolved; the PR is mergeable and current with origin/main. The external Copilot contributor CLA is the sole remaining gate.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c9ebc8c1cc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/Exceptionless.Core/Jobs/CleanupOrphanedDataJob.cs Outdated
@github-actions

Copy link
Copy Markdown

Code Coverage

Package Line Rate Branch Rate Complexity Health
Exceptionless.Core 73% 65% 9533
Exceptionless.AppHost 38% 40% 139
Exceptionless.Insulation 23% 23% 205
Exceptionless.Web 84% 67% 6935
Summary 77% (22576 / 29509) 65% (10387 / 15894) 16812

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Investigate cluster CPU spikes

3 participants