Skip to content

fix: preserve partial benchmark failures in published results - #172

Open
lukekarrys wants to merge 1 commit into
mainfrom
codex/issue-153
Open

lukekarrys wants to merge 1 commit into
mainfrom
codex/issue-153

Conversation

@lukekarrys

Copy link
Copy Markdown
Member

When only one of three benchmark runs succeeds, processing currently publishes its timing as an ordinary result and loses the failed exit codes. Preserve attempted, successful, and dropped counts plus original exit codes; label partial results in tables and every chart layout; and omit affected commands from rankings and daily history comparisons. Synthetic average rows retain partial warnings.

The schema additions are optional for historical compatibility. Existing survivor arrays and total-failure DNF sentinels remain readable, while partial CPU aggregates are omitted and a single survivor has no reported standard deviation. The registry raw-failure scan remains before processing, and post-processing speed comparisons exclude partial results.

Validation:

  • 12 script tests pass, covering complete/partial/total failure, repeated processing, PM and registry chart output, historical compatibility, and raw-validation ordering.
  • 5 app regression tests pass for labels, ranking exclusion, synthetic averages, and historical daily aggregation.
  • Forced TypeScript check and Vite production build pass.
  • Adds a Node 24 PR workflow to run the data tests and app build.
  • Browser visual verification was unavailable because no browser was connected in this session.

Closes #153

@coderabbitai

coderabbitai Bot commented Sep 16, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 58 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: 66bc002f-6c72-4b19-acb9-10144b69545c

📥 Commits

Reviewing files that changed from the base of the PR and between 2a654d4 and 27eb07b.

📒 Files selected for processing (16)
  • .github/workflows/benchmark.yaml
  • .github/workflows/test.yml
  • README.md
  • app/package.json
  • app/src/components/history-chart.tsx
  • app/src/components/ui/chart.tsx
  • app/src/components/variation/chart.tsx
  • app/src/components/variation/table.tsx
  • app/src/hooks/use-history-data.ts
  • app/src/lib/run-completeness.ts
  • app/src/lib/utils.ts
  • app/src/types/chart-data.ts
  • app/tests/partial-results.test.js
  • scripts/clean-benchmarks.js
  • scripts/generate-chart.js
  • scripts/partial-results.test.js

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Preserve partial benchmark failures instead of publishing survivor-only stats

2 participants