Skip to content

Ingest string-bench results into the compression tables - #11044

Merged
joseph-isaacs merged 1 commit into
developfrom
ji/upbeat-brown-y7naim
Oct 30, 2026
Merged

joseph-isaacs merged 1 commit into
developfrom
ji/upbeat-brown-y7naim

Conversation

@joseph-isaacs

@joseph-isaacs joseph-isaacs commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Since #10988, develop stores no String Encoding results, because string-bench emitted no ingest records. This PR lets string-bench write its Vortex file results into the existing compression_sizes and compression_times tables. It needs no new table or schema change.

The website side is vortex-data/vortex.dev#152, which lists these rows in their own String Encoding and String Encoding Size groups. Either PR can merge first. If this one lands first, the string charts appear inside the Compression groups until the website deploys.

Changes

  • benchmarks/string-bench/src/serialized.rs: adds SerializedResult::v3_records(), built on the existing v3::compression_size_record and v3::compression_time_record helpers. Each (column, encoder) run emits:

    • one compression_size record: the Vortex file size, plus the column's canonical uncompressed size
    • compression_time records: encode for the file write and decode for the scan, each with its median and every run
  • Record keys. Every record uses:

    • dataset strings/<column>
    • the encoder (onpair-12, onpair-16, fsst) as dataset_variant
    • format vortex-file-compressed

    The prefix and the non-null variant keep these keys from colliding with compress-bench rows for the same columns (l_comment, ClickBench URL). The compress-bench summaries only use rows paired with Parquet, so string rows can't change them.

  • benchmarks/string-bench/src/main.rs: adds an --ingest-jsonl <path> flag. Only the Vortex file suite is recorded; the codec suite is a local diagnostic that CI doesn't run.

  • scripts/string-split.py: passes --ingest-jsonl for each run and merges the outputs into results.ingest.jsonl.

  • .github/workflows/develop-bench.yml: sets v4_ingest: true for string-bench and drops the comment saying its results aren't stored.

PR runs need no further change. pr-bench-runner.yml already fetches a develop baseline for string-bench from results.ingest.jsonl, so PR comments will compare against develop once develop has rows.

I haven't compiled this or run clippy locally; CI will cover it.

🤖 Generated with Claude Code

https://claude.ai/code/session_01UeniBE69Jeaf1h8ZkLq7QB

string-bench's Vortex file suite now writes v3 ingest records via a new
--ingest-jsonl flag, and develop-bench ingests them into the existing
compression_sizes and compression_times tables. Each (column, encoder)
run emits one size row and encode (write) / decode (read) timing rows,
with dataset `strings/<column>` and the encoder in dataset_variant so
the rows never collide with compress-bench keys.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UeniBE69Jeaf1h8ZkLq7QB
Signed-off-by: Joe Isaacs <joe.isaacs@live.co.uk>
@joseph-isaacs
joseph-isaacs force-pushed the ji/upbeat-brown-y7naim branch from b52d84f to b0d522f Compare October 30, 2026 15:30
@joseph-isaacs joseph-isaacs changed the title Enable v3 record emission for string-bench benchmark results Ingest string-bench results into the compression tables Oct 30, 2026
@joseph-isaacs
joseph-isaacs marked this pull request as ready for review October 30, 2026 15:30
@joseph-isaacs joseph-isaacs added the changelog/chore A trivial change label Oct 30, 2026
@joseph-isaacs
joseph-isaacs enabled auto-merge (squash) October 30, 2026 15:42
@joseph-isaacs
joseph-isaacs merged commit efc7889 into develop Oct 30, 2026
107 of 108 checks passed
@joseph-isaacs
joseph-isaacs deleted the ji/upbeat-brown-y7naim branch October 30, 2026 15:42
@codspeed

codspeed Bot commented Oct 30, 2026

Copy link
Copy Markdown

Merging this PR will improve performance by 10.38%

2 warnings

⚠️ Walltime results may be unreliable

Some benchmarks used the Walltime instrument outside CodSpeed Macro Runners, which can make the results inconsistent.

Learn more about Macro Runners

See affected benchmarks

⚠️ 1 benchmark is dominated by system calls

System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.

Learn more about system calls

See affected benchmarks

⚡ 1 improved benchmark
✅ 2317 untouched benchmarks

Performance Changes

Mode Benchmark BASE HEAD Efficiency
⚡ Walltime threshold_avx2[80] 3 µs 2.7 µs +10.38%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing ji/upbeat-brown-y7naim (b0d522f) with develop (38d8fbc)

Open in CodSpeed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/chore A trivial change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants