Skip to content

fix(asap-tools): complete ClickHouse runs and clean CPU for random E2E pilots - #902

Merged
milindsrivastava1997 merged 6 commits into
mainfrom
codex/random-e2e-followups
Oct 15, 2026
Merged

milindsrivastava1997 merged 6 commits into
mainfrom
codex/random-e2e-followups

Conversation

@milindsrivastava1997

Copy link
Copy Markdown
Contributor

Follow-up to #879, from running workload 01's five-system pilot.

Problems the pilot hit:

  • In a fed ClickHouse run, remote_monitor matched processes by clickhouse, which also matched the paced feeder's --sink clickhouse on the same node. The feeder's CPU was added to ClickHouse's, and when the feed ended, the monitor took it for ClickHouse exiting and stopped the query client mid-run. Raw ClickHouse lost about 60% of its queries.
  • The run's end was tied to the feed's duration, not to its queries.

Fixes:

  • remote_monitor matches the ClickHouse server only, with the pattern clickhouse.server (Docker and bare metal).
  • On SIGTERM, paced_feeder stops at the next tick and still writes --stats-out. Achieved rows/s now spans the ticks actually sent.
  • A fed ClickHouse run stops its feeders once its queries are done. The feed's duration only bounds a hung run, and the random-workload configs set it to 10× the query phase for every arm. The feeder runs on the next node, as in experiment_run_e2e.py, so it no longer takes ClickHouse's cores. Fed runs now need num_nodes >= 1.
  • compare_arms reads a stopped client's truncated results and splits each arm's CPU-seconds into warm-up and query phase.
  • The engine test e2e_avg_of_per_series_quantile_over_time gets its own port. It clashed with a test added on main.

Testing:

  • experiments: python3 -m unittest discover -s tests -p "test_*.py", 199 pass
  • paced-feeder: cargo test, 39 pass
  • query engine: cargo test --test e2e_precompute_equivalence, 22 pass (3 runs)
  • SIGTERM, checked live against a stub sink: the feeder stops after the tick in flight, writes stats for the ticks sent, and exits 0.

🤖 Generated with Claude Code

milindsrivastava1997 and others added 6 commits October 15, 2026 00:18
A client stopped mid-run leaves its gzipped JSONL unterminated;
compare_arms now keeps the complete lines before the cut.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
remote_monitor matched processes by "clickhouse", which also matched a
paced feeder's `--sink clickhouse` on the same node: the feeder's CPU was
summed into ClickHouse's, and when the feed ended the monitor took it for
ClickHouse exiting and stopped the query client mid-run. The pattern
clickhouse.server matches the server in Docker and bare metal only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A run can now stop its feeders when its queries are done instead of
waiting out their duration, without losing --stats-out: SIGTERM ends the
feed before the next tick and the stats cover the ticks sent. Achieved
rows/s spans those ticks, not the planned duration.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It shared 19429 with e2e_topk_of_sum_ranks_groups_from_one_planned_heap;
run in parallel, one of them got connection refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The run waited out its feed and the feed's end could cut the queries
short. Now it stops the feeders once the queries are done (SIGTERM, so
they still write stats), and a feed's duration only bounds a hung run;
the random-workload configs set it to 10x the query phase for every arm.
The feeder runs on the next node, as in experiment_run_e2e.py, so it no
longer takes ClickHouse's cores.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
compare_arms reports CPU-seconds over the whole monitored run and its
split at the client's starting delay (warm-up, ingest only) and after it
(queries plus any lag), with the monitored length. The per-second samples
stay as recorded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@milindsrivastava1997
milindsrivastava1997 merged commit 788d827 into main Oct 15, 2026
26 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant