Repository navigation
fix(asap-tools): complete ClickHouse runs and clean CPU for random E2E pilots - #902
Merged
Merged
Conversation
A client stopped mid-run leaves its gzipped JSONL unterminated; compare_arms now keeps the complete lines before the cut. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
remote_monitor matched processes by "clickhouse", which also matched a paced feeder's `--sink clickhouse` on the same node: the feeder's CPU was summed into ClickHouse's, and when the feed ended the monitor took it for ClickHouse exiting and stopped the query client mid-run. The pattern clickhouse.server matches the server in Docker and bare metal only. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A run can now stop its feeders when its queries are done instead of waiting out their duration, without losing --stats-out: SIGTERM ends the feed before the next tick and the stats cover the ticks sent. Achieved rows/s spans those ticks, not the planned duration. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
It shared 19429 with e2e_topk_of_sum_ranks_groups_from_one_planned_heap; run in parallel, one of them got connection refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The run waited out its feed and the feed's end could cut the queries short. Now it stops the feeders once the queries are done (SIGTERM, so they still write stats), and a feed's duration only bounds a hung run; the random-workload configs set it to 10x the query phase for every arm. The feeder runs on the next node, as in experiment_run_e2e.py, so it no longer takes ClickHouse's cores. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
compare_arms reports CPU-seconds over the whole monitored run and its split at the client's starting delay (warm-up, ingest only) and after it (queries plus any lag), with the monitored length. The per-second samples stay as recorded. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #879, from running workload 01's five-system pilot.
Problems the pilot hit:
remote_monitormatched processes byclickhouse, which also matched the paced feeder's--sink clickhouseon the same node. The feeder's CPU was added to ClickHouse's, and when the feed ended, the monitor took it for ClickHouse exiting and stopped the query client mid-run. Raw ClickHouse lost about 60% of its queries.Fixes:
remote_monitormatches the ClickHouse server only, with the patternclickhouse.server(Docker and bare metal).paced_feederstops at the next tick and still writes--stats-out. Achieved rows/s now spans the ticks actually sent.experiment_run_e2e.py, so it no longer takes ClickHouse's cores. Fed runs now neednum_nodes >= 1.compare_armsreads a stopped client's truncated results and splits each arm's CPU-seconds into warm-up and query phase.e2e_avg_of_per_series_quantile_over_timegets its own port. It clashed with a test added onmain.Testing:
python3 -m unittest discover -s tests -p "test_*.py", 199 passcargo test, 39 passcargo test --test e2e_precompute_equivalence, 22 pass (3 runs)🤖 Generated with Claude Code