Skip to content

Add reproducible scheduling benchmark and experimental comparison - #3

Merged
TohruSu merged 1 commit into
mainfrom
feat/reproducible-benchmark
Oct 15, 2026
Merged

TohruSu merged 1 commit into
mainfrom
feat/reproducible-benchmark

Conversation

@TohruSu

@TohruSu TohruSu commented Oct 15, 2026

Copy link
Copy Markdown
Owner

Add a fixed, paired experiment comparing three independently trained Q-Learning models with FIFO, SPT, EDF, Priority, and seeded Random. Run it with adaptivequeue benchmark --output-dir benchmark-results; cached models are validated against the protocol, and --retrain regenerates them.

Models train on mixed workloads for 1,000 episodes at seeds 42, 2042, and 4042 with existing defaults. Evaluation covers stable, bursty, and mixed workloads at seeds 20000–20029, with each job set generated once and reused for all policies. RL evaluation is greedy and leaves Q-values and RNG state unchanged.

The committed reference run includes 720 per-episode metric rows, 24 summary rows with paired cost differences/win rates, 36 action-frequency rows (including contended-only frequencies), both figures, three saved Q-tables, and protocol metadata. Summary statistics average episode metrics, including episode P95 waiting time; training runs remain separate.

Observed mean primary cost J (lower is better):

Policy / training seed Stable Bursty Mixed
Priority 9.852 30.867 9.212
Q-Learning / 42 9.270 28.804 9.284
Q-Learning / 2042 10.164 31.471 10.402
Q-Learning / 4042 9.210 27.130 9.900

Priority has the best classical mean cost in each workload. Two RL runs improve on it in stable and bursty workloads, but every RL run loses to Priority on mixed. SPT achieves the lowest deadline-miss rate in every workload. docs/experiments.md includes all five baselines, paired comparisons, action mixes, reproduction details, and the limitations of three training seeds at one load/deadline setting.

Validation: inspected six preliminary baseline episodes before the fixed final run; all 81 pytest tests pass, along with Ruff lint/format, strict mypy targeting Python 3.10, and whitespace checks. Fresh/cached reproducibility, identical evaluation job objects, aggregation, frozen evaluation, and figure generation are tested. Matplotlib is an optional benchmark dependency; existing baseline and RL CLI tests remain passing.

@TohruSu
TohruSu merged commit f977149 into main Oct 15, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant