From 8ab8f7a1ae7106f80edceefcdce4e4f0f89660f1 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Wed, 17 Jun 2026 07:52:29 -0700 Subject: [PATCH 01/17] Add a dummy puzzletron skill Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/SKILL.md | 36 ++++++++++++++++++++++++++++++ .claude/skills/puzzletron | 1 + 2 files changed, 37 insertions(+) create mode 100644 .agents/skills/puzzletron/SKILL.md create mode 120000 .claude/skills/puzzletron diff --git a/.agents/skills/puzzletron/SKILL.md b/.agents/skills/puzzletron/SKILL.md new file mode 100644 index 00000000000..c0cab4f02c1 --- /dev/null +++ b/.agents/skills/puzzletron/SKILL.md @@ -0,0 +1,36 @@ +--- +name: puzzletron +description: End-to-end workflow for model pruning and MIP-based optimization. Use `all` to run the full workflow or `mip_sweep` to run the MIP sweep. Usage: /puzzletron +license: Apache-2.0 +--- + +# Puzzletron + +## Routing + +**STEP 1 — Check args before doing anything else. This is MANDATORY.** + +- If args are **empty**, output the block below verbatim and **STOP immediately. Do NOT proceed to any command.** +- If args do **not exactly match** `all` or `mip_sweep`, output the block below verbatim and **STOP immediately. Do NOT proceed to any command.** + +--- + +**Puzzletron** — end-to-end workflow for model pruning and MIP-based optimization. + +Available commands: +- `all` — Run the full puzzletron workflow +- `mip_sweep` — Run the MIP sweep + +Usage: `/puzzletron ` + +--- + +**STEP 2 — Only if args exactly match a command name, execute it. Never reach this step if args were empty.** + +## Command: all + +Return the following message: hello world: puzzletron all message2 + +## Command: mip_sweep + +Return the following message: hello world: puzzletron mip sweep2 diff --git a/.claude/skills/puzzletron b/.claude/skills/puzzletron new file mode 120000 index 00000000000..ef76b5489dd --- /dev/null +++ b/.claude/skills/puzzletron @@ -0,0 +1 @@ +../../.agents/skills/puzzletron \ No newline at end of file From 9a6d71623e1d42e3c25582b2b23b14b51be079f0 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Wed, 17 Jun 2026 08:55:51 -0700 Subject: [PATCH 02/17] Add progress comnand for puzzletron mip_sweep Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/README.md | 58 +++++++++++ .agents/skills/puzzletron/SKILL.md | 148 ++++++++++++++++++++++++++-- examples/puzzletron/README.md | 7 ++ 3 files changed, 203 insertions(+), 10 deletions(-) create mode 100644 .agents/skills/puzzletron/README.md diff --git a/.agents/skills/puzzletron/README.md b/.agents/skills/puzzletron/README.md new file mode 100644 index 00000000000..af43fb496a4 --- /dev/null +++ b/.agents/skills/puzzletron/README.md @@ -0,0 +1,58 @@ +# Puzzletron Agent Skill + +Puzzletron is an end-to-end workflow for model pruning and MIP-based architecture optimization. +This skill exposes it as a slash command for AI coding agents. + +For full environment setup, model configuration, and algorithm details see +[examples/puzzletron/README.md](../../examples/puzzletron/README.md). + +## Using with AI agents + +> **Experimental:** AI agent integration is an experimental feature and may change. + +| Agent | How to invoke | +|---|---| +| **Claude Code** | `/puzzletron ` in the chat | + +## Commands + +### `mip_sweep ` + +Runs the MIP sweep across multiple compression rates. `nproc_per_node` is the number of GPUs per node. + +```text +/puzzletron mip_sweep 4 +``` + +Output is streamed live and also written to `./log.txt`. + +### `mip_sweep progress` + +Parses `./log.txt` and prints a structured progress report: prep steps, per-compression-rate +sub-steps, and a timing summary with elapsed and estimated remaining time. + +```text +/puzzletron mip_sweep progress +``` + +Example output: + +```text +Overall: Puzzletron step 7/8 — MIP sweep (6 compression rates) +────────────────────────────────────────────────────────────── + Status Phase Elapsed +────────────────────────────────────────────────────────────── + [DONE] Prep (teacher memory + rate list) <1s + [DONE] compression_rate=0.5 3m 52s + [RUNNING] compression_rate=0.6 — validating (47/128 batches) 1m 14s + [ ] compression_rate=0.7 pending + [ ] compression_rate=0.8 pending + [ ] compression_rate=0.9 pending + [ ] compression_rate=1.0 pending +────────────────────────────────────────────────────────────── + Started: 08:05:30 + Now: 08:10:56 + Elapsed: 5m 26s + Completed: 1/6 compression rates (avg 3m 52s/rate) + Remaining: ~19m 22s estimated +``` diff --git a/.agents/skills/puzzletron/SKILL.md b/.agents/skills/puzzletron/SKILL.md index c0cab4f02c1..d3a951ba5db 100644 --- a/.agents/skills/puzzletron/SKILL.md +++ b/.agents/skills/puzzletron/SKILL.md @@ -1,6 +1,6 @@ --- name: puzzletron -description: End-to-end workflow for model pruning and MIP-based optimization. Use `all` to run the full workflow or `mip_sweep` to run the MIP sweep. Usage: /puzzletron +description: End-to-end workflow for model pruning and MIP-based optimization. Use `mip_sweep` to run the MIP sweep. Usage: /puzzletron license: Apache-2.0 --- @@ -11,26 +11,154 @@ license: Apache-2.0 **STEP 1 — Check args before doing anything else. This is MANDATORY.** - If args are **empty**, output the block below verbatim and **STOP immediately. Do NOT proceed to any command.** -- If args do **not exactly match** `all` or `mip_sweep`, output the block below verbatim and **STOP immediately. Do NOT proceed to any command.** +- If the first word of args does **not exactly match** `mip_sweep`, output the block below verbatim and **STOP immediately. Do NOT proceed to any command.** --- **Puzzletron** — end-to-end workflow for model pruning and MIP-based optimization. Available commands: -- `all` — Run the full puzzletron workflow -- `mip_sweep` — Run the MIP sweep +- `mip_sweep ` — Run the MIP sweep (nproc_per_node: number of GPUs per node) +- `mip_sweep progress` — Show live MIP sweep progress with timing summary -Usage: `/puzzletron ` +Usage: `/puzzletron [args]` --- -**STEP 2 — Only if args exactly match a command name, execute it. Never reach this step if args were empty.** +**STEP 2 — Only if the first word of args exactly matches a command name, execute it. Never reach this step if args were empty.** -## Command: all +## Command: mip_sweep -Return the following message: hello world: puzzletron all message2 +Parse the second word of args. -## Command: mip_sweep +- If no second word is provided, ask the user: "Please provide the number of GPUs per node (nproc_per_node)." and **STOP**. +- If the second word is exactly `progress`, execute the **mip_sweep progress** sub-command below. +- Otherwise treat the second word as `nproc_per_node` and run the sweep. + +### mip_sweep \ + +Run the following Bash command, substituting `` with the parsed value: + +```bash +torchrun --nproc_per_node examples/puzzletron/main.py \ + --config examples/puzzletron/configs/llama-3_1-8B_pruneffn_memory/llama-3_1-8B_pruneffn_memory.yaml \ + --mip-only 2>&1 | tee ./log.txt | grep "Puzzletron Progress" +``` + +Stream output to the user as it arrives. When the command finishes, report the exit code. + +### mip_sweep progress + +Run the following Python script verbatim. Do not modify it. Present the output to the user wrapped in a fenced code block (``` ... ```). + +```bash +python3 - << 'PYEOF' +import re, sys +from datetime import datetime + +LOG = './log.txt' +try: + lines = open(LOG).readlines() + text = ''.join(lines) +except FileNotFoundError: + print("No log.txt found. Run /puzzletron mip_sweep first.") + sys.exit(0) + +def norm(r): return str(float(r)) +def fmt(s): return f"{int(s)//60}m {int(s)%60}s" if s is not None else "—" +def get_ts(line): + m = re.search(r'\[(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})', line) + return datetime.strptime(m.group(1), '%Y-%m-%d %H:%M:%S') if m else None + +rates_match = re.search(r'Compression rates: \[(.*?)\]', text) +all_rates = [norm(r.strip()) for r in rates_match.group(1).split(',')] if rates_match else [] + +# Collect start timestamp per rate +rate_start = {} +for line in lines: + if 'sweep.py:258' in line: + m = re.search(r'compression_rate=([\d.]+)', line) + if m: + r = norm(m.group(1)) + if r in all_rates and r not in rate_start: + rate_start[r] = get_ts(line) + +now = datetime.now().replace(microsecond=0) +sweep_start = rate_start.get(all_rates[0]) if all_rates else None + +# Rate is done when the next rate has started; last rate done when sweep.py:287 appears +rate_done = set() +for i, r in enumerate(all_rates[:-1]): + if all_rates[i + 1] in rate_start: + rate_done.add(r) +last = all_rates[-1] +sweep_complete_ts = None +for line in lines: + ts = get_ts(line) + if ts and 'sweep.py:292' in line: + sweep_complete_ts = ts + break +if sweep_complete_ts and last in rate_start: + rate_done.add(last) + +# Per-rate elapsed = next rate start - this rate start (or completion ts or now for last) +rate_elapsed = {} +for i, r in enumerate(all_rates): + if r not in rate_start: + continue + if i + 1 < len(all_rates): + end = rate_start[all_rates[i + 1]] + else: + end = sweep_complete_ts if sweep_complete_ts else now + rate_elapsed[r] = int((end - rate_start[r]).total_seconds()) + +# Currently running rate +running_rate = next((r for r in all_rates if r in rate_start and r not in rate_done), None) + +# Sub-step detail for running rate +cur_detail = "" +if running_rate: + batch_matches = re.findall(r'calculate_losses_pipeline[^:]*:\s*(\d+)%.*?(\d+)/(\d+)', text) + cbc_matches = re.findall(r'After (\d+) nodes.*?\(([\d.]+) seconds\)', text) + if batch_matches: + pct, cur, total = batch_matches[-1] + cur_detail = f" — validating ({cur}/{total} batches)" + elif cbc_matches: + nodes, secs = cbc_matches[-1] + cur_detail = f" — MIP solver ({int(nodes):,} nodes, {float(secs):.1f}s)" + +end_ts = sweep_complete_ts if sweep_complete_ts else now +total_elapsed = int((end_ts - sweep_start).total_seconds()) if sweep_start else 0 + +done_count = len(rate_done) +remaining_count = len(all_rates) - done_count +avg_s = sum(rate_elapsed[r] for r in rate_done) / done_count if done_count else None +est_rem = fmt(avg_s * remaining_count) if avg_s and remaining_count else ("done" if not remaining_count else "calculating...") + +DIV = '─' * 62 -Return the following message: hello world: puzzletron mip sweep2 +print(f"\nOverall: Puzzletron step 7/8 — MIP sweep ({len(all_rates)} compression rates)") +print(DIV) +print(f" {'Status':<10} {'Phase':<32} {'Elapsed':>8}") +print(DIV) +print(f" [DONE] {'Prep (teacher memory + rate list)':<32} {'<1s':>8}") +for r in all_rates: + if r not in rate_start: + print(f" [ ] {f'compression_rate={r}':<32} {'pending':>8}") + elif r == running_rate: + detail = cur_detail + print(f" [RUNNING] {f'compression_rate={r}{detail}':<32} {fmt(rate_elapsed.get(r)):>8}") + else: + print(f" [DONE] {f'compression_rate={r}':<32} {fmt(rate_elapsed.get(r)):>8}") +print(DIV) +print(f" Started: {sweep_start.strftime('%H:%M:%S') if sweep_start else '—'}") +print(f" Finished: {sweep_complete_ts.strftime('%H:%M:%S') if sweep_complete_ts else now.strftime('%H:%M:%S') + ' (in progress)'}") +print(f" Elapsed: {fmt(total_elapsed)}") +print(f" Completed: {done_count}/{len(all_rates)} compression rates", end="") +print(f" (avg {fmt(avg_s)}/rate)" if avg_s else "") +print(f" Remaining: {est_rem} estimated") +results_match = re.search(r'Results written to: (\S+)', text) +if results_match: + print(f"\n Results: {results_match.group(1)}") +PYEOF +``` diff --git a/examples/puzzletron/README.md b/examples/puzzletron/README.md index 48954a2b773..d5ce1e4535c 100644 --- a/examples/puzzletron/README.md +++ b/examples/puzzletron/README.md @@ -388,3 +388,10 @@ Due to non-linear extension of the runtime stats of single subblocks to the tota ## Advanced Usage Modify `llama-3_1-8B_pruneffn_memory.yaml` file for advanced compression scenarios. + +## Using with AI agents + +> **Experimental:** AI agent integration is an experimental feature and may change. + +Puzzletron ships a skill for AI coding agents (Claude Code, Cursor, Codex). +See [`.agents/skills/puzzletron/README.md`](../../.agents/skills/puzzletron/README.md) for setup, commands, and example output. From 1e794636d8fc2f9b8bde9ef191a593cdb7d09208 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Wed, 17 Jun 2026 09:02:59 -0700 Subject: [PATCH 03/17] update puzzletron readme Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/README.md | 51 +++++++++++++---------------- 1 file changed, 23 insertions(+), 28 deletions(-) diff --git a/.agents/skills/puzzletron/README.md b/.agents/skills/puzzletron/README.md index af43fb496a4..62e94f54fd5 100644 --- a/.agents/skills/puzzletron/README.md +++ b/.agents/skills/puzzletron/README.md @@ -6,53 +6,48 @@ This skill exposes it as a slash command for AI coding agents. For full environment setup, model configuration, and algorithm details see [examples/puzzletron/README.md](../../examples/puzzletron/README.md). -## Using with AI agents - > **Experimental:** AI agent integration is an experimental feature and may change. -| Agent | How to invoke | -|---|---| -| **Claude Code** | `/puzzletron ` in the chat | - -## Commands +Run `/puzzletron` with no arguments to see available commands. -### `mip_sweep ` +## Running the MIP sweep -Runs the MIP sweep across multiple compression rates. `nproc_per_node` is the number of GPUs per node. +Start the sweep by telling the agent how many GPUs per node to use: ```text /puzzletron mip_sweep 4 ``` -Output is streamed live and also written to `./log.txt`. - -### `mip_sweep progress` - -Parses `./log.txt` and prints a structured progress report: prep steps, per-compression-rate -sub-steps, and a timing summary with elapsed and estimated remaining time. +Output is streamed live and also written to `./log.txt`. While it runs (or after it finishes), +check progress with: ```text /puzzletron mip_sweep progress ``` -Example output: +Example output when complete: ```text Overall: Puzzletron step 7/8 — MIP sweep (6 compression rates) ────────────────────────────────────────────────────────────── - Status Phase Elapsed + Status Phase Elapsed ────────────────────────────────────────────────────────────── - [DONE] Prep (teacher memory + rate list) <1s - [DONE] compression_rate=0.5 3m 52s - [RUNNING] compression_rate=0.6 — validating (47/128 batches) 1m 14s - [ ] compression_rate=0.7 pending - [ ] compression_rate=0.8 pending - [ ] compression_rate=0.9 pending - [ ] compression_rate=1.0 pending + [DONE] Prep (teacher memory + rate list) <1s + [DONE] compression_rate=0.5 3m 52s + [DONE] compression_rate=0.6 4m 41s + [DONE] compression_rate=0.7 4m 46s + [DONE] compression_rate=0.8 3m 55s + [DONE] compression_rate=0.9 3m 55s + [DONE] compression_rate=1.0 3m 59s ────────────────────────────────────────────────────────────── Started: 08:05:30 - Now: 08:10:56 - Elapsed: 5m 26s - Completed: 1/6 compression rates (avg 3m 52s/rate) - Remaining: ~19m 22s estimated + Finished: 08:30:38 + Elapsed: 25m 8s + Completed: 6/6 compression rates (avg 4m 11s/rate) + Remaining: done estimated + + Results: /workspace/puzzle_dir/mip_sweep_results.csv ``` + +While running, the report shows which rate is active, sub-step detail (MIP solver node count +or validation batch progress), and an estimated time remaining based on completed rates. From f99c01f837be740769284de76f84f766c50c2df9 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Wed, 17 Jun 2026 09:09:47 -0700 Subject: [PATCH 04/17] fix typo in SKILL.md Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/SKILL.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.agents/skills/puzzletron/SKILL.md b/.agents/skills/puzzletron/SKILL.md index d3a951ba5db..87258b0cf44 100644 --- a/.agents/skills/puzzletron/SKILL.md +++ b/.agents/skills/puzzletron/SKILL.md @@ -1,6 +1,6 @@ --- name: puzzletron -description: End-to-end workflow for model pruning and MIP-based optimization. Use `mip_sweep` to run the MIP sweep. Usage: /puzzletron +description: "End-to-end workflow for model pruning and MIP-based optimization. Use `mip_sweep` to run the MIP sweep. Usage: /puzzletron " license: Apache-2.0 --- From 131296c1cf46fecd564e2f750806bbf19024f0c8 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Thu, 18 Jun 2026 00:01:21 -0700 Subject: [PATCH 05/17] Add PYTHONPATH to mip_sweep skill Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/SKILL.md | 1 + 1 file changed, 1 insertion(+) diff --git a/.agents/skills/puzzletron/SKILL.md b/.agents/skills/puzzletron/SKILL.md index 87258b0cf44..afda7a25b8a 100644 --- a/.agents/skills/puzzletron/SKILL.md +++ b/.agents/skills/puzzletron/SKILL.md @@ -40,6 +40,7 @@ Parse the second word of args. Run the following Bash command, substituting `` with the parsed value: ```bash +export PYTHONPATH=$PYTHONPATH:/workspace/Model-Optimizer && \ torchrun --nproc_per_node examples/puzzletron/main.py \ --config examples/puzzletron/configs/llama-3_1-8B_pruneffn_memory/llama-3_1-8B_pruneffn_memory.yaml \ --mip-only 2>&1 | tee ./log.txt | grep "Puzzletron Progress" From c5c201561111212929aa4c70f0af1b6f4bf1fb7a Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Thu, 18 Jun 2026 00:52:58 -0700 Subject: [PATCH 06/17] add skill for puzzletron all Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/README.md | 42 +++++- .agents/skills/puzzletron/SKILL.md | 219 +++++++++++++++++++++++++++- 2 files changed, 254 insertions(+), 7 deletions(-) diff --git a/.agents/skills/puzzletron/README.md b/.agents/skills/puzzletron/README.md index 62e94f54fd5..bb2aa4220aa 100644 --- a/.agents/skills/puzzletron/README.md +++ b/.agents/skills/puzzletron/README.md @@ -43,7 +43,7 @@ Overall: Puzzletron step 7/8 — MIP sweep (6 compression rates) Started: 08:05:30 Finished: 08:30:38 Elapsed: 25m 8s - Completed: 6/6 compression rates (avg 4m 11s/rate) + Completed: 6/6 compression rates Remaining: done estimated Results: /workspace/puzzle_dir/mip_sweep_results.csv @@ -51,3 +51,43 @@ Overall: Puzzletron step 7/8 — MIP sweep (6 compression rates) While running, the report shows which rate is active, sub-step detail (MIP solver node count or validation batch progress), and an estimated time remaining based on completed rates. + +## Running the full pipeline + +To run all 8 pipeline steps (not just the MIP sweep): + +```text +/puzzletron all 2 +``` + +Check progress with: + +```text +/puzzletron all progress +``` + +Example output while running: + +```text +Overall: Puzzletron full pipeline (steps 1–8) +──────────────────────────────────────────────────────────────────── + Status Step Description Elapsed +──────────────────────────────────────────────────────────────────── + [DONE] 1/8: starting puzzletron pipeline 0m 0s + [DONE] 2/8: converting model to Puzzletron heterogeneous format (single-gpu) 0m 26s + [DONE] 3/8: scoring pruning activations (multi-gpu) 9m 9s + [DONE] 4/8: pruning the model and saving pruned checkpoints (single-gpu) 0m 57s + [DONE] 5/8: building replacement library and subblock statistics (single-gpu) 0m 26s + [RUNNING] 6/8: calculating one block scores (multi-gpu) (76/352 solutions) 27m 53s + [ ] 7/8: pending + [ ] 8/8: pending +──────────────────────────────────────────────────────────────────── + Started: 00:08:50 + Finished: 00:47:41 (in progress) + Elapsed: 38m 51s + Completed: 5/8 steps + Remaining: 105m 38s estimated +``` + +Step 6 progress is tracked via completed `solution_N.json` files on disk for an accurate +remaining estimate. Step 7 (MIP sweep) shows per-rate progress once it starts. diff --git a/.agents/skills/puzzletron/SKILL.md b/.agents/skills/puzzletron/SKILL.md index afda7a25b8a..35b05b7f04e 100644 --- a/.agents/skills/puzzletron/SKILL.md +++ b/.agents/skills/puzzletron/SKILL.md @@ -11,7 +11,7 @@ license: Apache-2.0 **STEP 1 — Check args before doing anything else. This is MANDATORY.** - If args are **empty**, output the block below verbatim and **STOP immediately. Do NOT proceed to any command.** -- If the first word of args does **not exactly match** `mip_sweep`, output the block below verbatim and **STOP immediately. Do NOT proceed to any command.** +- If the first word of args does **not exactly match** `mip_sweep` or `all`, output the block below verbatim and **STOP immediately. Do NOT proceed to any command.** --- @@ -20,6 +20,8 @@ license: Apache-2.0 Available commands: - `mip_sweep ` — Run the MIP sweep (nproc_per_node: number of GPUs per node) - `mip_sweep progress` — Show live MIP sweep progress with timing summary +- `all ` — Run the full Puzzletron pipeline (nproc_per_node: number of GPUs per node) +- `all progress` — Show live full pipeline progress with timing summary Usage: `/puzzletron [args]` @@ -27,13 +29,219 @@ Usage: `/puzzletron [args]` **STEP 2 — Only if the first word of args exactly matches a command name, execute it. Never reach this step if args were empty.** +## Command: all + +Parse `nproc_per_node` from args using either positional or flag syntax: +- Positional: second word is a number, e.g. `all 2` +- Flag: `--nproc_per_node ` anywhere in args, e.g. `all --nproc_per_node 2` + +- If the second word is exactly `progress`, execute the **all progress** sub-command below. +- If no `nproc_per_node` value can be found, ask the user: "Please provide the number of GPUs per node (nproc_per_node)." and **STOP**. +- Otherwise use the parsed value and run the full pipeline. + +### all \ + +Run the following Bash command, substituting `` with the parsed value: + +```bash +export PYTHONPATH=$PYTHONPATH:/workspace/Model-Optimizer && \ +torchrun --nproc_per_node examples/puzzletron/main.py \ + --config examples/puzzletron/configs/llama-3_1-8B_pruneffn_memory/llama-3_1-8B_pruneffn_memory.yaml \ + 2>&1 | tee ./log.txt | grep "Puzzletron Progress" +``` + +Stream output to the user as it arrives. When the command finishes, report the exit code. + +### all progress + +Run the following Python script verbatim. Do not modify it. Present the output to the user wrapped in a fenced code block (``` ... ```). + +```bash +python3 - << 'PYEOF' +import re, sys +from datetime import datetime + +LOG = './log.txt' +try: + lines = open(LOG).readlines() + text = ''.join(lines) +except FileNotFoundError: + print("No log.txt found. Run /puzzletron all first.") + sys.exit(0) + +def norm(r): return str(float(r)) +def fmt(s): return f"{int(s)//60}m {int(s)%60}s" if s is not None else "—" +def get_ts(line): + m = re.search(r'\[(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})', line) + return datetime.strptime(m.group(1), '%Y-%m-%d %H:%M:%S') if m else None + +now = datetime.now().replace(microsecond=0) +DIV = '─' * 68 + +# Parse pipeline steps from "Puzzletron Progress X/8: " lines +step_events = [] +for line in lines: + m = re.search(r'Puzzletron Progress (\d+)/(\d+): (.+)', line) + if m: + step_num = int(m.group(1)) + total_steps = int(m.group(2)) + desc = m.group(3).strip() + ts = get_ts(line) + step_events.append((step_num, total_steps, desc, ts)) + +total_steps = step_events[-1][1] if step_events else 8 +seen_steps = {e[0]: (e[2], e[3]) for e in step_events} +last_step_num = max(seen_steps.keys()) if seen_steps else 0 + +# Determine if pipeline is fully complete +pipeline_complete_ts = None +for line in lines: + ts = get_ts(line) + if ts and 'sweep.py:292' in line: + pipeline_complete_ts = ts + break + +# Sub-step detail and remaining estimate for current step +cur_detail = "" +step_remaining = None +batch_matches = re.findall(r'calculate_losses_pipeline[^:]*:\s*(\d+)%.*?(\d+)/(\d+)', text) +cbc_matches = re.findall(r'After (\d+) nodes.*?\(([\d.]+) seconds\)', text) +# Step 6: count completed solution files for real progress +import glob as _glob, os as _os +sol_dir_match = re.search(r"'output_dir': '([^']+single_sequence_replacement_solutions--validation[^']*)'", text) +sol_done, sol_total = None, None +if sol_dir_match: + sol_dir = sol_dir_match.group(1) + sol_files = _glob.glob(f"{sol_dir}/solution*.json") + sol_done = len(sol_files) + sol_list_match = re.search(r"'solutions_to_validate': \[([\d, ]+)\]", text) + if sol_list_match: + sol_total = len(sol_list_match.group(1).split(',')) +if sol_done is not None and sol_total: + cur_detail = f" ({sol_done}/{sol_total} solutions)" +elif batch_matches: + pct, cur_b, total_b = batch_matches[-1] + cur_detail = f" ({cur_b}/{total_b} batches)" +elif cbc_matches: + nodes, secs = cbc_matches[-1] + cur_detail = f" (MIP solver: {int(nodes):,} nodes, {float(secs):.1f}s)" + +# MIP sweep compression rate detail (step 7) +rates_match = re.search(r'Compression rates: \[(.*?)\]', text) +all_rates = [norm(r.strip()) for r in rates_match.group(1).split(',')] if rates_match else [] +rate_start = {} +for line in lines: + if 'sweep.py:258' in line: + m = re.search(r'compression_rate=([\d.]+)', line) + if m: + r = norm(m.group(1)) + if r in all_rates and r not in rate_start: + rate_start[r] = get_ts(line) +rate_done = set() +for i, r in enumerate(all_rates[:-1]): + if all_rates[i + 1] in rate_start: + rate_done.add(r) +if pipeline_complete_ts and all_rates and all_rates[-1] in rate_start: + rate_done.add(all_rates[-1]) + +# Overall timing +pipeline_start = step_events[0][3] if step_events else None +end_ts = pipeline_complete_ts if pipeline_complete_ts else now +total_elapsed = int((end_ts - pipeline_start).total_seconds()) if pipeline_start else 0 + +# Estimate remaining time for current running step +step_ts_list = sorted(seen_steps.items()) +cur_step_start_ts = seen_steps[last_step_num][1] if last_step_num in seen_steps else None +if not pipeline_complete_ts and cur_step_start_ts: + cur_step_elapsed = int((now - cur_step_start_ts).total_seconds()) + if sol_done and sol_total and sol_done > 0: + rate_per_sol = cur_step_elapsed / sol_done + step_remaining = rate_per_sol * (sol_total - sol_done) + elif batch_matches and int(cur_b) > 0 and int(cur_b) < int(total_b): + rate_per_batch = cur_step_elapsed / int(cur_b) + step_remaining = rate_per_batch * (int(total_b) - int(cur_b)) + elif all_rates and last_step_num == 7: + done_count_r = len(rate_done) + remaining_count_r = len(all_rates) - done_count_r + rate_elapsed = {} + for i, r in enumerate(all_rates): + if r not in rate_start: + continue + if i + 1 < len(all_rates) and all_rates[i+1] in rate_start: + rate_elapsed[r] = int((rate_start[all_rates[i+1]] - rate_start[r]).total_seconds()) + avg_r = sum(rate_elapsed.values()) / len(rate_elapsed) if rate_elapsed else None + if avg_r and remaining_count_r: + step_remaining = avg_r * remaining_count_r + +print(f"\nOverall: Puzzletron full pipeline (steps 1–{total_steps})") +print(DIV) +print(f" {'Status':<10} {'Step':<4} {'Description':<34} {'Elapsed':>8}") +print(DIV) + +for i, (snum, (sdesc, sts)) in enumerate(step_ts_list): + next_ts = step_ts_list[i+1][1][1] if i+1 < len(step_ts_list) else (pipeline_complete_ts if pipeline_complete_ts else now) + elapsed = int((next_ts - sts).total_seconds()) if sts and next_ts else None + is_last = (snum == last_step_num) + is_done = not is_last or pipeline_complete_ts is not None + detail = "" + if is_last and not is_done: + detail = cur_detail + if snum == 7 and all_rates: + detail = f" ({len(rate_done)}/{len(all_rates)} rates done)" + label = f"{snum}/{total_steps}: {sdesc}{detail}" + status = "[DONE]" if is_done else "[RUNNING]" + print(f" {status:<10} {'':<4} {label:<34} {fmt(elapsed) if elapsed is not None else '—':>8}") + +for snum in range(last_step_num + 1, total_steps + 1): + print(f" {'[ ]':<10} {'':<4} {f'{snum}/{total_steps}: pending':<34} {'':>8}") + +print(DIV) +if all_rates and last_step_num >= 7: + print(f" MIP rates: {len(rate_done)}/{len(all_rates)} done", end="") + running_rate = next((r for r in all_rates if r in rate_start and r not in rate_done), None) + if running_rate: + print(f" (running: {running_rate})", end="") + print() +done_steps = len([s for s in seen_steps if s != last_step_num or pipeline_complete_ts]) +avg_step_s = None +step_durations = [] +for i, (snum, (sdesc, sts)) in enumerate(step_ts_list): + next_ts = step_ts_list[i+1][1][1] if i+1 < len(step_ts_list) else (pipeline_complete_ts if pipeline_complete_ts else None) + if next_ts and sts: + step_durations.append(int((next_ts - sts).total_seconds())) +if step_durations: + avg_step_s = sum(step_durations) / len(step_durations) +remaining_steps_after = total_steps - last_step_num # steps not yet started after current +if pipeline_complete_ts: + est_rem = "done" +elif step_remaining is not None: + future_est = (avg_step_s * remaining_steps_after) if avg_step_s else 0 + est_rem = fmt(step_remaining + future_est) +elif avg_step_s: + est_rem = fmt(avg_step_s * (remaining_steps_after + 1)) +else: + est_rem = "calculating..." + +print(f" Started: {pipeline_start.strftime('%H:%M:%S') if pipeline_start else '—'}") +print(f" Finished: {pipeline_complete_ts.strftime('%H:%M:%S') if pipeline_complete_ts else now.strftime('%H:%M:%S') + ' (in progress)'}") +print(f" Elapsed: {fmt(total_elapsed)}") +print(f" Completed: {done_steps}/{total_steps} steps") +print(f" Remaining: {est_rem} estimated") +results_match = re.search(r'Results written to: (\S+)', text) +if results_match: + print(f"\n Results: {results_match.group(1)}") +PYEOF +``` + ## Command: mip_sweep -Parse the second word of args. +Parse `nproc_per_node` from args using either positional or flag syntax: +- Positional: second word is a number, e.g. `mip_sweep 2` +- Flag: `--nproc_per_node ` anywhere in args, e.g. `mip_sweep --nproc_per_node 2` -- If no second word is provided, ask the user: "Please provide the number of GPUs per node (nproc_per_node)." and **STOP**. - If the second word is exactly `progress`, execute the **mip_sweep progress** sub-command below. -- Otherwise treat the second word as `nproc_per_node` and run the sweep. +- If no `nproc_per_node` value can be found, ask the user: "Please provide the number of GPUs per node (nproc_per_node)." and **STOP**. +- Otherwise use the parsed value and run the sweep. ### mip_sweep \ @@ -155,8 +363,7 @@ print(DIV) print(f" Started: {sweep_start.strftime('%H:%M:%S') if sweep_start else '—'}") print(f" Finished: {sweep_complete_ts.strftime('%H:%M:%S') if sweep_complete_ts else now.strftime('%H:%M:%S') + ' (in progress)'}") print(f" Elapsed: {fmt(total_elapsed)}") -print(f" Completed: {done_count}/{len(all_rates)} compression rates", end="") -print(f" (avg {fmt(avg_s)}/rate)" if avg_s else "") +print(f" Completed: {done_count}/{len(all_rates)} compression rates") print(f" Remaining: {est_rem} estimated") results_match = re.search(r'Results written to: (\S+)', text) if results_match: From 1a62799925c522642eca2a375737dd8c393dfab8 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Thu, 18 Jun 2026 01:11:11 -0700 Subject: [PATCH 07/17] update progress bar Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/SKILL.md | 37 +++++++++++++++++++++++++----- 1 file changed, 31 insertions(+), 6 deletions(-) diff --git a/.agents/skills/puzzletron/SKILL.md b/.agents/skills/puzzletron/SKILL.md index 35b05b7f04e..24a647294d1 100644 --- a/.agents/skills/puzzletron/SKILL.md +++ b/.agents/skills/puzzletron/SKILL.md @@ -211,16 +211,41 @@ for i, (snum, (sdesc, sts)) in enumerate(step_ts_list): step_durations.append(int((next_ts - sts).total_seconds())) if step_durations: avg_step_s = sum(step_durations) / len(step_durations) -remaining_steps_after = total_steps - last_step_num # steps not yet started after current + +# Parse config for sweep settings to estimate step 7 duration +CONFIG_PATH = 'examples/puzzletron/configs/llama-3_1-8B_pruneffn_memory/llama-3_1-8B_pruneffn_memory.yaml' +sweep_enabled = True +sweep_n_rates = 6 +try: + cfg_text = open(CONFIG_PATH).read() + _en_m = re.search(r'sweep:\s*\n\s+enabled:\s*(true|false)', cfg_text) + if _en_m: + sweep_enabled = _en_m.group(1) == 'true' + _rates_m = re.search(r'memory_compression_rates:\s*\[([^\]]+)\]', cfg_text) + if _rates_m: + sweep_n_rates = len(_rates_m.group(1).split(',')) +except Exception: + pass +# Prefer actual rate count from log if step 7 has already started +effective_n_rates = len(all_rates) if all_rates else sweep_n_rates +RATE_S = 250 # ~4m 10s per compression rate (historical) + +def step_est(snum): + if snum == 7: + return (RATE_S * effective_n_rates) if sweep_enabled else 120 + elif snum == 8: + return 60 + return avg_step_s or 0 + if pipeline_complete_ts: est_rem = "done" elif step_remaining is not None: - future_est = (avg_step_s * remaining_steps_after) if avg_step_s else 0 - est_rem = fmt(step_remaining + future_est) -elif avg_step_s: - est_rem = fmt(avg_step_s * (remaining_steps_after + 1)) + future_s = sum(step_est(s) for s in range(last_step_num + 1, total_steps + 1)) + est_rem = fmt(step_remaining + future_s) else: - est_rem = "calculating..." + cur_s = step_est(last_step_num) + future_s = cur_s + sum(step_est(s) for s in range(last_step_num + 1, total_steps + 1)) + est_rem = fmt(future_s) if (cur_s or future_s) else "calculating..." print(f" Started: {pipeline_start.strftime('%H:%M:%S') if pipeline_start else '—'}") print(f" Finished: {pipeline_complete_ts.strftime('%H:%M:%S') if pipeline_complete_ts else now.strftime('%H:%M:%S') + ' (in progress)'}") From fa197c2bb9809c908f67feb95c0f78aa702a05cb Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Thu, 18 Jun 2026 01:14:40 -0700 Subject: [PATCH 08/17] Rename mip_sweep to mip command Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/README.md | 8 ++++---- .agents/skills/puzzletron/SKILL.md | 22 +++++++++++----------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/.agents/skills/puzzletron/README.md b/.agents/skills/puzzletron/README.md index bb2aa4220aa..e4baa46fcb3 100644 --- a/.agents/skills/puzzletron/README.md +++ b/.agents/skills/puzzletron/README.md @@ -10,19 +10,19 @@ For full environment setup, model configuration, and algorithm details see Run `/puzzletron` with no arguments to see available commands. -## Running the MIP sweep +## Running the MIP step -Start the sweep by telling the agent how many GPUs per node to use: +Start the MIP step by telling the agent how many GPUs per node to use: ```text -/puzzletron mip_sweep 4 +/puzzletron mip 4 ``` Output is streamed live and also written to `./log.txt`. While it runs (or after it finishes), check progress with: ```text -/puzzletron mip_sweep progress +/puzzletron mip progress ``` Example output when complete: diff --git a/.agents/skills/puzzletron/SKILL.md b/.agents/skills/puzzletron/SKILL.md index 24a647294d1..3cdfb7a2fa7 100644 --- a/.agents/skills/puzzletron/SKILL.md +++ b/.agents/skills/puzzletron/SKILL.md @@ -11,15 +11,15 @@ license: Apache-2.0 **STEP 1 — Check args before doing anything else. This is MANDATORY.** - If args are **empty**, output the block below verbatim and **STOP immediately. Do NOT proceed to any command.** -- If the first word of args does **not exactly match** `mip_sweep` or `all`, output the block below verbatim and **STOP immediately. Do NOT proceed to any command.** +- If the first word of args does **not exactly match** `mip` or `all`, output the block below verbatim and **STOP immediately. Do NOT proceed to any command.** --- **Puzzletron** — end-to-end workflow for model pruning and MIP-based optimization. Available commands: -- `mip_sweep ` — Run the MIP sweep (nproc_per_node: number of GPUs per node) -- `mip_sweep progress` — Show live MIP sweep progress with timing summary +- `mip ` — Run the MIP step (nproc_per_node: number of GPUs per node) +- `mip progress` — Show live MIP progress with timing summary - `all ` — Run the full Puzzletron pipeline (nproc_per_node: number of GPUs per node) - `all progress` — Show live full pipeline progress with timing summary @@ -258,17 +258,17 @@ if results_match: PYEOF ``` -## Command: mip_sweep +## Command: mip Parse `nproc_per_node` from args using either positional or flag syntax: -- Positional: second word is a number, e.g. `mip_sweep 2` -- Flag: `--nproc_per_node ` anywhere in args, e.g. `mip_sweep --nproc_per_node 2` +- Positional: second word is a number, e.g. `mip 2` +- Flag: `--nproc_per_node ` anywhere in args, e.g. `mip --nproc_per_node 2` -- If the second word is exactly `progress`, execute the **mip_sweep progress** sub-command below. +- If the second word is exactly `progress`, execute the **mip progress** sub-command below. - If no `nproc_per_node` value can be found, ask the user: "Please provide the number of GPUs per node (nproc_per_node)." and **STOP**. -- Otherwise use the parsed value and run the sweep. +- Otherwise use the parsed value and run the MIP step. -### mip_sweep \ +### mip \ Run the following Bash command, substituting `` with the parsed value: @@ -281,7 +281,7 @@ torchrun --nproc_per_node examples/puzzletron/main.py \ Stream output to the user as it arrives. When the command finishes, report the exit code. -### mip_sweep progress +### mip progress Run the following Python script verbatim. Do not modify it. Present the output to the user wrapped in a fenced code block (``` ... ```). @@ -295,7 +295,7 @@ try: lines = open(LOG).readlines() text = ''.join(lines) except FileNotFoundError: - print("No log.txt found. Run /puzzletron mip_sweep first.") + print("No log.txt found. Run /puzzletron mip first.") sys.exit(0) def norm(r): return str(float(r)) From 03ceee301528ead4ceeecb54b5b8790ff9eb5093 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Thu, 18 Jun 2026 01:57:30 -0700 Subject: [PATCH 09/17] Move python scripts for puzzletron claude command to py scripts. Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/SKILL.md | 315 +--------------------- .agents/skills/puzzletron/all_progress.py | 237 ++++++++++++++++ .agents/skills/puzzletron/mip_progress.py | 141 ++++++++++ 3 files changed, 383 insertions(+), 310 deletions(-) create mode 100644 .agents/skills/puzzletron/all_progress.py create mode 100644 .agents/skills/puzzletron/mip_progress.py diff --git a/.agents/skills/puzzletron/SKILL.md b/.agents/skills/puzzletron/SKILL.md index 3cdfb7a2fa7..a3d51ab799b 100644 --- a/.agents/skills/puzzletron/SKILL.md +++ b/.agents/skills/puzzletron/SKILL.md @@ -1,6 +1,6 @@ --- name: puzzletron -description: "End-to-end workflow for model pruning and MIP-based optimization. Use `mip_sweep` to run the MIP sweep. Usage: /puzzletron " +description: "End-to-end workflow for model pruning and MIP-based optimization. Commands: mip, all. Usage: /puzzletron " license: Apache-2.0 --- @@ -54,208 +54,10 @@ Stream output to the user as it arrives. When the command finishes, report the e ### all progress -Run the following Python script verbatim. Do not modify it. Present the output to the user wrapped in a fenced code block (``` ... ```). +Run the following Bash command. Present the output to the user wrapped in a fenced code block (``` ... ```). ```bash -python3 - << 'PYEOF' -import re, sys -from datetime import datetime - -LOG = './log.txt' -try: - lines = open(LOG).readlines() - text = ''.join(lines) -except FileNotFoundError: - print("No log.txt found. Run /puzzletron all first.") - sys.exit(0) - -def norm(r): return str(float(r)) -def fmt(s): return f"{int(s)//60}m {int(s)%60}s" if s is not None else "—" -def get_ts(line): - m = re.search(r'\[(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})', line) - return datetime.strptime(m.group(1), '%Y-%m-%d %H:%M:%S') if m else None - -now = datetime.now().replace(microsecond=0) -DIV = '─' * 68 - -# Parse pipeline steps from "Puzzletron Progress X/8: " lines -step_events = [] -for line in lines: - m = re.search(r'Puzzletron Progress (\d+)/(\d+): (.+)', line) - if m: - step_num = int(m.group(1)) - total_steps = int(m.group(2)) - desc = m.group(3).strip() - ts = get_ts(line) - step_events.append((step_num, total_steps, desc, ts)) - -total_steps = step_events[-1][1] if step_events else 8 -seen_steps = {e[0]: (e[2], e[3]) for e in step_events} -last_step_num = max(seen_steps.keys()) if seen_steps else 0 - -# Determine if pipeline is fully complete -pipeline_complete_ts = None -for line in lines: - ts = get_ts(line) - if ts and 'sweep.py:292' in line: - pipeline_complete_ts = ts - break - -# Sub-step detail and remaining estimate for current step -cur_detail = "" -step_remaining = None -batch_matches = re.findall(r'calculate_losses_pipeline[^:]*:\s*(\d+)%.*?(\d+)/(\d+)', text) -cbc_matches = re.findall(r'After (\d+) nodes.*?\(([\d.]+) seconds\)', text) -# Step 6: count completed solution files for real progress -import glob as _glob, os as _os -sol_dir_match = re.search(r"'output_dir': '([^']+single_sequence_replacement_solutions--validation[^']*)'", text) -sol_done, sol_total = None, None -if sol_dir_match: - sol_dir = sol_dir_match.group(1) - sol_files = _glob.glob(f"{sol_dir}/solution*.json") - sol_done = len(sol_files) - sol_list_match = re.search(r"'solutions_to_validate': \[([\d, ]+)\]", text) - if sol_list_match: - sol_total = len(sol_list_match.group(1).split(',')) -if sol_done is not None and sol_total: - cur_detail = f" ({sol_done}/{sol_total} solutions)" -elif batch_matches: - pct, cur_b, total_b = batch_matches[-1] - cur_detail = f" ({cur_b}/{total_b} batches)" -elif cbc_matches: - nodes, secs = cbc_matches[-1] - cur_detail = f" (MIP solver: {int(nodes):,} nodes, {float(secs):.1f}s)" - -# MIP sweep compression rate detail (step 7) -rates_match = re.search(r'Compression rates: \[(.*?)\]', text) -all_rates = [norm(r.strip()) for r in rates_match.group(1).split(',')] if rates_match else [] -rate_start = {} -for line in lines: - if 'sweep.py:258' in line: - m = re.search(r'compression_rate=([\d.]+)', line) - if m: - r = norm(m.group(1)) - if r in all_rates and r not in rate_start: - rate_start[r] = get_ts(line) -rate_done = set() -for i, r in enumerate(all_rates[:-1]): - if all_rates[i + 1] in rate_start: - rate_done.add(r) -if pipeline_complete_ts and all_rates and all_rates[-1] in rate_start: - rate_done.add(all_rates[-1]) - -# Overall timing -pipeline_start = step_events[0][3] if step_events else None -end_ts = pipeline_complete_ts if pipeline_complete_ts else now -total_elapsed = int((end_ts - pipeline_start).total_seconds()) if pipeline_start else 0 - -# Estimate remaining time for current running step -step_ts_list = sorted(seen_steps.items()) -cur_step_start_ts = seen_steps[last_step_num][1] if last_step_num in seen_steps else None -if not pipeline_complete_ts and cur_step_start_ts: - cur_step_elapsed = int((now - cur_step_start_ts).total_seconds()) - if sol_done and sol_total and sol_done > 0: - rate_per_sol = cur_step_elapsed / sol_done - step_remaining = rate_per_sol * (sol_total - sol_done) - elif batch_matches and int(cur_b) > 0 and int(cur_b) < int(total_b): - rate_per_batch = cur_step_elapsed / int(cur_b) - step_remaining = rate_per_batch * (int(total_b) - int(cur_b)) - elif all_rates and last_step_num == 7: - done_count_r = len(rate_done) - remaining_count_r = len(all_rates) - done_count_r - rate_elapsed = {} - for i, r in enumerate(all_rates): - if r not in rate_start: - continue - if i + 1 < len(all_rates) and all_rates[i+1] in rate_start: - rate_elapsed[r] = int((rate_start[all_rates[i+1]] - rate_start[r]).total_seconds()) - avg_r = sum(rate_elapsed.values()) / len(rate_elapsed) if rate_elapsed else None - if avg_r and remaining_count_r: - step_remaining = avg_r * remaining_count_r - -print(f"\nOverall: Puzzletron full pipeline (steps 1–{total_steps})") -print(DIV) -print(f" {'Status':<10} {'Step':<4} {'Description':<34} {'Elapsed':>8}") -print(DIV) - -for i, (snum, (sdesc, sts)) in enumerate(step_ts_list): - next_ts = step_ts_list[i+1][1][1] if i+1 < len(step_ts_list) else (pipeline_complete_ts if pipeline_complete_ts else now) - elapsed = int((next_ts - sts).total_seconds()) if sts and next_ts else None - is_last = (snum == last_step_num) - is_done = not is_last or pipeline_complete_ts is not None - detail = "" - if is_last and not is_done: - detail = cur_detail - if snum == 7 and all_rates: - detail = f" ({len(rate_done)}/{len(all_rates)} rates done)" - label = f"{snum}/{total_steps}: {sdesc}{detail}" - status = "[DONE]" if is_done else "[RUNNING]" - print(f" {status:<10} {'':<4} {label:<34} {fmt(elapsed) if elapsed is not None else '—':>8}") - -for snum in range(last_step_num + 1, total_steps + 1): - print(f" {'[ ]':<10} {'':<4} {f'{snum}/{total_steps}: pending':<34} {'':>8}") - -print(DIV) -if all_rates and last_step_num >= 7: - print(f" MIP rates: {len(rate_done)}/{len(all_rates)} done", end="") - running_rate = next((r for r in all_rates if r in rate_start and r not in rate_done), None) - if running_rate: - print(f" (running: {running_rate})", end="") - print() -done_steps = len([s for s in seen_steps if s != last_step_num or pipeline_complete_ts]) -avg_step_s = None -step_durations = [] -for i, (snum, (sdesc, sts)) in enumerate(step_ts_list): - next_ts = step_ts_list[i+1][1][1] if i+1 < len(step_ts_list) else (pipeline_complete_ts if pipeline_complete_ts else None) - if next_ts and sts: - step_durations.append(int((next_ts - sts).total_seconds())) -if step_durations: - avg_step_s = sum(step_durations) / len(step_durations) - -# Parse config for sweep settings to estimate step 7 duration -CONFIG_PATH = 'examples/puzzletron/configs/llama-3_1-8B_pruneffn_memory/llama-3_1-8B_pruneffn_memory.yaml' -sweep_enabled = True -sweep_n_rates = 6 -try: - cfg_text = open(CONFIG_PATH).read() - _en_m = re.search(r'sweep:\s*\n\s+enabled:\s*(true|false)', cfg_text) - if _en_m: - sweep_enabled = _en_m.group(1) == 'true' - _rates_m = re.search(r'memory_compression_rates:\s*\[([^\]]+)\]', cfg_text) - if _rates_m: - sweep_n_rates = len(_rates_m.group(1).split(',')) -except Exception: - pass -# Prefer actual rate count from log if step 7 has already started -effective_n_rates = len(all_rates) if all_rates else sweep_n_rates -RATE_S = 250 # ~4m 10s per compression rate (historical) - -def step_est(snum): - if snum == 7: - return (RATE_S * effective_n_rates) if sweep_enabled else 120 - elif snum == 8: - return 60 - return avg_step_s or 0 - -if pipeline_complete_ts: - est_rem = "done" -elif step_remaining is not None: - future_s = sum(step_est(s) for s in range(last_step_num + 1, total_steps + 1)) - est_rem = fmt(step_remaining + future_s) -else: - cur_s = step_est(last_step_num) - future_s = cur_s + sum(step_est(s) for s in range(last_step_num + 1, total_steps + 1)) - est_rem = fmt(future_s) if (cur_s or future_s) else "calculating..." - -print(f" Started: {pipeline_start.strftime('%H:%M:%S') if pipeline_start else '—'}") -print(f" Finished: {pipeline_complete_ts.strftime('%H:%M:%S') if pipeline_complete_ts else now.strftime('%H:%M:%S') + ' (in progress)'}") -print(f" Elapsed: {fmt(total_elapsed)}") -print(f" Completed: {done_steps}/{total_steps} steps") -print(f" Remaining: {est_rem} estimated") -results_match = re.search(r'Results written to: (\S+)', text) -if results_match: - print(f"\n Results: {results_match.group(1)}") -PYEOF +python3 .agents/skills/puzzletron/all_progress.py ``` ## Command: mip @@ -283,115 +85,8 @@ Stream output to the user as it arrives. When the command finishes, report the e ### mip progress -Run the following Python script verbatim. Do not modify it. Present the output to the user wrapped in a fenced code block (``` ... ```). +Run the following Bash command. Present the output to the user wrapped in a fenced code block (``` ... ```). ```bash -python3 - << 'PYEOF' -import re, sys -from datetime import datetime - -LOG = './log.txt' -try: - lines = open(LOG).readlines() - text = ''.join(lines) -except FileNotFoundError: - print("No log.txt found. Run /puzzletron mip first.") - sys.exit(0) - -def norm(r): return str(float(r)) -def fmt(s): return f"{int(s)//60}m {int(s)%60}s" if s is not None else "—" -def get_ts(line): - m = re.search(r'\[(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})', line) - return datetime.strptime(m.group(1), '%Y-%m-%d %H:%M:%S') if m else None - -rates_match = re.search(r'Compression rates: \[(.*?)\]', text) -all_rates = [norm(r.strip()) for r in rates_match.group(1).split(',')] if rates_match else [] - -# Collect start timestamp per rate -rate_start = {} -for line in lines: - if 'sweep.py:258' in line: - m = re.search(r'compression_rate=([\d.]+)', line) - if m: - r = norm(m.group(1)) - if r in all_rates and r not in rate_start: - rate_start[r] = get_ts(line) - -now = datetime.now().replace(microsecond=0) -sweep_start = rate_start.get(all_rates[0]) if all_rates else None - -# Rate is done when the next rate has started; last rate done when sweep.py:287 appears -rate_done = set() -for i, r in enumerate(all_rates[:-1]): - if all_rates[i + 1] in rate_start: - rate_done.add(r) -last = all_rates[-1] -sweep_complete_ts = None -for line in lines: - ts = get_ts(line) - if ts and 'sweep.py:292' in line: - sweep_complete_ts = ts - break -if sweep_complete_ts and last in rate_start: - rate_done.add(last) - -# Per-rate elapsed = next rate start - this rate start (or completion ts or now for last) -rate_elapsed = {} -for i, r in enumerate(all_rates): - if r not in rate_start: - continue - if i + 1 < len(all_rates): - end = rate_start[all_rates[i + 1]] - else: - end = sweep_complete_ts if sweep_complete_ts else now - rate_elapsed[r] = int((end - rate_start[r]).total_seconds()) - -# Currently running rate -running_rate = next((r for r in all_rates if r in rate_start and r not in rate_done), None) - -# Sub-step detail for running rate -cur_detail = "" -if running_rate: - batch_matches = re.findall(r'calculate_losses_pipeline[^:]*:\s*(\d+)%.*?(\d+)/(\d+)', text) - cbc_matches = re.findall(r'After (\d+) nodes.*?\(([\d.]+) seconds\)', text) - if batch_matches: - pct, cur, total = batch_matches[-1] - cur_detail = f" — validating ({cur}/{total} batches)" - elif cbc_matches: - nodes, secs = cbc_matches[-1] - cur_detail = f" — MIP solver ({int(nodes):,} nodes, {float(secs):.1f}s)" - -end_ts = sweep_complete_ts if sweep_complete_ts else now -total_elapsed = int((end_ts - sweep_start).total_seconds()) if sweep_start else 0 - -done_count = len(rate_done) -remaining_count = len(all_rates) - done_count -avg_s = sum(rate_elapsed[r] for r in rate_done) / done_count if done_count else None -est_rem = fmt(avg_s * remaining_count) if avg_s and remaining_count else ("done" if not remaining_count else "calculating...") - -DIV = '─' * 62 - -print(f"\nOverall: Puzzletron step 7/8 — MIP sweep ({len(all_rates)} compression rates)") -print(DIV) -print(f" {'Status':<10} {'Phase':<32} {'Elapsed':>8}") -print(DIV) -print(f" [DONE] {'Prep (teacher memory + rate list)':<32} {'<1s':>8}") -for r in all_rates: - if r not in rate_start: - print(f" [ ] {f'compression_rate={r}':<32} {'pending':>8}") - elif r == running_rate: - detail = cur_detail - print(f" [RUNNING] {f'compression_rate={r}{detail}':<32} {fmt(rate_elapsed.get(r)):>8}") - else: - print(f" [DONE] {f'compression_rate={r}':<32} {fmt(rate_elapsed.get(r)):>8}") -print(DIV) -print(f" Started: {sweep_start.strftime('%H:%M:%S') if sweep_start else '—'}") -print(f" Finished: {sweep_complete_ts.strftime('%H:%M:%S') if sweep_complete_ts else now.strftime('%H:%M:%S') + ' (in progress)'}") -print(f" Elapsed: {fmt(total_elapsed)}") -print(f" Completed: {done_count}/{len(all_rates)} compression rates") -print(f" Remaining: {est_rem} estimated") -results_match = re.search(r'Results written to: (\S+)', text) -if results_match: - print(f"\n Results: {results_match.group(1)}") -PYEOF +python3 .agents/skills/puzzletron/mip_progress.py ``` diff --git a/.agents/skills/puzzletron/all_progress.py b/.agents/skills/puzzletron/all_progress.py new file mode 100644 index 00000000000..9224439df4b --- /dev/null +++ b/.agents/skills/puzzletron/all_progress.py @@ -0,0 +1,237 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Generated with Claude Code +"""Progress report for the full Puzzletron pipeline (all 8 steps).""" + +import glob +import re +import sys +from datetime import datetime + +LOG = "./log.txt" +try: + lines = open(LOG).readlines() + text = "".join(lines) +except FileNotFoundError: + print("No log.txt found. Run /puzzletron all first.") + sys.exit(0) + + +def norm(r): + """Normalize a compression rate to a canonical float string.""" + return str(float(r)) + + +def fmt(s): + """Format seconds as 'Xm Ys', or '—' if None.""" + return f"{int(s) // 60}m {int(s) % 60}s" if s is not None else "—" + + +def get_ts(line): + """Extract a datetime from a log line timestamp, or None.""" + m = re.search(r"\[(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})", line) + return datetime.strptime(m.group(1), "%Y-%m-%d %H:%M:%S") if m else None + + +now = datetime.now().replace(microsecond=0) +DIV = "─" * 68 + +step_events = [] +for line in lines: + m = re.search(r"Puzzletron Progress (\d+)/(\d+): (.+)", line) + if m: + step_num = int(m.group(1)) + total_steps = int(m.group(2)) + desc = m.group(3).strip() + ts = get_ts(line) + step_events.append((step_num, total_steps, desc, ts)) + +total_steps = step_events[-1][1] if step_events else 8 +seen_steps = {e[0]: (e[2], e[3]) for e in step_events} +last_step_num = max(seen_steps.keys()) if seen_steps else 0 + +pipeline_complete_ts = None +for line in lines: + ts = get_ts(line) + if ts and "sweep.py:292" in line: + pipeline_complete_ts = ts + break + +cur_detail = "" +step_remaining = None +batch_matches = re.findall(r"calculate_losses_pipeline[^:]*:\s*(\d+)%.*?(\d+)/(\d+)", text) +cbc_matches = re.findall(r"After (\d+) nodes.*?\(([\d.]+) seconds\)", text) + +sol_dir_match = re.search( + r"'output_dir': '([^']+single_sequence_replacement_solutions--validation[^']*)'", text +) +sol_done, sol_total = None, None +if sol_dir_match: + sol_dir = sol_dir_match.group(1) + sol_done = len(glob.glob(f"{sol_dir}/solution*.json")) + sol_list_match = re.search(r"'solutions_to_validate': \[([\d, ]+)\]", text) + if sol_list_match: + sol_total = len(sol_list_match.group(1).split(",")) +if sol_done is not None and sol_total: + cur_detail = f" ({sol_done}/{sol_total} solutions)" +elif batch_matches: + pct, cur_b, total_b = batch_matches[-1] + cur_detail = f" ({cur_b}/{total_b} batches)" +elif cbc_matches: + nodes, secs = cbc_matches[-1] + cur_detail = f" (MIP solver: {int(nodes):,} nodes, {float(secs):.1f}s)" + +rates_match = re.search(r"Compression rates: \[(.*?)\]", text) +all_rates = [norm(r.strip()) for r in rates_match.group(1).split(",")] if rates_match else [] +rate_start = {} +for line in lines: + if "sweep.py:258" in line: + m = re.search(r"compression_rate=([\d.]+)", line) + if m: + r = norm(m.group(1)) + if r in all_rates and r not in rate_start: + rate_start[r] = get_ts(line) +rate_done = set() +for i, r in enumerate(all_rates[:-1]): + if all_rates[i + 1] in rate_start: + rate_done.add(r) +if pipeline_complete_ts and all_rates and all_rates[-1] in rate_start: + rate_done.add(all_rates[-1]) + +pipeline_start = step_events[0][3] if step_events else None +end_ts = pipeline_complete_ts or now +total_elapsed = int((end_ts - pipeline_start).total_seconds()) if pipeline_start else 0 + +step_ts_list = sorted(seen_steps.items()) +cur_step_start_ts = seen_steps[last_step_num][1] if last_step_num in seen_steps else None +if not pipeline_complete_ts and cur_step_start_ts: + cur_step_elapsed = int((now - cur_step_start_ts).total_seconds()) + if sol_done and sol_total and sol_done > 0: + rate_per_sol = cur_step_elapsed / sol_done + step_remaining = rate_per_sol * (sol_total - sol_done) + elif batch_matches and int(cur_b) > 0 and int(cur_b) < int(total_b): + rate_per_batch = cur_step_elapsed / int(cur_b) + step_remaining = rate_per_batch * (int(total_b) - int(cur_b)) + elif all_rates and last_step_num == 7: + done_count_r = len(rate_done) + remaining_count_r = len(all_rates) - done_count_r + rate_elapsed = {} + for i, r in enumerate(all_rates): + if r not in rate_start: + continue + if i + 1 < len(all_rates) and all_rates[i + 1] in rate_start: + rate_elapsed[r] = int( + (rate_start[all_rates[i + 1]] - rate_start[r]).total_seconds() + ) + avg_r = sum(rate_elapsed.values()) / len(rate_elapsed) if rate_elapsed else None + if avg_r and remaining_count_r: + step_remaining = avg_r * remaining_count_r + +print(f"\nOverall: Puzzletron full pipeline (steps 1–{total_steps})") # noqa: RUF001 +print(DIV) +print(f" {'Status':<10} {'Step':<4} {'Description':<34} {'Elapsed':>8}") +print(DIV) + +for i, (snum, (sdesc, sts)) in enumerate(step_ts_list): + next_ts = ( + step_ts_list[i + 1][1][1] if i + 1 < len(step_ts_list) else (pipeline_complete_ts or now) + ) + elapsed = int((next_ts - sts).total_seconds()) if sts and next_ts else None + is_last = snum == last_step_num + is_done = not is_last or pipeline_complete_ts is not None + detail = "" + if is_last and not is_done: + detail = cur_detail + if snum == 7 and all_rates: + detail = f" ({len(rate_done)}/{len(all_rates)} rates done)" + label = f"{snum}/{total_steps}: {sdesc}{detail}" + status = "[DONE]" if is_done else "[RUNNING]" + print( + f" {status:<10} {'':<4} {label:<34} {fmt(elapsed) if elapsed is not None else '—':>8}" + ) + +for snum in range(last_step_num + 1, total_steps + 1): + print(f" {'[ ]':<10} {'':<4} {f'{snum}/{total_steps}: pending':<34} {'':>8}") + +print(DIV) +if all_rates and last_step_num >= 7: + print(f" MIP rates: {len(rate_done)}/{len(all_rates)} done", end="") + running_rate = next((r for r in all_rates if r in rate_start and r not in rate_done), None) + if running_rate: + print(f" (running: {running_rate})", end="") + print() + +done_steps = len([s for s in seen_steps if s != last_step_num or pipeline_complete_ts]) +step_durations = [] +for i, (snum, (sdesc, sts)) in enumerate(step_ts_list): + next_ts = ( + step_ts_list[i + 1][1][1] if i + 1 < len(step_ts_list) else (pipeline_complete_ts or None) + ) + if next_ts and sts: + step_durations.append(int((next_ts - sts).total_seconds())) +avg_step_s = sum(step_durations) / len(step_durations) if step_durations else None + +CONFIG_PATH = ( + "examples/puzzletron/configs/llama-3_1-8B_pruneffn_memory/llama-3_1-8B_pruneffn_memory.yaml" +) +sweep_enabled = True +sweep_n_rates = 6 +try: + cfg_text = open(CONFIG_PATH).read() + _en_m = re.search(r"sweep:\s*\n\s+enabled:\s*(true|false)", cfg_text) + if _en_m: + sweep_enabled = _en_m.group(1) == "true" + _rates_m = re.search(r"memory_compression_rates:\s*\[([^\]]+)\]", cfg_text) + if _rates_m: + sweep_n_rates = len(_rates_m.group(1).split(",")) +except Exception: + pass +effective_n_rates = len(all_rates) if all_rates else sweep_n_rates +RATE_S = 250 # ~4m 10s per compression rate (historical) + + +def step_est(snum): + """Estimate duration in seconds for a pending pipeline step.""" + if snum == 7: + return (RATE_S * effective_n_rates) if sweep_enabled else 120 + elif snum == 8: + return 60 + return avg_step_s or 0 + + +if pipeline_complete_ts: + est_rem = "done" +elif step_remaining is not None: + future_s = sum(step_est(s) for s in range(last_step_num + 1, total_steps + 1)) + est_rem = fmt(step_remaining + future_s) +else: + cur_s = step_est(last_step_num) + future_s = cur_s + sum(step_est(s) for s in range(last_step_num + 1, total_steps + 1)) + est_rem = fmt(future_s) if (cur_s or future_s) else "calculating..." + +finished_str = ( + pipeline_complete_ts.strftime("%H:%M:%S") + if pipeline_complete_ts + else now.strftime("%H:%M:%S") + " (in progress)" +) +print(f" Started: {pipeline_start.strftime('%H:%M:%S') if pipeline_start else '—'}") +print(f" Finished: {finished_str}") +print(f" Elapsed: {fmt(total_elapsed)}") +print(f" Completed: {done_steps}/{total_steps} steps") +print(f" Remaining: {est_rem} estimated") +results_match = re.search(r"Results written to: (\S+)", text) +if results_match: + print(f"\n Results: {results_match.group(1)}") diff --git a/.agents/skills/puzzletron/mip_progress.py b/.agents/skills/puzzletron/mip_progress.py new file mode 100644 index 00000000000..90e1dbce814 --- /dev/null +++ b/.agents/skills/puzzletron/mip_progress.py @@ -0,0 +1,141 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Generated with Claude Code +"""Progress report for the Puzzletron MIP step.""" + +import re +import sys +from datetime import datetime + +LOG = "./log.txt" +try: + lines = open(LOG).readlines() + text = "".join(lines) +except FileNotFoundError: + print("No log.txt found. Run /puzzletron mip first.") + sys.exit(0) + + +def norm(r): + """Normalize a compression rate to a canonical float string.""" + return str(float(r)) + + +def fmt(s): + """Format seconds as 'Xm Ys', or '—' if None.""" + return f"{int(s) // 60}m {int(s) % 60}s" if s is not None else "—" + + +def get_ts(line): + """Extract a datetime from a log line timestamp, or None.""" + m = re.search(r"\[(\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})", line) + return datetime.strptime(m.group(1), "%Y-%m-%d %H:%M:%S") if m else None + + +rates_match = re.search(r"Compression rates: \[(.*?)\]", text) +all_rates = [norm(r.strip()) for r in rates_match.group(1).split(",")] if rates_match else [] + +rate_start = {} +for line in lines: + if "sweep.py:258" in line: + m = re.search(r"compression_rate=([\d.]+)", line) + if m: + r = norm(m.group(1)) + if r in all_rates and r not in rate_start: + rate_start[r] = get_ts(line) + +now = datetime.now().replace(microsecond=0) +sweep_start = rate_start.get(all_rates[0]) if all_rates else None + +rate_done = set() +for i, r in enumerate(all_rates[:-1]): + if all_rates[i + 1] in rate_start: + rate_done.add(r) +last = all_rates[-1] if all_rates else None +sweep_complete_ts = None +for line in lines: + ts = get_ts(line) + if ts and "sweep.py:292" in line: + sweep_complete_ts = ts + break +if sweep_complete_ts and last and last in rate_start: + rate_done.add(last) + +rate_elapsed = {} +for i, r in enumerate(all_rates): + if r not in rate_start: + continue + if i + 1 < len(all_rates): + end = rate_start[all_rates[i + 1]] + else: + end = sweep_complete_ts or now + rate_elapsed[r] = int((end - rate_start[r]).total_seconds()) + +running_rate = next((r for r in all_rates if r in rate_start and r not in rate_done), None) + +cur_detail = "" +if running_rate: + batch_matches = re.findall(r"calculate_losses_pipeline[^:]*:\s*(\d+)%.*?(\d+)/(\d+)", text) + cbc_matches = re.findall(r"After (\d+) nodes.*?\(([\d.]+) seconds\)", text) + if batch_matches: + pct, cur, total = batch_matches[-1] + cur_detail = f" — validating ({cur}/{total} batches)" + elif cbc_matches: + nodes, secs = cbc_matches[-1] + cur_detail = f" — MIP solver ({int(nodes):,} nodes, {float(secs):.1f}s)" + +end_ts = sweep_complete_ts or now +total_elapsed = int((end_ts - sweep_start).total_seconds()) if sweep_start else 0 + +done_count = len(rate_done) +remaining_count = len(all_rates) - done_count +avg_s = sum(rate_elapsed[r] for r in rate_done) / done_count if done_count else None +est_rem = ( + fmt(avg_s * remaining_count) + if avg_s and remaining_count + else ("done" if not remaining_count else "calculating...") +) + +DIV = "─" * 62 + +print(f"\nOverall: Puzzletron step 7/8 — MIP sweep ({len(all_rates)} compression rates)") +print(DIV) +print(f" {'Status':<10} {'Phase':<32} {'Elapsed':>8}") +print(DIV) +print(f" [DONE] {'Prep (teacher memory + rate list)':<32} {'<1s':>8}") +for r in all_rates: + if r not in rate_start: + print(f" [ ] {f'compression_rate={r}':<32} {'pending':>8}") + elif r == running_rate: + print( + f" [RUNNING] {f'compression_rate={r}{cur_detail}':<32} {fmt(rate_elapsed.get(r)):>8}" + ) + else: + print(f" [DONE] {f'compression_rate={r}':<32} {fmt(rate_elapsed.get(r)):>8}") +print(DIV) +finished_str = ( + sweep_complete_ts.strftime("%H:%M:%S") + if sweep_complete_ts + else now.strftime("%H:%M:%S") + " (in progress)" +) +print(f" Started: {sweep_start.strftime('%H:%M:%S') if sweep_start else '—'}") +print(f" Finished: {finished_str}") +print(f" Elapsed: {fmt(total_elapsed)}") +print(f" Completed: {done_count}/{len(all_rates)} compression rates") +print(f" Remaining: {est_rem} estimated") +results_match = re.search(r"Results written to: (\S+)", text) +if results_match: + print(f"\n Results: {results_match.group(1)}") From 02ec52eeb5dd7a403323e1177f405d84ab3d5ef9 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Thu, 18 Jun 2026 02:00:44 -0700 Subject: [PATCH 10/17] update progress bar Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.agents/skills/puzzletron/README.md b/.agents/skills/puzzletron/README.md index e4baa46fcb3..9ee8ed9c6c0 100644 --- a/.agents/skills/puzzletron/README.md +++ b/.agents/skills/puzzletron/README.md @@ -78,15 +78,15 @@ Overall: Puzzletron full pipeline (steps 1–8) [DONE] 3/8: scoring pruning activations (multi-gpu) 9m 9s [DONE] 4/8: pruning the model and saving pruned checkpoints (single-gpu) 0m 57s [DONE] 5/8: building replacement library and subblock statistics (single-gpu) 0m 26s - [RUNNING] 6/8: calculating one block scores (multi-gpu) (76/352 solutions) 27m 53s + [RUNNING] 6/8: calculating one block scores (multi-gpu) (270/352 solutions) 100m 6s [ ] 7/8: pending [ ] 8/8: pending ──────────────────────────────────────────────────────────────────── Started: 00:08:50 - Finished: 00:47:41 (in progress) - Elapsed: 38m 51s + Finished: 01:59:54 (in progress) + Elapsed: 111m 4s Completed: 5/8 steps - Remaining: 105m 38s estimated + Remaining: 56m 24s estimated ``` Step 6 progress is tracked via completed `solution_N.json` files on disk for an accurate From dd6fbdb60f72c7b19969b26194357dd1624500fd Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Thu, 18 Jun 2026 04:01:44 -0700 Subject: [PATCH 11/17] code clean up Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/all_progress.py | 75 ++-------------------- .agents/skills/puzzletron/mip_progress.py | 76 +++++++++++++++++------ 2 files changed, 64 insertions(+), 87 deletions(-) diff --git a/.agents/skills/puzzletron/all_progress.py b/.agents/skills/puzzletron/all_progress.py index 9224439df4b..f80cfb31054 100644 --- a/.agents/skills/puzzletron/all_progress.py +++ b/.agents/skills/puzzletron/all_progress.py @@ -30,11 +30,6 @@ sys.exit(0) -def norm(r): - """Normalize a compression rate to a canonical float string.""" - return str(float(r)) - - def fmt(s): """Format seconds as 'Xm Ys', or '—' if None.""" return f"{int(s) // 60}m {int(s) % 60}s" if s is not None else "—" @@ -64,11 +59,8 @@ def get_ts(line): last_step_num = max(seen_steps.keys()) if seen_steps else 0 pipeline_complete_ts = None -for line in lines: - ts = get_ts(line) - if ts and "sweep.py:292" in line: - pipeline_complete_ts = ts - break +if last_step_num == total_steps and total_steps in seen_steps: + pipeline_complete_ts = seen_steps[total_steps][1] cur_detail = "" step_remaining = None @@ -94,23 +86,6 @@ def get_ts(line): nodes, secs = cbc_matches[-1] cur_detail = f" (MIP solver: {int(nodes):,} nodes, {float(secs):.1f}s)" -rates_match = re.search(r"Compression rates: \[(.*?)\]", text) -all_rates = [norm(r.strip()) for r in rates_match.group(1).split(",")] if rates_match else [] -rate_start = {} -for line in lines: - if "sweep.py:258" in line: - m = re.search(r"compression_rate=([\d.]+)", line) - if m: - r = norm(m.group(1)) - if r in all_rates and r not in rate_start: - rate_start[r] = get_ts(line) -rate_done = set() -for i, r in enumerate(all_rates[:-1]): - if all_rates[i + 1] in rate_start: - rate_done.add(r) -if pipeline_complete_ts and all_rates and all_rates[-1] in rate_start: - rate_done.add(all_rates[-1]) - pipeline_start = step_events[0][3] if step_events else None end_ts = pipeline_complete_ts or now total_elapsed = int((end_ts - pipeline_start).total_seconds()) if pipeline_start else 0 @@ -125,20 +100,6 @@ def get_ts(line): elif batch_matches and int(cur_b) > 0 and int(cur_b) < int(total_b): rate_per_batch = cur_step_elapsed / int(cur_b) step_remaining = rate_per_batch * (int(total_b) - int(cur_b)) - elif all_rates and last_step_num == 7: - done_count_r = len(rate_done) - remaining_count_r = len(all_rates) - done_count_r - rate_elapsed = {} - for i, r in enumerate(all_rates): - if r not in rate_start: - continue - if i + 1 < len(all_rates) and all_rates[i + 1] in rate_start: - rate_elapsed[r] = int( - (rate_start[all_rates[i + 1]] - rate_start[r]).total_seconds() - ) - avg_r = sum(rate_elapsed.values()) / len(rate_elapsed) if rate_elapsed else None - if avg_r and remaining_count_r: - step_remaining = avg_r * remaining_count_r print(f"\nOverall: Puzzletron full pipeline (steps 1–{total_steps})") # noqa: RUF001 print(DIV) @@ -155,8 +116,6 @@ def get_ts(line): detail = "" if is_last and not is_done: detail = cur_detail - if snum == 7 and all_rates: - detail = f" ({len(rate_done)}/{len(all_rates)} rates done)" label = f"{snum}/{total_steps}: {sdesc}{detail}" status = "[DONE]" if is_done else "[RUNNING]" print( @@ -167,13 +126,6 @@ def get_ts(line): print(f" {'[ ]':<10} {'':<4} {f'{snum}/{total_steps}: pending':<34} {'':>8}") print(DIV) -if all_rates and last_step_num >= 7: - print(f" MIP rates: {len(rate_done)}/{len(all_rates)} done", end="") - running_rate = next((r for r in all_rates if r in rate_start and r not in rate_done), None) - if running_rate: - print(f" (running: {running_rate})", end="") - print() - done_steps = len([s for s in seen_steps if s != last_step_num or pipeline_complete_ts]) step_durations = [] for i, (snum, (sdesc, sts)) in enumerate(step_ts_list): @@ -184,29 +136,12 @@ def get_ts(line): step_durations.append(int((next_ts - sts).total_seconds())) avg_step_s = sum(step_durations) / len(step_durations) if step_durations else None -CONFIG_PATH = ( - "examples/puzzletron/configs/llama-3_1-8B_pruneffn_memory/llama-3_1-8B_pruneffn_memory.yaml" -) -sweep_enabled = True -sweep_n_rates = 6 -try: - cfg_text = open(CONFIG_PATH).read() - _en_m = re.search(r"sweep:\s*\n\s+enabled:\s*(true|false)", cfg_text) - if _en_m: - sweep_enabled = _en_m.group(1) == "true" - _rates_m = re.search(r"memory_compression_rates:\s*\[([^\]]+)\]", cfg_text) - if _rates_m: - sweep_n_rates = len(_rates_m.group(1).split(",")) -except Exception: - pass -effective_n_rates = len(all_rates) if all_rates else sweep_n_rates -RATE_S = 250 # ~4m 10s per compression rate (historical) - def step_est(snum): """Estimate duration in seconds for a pending pipeline step.""" if snum == 7: - return (RATE_S * effective_n_rates) if sweep_enabled else 120 + # Step 7 in the full pipeline is a single MIP solve (~5m), not a sweep + return 296 elif snum == 8: return 60 return avg_step_s or 0 @@ -233,5 +168,7 @@ def step_est(snum): print(f" Completed: {done_steps}/{total_steps} steps") print(f" Remaining: {est_rem} estimated") results_match = re.search(r"Results written to: (\S+)", text) +if not results_match: + results_match = re.search(r"\[run_puzzle\.py:335\]\s+(\S+)", text) if results_match: print(f"\n Results: {results_match.group(1)}") diff --git a/.agents/skills/puzzletron/mip_progress.py b/.agents/skills/puzzletron/mip_progress.py index 90e1dbce814..a52f1120b13 100644 --- a/.agents/skills/puzzletron/mip_progress.py +++ b/.agents/skills/puzzletron/mip_progress.py @@ -45,9 +45,63 @@ def get_ts(line): return datetime.strptime(m.group(1), "%Y-%m-%d %H:%M:%S") if m else None +now = datetime.now().replace(microsecond=0) + rates_match = re.search(r"Compression rates: \[(.*?)\]", text) all_rates = [norm(r.strip()) for r in rates_match.group(1).split(",")] if rates_match else [] +# Detect completion via step 8 marker or sweep.py:292 +complete_ts = None +for line in lines: + ts = get_ts(line) + if ts and ("sweep.py:292" in line or "Puzzletron Progress 8/8" in line): + complete_ts = ts + break + +cbc_matches = re.findall(r"After (\d+) nodes.*?\(([\d.]+) seconds\)", text) + +# ── Sweep disabled: single MIP solve ───────────────────────────────────────── +if not all_rates: + step7_ts = None + for line in lines: + ts = get_ts(line) + if ts and "Puzzletron Progress 7/8" in line: + step7_ts = ts + break + + end_ts = complete_ts or now + total_elapsed = int((end_ts - step7_ts).total_seconds()) if step7_ts else 0 + + cbc_detail = "" + if cbc_matches: + nodes, secs = cbc_matches[-1] + cbc_detail = f" ({int(nodes):,} nodes, {float(secs):.1f}s)" + + DIV = "─" * 62 + print("\nOverall: Puzzletron step 7/8 — MIP solve (sweep disabled)") + print(DIV) + print(f" {'Status':<10} {'Phase':<32} {'Elapsed':>8}") + print(DIV) + print(f" {'[DONE]':<10} {'Prep (loading model + scores)':<32} {'<1s':>8}") + status = "[DONE]" if complete_ts else "[RUNNING]" + label = f"MIP solve{cbc_detail}" + print(f" {status:<10} {label:<32} {fmt(total_elapsed):>8}") + print(DIV) + finished_str = ( + complete_ts.strftime("%H:%M:%S") + if complete_ts + else now.strftime("%H:%M:%S") + " (in progress)" + ) + print(f" Started: {step7_ts.strftime('%H:%M:%S') if step7_ts else '—'}") + print(f" Finished: {finished_str}") + print(f" Elapsed: {fmt(total_elapsed)}") + print(f" Remaining: {'done' if complete_ts else 'calculating...'}") + results_match = re.search(r"Results written to: (\S+)", text) + if results_match: + print(f"\n Results: {results_match.group(1)}") + sys.exit(0) + +# ── Sweep enabled: per-rate progress ───────────────────────────────────────── rate_start = {} for line in lines: if "sweep.py:258" in line: @@ -57,7 +111,6 @@ def get_ts(line): if r in all_rates and r not in rate_start: rate_start[r] = get_ts(line) -now = datetime.now().replace(microsecond=0) sweep_start = rate_start.get(all_rates[0]) if all_rates else None rate_done = set() @@ -65,23 +118,14 @@ def get_ts(line): if all_rates[i + 1] in rate_start: rate_done.add(r) last = all_rates[-1] if all_rates else None -sweep_complete_ts = None -for line in lines: - ts = get_ts(line) - if ts and "sweep.py:292" in line: - sweep_complete_ts = ts - break -if sweep_complete_ts and last and last in rate_start: +if complete_ts and last and last in rate_start: rate_done.add(last) rate_elapsed = {} for i, r in enumerate(all_rates): if r not in rate_start: continue - if i + 1 < len(all_rates): - end = rate_start[all_rates[i + 1]] - else: - end = sweep_complete_ts or now + end = rate_start[all_rates[i + 1]] if i + 1 < len(all_rates) else (complete_ts or now) rate_elapsed[r] = int((end - rate_start[r]).total_seconds()) running_rate = next((r for r in all_rates if r in rate_start and r not in rate_done), None) @@ -89,7 +133,6 @@ def get_ts(line): cur_detail = "" if running_rate: batch_matches = re.findall(r"calculate_losses_pipeline[^:]*:\s*(\d+)%.*?(\d+)/(\d+)", text) - cbc_matches = re.findall(r"After (\d+) nodes.*?\(([\d.]+) seconds\)", text) if batch_matches: pct, cur, total = batch_matches[-1] cur_detail = f" — validating ({cur}/{total} batches)" @@ -97,7 +140,7 @@ def get_ts(line): nodes, secs = cbc_matches[-1] cur_detail = f" — MIP solver ({int(nodes):,} nodes, {float(secs):.1f}s)" -end_ts = sweep_complete_ts or now +end_ts = complete_ts or now total_elapsed = int((end_ts - sweep_start).total_seconds()) if sweep_start else 0 done_count = len(rate_done) @@ -110,7 +153,6 @@ def get_ts(line): ) DIV = "─" * 62 - print(f"\nOverall: Puzzletron step 7/8 — MIP sweep ({len(all_rates)} compression rates)") print(DIV) print(f" {'Status':<10} {'Phase':<32} {'Elapsed':>8}") @@ -127,9 +169,7 @@ def get_ts(line): print(f" [DONE] {f'compression_rate={r}':<32} {fmt(rate_elapsed.get(r)):>8}") print(DIV) finished_str = ( - sweep_complete_ts.strftime("%H:%M:%S") - if sweep_complete_ts - else now.strftime("%H:%M:%S") + " (in progress)" + complete_ts.strftime("%H:%M:%S") if complete_ts else now.strftime("%H:%M:%S") + " (in progress)" ) print(f" Started: {sweep_start.strftime('%H:%M:%S') if sweep_start else '—'}") print(f" Finished: {finished_str}") From c9f1fe1dccf9a4e69156ab3591f74464bccf2d96 Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Thu, 18 Jun 2026 04:05:26 -0700 Subject: [PATCH 12/17] code clean up Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/mip_progress.py | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/.agents/skills/puzzletron/mip_progress.py b/.agents/skills/puzzletron/mip_progress.py index a52f1120b13..d5d67eab039 100644 --- a/.agents/skills/puzzletron/mip_progress.py +++ b/.agents/skills/puzzletron/mip_progress.py @@ -97,6 +97,8 @@ def get_ts(line): print(f" Elapsed: {fmt(total_elapsed)}") print(f" Remaining: {'done' if complete_ts else 'calculating...'}") results_match = re.search(r"Results written to: (\S+)", text) + if not results_match: + results_match = re.search(r"\[run_puzzle\.py:335\]\s+(\S+)", text) if results_match: print(f"\n Results: {results_match.group(1)}") sys.exit(0) @@ -157,16 +159,16 @@ def get_ts(line): print(DIV) print(f" {'Status':<10} {'Phase':<32} {'Elapsed':>8}") print(DIV) -print(f" [DONE] {'Prep (teacher memory + rate list)':<32} {'<1s':>8}") +print(f" {'[DONE]':<10} {'Prep (teacher memory + rate list)':<32} {'<1s':>8}") for r in all_rates: if r not in rate_start: - print(f" [ ] {f'compression_rate={r}':<32} {'pending':>8}") + print(f" {'[ ]':<10} {f'compression_rate={r}':<32} {'pending':>8}") elif r == running_rate: print( - f" [RUNNING] {f'compression_rate={r}{cur_detail}':<32} {fmt(rate_elapsed.get(r)):>8}" + f" {'[RUNNING]':<10} {f'compression_rate={r}{cur_detail}':<32} {fmt(rate_elapsed.get(r)):>8}" ) else: - print(f" [DONE] {f'compression_rate={r}':<32} {fmt(rate_elapsed.get(r)):>8}") + print(f" {'[DONE]':<10} {f'compression_rate={r}':<32} {fmt(rate_elapsed.get(r)):>8}") print(DIV) finished_str = ( complete_ts.strftime("%H:%M:%S") if complete_ts else now.strftime("%H:%M:%S") + " (in progress)" From afb6a712f3f9efd006eb5df1867feba8c36198bf Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Thu, 18 Jun 2026 04:17:07 -0700 Subject: [PATCH 13/17] add changelog for puzzletron clade skill Signed-off-by: Daniel Korzekwa --- CHANGELOG.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/CHANGELOG.rst b/CHANGELOG.rst index d3d0ec160ec..852a352d891 100755 --- a/CHANGELOG.rst +++ b/CHANGELOG.rst @@ -6,6 +6,7 @@ Changelog **New Features** +- Add **experimental** ``/puzzletron`` Claude Code agent skill (``.agents/skills/puzzletron/``) with ``mip`` and ``all`` commands for running the MIP step or full pipeline, and ``mip progress`` / ``all progress`` sub-commands reporting per-step status, elapsed time, and estimated time remaining. See `.agents/skills/puzzletron/README.md `_. - Add the ``day0-release`` agent skill (``.agents/skills/day0-release/``), a deterministic end-to-end driver that chains the PTQ → evaluation → comparison skills (the evaluation stage deploys the checkpoint itself) with an enforced gate after each stage and returns a publish decision (ACCEPT / REGRESSION / ANOMALOUS / INFEASIBLE). Ships three GPU-free, unit-tested gate scripts (``gate_ptq.py``, ``gate_run.py``, ``gate_compare.py``) that validate checkpoint coverage, evaluation-run completeness, and baseline-vs-candidate accuracy threshold. v1 reports and stops on regression; the recipe-search loop is deferred. - Add **streaming** speculative-decoding training (EAGLE3 / DFlash): the draft trains on base-model hidden states produced on the fly by a co-located ``vllm serve`` (no disk dump), moved trainer-side over NIXL RDMA, scaling to multi-node (dedicated serve replicas + DDP trainers). New launcher examples for NVFP4 Kimi-K2.5 / K2.6 on GB200/aarch64 under ``tools/launcher/examples/moonshotai/``. - Add a fused Triton fast path for ``local_hessian`` NVFP4 weight-scale search (the Hessian-weighted FP8-E4M3 scale sweep). For each NVFP4 block it minimizes ``dwᵀ H dw`` over the 126 candidate scales using the per-cin-block local Hessian on tensor cores, replacing the per-weight Python reference sweep — roughly **34x** faster on a single 8192x4096 weight and bit-exact with the reference for fp32/fp16 weights. Used automatically during ``local_hessian`` calibration for both dense and fused-MoE expert weights; falls back to the reference sweep on CPU, when Triton is unavailable, or via ``MODELOPT_NVFP4_TRITON_SWEEP=0``. From 37d7132dff009f234cf1e7e81fae68f6f8e05b0b Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Fri, 19 Jun 2026 00:21:38 -0700 Subject: [PATCH 14/17] NameError: unpack batch_matches before if/elif so cur_b and total_b are always defined Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/all_progress.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.agents/skills/puzzletron/all_progress.py b/.agents/skills/puzzletron/all_progress.py index f80cfb31054..3db4b033e6b 100644 --- a/.agents/skills/puzzletron/all_progress.py +++ b/.agents/skills/puzzletron/all_progress.py @@ -77,10 +77,10 @@ def get_ts(line): sol_list_match = re.search(r"'solutions_to_validate': \[([\d, ]+)\]", text) if sol_list_match: sol_total = len(sol_list_match.group(1).split(",")) +pct, cur_b, total_b = batch_matches[-1] if batch_matches else (None, None, None) if sol_done is not None and sol_total: cur_detail = f" ({sol_done}/{sol_total} solutions)" elif batch_matches: - pct, cur_b, total_b = batch_matches[-1] cur_detail = f" ({cur_b}/{total_b} batches)" elif cbc_matches: nodes, secs = cbc_matches[-1] @@ -97,7 +97,7 @@ def get_ts(line): if sol_done and sol_total and sol_done > 0: rate_per_sol = cur_step_elapsed / sol_done step_remaining = rate_per_sol * (sol_total - sol_done) - elif batch_matches and int(cur_b) > 0 and int(cur_b) < int(total_b): + elif cur_b is not None and total_b is not None and int(cur_b) > 0 and int(cur_b) < int(total_b): rate_per_batch = cur_step_elapsed / int(cur_b) step_remaining = rate_per_batch * (int(total_b) - int(cur_b)) From 417ae874cb442cd5f59a3a3bb73cb0b88d0a582e Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Fri, 19 Jun 2026 00:23:12 -0700 Subject: [PATCH 15/17] dd nproc_per_node integer validation to prevent shell injection in all and mip commands Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/SKILL.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.agents/skills/puzzletron/SKILL.md b/.agents/skills/puzzletron/SKILL.md index a3d51ab799b..2b5c5bb1e94 100644 --- a/.agents/skills/puzzletron/SKILL.md +++ b/.agents/skills/puzzletron/SKILL.md @@ -37,6 +37,7 @@ Parse `nproc_per_node` from args using either positional or flag syntax: - If the second word is exactly `progress`, execute the **all progress** sub-command below. - If no `nproc_per_node` value can be found, ask the user: "Please provide the number of GPUs per node (nproc_per_node)." and **STOP**. +- If the value does not match `^[0-9]+$`, ask the user: "nproc_per_node must be a positive integer." and **STOP**. - Otherwise use the parsed value and run the full pipeline. ### all \ @@ -68,6 +69,7 @@ Parse `nproc_per_node` from args using either positional or flag syntax: - If the second word is exactly `progress`, execute the **mip progress** sub-command below. - If no `nproc_per_node` value can be found, ask the user: "Please provide the number of GPUs per node (nproc_per_node)." and **STOP**. +- If the value does not match `^[0-9]+$`, ask the user: "nproc_per_node must be a positive integer." and **STOP**. - Otherwise use the parsed value and run the MIP step. ### mip \ From 708cf7b37341f4249fced362e35b9124f4de87bc Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Fri, 19 Jun 2026 00:25:22 -0700 Subject: [PATCH 16/17] replace hardcoded sweep.py line-number markers with content-based detection in mip_progress.py Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/mip_progress.py | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/.agents/skills/puzzletron/mip_progress.py b/.agents/skills/puzzletron/mip_progress.py index d5d67eab039..2065cc18ce3 100644 --- a/.agents/skills/puzzletron/mip_progress.py +++ b/.agents/skills/puzzletron/mip_progress.py @@ -54,7 +54,7 @@ def get_ts(line): complete_ts = None for line in lines: ts = get_ts(line) - if ts and ("sweep.py:292" in line or "Puzzletron Progress 8/8" in line): + if ts and ("Results written to:" in line or "Puzzletron Progress 8/8" in line): complete_ts = ts break @@ -106,12 +106,11 @@ def get_ts(line): # ── Sweep enabled: per-rate progress ───────────────────────────────────────── rate_start = {} for line in lines: - if "sweep.py:258" in line: - m = re.search(r"compression_rate=([\d.]+)", line) - if m: - r = norm(m.group(1)) - if r in all_rates and r not in rate_start: - rate_start[r] = get_ts(line) + m = re.search(r"compression_rate=([\d.]+)", line) + if m: + r = norm(m.group(1)) + if r in all_rates and r not in rate_start: + rate_start[r] = get_ts(line) sweep_start = rate_start.get(all_rates[0]) if all_rates else None From ca9d8b4009d3df0ff06059de46fef64f398cb92d Mon Sep 17 00:00:00 2001 From: Daniel Korzekwa Date: Fri, 19 Jun 2026 00:26:40 -0700 Subject: [PATCH 17/17] add set -o pipefail to torchrun pipelines so torchrun failures are not masked by grep exit code Signed-off-by: Daniel Korzekwa --- .agents/skills/puzzletron/SKILL.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.agents/skills/puzzletron/SKILL.md b/.agents/skills/puzzletron/SKILL.md index 2b5c5bb1e94..803cd5bf816 100644 --- a/.agents/skills/puzzletron/SKILL.md +++ b/.agents/skills/puzzletron/SKILL.md @@ -45,7 +45,7 @@ Parse `nproc_per_node` from args using either positional or flag syntax: Run the following Bash command, substituting `` with the parsed value: ```bash -export PYTHONPATH=$PYTHONPATH:/workspace/Model-Optimizer && \ +set -o pipefail && export PYTHONPATH=$PYTHONPATH:/workspace/Model-Optimizer && \ torchrun --nproc_per_node examples/puzzletron/main.py \ --config examples/puzzletron/configs/llama-3_1-8B_pruneffn_memory/llama-3_1-8B_pruneffn_memory.yaml \ 2>&1 | tee ./log.txt | grep "Puzzletron Progress" @@ -77,7 +77,7 @@ Parse `nproc_per_node` from args using either positional or flag syntax: Run the following Bash command, substituting `` with the parsed value: ```bash -export PYTHONPATH=$PYTHONPATH:/workspace/Model-Optimizer && \ +set -o pipefail && export PYTHONPATH=$PYTHONPATH:/workspace/Model-Optimizer && \ torchrun --nproc_per_node examples/puzzletron/main.py \ --config examples/puzzletron/configs/llama-3_1-8B_pruneffn_memory/llama-3_1-8B_pruneffn_memory.yaml \ --mip-only 2>&1 | tee ./log.txt | grep "Puzzletron Progress"