Upgrade to v1.0.0 - #539
Open
Zhiyuan He (hzy46) wants to merge 372 commits into
Open
Conversation
- gateway-config.yaml: wildcard route adds return_token_ids=true to all requests - run.sh: in vLLM mode, creates ConfigMap from gateway config and patches agl-lite deployment with volume mount + --gateway-config flag - proxy: event captures prepared body (with injected params), not original — RL algorithm sees exactly what was sent to model server - rl_loop.py: logs prompt_token_ids + response token_ids from events, 2 new structural checks (return_token_ids in request, token_ids in response) - Verified: 102 prompt tokens + 253 response tokens captured per rollout - Updated reference_output_vllm.log with token_ids - 271 unit tests passing
Topology change for vLLM mode:
Before: agl-lite in minikube pod, port-forward to host, gateway→vLLM
crosses minikube↔host network boundary
After: agl-lite on host (process), gateway→vLLM is localhost,
no port-forward needed, only controller+agents in minikube
Changes:
- deploy.sh: --no-serve flag skips agl-lite Deployment (controller-only)
- run.sh: vLLM mode starts 'agl-lite serve' as host process with gateway
config, waits for healthz; mock mode unchanged (all in K8s + port-forward)
- .env.vllm.example: AGL_LITE_URL=host.minikube.internal:8080 (K8s→host),
AGL_MODEL_ENDPOINT=localhost:8010 (gateway→vLLM, both on host)
- README: updated architecture diagram showing colocated topology
- Both modes E2E verified: mock 10/10, vLLM 4/5 (80%) all checks pass
- 271 unit tests passing
…on, new decisions
Detailed daemon breakdown (1154 lines: 30% replace, 63% reuse, 6% simplify). Phase 5a: triplet API in agl-lite (events→triplets server-side). Phase 5b: two options for VERL-side — daemon subclass vs standalone interface. Phase 5c: full training loop E2E.
…, not vice versa Corrected dependency direction: agl-lite is a standalone HTTP service with no VERL/torch knowledge. The daemon subclass (Option A) lives in agent-lightning repo, talks to agl-lite over HTTP. Recommended Option A (~150 lines) over Option B (~650 lines with copied tensor math).
Adds format=triplet query param that trims events for RL training: - model_request: extracts prompt_token_ids + response_token_ids from streaming (list of SSE chunks) or non-streaming (dict) responses, strips full request/response bodies - reward: keeps only scalar value, strips message - other event types: pass through unchanged No new endpoint — same auth, filtering, pagination. Raw events still available without the flag. 4 new tests (streaming, non-streaming, no token_ids, full-event baseline). 275 total tests passing.
Phase 5a+5b: agl-lite side of VERL integration.
Server-side (5a):
- format=triplet on GET /api/events trims model_request to
prompt_token_ids + response_token_ids, reward to scalar value
- AglLiteClient.get_events() accepts format param
Daemon (5b) — agl_lite/verl/daemon.py (851 lines):
NEW (187 lines): store interaction via AglLiteClient
- _async_set_up: register_models + enqueue_rollouts
- _async_validate_data: get_events(format=triplet) → Triplet/RolloutLegacy
- _async_run_until_finished: poll get_rollout for succeeded status
- No proxy server, no adapter, no LightningStore
COPIED (510 lines): from agent-lightning AgentModeDaemon
- get_train_data_batch: triplets → padded tensors → DataProto
- Multimodal (mrope, image handling)
- Utilities (padding, token matching)
- Validation/metrics
Tests: 9 new (5 utility, 4 daemon with real agl-lite server via ASGI transport)
Total: 284 tests passing
Slidev deck covering: 1. Why agl-lite (dependency problem, what we actually need) 2. Four key simplifications (LiteLLM, OTEL, Store, execution) 3. Architecture (high-level, data flow, weight updates) 4. VERL integration (AglLiteDaemon, triplet format, trainer code) 5. Developer guide (what lives where, agent contract, gateway config) 6. Status and next steps 21 content slides + 6 section dividers. ~25 min presentation.
Restructured around three design choices (from README): 1. Self-owned gateway (replaces LiteLLM dep) 2. Gateway-level data capture (replaces OTEL dep) 3. K8s-native runner (store simplification as consequence) Technical deep dives on gateway and store+controller. Architecture diagram referenced from docs/images/. No before/after comparisons with Agent Lightning.
… restructured Changes: - Remove all v-clicks/animations (technical discussion) - Add agent_output as third reserved event type - Move agent contract slide into gateway section - Add deployment section with math-poc vLLM example - Remove 'How You Can Help' page - Use markdown image syntax for architecture diagram - Add 7-item agenda matching new structure
Shows merge flow: job_template (raw pod spec) + controller injection + rollout.config overrides → K8s Job. Two examples: simple math-poc and multi-container coding tasks with scorer sidecar.
…te into two pages
Add examples/swe_bench design item covering: - Per-instance SWE-bench Docker images via rollout overrides - Pluggable coding agents (claude_code, mini_swe_agent) with install/run scripts - Mountable config files (CLAUDE.md) via ConfigMap - Reward function: separate evaluation container applies patch + runs golden tests - Algorithm script structure following math-poc pattern with vLLM backend - File layout and open questions for discussion
(A) Use RolloutConfig.image (first-class field) instead of overrides
(D) Evaluation runs inside K8s as a second rollout, not on algorithm host
- eval_script generated from swebench.harness.make_test_spec (pure Python)
- evaluator job: apply patch + run eval_script + parse log + post reward
- no Docker SDK needed on algorithm host
(C) ConfigMap for agent scripts + CLAUDE.md, mounted via existing Mount schema
(1) Naive image pull for now, IfNotPresent; Epoch AI trimmed images as fallback
(2) Volume mount via ConfigMap confirmed; open question on large files deferred
Key insight: eval_script only resets TEST files (from test_patch), not source files. Agent's code modifications are untouched. So evaluation can run in the same container right after the agent finishes. - eval_script pre-generated by algorithm via make_test_spec() (~2KB bash) - Passed to container via AGL_EVAL_SCRIPT env var - entrypoint.sh: install agent → run agent → git diff → eval_script → grade → post reward - grade.py: minimal log parser (~30 lines), no swebench package needed in container - Eliminates second rollout, patch-passing, and Docker-on-host requirement - Simplified file layout (no evaluation/ dir, no eval-job-template)
…iscuss] Key insight: separate task-specific logic (dataset parsing, image selection, eval_script generation, official grading) from task-agnostic logic (model registration, rollout polling, triplet→tensor construction). TaskController interface: - prepare_rollouts(data, is_train) → List[EnqueueRolloutRequest] - compute_rewards(rollout_ids, volume_path) → Dict[str, float] Volume-based grading: container writes test_output.txt + patch.diff to shared volume. Algorithm-side TaskController reads files and calls official grading tools (e.g., swebench get_eval_report). No grade.py needed in container. Examples: SWEBenchController (~100 lines), MathController (~30 lines). Daemon becomes task-agnostic (~300 lines). New task = new controller only.
…erver Replace TaskController (separate process) with RolloutHooks (in-server): - on_enqueue: pre-processor, transforms request BEFORE persist - on_succeeded: post-transition, runs inside update_rollout() atomically Key insight: single-threaded sync store means hooks are atomic — no reader can see intermediate state. No flags (reward_pending) or intermediate states needed. Reward is in the store before update_rollout() returns. User workflow: write hooks.py → build custom Docker image → done. SWEBenchHooks: on_enqueue maps instance→image+eval_script, on_succeeded reads volume + calls official get_eval_report(). Updated SWE-bench file layout: add hooks.py, Dockerfile.server, remove grade.py (grading now in server-side hook).
Hook-facing context (e.g., original dataset row, ground_truth, grading info). Not sent to container — only accessible to store hooks for task-specific logic like reward computation. Three fields now have clear consumers: input → agent (AGL_TASK_INPUT env var) config → K8s controller (image, command, resources) metadata → hooks (dataset context, grading info) 284 tests passing.
RolloutHooks base class with 3 hook points: - on_enqueue: pre-processor, transforms request before persist - on_succeeded: post-transition, fires atomically after SUCCEEDED - on_failed: post-transition, fires after TERMINAL_FAILED Store integration (memory.py): - __init__ accepts optional hooks parameter - enqueue_rollouts: calls on_enqueue before creating each rollout - update_rollout: calls on_succeeded/on_failed after status transition - Hook errors are logged but don't crash the transition (except on_enqueue which prevents rollout creation — the request is invalid) Infrastructure: - agl_lite/hooks.py: RolloutHooks ABC + load_hooks() dynamic loader - ServerSettings.hooks: path to hooks module - CLI: agl-lite serve --hooks path/to/hooks.py - create_app: loads hooks at startup, passes to InMemoryStore 12 new tests covering: - on_enqueue: transforms request, passthrough without hooks, error prevents creation - on_succeeded: reward posted atomically, wrong answer, hook error resilience - on_failed: zero reward posted - load_hooks: file loading, missing file, no subclass, multiple subclasses 296 tests passing (284 existing + 12 new).
The controller no longer reads rollout.input — it only applies config. Task input to the agent is now explicitly set by hooks via config.environment_variables['AGL_TASK_INPUT']. This cleanly separates: input → algorithm data (raw dataset row, read by hooks) config → K8s execution (env vars, image, mounts — set by hooks) metadata → algorithm control indexes (batch_idx, etc.) Without hooks, agents get task from baked-in Docker image or from config.environment_variables set directly by the caller. 296 tests passing.
RolloutMetadata replaces dict[str, Any] with typed fields: - batch_idx, sample_idx_in_batch, trial_idx_in_group (algorithm tracking) - data: dict (raw dataset content for hooks to use in grading) - extra='allow' for task-specific extensions Three fields now have clear consumers: input → algorithm data (raw dataset row, read by hooks) config → K8s controller (image, env vars, mounts) metadata → algorithm indexes + hook grading context (metadata.data) 296 tests passing.
input already holds the raw dataset content. extra='allow' lets hooks stash grading context (e.g., ground_truth) directly as extra fields on metadata. No need for a reserved data dict. RolloutMetadata is now minimal: batch_idx, sample_idx_in_batch, trial_idx_in_group + extra fields 296 tests passing.
New structure:
examples/math-poc/
├── rl_loop_v2.py # unified, task-agnostic (~200 lines)
├── run_v2.sh # run.sh [mock|vllm] (default: vllm)
├── mock/
│ ├── hooks.py # MathMockHooks: boxed embedding, exact match
│ ├── .env.example
│ ├── gateway-config.yaml
│ ├── job-template.yaml
│ └── k8s-mockai.yaml
└── vllm/
├── hooks.py # MathVllmHooks: plain questions, numeric reward
├── .env.example
├── gateway-config.yaml
└── job-template.yaml
Hook responsibilities:
on_enqueue: set image, AGL_TASK_INPUT, AGL_MODEL_NAME, stash ground_truth
on_succeeded: extract answer from events, compute reward, post reward event
rl_loop_v2.py is fully task-agnostic: sends raw JSONL rows as input,
hooks do all transformation and grading. ~200 lines vs ~950 combined before.
7 new tests for mock + vllm hooks. 296 + 7 = 303 tests passing.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix: llm-in-sandbox-minikube
fix typo and clean deploy folder
…nc training (#27) * feat(is): add bypass-mode rollout importance-sampling support Capture chosen-token rollout logprobs end to end and feed them into VERL bypass mode so old_log_probs = rollout_log_probs, avoiding actor recompute. - proxy: inject logprobs=True on train-mode requests - events: extract finite chosen-token logprobs (chat + completions schemas); never fail the triplet query on bad logprobs, report status/error instead - rollout_bridge: thread response_log_probs through triplets/trace/trajectory, pad/truncate in lockstep with response_mask, fill masked/dropped positions with finite 0.0, drop rows with missing/invalid/length-mismatched logprobs (with metrics), emit float32 rollout_log_probs only when present - trainer: import apply_bypass_mode; in _async_train_step, when bypass_mode is enabled, require finite rollout_log_probs and skip _compute_old_log_prob * feat(science_world): enable bypass-mode rollout correction (IS) Add algorithm.rollout_correction with bypass_mode + ppo_clip so the science_world example trains using rollout logprobs (RS/IS off for first pass).
* feat: add search_r1 example * Handle null rollout metadata Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* chore: simplify installation dependencies * chore: simplify verl installation scripts * docs: add installation guide * docs: remove verl sync warning * fix: use local calc-x data paths * chore: streamline verl setup * docs: clarify verl cuda variants
* Add Search-R1 completion API agent * End Search-R1 rollouts on invalid action
* example/async_swe_smith: online GRPO training example for SWE-smith
Add an end-to-end async GRPO example that trains a SWE agent on the
SWE-smith dataset via agl-lite, plus supporting infrastructure:
- examples/async_swe_smith: agent, chat template, k8s job template,
image pre-pull, FSDP + Megatron trainers, Qwen3-30B-A3B / Qwen3.5-9B
launch scripts, README, and a smoke test.
- agl_lite/verl/trainer.py: per-step GRPO group / zero-advantage group
counts and a per-stage perf/mfu/actor_infer metric.
- examples/calc_x/comparison: async-vs-sync runner scripts.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* feat(verl): record per-rollout pod lifecycle timing
Capture server-authoritative running_at/finished_at while polling and
emit, per training step, a wandb Table of per-rollout submitted/running/
finished timestamps plus queue-wait/run-duration scalar aggregates. The
queue wait (running_at - submitted) exposes how long pods sit waiting
when jobs are launched in CPU-limited batches.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* fix(setup_verl): install matching CUDA 13.0 toolchain for cu130 flash-attn build
The cu130 variant builds flash-attn from source against torch's CUDA 13.0
runtime, but the system CUDA toolkit is often 12.x, so nvcc rejects the
build. Install a CUDA 13.0 pip toolchain (nvcc/crt/nvvm/cccl/runtime) pinned
to >=13.0,<13.1 and point the build at it via CUDA_HOME/PATH/LIBRARY_PATH/
LD_LIBRARY_PATH/CPATH. Pinning to 13.0.x keeps nvcc's version aligned with
torch's CUDART (13000); a mismatched minor (e.g. 13.3) trips cccl's
"CUDA compiler and CUDA toolkit headers are incompatible" check.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* smaller cpu request
* feat(async_swe_smith): add Qwen3-8B sync-rollout trainer launch script
Mirrors the async async_swe_smith_qwen3_8b run (wandb fxaj7bcw) but flips
agentlightning.async_rollout.enabled=False for synchronous rollouts, keeping
all other hyperparameters identical (TP=2, GRPO n=8, train_batch_size=32,
ppo_micro_batch_size_per_gpu=2, lr=1e-6, clip 0.2/0.28, max_model_len=32768).
rollout.mode stays async (vLLM OpenAI server mode for tool calling). Sizing
knobs are env-overridable (AGL_GPU_MEM_UTIL, AGL_ROLLOUT_TP, etc.).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* example/async_swe_smith: add Qwen3-8B async runner; enable chunked prefill
Add run_qwen3_8b_async.sh (async counterpart of run_qwen3_8b_sync.sh):
identical training hyperparameters, only agentlightning.async_rollout.
enabled=True, for the sync-vs-async speedup A/B. Also flip the example's
vLLM enable_chunked_prefill to True.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* example/async_swe_smith: lower agent pod memory limit to 2Gi
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* example/async_swe_smith: expose train/async batch sizes as env-overridable knobs
Surface data.train_batch_size (32) and async_rollout.async_train_batch_size
(48) in run_qwen3_8b_async.sh via AGL_TRAIN_BATCH_SIZE / AGL_ASYNC_TRAIN_BATCH_SIZE,
passed through SIZING_OVERRIDES. Also bump default total_epochs 2 -> 4.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* example/async_swe_smith: disable chunked prefill + val-before-train; add requirements.txt
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* update config
* smaller poll interval for less cpu pressure
* example/async_swe_smith: auto-create AGL_NAMESPACE before controller start
The controller does not create its target namespace; with a non-default
AGL_NAMESPACE the ConfigMap step would fail on a missing namespace. Ensure
it idempotently before launch so per-user namespace isolation works.
Co-Authored-By: Claude Opus 4 (1M context) <noreply@anthropic.com>
* example/async_swe_smith: add openai-preinstalled job template variant
Uses the :openai SWE-smith images (openai library baked in) and drops the runtime 'pip install openai', removing the per-rollout cold-start cost.
* async_swe_smith: use openai-preinstalled job template for training
Point the async trainer at job-template-openai.yaml (":openai" images
with the openai library prebuilt) so rollout pods skip the runtime
pip install openai and start the agent directly.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* example/async_swe_smith: add Qwen3-30B-A3B Megatron/R3 async runner
Conservative single-node 4x B200 defaults (n=4, gpu_mem_util=0.6,
train_batch=16, full param/optim/grad offload + recompute) sized to fit
30B-A3B; all knobs env-overridable. Set vLLM moe_backend=triton.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: support explicit pre-split train/val datasets
Add --train-dataset-path / --val-dataset-path (default to
train_datasets.jsonl / val_datasets.jsonl) so the trainer consumes
pre-split, pre-curated datasets as-is: no FAIL_TO_PASS curation and no
train/val split. Falls back to the legacy single-file + split path when
those files are absent. Adds --max-val-instances to bound validation
eval time.
Large local datasets (subset0 / train / val .jsonl) are excluded via
.git/info/exclude rather than .gitignore, so this drops the tracked
subset0 ignore line.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* async_swe_smith: support explicit pre-split train/val datasets in megatron entrypoint
Mirror the load_split_file path from train_smith_agent.py (27ec14d) into the
Megatron+R3 entrypoint: add --train-dataset-path/--val-dataset-path/
--max-val-instances and consume the pre-split files as-is (no FAIL_TO_PASS
curation, no internal train/val split) when both exist, else fall back to the
single-file split. Also add --ci so the run script's smoke flag is a real arg
instead of leaking into the hydra config overrides.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* async_swe_smith: add rollout correction (IS/RS) knobs to megatron runner
Wire algorithm.rollout_correction.* (bypass_mode / rollout_is / rollout_rs
and thresholds) into the Qwen3-30B-A3B megatron runner as env-overridable
sizing overrides, mirroring uni-agent's train_qwen3_moe_rc.sh defaults.
loss_mode is left at the actor default (verl 0.8.0 has no gspo).
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: match fsdp rollout sizing in megatron runner
Bump default ROLLOUT_N 4->8, TRAIN_BATCH_SIZE 16->32, ASYNC_TRAIN_BATCH_SIZE
24->48 to mirror the fsdp distributed run's rollout workload, enabling an
apples-to-apples engine comparison at equal generation volume.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: standardize wandb experiment naming
Name runs as swe_smith_{sync|async}_{model}_{backend} so FSDP and Megatron
runs are distinguishable in wandb. Mode is read from rollout.mode, model
from the basename of the model path; backend is fsdp / megatron per script.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: add pure-rollout collection harness
Collect reward (all-0/all-1 GRPO degeneration), completion-token length, and
turn-count stats over the train/val datasets without running a VERL trainer.
- serve_vllm_qwen3_8b.sh: stand up Qwen3-8B vLLM (128K ctx, chunked prefill,
TP=1, CUDA graph, hermes tools, return-tokens-as-token-ids) and register it
with the server proxy via POST /api/models.
- enqueue_rollouts.py: POST each dataset row N times (GRPO group) to
/api/rollouts with the openai job template + is_train flag.
- rollout_stats.py: poll finished rollouts, aggregate from reward/model_request/
agent_output events, group by data_id, report train/val separately.
- job-template-openai.yaml: SMITH_MAX_TURNS=1000 so turns (not the 40-cap) bound
self-completion.
- rollout_test.md: design doc.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: retry reward post until it succeeds
Reward events are training signal, so a single timeout silently dropped
the sample. Retry with capped exponential backoff until the post lands.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: full-run coverage for pure-rollout stats harness
The rollout_stats collector previously polled GET /api/rollouts?state_in=
succeeded&failed&limit=10000. That list route returns matches[:limit] in
insertion order with no offset, and each rollout is ~77KB (job_template is
embedded), so for the 164k-rollout SWE-smith run stats would silently plateau
at ~10k terminal rollouts (~6% of the run) — never seeing later completions.
Server: add an append-only completion log (_terminal_order, appended on
terminal state transitions in patch_rollout) and a cursor-paginated, lightweight
projection endpoint GET /api/rollouts/terminal?after=&limit= returning only
rollout_id/state/data_id/is_train + a next_after cursor. Because the log is
ordered by completion, an index cursor never misses out-of-order completions
and needs no full rescan of the store.
Collector: drain the cursor instead of re-listing; O(page_size) work per poll,
no 10k cap. Atomic snapshot writes (tmp + os.replace), per-rollout event-fetch
retry that skips unreadable rollouts without stalling the cursor, and a
--page-size flag. Snapshot now reports processed/total_terminal/backlog.
serve_vllm_qwen3_8b.sh: Qwen3-8B native ctx is 40960 and vLLM 0.20.2 dropped
--rope-scaling, so --max-model-len 131072 was rejected outright. Add YaRN via
--hf-overrides (factor 4.0 over original 32768) when extending beyond native
context — the documented 128K goal now actually serves.
Tests: cover the new endpoint (completion-order pagination, cursor advance,
projection fields, non-terminal exclusion); reset _terminal_order between tests.
.gitignore: ignore the generated rollout_stats.json output.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* async_swe_smith: checkout bug branch so the agent sees the injected bug
The SWE-smith image defaults /testbed to clean `main`, so the agent saw fixed
code, had nothing to fix, and evaluate() false-passed (reward 1.0 on a 0-byte
patch). Now the agent checks out the bug branch HEAD (`Remove F2P Tests`):
buggy source with the FAIL_TO_PASS tests removed, so it cannot read the tests
to reverse the fix. At evaluation time the F2P test files are restored from the
parent `Bug Patch` commit, keeping the agent's edits, so the real
FAIL_TO_PASS/PASS_TO_PASS suite runs against the fix.
Co-Authored-By: Claude <noreply@anthropic.com>
* Align SWE-smith agent loop with mini-swe-agent
Replace OpenAI tool-calling with mini-swe-agent's text/bash action space:
- one bash code block per turn (parse_action); submit via the
COMPLETE_TASK_AND_SUBMIT_FINAL_OUTPUT marker instead of a submit tool
- swebench-style system/instance prompts adapted for /testbed
- head/tail observation elision + format-error nudges (max_format_errors)
- execute commands via 'bash -c' with pagers/progress bars silenced
Keeps AGL glue (events, eval-meta, bug-commit checkout, F2P restore,
git-diff patch capture, FAIL_TO_PASS/PASS_TO_PASS eval) and the
context-overflow / transient-error loop contracts. Adds SMITH_MAX_FORMAT_ERRORS.
* async_swe_smith: distinguish eval timeout from real failure + rollout tooling
- smith_agent: evaluate() now flags eval_timeout (pytest rc=124 / "timed out"),
surfaced in the reward event; fix p2p_ok counting None status as PASSED (which
let timeouts false-pass as resolved). Set temperature=1.0 explicitly; cap
SMITH_MAX_TURNS at 100 in job-template-openai.
- rollout_stats: tolerate transient connection errors (RemoteDisconnected).
- add monitor_controller.sh (periodic health probe), trace.md (4-rollout trace
example), reward_table.md (per-instance reward/patch snapshot).
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: add SWE-smith --f2p_only eval mode (on by default)
Mirror swesmith/harness/grading.py: when f2p_only, restrict evaluation to the
test files containing FAIL_TO_PASS. F2P is kept whole; PASS_TO_PASS is filtered
to only the P2P tests in those same files — same-file regressions still count,
but the many unrelated cross-file P2P tests (the eval-time/timeout driver) are
dropped. Resolution still needs F2P AND filtered P2P to pass. Toggled by
SMITH_F2P_ONLY (default on).
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: add rollout-collection runbook + full f2p_only result table
- plan.md: end-to-end runbook for launching a local pure-rollout collection
batch (cold-start vLLM/server/controller, enqueue, monitor, gold-patch
comparison), with the known pitfalls and how to run the second half.
- reward_table.md: full 367-instance agent-vs-gold patch comparison under
official f2p_only (all-0 98.1%, eval timeouts 0/1468).
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: lightweight branch switch to avoid checkout timeouts
git checkout --force <branch> stats/rewrites the whole /testbed working tree
(thousands of files for repos like pandas), which times out under high pod
concurrency (node at 6x oversubscription) and fails jobs. Instead, diff
main..branch at the object level to get just the changed files and apply only
those: modified/added paths via checkout, deleted F2P tests via git rm. All
git calls now go through a retry helper. Measured 1234ms -> 32ms on pandas;
working-tree state is identical to the old full checkout.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat(swe-smith): Qwen3.5-9B pure-rollout eval + 8B vs 9B reward table
- 4x vLLM Qwen3.5-9B (TP=1, round-robin) pure-rollout over subset0
- gen_compare_table.py / gen_reward_table_9b.py reward-table builders
- reward_table.md: 8B(all-0 98%) vs 9B(~47%), 116/230 improved
- baseline 8B table snapshot; trainer wandb table trimmed; async sizing
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe-smith: per-call max_tokens 16384, vLLM ctx 128k for 9B rollout
- smith_agent default AGL_MAX_TOKENS 4096->16384
- job-template-openai: AGL_MAX_TOKENS=16384 env
- vLLM restarted max_model_len 131072 (4x, registered)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix shuffling issue & zero sample issue
* async_swe_smith: align reward with official XFAIL grading + 128k train ctx
Count XFAIL as pass (F2P/P2P) to match swesmith/harness/grading.py; set
data/trajectory max lengths to 65536 = 128k. Add event-driven fail/starve
pod watcher.
Co-Authored-By: Claude <noreply@anthropic.com>
* swe-smith: reward_table 9B 321/367, add per-rollout total token length
- gen_compare_table: 9B total-len column (avg) + p50/p90/max summary
- 9B vs 8B: all-0 98%->52%, improved 147, regress 0
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe-smith: annotate units in reward_table (patch=bytes, total=tokens)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* async_swe_smith: tee rollout pod logs to hostPath for post-mortem
Mount host /agl-logs and tee stdout/stderr to agl-rollout-<id>.log so a
killed/evicted pod's trace survives the TTL reaper. Both templates.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: bump pod memory to 4Gi to stop OOMKills
Long 128k-ctx trajectories were killed at 2Gi. Raise request+limit to 4Gi.
Co-Authored-By: Claude <noreply@anthropic.com>
* refactor: promote async SWE-smith example
Rename async_swe_smith to swe_smith, update startup docs/scripts, and prepare local :openai rollout images from minikube-loaded tags.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: log trace merge mismatches to wandb
Record capped unmerged trajectory triplets as a wandb table and expose mismatch row metrics from the rollout adapter.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* async_swe_smith: add delete.jsonl — instances with missing bug branches
35 instances across 11 repos (mostly Go) whose images lack the
origin/<instance_id> branch, so checkout fails with "unknown revision"
and the rollout dies before turn 1. Listed for dataset filtering.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: wait out gateway pause instead of burning a turn
On 429 'gateway paused' (async weight sync), the agent now retries in place
every 5s up to SMITH_GATEWAY_WAIT_S (default 600s) without incrementing the
turn or appending an empty assistant message. Previously the pause was
treated as a generic error, burning turns and polluting the trajectory.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: harden fetch_eval_meta against transient server errors
RemoteDisconnected escaped the except (it is http.client.HTTPException, not
URLError) and crashed the agent; timeouts were swallowed but left instance_id
empty, wasting a full rollout until checkout failed. Catch connection-reset /
HTTP exceptions, retry 4x with backoff for the bursty transient failures, and
abort immediately when no instance_id can be fetched instead of running on.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: force pytest serial in eval to stop OOMKills
The eval pytest command passed no -n flag, so a testbed project's own config
(e.g. scipy's `-n auto`) spawned workers = HOST core count (dozens), ignoring
the pod's cpu:1 / 4Gi cgroup. ~70 python workers @ ~112MB each exhausted the
4Gi limit and triggered cgroup OOMKill mid-rollout — verified live: OOMed pods
had 70+ python procs, memory.current pinned at 4094/4096Mi. Append `-p no:xdist`
to disable the plugin (overrides any ini `-n auto`; no-op when xdist absent).
Confirmed fixed: new pods now run 1 python proc instead of 70.
Co-Authored-By: Claude <noreply@anthropic.com>
* async_swe_smith: update reward_table.md — Qwen3.5-9B pure-rollout (620 instances)
Snapshot of the in-progress Qwen3.5-9B pure-rollout collection over
train_datasets.jsonl (is_train=false, n=4 GRPO groups), throttled to 50
in-flight. 620 instances / 2490 rollouts terminal so far; all-0 544, all-1 51,
mixed 25; 0 eval timeouts (SMITH_EVAL_TIMEOUT=3000, SMITH_MAX_TURNS=100).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* async_swe_smith: cap pytest at -n4, bump pod mem 6Gi, throttle job rate
Eval-time OOMKills traced to testbed `-n auto` reading host cores (~70 xdist
workers) and busting the cgroup. Pin -n4 (~450MB) with serial fallback when
xdist is absent. Raise pod memory to 6Gi and lower max_jobs_per_minute to 100.
Co-Authored-By: Claude <noreply@anthropic.com>
* fix: harden SWE-smith rollout checkout
* docs: document SWE-smith Qwen3.5 sync run
* swe_smith: point hostPath logs at host dir after kubeadm migration
minikube is replaced by a kubeadm single-node cluster, so pods run directly
on host docker. hostPath /agl-logs (minikube VM path) now points at the real
host dir /home/v-zhiwenzhou/agl-logs — no minikube mount needed. Also lower
openai template pod memory to 4Gi.
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: update reward_table.md — Qwen3.5-9B pure-rollout (7300 instances)
Snapshot of the Qwen3.5-9B pure-rollout collection over the verified + full
train datasets (is_train=false, n=4 GRPO groups), throttled to 50 in-flight,
deduped by data_id. 7300 instances / 29174 rollouts: all-0 4510 (61.8%),
all-1 1667 (22.8%), mixed 1123 (15.4%); 50 eval timeouts
(SMITH_EVAL_TIMEOUT=3000, SMITH_MAX_TURNS=100).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: store reward table as SQLite db, drop markdown
Switch the reward table from reward_table.md to reward_table.db (SQLite).
Column names match the old markdown detail table exactly: instance, rollouts,
"r=0", "r=1", "分类", "best patch_size", "avg turns", "超时".
gen_reward_db.py is incremental — it caches each terminal rollout by id in a
rollout_events table and only fetches events for NEW rollouts, so a refresh
takes ~7s instead of ~10min at 30k+ rollouts. reward_table is rebuilt from
rollout_events (is_train=0, grouped by data_id).
Snapshot: 30905 rollouts / 7718 instances (all-0 4606, all-1 1893, mixed 1219).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: retry idempotent /proxy/pause with 5min timeout; pod mem 6Gi
/proxy/pause only sets state.paused=True (idempotent, lock-guarded), so retrying
is safe. When the server is saturated with in-flight rollouts the pause POST can
be slow to get a connection and the default 30s timeout is not enough — give it
300s and 5 retries with backoff. Also bump openai template pod memory to 6Gi
(kernel OOM traced anon-rss ~4.1GB busting the 4Gi limit).
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: self-documenting timestamped run logs in launcher
Redirect each role's stdout+stderr to a timestamped file
(/tmp/agl_logs/{role}_{model}_{YYYYmmdd-HHMMSS}.log) with a stable
{role}_latest.log symlink, so repeated server/trainer restarts no longer
overwrite each other's logs. A fixed name like /tmp/train_9b_async.log
silently loses the previous run's crash trace on relaunch.
Opt out with AGL_LOG_TO_FILE=0; override dir/name via AGL_LOG_DIR/AGL_LOG_FILE;
auto-skipped for --ci smoke tests.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* fix: dedupe triplet model requests by prompt tokens
* feat: add retry logic to non-GET HTTP requests to harden the trainer
Make every non-GET request in the trainer process resilient to transient
failures so a single HTTP error can no longer crash training mid-step:
- register_model (POST /api/models): retry with backoff (idempotent upsert)
- _resume_gateway (POST /proxy/resume): retry with backoff (idempotent)
- _create_rollouts (POST /api/rollouts): pre-assign client rollout_ids and
retry; server-side enqueue is now idempotent (existing id returns the
existing rollout, events untouched), so retries never duplicate rollouts
- delete_model / _delete_rollout: best-effort, swallow errors
- add DELETE /rollouts/{id} (idempotent, cascades to events) and delete
completed rollouts from the managers to keep server-side state bounded
Also switch AglLiteSyncClient's retrying GET to print over logging for
consistency with the rest of the codebase.
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: prevent git-history reward hacking in smith_agent
SWE-smith testbed repos ship the full git history, so the agent can recover
the injected-bug fix (and the deleted FAIL_TO_PASS tests) via
`git checkout <pre-bug-sha> -- <src>` / `git show` / `git log -p`. On the
Qwen3.5-9B run this inflated val/reward to ~0.80 (96% of resolved val rollouts
used the git hack; ~0% solved cleanly).
Agent-side mitigation:
- relocate_git(): move /testbed/.git out of the worktree after checkout (O(1)
same-fs rename); the harness still reaches it via _git_base()
(--git-dir/--work-tree) for restore_f2p_tests + capture_patch.
- _forbidden_action(): reject any agent command that invokes git or reads git
metadata (.git / --git-dir / the relocated dir), returned as an observation;
wired into run_agent_loop before _run.
- _agent_env(): strip SMITH_HIDDEN_GIT_DIR and any var leaking the path so the
relocated dir is invisible in the pod env.
- Drop the git-based submission step from the prompt (harness captures the
patch independently) and note that git is disabled.
Verified on 50 random val instances (base Qwen3.5-9B, isolated pods):
relocation 50/50 (O(1)), 0 leaks, 0 false-positive blocks; submission and
eval/patch-capture work via --git-dir; blocked git attempts recover to manual
edits without hindering legitimate solves.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* feat: log rollout trajectories as wandb artifacts
* feat: add rollout DELETE endpoint and auto-cleanup in rollout managers
Add DELETE /rollouts/{rollout_id} (idempotent, cascades to events) and a
retrying GET on AglLiteSyncClient. Sync manager deletes each round's
completed rollouts; async manager deletes non-carry-over groups on
completion, keeping server-side state bounded during training.
Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit 149400351dc1fe16c2bd98a0a8eae34d3931da85)
* feat: add retry logic to non-GET HTTP requests to harden the trainer
Make every non-GET request in the trainer process resilient to transient
failures so a single HTTP error can no longer crash training mid-step:
- register_model (POST /api/models): retry with backoff (idempotent upsert)
- _resume_gateway (POST /proxy/resume): retry with backoff (idempotent)
- _create_rollouts (POST /api/rollouts): pre-assign client rollout_ids and
retry; server-side enqueue is now idempotent (existing id returns the
existing rollout, events untouched), so retries never duplicate rollouts
- delete_model / _delete_rollout: best-effort, swallow errors
- add DELETE /rollouts/{id} (idempotent, cascades to events) and delete
completed rollouts from the managers to keep server-side state bounded
Also switch AglLiteSyncClient's retrying GET to print over logging for
consistency with the rest of the codebase.
Co-Authored-By: Claude <noreply@anthropic.com>
(cherry picked from commit 137506f293f5287ee62dcacdab358c43f9c40825)
* fix: align rollout cleanup retry merge
* swe_smith: update reward_table.db — 24836 instances (85% of verified)
Qwen3.5-9B pure-rollout over train_dataset_verified.jsonl (is_train=false, n=4,
throttled 50 in-flight, deduped by data_id). 99647 rollouts / 24836 instances:
all-0 13789 (55.5%), all-1 6575 (26.5%), mixed 4472 (18.0%).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: lower pod memory request to 2Gi, keep 6Gi limit
request=limit=6Gi over-reserved: the scheduler pinned ~83 pods against the
node's 510Gi allocatable while pods actually use ~1-2GB, leaving CPU at 16%
and 150Gi RAM idle. Drop request to 2Gi so more pods schedule; keep limit at
6Gi so long-trajectory pods (kernel-observed anon-rss ~4.1GB) don't OOM.
Co-Authored-By: Claude <noreply@anthropic.com>
* feat: route rollouts deterministically by rollout_id hash for prefix caching
Replace round-robin server selection with a stable sha256(rollout_id) mod
pool-size mapping so every request from a rollout lands on the same endpoint,
maximizing prefix-cache hits. Sort the pool by endpoint to keep the mapping
independent of registration order.
Co-Authored-By: Claude <noreply@anthropic.com>
* Log compact rollout trajectory artifacts
* swe_smith: lower pod memory request to 1Gi, keep 6Gi limit
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: Qwen3.5-9B run defaults + tighten token caps, drop internal plan.md
- run_qwen3_8b_async.sh: default model Qwen3-8B->Qwen3.5-9B, max_model_len
32768->65536, expose AGL_ROLLOUT_N, tie ppo_mini_batch_size to train_batch_size,
async overrides (val_before_train=False, test_freq=-1), run-name qwen35_9b_async.
- job-template-openai.yaml: AGL_MAX_TOKENS 16384->12288, add SMITH_OBS_CHAR_CAP=6000
to keep agent trajectories shorter under the 65536 context window.
- remove examples/swe_smith/plan.md (internal planning notes, not shipped code).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Update SWE-smith training settings
* swe_smith: final Qwen3.5-9B reward table + 5245 mixed instances
Collection complete over train_dataset_verified.jsonl (is_train=false, n=4,
throttled 50 in-flight, deduped by data_id via reward db). Final:
29976 instances / 120283 rollouts — all-0 17109 (57.1%), all-1 7622 (25.4%),
mixed 5245 (17.5%). 82.5% are degenerate (zero GRPO advantage); the 5245 mixed
instances are the trainable set.
- reward_table.db: final SQLite snapshot (30MB).
- mixed_instance_ids.txt: the 5245 mixed data_ids (instances with GRPO signal).
- enqueue_throttled.py: throttled enqueuer with --skip-from-db (dedup that
survives a server restart — needed after the in-memory store was wiped).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: prune unused scripts, add log/reward analysis tooling
- Remove no-longer-used enqueue/gen/monitor/serve scripts and stale docs.
- job-template-openai: point rollout-logs hostPath at /agl-logs (minikube 9p
mountpoint) so pod logs reach the host under the docker driver.
- Add filter.md (full->verified dataset filtering rationale), scan_abnormal_pods.sh
(k8s abnormal-pod probe), stat_pod_logs.sh (bulk pod-log anomaly stats), and
report_qwen35_9b.md (Qwen3.5-9B rollout report: 34% resolve, 83% ctx overflow).
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: remove unused pod monitoring/log scripts
Co-Authored-By: Claude <noreply@anthropic.com>
* Add SWE Verify checkpoint evaluation
* Add B200 Qwen3.5-9B vLLM pure-decode roofline benchmark
Measure single-card B200 decode performance vs theoretical roofline for
Qwen3.5-9B (hybrid GatedDeltaNet + full-attention). Includes:
- theoretical roofline (mem/compute roofs, ridge, TPOT floor)
- batch sweep + context sweep via latency-subtraction (pure decode isolation)
- reusable bench + plot scripts, CSV data, roofline.png, README + summary
Key: ~24k tok/s/card decode ceiling (21% of compute roof); batch-1 at 51%
of bandwidth roof (hybrid recurrent-state overhead); empirical ridge ~batch
32-64; context 128->64k drops 4x but <=4k near-lossless (hybrid KV advantage).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* roofline: document measured HBM BW / BF16 TFLOPS in README
Roof charts & theory keep datasheet spec (8.0 TB/s, 2250 TFLOPS) per roofline
convention. Add README section 1.4 with empirically-measured achievable ceilings
(HBM 7.11 TB/s = 89% of spec; BF16 dense 1623 TFLOPS = 72% of spec) plus two
reusable microbenchmark scripts, so real attainment can be cross-referenced
(batch-1 = 57% of measured BW roof; batch-256 = 29% of measured compute roof).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add per-step + per-minute rollout stats collector with long-tail metrics
Instrumentation for the async-GRPO SWE-smith run (Qwen3.5-9B, 4xB200) to
quantify per-vLLM-call long tails and their effect on rollout throughput.
- collect_rollout_stats.py: 60s collector writing two tables from vLLM
Prometheus histograms (e2e latency, response/prompt tokens).
- rollout_step_stats: per-call avg prompt/resp len, per-card out tokens +
throughput, max response, and per-call long-tail percentiles
(resp_tok/e2e_lat p50/p90/p99 + tail_ratio_lat_p99_p50).
- gen_window_throughput: per-~60s throughput time-series inside each gen
window, showing tail-segment collapse.
- rollout_stats.db: 18 step rows + 177 per-minute rows.
- README documents schema + findings: per-call e2e p99/p50 ~12-18x every
step (driven by ~21x response-token tail); within-window throughput
collapses 88-99.7% from peak; effective throughput ~62% of peak.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: reward==1 long-turn penalty + val temperature 0.7; prune scratch files
- smith_agent.py: add length_penalized_reward() that penalizes ONLY solved
(reward==1) trajectories: reward = 1 - λ·clip((n_turns-T0)/(max_turns-T0),0,1);
run_agent_loop now returns (submitted, turns_used). Tunable via env
SMITH_LEN_PEN_T0 (default 55) / SMITH_LEN_PEN_LAMBDA (default 0.2). Reports
raw_value + n_turns on the reward event for monitoring.
- job-template-openai.yaml: surface SMITH_LEN_PEN_T0=55 / SMITH_LEN_PEN_LAMBDA=0.2.
- tests: cover the penalty (solved-only, within-budget, ramp-to-cap, degenerate
span) and unpack the new run_agent_loop tuple.
- server.yaml: default_proxy.val.temperature 0 -> 0.7 (validation samples at 0.7).
- README: add `export FLA_TILELANG=0` to the run recipe.
- Remove scratch/dev files (enqueue_rollouts.py, rollout_stats.py, gen_*.py,
monitor_controller.sh, run_qwen3_8b_async.sh, serve_vllm_qwen3_8b.sh,
watch_failpods.sh, reward_table.db, reward_table_8b_baseline.md, rollout_test.md,
delete.jsonl).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: update mixed_instance_ids.txt to 5407 (add 162 rescued from 128k rerun)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: block network-based reward hacking (NetworkPolicy + agent guards)
Agents were downloading upstream correct source (curl raw.githubusercontent,
pip install <target pkg>, wget, urllib) to overwrite the buggy file — ~26% of
rollouts — after git was already blocked. Two layers now cut this off:
- Network root fix: deploy Calico VXLAN (replacing flannel, which cannot enforce
NetworkPolicy) + default-deny egress that only allows the agl-lite server IP,
so pods have no public internet. See canal.md for the full deploy/verify/
rollback runbook. Adds canal-policy-only.yaml, networkpolicy-egress-lockdown.yaml,
and an app=agl-rollout label on the job template pod.
- Code backstop: extend smith_agent.py _forbidden_action to reject curl/wget/
pip-install/urllib and test-harness file writes (conftest/pytest.ini/
sitecustomize), and align the prompt (drop "you may install it").
Also enables fused kernels in the train configs (unrelated perf tweak).
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: gate long-turn penalty to training only; README -> mixed dataset
- smith_agent.py: length_penalized_reward() now takes is_train and only shapes
reward when is_train AND solved. Validation reward stays the true, unshaped
metric (it drives checkpoint selection). main() derives is_train from
AGL_OPENAI_BASE_URL (/mode/train/ marker; fail-safe: never penalize val) and
logs mode. Fixes the earlier bug where the penalty applied to val too.
- tests: pass is_train explicitly; add test_length_penalty_skips_validation
asserting val reward is never reshaped (even a long solved rollout keeps 1.0).
- README: point the active Qwen3.5-9B recipe at train_dataset_mixed.jsonl (6343
instances = mixed_dataset minus the 3 network-dependent repos dspy/pydantic/
MONAI whose eval needs runtime pip install and breaks once pods are offline).
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* swe_smith: add prompt-length penalty (plan B) + align train config to bh103or9
Penalize context bloat on SOLVED training rollouts: track the largest
server-reported usage.prompt_tokens per rollout (no local tokenizer) and
subtract up to 0.1 as the longest prompt grows from 50K to 64K tokens.
Gated on raw solved status so it stacks with the long-turn penalty (plan A)
and never reshapes validation reward. Tunable via SMITH_PROMPT_PEN_*.
Align train config to the bh103or9 run: model Qwen3.5-9B, gpu_mem_util 0.8,
max_num_batched_tokens 8192, ppo_max_token_len_per_gpu 16384.
Co-Authored-By: Claude <noreply@anthropic.com>
* add max_ppo_update_times
* Merge rollout-level advantage computation (cherry-pick ee78dbe)
Add optional rollout-level advantage (algorithm.enable_rollout_level_advantage)
and its module + tests, wired into AglLiteRayPPOTrainer alongside the existing
compute_advantage path. Cherry-picked from SiweiPro's ee78dbe to take only the
advantage change without the unrelated swe_verify work on that branch.
Co-Authored-By: SiweiPro <18474108006@163.com>
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: turn penalty T0=80, enable max_ppo_update_times=2 + bypass_mode
- Long-turn penalty now starts at 80 turns (was 55), max penalty 0.2 at the
100-turn cap.
- Set agentlightning.max_ppo_update_times=2 to bound PPO updates per step.
- Enable rollout_correction.bypass_mode: reuse rollout log-probs as old_log_prob
for importance sampling, skipping the old-logprob forward pass.
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: untrack examples/swe_smith/infra (local-only tooling)
Roofline benchmarks + rollout-stats tooling are local infra, not part of the
training example. Remove from the tree and exclude locally so they stay off origin.
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith: untrack reward_table.db (large local-only artifact)
30MB reward cache is a local build artifact, not source. Remove from the tree
and exclude locally so it stays off origin.
Co-Authored-By: Claude <noreply@anthropic.com>
* swe_smith / swe_verify: untrack local-only files (docs, dataset curation, network policies, comparison scripts, swe_verify example)
Move these to .git/info/exclude so they stay off origin while remaining local:
- examples/swe_verify/ (7 files, plus tests/examples/test_swe_verify.py to avoid CI ImportError)
- examples/swe_smith/README.md
- examples/swe_smith/canal.md, filter.md, report_qwen35_9b.md (internal notes)
- examples/swe_smith/canal-policy-only.yaml, networkpolicy-egress-lockdown.yaml
(cluster-specific network policies)
- examples/swe_smith/mixed_instance_ids.txt (local dataset curation, 5407 rows)
- examples/calc_x/comparison/ (local benchmark scripts)
Co-Authored-By: Claude <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4 (1M context) <noreply@anthropic.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: hzy46 <362583303@qq.com>
Co-authored-by: SiweiPro <18474108006@163.com>
Register a VERL policy loss that gives each rollout equal weight across its expanded training rows. Normalize advantages by rollout token count and trained batch size, and load the custom loss in Ray actor workers. Add focused coverage for registration, normalization, PPO aggregation, and input validation.
* docs: add agl-lite documentation * docs: refine project messaging * docs: refresh project overview and examples * Update README.md * docs: adopt Agent Lightning v1.0 branding * Update README.md * docs & license * Tune SWE-smith turn penalty * Reorganize example documentation * Refine README tagline * docs: reorganize setup and configuration guides * docs: clarify shared gateway key * docs: simplify configuration and async training guides
* Rename package to Agent Lightning * Keep package initialization minimal
|
You are not allowed to delete the mandatory files in this repo. Total execution time: 6.54 seconds |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR refactors the whole code base and upgrade agent lightning to v1.0.0.
New commits are introduced by --allow-unrelated-histories and all histories are preserved.