test(v1): add taskset-only Oolong Prime Agent smoke adapter - #2333
test(v1): add taskset-only Oolong Prime Agent smoke adapter#2333sethkarten wants to merge 7 commits into
Conversation
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| match = VERDICT_RE.search(response.text.lower()) | ||
| return bool(match and match.group(1) == "yes") |
There was a problem hiding this comment.
🟡 Medium oolong_synth_v1/judge.py:26
parse uses VERDICT_RE.search to find the first standalone yes or no anywhere in the judge response, so a judge reply like "I considered yes, but the answer is no" returns True — the wrong reward is awarded whenever the judge explains its reasoning before giving the final verdict. Consider matching only a single-token verdict or parsing the final yes/no in the response instead of the first one.
| match = VERDICT_RE.search(response.text.lower()) | |
| return bool(match and match.group(1) == "yes") | |
| def parse(self, response: vf.JudgeResponse[bool]) -> bool: | |
| text = response.text.lower().strip() | |
| match = re.fullmatch(r"yes|no", text) or re.search(r"\b(yes|no)\b\s*\.?\s*$", text) | |
| return bool(match and match.group(1) == "yes") |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @environments/oolong_synth_v1/oolong_synth_v1/judge.py around lines 26-27:
`parse` uses `VERDICT_RE.search` to find the first standalone `yes` or `no` anywhere in the judge response, so a judge reply like `"I considered yes, but the answer is no"` returns `True` — the wrong reward is awarded whenever the judge explains its reasoning before giving the final verdict. Consider matching only a single-token verdict or parsing the final `yes`/`no` in the response instead of the first one.
ApprovabilityVerdict: Needs human review 3 blocking correctness issues found. This PR adds a new evaluation taskset with multiple unresolved findings: an ephemeral tunnel URL that will break, timezone-aware/naive datetime comparison bugs in scoring, and regex parsing issues in the judge. These substantive issues in the evaluation infrastructure warrant human review. You can customize Macroscope's approvability policy. Learn more. |
|
|
||
| [env.taskset] | ||
| id = "oolong-synth-v1" | ||
| context_len = 1024 |
There was a problem hiding this comment.
🟡 Medium configs/oolong_synth_v1_luna_n1.toml:15
The config header describes a 256K-context evaluation, but context_len = 1024 selects the 1K dataset bucket. A live run therefore evaluates a much shorter task set than intended, producing misleading results. Set context_len to 262144 to match the stated 256K bucket.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @configs/oolong_synth_v1_luna_n1.toml around line 15:
The config header describes a 256K-context evaluation, but `context_len = 1024` selects the 1K dataset bucket. A live run therefore evaluates a much shorter task set than intended, producing misleading results. Set `context_len` to `262144` to match the stated 256K bucket.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
| id = "prime_agent" | ||
| version = "0.7.1" | ||
| tarball_url = "https://t-1-1916d097074a4b51.tunnel.pinfra.io/prime-agent-0.7.1.tgz" | ||
| tarball_sha256 = "d601e0dd88103cddfe93b7a1da286a1869259b2fb2b7430c0ce5472eaec93cf9" |
There was a problem hiding this comment.
Ephemeral tunnel artifact URL
High Severity
tarball_url now points at an ephemeral tunnel.pinfra.io host instead of a stable release artifact. Those tunnels expire, so any non-dry-run install of prime_agent 0.7.1 will fail once the endpoint disappears. This also removes the prior intentional placeholder blocker meant to hold until an approved pin lands.
Reviewed by Cursor Bugbot for commit f05b55a. Configure here.
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
There are 2 total unresolved issues (including 1 from previous review).
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit a5b09a0. Configure here.
| """Parse the dataset's serialized gold answer (a list literal, or a wrapped date).""" | ||
| if "datetime" not in answer_raw: | ||
| return ast.literal_eval(answer_raw)[0] | ||
| return datetime.strptime(answer_raw, "[datetime.date(%Y, %m, %d)]").replace(tzinfo=timezone.utc) |
There was a problem hiding this comment.
Timezone breaks date score equality
High Severity
parse_gold now returns a UTC-aware datetime, but the date branch still compares it to a typically naive dateutil.parser.parse result. That comparison raises TypeError, which the handler turns into 0.0, so correct ANSWER_TYPE.DATE answers score as wrong.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit a5b09a0. Configure here.
|
|
||
| return 1.0 if dateutil.parser.parse(str(trimmed_output)) == gold else 0.0 |
There was a problem hiding this comment.
🟠 High oolong_synth_v1/taskset.py:72
Date-only answers can never receive credit under the ANSWER_TYPE.DATE branch. parse_gold makes the gold value a UTC-aware datetime, but dateutil.parser.parse on a date-only string like "2024-01-01" returns a timezone-naive datetime. The == comparison between aware and naive datetimes raises TypeError, which is caught and returns 0.0 — so a correct date-only answer always scores zero. Normalize both sides to date (or apply the same timezone) before comparing.
| return 1.0 if dateutil.parser.parse(str(trimmed_output)) == gold else 0.0 | |
| import dateutil.parser | |
| gold_date = gold.date() if hasattr(gold, "date") else gold | |
| return 1.0 if dateutil.parser.parse(str(trimmed_output)).date() == gold_date else 0.0 |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @environments/oolong_synth_v1/oolong_synth_v1/taskset.py around lines 72-73:
Date-only answers can never receive credit under the `ANSWER_TYPE.DATE` branch. `parse_gold` makes the gold value a UTC-aware `datetime`, but `dateutil.parser.parse` on a date-only string like `"2024-01-01"` returns a timezone-naive `datetime`. The `==` comparison between aware and naive datetimes raises `TypeError`, which is caught and returns `0.0` — so a correct date-only answer always scores zero. Normalize both sides to `date` (or apply the same timezone) before comparing.
|
Closing because this local OOLONG taskset adapter is not part of the production ACP evaluation stack. The Yahoo 128k evaluation will consume the canonical Environments Hub package ( No evaluation was run from this PR. The ephemeral artifact pin in this branch is rejected and will not be used. |
1 participant
True Prime Agent ACP Oolong evaluation adapter
Append-only child of lock remediation #2332.
Adds a taskset-only
oolong_synth_v1package so the v1 loader falls back toSingleAgentEnvand permits explicitPrimeAgentHarness; published Hubrlm-oolong@0.2.5hardwires the generic RLM harness and cannot satisfy the requested ACP path.Dry-load proof (no model/sandbox):
SingleAgentEnvOolongSynthTasksetPrimeAgentHarnessprime,vm=TrueIncludes bounded Luna n=1 config with exact Prime Agent artifact URL/SHA placeholders. Offline
uv lock, disposable locked sync, dry-run, and resolved class proof passed. No model call/live eval. Draft/HOLD until exact #1239 artifact builds and is pinned.Note
Medium Risk
New sandbox evaluation path with custom scoring and a Prime Agent harness pinned to a tunnel tarball URL/SHA; incorrect scoring or harness pins could skew eval results, but it does not touch auth or core infra.
Overview
Adds a taskset-only
oolong-synth-v1environment so v1 can run Oolong synth long-context tasks underSingleAgentEnvwith an explicitPrimeAgentHarness(instead of the Hub RLM harness).Each task streams
oolongbench/oolong-synth, filters to acontext_lenbucket, and uploads the long context to/workspace/context.txtfor REPL scanning. The agent writes a single-token answer to/workspace/answer.txt; scoring uses official Oolong rules (exact match, numeric partial credit, date equality) or an optional host-side binary LLM judge.Registers the package in the examples dependency group and adds a bounded Luna n=1 dry-load config targeting the Prime VM runtime with pinned harness artifact URL/SHA.
Reviewed by Cursor Bugbot for commit 141d48a. Bugbot is set up for automated code reviews on this repo. Configure here.
Note
Add oolong-synth-v1 taskset environment with deterministic and judge-based scoring
oolong_synth_v1environment package withOolongSynthTaskset, which loads tasks from theoolongbench/oolong-synthdataset filtered by context length bucket.taskset.py: deterministic partial-credit scoring (exact match, exponential decay for numerics, exact date equality) and optional LLM judge-based binary scoring viaOolongJudge./workspace/context.txtin the agent sandbox; the agent writes its answer to/workspace/answer.txtor falls back to its last message.configs/oolong_synth_v1_luna_n1.tomlthat runs one task/rollout withopenai/gpt-5.6-lunaat the 1K context bucket using the Prime VM agent harness.Macroscope summarized 141d48a.