Repository navigation
Python: fix(lab-gaia): accept local parquet data in _ensure_data - #9609
Manohar Paturi (ManoharPaturi) wants to merge 9 commits into
Conversation
_ensure_data only recognized the old metadata.jsonl layout, so a data dir holding only the parquet files the loader itself prefers was ignored and the runner demanded HF_TOKEN for a fresh download. Accept metadata*.parquet in the same existence check. Fixes microsoft#9337 Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
…ing local data A partial metadata-XXXXX-of-NNNNN family from an interrupted copy would have counted as ready and run the benchmark over a truncated task set. Group shards by directory and prefix and only treat the snapshot as complete when each group holds all its declared shards. Follow-up to the review on microsoft#9340. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
… fix/gaia-parquet-ensure-data
An unsharded metadata parquet made the completeness check return early, so a directory mixing a plain file with an incomplete sharded family was treated as ready. Keep scanning after unsharded files and let any incomplete family disqualify the directory. Also clear HF_TOKEN in the incomplete snapshot test so it cannot attempt a real download. Follow-up to the review on microsoft#9340. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
… is present Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
|
The The job crashes inside griffe while loading the base (main) snapshot, before it compares the PR changes — the traceback points at the base copy of the package: Example run: https://github.com/microsoft/agent-framework/actions/runs/39810016173/job/119933441343 None of the changes in this PR touch |
| return unsharded_seen | ||
| # The loader reads every matching parquet file, so a single incomplete sharded | ||
| # family disqualifies the directory even when unsharded files also exist. | ||
| return all(len(indexes) == total for indexes, total in sharded.values()) |
There was a problem hiding this comment.
Manohar Paturi (@ManoharPaturi), this checks only the number of observed shard indexes against the largest declared total. A family missing shard 0, a family with mixed of-N totals, or even an index at or above the declared total can still pass and run GAIA on a truncated/incoherent task set. Please require one coherent total and indexes == set(range(total)), with regressions for missing shard 0, mixed totals, and an out-of-range index.
Counting observed indexes against the largest declared total passed a family missing shard 0, a family whose files declared mixed totals, and a family holding an index at or beyond the declared total, so the benchmark could run over a truncated task set. A family now counts as complete only when every file declares the same total and the indexes are exactly zero through that total minus one. Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
|
All three holes are closed in ea0838c, each reproduced on the previous head first. A family now counts as complete only when every file declares the same total and the indexes are exactly zero through that total minus one. Counting observed indexes against the largest declared total was the whole gap: the missing shard 0 regression seeds indexes one and two against a declared total of two, the mixed totals regression seeds three distinct indexes of three plus one file declaring two, which the old count against the max passes, and the out of range regression seeds indexes zero and two against two, where a garbage high index stood in for the missing shard one. All three fail on the previous head by accepting the directory and only fail to download when the token check runs, and pass now by forcing the download path. The gaia suite passes at 12 tests and ruff is clean. |
|
/review |
There was a problem hiding this comment.
MAF Automated Review — Iteration 1
Result: Findings reported
Scope: full PR (8 commit(s)): ddd62b7f05d2, a367690bddab, 47891ef94bfe, d172e602bca8, a16b33d15378, 60b4e0aa9752, 15e7f0bad102, ea0838c64e5f
Model: gpt-5.6-sol
Overview
The change aligns local-data detection with the loader's parquet-first behavior and adds strong filename-level guards for missing indexes, mixed totals, and incomplete families. The new tests preserve JSONL-only compatibility and cover the previously reported shard-index cases. Two gaps remain: complete-looking corrupt shards can still yield partial runs, and the shard parser does not cover every parquet filename accepted by the loader.
Reviewed the supplied pull-request change set across correctness, security/reliability, architecture, and failure behavior.
2 verified findings remained after source verification (2 medium) across 1 file. Details are attached to the affected lines below.
Affected areas: python/packages/lab/gaia/agent_framework_lab_gaia/gaia.py
| indexes = {index for index, _ in shards} | ||
| if indexes != set(range(totals.pop())): | ||
| return False | ||
| return True |
There was a problem hiding this comment.
Completeness here is filename-only. If all declared files exist but one shard is truncated or corrupt, this returns true; _load_gaia_local then catches that shard's read failure and runs with tasks from the remaining shards, silently producing a partial benchmark. Please validate that every accepted shard is readable before reporting the snapshot as usable, and fail the family if any shard cannot be loaded.
| yield record | ||
|
|
||
|
|
||
| _SHARDED_PARQUET_RE = re.compile(r"(metadata(?:-[^-]+)?)-(\d{5})-of-(\d{5})\.parquet") |
There was a problem hiding this comment.
This only recognizes metadata or one hyphen-delimited prefix component before the shard suffix. A family such as metadata-2023-validation-00000-of-00002.parquet still matches the loader's metadata*.parquet glob but misses this regex, is classified as unsharded, and is accepted with shard 1 absent. Please parse the -NNNNN-of-NNNNN suffix independently of the metadata prefix so every loader-visible sharded family is checked.
|
Manohar Paturi (@ManoharPaturi), this pull request is waiting on your response. If there is no further activity within 4 days, it will be closed automatically. (This is an automated message.) |
|
Both findings are addressed in 69cc456, each reproduced on the previous head first. The shard suffix now decides the family, whatever the prefix before it is. The regex takes any name through the -NNNNN-of-NNNNN suffix, so metadata-2023-validation-00000-of-00002.parquet counts as sharded exactly like the plain spelling instead of slipping past as an unsharded file; the regression seeds that family with shard 0 of 2 only and asserts the download path runs, then completes the family and asserts the directory becomes ready again. A complete looking name is no longer enough. After the totals and index set checks pass, every declared shard is opened through pyarrow's footer reader, and a shard that fails to read makes its family incomplete: the loader would otherwise skip the failing file and quietly run the benchmark over the remaining shards. Without pyarrow installed there is nothing to validate and nothing the loader could read either, so the check degrades to the name alone there. The regression writes a valid shard 0 of 2 beside a corrupt shard 1 and asserts the download path runs. Both regressions fail on the previous head. The gaia suite passes at 14 tests and ruff check and format are clean. |
Motivation & Context
_load_gaia_localprefers parquet files (the comment calls it the new format, validation split first) butGAIA._ensure_dataonly treated a data dir as ready when it containedmetadata.jsonl. Pointingdata_dirat a parquet-only mirror was ignored and the runner demanded HF_TOKEN for a fresh download even though the local data was fully usable. Issue #9337 has the repro.This PR replaces #9340, which was closed while it still had outstanding review feedback:
_ensure_data()still returned successfully whenever anymetadata.jsonlexisted, without checking incomplete parquet shards in the same directory. Per the close reason, it is being resubmitted here with that feedback addressed on top of the original fix.Description & Review Guide
_ensure_datanow accepts a data dir through_has_usable_local_data(): if anymetadata*.parquetfile is present (the same glob the loader itself reads), the parquet snapshot must be complete; otherwise the dir is usable whenmetadata.jsonlis present. A JSONL-only layout keeps working exactly as before.metadata-XXXXX-of-NNNNN.parquetfamily) no longer silently passes. The loader reads parquet first and only falls back to JSONL when no parquet file is present, so such a mixed dir would have loaded a truncated task set without any error — this is the review feedback from Python: fix(lab-gaia): accept local parquet data in _ensure_data #9340, addressed by requiring a complete parquet snapshot whenever any metadata parquet file is present._has_usable_local_datahelper ingaia.py, plus the two new regression tests:test_jsonl_does_not_rescue_an_incomplete_parquet_family(mixed layout with shard 0 of 2 forces download; completing the family makes the dir ready again) andtest_jsonl_only_layout_is_still_accepted.The full GAIA suite (9 tests), Ruff (pinned 0.16.10) check/format, pyright, and mypy are clean locally.
Related Issue
Fixes #9337
AI Assistance
Contribution Checklist