Repository navigation
Open
Copy link
Labels
pythonUsage: [Issues, PRs], Target: PythonUsage: [Issues, PRs], Target: Pythonready-for-implementationMaintainers agree the problem and direction are ready for implementation.Maintainers agree the problem and direction are ready for implementation.
Description
Observed Behavior
_load_gaia_local prefers parquet files (comment calls it the new format, validation split first) but GAIA._ensure_data only treats the data dir as ready when it contains metadata.jsonl. So a data_dir with only parquet files is ignored and the runner demands HF_TOKEN and tries a fresh download even though the local data is fully usable.
Expected Behavior
_ensure_data should accept the parquet layout the loader itself supports.
Steps to Reproduce
- Create a data dir with
2023/validation/metadata-00000-of-00001.parquetholding two task rows. _load_gaia_local(dir)returns both tasks.GAIA(data_dir=dir)._ensure_data()raises RuntimeError asking for HF_TOKEN.
Minimal Reproduction
import tempfile
from pathlib import Path
import pyarrow as pa
import pyarrow.parquet as pq
from agent_framework_lab_gaia.gaia import _load_gaia_local, GAIA
tmp = Path(tempfile.mkdtemp())
val = tmp / "2023" / "validation"
val.mkdir(parents=True)
rows = [
{"task_id": "v1", "Question": "What is 2+2?", "Final answer": "4", "Level": 1, "file_name": "", "Annotator Metadata": "{}"},
{"task_id": "v2", "Question": "Capital of France?", "Final answer": "Paris", "Level": 1, "file_name": "", "Annotator Metadata": "{}"},
]
pq.write_table(pa.Table.from_pylist(rows), val / "metadata-00000-of-00001.parquet")
assert len(_load_gaia_local(tmp)) == 2 # loader is fine with parquet
GAIA(data_dir=str(tmp))._ensure_data() # RuntimeError: HF_TOKEN ... requiredError Messages and Stack Traces
RuntimeError: HF_TOKEN environment variable or hf_token parameter is required to access the GAIA dataset. Please set your Hugging Face token with access to gaia-benchmark/GAIA.
Package Versions
agent-framework-lab built from main (dcbbf79)
Python Version
Python 3.14.8
Operating System
macOS
Regression
Unknown
Additional Context
Looks like _ensure_data was written for the old jsonl layout and never updated when parquet loading was added. Accepting metadata*.parquet in the same existence check would fix it. Happy to send a PR.
Acknowledgements
- I searched existing issues and did not find a duplicate.
Metadata
Metadata
Labels
pythonUsage: [Issues, PRs], Target: PythonUsage: [Issues, PRs], Target: Pythonready-for-implementationMaintainers agree the problem and direction are ready for implementation.Maintainers agree the problem and direction are ready for implementation.
Type
Projects
- StatusShow more project fieldsNo status
Activity
ManoharPaturi commentedon Oct 22, 2026
The fix for this issue is resubmitted as #9609 (replacing the closed #9340, which was closed with a request to address the outstanding review feedback:
_ensure_data()accepted a directory whenever anymetadata.jsonlexisted, without checking incomplete parquet shards in the same directory). The new PR contains the original parquet-only dir fix plus that feedback: any mixed layout with metadata parquet present now requires the complete parquet snapshot, with a regression test for the mixed layout.eavanvalkenburg commentedon Nov 3, 2026
/plan Plan the smallest complete supported-local-parquet readiness repair for #9337 through current PR #9609, not competing implementation. Main _ensure_data accepts only metadata.jsonl before demanding HF_TOKEN, although _load_gaia_local supports metadata*.parquet. Prior #9340 review already requires a complete selected parquet snapshot or an explicit safe fallback; #9609 addresses the old JSONL/unsharded shortcuts, so do not repeat those as still unfixed. Check remaining exact shard-index/declared-total validation: source-extracted current-head filename predicates accept shards 1 and 2 of 2 with missing shard 0, and a group containing 0-of-2 plus 1-of-3 and 2-of-3. This was an empty-filename predicate characterization at head 15e7f0b, not a pyarrow/GAIA integration run. Correct an existing description mismatch: the actual loader falls back to JSONL when no tasks load, not only when no parquet files exist; partial valid parquet can suppress that fallback. Preserve valid parquet-only and JSONL-only offline behavior, reconcile mixed layouts/recognized shard families/unsharded files and level or split filtering without an unrelated loader redesign. Require actual parquet/load regression coverage, exact index bounds/coherent totals, incomplete and mixed controls, no accidental token/network access in tests, and failed-download behavior. Estimate executable production-code scope excluding docs/tests and agree any compatibility boundary before calling this minor/implementation-ready or assigning; current completeness/readiness scope is not confidently below the threshold.
eavanvalkenburg commentedon Nov 12, 2026
/plan Plan the smallest complete supported-local-parquet readiness repair for #9337 through current PR #9609, not competing implementation. Main _ensure_data accepts only metadata.jsonl before demanding HF_TOKEN, although _load_gaia_local supports metadata*.parquet. Prior #9340 review already requires a complete selected parquet snapshot or an explicit safe fallback; #9609 addresses the old JSONL/unsharded shortcuts, so do not repeat those as still unfixed. Check remaining exact shard-index/declared-total validation: source-extracted current-head filename predicates accept shards 1 and 2 of 2 with missing shard 0, and a group containing 0-of-2 plus 1-of-3 and 2-of-3. This was an empty-filename predicate characterization at head 15e7f0b, not a pyarrow/GAIA integration run. Correct an existing description mismatch: the actual loader falls back to JSONL when no tasks load, not only when no parquet files exist; partial valid parquet can suppress that fallback. Preserve valid parquet-only and JSONL-only offline behavior, reconcile mixed layouts/recognized shard families/unsharded files and level or split filtering without an unrelated loader redesign. Require actual parquet/load regression coverage, exact index bounds/coherent totals, incomplete and mixed controls, no accidental token/network access in tests, and failed-download behavior. Estimate executable production-code scope excluding docs/tests and agree any compatibility boundary before calling this minor/implementation-ready or assigning; current completeness/readiness scope is not confidently below the threshold.