Skip to content

Python: [Bug]: GAIA runner refuses local parquet data and asks for HF_TOKEN #9337

Description

Observed Behavior

_load_gaia_local prefers parquet files (comment calls it the new format, validation split first) but GAIA._ensure_data only treats the data dir as ready when it contains metadata.jsonl. So a data_dir with only parquet files is ignored and the runner demands HF_TOKEN and tries a fresh download even though the local data is fully usable.

Expected Behavior

_ensure_data should accept the parquet layout the loader itself supports.

Steps to Reproduce

  1. Create a data dir with 2023/validation/metadata-00000-of-00001.parquet holding two task rows.
  2. _load_gaia_local(dir) returns both tasks.
  3. GAIA(data_dir=dir)._ensure_data() raises RuntimeError asking for HF_TOKEN.

Minimal Reproduction

import tempfile
from pathlib import Path
import pyarrow as pa
import pyarrow.parquet as pq
from agent_framework_lab_gaia.gaia import _load_gaia_local, GAIA

tmp = Path(tempfile.mkdtemp())
val = tmp / "2023" / "validation"
val.mkdir(parents=True)
rows = [
    {"task_id": "v1", "Question": "What is 2+2?", "Final answer": "4", "Level": 1, "file_name": "", "Annotator Metadata": "{}"},
    {"task_id": "v2", "Question": "Capital of France?", "Final answer": "Paris", "Level": 1, "file_name": "", "Annotator Metadata": "{}"},
]
pq.write_table(pa.Table.from_pylist(rows), val / "metadata-00000-of-00001.parquet")

assert len(_load_gaia_local(tmp)) == 2   # loader is fine with parquet
GAIA(data_dir=str(tmp))._ensure_data()   # RuntimeError: HF_TOKEN ... required

Error Messages and Stack Traces

RuntimeError: HF_TOKEN environment variable or hf_token parameter is required to access the GAIA dataset. Please set your Hugging Face token with access to gaia-benchmark/GAIA.

Package Versions

agent-framework-lab built from main (dcbbf79)

Python Version

Python 3.14.8

Operating System

macOS

Regression

Unknown

Additional Context

Looks like _ensure_data was written for the old jsonl layout and never updated when parquet loading was added. Accepting metadata*.parquet in the same existence check would fix it. Happy to send a PR.

Acknowledgements

  • I searched existing issues and did not find a duplicate.

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Oct 13, 2026
  2. ManoharPaturi commented on Oct 22, 2026

    @ManoharPaturi
    ContributorAuthor

    The fix for this issue is resubmitted as #9609 (replacing the closed #9340, which was closed with a request to address the outstanding review feedback: _ensure_data() accepted a directory whenever any metadata.jsonl existed, without checking incomplete parquet shards in the same directory). The new PR contains the original parquet-only dir fix plus that feedback: any mixed layout with metadata parquet present now requires the complete parquet snapshot, with a regression test for the mixed layout.

  3. eavanvalkenburg commented on Nov 3, 2026

    @eavanvalkenburg
    Member

    /plan Plan the smallest complete supported-local-parquet readiness repair for #9337 through current PR #9609, not competing implementation. Main _ensure_data accepts only metadata.jsonl before demanding HF_TOKEN, although _load_gaia_local supports metadata*.parquet. Prior #9340 review already requires a complete selected parquet snapshot or an explicit safe fallback; #9609 addresses the old JSONL/unsharded shortcuts, so do not repeat those as still unfixed. Check remaining exact shard-index/declared-total validation: source-extracted current-head filename predicates accept shards 1 and 2 of 2 with missing shard 0, and a group containing 0-of-2 plus 1-of-3 and 2-of-3. This was an empty-filename predicate characterization at head 15e7f0b, not a pyarrow/GAIA integration run. Correct an existing description mismatch: the actual loader falls back to JSONL when no tasks load, not only when no parquet files exist; partial valid parquet can suppress that fallback. Preserve valid parquet-only and JSONL-only offline behavior, reconcile mixed layouts/recognized shard families/unsharded files and level or split filtering without an unrelated loader redesign. Require actual parquet/load regression coverage, exact index bounds/coherent totals, incomplete and mixed controls, no accidental token/network access in tests, and failed-download behavior. Estimate executable production-code scope excluding docs/tests and agree any compatibility boundary before calling this minor/implementation-ready or assigning; current completeness/readiness scope is not confidently below the threshold.

  4. eavanvalkenburg commented on Nov 12, 2026

    @eavanvalkenburg
    Member

    /plan Plan the smallest complete supported-local-parquet readiness repair for #9337 through current PR #9609, not competing implementation. Main _ensure_data accepts only metadata.jsonl before demanding HF_TOKEN, although _load_gaia_local supports metadata*.parquet. Prior #9340 review already requires a complete selected parquet snapshot or an explicit safe fallback; #9609 addresses the old JSONL/unsharded shortcuts, so do not repeat those as still unfixed. Check remaining exact shard-index/declared-total validation: source-extracted current-head filename predicates accept shards 1 and 2 of 2 with missing shard 0, and a group containing 0-of-2 plus 1-of-3 and 2-of-3. This was an empty-filename predicate characterization at head 15e7f0b, not a pyarrow/GAIA integration run. Correct an existing description mismatch: the actual loader falls back to JSONL when no tasks load, not only when no parquet files exist; partial valid parquet can suppress that fallback. Preserve valid parquet-only and JSONL-only offline behavior, reconcile mixed layouts/recognized shard families/unsharded files and level or split filtering without an unrelated loader redesign. Require actual parquet/load regression coverage, exact index bounds/coherent totals, incomplete and mixed controls, no accidental token/network access in tests, and failed-download behavior. Estimate executable production-code scope excluding docs/tests and agree any compatibility boundary before calling this minor/implementation-ready or assigning; current completeness/readiness scope is not confidently below the threshold.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

pythonUsage: [Issues, PRs], Target: Pythonready-for-implementationMaintainers agree the problem and direction are ready for implementation.

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions