Send HUGGING_FACE_TOKEN for public but gated Hugging Face repos by MaxGhenis · Pull Request #538 · PolicyEngine/policyengine-core

MaxGhenis · 2026-09-03T18:27:28Z

Fixes #529.

Summary

download_huggingface_dataset decided whether to send a token by testing ModelInfo.private only. policyengine/policyengine-uk-data-private has been public but gated (private=False, gated="manual") since 31 July 2026, so the helper passed token=None to hf_hub_download and the gate returned 401 GatedRepoError even when a gate-approved HUGGING_FACE_TOKEN was in the environment. The Hub only falls back to its own HF_TOKEN variable when no explicit token is passed, which is why PolicyEngine/policyengine-uk#1817 had to export both names as a workaround (see the comment in .github/workflows/code_changes.yaml and pr_code_changes.yaml there).

Behaviour change

A repo now requires authentication when private is true or gated is truthy. ModelInfo.gated is False for ungated repos, "auto" or "manual" for gated repos, and None when the field is absent, so a truthiness test covers every state. In that case the existing get_or_prompt_hf_token() supplies the token from HUGGING_FACE_TOKEN (or an interactive prompt on a TTY).

Unchanged:

Public, ungated repos still download with token=None and never prompt.
In a non-interactive environment without secrets, the token is still None rather than an empty string (the Handle empty HUGGING_FACE_TOKEN gracefully #422 behaviour), so Dependabot-style CI keeps working for public repos.
The RepositoryNotFoundError path still treats the repo as private.
On a TTY with HUGGING_FACE_TOKEN unset, a gated repo now prompts for a token exactly as a private repo always has. Pressing Enter passes token=None, and huggingface_hub then applies its own implicit token (HF_TOKEN or the login file) if one is configured.

Verified live: GET /api/models/policyengine/policyengine-uk-data-private returns private: false, gated: "manual", and an unauthenticated HEAD on a resolve URL returns 401.

Tests

New tests in tests/core/tools/test_hugging_face.py:

test_download_gated_public_repo_passes_env_token[manual|auto] mocks model_info returning private=False, gated=... and asserts the token read from HUGGING_FACE_TOKEN by the real get_or_prompt_hf_token reaches hf_hub_download. Both fail on master and pass here.
test_download_private_flag_repo_passes_env_token covers private=True returned by model_info (the existing private test only covers the 404 path).
test_download_gated_repo_non_interactive_without_token asserts CI without secrets gets token=None and no prompt.
test_download_public_ungated_repo_never_prompts[False|None] guards against widening the predicate into a prompt on every public download.

Checks on the rebased head (493b930, on master a0ccbda):

Rebased onto master after Document Hugging Face dataset download helper #545 rewrote the same lines. The resolution keeps Document Hugging Face dataset download helper #545's docstring (Args:/Returns:), type hints and from pathlib import Path, and Fix vacuous test for private Hugging Face repo download without a token #539's parametrised test_download_private_repo_no_token byte-for-byte; the only removed lines are the four is_repo_private lines this PR renames to requires_authentication.
uv run --frozen pytest tests/core/tools/test_hugging_face.py -q: 23 passed.
With master's hugging_face.py swapped in, the two test_download_gated_public_repo_passes_env_token cases fail and the other 21 pass.
uvx ruff format --check . and uvx ruff check on the changed files pass.
CI run 36566824816 on this head: all 18 checks pass (Linux and Windows, py3.11 to 3.14, core and country packages).

Invariants (checked by an independent reviewer over a 168-case grid of private × gated × environment × TTY, 0 violations):

The token passed to hf_hub_download is either None or a non-empty string, never "".
A repo that is neither private nor gated always gets token=None and never prompts.
A private or gated repo with a non-empty HUGGING_FACE_TOKEN always gets that token, without a prompt.

Not run locally: mypy and the full suite (CI covers them), and make documentation (no docs changed).

axiom: n/a: infrastructure (Hugging Face download authentication), no policy change.

Issue: download_huggingface_dataset sends no token for public+gated repos (private is the wrong predicate) #529
Incident in policyengine-uk: CI HUGGING_FACE_TOKEN failing with 401 on private dataset downloads policyengine-uk#1816
Workaround that this makes unnecessary: Send HF_TOKEN so gated dataset downloads authenticate policyengine-uk#1817
Earlier token handling: Handle empty HUGGING_FACE_TOKEN gracefully #422

🤖 Generated with Claude Code

download_huggingface_dataset only authenticated when ModelInfo.private was true. A public but gated repo reports private=False with gated="auto" or "manual", so the helper passed token=None and the Hub answered 401 unless the caller happened to export HF_TOKEN. Treat a repo as requiring authentication when private is true or gated is truthy, and pass the HUGGING_FACE_TOKEN through the existing get_or_prompt_hf_token() path. Public, ungated repos still download with token=None and never prompt, and non-interactive runs without secrets still pass None rather than an empty string. Fixes #529. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

With #538, a public but gated repo (private=False, gated="manual", as policyengine/policyengine-uk-data-private is) also requires authentication, so it takes the same path that now warns when no HUGGING_FACE_TOKEN resolves. Add a "gated" lookup to TestNoTokenWarning: the three non-interactive no-token environments must warn, and an environment token must reach hf_hub_download without a warning. With the private-only predicate restored, the four new gated cases fail; with the warning removed, all ten warning-asserting cases fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

MaxGhenis · 2026-09-29T16:48:09Z

Merge audit (d051, Max ruled 2026-09-29 to land this fix).

Head reviewed and merged: 493b930, rebased onto master a0ccbda. Conflict with Document Hugging Face dataset download helper #545 resolved by keeping Document Hugging Face dataset download helper #545's docstring and type hints plus this PR's requires_authentication predicate; Fix vacuous test for private Hugging Face repo download without a token #539's parametrised test is unchanged.
Independent review: Opus 5.5 via subfleet run --task review --tier standard (job 20260929-114521-review-core-538-v3), VERDICT: APPROVE with no blocking findings. It ran the test file (23 passed), five mutants (master's predicate, always-auth, private dropped, not-found branch off, empty-string token; each caught), a 168-case grid over private × gated × env × TTY (0 violations), and ruff.
Non-blocking notes carried forward: the getpass mock in test_download_public_ungated_repo_never_prompts gets a return_value in Warn when a restricted Hugging Face download resolves no HUGGING_FACE_TOKEN #540's rebase, and "skip the prompt when huggingface_hub already has a cached token" is tracked in Skip the Hugging Face token prompt and no-token warning when huggingface_hub already has a cached token #553.
gh pr checks 538 exit 0 on run 36566824816 (18/18), MERGEABLE, not a draft, no change requests. Squash merge, matching recent history (Fix variable uprating when the uprating parameter is undefined, and pick the latest earlier period #551, Fix vacuous test for private Hugging Face repo download without a token #539).

download_huggingface_dataset keeps passing token=None through to hf_hub_download when a repo that requires authentication (private, or gated since #538) yields no HUGGING_FACE_TOKEN (per #422: huggingface_hub then applies its own cached token, so `hf auth login` users keep working), but now emits a UserWarning first so that the bare 401 huggingface_hub raises when that fallback is empty too can be traced to the missing or unapproved token (#529). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

With #538, a public but gated repo (private=False, gated="manual", as policyengine/policyengine-uk-data-private is) also requires authentication, so it takes the same path that now warns when no HUGGING_FACE_TOKEN resolves. Add a "gated" lookup to TestNoTokenWarning: the three non-interactive no-token environments must warn, and an environment token must reach hf_hub_download without a warning. With the private-only predicate restored, the four new gated cases fail; with the warning removed, all ten warning-asserting cases fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…vely Fold in the non-blocking findings from the independent review of #540: - Say a 401 means no token was sent and a 403 means a gated repo has not approved the token; huggingface_hub raises GatedRepoError for both and documents the unapproved case as 403. - Say the fallback to huggingface_hub's cached token is what normally happens: HF_HUB_DISABLE_IMPLICIT_TOKEN turns it off. - Name `huggingface-cli login` as well as `hf auth login`: the `hf` command first ships in huggingface_hub 0.34, and pyproject still allows 0.25.1. - Assert the warning in the two older no-token tests that now emit it, and give the ungated never-prompts test's getpass mock a string return value so a widened predicate fails on the assertion, not on os.environ. - Reword the changelog fragment: after #538 the repo may be private or gated, and a cached token that is not approved gives a 403. Add TestTokenRoutingInvariants, which runs every combination of repo state (ungated, gated None, fields missing, gated auto, gated manual, private, not found), environment (token unset, empty, only HF_TOKEN, set), TTY and prompt entry (112 cases) through the real function and checks it against a spec written out in the test: the token passed on, whether getpass is called, that the token is never "", and that exactly one warning fires when a repo requiring authentication ends up with no token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

download_huggingface_dataset keeps passing token=None through to hf_hub_download when a repo that requires authentication (private, or gated since #538) yields no HUGGING_FACE_TOKEN (per #422: huggingface_hub then applies its own cached token, so `hf auth login` users keep working), but now emits a UserWarning first so that the bare 401 huggingface_hub raises when that fallback is empty too can be traced to the missing or unapproved token (#529). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

With #538, a public but gated repo (private=False, gated="manual", as policyengine/policyengine-uk-data-private is) also requires authentication, so it takes the same path that now warns when no HUGGING_FACE_TOKEN resolves. Add a "gated" lookup to TestNoTokenWarning: the three non-interactive no-token environments must warn, and an environment token must reach hf_hub_download without a warning. With the private-only predicate restored, the four new gated cases fail; with the warning removed, all ten warning-asserting cases fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…vely Fold in the non-blocking findings from the independent review of #540: - Say a 401 means no token was sent and a 403 means a gated repo has not approved the token; huggingface_hub raises GatedRepoError for both and documents the unapproved case as 403. - Say the fallback to huggingface_hub's cached token is what normally happens: HF_HUB_DISABLE_IMPLICIT_TOKEN turns it off. - Name `huggingface-cli login` as well as `hf auth login`: the `hf` command first ships in huggingface_hub 0.34, and pyproject still allows 0.25.1. - Assert the warning in the two older no-token tests that now emit it, and give the ungated never-prompts test's getpass mock a string return value so a widened predicate fails on the assertion, not on os.environ. - Reword the changelog fragment: after #538 the repo may be private or gated, and a cached token that is not approved gives a 403. Add TestTokenRoutingInvariants, which runs every combination of repo state (ungated, gated None, fields missing, gated auto, gated manual, private, not found), environment (token unset, empty, only HF_TOKEN, set), TTY and prompt entry (112 cases) through the real function and checks it against a spec written out in the test: the token passed on, whether getpass is called, that the token is never "", and that exactly one warning fires when a repo requiring authentication ends up with no token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

…_TOKEN (#540) * Warn when a restricted Hugging Face download resolves no token download_huggingface_dataset keeps passing token=None through to hf_hub_download when a repo that requires authentication (private, or gated since #538) yields no HUGGING_FACE_TOKEN (per #422: huggingface_hub then applies its own cached token, so `hf auth login` users keep working), but now emits a UserWarning first so that the bare 401 huggingface_hub raises when that fallback is empty too can be traced to the missing or unapproved token (#529). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * Cover gated repos in the no-token warning tests With #538, a public but gated repo (private=False, gated="manual", as policyengine/policyengine-uk-data-private is) also requires authentication, so it takes the same path that now warns when no HUGGING_FACE_TOKEN resolves. Add a "gated" lookup to TestNoTokenWarning: the three non-interactive no-token environments must warn, and an environment token must reach hf_hub_download without a warning. With the private-only predicate restored, the four new gated cases fail; with the warning removed, all ten warning-asserting cases fail. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Tighten the no-token warning text and pin the token contract exhaustively Fold in the non-blocking findings from the independent review of #540: - Say a 401 means no token was sent and a 403 means a gated repo has not approved the token; huggingface_hub raises GatedRepoError for both and documents the unapproved case as 403. - Say the fallback to huggingface_hub's cached token is what normally happens: HF_HUB_DISABLE_IMPLICIT_TOKEN turns it off. - Name `huggingface-cli login` as well as `hf auth login`: the `hf` command first ships in huggingface_hub 0.34, and pyproject still allows 0.25.1. - Assert the warning in the two older no-token tests that now emit it, and give the ungated never-prompts test's getpass mock a string return value so a widened predicate fails on the assertion, not on os.environ. - Reword the changelog fragment: after #538 the repo may be private or gated, and a cached token that is not approved gives a 403. Add TestTokenRoutingInvariants, which runs every combination of repo state (ungated, gated None, fields missing, gated auto, gated manual, private, not found), environment (token unset, empty, only HF_TOKEN, set), TTY and prompt entry (112 cases) through the real function and checks it against a spec written out in the test: the token passed on, whether getpass is called, that the token is never "", and that exactly one warning fires when a repo requiring authentication ends up with no token. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Cover private-and-gated repos and count prompts in the token grid Add a "private-gated" repo state (private=True, gated="manual") to TestTokenRoutingInvariants, so a predicate that is true for private or gated but false for both (for example an xor) now fails 16 cases instead of passing all of them. Assert getpass's call_count rather than whether it was called, so a double prompt fails 30 grid cases, not one side test. The grid grows from 112 to 128 cases. Also give the Warns: docstring the same `huggingface-cli login` caveat the runtime warning already carries. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin the 403, GatedRepoError and huggingface-cli parts of the warning text The non-interactive warning test now also asserts that the message names RepositoryNotFoundError, GatedRepoError, the 403 case and the pre-0.34 `huggingface-cli login` command. Each of those clauses could previously be deleted without failing a test; each deletion now fails 9 of 167. The TestNoTokenWarning docstring no longer calls the 401 "missing or unapproved": with an empty fallback there is no token to be unapproved. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

This was referenced Sep 3, 2026

Fix vacuous test for private Hugging Face repo download without a token #539

Merged

Warn when a restricted Hugging Face download resolves no HUGGING_FACE_TOKEN #540

Merged

MaxGhenis mentioned this pull request Sep 15, 2026

download_huggingface_dataset hardcodes repo_type="model", so it cannot fetch datasets repos #549

Open

MaxGhenis force-pushed the fix/hf-gated-repo-token branch from fe14370 to 493b930 Compare September 29, 2026 12:14

MaxGhenis merged commit 881173b into master Sep 29, 2026
18 checks passed

MaxGhenis deleted the fix/hf-gated-repo-token branch September 29, 2026 16:46

MaxGhenis mentioned this pull request Sep 29, 2026

Skip the Hugging Face token prompt and no-token warning when huggingface_hub already has a cached token #553

Open

MaxGhenis mentioned this pull request Sep 29, 2026

Require policyengine-core 3.32.9 and stop exporting HF_TOKEN in CI PolicyEngine/policyengine-uk#1898

Merged

Navigation Menu

Send HUGGING_FACE_TOKEN for public but gated Hugging Face repos - #538

Send HUGGING_FACE_TOKEN for public but gated Hugging Face repos#538
MaxGhenis merged 1 commit into
masterfrom
fix/hf-gated-repo-token

MaxGhenis commented Sep 3, 2026 •

edited

Loading

Uh oh!

Uh oh!

MaxGhenis commented Sep 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

MaxGhenis commented Sep 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Behaviour change

Tests

Related

Uh oh!

Uh oh!

MaxGhenis commented Sep 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

MaxGhenis commented Sep 3, 2026 •

edited

Loading