Skip to content

Skip the Hugging Face token prompt and no-token warning when huggingface_hub already has a cached token #553

Description

@MaxGhenis

Problem

download_huggingface_dataset resolves its token only from HUGGING_FACE_TOKEN (or an interactive prompt) through get_or_prompt_hf_token(). huggingface_hub keeps its own token too: huggingface_hub.get_token() returns, in order, a Colab secret, HF_TOKEN / HUGGING_FACE_HUB_TOKEN, or the file written by hf auth login (utils/_auth.py:49 and :118 in huggingface_hub 1.4.1). When core passes token=None, get_token_to_send falls back to that cached token unless HF_HUB_DISABLE_IMPLICIT_TOKEN is set.

Two consequences, both raised as non-blocking findings by the independent reviews of #538 and #540:

  1. Prompt (since Send HUGGING_FACE_TOKEN for public but gated Hugging Face repos #538). On a TTY with HUGGING_FACE_TOKEN unset, a developer who already ran hf auth login or set HF_TOKEN is now prompted for a token when downloading a gated repo, as they always were for private repos. Pressing Enter works (core passes token=None and the cached token is used), but the prompt is unnecessary.
  2. Warning (Warn when a restricted Hugging Face download resolves no HUGGING_FACE_TOKEN #540). When no HUGGING_FACE_TOKEN resolves, core warns that the download may 401 even if huggingface_hub holds a cached token and the download succeeds.

Proposal

Before prompting or warning, check whether huggingface_hub will send a token anyway:

from huggingface_hub import get_token, constants

implicit_token_available = (
    get_token() is not None and not constants.HF_HUB_DISABLE_IMPLICIT_TOKEN
)

The #540 reviewer checked that get_token and constants.HF_HUB_DISABLE_IMPLICIT_TOKEN exist from 0.25.1, the current huggingface_hub floor, so this works across the supported range.

Tests to update: the hf-token-only cases in tests/core/tools/test_hugging_face.py currently pin the prompt and the warning on purpose; they would instead assert no prompt and no warning when get_token() returns a token, and keep asserting the old behaviour when it returns None or implicit tokens are disabled.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions