You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
download_huggingface_dataset resolves its token only from HUGGING_FACE_TOKEN (or an interactive prompt) through get_or_prompt_hf_token(). huggingface_hub keeps its own token too: huggingface_hub.get_token() returns, in order, a Colab secret, HF_TOKEN / HUGGING_FACE_HUB_TOKEN, or the file written by hf auth login (utils/_auth.py:49 and :118 in huggingface_hub 1.4.1). When core passes token=None, get_token_to_send falls back to that cached token unless HF_HUB_DISABLE_IMPLICIT_TOKEN is set.
Two consequences, both raised as non-blocking findings by the independent reviews of #538 and #540:
Prompt (since Send HUGGING_FACE_TOKEN for public but gated Hugging Face repos #538). On a TTY with HUGGING_FACE_TOKEN unset, a developer who already ran hf auth login or set HF_TOKEN is now prompted for a token when downloading a gated repo, as they always were for private repos. Pressing Enter works (core passes token=None and the cached token is used), but the prompt is unnecessary.
The #540 reviewer checked that get_token and constants.HF_HUB_DISABLE_IMPLICIT_TOKEN exist from 0.25.1, the current huggingface_hub floor, so this works across the supported range.
Tests to update: the hf-token-only cases in tests/core/tools/test_hugging_face.py currently pin the prompt and the warning on purpose; they would instead assert no prompt and no warning when get_token() returns a token, and keep asserting the old behaviour when it returns None or implicit tokens are disabled.
Problem
download_huggingface_datasetresolves its token only fromHUGGING_FACE_TOKEN(or an interactive prompt) throughget_or_prompt_hf_token(). huggingface_hub keeps its own token too:huggingface_hub.get_token()returns, in order, a Colab secret,HF_TOKEN/HUGGING_FACE_HUB_TOKEN, or the file written byhf auth login(utils/_auth.py:49and:118in huggingface_hub 1.4.1). When core passestoken=None,get_token_to_sendfalls back to that cached token unlessHF_HUB_DISABLE_IMPLICIT_TOKENis set.Two consequences, both raised as non-blocking findings by the independent reviews of #538 and #540:
HUGGING_FACE_TOKENunset, a developer who already ranhf auth loginor setHF_TOKENis now prompted for a token when downloading a gated repo, as they always were for private repos. Pressing Enter works (core passestoken=Noneand the cached token is used), but the prompt is unnecessary.HUGGING_FACE_TOKENresolves, core warns that the download may 401 even if huggingface_hub holds a cached token and the download succeeds.Proposal
Before prompting or warning, check whether huggingface_hub will send a token anyway:
getpassprompt whenimplicit_token_availableis true.token=Nonein that case, so what goes over the wire is unchanged and core still does not readHF_TOKENitself (Handle empty HUGGING_FACE_TOKEN gracefully #422).The #540 reviewer checked that
get_tokenandconstants.HF_HUB_DISABLE_IMPLICIT_TOKENexist from 0.25.1, the currenthuggingface_hubfloor, so this works across the supported range.Tests to update: the
hf-token-onlycases intests/core/tools/test_hugging_face.pycurrently pin the prompt and the warning on purpose; they would instead assert no prompt and no warning whenget_token()returns a token, and keep asserting the old behaviour when it returnsNoneor implicit tokens are disabled.