fix(tests): Mock HF in integration tests#259
Draft
matthewgrossman wants to merge 4 commits into
Draft
Conversation
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
The conftest.py patches hf_hub_download and AutoConfig.from_pretrained to serve from local fixtures, preventing HuggingFace API calls. The download_fixtures.py script regenerates fixtures when new models are added to tests. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Contributor
|
The 5K lines of HF config fixtures were unnecessary — the parallelism tests are always skipped in CI (torch not installed). Stripped down to only the change that fixes the actual CI failures: mocking HfApi in TestTrustRemoteCodePermission. Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
TestTrustRemoteCodePermissionintegration tests create HuggingFace-backed filesets (storage.type=huggingface), which triggers real HF Hub API calls (repo_info,get_hf_file_metadata) duringvalidate_storage()andresolve_config(). These calls are rate-limited by HuggingFace, causing all 6 tests to fail with:Setting
HF_TOKENin CI (PR #202) reduced the frequency but didn't eliminate the issue — rate limits still apply to authenticated users, and concurrent CI runs sharing the same token compound the problem.Strategy
Mock at the boundary, not in the test logic. These tests verify authorization behavior (who can set
trust_remote_code), not HuggingFace connectivity. The real HF API calls are an unnecessary side effect of fileset creation that adds fragility without testing anything meaningful.Added an autouse
_mock_hf_storagefixture toTestTrustRemoteCodePermissionthat patchesHfApi— the same pattern used by the existing unit tests inservices/core/files/tests/test_huggingface_backend.py. The mock returns a syntheticrepo_infowith an emptysiblingslist (skipping the file metadata check) and a fake commit SHA forresolve_config().No fixture files needed — just a 16-line mock that prevents the 6 failing tests from ever hitting the network.
Test plan
TestTrustRemoteCodePermission— the 6 tests that were failing)Fixes AIRCORE-744