Gemma-4 small (E2B/E4B): emit vision-tower shapes in HF export map - #4860
Open
lokic233 wants to merge 1 commit into
Open
Gemma-4 small (E2B/E4B): emit vision-tower shapes in HF export map#4860lokic233 wants to merge 1 commit into
lokic233 wants to merge 1 commit into
Conversation
GEMMA4_SMALL_HF_WEIGHTS_TO_SHAPE returned only language-model / PLE shapes and
emitted zero model.vision_tower.* / model.embed_vision.* keys, while
GEMMA4_SMALL_MAXTEXT_TO_HF_PARAM_MAPPING already maps the full vision encoder.
Since utils._process requires every mapped HF target to be present in the shape
map, 'to_huggingface use_multimodal=true' on a converted gemma4-e2b/e4b
checkpoint died on the first vision tensor:
ValueError: HF path 'model.vision_tower.patch_embedder.input_proj.weight'
not found in hf_shape_map
Add the vision block to GEMMA4_SMALL_HF_WEIGHTS_TO_SHAPE, gated on vision_config
presence (matching the mapping's 'use_multimodal and vcfg' gate) and mirroring
the already-landed GEMMA4_HF_WEIGHTS_TO_SHAPE (26B/31B) vision convention: each
clipped-linear vision projection emits a plain '.linear.weight' (no clip-bound
keys, which the small mapping does not map); attention q/k/v/o + q_norm/k_norm,
the four layernorms, gate/up/down MLP, patch-embedder input_proj +
position_embedding_table, and the embed_vision projection. std_scale / std_bias
are emitted only under standardize=true (E2B/E4B ship standardize=false).
Add tests/unit/hf_shape_test.py: fail-closed coverage asserting every vision HF
target the small param map produces exists in the shape map, plus std-conditional
and text-only-config checks.
Text-only and 26B/31B export paths are unaffected.
Signed-off-by: Loki Chen <dengcchi@meta.com>
There was a problem hiding this comment.
Code Review
This pull request adds support for the vision tower (multimodal E2B / E4B) in the GEMMA4_SMALL_HF_WEIGHTS_TO_SHAPE function within hf_shape.py. It maps the shapes of the vision tower components, including the patch embedder, position embedding table, standardization parameters, projection layers, and encoder layers, when a vision_config is present. Additionally, a comprehensive suite of unit tests has been introduced in hf_shape_test.py to validate this mapping behavior. There are no review comments, and I have no additional feedback to provide.
lokic233
marked this pull request as ready for review
August 12, 2026 21:08
lokic233
requested review from
A9isha,
NuojCheng,
RissyRan,
SurbhiJainUSC,
abhinavclemson,
aireenmei,
bvandermoon,
darisoy,
dipannita08,
gagika,
gobbleturk,
hengtaoguo,
huytransformer,
igorts-git,
jiangjy1982,
khatwanimohit,
parambole,
richjames0,
shralex,
shuningjin,
vipannalla and
xibinliu
as code owners
August 12, 2026 21:08
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
GEMMA4_SMALL_HF_WEIGHTS_TO_SHAPE(used bygemma4-e2b/gemma4-e4b) returnedonly language-model / Per-Layer-Embedding shapes and emitted zero
model.vision_tower.*/model.embed_vision.*keys — even thoughGEMMA4_SMALL_MAXTEXT_TO_HF_PARAM_MAPPINGalready maps the full vision encoder.Because
checkpoint_conversion/utils/utils.py::_processrequires every mapped HFtarget to be present in the shape map,
to_huggingface use_multimodal=trueon aconverted Gemma-4 E2B/E4B checkpoint died on the first vision tensor:
This change adds the vision block to
GEMMA4_SMALL_HF_WEIGHTS_TO_SHAPE:vision_configpresence, matching the mapping'suse_multimodal and vcfggate (no effect on text-only exports).GEMMA4_HF_WEIGHTS_TO_SHAPE(26B/31B) visionconvention: each clipped-linear vision projection emits a plain
.linear.weight— no clip-bound keys, because the small param map does notmap
input_min/input_max/output_min/output_max.q/k/v/o_proj.linear.weight+q_norm/k_norm, the fourlayernorms,
gate/up/down_proj.linear.weight, the patch-embedderinput_proj+position_embedding_table, and theembed_visionprojection.std_scale/std_biasare emitted only understandardize=true; E2B/E4Bship
standardize=falseand store no std keys (consistent with the mapping'splaceholder path).
The 26B/31B (
GEMMA4_HF_WEIGHTS_TO_SHAPE) and text-only paths are untouched.Tests
Added
tests/unit/hf_shape_test.py(pytest.mark.decoupled_target, CPU-only, nocheckpoint / no gated weights — builds a config dict shaped like the HF
gemma4-e2bconfig):test_shape_map_covers_every_mapped_vision_target— fail-closed: every HFtarget
GEMMA4_SMALL_MAXTEXT_TO_HF_PARAM_MAPPINGproduces must exist inGEMMA4_SMALL_HF_WEIGHTS_TO_SHAPE. This is the exact invariant_processenforces; it fails on
main(missing vision keys) and passes with this change.test_vision_tower_keys_present— canonical vision entry / projector / per-layer keys.test_no_clip_bound_keys— noinput_min/max/output_min/output_maxshape keys.test_std_keys_only_when_standardize— std keys appear only understandardize=true.test_text_only_config_emits_no_vision_keys— no regression to the text-only path.Reproduce:
Validation performed on this branch (grr-maxtext image):
standardize=false(42/42 vision targets) andstandardize=true(44/44);the same check fails on
main(vision keys absent) — confirming the fix.pytest tests/unit/hf_shape_test.py→ 5 passed.pyink --pyink-indentation=2 --line-length=122 --check→ clean.pylint --rcfile=pylintrc --disable=R0401,R0917,W0201,W0613→ 10.00/10.codespell→ clean.Checklist
Before submitting this PR, please make sure (put X in square brackets):
gemini-reviewlabel.Authored with Navi on behalf of Loki Chen (@lokic233).