Skip to content

test: add adapter unit tests for Llama#1413

Open
mukund1985 wants to merge 1 commit into
TransformerLensOrg:devfrom
mukund1985:test/llama-adapter
Open

test: add adapter unit tests for Llama#1413
mukund1985 wants to merge 1 commit into
TransformerLensOrg:devfrom
mukund1985:test/llama-adapter

Conversation

@mukund1985

Copy link
Copy Markdown

Adds a focused test suite for LlamaArchitectureAdapter per issue #1302.

What this adds

test_llama_adapter.py (22 tests)

  • Config flags: normalization_type="RMS", positional_embedding_type="rotary", final_rms=True, gated_mlp=True, attn_only=False, uses_rms_norm=True
  • GQA support via n_key_value_heads: propagation to cfg, presence in default_config when set, and absence when not set
  • Component mapping: top-level keys, bridge types, and HF module paths (model.embed_tokens, model.rotary_emb, model.layers, model.norm, lm_head)
  • Block submodules: ln1/ln2/attn/mlp bridge types and HF paths (input_layernorm, post_attention_layernorm, self_attn, mlp)
  • Attention Q/K/V/O and MLP gate/in/out projections: keys, HF paths, and LinearBridge types
  • Weight conversion: exact weight_processing_conversions key set

All 22 tests pass locally.

Checklist

  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing unit tests pass locally with my changes
  • I have not rewritten tests relating to key interfaces which would affect backward compatibility

Covers LlamaArchitectureAdapter — config flags (including uses_rms_norm),
GQA support via n_key_value_heads, component mapping (bridge types and HF
module paths), and weight conversion key set.

Part of TransformerLensOrg#1302.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant