Commit 7b28d78
authored
Add Exaone4 AWQ mapping (#2046)
EXAONE4 uses
[QK-Reorder-Norm](https://huggingface.co/LGAI-EXAONE/EXAONE-4.0-32B#introduction).
Transformers:
[Exaone4DecoderLayer](https://github.com/huggingface/transformers/blob/v4.57.1/src/transformers/models/exaone4/modeling_exaone4.py#L284-L314)
```python
class Exaone4DecoderLayer(GradientCheckpointingLayer):
def forward(
self,
hidden_states: torch.Tensor,
attention_mask: Optional[torch.Tensor] = None,
position_ids: Optional[torch.LongTensor] = None,
past_key_values: Optional[Cache] = None,
use_cache: Optional[bool] = False,
cache_position: Optional[torch.LongTensor] = None,
position_embeddings: Optional[tuple[torch.Tensor, torch.Tensor]] = None,
**kwargs: Unpack[TransformersKwargs],
) -> torch.Tensor:
residual = hidden_states
hidden_states, _ = self.self_attn(
hidden_states=hidden_states,
attention_mask=attention_mask,
position_ids=position_ids,
past_key_values=past_key_values,
use_cache=use_cache,
cache_position=cache_position,
position_embeddings=position_embeddings,
**kwargs,
)
hidden_states = self.post_attention_layernorm(hidden_states)
hidden_states = residual + hidden_states
# Fully Connected
residual = hidden_states
hidden_states = self.mlp(hidden_states)
hidden_states = self.post_feedforward_layernorm(hidden_states)
hidden_states = residual + hidden_states
return hidden_states
```
Signed-off-by: lkm2835 <lkm2835@gmail.com>1 parent b77175d commit 7b28d78
1 file changed
+11
-0
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
142 | 142 | | |
143 | 143 | | |
144 | 144 | | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
145 | 155 | | |
146 | 156 | | |
147 | 157 | | |
148 | 158 | | |
149 | 159 | | |
150 | 160 | | |
| 161 | + | |
151 | 162 | | |
152 | 163 | | |
153 | 164 | | |
| |||
0 commit comments