[Bug] ft checkpoint fails on Qwen3.6-35B-A3B abliterated NVFP4 checkpoints: Missing MoE expert source layers

[Redro38]

Summary

ft checkpoint fails when converting two community Qwen3.6-35B-A3B abliterated NVFP4 checkpoints to FreeToken's FTW format.

Both checkpoints are fully downloaded, detected by FreeToken Desktop, and their dense weights are loaded successfully. The conversion consistently fails when FreeToken starts building the MoE expert banks.

The official nvidia/Qwen3.6-35B-A3B-NVFP4 checkpoint works correctly on the exact same Windows machine and FreeToken installation.

Environment

• FreeToken Desktop: v0.2.0-beta.14
• OS: Windows
• GPU: NVIDIA GeForce RTX 4070 SUPER, 12 GB VRAM
• System RAM: 32 GB
• Page file: 64 GB initial / 96 GB maximum on a secondary SSD
• Model location: C:\Users\<user>\.freetoken\models\...

This does not appear to be an out-of-memory issue:

• System RAM was not exhausted.
• The page file had been configured and Windows restarted before retrying.
• The process exits quickly with a Python ValueError.
• Dense weights are fully processed before the failure.

Checkpoints

Working checkpoint

• nvidia/Qwen3.6-35B-A3B-NVFP4

This checkpoint downloads, converts, and runs successfully in FreeToken on this machine.

Failing checkpoints

• sakamakismile/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4
• joshebbs/qwen3.6-35b-abliterated-nvfp4-modelopt

Both are Qwen3.6-35B-A3B abliterated NVFP4 variants. The first one is detected as COMPRESSED-TENSORS by FreeToken Desktop; the second is a ModelOpt NVFP4 export.

Steps to reproduce

1. Download one of the failing Hugging Face repositories locally.
2. Place it under:

   C:\Users\<user>\.freetoken\models\
   

3. Restart FreeToken Desktop so the model is detected.
4. Select the model and click Convert / Retry convert.
5. Wait for the ft checkpoint conversion to begin.

Expected behavior

FreeToken should convert compatible Qwen3.6-35B-A3B NVFP4 Hugging Face checkpoints into the FTW format, creating files such as:

freetoken_weight.json
freetoken-00000.ftw
freetoken-00001.ftw
...

At minimum, if a particular checkpoint layout or quantization export is unsupported, the error should identify the unsupported checkpoint format or tensor layout.

Actual behavior

The conversion stops after dense weights have been loaded and when the MoE expert banks are being built.

FreeToken Desktop only displays:

Convert failed
ft checkpoint failed (exit 1)

The detailed logs show:

[INFO] Converting dense weights: 92938it [01:01, 1926.95it/s]
[INFO] Loading compressed-tensors weights: 100%|██████████| 1/1
[INFO] expert banks: slow path (serial build)

ValueError: Missing MoE expert source layers:
{
  'gate_up':,[0][1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37][38][39]
  'down':[0][1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37][38][39]
}

The ModelOpt checkpoint fails with the same exception:

[INFO] Converting dense weights: 443it [00:06, 162.88it/s]
[INFO] Loading weights: 100%|██████████| 1/1
[INFO] expert banks: slow path (serial build)

ValueError: Missing MoE expert source layers:
{
  'gate_up':,[0][1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37][38][39]
  'down':[0][1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37][38][39]
}

Relevant traceback

File "freetoken\checkpoint\convert.py", line 230, in convert_checkpoint
  banks = load_expert_banks(model_path, mc, device=dev, dtype=dtype, layer_sink=sink)

File "freetoken\moe\expert_banks.py", line 337, in _build_expert_banks

File "freetoken\models\qwen3_5_moe\weight.py", line 857, in setup_offload_expert_banks

File "freetoken\models\loader.py", line 424, in _load

ValueError: Missing MoE expert source layers:
{'gate_up': [0, ..., 39], 'down': [0, ..., 39]}

Question / suspected cause

Is the Qwen3.6/Qwen3.5 MoE loader currently assuming a specific MoE tensor naming or storage layout used by the official NVIDIA checkpoint?

The failure occurs for all 40 MoE layers and only for expert sources (gate_up and down), while dense weights are read successfully. This suggests that FreeToken may not currently map the expert tensor layout or metadata used by these community NVFP4 exports.

The issue may not be specific to abliterated weights themselves. A likely common factor is the checkpoint conversion/export format:

• sakamakismile/... is a compressed-tensors NVFP4 checkpoint.
• joshebbs/... is a ModelOpt NVFP4 checkpoint.
• nvidia/Qwen3.6-35B-A3B-NVFP4 works on the same system.

Could support be added for these MoE expert layouts, or could the converter emit a more specific compatibility error identifying the expected and discovered expert tensor keys?

Additional information

I can provide, if useful:

• The full ft checkpoint command invoked by FreeToken Desktop.
• The complete FreeToken log files.
• The exact list of tensor keys from each checkpoint's model.safetensors.
• A comparison of tensor keys between the working NVIDIA checkpoint and the failing checkpoints.
• Screenshots of the Desktop model entries and conversion failure.

MoE tensor keys

Attached files:

• joshebbs-moe-tensor-keys.txt
• sakamakismile-moe-tensor-keys.txt

These files contain the actual MoE-related safetensors key names reported by the checkpoints.

joshebbs-moe-tensor-keys.txt
joshebbs-tensor-keys.txt

[Redro38]

UPDATE

Updated tensor-layout comparison

I downloaded the original Hugging Face source checkpoint for the working NVIDIA model and inspected its SafeTensors keys.

The three checkpoints all contain the expected Qwen3.6 MoE expert tensors:

Checkpoint	SafeTensors keys	MoE-related keys
nvidia/Qwen3.6-35B-A3B-NVFP4	124,468	123,818
sakamakismile/Huihui-Qwen3.6-35B-A3B-abliterated-NVFP4	124,756	124,118
joshebbs/qwen3.6-35b-abliterated-nvfp4-modelopt	124,423	124,010

Therefore, the failing models are not missing their MoE expert weights. The failure appears to be caused by unsupported tensor naming and/or NVFP4 tensor-layout variants.

Working NVIDIA checkpoint layout

Representative working keys:

model.language_model.layers.0.mlp.experts.121.gate_proj.weight
model.language_model.layers.0.mlp.experts.121.gate_proj.input_scale
model.language_model.layers.0.mlp.experts.121.gate_proj.weight_scale
model.language_model.layers.0.mlp.experts.121.gate_proj.weight_scale_2

model.language_model.layers.0.mlp.experts.121.up_proj.weight
model.language_model.layers.0.mlp.experts.121.down_proj.weight

Representative shapes/dtypes:

gate_proj.weight: U8 [1024][512]
up_proj.weight:   U8 [1024][512]
down_proj.weight: U8 [2048][256]

gate/up weight_scale: F8E4M3[128][512]
down weight_scale:    F8E4M3 [2048][32]

joshebbs ModelOpt checkpoint layout

Representative failing keys:

language_model.model.layers.0.mlp.experts.124.gateproj.weight
language_model.model.layers.0.mlp.experts.124.gateproj.inputscale
language_model.model.layers.0.mlp.experts.124.gateproj.weightscale
language_model.model.layers.0.mlp.experts.124.gateproj.weightscale2

language_model.model.layers.0.mlp.experts.124.upproj.weight
language_model.model.layers.0.mlp.experts.124.downproj.weight

The tensor dtypes and shapes appear similar to the working NVIDIA checkpoint, but the key layout differs:

language_model.model.layers   vs   model.language_model.layers
gateproj                      vs   gate_proj
upproj                        vs   up_proj
downproj                      vs   down_proj
inputscale                    vs   input_scale
weightscale                   vs   weight_scale
weightscale2                  vs   weight_scale_2

This checkpoint reports ModelOpt NVFP4 quantization (modelopt version 0.43.0, group size 16).

Could FreeToken normalize these ModelOpt-compatible aliases before building the MoE expert banks?

Huihui / Compressed Tensors checkpoint layout

Representative failing keys:

model.language_model.layers.0.mlp.experts.2.gateproj.weightpacked
model.language_model.layers.0.mlp.experts.2.gateproj.weightscale
model.language_model.layers.0.mlp.experts.2.gateproj.inputglobalscale
model.language_model.layers.0.mlp.experts.2.gateproj.weightglobalscale

model.language_model.layers.0.mlp.experts.2.upproj.weightpacked
model.language_model.layers.0.mlp.experts.2.downproj.weightpacked

This checkpoint reports:

quant_method: compressed-tensors
format: nvfp4-pack-quantized
group_size: 16
quantization_status: compressed

Unlike the joshebbs checkpoint, this appears to be a different packed NVFP4 layout rather than only a naming variation. It likely needs explicit compressed-tensors support and cannot safely be fixed by key renaming alone.

Request

Could the Qwen3.5/Qwen3.6 MoE loader:

1. Normalize the joshebbs-style ModelOpt aliases before resolving MoE expert sources; and/or
2. Add support for the compressed-tensors nvfp4-pack-quantized expert layout used by the Huihui checkpoint?

The current Missing MoE expert source layers: {'gate_up': [...], 'down': [...]} error is misleading because the expert tensors exist in both checkpoints.

huihui_failing_config_subset.json
huihui_failing_moe_keys.txt
joshebbs_failing_config_subset.json
joshebbs_failing_moe_keys.txt
nvidia_working_config_subset.json
nvidia_working_moe_keys.txt

[Redro38]

Additional investigation: exact FreeToken NVFP4 regex + non-destructive compatibility test

I performed additional local inspection of the installed FreeToken Desktop v0.2.0-beta.14 modules and a controlled SafeTensors-header-only compatibility experiment on the joshebbs/qwen3.6-35b-abliterated-nvfp4-modelopt checkpoint.

This does not modify the original checkpoint. I worked only on a separate copied model directory.

Important correction to my previous comment

The actual joshebbs source keys use the standard underscore projection names:

language_model.model.layers.0.mlp.experts.0.down_proj.input_scale
language_model.model.layers.0.mlp.experts.0.down_proj.weight
language_model.model.layers.0.mlp.experts.0.down_proj.weight_scale
language_model.model.layers.0.mlp.experts.0.down_proj.weight_scale_2

language_model.model.layers.0.mlp.experts.0.gate_proj.weight
language_model.model.layers.0.mlp.experts.0.up_proj.weight

So the relevant difference from FreeToken's expected NVFP4 expert naming is the namespace prefix:

joshebbs:
language_model.model.layers.<layer>.mlp.experts.<expert>...

FreeToken expected:
model.language_model.layers.<layer>.mlp.experts.<expert>...

FreeToken's exported NVFP4 expert source specification

The installed compiled module freetoken.models.qwen3_5_moe.weight exposes this exact regex:

_NVFP4_EXPERT_KEY_RE = re.compile(
    r"^model\.language_model\.layers\."
    r"(?P<layer>\d+)\.mlp\.experts\."
    r"(?P<expert>\d+)\."
    r"(?P<proj>gate_proj|up_proj|down_proj)\."
    r"(?P<kind>weight|weight_scale|weight_scale_2)$"
)

It also exposes:

_NVFP4_SOURCE_SPEC
desc = "Qwen3.5 NVFP4 experts"

proj_to_role = {
  "gate_proj": "gate",
  "up_proj": "up",
  "down_proj": "down"
}

This appears to be the expected source layout for Qwen3.5/Qwen3.6 ModelOpt NVFP4 scattered experts.

Header-only compatibility experiment

I made a separate copy of the joshebbs checkpoint and rewrote only the SafeTensors JSON header, only for keys containing .mlp.experts.:

language_model.model.layers.<layer>.mlp.experts.<expert>...
-> model.language_model.layers.<layer>.mlp.experts.<expert>...

No tensor bytes were dequantized, reshaped, or changed.

Validation results:

Total SafeTensors keys: 124423
MoE expert keys renamed: 122880
Non-expert keys left unchanged: 1543
Output-name collisions: 0
Source header bytes: 16890584
New header bytes: 16890584
SafeTensors validation: passed
Payload SHA-256 before/after: identical

The rewritten checkpoint contained keys such as:

model.language_model.layers.0.mlp.experts.0.down_proj.weight
model.language_model.layers.0.mlp.experts.0.down_proj.weight_scale
model.language_model.layers.0.mlp.experts.0.down_proj.weight_scale_2

model.language_model.layers.0.mlp.experts.0.gate_proj.weight
model.language_model.layers.0.mlp.experts.0.gate_proj.weight_scale
model.language_model.layers.0.mlp.experts.0.gate_proj.weight_scale_2

model.language_model.layers.0.mlp.experts.0.up_proj.weight
model.language_model.layers.0.mlp.experts.0.up_proj.weight_scale
model.language_model.layers.0.mlp.experts.0.up_proj.weight_scale_2

Result

Despite the rewritten MoE keys matching the exported _NVFP4_EXPERT_KEY_RE pattern, ft checkpoint still fails with the same exception for every one of the 40 MoE layers:

ValueError: Missing MoE expert source layers:
{
  'gate_up': [0, 1, ..., 39],
  'down': [0, 1, ..., 39]
}

Dense-weight conversion still completes successfully before the error.

Request / question

This suggests the failure is not resolved by the expert-key prefix normalization alone.

Could you please advise which condition prevents load_nvfp4_moe_expert_sources / stream_moe_expert_sources from registering these otherwise regex-matching Qwen3.5 NVFP4 expert tensors?

Possible areas to check may include:

• checkpoint-format detection before the scattered-expert NVFP4 reader is selected;
• key enumeration/filtering before _NVFP4_EXPERT_KEY_RE is applied;
• dependence on config.json, model.safetensors.index.json, or another ModelOpt metadata convention;
• a requirement involving input_scale or another auxiliary NVFP4 tensor;
• a difference between the official NVIDIA ModelOpt export and ModelOpt 0.43.0 community exports.

An additional debug message showing:

1. the selected MoE/NVFP4 reader,
2. the number of expert keys scanned,
3. the number matching _NVFP4_EXPERT_KEY_RE, and
4. the first few rejected expert keys,

would make this incompatibility much easier to diagnose.

[Yuankus]

same issue

[JIAQI13]

Update: fix submitted as #296 (feat(qwen3_5_moe): support compressed-tensors NVFP4 experts with block-fp8 dense side). It is independent of #294 — the two PRs touch different hunks in the shared files and do not conflict.

Root cause of the sakamakismile/Huihui-...-abliterated-NVFP4 failure: it is a compressed-tensors checkpoint — routed experts are stored NVFP4 with weight_packed / weight_scale / weight_global_scale keys (4-bit, group_size 16), while the dense side (attn/GDN + shared expert) is 128×128 block FP8. The expert bank loader only recognized the modelopt layout (weight / weight_scale / weight_scale_2), so the bank spec matched zero layers and conversion failed with ValueError: Missing MoE expert source layers (dense weights already read). On current main the same checkpoint instead crashes earlier on the dense block-FP8 dequant (RuntimeError: tensor size 2048 vs 16).

#296 adds a compressed-tensors NVFP4 expert bank provider (reciprocal global scale, single-file safetensors header mapping), format detection from config_groups, and routes the dense side through the existing block-FP8 loader (shared-expert/GDN block-fp8 modules now key off dense_quant/attn_quant).

Verified on kyaky/Qwen3.6-35B-A3B-Uncensored-NVFP4 (same compressed-tensors mixed-precision family), RTX 3080 Laptop 16 GB:

• A/B ft checkpoint --moe-backend offload --shard-gib 8: main fails, this branch converts in 33 s — 20.19 GiB FTW, 653 weight + 240 expert banks
• ft serve --moe-backend offload --moe-cache-auto loads via the FTW fast path and serves /v1/chat/completions

Scope: compressed-tensors layout only. The other failing checkpoint in this report, joshebbs/qwen3.6-35b-abliterated-nvfp4-modelopt, uses an unusual doubled key prefix (language_model.model.layers.N.mlp.experts.*) that neither the existing modelopt regex nor #296 matches — looks like a separate export issue.