Qwen3.5 hybrid silently ignores --kv-bits and creates F16 K/V caches

[chaoshou-coder]

TL;DR

• BaseRT 0.2.2 accepts --kv-bits 4|8|84 for a qwen35moe model without an
  error or warning.
• serve --kv-bits 84 still passes F16/F16 to both contiguous and paged KV
  cache constructors; expected mixed mode is K=Q8, V=Q4.
• basert bench also defaults to F16/F16 for this model. Its internal
  BASERT_KV_BITS=4|8|16|84 override calls the setter, but cache creation still
  receives F16/F16.
• At 262K context, F16 KV is about 5.0 GiB versus about 1.875 GiB plus
  metadata for requested mixed 84.
• If quantized KV is unsupported for Qwen3.5 hybrid, BaseRT should reject the
  option or warn instead of silently falling back.

Environment

BaseRT:              0.2.2
OS:                  macOS 26.5.2 (25F84)
Machine:             MacBook Pro (Mac17,9)
Chip:                Apple M5 Pro
Unified memory:      48 GB

Model provenance

This is a locally converted model, not a pre-converted BaseCompute bundle.

Field	Value
Official source	https://huggingface.co/Qwen/Qwen-AgentWorld-35B-A3B
HF revision	60d2b0434a53d2e62a7c00a489586815d94ebffb
Source format	BF16 safetensors
Conversion	Local basert convert, custom Q4-mix profile
.base size	20,276,645,888 bytes
Runtime architecture	qwen35moe
Layers	40: 10 full-attention + 30 linear-attention
KV shape	2 KV heads, head_dim 256
Trained context	262,144 tokens

basert inspect

arch:          qwen35moe
quant_scheme:  BaseQ4
min_hw:        apple_m1
base_rt:       0.2.0
created:       1786461501
flags:         HeaderFlags(QUANTIZED | HAS_MOE)
n_layers:      40
n_tensors:     693
signed:        false
weights bytes: 20265235456
dtypes:
  BaseQ4    331 tensors   19195822080 bytes
  Bf16       80 tensors      42106880 bytes
  F16       222 tensors    1027298816 bytes
  F32        60 tensors          7680 bytes
n_slots:       0

User-visible reproduction

All documented KV widths are accepted without an unsupported/fallback warning:

MODEL=/path/to/Qwen-AgentWorld-35B-A3B.base

basert serve "$MODEL" --max-context 4096 --kv-bits 4
basert serve "$MODEL" --max-context 4096 --kv-bits 8
basert serve "$MODEL" --max-context 4096 --kv-bits 16
basert serve "$MODEL" --max-context 4096 --kv-bits 84

# Paged mode is also accepted:
basert serve "$MODEL" --max-context 4096 --kv-bits 84 --paged-kv

The selected runtime KV dtype is not shown by -v, basert inspect, /props,
or /slots, so there is no user-visible way to verify that the option took
effect.

Diagnostic method

The shipped dylib retains symbols for the cache constructors. Breakpoints were
set at:

baseRT::KVCache::create(
  Device&, int, int, int, int, KVDtype, KVDtype)

baseRT::PagedKVCache::create_hybrid(
  Device&, BaseRTModelConfig const&, int, int, KVDtype, KVDtype, int)

The K/V dtype arguments were read at function entry.

Observed cache dtypes

Executable/path	Requested setting	Actual K dtype	Actual V dtype	Warning
serve, contiguous	--kv-bits 84	1 (F16)	1 (F16)	None
serve, paged	--kv-bits 84	1 (F16)	1 (F16)	None
bench, contiguous	default	1 (F16)	1 (F16)	None
bench, paged	default	1 (F16)	1 (F16)	None
bench, contiguous	BASERT_KV_BITS=4/8/16/84	1 (F16)	1 (F16)	None
bench, paged	BASERT_KV_BITS=4/8/16/84	1 (F16)	1 (F16)	None

BASERT_KV_BITS=8 was separately verified to call
baseRT_set_kv_bits(8) before model load; the value is lost or overridden
before final cache creation.

Dtype mapping confirmation

The cache allocator's size-selection logic maps the constructor enum as follows:

Enum	Allocation	Dtype
1	2 bytes/element	F16
2	34 bytes / 32 elements	Q8 + metadata
3	18 bytes / 32 elements	Q4 + metadata

Expected mixed 84 constructor arguments:

K = enum 2 (Q8)
V = enum 3 (Q4)

Actual:

K = enum 1 (F16)
V = enum 1 (F16)

Memory impact at 262,144 tokens

The model has 10 full-attention layers, 2 KV heads, and head_dim 256.

Requested/actual mode	Approximate KV memory	Difference
Actual F16 K + F16 V	5.0 GiB	baseline
Requested mixed 84, before metadata/alignment	1.875 GiB	about 3.1 GiB less

Calculation:

F16:
10 layers * 2 KV heads * 256 dim * 2 (K,V) * 2 bytes * 262144
= 5.0 GiB

Mixed 84 raw payload:
10 layers * 2 KV heads * 256 dim * (1 byte K + 0.5 byte V) * 262144
= 1.875 GiB, plus quantization metadata/alignment

Expected vs actual

	Expected	Actual
Supported mode	Honor requested K/V width	Always F16/F16
Unsupported mode	Error or explicit warning	Silently accepted
Runtime diagnostics	Show selected K/V dtype	Not exposed
Memory planning	User can rely on selected width	Actual cache is ~3 GiB larger at 262K

Acceptable fixes include:

1. Support quantized KV for Qwen3.5 full-attention layers;
2. Reject unsupported --kv-bits values for this architecture; or
3. Print an explicit fallback warning and expose the actual selected dtype in
   -v, /props, /slots, or basert inspect.