Symmetric turbo3 KV catastrophic on Qwen2.5 (GQA 7:1) — auto-asymmetric fix proposed

[signalnine]

Problem

Symmetric turbo3 K+V produces catastrophic PPL on Qwen2.5-7B (PPL 2,887 vs baseline 7.43). This has never worked — bisect confirms PPL 10,829 at the very first CUDA turbo commit (97dddac).

Root cause

Qwen2.5-7B has 4 KV heads serving 28 Q heads (GQA ratio 7:1). Each K head's turbo3 quantization error gets broadcast to 7 Q heads, amplifying the error. Models with GQA ≤ 4:1 are unaffected (Mistral-7B: 8 KV heads, turbo3 PPL = 7.71, only +4.4%).

Bisect data

Commit	Label	Qwen turbo3/turbo3 PPL
97dddac	first VEC FA fix	10,829
f2b3936	asymmetric KV support	2,887
53180d9	block-size 128	106,035
fe15d61	InnerQ equalization	2,886
bc05a68	current HEAD	2,887

Never worked. The asymmetric KV support (q8_0-K + turbo3-V) was introduced as a manual fix for exactly this issue.

Proposed fix

Auto-detect high GQA ratio at KV cache init and silently upgrade K to q8_0:

const uint32_t gqa_ratio = n_head / n_head_kv;
if (gqa_ratio >= 6 && type_k == type_v && k_is_turbo) {
    type_k = GGML_TYPE_Q8_0;  // auto-asymmetric
}

Results with fix:

• Qwen2.5-7B turbo3/turbo3: PPL 7.06 (was 2,887), NIAH 5/5
• Mistral-7B: unaffected (GQA 4:1, below threshold)
• Opt-out: TURBO_AUTO_ASYMMETRIC=0

Implementation: https://github.com/signalnine/llama-cpp-turboquant/tree/fix/auto-asymmetric-gqa
Also included in PR #53.

Affected models

Any model with n_head_kv ≤ 4 and n_head ≥ 24 (GQA ≥ 6:1):

• Qwen2.5 family (4 KV heads, 28 Q heads)
• Qwen2 family (same architecture)
• Potentially other models with aggressive GQA

[seanrasch]

Some supporting numbers from a different model family (Qwen3 8B, pure transformer, GQA 4:1, head_dim=128) on RTX 3080 Ti.

wikitext-2, 50 chunks @ 2K context, build at bc05a6803 (feature/turboquant-kv-cache HEAD):

Config (K/V)	PPL	Δ vs f16
f16/f16	8.5253	—
turbo3/turbo3 (symmetric)	9.0769	+6.47%
q8_0/turbo3 (asymmetric)	8.5400	+0.17%

Direction matches your Qwen2.5 GQA 7:1 finding — symmetric turbo3 K is significantly worse on GQA models, and q8_0 K + turbo3 V is essentially free. On 4:1 GQA the damage is +6.47% (significant but recoverable); on 7:1 GQA you're seeing catastrophic, which fits a scaling-with-ratio story.

For contrast, Qwen3.5 9B (hybrid GDN+attn, GQA 4:1, head_dim=256) shows symmetric turbo3 = +0.82% PPL at 16K context — the GDN layers seem to dampen the GQA-K-symmetric problem dramatically. So the issue isn't purely GQA-ratio; pure-transformer GQA is the worst case.

Auto-asymmetric KV looks like the right default. Speed picture matches too: on Qwen3 8B, symmetric turbo3 K cuts prefill from 4922→2114 t/s (-57%) on Ampere, while q8_0 K recovers to 4707 t/s.