feat: ROCm/HIP support for turbo3 KV cache (gfx1100/RDNA3)

[apollosenvy]

Summary

Adds complete ROCm/HIP backend support for turbo3 KV cache quantization, enabling 4.6x KV compression on AMD GPUs.

Tested on: AMD Radeon RX 7900 XTX (gfx1100), ROCm 7.1, Qwen3.5-27B Q4_K_M

What's included

• GPU kernels: dequant, vec_dot, Walsh-Hadamard Transform, quantize, GET_ROWS, CPY
• Flash attention: F16 dequant bridge (turbo3 -> F16 temp buffer -> F16 FA kernel). The native turbo3 FA template compiles but produces broken kernels on HIP/RDNA3, so we route through the existing need_f16_K/V conversion path in launch_fattn.
• CPU fallback: vec_dot + type traits for graph scheduler fallback paths
• All dispatch wiring: supports_op for MUL_MAT, GET_ROWS, SET_ROWS, CPY, FLASH_ATTN_EXT, TURBO_WHT

Critical bugfix: K/V rotation before cache write

Found and fixed a quality issue: K and V vectors were being quantized to turbo3 without WHT rotation. The graph rotated Q (forward) and inverse-rotated the attention output, but the K/V write path in cpy_k()/cpy_v() was missing the forward rotation step.

Before fix: <accepts high pressure "Itself" - ount>& & nsp; a- xii'
After fix: 7 x 8 = 56. This is one of the most common multiplication facts.

This fix is in src/llama-kv-cache.cpp and may also benefit the Metal path if it has the same omission.

Performance

Metric	FP16 KV	turbo3 KV	Ratio
Prompt	90 tok/s	82 tok/s	91%
Generation	21 tok/s	12 tok/s	57%*
KV cache (32K ctx)	2048 MB	448 MB	4.6x

*Generation speed regression is from AMD_SERIALIZE_KERNEL=3 requirement (see below).

Known limitation

Requires AMD_SERIALIZE_KERNEL=3 environment variable. Without it, the multi-stream graph scheduler has a race condition between KV cache write and FA read ops. Per-op cudaStreamSynchronize is not sufficient. This is a ROCm runtime issue, not a turbo3 bug.

Files changed (24 files, +595 lines)

New files (5):

• dequantize-turbo.cuh - Device dequant (centroid LUT)
• vecdot-turbo.cuh - MMVQ vec_dot
• turbo-wht.cu/cuh - WHT butterfly kernel
• fattn-vec-instance-turbo3_0-turbo3_0.cu - FA template (D=256)

Modified (19): common.cuh, mmvq.cu, fattn-common.cuh, fattn-vec.cuh, fattn.cu, convert.cu, cpy-utils.cuh, cpy.cu, set-rows.cu, getrows.cu, ggml-cuda.cu, CMakeLists.txt (cuda+hip), ggml-turbo-quant.c, ggml-quants.h, ggml-cpu.c, quants.h, llama-kv-cache.cpp

Test plan

• Standalone GPU kernel test (quantize/dequant round-trip)
• Server starts with --cache-type-k turbo3 --cache-type-v turbo3
• KV cache allocates at expected compression ratio
• Inference produces coherent output
• Math reasoning correct (7*8=56)
• Perplexity benchmark vs upstream Metal numbers
• Multi-turn conversation stability

[apollosenvy]

Synced to TOT — HIP port on rocm-turbo3-v2

Rebased onto your latest feature/turboquant-kv-cache (172fc85). Clean single commit: HIP/ROCm porting for your warp-cooperative turbo3 kernel + turbo2/turbo3 FA templates.

What was needed for HIP

1. Vendor header (hip.h): Added cudaMemcpyToSymbol/FromSymbol mappings. Fixed __shfl_sync, __shfl_xor_sync, __shfl_up_sync, __shfl_down_sync to support 3-arg calls (CUDA defaults width to warpSize, old HIP macros required 4 args). Added __ballot_sync -> __ballot with uint32_t cast.

2. HIP CMakeLists: Added turbo3 and turbo2 FA template instances (turbo3-turbo3, turbo3-q8_0, q8_0-turbo3, turbo2-turbo2, turbo2-q8_0, q8_0-turbo2).

PPL Confirmed

You were right about the pipeline bugs. With the CPU quantize stub fix (53f1298):

Model	KV Type	PPL	vs F16
Mistral-Small-24B (hd128)	F16	5.16	baseline
Mistral-Small-24B (hd128)	turbo3	5.28	+2.4%
Mistral-Small-24B (hd128)	q4_0	5.26	+1.9%

PolarQuant IS sound. The catastrophic PPL was 100% the CPU quantize stub zeroing qs/signs on layer 0. My earlier "fundamental limitation" analysis was wrong — thanks for pushing back on that.

Branch is rocm-turbo3-v2 on fork. Ready for review.

[apollosenvy]

TurboQuant+ Temporal Decay -- Implemented

Pushed to rocm-turbo3-v2: position-aware KV cache quality degradation. Old tokens get their signs field zeroed in-place, reducing turbo3 from 8-level to 4-level centroid reconstruction.

How it works

Three dynamic tiers with proportional boundaries:

• Hot (newest 15%): full quality
• Warm (middle 35%): base turbo3 quality
• Cold (oldest 50%): signs zeroed, 4-level reconstruction

Boundaries shift proportionally as context grows. Demotion is a single ggml_turbo_decay op appended to the graph after SET_ROWS. One kernel launch per layer per boundary advance.

Configuration

TURBO_DECAY=balanced          # 15/35/50 (default)
TURBO_DECAY=conservative      # 25/40/35
TURBO_DECAY=aggressive        # 5/25/70
TURBO_DECAY=10,30,60          # custom percentages

Benchmark (Qwen3.5-27B Q4_K_M, 7900 XTX)

PPL (ctx=2048, 4 chunks):

Mode	PPL	vs No-Decay
off	7.58	baseline
balanced	8.43	+11.2%
aggressive	9.81	+29.4%

Speed: Zero measurable overhead (within noise).

The PPL hit at 2K is expected -- at short context, the cold tier covers tokens the model still actively attends to. At longer contexts (16K+), the cold tier covers much older tokens with naturally lower attention weight, so the quality impact should decrease. This is the design tradeoff: sacrifice some short-context quality for better long-context compression efficiency.

Implementation (8 commits)

1. Design spec + reviewer feedback fixes
2. Implementation plan
3. Config struct + env var parsing
4. GGML_OP_TURBO_DECAY op
5. CUDA/HIP/CPU decay kernels
6. Wire into cpy_k/cpy_v graph
7. PPL validation
8. Push

Phase 2 (hot tier q8_0 promotion) and Phase 3 (attention-guided decay) designed but deferred.

[TheTom]

Tested on M5 Max 128GB (Metal) and M2 Pro 32GB (Metal). Thanks for the work here, some notes below.

What passes (head_dim=128 models)

PPL and speed are clean for standard models:

Config	PPL (8-chunk)	pp512 t/s	tg128 t/s	Known baseline
turbo3	6.1756	2728	75.14	6.1756 (match)
turbo4	6.1250	—	—	6.1250 (match)

Build passes on Metal with no warnings.

Issue 1: TURBO_DECAY crashes on Metal

The temporal decay op has CUDA/HIP/CPU kernels but no Metal kernel. When TURBO_DECAY=balanced is set with a GPU-offloaded model, it aborts:

pre-allocated tensor (cache_k_l3 (view) (view)) in a buffer (MTL0) that cannot run the operation (TURBO_DECAY)

Repro:

TURBO_DECAY=balanced ./build/bin/llama-cli \
  -m model.gguf -ngl 99 -fa on \
  -ctk turbo3 -ctv turbo3 \
  -p "Write a 500 word essay about the history of computing" -n 500

This is a crash, not a graceful fallback. The decay op needs either a Metal kernel or a supports_op guard that prevents dispatch to the Metal backend. As-is, any Metal user who sets the env var will hit an abort.

Issue 2: Boundary V modes 5/6/7 removed

The PR removes layer-adaptive modes 5, 6, and 7 (Boundary V) from the constructor. These were shipped in PR #30 and are part of our current feature set. TURBO_LAYER_ADAPTIVE=5 no longer produces a "Boundary V" log line, confirming the code path is gone.

Suggestion

Consider splitting this into separate PRs:

1. ROCm/HIP kernel support (the core contribution, looks solid)
2. Temporal decay (experimental feature, needs Metal kernel + more validation)
3. Non-128 head_dim q8_0 fallback (architectural decision worth discussing separately)

Happy to re-review individual pieces. The ROCm work is great and would be straightforward to land on its own.

[apollosenvy]

Split per your suggestion:

• PR feat: HIP/ROCm support for turbo3/turbo2 (7900 XTX) #31 — ROCm/HIP port only. Single commit, minimal. Rebased on your latest (includes Boundary V). Also excludes D>=576 fattn-tile instances that exceed HIP's 64KB local memory limit.

• Temporal decay — Will open as separate PR once Metal kernel or supports_op guard is added. The decay op shouldn't dispatch to backends that don't implement it.

• head_dim fallback — Your commit 4c4511c14 already handles this. No separate PR needed from our side.

Re: modes 5/6/7 — we didn't remove them. Our branch was based on a commit before PR #30. The new PR #31 is rebased on your latest which includes Boundary V, so modes 5/6/7 are intact.