Eval bug: pre-allocated tensor (cache_k_l3 (view)) in a buffer that cannot run the operation (SET_ROWS)

[WizardlyBump17]

Name and Version

root@ed00659cbc64:/app# ./llama-cli --version
load_backend: loaded SYCL backend from /app/libggml-sycl.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
version: 8793 (bc05a6803)
built with IntelLLVM 2025.3.2 for Linux x86_64

root@676ea1faa4ff:/app# ./llama-cli --version
load_backend: loaded Vulkan backend from /app/libggml-vulkan.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
version: 8793 (bc05a6803)
built with GNU 15.2.0 for Linux x86_64

root@f39e7db7b25c:/app# ./llama-cli --version
load_backend: loaded Vulkan backend from /app/libggml-vulkan.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
version: 8772 (e9c54d557)
built with GNU 15.2.0 for Linux x86_64

Operating systems

Linux

GGML backends

SYCL

Hardware

Ryzen 7 5700X3D
B580

Models

No response

Problem description & steps to reproduce

root@f39e7db7b25c:/app# ./llama-bench --model /models/Qwen3.5-0.8B-F16.gguf --flash-attn 1 --cache-type-k turbo4 --cache-type-v turbo4
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(tm) B580 Graphics (BMG G21) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: KHR_coopmat
load_backend: loaded Vulkan backend from /app/libggml-vulkan.so
load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so
| model                          |       size |     params | backend    | ngl | type_k | type_v | fa |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | -----: | -----: | -: | --------------: | -------------------: |
/app/ggml/src/ggml-backend.cpp:809: pre-allocated tensor (cache_k_l3 (view)) in a buffer (Vulkan0) that cannot run the operation (SET_ROWS)
libggml-base.so.0(+0x18ab6) [0x7c69f00e3ab6]
libggml-base.so.0(ggml_print_backtrace+0x20d) [0x7c69f00e3f1d]
libggml-base.so.0(ggml_abort+0x166) [0x7c69f00e4106]
libggml-base.so.0(+0x31b7c) [0x7c69f00fcb7c]
libggml-base.so.0(ggml_backend_sched_split_graph+0xc8f) [0x7c69f00feb9f]
libllama.so.0(_ZN13llama_context13graph_reserveEjjjPK22llama_memory_context_ibPm+0x616) [0x7c69f0282a06]
libllama.so.0(_ZN13llama_context13sched_reserveEv+0x1000) [0x7c69f0284a40]
libllama.so.0(_ZN13llama_contextC2ERK11llama_model20llama_context_params+0xa75) [0x7c69f0286ac5]
libllama.so.0(llama_init_from_model+0x126) [0x7c69f02877c6]
./llama-bench(+0x3ab84) [0x56333c439b84]
/usr/lib/x86_64-linux-gnu/libc.so.6(+0x2a601) [0x7c69ef3b3601]
/usr/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x88) [0x7c69ef3b3718]
./llama-bench(+0x3d3d5) [0x56333c43c3d5]
Aborted                    (core dumped) ./llama-bench --model /models/Qwen3.5-0.8B-F16.gguf --flash-attn 1 --cache-type-k turbo4 --cache-type-v turbo4

Tried on both SYCL and Vulkan backends

First Bad Commit

No response

Relevant log output

Above

[Ezzz-dev]

Can reproduce this right now in latest commit as well.
Vulkan.
7900XTX

[apollosenvy]

Can no longer reproduce on a 7900 XTX (gfx1100, RADV NAVI31, Mesa 25.x, Vulkan 1.4.341) against current feature/turboquant-kv-cache (59798f1) + the FA pipeline registration follow-up on my fork branch apollosenvy/pr/vulkan-turbo3-april-fix.

Without the follow-up, origin's current tip crashes earlier for me — not with the SET_ROWS error in this issue but with ggml-vulkan.cpp:9110: GGML_ASSERT(Br == pipeline->wg_denoms[0]) in ggml_vk_flash_attn. Turns out CREATE_FA(GGML_TYPE_TURBO3_0, turbo3_0, FA_COOPMAT1, _cm1) was missing, so coopmat-capable devices (7900 XTX falls into that) dispatched against an uninitialized FA pipeline. Registering the cm1 and cm2 variants clears it.

If you're still on the older commit from your repro, give that branch a try. test-backend-ops -o SET_ROWS -b Vulkan0 passes 147/147 for me, -o FLASH_ATTN_EXT -b Vulkan0 -p type_KV=turbo3 passes all 530 cases, and llama-cli -ngl 99 -fa on -ctk turbo3 -ctv turbo3 -dev Vulkan0 no longer aborts on prefill.

[theoparis]

I encountered the same error with turbo4 but not turbo3 👀

[cclecle]

Same error on SYCL, latest oneAPI , Intel A380:

./llama-bench -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --mmap 0 -t 8 -d 2048,4096,8192 -fitt 2048 --n-cpu-moe 60 -ctk turbo3 -ctv q8_0 -fa 1
| model                          |       size |     params | backend    |  n_cpu_moe | threads | type_k | type_v | fa | mmap |       fitt |            test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | ---------: | ------: | -----: | -----: | -: | ---: | ---------: | --------------: | -------------------: |
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
/root/llama-cpp-turboquant/ggml/src/ggml-backend.cpp:898: pre-allocated tensor (cache_k_l3 (view)) in a buffer (SYCL0) that cannot run the operation (SET_ROWS)
libggml-base.so.0(+0x15b78) [0x7c7ff4cfcb78]
libggml-base.so.0(ggml_print_backtrace+0x285) [0x7c7ff4cfcb55]
libggml-base.so.0(ggml_abort+0x109) [0x7c7ff4cfbbd9]
libggml-base.so.0(+0x39a8d) [0x7c7ff4d20a8d]
libggml-base.so.0(ggml_backend_sched_split_graph+0x389) [0x7c7ff4d1d069]
libllama.so.0(_ZN13llama_context13graph_reserveEjjjPK22llama_memory_context_ibPm+0x2fa) [0x7c7ff4e95dca]
libllama.so.0(_ZN13llama_context13sched_reserveEv+0x4e8) [0x7c7ff4e947b8]
libllama.so.0(_ZN13llama_contextC2ERK11llama_model20llama_context_params+0x189c) [0x7c7ff4e933ec]
libllama.so.0(llama_init_from_model+0x302) [0x7c7ff4e9da62]
libllama.so.0(+0x78474) [0x7c7ff4e78474]
libllama.so.0(+0x71524) [0x7c7ff4e71524]
libllama.so.0(llama_params_fit+0x47) [0x7c7ff4e71317]
./llama-bench() [0x421824]
/lib/x86_64-linux-gnu/libc.so.6(+0x29ca8) [0x7c7ff1235ca8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7c7ff1235d65]
./llama-bench() [0x41f2c1]
Aborted

Edit:
Just asked Claude to implement the missing SYCL Kernel..... benchmark works ! Need to test quality now.
If one want to try:
https://github.com/cclecle/llama-cpp-turboquant/tree/feature/turboquant-kv-cache