Misc. bug: crashes when trying to use qwen3.5-27b and gemma4-26b-4a using tensor parallelism.

[nikitabalakin]

Name and Version

./llama-cli --version
ggml_cuda_init: found 3 CUDA devices (Total VRAM: 72375 MiB):
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24126 MiB
Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24126 MiB
Device 2: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes, VRAM: 24123 MiB
version: 8760 (865ff06)
built with GNU 15.2.0 for Linux x86_64

Operating systems

No response

Which llama.cpp modules do you know to be affected?

No response

Command line

./llama-server -m /home/kotokin/model/google_gemma-4-26B-A4B-it-Q8_0.gguf --no-mmap -ngl 999 -c 131072 -np 1 -sm tensor --cache-ram 2048 -ctxcp 2
./llama-server -m /home/kotokin/model/Qwen3.5-27B-ultra-uncensored-heretic-v2.Q8_0.gguf --no-mmap -ngl 999 -c 131072 -np 1 -sm tensor --cache-ram 2048 -ctxcp 2

Problem description & steps to reproduce

I get strange behavior when using the two models listed above. In the case of the 27b, I can successfully upload the model, and even use it in the built-in webui from llamacpp, the first message succeeds, and after the second I get an error.

In the case of the gemma4-26b-4A model, I can't even load the model, it crashes like a cache allocation attempt.

First Bad Commit

No response

Relevant log output

Logs

       eval time =   26828.29 ms /  1132 tokens (   23.70 ms per token,    42.19 tokens per second)
      total time =   27013.15 ms /  1146 tokens
slot      release: id  0 | task 0 | stop processing: n_tokens = 1145, truncated = 0
srv  update_slots: all slots are idle
srv  params_from_: Chat format: peg-native
slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.226 (> 0.100 thold), f_keep = 0.010
srv  get_availabl: updating prompt cache
srv   prompt_save:  - saving prompt with length 1145, total state size = 221.211 MiB
/home/kotokin/llama.cpp/ggml/src/ggml-backend-meta.cpp:1276: GGML_ASSERT(split_state.n_segments == 1) failed
[New LWP 45990]
[New LWP 45989]
[New LWP 45988]
[New LWP 45987]
[New LWP 45986]
[New LWP 45985]
[New LWP 45984]
[New LWP 45983]
[New LWP 45982]
[New LWP 45981]
[New LWP 45980]
[New LWP 45979]
[New LWP 45978]
[New LWP 45977]
[New LWP 45976]
[New LWP 45975]
[New LWP 45974]
[New LWP 45973]
[New LWP 45972]
[New LWP 45971]
[New LWP 45970]
[New LWP 45969]
[New LWP 45968]
[New LWP 45967]
[New LWP 45966]
[New LWP 45965]
[New LWP 45964]
[New LWP 45963]
[New LWP 45962]
[New LWP 45961]
[New LWP 45960]
[New LWP 45955]
[New LWP 45954]
[New LWP 45953]
[New LWP 45952]
[New LWP 45951]
[New LWP 45950]
[New LWP 45949]
[New LWP 45948]
[New LWP 45935]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
warning: 56     ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory
#0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56      in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1  0x000076bfc4aa013c in __internal_syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=0, a6=0, nr=61) at ./nptl/cancellation.c:49
warning: 49     ./nptl/cancellation.c: No such file or directory
#2  __syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75      in ./nptl/cancellation.c
#3  0x000076bfc4b1ca0f in __GI___wait4 (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>, usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30     ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4  0x000076bfc5619cd3 in ggml_print_backtrace () from /home/kotokin/llama.cpp/build/bin/libggml-base.so.0
#5  0x000076bfc5619e86 in ggml_abort () from /home/kotokin/llama.cpp/build/bin/libggml-base.so.0
#6  0x000076bfc5640713 in ggml_backend_meta_buffer_get_tensor(ggml_backend_buffer*, ggml_tensor const*, void*, unsigned long, unsigned long) () from /home/kotokin/llama.cpp/build/bin/libggml-base.so.0
#7  0x000076bfc52cfe7f in llama_io_write_buffer::write_tensor(ggml_tensor const*, unsigned long, unsigned long) () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#8  0x000076bfc5331d1d in llama_memory_recurrent::state_write_data(llama_io_write_i&, std::vector<std::pair<unsigned int, unsigned int>, std::allocator<std::pair<unsigned int, unsigned int> > > const&) const () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#9  0x000076bfc53320b8 in llama_memory_recurrent::state_write(llama_io_write_i&, int, unsigned int) const () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#10 0x000076bfc52bfc9f in llama_context::state_seq_write_data(llama_io_write_i&, int, unsigned int) () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#11 0x000076bfc52bfd8d in llama_context::state_seq_get_data(int, unsigned char*, unsigned long, unsigned int) () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#12 0x000056ed206a5bef in server_context_impl::get_available_slot(server_task const&) ()
#13 0x000056ed206bed47 in server_context_impl::process_single_task(server_task&&) ()
#14 0x000056ed20741ac7 in server_queue::start_loop(long) ()
#15 0x000056ed206086f7 in main ()
[Inferior 1 (process 45934) detached]
Aborted (core dumped)

common_init_result: added <eos> logit bias = -inf
common_init_result: added <|tool_response> logit bias = -inf
common_init_result: added <turn|> logit bias = -inf
llama_init_from_model: enabling flash_attn since it is required for SPLIT_MODE_TENSOR
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 131072
llama_context: n_ctx_seq     = 131072
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = false
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (131072) < n_ctx_train (262144) -- the full capacity of the model will not be utilized
llama_context:  CUDA_Host  output buffer size =     1.00 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 131072 cells
/home/kotokin/llama.cpp/ggml/src/ggml-backend.cpp:119: GGML_ASSERT(buffer) failed
[New LWP 46272]
[New LWP 46271]
[New LWP 46270]
[New LWP 46269]
[New LWP 46268]
[New LWP 46267]
[New LWP 46266]
[New LWP 46265]
[New LWP 46264]
[New LWP 46263]
[New LWP 46262]
[New LWP 46261]
[New LWP 46260]
[New LWP 46259]
[New LWP 46258]
[New LWP 46257]
[New LWP 46256]
[New LWP 46255]
[New LWP 46254]
[New LWP 46253]
[New LWP 46252]
[New LWP 46251]
[New LWP 46250]
[New LWP 46249]
[New LWP 46248]
[New LWP 46247]
[New LWP 46246]
[New LWP 46245]
[New LWP 46244]
[New LWP 46243]
[New LWP 46242]
[New LWP 46236]
[New LWP 46235]
[New LWP 46234]
[New LWP 46233]
[New LWP 46232]
[New LWP 46231]
[New LWP 46230]
[New LWP 46229]
[New LWP 46218]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
warning: 56     ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory
#0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56      in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1  0x00007e885e8a013c in __internal_syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=0, a6=0, nr=61) at ./nptl/cancellation.c:49
warning: 49     ./nptl/cancellation.c: No such file or directory
#2  __syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75      in ./nptl/cancellation.c
#3  0x00007e885e91ca0f in __GI___wait4 (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>, usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30     ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4  0x00007e885ef54cd3 in ggml_print_backtrace () from /home/kotokin/llama.cpp/build/bin/libggml-base.so.0
#5  0x00007e885ef54e86 in ggml_abort () from /home/kotokin/llama.cpp/build/bin/libggml-base.so.0
#6  0x00007e885ef6bfa0 in ggml_backend_buffer_get_size () from /home/kotokin/llama.cpp/build/bin/libggml-base.so.0
#7  0x00007e885ef7f355 in ggml_backend_meta_alloc_ctx_tensors_from_buft () from /home/kotokin/llama.cpp/build/bin/libggml-base.so.0
#8  0x00007e885f113211 in llama_kv_cache::llama_kv_cache(llama_model const&, ggml_type, ggml_type, bool, bool, bool, unsigned int, unsigned int, unsigned int, unsigned int, llama_swa_type, std::function<bool (int)> const&, std::function<int (int)> const&) () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#9  0x00007e885f120eae in llama_kv_cache_iswa::llama_kv_cache_iswa(llama_model const&, ggml_type, ggml_type, bool, bool, bool, bool, unsigned int, unsigned int, unsigned int, unsigned int, std::function<bool (int)> const&, std::function<int (int)> const&) () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#10 0x00007e885f150936 in llama_model::create_memory(llama_memory_params const&, llama_cparams const&) const () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#11 0x00007e885f0c690c in llama_context::llama_context(llama_model const&, llama_context_params) () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#12 0x00007e885f0c73b1 in llama_init_from_model () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#13 0x000060de66a70f9b in common_init_result::common_init_result(common_params&) ()
#14 0x000060de66a72d8a in common_init_from_params(common_params&) ()
#15 0x000060de6697205e in server_context_impl::load_model(common_params&) ()
#16 0x000060de668b8175 in main ()
[Inferior 1 (process 46217) detached]
Aborted (core dumped)

[nikitabalakin]

The problem with 27b occurs exactly at the moment the checkpoint is created. However, the cuda version does not affect the error.

Logs

lot update_slots: id  0 | task 635 | prompt processing progress, n_tokens = 2045, batch.n_tokens = 2048, progress = 0.033042
slot update_slots: id  0 | task 635 | n_tokens = 2045, memory_seq_rm [2045, end)
slot update_slots: id  0 | task 635 | prompt processing progress, n_tokens = 4090, batch.n_tokens = 2048, progress = 0.066085
slot update_slots: id  0 | task 635 | n_tokens = 4090, memory_seq_rm [4090, end)
slot update_slots: id  0 | task 635 | prompt processing progress, n_tokens = 6135, batch.n_tokens = 2048, progress = 0.099127
slot update_slots: id  0 | task 635 | n_tokens = 6135, memory_seq_rm [6135, end)
slot update_slots: id  0 | task 635 | prompt processing progress, n_tokens = 8180, batch.n_tokens = 2048, progress = 0.132170
slot update_slots: id  0 | task 635 | n_tokens = 8180, memory_seq_rm [8180, end)
slot update_slots: id  0 | task 635 | prompt processing progress, n_tokens = 10225, batch.n_tokens = 2048, progress = 0.165212
slot update_slots: id  0 | task 635 | n_tokens = 10225, memory_seq_rm [10225, end)
slot update_slots: id  0 | task 635 | 8192 tokens since last checkpoint at 0, creating new checkpoint during processing at position 12270
slot update_slots: id  0 | task 635 | prompt processing progress, n_tokens = 12270, batch.n_tokens = 2048, progress = 0.198255
/home/kotokin/llama.cpp/ggml/src/ggml-backend-meta.cpp:1276: GGML_ASSERT(split_state.n_segments == 1) failed
[New LWP 164110]
[New LWP 164109]
[New LWP 164108]
[New LWP 164107]
[New LWP 164106]
[New LWP 164105]
[New LWP 164104]
[New LWP 164103]
[New LWP 164102]
[New LWP 164101]
[New LWP 164100]
[New LWP 164099]
[New LWP 164098]
[New LWP 164097]
[New LWP 164096]
[New LWP 164095]
[New LWP 164094]
[New LWP 164093]
[New LWP 164092]
[New LWP 164091]
[New LWP 164090]
[New LWP 164089]
[New LWP 164088]
[New LWP 164087]
[New LWP 164086]
[New LWP 164085]
[New LWP 164084]
[New LWP 164083]
[New LWP 164082]
[New LWP 164081]
[New LWP 164080]
[New LWP 164073]
[New LWP 164072]
[New LWP 164071]
[New LWP 164070]
[New LWP 164069]
[New LWP 164068]
[New LWP 164067]
[New LWP 164066]
[New LWP 164055]

This GDB supports auto-downloading debuginfo from the following URLs:
  <https://debuginfod.ubuntu.com>
Enable debuginfod for this session? (y or [n]) [answered N; input not from terminal]
Debuginfod has been disabled.
To make this setting permanent, add 'set debuginfod enabled off' to .gdbinit.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
__syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
warning: 56     ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S: No such file or directory
#0  __syscall_cancel_arch () at ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S:56
56      in ../sysdeps/unix/sysv/linux/x86_64/syscall_cancel.S
#1  0x00007072858a013c in __internal_syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=0, a6=0, nr=61) at ./nptl/cancellation.c:49
warning: 49     ./nptl/cancellation.c: No such file or directory
#2  __syscall_cancel (a1=<optimized out>, a2=<optimized out>, a3=<optimized out>, a4=<optimized out>, a5=a5@entry=0, a6=a6@entry=0, nr=61) at ./nptl/cancellation.c:75
75      in ./nptl/cancellation.c
#3  0x000070728591ca0f in __GI___wait4 (pid=<optimized out>, stat_loc=<optimized out>, options=<optimized out>, usage=<optimized out>) at ../sysdeps/unix/sysv/linux/wait4.c:30
warning: 30     ../sysdeps/unix/sysv/linux/wait4.c: No such file or directory
#4  0x0000707285f5a1b3 in ggml_print_backtrace () from /home/kotokin/llama.cpp/build/bin/libggml-base.so.0
#5  0x0000707285f5a35b in ggml_abort () from /home/kotokin/llama.cpp/build/bin/libggml-base.so.0
#6  0x0000707285f7ff93 in ggml_backend_meta_buffer_get_tensor(ggml_backend_buffer*, ggml_tensor const*, void*, unsigned long, unsigned long) () from /home/kotokin/llama.cpp/build/bin/libggml-base.so.0
#7  0x00007072860ce94f in llama_io_write_buffer::write_tensor(ggml_tensor const*, unsigned long, unsigned long) () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#8  0x000070728612ac05 in llama_memory_recurrent::state_write_data(llama_io_write_i&, std::vector<std::pair<unsigned int, unsigned int>, std::allocator<std::pair<unsigned int, unsigned int> > > const&) const () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#9  0x000070728612af5c in llama_memory_recurrent::state_write(llama_io_write_i&, int, unsigned int) const () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#10 0x00007072860bcada in llama_context::state_seq_write_data(llama_io_write_i&, int, unsigned int) () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#11 0x00007072860bcbac in llama_context::state_seq_get_data(int, unsigned char*, unsigned long, unsigned int) () from /home/kotokin/llama.cpp/build/bin/libllama.so.0
#12 0x00005d287a2721dd in server_context_impl::update_slots() ()
#13 0x00005d287a2f2bfe in server_queue::start_loop(long) ()
#14 0x00005d287a1c849a in main ()
[Inferior 1 (process 164054) detached]
Aborted (core dumped)

[misev]

I'm getting the same assert fail with 27B:

llama.cpp/ggml/src/ggml-backend-meta.cpp:1276: GGML_ASSERT(split_state.n_segments == 1) failed

I wonder how it worked for him, maybe there was a regression since 2 days ago?

[nikitabalakin]

I recompiled it in debugging mode, and launched it using gdb. The resulting file was fed to gemini pro, here's his opinion, maybe it will help a little:

Logs

Based on the attached GDB backtrace, the llama-server process aborts with a SIGABRT due to a failed assertion in the GGML backend.

The core issue appears to be an incompatibility between context state serialization (likely related to prompt caching or slot management) and tensor-based multi-GPU splitting (LLAMA_SPLIT_MODE_TENSOR).

Here is the exact trace of the problem:

    The server is running a large context (n_ctx = 131072) with the model Qwen_Qwen3.5-27B-Q8_0.gguf and is actively updating slots via server_context_impl::update_slots.

    During the slot update, the server attempts to retrieve and serialize sequence data by calling llama_state_seq_get_data_ext -> llama_context::state_seq_get_data -> llama_context::state_seq_write_data.

    This propagates down to the memory management layer: llama_memory_hybrid::state_write -> llama_memory_recurrent::state_write -> llama_memory_recurrent::state_write_data.

    To write the tensor data, llama_io_write_buffer::write_tensor calls ggml_backend_tensor_get.

    The crash ultimately occurs in ggml_backend_meta_buffer_get_tensor at ggml/src/ggml-backend-meta.cpp:1276.

The specific failure:
The code asserts that the tensor must be in a single segment:

GGML_ASSERT(split_state.n_segments == 1) failed However, the local variables in the GDB dump clearly show that split_state.n_segments = 5.

My Suspicions / Hypothesis:
Because the server is launched with split_mode = LLAMA_SPLIT_MODE_TENSOR, the tensor is physically sharded across multiple devices (5 segments). When the server tries to dump/serialize the KV cache or state data, the current implementation of llama_memory_recurrent::state_write_data (or the underlying ggml_backend_tensor_get ) does not seem to support gathering a fragmented tensor back into a contiguous buffer. It strictly expects n_segments == 1, which leads to the fatal crash when it encounters the multi-GPU shards.

Steps to reproduce:
Run llama-server with --split-mode tensor, using multiple GPUs, and trigger a scenario where the server attempts to save/extract slot states or prompt cache sequences.

[JohannesGaessler]

Qwen 3.5 26b a4b 27b should be fixed by #21808 .

[nikitabalakin]

Qwen 3.5 26b a4b should be fixed by #21808 .

It also looks like the video memory is overflowing. So much so that in a few hours of use, more than twenty gigabytes are filled.

[JohannesGaessler]

Yes, as I said in #19378 :

There is currently an issue with a VRAM leak due to CUDA graphs being continually created but never destroyed (negligible for -sm layer but not for -sm tensor).

[JohannesGaessler]

Gemma 4 26b a4b should now also work with #21808 .

[lapy]

Hi @JohannesGaessler, you mention Qwen3.5 26B in multiple places but probably you mean Gemma 4 26B , right?

Regarding the bug, I also get GGML_ASSERT(split_state.n_segments == 1) failed with Qwen3.5 27B and derivatives/finetunes.

[JohannesGaessler]

Yes, you're right, I said the wrong parameter counts for Qwen 3.5 27b but the local file I had for that model does work correctly.