Eval bug: Gemma 4 audio support is missing

[coder543]

Name and Version

version: 8638 (5803c8d)
built with GNU 13.3.0 for Linux aarch64

Operating systems

Linux

GGML backends

CUDA

Hardware

DGX Spark

Models

ggml-org/gemma-4-E4B-it-GGUF/gemma-4-e4b-it-f16.gguf

Problem description & steps to reproduce

webui claims this model does not support audio input

This model definitely should support audio input.

First Bad Commit

No response

Relevant log output

Logs

ggml_cuda_init: found 1 CUDA devices (Total VRAM: 124610 MiB):
  Device 0: NVIDIA GB10, compute capability 12.1, VMM: yes, VRAM: 124610 MiB
system info: n_threads = 20, n_threads_batch = 20, total_threads = 20

system_info: n_threads = 20 (n_threads_batch = 20) / 20 | CUDA : ARCHS = 1210 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | FA_ALL_QUANTS = 1 | BLACKWELL_NATIVE_FP4 = 1 | CPU : NEON = 1 | ARM_FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

Running without SSL
init: using 19 threads for HTTP server
start: binding port with default address family
main: loading model
srv    load_model: loading model '/home/coder/models/gemma-4-e4b-it-f16.gguf'
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GB10) (000f:01:00.0) - 119796 MiB free
llama_model_loader: loaded meta data with 44 key-value pairs and 720 tensors from /home/coder/models/gemma-4-e4b-it-f16.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = gemma4
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                     general.sampling.top_k i32              = 64
llama_model_loader: - kv   3:                     general.sampling.top_p f32              = 0.950000
llama_model_loader: - kv   4:                      general.sampling.temp f32              = 1.000000
llama_model_loader: - kv   5:                         general.size_label str              = 7.5B
llama_model_loader: - kv   6:                            general.license str              = apache-2.0
llama_model_loader: - kv   7:                       general.license.link str              = https://ai.google.dev/gemma/docs/gemm...
llama_model_loader: - kv   8:                               general.tags arr[str,1]       = ["any-to-any"]
llama_model_loader: - kv   9:                         gemma4.block_count u32              = 42
llama_model_loader: - kv  10:                      gemma4.context_length u32              = 131072
llama_model_loader: - kv  11:                    gemma4.embedding_length u32              = 2560
llama_model_loader: - kv  12:                 gemma4.feed_forward_length u32              = 10240
llama_model_loader: - kv  13:                gemma4.attention.head_count u32              = 8
llama_model_loader: - kv  14:             gemma4.attention.head_count_kv u32              = 2
llama_model_loader: - kv  15:                      gemma4.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  16:                  gemma4.rope.freq_base_swa f32              = 10000.000000
llama_model_loader: - kv  17:    gemma4.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  18:                gemma4.attention.key_length u32              = 512
llama_model_loader: - kv  19:              gemma4.attention.value_length u32              = 512
llama_model_loader: - kv  20:                          general.file_type u32              = 1
llama_model_loader: - kv  21:             gemma4.final_logit_softcapping f32              = 30.000000
llama_model_loader: - kv  22:            gemma4.attention.sliding_window u32              = 512
llama_model_loader: - kv  23:          gemma4.attention.shared_kv_layers u32              = 18
llama_model_loader: - kv  24:    gemma4.embedding_length_per_layer_input u32              = 256
llama_model_loader: - kv  25:    gemma4.attention.sliding_window_pattern arr[bool,42]     = [true, true, true, true, true, false,...
llama_model_loader: - kv  26:            gemma4.attention.key_length_swa u32              = 256
llama_model_loader: - kv  27:          gemma4.attention.value_length_swa u32              = 256
llama_model_loader: - kv  28:                gemma4.rope.dimension_count u32              = 512
llama_model_loader: - kv  29:            gemma4.rope.dimension_count_swa u32              = 256
llama_model_loader: - kv  30:               general.quantization_version u32              = 2
llama_model_loader: - kv  31:                       tokenizer.ggml.model str              = gemma4
llama_model_loader: - kv  32:                      tokenizer.ggml.tokens arr[str,262144]  = ["<pad>", "<eos>", "<bos>", "<unk>", ...
llama_model_loader: - kv  33:                      tokenizer.ggml.scores arr[f32,262144]  = [-1000.000000, -1000.000000, -1000.00...
llama_model_loader: - kv  34:                  tokenizer.ggml.token_type arr[i32,262144]  = [3, 3, 3, 3, 3, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  35:                      tokenizer.ggml.merges arr[str,514906]  = ["\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n \n", ...
llama_model_loader: - kv  36:                tokenizer.ggml.bos_token_id u32              = 2
llama_model_loader: - kv  37:                tokenizer.ggml.eos_token_id u32              = 1
llama_model_loader: - kv  38:            tokenizer.ggml.unknown_token_id u32              = 3
llama_model_loader: - kv  39:            tokenizer.ggml.padding_token_id u32              = 0
llama_model_loader: - kv  40:               tokenizer.ggml.mask_token_id u32              = 4
llama_model_loader: - kv  41:                    tokenizer.chat_template str              = {%- macro format_parameters(propertie...
llama_model_loader: - kv  42:            tokenizer.ggml.add_space_prefix bool             = false
llama_model_loader: - kv  43:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - type  f32:  339 tensors
llama_model_loader: - type  f16:  381 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = F16
print_info: file size   = 14.00 GiB (16.00 BPW) 
load: 0 unused tokens
load: control-looking token:    212 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 1 ('<eos>')
load:   - 106 ('<turn|>')
load:   - 212 ('</s>')
load: special tokens cache size = 25
load: token to piece cache size = 1.9445 MB
print_info: arch                  = gemma4
print_info: vocab_only            = 0
print_info: no_alloc              = 0
print_info: n_ctx_train           = 131072
print_info: n_embd                = 2560
print_info: n_embd_inp            = 2560
print_info: n_layer               = 42
print_info: n_head                = 8
print_info: n_head_kv             = 2
print_info: n_rot                 = 512
print_info: n_swa                 = 512
print_info: is_swa_any            = 1
print_info: n_embd_head_k         = 512
print_info: n_embd_head_v         = 512
print_info: n_gqa                 = 4
print_info: n_embd_k_gqa          = [512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024]
print_info: n_embd_v_gqa          = [512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024, 512, 512, 512, 512, 512, 1024]
print_info: f_norm_eps            = 0.0e+00
print_info: f_norm_rms_eps        = 1.0e-06
print_info: f_clamp_kqv           = 0.0e+00
print_info: f_max_alibi_bias      = 0.0e+00
print_info: f_logit_scale         = 0.0e+00
print_info: f_attn_scale          = 1.0e+00
print_info: n_ff                  = 10240
print_info: n_expert              = 0
print_info: n_expert_used         = 0
print_info: n_expert_groups       = 0
print_info: n_group_used          = 0
print_info: causal attn           = 1
print_info: pooling type          = -1
print_info: rope type             = 2
print_info: rope scaling          = linear
print_info: freq_base_train       = 1000000.0
print_info: freq_scale_train      = 1
print_info: freq_base_swa         = 10000.0
print_info: freq_scale_swa        = 1
print_info: n_embd_head_k_swa     = 256
print_info: n_embd_head_v_swa     = 256
print_info: n_rot_swa             = 256
print_info: n_ctx_orig_yarn       = 131072
print_info: rope_yarn_log_mul     = 0.0000
print_info: rope_finetuned        = unknown
print_info: model type            = E4B
print_info: model params          = 7.52 B
print_info: general.name          = n/a
print_info: vocab type            = SPM
print_info: n_vocab               = 262144
print_info: n_merges              = 0
print_info: BOS token             = 2 '<bos>'
print_info: EOS token             = 1 '<eos>'
print_info: UNK token             = 3 '<unk>'
print_info: PAD token             = 0 '<pad>'
print_info: MASK token            = 4 '<mask>'
print_info: LF token              = 248 '<0x0A>'
print_info: EOG token             = 1 '<eos>'
print_info: EOG token             = 106 '<turn|>'
print_info: EOG token             = 212 '</s>'
print_info: max token length      = 93
load_tensors: loading model tensors, this can take a while... (mmap = false, direct_io = false)
str: cannot properly format tensor name output with suffix=weight bid=-1 xid=-1
load_tensors: offloading output layer to GPU
load_tensors: offloading 41 repeating layers to GPU
load_tensors: offloaded 43/43 layers to GPU
load_tensors:        CUDA0 model buffer size = 14340.66 MiB
load_tensors:    CUDA_Host model buffer size =  1280.00 MiB
....................................................
common_init_result: added <eos> logit bias = -inf
common_init_result: added <turn|> logit bias = -inf
common_init_result: added </s> logit bias = -inf
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 131072
llama_context: n_ctx_seq     = 131072
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 2048
llama_context: causal_attn   = 1
llama_context: flash_attn    = enabled
llama_context: kv_unified    = false
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context:  CUDA_Host  output buffer size =     1.00 MiB
llama_kv_cache_iswa: creating non-SWA KV cache, size = 131072 cells
llama_kv_cache:      CUDA0 KV buffer size =  2048.00 MiB
llama_kv_cache: size = 2048.00 MiB (131072 cells,   4 layers,  1/1 seqs), K (f16): 1024.00 MiB, V (f16): 1024.00 MiB
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0
llama_kv_cache_iswa: creating     SWA KV cache, size = 2560 cells
llama_kv_cache:      CUDA0 KV buffer size =   100.00 MiB
llama_kv_cache: size =  100.00 MiB (  2560 cells,  20 layers,  1/1 seqs), K (f16):   50.00 MiB, V (f16):   50.00 MiB
llama_kv_cache: attn_rot_k = 0
llama_kv_cache: attn_rot_v = 0
sched_reserve: reserving ...
sched_reserve: resolving fused Gated Delta Net support:
sched_reserve: fused Gated Delta Net (autoregressive) enabled
sched_reserve: fused Gated Delta Net (chunked) enabled
sched_reserve:      CUDA0 compute buffer size =  2068.00 MiB
sched_reserve:  CUDA_Host compute buffer size =  1084.09 MiB
sched_reserve: graph nodes  = 1867
sched_reserve: graph splits = 2
sched_reserve: reserve took 345.38 ms, sched copies = 1
clip_model_loader: model name:   
clip_model_loader: description:  
clip_model_loader: GGUF version: 3
clip_model_loader: alignment:    32
clip_model_loader: n_tensors:    1411
clip_model_loader: n_kv:         32

clip_model_loader: has vision encoder
clip_model_loader: has audio encoder
clip_ctx: CLIP using CUDA0 backend
load_hparams: projector:          gemma4v
load_hparams: n_embd:             768
load_hparams: n_head:             12
load_hparams: n_ff:               3072
load_hparams: n_layer:            16
load_hparams: ffn_op:             gelu_quick
load_hparams: projection_dim:     2560

--- vision hparams ---
load_hparams: image_size:         224
load_hparams: patch_size:         16
load_hparams: has_llava_proj:     0
load_hparams: minicpmv_version:   0
load_hparams: n_merge:            3
load_hparams: n_wa_pattern: 0
load_hparams: image_min_pixels:   580608
load_hparams: image_max_pixels:   645120

load_hparams: model size:         944.39 MiB
load_hparams: metadata size:      0.50 MiB
srv    load_model: loaded multimodal model, '/home/coder/models/gemma-4-e4b-it-mmproj-f16.gguf'
srv    load_model: speculative decoding is not supported by multimodal, it will be disabled
srv    load_model: initializing slots, n_slots = 1
no implementations specified for speculative decoding
slot   load_model: id  0 | task -1 | speculative decoding context not initialized
slot   load_model: id  0 | task -1 | new slot, n_ctx = 131072
srv    load_model: prompt cache is enabled, size limit: 8192 MiB
srv    load_model: use `--cache-ram 0` to disable the prompt cache
srv    load_model: for more info see https://github.com/ggml-org/llama.cpp/pull/16391
init: chat template, example_format: '<bos><|turn>system
<|think|>You are a helpful assistant<turn|>
<|turn>user
Hello<turn|>
<|turn>model
Hi there<turn|>
<|turn>user
How are you?<turn|>
<|turn>model
'
srv          init: init: chat template, thinking = 1
main: model loaded
main: server is listening on http://0.0.0.0:5809
main: starting the main loop...
srv  update_slots: all slots are idle

[iz0eyj]

@coder543
Yes it is, I'm using the latest llama.cpp version (both Vulcan and ROCm) , and I get unknown model architecture: 'gemma4'

[coder543]

@iz0eyj No... the latest version works fine with the architecture. It also has nothing to do with audio support being missing.

Your issue is this one: #21320

And the issue is that you're using an outdated version of llama-server. You have to build it yourself.

[iz0eyj]

@coder543
I honestly don't understand the controversy.
The version I'm using is the b8635 binary, which is the latest in the repository, and just like you, I'm getting an error message indicating an unsupported version.
No problem, though; if it bothers you, I'll delete my report.

Done.

[coder543]

@iz0eyj I'm not getting an error about an unsupported version. I'm just not able to attach audio files as input. The model works fine. I can chat with it and get responses.

And again, if you look at the issue I linked, the issue is that the release binaries are out of date. The build server is running slow, so you have to build it yourself, like I did. I am on this version: 8638 (5803c8d)

Which is newer than 8635 that you have.

[iz0eyj]

@coder543
It's the same, the entire Gemma 4 is unsupported.
But it doesn't matter, as you can see I deleted the bug report.
Peace & Good.

[NickM-27]

It is likely a cached build that is being used, I have been running Gemma4 with my docker build on my branch

[delarun]

+1
Audio not detected with loaded mmproj
(builded latest a1cfb64)

[crashr]

{"error":{"code":500,"message":"audio input is not supported - hint: if this is unexpected, you may need to provide the mmproj","type":"server_error"}}

#21309

[pfn]

:~/src/llama.cpp/build-local/build/bin$ ./llama-mtmd-cli --jinja -m gemma-4-E4B-it-GGUF-Q8_K/gemma-4-E4B-it-UD-Q8_K_XL.gguf --mmproj gemma-4-E4B-it-GGUF-Q8_K/mmproj-BF16.gguf --audio ~/Music/some-song.flac -p "what do you hear?"
... snip ...
main: loading model: gemma-4-E4B-it-GGUF-Q8_K/gemma-4-E4B-it-UD-Q8_K_XL.gguf
WARN: This is an experimental CLI for testing multimodal capability.
      For normal use cases, please use the standard llama-cli
This model does not support audio input

tried using llama-mtmd-cli to eliminate everything else, the model reports audio is not supported

gemma-4-E4B-it-GGUF-Q8_K$ ~/python/bin/gguf-dump mmproj-BF16.gguf
INFO:gguf-dump:* Loading: mmproj-BF16.gguf
* File is LITTLE endian, script is running on a LITTLE endian host.
* Dumping 39 key/value pair(s)
... snip ...
     18: BOOL       |        1 | clip.has_vision_encoder = True
     19: UINT32     |        1 | clip.vision.projection_dim = 2560
     20: UINT32     |        1 | clip.vision.image_size = 224
     21: UINT32     |        1 | clip.vision.patch_size = 16
     22: UINT32     |        1 | clip.vision.embedding_length = 768
     23: UINT32     |        1 | clip.vision.feed_forward_length = 3072
     24: UINT32     |        1 | clip.vision.block_count = 16
     25: UINT32     |        1 | clip.vision.attention.head_count = 12
     26: [FLOAT32]  |        3 | clip.vision.image_mean = [0.0, 0.0, 0.0]
     27: [FLOAT32]  |        3 | clip.vision.image_std = [1.0, 1.0, 1.0]
     28: BOOL       |        1 | clip.has_audio_encoder = True
     29: UINT32     |        1 | clip.audio.projection_dim = 2560
     30: UINT32     |        1 | clip.audio.embedding_length = 1024
     31: UINT32     |        1 | clip.audio.feed_forward_length = 4096
     32: UINT32     |        1 | clip.audio.block_count = 12
     33: UINT32     |        1 | clip.audio.attention.head_count = 8
     34: STRING     |        1 | clip.vision.projector_type = 'gemma4v'
     35: FLOAT32    |        1 | clip.vision.attention.layer_norm_epsilon = 9.999999974752427e-07
     36: STRING     |        1 | clip.audio.projector_type = 'gemma4a'
     37: UINT32     |        1 | clip.audio.num_mel_bins = 128
     38: FLOAT32    |        1 | clip.audio.attention.layer_norm_epsilon = 9.999999747378752e-06
     39: UINT32     |        1 | general.quantization_version = 2
* Dumping 1411 tensor(s)
... snip ...

https://github.com/ggml-org/llama.cpp/blob/master/tools/mtmd/clip.cpp#L2438 <-- intentionally disabled, not supported yet

[kth8]

ngxson is working on it

#21309