UPSTREAM PR #21421: mtmd: add Gemma 4 audio conformer encoder support

[loci-dev]

Note

Source pull request: ggml-org/llama.cpp#21421

Overview

Add audio processing support for Gemma 4 models via a USM-style Conformer encoder.

Architecture:

• 12-layer Conformer: FFN -> Self-Attention -> Causal Conv1D -> FFN -> Norm
• Subsampling Conv Projection: 2x Conv2D(stride=2) with LayerNorm
• Chunked local attention with sinusoidal RPE (chunk_size=12, context_size=24)
• Logit softcapping at 50.0, ClippableLinear with per-tensor clamping
• Output projection -> RMSNorm -> multimodal embedder

Chunked local attention (matching PyTorch reference):

• Q split into non-overlapping blocks of 12
• K/V extracted as overlapping context windows of 24 via ggml_view_4d with stride 12
• Per-block causal mask: query at position q only attends to keys at positions <= q
• Blocked relative position shift (Transformer-XL appendix B)
• RPE: 13 sinusoidal position embeddings [12, 11, ..., 0]

Mel preprocessing (dedicated mtmd_audio_preprocessor_gemma4a):

• HTK mel scale, 128 bins, magnitude STFT, mel_floor=1e-3
• Standard periodic Hann window (320 samples), zero-padded to FFT size
• Semicausal left-padding (frame_length/2 samples)
• Frame count matched to PyTorch unfold formula
• No pre-emphasis, no Whisper-style normalization
• 30-second chunking (splits long audio into 30s segments)
• Mel cosine similarity vs PyTorch: 0.9998

Key fixes:

• Tensor loading dedup: get_tensor() now throws on duplicate tensor names (via std::unordered_set guard), and GEMMA4A-specific tensor loading no longer re-loads tensors already handled by the generic per-layer loop.
• ClippableLinear clamp_info loading moved after per-layer tensor loading to ensure all conformer weights have clamp data.
• Skip Whisper-style (x+4)/4 normalization for Gemma4 raw log-mel output.
• CUDA ggml_ssm_conv: added kernel_size=5 support (Gemma4 depthwise Conv1D uses kernel_size=5, previously only 3,4,9 were supported).

Test results (E4B Q4_K_M):

LibriSpeech test samples:

Ground truth: "MISTER QUILTER IS THE APOSTLE OF THE MIDDLE CLASSES AND WE ARE GLAD TO WELCOME HIS GOSPEL"
E4B output:   "Mr. Coulter is the apostle of the middle classes, and we are glad to welcome his gospel."

Ground truth: "NOR IS MISTER QUILTER'S MANNER LESS INTERESTING THAN HIS MATTER"
E4B output:   "Norris Mr. Coulter's manner less interesting than his manner."

Ground truth: "HE TELLS US THAT AT THIS FESTIVE SEASON OF THE YEAR WITH CHRISTMAS AND ROAST BEEF..."
E4B output:   "He tells us that this festive season of the year, with Christmas and New Year looming before us..."

Generation parameters (from model's generation_config.json):
--temp 1.0 --top-k 64 --top-p 0.95

Additional information

Test plan:

• test-mtmd-c-api passes
• test-llama-archs passes (gemma4 fixture added, skipped pending ISWA KV cache fix)
• E4B Q4_K_M transcription (Vulkan Intel Iris Xe, CUDA RTX 3060, CUDA Tesla T4)
• E2B Q4_K_M transcription (CUDA RTX 3060, CUDA Tesla T4)
• LibriSpeech samples with known ground truth
• Mel values verified against PyTorch (cosine 0.9998)
• Encoder output cosine vs PyTorch: 0.68 (expected for F16 through 12 conformer layers)
• CI ctest: 49/49 debug passed
• CUDA ssm_conv kernel_size=5 tested on RTX 3060 and Tesla T4

Ref: #21325

Requirements

• I have read and agree with the contributing guidelines
• AI usage disclosure: YES - Claude Code was used in an assistive capacity for iterative debugging (tensor tracing, mel spectrogram comparison) and code review. All architecture decisions, algorithm implementations, and code were manually reviewed and verified against the PyTorch reference.

[loci-review]

Overview

Analysis of 125,410 functions across 15 binaries shows 34 modified functions with performance changes isolated to the multimodal library. Changes add Gemma 4 audio conformer encoder support with duplicate tensor detection and safe map lookups.

Function counts: 34 modified, 290 new, 3 removed, 125,083 unchanged

Power consumption by binary:

Binary	Base (nJ)	Target (nJ)	Change
build.bin.libmtmd.so	196,832.42	201,026.65	+2.13%
build.bin.libllama.so	264,739.36	264,739.10	-0.00%
build.bin.llama-cvector-generator	367,100.44	367,098.51	-0.00%
build.bin.llama-tts	372,435.36	372,435.17	-0.00%
build.bin.llama-bench	160,327.90	160,328.23	+0.00%
build.bin.llama-quantize	44,468.23	44,468.23	0.00%
build.bin.llama-qwen2vl-cli	277.87	277.87	0.00%
build.bin.llama-tokenize	38,388.75	38,388.75	0.00%
build.bin.llama-gemma3-cli	277.87	277.87	0.00%
build.bin.llama-gguf-split	2,864.08	2,864.08	0.00%
build.bin.llama-llava-cli	277.87	277.87	0.00%
build.bin.llama-minicpmv-cli	277.87	277.87	0.00%
build.bin.libggml-cpu.so	177,792.05	177,792.05	0.00%
build.bin.libggml.so	5,136.91	5,136.91	0.00%
build.bin.libggml-base.so	74,169.64	74,169.64	0.00%

Function Analysis

clip_model_loader::load_tensors (build.bin.libmtmd.so)

• Response time: 446,935 ns → 1,035,599 ns (+588,664 ns, +132%)
• Throughput time: 5,468 ns → 5,851 ns (+384 ns, +7%)
• Adds ~150 lines for GEMMA4A audio conformer with 40+ specialized tensors per layer, duplicate detection via std::unordered_set, and safe iterator-based map lookups. Regression is feature-driven, not inefficiency.

clip_init (build.bin.libmtmd.so)

• Response time: 1,180,428 ns → 2,356,773 ns (+1,176,345 ns, +100%)
• Throughput time: 275 ns → 258 ns (-17 ns, -6%)
• Regression entirely driven by load_tensors dependency. Function's own code improved 6%, demonstrating efficient implementation.

Lambda operator in load_tensors (build.bin.libmtmd.so)

• Response time: 3,873 ns → 11,555 ns (+7,682 ns, +198%)
• Throughput time: 242 ns → 356 ns (+114 ns, +47%)
• Switches from vector storage to hash-based duplicate detection. Adds unordered_set::count() (+1,271 ns) and insert() (+4,616 ns) to prevent silent model corruption with 50+ new tensor types.

filter_params constructor (build.bin.libmtmd.so)

• Response time: 27 ns → 50 ns (+23 ns, +88%)
• Adds 3 fields for Gemma4 audio preprocessing: no_padding, use_magnitude, mel_floor. Proportional overhead for expanded functionality.

std::vector::end() (build.bin.libmtmd.so)

• Response time: 265 ns → 82 ns (-183 ns, -69%)
• Compiler optimization reduced CFG from 9 to 7 blocks with 89% faster entry block execution.

Other analyzed functions show regressions of 9-40% in memory allocation and vector operations due to clip_layer struct expansion (+40 bytes for 5 new tensor pointers), with negligible absolute overhead.

Flame Graph Comparison

Comparing flame graphs for clip_model_loader::load_tensors to illustrate the structural changes from vector-based to hash-based tensor tracking:

Base version:

Target version:

The target version shows 2.3x increase in execution time with new hash table operations (_M_insert, _M_bucket_index, _M_rehash) replacing simple vector operations, enabling robust duplicate detection for complex audio conformer models.

Additional Findings

All performance regressions occur in non-critical initialization code (model loading, one-time setup). Core inference libraries (libllama.so, libggml.so, libggml-cpu.so) show zero power consumption change, confirming inference hot paths are completely unaffected. The changes prioritize correctness over initialization speed—duplicate detection prevents silent model corruption, and safe lookups prevent crashes. Absolute overhead (~1.8 ms) is negligible in multi-second model loading operations. Chunked local attention reduces inference complexity from O(n²) to O(n·chunk_size), providing 50% computation reduction that will offset initialization costs during actual inference.

💬 Questions? Tag @loci-dev

[loci-review]

The analysis encountered an error. Please review the Processing Details for more information.