Feature Request: Add NVFP4 tensor mapping for GEMMA4 architecture

[rnett]

Prerequisites

• I am running the latest code.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue.
• I reviewed the Discussions.

Feature Description

The GEMMA4 architecture implementation in llama.cpp (specifically the C++ loader) does not yet map GGML_TYPE_NVFP4 weights to internal architectural slots. Even with a valid NVFP4 GGUF, the loader fails with a tensor count mismatch because it only recognizes the FP32/BF16 tensors.

Motivation

NVFP4 is the native 4-bit format for NVIDIA Blackwell GPUs. Supporting this mapping in the GEMMA4 architecture is essential for leveraging hardware acceleration on RTX 50-series and B200 hardware.

Possible Implementation

Update the architectural registry for GEMMA4 to accept GGML_TYPE_NVFP4 for weights and their associated scale tensors.

[mollahmasud977-sys]

Thanks for the detailed report and clear explanation. I can see why the current GEMMA4 implementation fails to recognize NVFP4 tensors—the loader indeed only expects BF16/F32, which explains the tensor count mismatch. في السبت، 11 أبريل 2026 22:19 Ryan Nett ***@***.***> كتب:

…

*rnett* created an issue (ggml-org/llama.cpp#21777) <#21777> Prerequisites - I am running the latest code. - I carefully followed the README.md. - I searched using keywords relevant to my issue. - I reviewed the Discussions. Feature Description The current \GEMMA4\ architecture implementation in \llama.cpp\ does not map \GGML_TYPE_NVFP4\ weights to its internal architectural slots. Even when a correctly formatted NVFP4 GGUF is generated (e.g., using the latest \convert_hf_to_gguf.py), the loader fails with a tensor count mismatch error. Specifically, for Gemma 4 E2B, the engine expects 1093 valid tensors but only "sees" the BF16/F32 ones (approx 541), ignoring the NVFP4 ones. Motivation NVFP4 is the hardware-native 4-bit floating point format for *NVIDIA Blackwell* GPUs (RTX 50-series). Adding support for this in the \GEMMA4\ architecture will allow users to leverage hardware acceleration on the latest NVIDIA hardware, potentially offering significant throughput improvements over standard integer-based quantization. Possible Implementation Update \gemma.cpp\ (or the \GEMMA4\ architectural registry) to include \GGML_TYPE_NVFP4\ in the list of allowed tensor types for weights and their associated scales (.weight.scale, .weight.input_scale). — Reply to this email directly, view it on GitHub <#21777>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/CBTQ2IVS6KBUPZWNNKU6YFD4VKZEVAVCNFSM6AAAAACXVGHHIKVHI2DSMVQWIX3LMV43ASLTON2WKOZUGI2DKOBQG4YDIMY> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>