Add HyperCLOVA X SEED Think 14B

[bigshanedogg]

It would be great to add native support for HyperCLOVA X SEED Think 14B to the Transformers library, so users can load it without trust_remote_code=True. In addition, this model is intended to serve as the backbone for future multimodal models to be released on the Hugging Face Hub. Without native Transformers support, every new model variant must bundle its own copy of modeling_hyperclovax.py, leading to code duplication, and increased maintenance burden.

Model description

HyperCLOVA X SEED Think 14B is a 14.74B-parameter reasoning LLM developed by NAVER Cloud. It is a LLaMA-style decoder-only transformer with two architectural modifications not present in standard LLaMA:

• Peri-Layer Normalization: an extra RMSNorm is applied after each sub-layer output (in addition to the standard pre-norm), controlled by a use_post_norm config flag.
• Maximal Update Parametrization (μP): per-config scaling factors (attention_multiplier, residual_multiplier, embedding_multiplier, logits_scaling) replace the standard fixed scaling, enabling stable training across model sizes.

The model supports dual-mode reasoning: Think (chain-of-thought before answering) and Non-Think (direct answer), switchable via apply_chat_template(force_reasoning=True/False). It also supports function calling via a custom ChatML dialect. The model is supported in vLLM as of March 2026.

I checked that no existing PR covers this. I have also prepared a draft PR (#44956) in case it is helpful for the discussion or review.

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

• Huggingface hub: naver-hyperclovax/HyperCLOVAX-SEED-Think-14B
• Technical report: arXiv 2506.22403
• vLLM upstream: vllm-project/vllm#37107 (merged 2026-03-16)

[zucchini-nlp]

Same model as in #44314?

[bigshanedogg]

#44314 aims to add a vision-language model, whereas this issue is more focused on native support for the language model standalone.
There is some overlap since #44314 uses HyperCLOVA X as its backbone, but as we originally wrote the LM code in our Hub repository as part of a vision-language model, directly reusing it as a standalone implementation may lead to unintended behavior.

This issue proposes a standalone implementation of the HyperCLOVA X language model with performance verified against the official tech report.

I thought keeping this as a separate model might be helpful for standalone use and benchmark evaluation — but would appreciate any feedback on whether thiseparation makes sense.

[zucchini-nlp]

#44314 uses HyperCLOVA X as its backbone

If the backbone used in 44312 is the same model (acrhitceture-wise), we shouldn't need a separate model class. The VLM model can use HyperClovaX as backbone but we also can load a bare LM for generation if needed. And also from our interaction when adding the VLM, seems like the HyperClovax has the same architecture as Granite model

So if the above is true we can allow users load HyperClovax with transformers with zero efforts using existing Granite model code

[bigshanedogg]

@zucchini-nlp , Thank you for the thoughtful and detailed feedback.
I fully agree with the goal of keeping the codebase lean, and think the suggestion to reuse the Granite model class is a very reasonable one.
That said, during a closer comparison, I found a few areas where additional implementation beyond Granite appears to be necessary. I'd like to share those findings, with the intention of inheriting from Granite as much as possible to minimize code duplication.

---

1. Structural Incompatibility: Peri-Layer Normalization

HyperCLOVAX introduces Peri-Layer Normalization (Peri-LN) — an additional RMSNorm applied after each sub-layer output (both attention and MLP), controlled by the use_post_norm config argument. Since Granite does not have this component, some weights go missing when loading with it.

Attempting to load with Granite produces the following log:

Key                                     | Status     |
----------------------------------------+------------+
model.layers.{0...37}.post_norm2.weight | UNEXPECTED |
model.layers.{0...37}.post_norm1.weight | UNEXPECTED |

These are not extraneous weights — they are load-bearing parameters that affect the forward pass. Since Granite has no equivalent structure, this mismatch is difficult to resolve via a config alias alone.

---

2. Logits Scaling: Opposite Semantics

Both models share a logits_scaling parameter, but apply it in opposite directions:

# Granite
logits = logits / self.config.logits_scaling   # division

# HyperCLOVAX
logits = self.lm_head(...) * self.logits_scaling   # multiplication

The actual HyperCLOVAX checkpoint uses logits_scaling = 0.125. Under Granite's semantics, this becomes a division by 0.125, scaling the logits ×8 — the inverse of what was intended. Under HyperCLOVAX's semantics, it correctly scales logits ×0.125, consistent with the MuP training setup.

---

Observed Output Difference

The differences in 1 and 2 above lead to the following empirically observed divergence in model output:

With GraniteForCausalLM (inverted logits scale + no Peri-LN):

# Model type: <class 'transformers.models.granite.modeling_granite.GraniteForCausalLM'>
# Query: Please explain in detail what you are capable of doing.
# Prediction:  역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 역대 

With HyperCLOVAXForCausalLM (correct implementation):

# Model type: <class 'transformers.models.hyperclovax.modeling_hyperclovax.HyperCLOVAXForCausalLM'>
# Query: Please explain in detail what you are capable of doing.
# Prediction: Okay, so the user wants a detailed explanation of my capabilities. Let me start by breaking down what I can do.

First, I need to outline the main areas where I can assist users. These areas typically include natural language processing (NLP), information retrieval, text generation, and more.

Now, let's dive deeper into each of these key areas where I can provide assistance to users.

In addition, the implementation in this draft PR — with use_post_norm=True and multiplicative logits_scaling — successfully reproduces the benchmark scores reported in the official HyperCLOVAX technical report, confirming that both features are essential for correctness.

---

I agree with minimizing redundant model classes and am happy to follow whatever direction you prefer. That said, the two issues above also apply to #44314, so some form of dedicated handling seems necessary either way.

Since the referenced draft PR reproduces the reported benchmark scores, one humble suggestion would be that #44314 might consider building on this implementation rather than including a separate bare LM implementation within the VLM addition PR — as I hope the performance of our released models can be reliably reproduced for users, not only for HyperCLOVAX (bare LM) but also for HyperCLOVAXVision.

Happy to discuss further. Thank you for your time and thorough review.

[zucchini-nlp]

Ah, so the LLM actually applied the norm but the multimodal one doesn't. I see

We can then add the LM as a separate model and take advantage of modular to copy identical parts. Most modules will be copied from Granite and the decoder layer with norms can be copied from GLM4. The modular should be a couple hundred LOC only with modular and easy to review-iterate

I see that you already have a PR, so I recommend to use modular and make sure CI is passing. Then ping me when you need a review

transformers/src/transformers/models/glm4/modeling_glm4.py

Lines 67 to 69 in b94fee0

	class Glm4DecoderLayer(GradientCheckpointingLayer):
	def __init__(self, config: Glm4Config, layer_idx: int):
	super().__init__()

[bigshanedogg]

Thank you for the clear guidance!
I'll proceed with:

• Copying only the necessary parts from Granite and GLM4
• Structuring the implementation via modular.py

I'll refactor the PR accordingly, ensure CI passes, and ping you when it's ready for review!

[jp1924]

한국어 버전(펼치지/접기)

@bigshanedogg 안녕하세요,

네이버 개발자 분깨서 HCX의 유지보수에 참여해 주신다는 소식을 듣고 기쁩니다.
솔직히 공식 지원이 없다고 판단하여 작업을 시작했었기에 더욱 기쁩니다.

다만 지금, #44314 에서 HyperCLOVA X Vision 모델 PR을 제가 상당 부분 진행한 상태라,
작업을 중단하기에는 저 스스로도 아쉬움이 너무나 큽니다.

그래서 대신 협업을 제안드리고 싶습니다.

현재 #44314의 HyperClovaXForCausalLM으로 naver-hyperclovax/HyperCLOVAX-SEED-Think-14B를 로드해 보면,
post_norm1과 post_norm2 가중치가 UNEXPECTED로 표시됩니다.
@zucchini-nlp 께서 이 스레드에서 지적해 주신 두 가지 차이점인 Peri-Layer Normalization와 multiplicative logits scaling를 반영하면 두 구현은 동일해집니다.

그리고 naver-hyperclovax/HyperCLOVAX-SEED-Think-14B과 naver-hyperclovax/HyperCLOVAX-SEED-Think-32B의 백본이 동일하기 때문에 별도로 진행할 경우 코드 중복이 불가피하다는 것입니다.

그래서 다음과 같은 역할 분담을 제안드립니다

• @jp1924: 코드 작업 및 CI 통과를 진행
• @bigshanedogg: 코드 리뷰어로 참여하여, 아키텍처 정확성 검증 및 코드 제안을 통한 세부 수정 가이드

네이버 개발자 관점에서 검증해 주신다면 PR의 품질이 크게 향상될 것이라 생각합니다.
어떻게 생각하시나요?

Hey @bigshanedogg,

Really glad to hear you're getting involved with HCX maintenance as a Naver dev!
Makes sense since official support seemed to be MIA, so this is great news.

The thing is, I've already put a lot of work into the HyperCLOVA X Vision PR over at #44314, and it'd be a shame to stop now.

How about we collaborate instead?

So right now, when you load naver-hyperclovax/HyperCLOVAX-SEED-Think-14B with HyperClovaXForCausalLM from #44314, the post_norm1 and post_norm2 weights show up as UNEXPECTED. But if we add those two architectural differences @zucchini-nlp mentioned Peri-Layer Normalization and multiplicative logits scaling they should match perfectly.

Plus, since the backbone code for naver-hyperclovax/HyperCLOVAX-SEED-Think-14B and naver-hyperclovax/HyperCLOVAX-SEED-Think-32B is identical, doing them separately just means we end up with duplicate code.

What if we split it up like this?

• @jp1924: Handle the implementation and get CI passing
• @bigshanedogg: Review the code, validate the architecture, and suggest improvements

With the original Naver team's perspective on this, I think we could really nail the PR quality. What do you think?

[bigshanedogg]

@jp1924 ,
Thank you for your interest and contribution to our released models!

We had actually been planning to provide better support following internal requests.
It would also be good to get this settled sooner rather than later, in preparation for upcoming model releases.

As a starting point, we'd suggest proceeding with #44314 importing from #44957 once it's ready, rather than maintaining separate implementations. I'll take a look at your PR and leave comments there.

[bigshanedogg]

Hi @zucchini-nlp,
Just a quick status update — the PR (#44956) is now ready for review with all CI checks passed and benchmark validation is complete.
As we discussed here, this bare LM PR and #44314 (HyperCLOVAX Vision 32B) are intended to be reviewed and merged in sequence.
Since #44314 relies on the native HyperCLOVAX modules added in this PR as its text backbone, landing #44956 first would unblock the Vision PR and avoid duplicating LM code.
No rush — just wanted to share the current status.
Happy to address any feedback when the time comes.
Thank you!
(cc. @jp1924 )

[jp1924]

한국어 버전 (펼치기/접기)

@bigshanedogg 업데이트 감사합니다.

다만 한 가지 명확히 할 점이 있습니다.

#44314는 #44956에 의존하지 않습니다. hyperclovax LM 구현이 이미 #44314 내에 포함되어 있으며, naver-hyperclovax/HyperCLOVAX-SEED-Think-14B 로드도 정상 동작하는 것을 확인한 상태입니다.

또한 #45099 스레드에서 리뷰어 께서 이미 명확하게 방향을 정해주셨습니다:

"Let's review and add the PR already in progress (#44314)"

현재 두 PR의 hyperclovax 구현이 사실상 동일한 상황에서, 리뷰어께서 같은 코드를 두 번 검토하시는 것은 비효율적이라고 생각합니다.

리뷰어의 방향에 따라 #44314를 먼저 진행하는 것이 맞다고 판단됩니다.

@bigshanedogg Thanks for the update.

That said, I'd like to clarify one important point.

#44314 does not depend on #44956. The hyperclovax LM implementation is already included in #44314, and I've verified that loading naver-hyperclovax/HyperCLOVAX-SEED-Think-14B works correctly.

Additionally, reviewer has already provided clear direction in the #45099 thread:

"Let's review and add the PR already in progress (#44314)"

Since both PRs implement essentially the same hyperclovax architecture, having the reviewer examine the same code twice would be inefficient.

Following the reviewer's guidance, I believe #44314 should proceed first.

[jp1924]

@bigshanedogg Do you have an email or Slack/DM where we could chat? It's a bit challenging to cover all the context through GitHub comments.