[Feature] Qwen3 VL eagle3 support

[dcw02]

Motivation

** Draft PR. This is currently WIP **

Add eagle3 support for qwen3_vl and qwen3_vl_moe models.

Modifications

Related Issues

Accuracy Test

Benchmark & Profiling

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
• For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
• Please feel free to join our Slack channel at https://sgl-fru7574.slack.com/archives/C09784E3EN6 to discuss your PR.

[dcw02]

Training finishes for tp size 1 and both sdpa/flex attention backends. Loss/acc curves look ok, going to add qwen3_vl_moe eagle3 support into sglang/vllm (so I can eval) before adding tp size > 1 support.

[dcw02]

Marking this ready for review, adds qwen3_vl and qwen3_vl_moe EAGLE3 draft training support for tp size 1. SGLang support is being worked on/cleaned up.

Training graphs:

• qwen3-vl-8b-instruct
• qwen3-vl-30b-a3b-instruct

[kevin19891229]

When will verification be supported?

[dcw02]

When will verification be supported?

There's a branch of sglang here that you can run. It's currently being cleaned up for upstreaming. We were able to confirm an accept length of almost 5 for a spec config of 4, 6, 24 for one of our tasks for qwen3-vl-8b.

[FrankLeeeee]

Can you rebase this PR to the latest main?

[dcw02]

Can you rebase this PR to the latest main?

yes I'll get to it this week

[dcw02]

sorry for the delay @FrankLeeeee I finished the rebase

[FrankLeeeee]

@KerwinKai this PR seems to overlap with yours, can you take a look?

[KerwinKai]

@KerwinKai this PR seems to overlap with yours, can you take a look?

Yes, I'll try.

[ooolmk]

` elif (
args.is_vlm
and draft_model_config.target_model_type == "qwen3_vl"
and args.tp_size == 1
):
from transformers import Qwen3VLForConditionalGeneration

        target_model = (
            Qwen3VLForConditionalGeneration.from_pretrained(
                pretrained_model_name_or_path=args.target_model_path,
                dtype=torch.bfloat16,
            )
            .eval()
            .cuda()
        )`

Initialize the target model using Qwen3VLForConditionalGeneration from the Transformers library, but the class definition does not include set_aux_hidden_states_layers, causing the error:
AttributeError: 'Qwen3VLForConditionalGeneration' object has no attribute 'set_aux_hidden_states_layers'.

How should I modify Qwen3VLForConditionalGeneration? I noticed there is a class Eagle3TargetModel(ABC), but I’m not sure how to use it.

[dcw02]

` elif ( args.is_vlm and draft_model_config.target_model_type == "qwen3_vl" and args.tp_size == 1 ): from transformers import Qwen3VLForConditionalGeneration

        target_model = (
            Qwen3VLForConditionalGeneration.from_pretrained(
                pretrained_model_name_or_path=args.target_model_path,
                dtype=torch.bfloat16,
            )
            .eval()
            .cuda()
        )`

Initialize the target model using Qwen3VLForConditionalGeneration from the Transformers library, but the class definition does not include set_aux_hidden_states_layers, causing the error: AttributeError: 'Qwen3VLForConditionalGeneration' object has no attribute 'set_aux_hidden_states_layers'.

How should I modify Qwen3VLForConditionalGeneration? I noticed there is a class Eagle3TargetModel(ABC), but I’m not sure how to use it.

you should specify backend as hf rather than sglang backend

[ooolmk]

Initialize the target model using Qwen3VLForConditionalGeneration from the Transformers library, but the class definition does not include set_aux_hidden_states_layers, causing the error: AttributeError: 'Qwen3VLForConditionalGeneration' object has no attribute 'set_aux_hidden_states_layers'.
How should I modify Qwen3VLForConditionalGeneration? I noticed there is a class Eagle3TargetModel(ABC), but I’m not sure how to use it.

you should specify backend as hf rather than sglang backend

Thanks a lot! But I finally resolved the 'set_aux_hidden_states_layers' issue by
target_model = HFEagle3TargetModel(target_model) and tune off all the fsdp & dp config
I successfully trained a qwen3-vl-2b-eagle3 model based on your commits. It can now perform inference on SLGANG. I'm preparing to switch to a larger dataset to further validate its performance.

[yossigandelsman]

Any update on this issue?

[mmdbhs]

Initialize the target model using Qwen3VLForConditionalGeneration from the Transformers library, but the class definition does not include set_aux_hidden_states_layers, causing the error: AttributeError: 'Qwen3VLForConditionalGeneration' object has no attribute 'set_aux_hidden_states_layers'.
How should I modify Qwen3VLForConditionalGeneration? I noticed there is a class Eagle3TargetModel(ABC), but I’m not sure how to use it.

you should specify backend as hf rather than sglang backend

Thanks a lot! But I finally resolved the 'set_aux_hidden_states_layers' issue by target_model = HFEagle3TargetModel(target_model) and tune off all the fsdp & dp config I successfully trained a qwen3-vl-2b-eagle3 model based on your commits. It can now perform inference on SLGANG. I'm preparing to switch to a larger dataset to further validate its performance.

[rank0]: File "/mnt/hwnas/xuyingbing/SpecForge/specforge/core/eagle3.py", line 380, in _prepare_data
[rank0]: outputs = self.target_model(**filtered_target_kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: TypeError: 'HFEagle3TargetModel' object is not callable
how do you fix it?

[ooolmk]

[rank0]: File "/mnt/hwnas/xuyingbing/SpecForge/specforge/core/eagle3.py", line 380, in _prepare_data [rank0]: outputs = self.target_model(**filtered_target_kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: TypeError: 'HFEagle3TargetModel' object is not callable how do you fix it?

target_model should be Qwen3VLForConditionalGeneration instead of HFEagle3TargetModel
so use targetmodel.model when init QwenVLOnlineEagle3Model：
eagle3_model = QwenVLOnlineEagle3Model(
target_model=target_model.model,
draft_model=draft_model,
processor=processor,
length=args.ttt_length,
attention_backend=args.attention_backend,
target_model_type=getattr(draft_model_config, "target_model_type", None),
)

[mmdbhs]

QwenVLOnlineEagle3Model

I have a better solution; you just need to comment out the code below

[pyzhang2008]

any update on this issue？

[dcw02]

closing because #431 exists!