refactor(vLLM): Move video support from example to backend

[rmccorm4]

Overview:

• replace model-name allowlists with capability-driven vision loading and multimodal handling
• add native video_url loading in the standard TokensPrompt multi_modal_data flow
• move the video agg/disagg launch scripts under examples/backends/vllm and update docs/tests
• Remove old Llava video model support for simplicity until explicitly requested

Details:

Quick Benchmark: Dynamo vs vllm serve for Video Inference

I ran a quick apples-to-apples comparison between Dynamo aggregate mode (examples/backends/vllm/launch/video_agg.sh) and plain vllm serve, both serving Qwen/Qwen2-VL-2B-Instruct on the same machine and GPU configuration.

Dynamo command:

bash examples/backends/vllm/launch/video_agg.sh \
    --model Qwen/Qwen3-VL-2B-Instruct

vllm serve command:

vllm serve \
    --model Qwen/Qwen3-VL-2B-Instruct \
    --served-model-name Qwen/Qwen3-VL-2B-Instruct \
    --host 0.0.0.0 \
    --port 8000 \
    --max-model-len 8192 \
    --allowed-local-media-path / \
    --limit-mm-per-prompt '{"video":1}' \
    --media-io-kwargs '{"video":{"num_frames":32}}'

Benchmark command:

aiperf profile \
    --model Qwen/Qwen3-VL-2B-Instruct \
    --endpoint-type chat \
    --endpoint /v1/chat/completions \
    --url localhost:8000 \
    --video-width 640 \
    --video-height 480 \
    --video-fps 4 \
    --video-duration 5.0 \
    --video-format mp4 \
    --video-codec libx264 \
    --request-count 20 \
    --concurrency 1 \
    --osl 1200 \
    --osl-stddev 0 \
    --extra-inputs '{"ignore_eos": true, "min_tokens": 1200}' \
    --use-server-token-count \
    --ui none \
    --no-server-metrics \
    --no-gpu-telemetry

Both runs completed successfully with identical prompt/completion lengths:

• Average ISL: 962
• Average OSL: 1200
• Success rate: 20/20

Deployment	Concurrency	Avg latency	Req/s	Output tok/s	Benchmark duration
vLLM serve	1	6459.49 ms	0.1548	185.72	129.23 s
Dynamo + vLLM	1	6483.57 ms	0.1542	185.02	129.71 s
vLLM serve	2	7295.91 ms	0.2740	328.85	72.98 s
Dynamo + vLLM	2	7341.10 ms	0.2724	326.82	73.43 s

Where should the reviewer start?

Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)

• closes GitHub issue: #xxx

Future

• Can probably condense image and video bash scripts into single vision script as the Qwen3-VL model should work for both cases

Summary by CodeRabbit

Release Notes

• New Features

  • Video input support now integrated into vLLM multimodal backend with configurable frame sampling
  • Unified video serving available for both aggregated and disaggregated inference modes

• Documentation

  • Updated multimodal documentation to reflect video support in vLLM backend
  • Added launch examples for video-enabled deployments

• Chores

  • Removed legacy video encoding components
  • Updated example configurations to use standardized video infrastructure

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-c855d5ab-4ecb-45d1-b97b-b1d16d12fa36.docs.buildwithfern.com/dynamo/dev

[rmccorm4]

/ok to test 3057755

[krishung5]

LGTM! Left two minor comments which are not blocking. Great to to see the quick benchmark result for the video pipeline.