[Issue]: MIGraphX Dynamic Shape Issue ONNXRuntime

[DiarmuidKelly]

Problem Description

MIGraphX Dynamic Shape Issue ONNXRuntime

Environment

OS:
  NAME="Ubuntu"
  VERSION="25.10 (Questing Quokka)"
  Kernel: 6.17.0-14-generic
CPU:
  13th Gen Intel(R) Core(TM) i5-13600KF
GPU:
  Name: gfx1101
  Marketing Name: AMD Radeon RX 7800 XT
ROCm: 7.2
Python: 3.12.9

Env Deps (from repo.radeon.com/rocm/manylinux/rocm-rel-7.2/):

• triton-3.5.1+rocm7.2.0.gita272dfa8-cp312-cp312-linux_x86_64.whl
• torch-2.9.1+rocm7.2.0.lw.git7e1940d4-cp312-cp312-linux_x86_64.whl
• torchvision-0.24.0+rocm7.2.0.gitb919bd0c-cp312-cp312-linux_x86_64.whl
• onnxruntime_migraphx-1.23.2-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl

Issue Summary

MIGraphX crashes on second inference when input tensor dimensions change. The first inference succeeds, but subsequent inferences with different sequence lengths fail with a shape mismatch error.

Steps to Reproduce

1. Load any ONNX model with dynamic input shapes via MIGraphXExecutionProvider
2. Run inference with input shape [1, 512, 16] - SUCCEEDS
3. Run inference with input shape [1, 512, 10] - FAILS

The model compiles successfully on first run, but fails when subsequent inferences have different tensor dimensions (e.g. different sequence lengths).

Observed behaviour with a TTS model (dynamic sequence length):

Inference	Input	Result	Time
1	16 chars	OK	94s (initial compile)
2	16 chars	OK	0.06s (cached, same shape)
3	10 chars	OK	150s (recompiles for new shape)
4	55 chars	FAILED	shape mismatch error
5	5 chars	FAILED	shape mismatch error

Error Message

migraphx_parse_onnx_buffer: Error: .../AMDMIGraphX/src/include/migraphx/op/dot.hpp:117:
compute_shape: DOT: static inner dimensions do not match: {1, 512, 70} x {1, 13, 24}

[ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running
MGXKernel_graph_main_graph_3837677502492941375_6 node

Root Cause Analysis

MIGraphX compiles GPU kernels for specific tensor shapes during the first inference. When subsequent inferences have different input dimensions (different text lengths = different sequence lengths), the pre-compiled kernels fail because they expect the original shapes.

This is a fundamental limitation in how MIGraphX handles dynamic shapes. Unlike CPUExecutionProvider or CUDAExecutionProvider which handle dynamic shapes at runtime, MIGraphX requires static shapes known at compile time.

Related Issues

• Dynamic Shape Support in IR to handle shape changes #1023 - "Dynamic Shape Support in IR to handle shape changes" (OPEN, created 2021-12-09, no description)
• Invalid MIGraphX EP option: migraphx_load_compiled_path microsoft/onnxruntime#25379

Impact

This makes MIGraphX unsuitable for any model with dynamic input shapes, which includes most NLP/TTS models where sequence length varies with input.

Workarounds Attempted

1. Model caching (migraphx_model_cache_dir) - Does not help, shapes still mismatch
2. Warming with large input first - Does not help, smaller inputs still fail

Requested Resolution

Implement dynamic shape support in MIGraphX, or document this limitation clearly in the ONNX Runtime MIGraphX provider documentation.

At minimum, MIGraphX should fall back gracefully to recompilation when shapes change, rather than crashing with an assertion error.

[DiarmuidKelly]

I think I have a solution. Thoughts? I would try get to a PR if so.

https://github.com/ROCm/AMDMIGraphX/blob/develop/src/onnx/onnx_parser.cpp#L387

if(parser.map_input_dims.count(name) > 0)
{
    std::vector<std::size_t> dims = parser.map_input_dims.at(name);
    s                             = parser.parse_type(input.type(), dims);
}

Proposed solution:

ENV vars

MIGRAPHX_DYNAMIC_SHAPE_MIN=1      # Minimum dimension for dynamic range
MIGRAPHX_DYNAMIC_SHAPE_MAX=256    # Maximum dimension for dynamic range

And in the onnx_parser.cpp

#include <migraphx/env.hpp>

MIGRAPHX_DECLARE_ENV_VAR(MIGRAPHX_DYNAMIC_SHAPE_MIN)
MIGRAPHX_DECLARE_ENV_VAR(MIGRAPHX_DYNAMIC_SHAPE_MAX)

// In parse logic around line 387:
auto dyn_min = value_of(MIGRAPHX_DYNAMIC_SHAPE_MIN{}, 0);
auto dyn_max = value_of(MIGRAPHX_DYNAMIC_SHAPE_MAX{}, 0);
bool force_dynamic = (dyn_max > 0);

if(force_dynamic || parser.map_input_dims.count(name) == 0)
{
    // Dynamic path
    s = parser.parse_type(input.type());
}
else
{
    // Static path (original behaviour)
    s = parser.parse_type(input.type(), parser.map_input_dims.at(name));
}

Still looking at some binning, and padding. Like TensorRT

[huanrwan-amd]

Hi @DiarmuidKelly, thanks for posting, before moving to the next step. Can you please provide a minimum reproducer for "Steps to Reproduce"?

[DiarmuidKelly]

Minimal Reproducer 1: fp32 Model (Immediate Failure)

Environment:

• ROCm 7.2, gfx1101 (RX 7800 XT)
• onnxruntime_migraphx-1.23.2
• Model: Kokoro-82M-v1.0-ONNX (fp32)

Requirements:

pip install onnxruntime-migraphx huggingface_hub numpy

Script (click to expand)

#!/usr/bin/env python3
"""Minimal reproducer - fp32 model from HuggingFace."""

import numpy as np
import os
import sys
from pathlib import Path


def download_model():
    from huggingface_hub import hf_hub_download
    cache_dir = Path.home() / ".cache" / "migraphx_repro"
    cache_dir.mkdir(parents=True, exist_ok=True)
    print("Downloading Kokoro ONNX model from HuggingFace...")
    return hf_hub_download(
        repo_id="onnx-community/Kokoro-82M-v1.0-ONNX",
        filename="onnx/model.onnx",
        cache_dir=cache_dir,
    )


def create_dummy_inputs(seq_len: int) -> dict:
    return {
        "input_ids": np.zeros((1, seq_len), dtype=np.int64),
        "style": np.random.randn(1, 256).astype(np.float32),
        "speed": np.ones(1, dtype=np.float32),
    }


def main():
    import onnxruntime as ort

    print(f"Available providers: {ort.get_available_providers()}")
    model_path = download_model()

    print("\nCreating ONNX session with MIGraphXExecutionProvider...")
    providers = [
        ("MIGraphXExecutionProvider", {"device_id": "0"}),
        "CPUExecutionProvider",
    ]
    session = ort.InferenceSession(model_path, providers=providers)

    print("\nModel inputs:")
    for inp in session.get_inputs():
        print(f"  {inp.name}: {inp.shape}")

    test_sequence = [16, 16, 10, 55, 16]
    print("\nRunning inference with varying sequence lengths...")

    for i, seq_len in enumerate(test_sequence, 1):
        inputs = create_dummy_inputs(seq_len)
        print(f"\nInference {i}: seq_len={seq_len}")
        try:
            outputs = session.run(None, inputs)
            print(f"  Result: OK, output shape: {outputs[0].shape}")
        except Exception as e:
            print(f"  Result: FAILED - {e}")


if __name__ == "__main__":
    main()

Output:

Model inputs:
  input_ids: [1, 'sequence_length'] (tensor(int64))
  style: [1, 256] (tensor(float))
  speed: [1] (tensor(float))

Running inference with varying sequence lengths...

Inference 1: seq_len=16
  Result: OK, output shape: (1, 52200)

Inference 2: seq_len=16
  Result: OK, output shape: (1, 61200)

Inference 3: seq_len=10
  Result: FAILED
  Error: DOT: static inner dimensions do not match: {1, 512, 10} x {1, 16, 102}

Inference 4: seq_len=55
  Result: FAILED
  Error: DOT: static inner dimensions do not match: {1, 512, 55} x {1, 16, 102}

Inference 5: seq_len=16
  Result: OK, output shape: (1, 61200)

Summary:

• Different shapes fail immediately with shape mismatch
• Original shape (16) still works after failures
• No recompilation attempted

**RAW Log** (click to expand)

�[0;93m2026-02-19 09:23:37.631036160 [W:onnxruntime:Default, migraphx_execution_provider.cc:167 MIGraphXExecutionProvider] [MIGraphX EP] MIGraphX ENV Override Variables Set:�[m
�[0;93m2026-02-19 09:23:37.999697360 [W:onnxruntime:Default, migraphx_execution_provider_utils.h:155 canEvalNodeArgument] Node:/encoder/Range Input:/encoder/Cast_1_output_0 Can't eval shape�[m
�[0;93m2026-02-19 09:23:38.000459252 [W:onnxruntime:Default, migraphx_execution_provider.cc:1145 GetCapability] ============= Unsupported nodes ====================�[m
�[0;93m2026-02-19 09:23:38.000462091 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] Range�[m
�[0;93m2026-02-19 09:23:38.000463674 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] STFT�[m
�[0;93m2026-02-19 09:23:38.000464964 [W:onnxruntime:Default, migraphx_execution_provider.cc:1149 GetCapability] ************* Unsupported nodes ********************�[m
�[0;93m2026-02-19 09:23:42.701647311 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin�[m
�[0;93m2026-02-19 09:25:01.727063130 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete�[m
�[0;93m2026-02-19 09:25:03.227806120 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin�[m
�[0;93m2026-02-19 09:26:12.688969830 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete�[m
�[0;93m2026-02-19 09:26:14.641009762 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin�[m
�[0;93m2026-02-19 09:26:40.615633188 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete�[m
migraphx_parse_onnx_buffer: Error: /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/AMDMIGraphX/src/include/migraphx/op/dot.hpp:117: compute_shape: DOT: static inner dimensions do not match: {1, 512, 10} x {1, 16, 102}
�[1;31m2026-02-19 09:26:42.998442844 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_main_graph_10674661227598027056_2 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_main_graph_10674661227598027056_2_2' Status Message: Failed to call function�[m
migraphx_parse_onnx_buffer: Error: /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/AMDMIGraphX/src/include/migraphx/op/dot.hpp:117: compute_shape: DOT: static inner dimensions do not match: {1, 512, 55} x {1, 16, 102}
�[1;31m2026-02-19 09:26:45.502057961 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_main_graph_10674661227598027056_2 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_main_graph_10674661227598027056_2_2' Status Message: Failed to call function�[m
Available providers: ['MIGraphXExecutionProvider', 'CPUExecutionProvider']
Downloading Kokoro ONNX model from HuggingFace...
Model downloaded: /xxxx/xxxx/.cache/migraphx_repro/models--onnx-community--Kokoro-82M-v1.0-ONNX/snapshots/1939ad2a8e416c0acfeecc08a694d14ef25f2231/onnx/model.onnx

[DiarmuidKelly]

Minimal Reproducer 2: int8 Model (Session Corruption)

Environment:

• ROCm 7.2, gfx1101 (RX 7800 XT)
• onnxruntime_migraphx-1.23.2
• Model: kokoro-v1.0.int8.onnx (int8 quantized)

Requirements:

pip install onnxruntime-migraphx requests numpy

Script (click to expand)

#!/usr/bin/env python3
"""Minimal reproducer - int8 model from thewh1teagle/kokoro-onnx."""

import numpy as np
import os
import sys
from pathlib import Path
import requests

MODEL_URL = "https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.0/kokoro-v1.0.int8.onnx"
VOICES_URL = "https://github.com/thewh1teagle/kokoro-onnx/releases/download/model-files-v1.0/voices-v1.0.bin"


def download_file(url: str, dest: Path) -> None:
    if dest.exists():
        print(f"Already downloaded: {dest.name}")
        return
    print(f"Downloading {dest.name}...")
    dest.parent.mkdir(parents=True, exist_ok=True)
    response = requests.get(url, stream=True)
    response.raise_for_status()
    with open(dest, 'wb') as f:
        for chunk in response.iter_content(chunk_size=8192):
            f.write(chunk)


def download_models() -> tuple[Path, Path]:
    cache_dir = Path.home() / ".cache" / "migraphx_repro"
    model_path = cache_dir / "kokoro-v1.0.int8.onnx"
    voices_path = cache_dir / "voices-v1.0.bin"
    download_file(MODEL_URL, model_path)
    download_file(VOICES_URL, voices_path)
    return model_path, voices_path


def create_dummy_inputs(seq_len: int, voices: np.ndarray) -> dict:
    voice_idx = min(seq_len, len(voices) - 1)
    style = voices["af_heart"][voice_idx]
    return {
        "tokens": np.zeros((1, seq_len), dtype=np.int64),
        "style": style.reshape(1, -1).astype(np.float32),
        "speed": np.ones(1, dtype=np.float32),
    }


def main():
    import onnxruntime as ort

    print(f"Available providers: {ort.get_available_providers()}")
    model_path, voices_path = download_models()
    voices = np.load(voices_path)

    print("\nCreating ONNX session with MIGraphXExecutionProvider...")
    providers = [
        ("MIGraphXExecutionProvider", {"device_id": "0"}),
        "CPUExecutionProvider",
    ]
    session = ort.InferenceSession(str(model_path), providers=providers)

    print("\nModel inputs:")
    for inp in session.get_inputs():
        print(f"  {inp.name}: {inp.shape}")

    test_sequence = [16, 16, 10, 55, 16]
    print("\nRunning inference with varying sequence lengths...")

    for i, seq_len in enumerate(test_sequence, 1):
        inputs = create_dummy_inputs(seq_len, voices)
        print(f"\nInference {i}: seq_len={seq_len}")
        try:
            outputs = session.run(None, inputs)
            print(f"  Result: OK, output shape: {outputs[0].shape}")
        except Exception as e:
            print(f"  Result: FAILED - {e}")


if __name__ == "__main__":
    main()

Output (stderr + stdout combined):

Unsupported nodes: DynamicQuantizeLSTM (x6), Range, FusedMatMul, STFT

[W] Model Compile: Begin
[W] Model Compile: Complete (21s)

Model inputs:
  tokens: [1, 'sequence_length'] (tensor(int64))
  style: [1, 256] (tensor(float))
  speed: [1] (tensor(float))

Running inference with varying sequence lengths...

Inference 1: seq_len=16
  Result: OK, output shape: (25200,)

Inference 2: seq_len=16       <-- SAME SHAPE AS INFERENCE 1
  migraphx_parse_onnx_buffer: Error: .../migraphx/op/dot.hpp:117:
    compute_shape: DOT: static inner dimensions do not match: {1, 512, 16} x {1, 13, 13}
  [E:onnxruntime] Non-zero status code returned while running MGXKernel node
  Result: FAILED

  [W] Model Compile: Begin    <-- Recompilation triggered
  [W] Model Compile: Complete (13s)

Inference 3: seq_len=10
  migraphx_parse_onnx_buffer: Error: DOT: static inner dimensions do not match: {1, 512, 10} x {1, 13, 13}
  Result: FAILED

Inference 4: seq_len=55
  migraphx_parse_onnx_buffer: Error: DOT: static inner dimensions do not match: {1, 512, 55} x {1, 13, 13}
  Result: FAILED

Inference 5: seq_len=16       <-- Original shape now permanently broken
  migraphx_parse_onnx_buffer: Error: DOT: static inner dimensions do not match: {1, 512, 16} x {1, 13, 13}
  Result: FAILED

Summary:

• Inference 1 works, then Inference 2 with same shape fails
• MIGraphX attempts recompilation after failures
• Recompilation doesn't fix the issue
• Session enters corrupted state - even previously-working shapes stop working
• This makes MIGraphX unusable for any model with dynamic input dimensions

**Raw Log** (click to expand)

2026-02-19 10:16:31.121198800 [W:onnxruntime:Default, migraphx_execution_provider.cc:167 MIGraphXExecutionProvider] [MIGraphX EP] MIGraphX ENV Override Variables Set:
2026-02-19 10:16:31.257599586 [W:onnxruntime:Default, migraphx_execution_provider_utils.h:155 canEvalNodeArgument] Node:/encoder/Range Input:/encoder/Cast_1_output_0 Can't eval shape
2026-02-19 10:16:31.258767497 [W:onnxruntime:Default, migraphx_execution_provider.cc:1145 GetCapability] ============= Unsupported nodes ====================
2026-02-19 10:16:31.258771862 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] DynamicQuantizeLSTM
2026-02-19 10:16:31.258773422 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] DynamicQuantizeLSTM
2026-02-19 10:16:31.258774711 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] DynamicQuantizeLSTM
2026-02-19 10:16:31.258776277 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] DynamicQuantizeLSTM
2026-02-19 10:16:31.258777996 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] Range
2026-02-19 10:16:31.258779727 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] FusedMatMul
2026-02-19 10:16:31.258781225 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] DynamicQuantizeLSTM
2026-02-19 10:16:31.258782756 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] DynamicQuantizeLSTM
2026-02-19 10:16:31.258784675 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] STFT
2026-02-19 10:16:31.258785975 [W:onnxruntime:Default, migraphx_execution_provider.cc:1149 GetCapability] ************* Unsupported nodes ********************
migraphx_parse_onnx_buffer: Error: /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/AMDMIGraphX/src/include/migraphx/op/dot.hpp:117: compute_shape: DOT: static inner dimensions do not match: {1, 512, 16} x {1, 13, 13}
2026-02-19 10:16:37.238773915 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_main_graph_2828712196626593480_6 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_main_graph_2828712196626593480_6_6' Status Message: Failed to call function
migraphx_parse_onnx_buffer: Error: /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/AMDMIGraphX/src/include/migraphx/op/dot.hpp:117: compute_shape: DOT: static inner dimensions do not match: {1, 512, 10} x {1, 13, 13}
2026-02-19 10:16:38.753620896 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_main_graph_2828712196626593480_6 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_main_graph_2828712196626593480_6_6' Status Message: Failed to call function
migraphx_parse_onnx_buffer: Error: /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/AMDMIGraphX/src/include/migraphx/op/dot.hpp:117: compute_shape: DOT: static inner dimensions do not match: {1, 512, 55} x {1, 13, 13}
2026-02-19 10:16:40.293013694 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_main_graph_2828712196626593480_6 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_main_graph_2828712196626593480_6_6' Status Message: Failed to call function
migraphx_parse_onnx_buffer: Error: /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/AMDMIGraphX/src/include/migraphx/op/dot.hpp:117: compute_shape: DOT: static inner dimensions do not match: {1, 512, 16} x {1, 13, 13}
2026-02-19 10:16:41.878289650 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_main_graph_2828712196626593480_6 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_main_graph_2828712196626593480_6_6' Status Message: Failed to call function
Available providers: ['MIGraphXExecutionProvider', 'CPUExecutionProvider']
Already downloaded: /xxxx/xxxx/.cache/migraphx_repro/kokoro-v1.0.int8.onnx
Already downloaded: /xxxx/xxxx/.cache/migraphx_repro/voices-v1.0.bin

Creating ONNX session with MIGraphXExecutionProvider...
Session created. Active provider: MIGraphXExecutionProvider

[huanrwan-amd]

Hi @DiarmuidKelly, thanks for posting the reproducer. Notice you are on Ubuntu25 which is not fully supported and work in progress. Did you try with the official docker image on ROCm ONNX (https://hub.docker.com/r/rocm/onnxruntime/tags e.g. rocm/onnxruntime:rocm7.2_ub24.04_ort1.23_torch2.9.1). It works on both examples.
1.

MIGraphX dynamic shape repro (AMDMIGraphX#4618)
Single session, multiple sess.run() with different shapes (same-session repro).
Model: dynamic_shape_model.onnx (create with create_model.py if missing)
Using providers: ['MIGraphXExecutionProvider', 'CPUExecutionProvider']

[Run1] OK — shapes A=(1, 512, 16), B=(1, 16, 24) -> Y=(1, 512, 24)
[Run2] OK — shapes A=(1, 512, 10), B=(1, 10, 24) -> Y=(1, 512, 24)
[Run3] OK — shapes A=(1, 512, 16), B=(1, 16, 24) -> Y=(1, 512, 24)

All runs succeeded (MIGraphX may have recompiled or your build handles dynamic shapes).

Available providers: ['MIGraphXExecutionProvider', 'CPUExecutionProvider']
Downloading kokoro-v1.0.int8.onnx...
Downloading voices-v1.0.bin...

Creating ONNX session with MIGraphXExecutionProvider...

Model inputs:
  tokens: [1, 'sequence_length']
  style: [1, 256]
  speed: [1]

Running inference with varying sequence lengths...

Inference 1: seq_len=16
  Result: OK, output shape: (9600,)

Inference 2: seq_len=16
  Result: OK, output shape: (9600,)

Inference 3: seq_len=10
  Result: OK, output shape: (6000,)

Inference 4: seq_len=55
  Result: OK, output shape: (34200,)

Inference 5: seq_len=16
  Result: OK, output shape: (9600,)

[DiarmuidKelly]

@huanrwan-amd it does appear Kernel 6.17 or Ubuntu25.10 are the issue. Running the container (with amdgpu, amdkfd pass-through) errors out in the same place.

docker run -it --rm \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add video \
  --group-add render \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  -v $(pwd):/workspace \
  rocm/onnxruntime:rocm7.2_ub24.04_ort1.23_torch2.9.1 \
  bash

I may try 24.04/Kernel 6.8, or sideline Onnx runtime until 26.04LTS support.

Thanks for your help.

Raw Logs

see below

Creating ONNX session with MIGraphXExecutionProvider...
2026-02-20 07:37:43.662618330 [W:onnxruntime:Default, migraphx_execution_provider.cc:167 MIGraphXExecutionProvider] [MIGraphX EP] MIGraphX ENV Override Variables Set:
2026-02-20 07:37:43.800095278 [W:onnxruntime:Default, migraphx_execution_provider_utils.h:155 canEvalNodeArgument] Node:/encoder/Range Input:/encoder/Cast_1_output_0 Can't eval shape
2026-02-20 07:37:43.801331681 [W:onnxruntime:Default, migraphx_execution_provider.cc:1145 GetCapability] ============= Unsupported nodes ====================
2026-02-20 07:37:43.801337191 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] DynamicQuantizeLSTM
2026-02-20 07:37:43.801339274 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] DynamicQuantizeLSTM
2026-02-20 07:37:43.801341062 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] DynamicQuantizeLSTM
2026-02-20 07:37:43.801342542 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] DynamicQuantizeLSTM
2026-02-20 07:37:43.801343958 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] Range
2026-02-20 07:37:43.801345853 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] FusedMatMul
2026-02-20 07:37:43.801347350 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] DynamicQuantizeLSTM
2026-02-20 07:37:43.801348819 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] DynamicQuantizeLSTM
2026-02-20 07:37:43.801350338 [W:onnxruntime:Default, migraphx_execution_provider.cc:1147 GetCapability] STFT
2026-02-20 07:37:43.801351874 [W:onnxruntime:Default, migraphx_execution_provider.cc:1149 GetCapability] ************* Unsupported nodes ********************
Session created. Active provider: MIGraphXExecutionProvider

Model inputs:
  tokens: [1, 'sequence_length'] (tensor(int64))
  style: [1, 256] (tensor(float))
  speed: [1] (tensor(float))

============================================================
Running inference with varying sequence lengths...
============================================================

Inference 1: seq_len=16
  tokens shape: (1, 16)
2026-02-20 07:37:44.223819031 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:38:17.596802654 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:38:18.428641337 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:38:20.853716277 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:38:20.876771071 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:38:21.623953484 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:38:21.636147343 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:38:22.140078701 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:38:22.152117528 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:38:25.203555019 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:38:25.241446453 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:38:53.218058117 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:38:53.682424705 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:39:25.493576245 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:39:25.932100224 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:40:38.781621125 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
  Result: OK
  Output shape: (9600,)

Inference 2: seq_len=16
  tokens shape: (1, 16)
  Result: OK
  Output shape: (9600,)

Inference 3: seq_len=10
  tokens shape: (1, 10)
2026-02-20 07:40:40.381611878 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:41:11.068573189 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:41:11.773966371 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:41:12.987256049 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:41:13.007540192 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:41:13.565475946 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:41:13.581576888 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:41:14.132868369 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:41:14.148826975 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:41:16.704393524 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:41:16.763022035 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:41:38.491165691 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:41:38.977250098 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:42:05.702236892 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
2026-02-20 07:42:06.202580083 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:43:13.548512035 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
  Result: OK
  Output shape: (6000,)

Inference 4: seq_len=55
  tokens shape: (1, 55)
2026-02-20 07:43:15.983642868 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:43:43.492972793 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
migraphx_parse_onnx_buffer: Error: /longer_pathname_so_that_rpms_can_support_packaging_the_debug_info_for_all_os_profiles/src/AMDMIGraphX/src/include/migraphx/op/dot.hpp:117: compute_shape: DOT: static inner dimensions do not match: {1, 512, 55} x {1, 10, 19}
2026-02-20 07:43:43.980692421 [E:onnxruntime:, sequential_executor.cc:572 ExecuteKernel] Non-zero status code returned while running MGXKernel_graph_main_graph_11710926676126528799_6 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_main_graph_11710926676126528799_6_6' Status Message: Failed to call function
  Result: FAILED
  Error: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Non-zero status code returned while running MGXKernel_graph_main_graph_11710926676126528799_6 node. Name:'MIGraphXExecutionProvider_MGXKernel_graph_main_graph_11710926676126528799_6_6' Status Message: Failed to call function

Inference 5: seq_len=16
  tokens shape: (1, 16)
2026-02-20 07:43:44.875763511 [W:onnxruntime:Default, migraphx_execution_provider.cc:1309 compile_program] Model Compile: Begin
2026-02-20 07:43:57.344442881 [W:onnxruntime:Default, migraphx_execution_provider.cc:1314 compile_program] Model Compile: Complete
  Result: OK
  Output shape: (6000,)

============================================================
Test complete.
============================================================