Add MmapUseMlockIgnoreErrors as default load mode when creating creat…

[psiddh]

…e_text_llm_runner

Users can still override the default mode

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/17613

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❌ 7 New Failures, 1 Unrelated Failure

As of commit 496a321 with merge base 9a58ce8 ():

NEW FAILURES - The following jobs have failed:

• pull / test-mediatek-models-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 60ec64061cc44bfbabd41fa9a3c8c4df185f169cc89570295bed0d39516a31ac /exec failed with exit code 2
• pull / test-openvino-linux / linux-job (gh)
  RuntimeError: Command docker exec -t 5d26822212a76a8c5406622515276a7d674689e90e5af4aa34ffcf9da3571200 /exec failed with exit code 1
• pull / unittest-buck / linux / linux-job (gh)
  RuntimeError: Command docker exec -t dd2a25dadfe34730c7d6d309c920969e0906870c974cd4054ca96358f8c4b2a8 /exec failed with exit code 3
• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (mistralai, Voxtral-Mini-3B-2507, non-quantized) / windows-job (gh)
  Process completed with exit code 1.
• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (mistralai, Voxtral-Mini-3B-2507, quantized-int4-weight-only) / windows-job (gh)
  Process completed with exit code 1.
• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (nvidia, parakeet-tdt, non-quantized) / windows-job (gh)
  Process completed with exit code 1.
• Test CUDA Windows Export and E2E / test-model-cuda-windows-e2e (nvidia, parakeet-tdt, quantized-int4-weight-only) / windows-job (gh)
  Process completed with exit code 1.

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• pull / unittest-buck / macos / macos-job (gh) (trunk failure)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 3

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[github-actions]

This PR needs a release notes: label

If your change should be included in the release notes (i.e. would users of this library care about this change?), please use a label starting with release notes:. This helps us keep track and include your important work in the next release notes.

To add a label, you can comment to pytorchbot, for example
@pytorchbot label "release notes: none"

For more information, see
https://github.com/pytorch/pytorch/wiki/PyTorch-AutoLabel-Bot#why-categorize-for-release-notes-and-how-does-it-work.

[Copilot]

Pull request overview

This PR adds a new optional parameter load_mode to the create_text_llm_runner factory functions, with a default value of Module::LoadMode::MmapUseMlockIgnoreErrors (previously, the load mode was hardcoded to File). This change allows users to customize the memory loading strategy while providing a more efficient default for LLM workloads.

Changes:

• Added load_mode parameter to both overloads of create_text_llm_runner with default value MmapUseMlockIgnoreErrors
• Updated function implementation to use the new parameter when constructing Module instances
• Removed hardcoded Module::LoadMode::File usage in module construction

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
extension/llm/runner/llm_runner_helper.h	Added load_mode parameter with default value to both create_text_llm_runner function declarations
extension/llm/runner/llm_runner_helper.cpp	Implemented parameter propagation and replaced hardcoded LoadMode::File with configurable load_mode parameter

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mergennachin]

Why ignore errors variant over MmapUseMlock?

[psiddh]

The codebase only uses MmapUseMlockIgnoreErrors over MmapUseMlock for LLM runners. I feel that is the right default.

• MmapUseMlock: mlock failure → logs error, unmaps the pages, returns Error::NotSupported.
  The model fails to load entirely. (Stricter!)
• MmapUseMlockIgnoreErrors: mlock failure → logs at Debug level and continues. The model
  loads via normal mmap, just without pages pinned in RAM.

For LLM runners, a hard failure is almost never the right behavior. Failing to load the model at all will be
bad UX / functionality experience for the End User.

By using mmap-based loading in our LLM runners, we avoid loading the entire model into RAM
upfront, which reduces peak memory usage and OOM risk. The MmapUseMlockIgnoreErrors variant
additionally attempts to pin pages in memory for better inference latency, but gracefully
falls back to standard mmap if the system can't support it, giving us the best of both
worlds without hard failures.

[GregoryComer]

Realistically, I think that we're unlikely to actually be able to lock the entire LLM PTE on most systems, so using the base mmap might be better? The mlock ignore errors variant seems functionally fine, though. It'll just fall through to mmap pretty much 100% of the time.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.