Add live audio transcription streaming support to Foundry Local C# SDK

[rui-ren]

Here's the updated PR description based on the latest changes (renamed types, CoreInterop routing fix, mermaid updates):

---

Title: Add live audio transcription streaming support to Foundry Local C# SDK

Description:

Adds real-time audio streaming support to the Foundry Local C# SDK, enabling live microphone-to-text transcription via ONNX Runtime GenAI's StreamingProcessor API (Nemotron ASR).

The existing OpenAIAudioClient only supports file-based transcription. This PR introduces LiveAudioTranscriptionSession that accepts continuous PCM audio chunks (e.g., from a microphone) and returns partial/final transcription results as an async stream.

What's included

New files

• src/OpenAI/LiveAudioTranscriptionClient.cs — Streaming session with StartAsync(), AppendAsync(), GetTranscriptionStream(), StopAsync()
• src/OpenAI/LiveAudioTranscriptionTypes.cs — LiveAudioTranscriptionResult and CoreErrorResponse types

Modified files

• src/OpenAI/AudioClient.cs — Added CreateLiveTranscriptionSession() factory method
• src/Detail/ICoreInterop.cs — Added StreamingRequestBuffer struct, StartAudioStream, PushAudioData, StopAudioStream interface methods
• src/Detail/CoreInterop.cs — Routes audio commands through existing execute_command / execute_command_with_binary native entry points (no separate audio exports needed)
• src/Detail/JsonSerializationContext.cs — Registered LiveAudioTranscriptionResult for AOT compatibility
• test/FoundryLocal.Tests/Utils.cs — Updated to use CreateLiveTranscriptionSession()

Documentation

API surface

var audioClient = await model.GetAudioClientAsync();
var session = audioClient.CreateLiveTranscriptionSession();

session.Settings.SampleRate = 16000;
session.Settings.Channels = 1;
session.Settings.Language = "en";

await session.StartAsync();

// Push audio from microphone callback
await session.AppendAsync(pcmBytes);

// Read results as async stream
await foreach (var result in session.GetTranscriptionStream())
{
    Console.Write(result.Text);
}

await session.StopAsync();

Design highlights

• Internal push queue — Bounded Channel<T> serializes audio pushes from any thread (safe for mic callbacks) with backpressure
• Retry policy — Transient native errors retried with exponential backoff (3 attempts); permanent errors terminate the session
• Settings freeze — Audio format settings are snapshot-copied at StartAsync() and immutable during the session
• Cancellation-safe stop — StopAsync always calls native stop even if cancelled, preventing native session leaks
• Dedicated session CTS — Push loop uses its own CancellationTokenSource, decoupled from the caller's token
• Routes through existing exports — StartAudioStream and StopAudioStream route through execute_command; PushAudioData routes through execute_command_with_binary — no new native entry points required

Core integration (neutron-server)

The Core side (AudioStreamingSession.cs) uses StreamingProcessor + Generator + Tokenizer + TokenizerStream from onnxruntime-genai to perform real-time RNNT decoding. The native commands (audio_stream_start/push/stop) are handled as cases in NativeInterop.ExecuteCommandManaged / ExecuteCommandWithBinaryManaged.

Verified working

• ✅ SDK build succeeds (0 errors)
• ✅ GenAI StreamingProcessor pipeline verified with WAV file (correct transcript)
• ✅ Core TranscribeChunk byte[] PCM path matches reference float[] path exactly
• ✅ Full E2E simulation: SDK Channel + JSON serialization + session management (32 partial + 1 final result)
• ✅ Live microphone test: 67s real-time transcription through SDK → Core → GenAI
• ✅ Full SDK → Core → GenAI E2E with locally built Core DLL and GenAI NuGet 0.13.0-dev

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview, Comment	Mar 25, 2026 8:45am

[Copilot]

Pull request overview

Adds a new C# SDK API for live/streaming audio transcription sessions (push PCM chunks, receive incremental/final text results) and includes a Windows microphone demo sample.

Changes:

• Introduces LiveAudioTranscriptionSession + result/error types for streaming ASR over Core interop.
• Extends Core interop to support audio stream start/push/stop (including binary payload routing).
• Adds a samples/cs/LiveAudioTranscription demo project and updates the audio client factory API.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
sdk_v2/cs/test/FoundryLocal.Tests/Utils.cs	Replaced prior test utilities with ad-hoc top-level streaming harness code (currently breaks test build).
sdk_v2/cs/test/FoundryLocal.Tests/ModelTests.cs	Adds trailing blank lines (formatting noise).
sdk_v2/cs/src/OpenAI/LiveAudioTranscriptionTypes.cs	Adds LiveAudioTranscriptionResult and a structured Core error type.
sdk_v2/cs/src/OpenAI/LiveAudioTranscriptionClient.cs	Adds LiveAudioTranscriptionSession implementation (channels, retry, stop semantics).
sdk_v2/cs/src/OpenAI/AudioClient.cs	Adds CreateLiveTranscriptionSession() and removes the public file streaming transcription API.
sdk_v2/cs/src/Detail/JsonSerializationContext.cs	Registers new audio streaming types for source-gen JSON.
sdk_v2/cs/src/Detail/ICoreInterop.cs	Adds interop structs + methods for audio stream start/push/stop.
sdk_v2/cs/src/Detail/CoreInterop.cs	Implements binary command routing via execute_command_with_binary and start/stop routing via execute_command.
sdk_v2/cs/src/AssemblyInfo.cs	Adds InternalsVisibleTo("AudioStreamTest").
samples/cs/LiveAudioTranscription/README.md	Documentation for the live transcription demo sample.
samples/cs/LiveAudioTranscription/Program.cs	Windows microphone demo using NAudio + new session API.
samples/cs/LiveAudioTranscription/LiveAudioTranscription.csproj	Adds sample project dependencies and references the SDK project (path currently incorrect).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

You can also share your feedback on Copilot code review. Take the survey.

[hanbitmyths]

Is there any reason to add zip and dlls at LiveAudioTranscription?

[rui-ren]

Is there any reason to add zip and dlls at LiveAudioTranscription?

Yes, I am deleting this. It was caused by the rename a lot of folders and mistakenly upload the test folder etc,, let me revert it back.