Add live audio transcription streaming support to Foundry Local JS SDK

[rui-ren]

Here's the updated PR description with the renamed types:

---

Title: Add live audio transcription streaming support to Foundry Local JS SDK

Description:

Adds real-time audio streaming support to the Foundry Local JS SDK, enabling live microphone-to-text transcription via ONNX Runtime GenAI ASR.

The existing AudioClient only supports file-based transcription. This PR introduces LiveAudioTranscriptionClient that accepts continuous PCM audio chunks (e.g., from a microphone) and returns partial/final transcription results as an async iterable.

What's included

New files

• src/openai/liveAudioTranscriptionClient.ts — Streaming client with start(), pushAudioData(), getTranscriptionStream(), stop(), dispose()
• src/openai/liveAudioTranscriptionTypes.ts — LiveAudioTranscriptionResult and CoreErrorResponse interfaces, tryParseCoreError() helper

Modified files

• src/imodel.ts — Added createLiveTranscriptionClient() to interface
• src/model.ts — Delegates to selectedVariant.createLiveTranscriptionClient()
• src/modelVariant.ts — Implementation (creates new LiveAudioTranscriptionClient(modelId, coreInterop))
• src/index.ts — Exports LiveAudioTranscriptionClient, LiveAudioTranscriptionSettings, LiveAudioTranscriptionResult, CoreErrorResponse

API surface

const audioClient = model.createAudioClient();
const session = model.createLiveTranscriptionClient();

session.settings.sampleRate = 16000;
session.settings.channels = 1;
session.settings.language = "en";

await session.start();

// Push audio from microphone callback
await session.pushAudioData(pcmBytes);

// Read results as async iterable
for await (const result of session.getTranscriptionStream()) {
    console.log(result.text);
}

await session.stop();

Design highlights

• Internal async push queue — Bounded AsyncQueue<T> serializes audio pushes from any context (safe for mic callbacks) and provides backpressure. Mirrors C#'s Channel<T> pattern.
• Retry policy — Transient native errors retried with exponential backoff (3 attempts); permanent errors terminate the session
• Settings freeze — Audio format settings are snapshot-copied and Object.freeze()d at start(), immutable during the session
• Buffer copy — pushAudioData() copies the input Uint8Array before queueing, safe when caller reuses buffers
• Drain-on-stop — stop() completes the push queue, waits for the push loop to drain, then calls native stop
• Dispose safety — dispose() wraps stop() in try/catch, never throws

Native core dependency

This PR adds the JS SDK surface. The 3 native commands (audio_stream_start, audio_stream_push, audio_stream_stop) are routed through the existing execute_command / execute_command_with_binary exports. The code compiles with zero TypeScript errors without the native library.

Testing

• ✅ TypeScript compilation — 0 errors across all source files
• ⏳ Integration tests pending native core delivery

Parity with C# SDK

This implementation mirrors the C# LiveAudioTranscriptionSession (branch ruiren/audio-streaming-support-sdk) with identical logic:

• Same session lifecycle: start → push → getStream → stop
• Same push loop with retry and permanent error handling
• Same settings freeze and buffer copy semantics
• Same drain-before-stop ordering
• Same renamed types: LiveAudioTranscription* (matching C# rename)

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
foundry-local	Ready	Preview, Comment	Mar 25, 2026 3:17am

[Copilot]

Pull request overview

This PR introduces live (microphone-style) audio transcription streaming support, adding a new streaming client to the JS SDK and a parallel end-to-end implementation in the C# SDK (interop, types, tests, and samples). It also bumps Foundry Local Core package versions to 0.9.0 in JS install scripts and C# projects.

Changes:

• JS: Add LiveAudioTranscriptionClient/types and wire a createLiveTranscriptionClient() factory through IModel/Model/ModelVariant, plus export from the package entrypoint.
• C#: Add LiveAudioTranscriptionSession + types, CoreInterop binary push plumbing, tests, docs, and sample projects.
• Versioning: Update Foundry Local Core package versions from 0.9.0.8-rc3 to 0.9.0 in relevant JS/C# build/install locations.

Reviewed changes

Copilot reviewed 23 out of 25 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
sdk/js/src/openai/liveAudioTranscriptionClient.ts	New JS streaming transcription client implementation (session lifecycle, push queue, async stream).
sdk/js/src/openai/liveAudioTranscriptionTypes.ts	New JS result/error interfaces and error parsing helper.
sdk/js/src/imodel.ts	Adds createLiveTranscriptionClient() to the JS model interface.
sdk/js/src/model.ts	Exposes createLiveTranscriptionClient() on Model by delegating to the selected variant.
sdk/js/src/modelVariant.ts	Implements createLiveTranscriptionClient() for a specific model variant.
sdk/js/src/index.ts	Exports live transcription client/settings/types from the JS SDK entrypoint.
sdk/js/script/install.cjs	Bumps core artifact versions used by the JS install flow to 0.9.0.
sdk/cs/src/OpenAI/LiveAudioTranscriptionClient.cs	Adds C# live transcription session (push loop, stream results, lifecycle).
sdk/cs/src/OpenAI/LiveAudioTranscriptionTypes.cs	Adds C# response/error types + JSON mapping for streaming transcription.
sdk/cs/src/OpenAI/AudioClient.cs	Adds CreateLiveTranscriptionSession() to the C# audio client.
sdk/cs/src/OpenAI/ChatClient.cs	Adjusts streaming chat completion implementation (notably error propagation behavior).
sdk/cs/src/Detail/ICoreInterop.cs	Extends interop contract with audio streaming methods and a binary request buffer struct.
sdk/cs/src/Detail/CoreInterop.cs	Implements binary push via execute_command_with_binary and adds audio streaming helpers.
sdk/cs/src/Detail/JsonSerializationContext.cs	Registers streaming types for System.Text.Json source generation.
sdk/cs/src/Microsoft.AI.Foundry.Local.csproj	Bumps core package versions to 0.9.0.
sdk/cs/test/FoundryLocal.Tests/LiveAudioTranscriptionTests.cs	Adds unit tests for streaming types/options and session guards.
sdk/cs/test/FoundryLocal.Tests/Microsoft.AI.Foundry.Local.Tests.csproj	Adds a conditional package reference related to ORT Linux GPU package versioning.
sdk/cs/test/FoundryLocal.Tests/ModelTests.cs	Minor formatting adjustment (closing brace).
sdk/cs/README.md	Documents the new C# live audio transcription streaming API and lifecycle.
samples/cs/GettingStarted/src/LiveAudioTranscriptionExample/Program.cs	Adds a console sample showing microphone streaming transcription.
samples/cs/GettingStarted/windows/LiveAudioTranscriptionExample/LiveAudioTranscriptionExample.csproj	Adds Windows sample project configuration and dependencies (WinML + NAudio).
samples/cs/GettingStarted/cross-platform/LiveAudioTranscriptionExample/LiveAudioTranscriptionExample.csproj	Adds cross-platform sample project configuration and dependencies.
sdk_v2/js/src/index.ts	Adds a new sdk_v2/js entrypoint exporting the JS surface (including live transcription).
sdk_v2/cs/src/Microsoft.AI.Foundry.Local.csproj	Adds/updates a sdk_v2/cs project file with core package version settings and dependencies.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

CoreErrorResponse is marked @internal but is exported from the package entrypoint (sdk/js/src/index.ts). Either remove the @internal tag (if it’s intended to be public API) or stop exporting it from the public surface to keep docs/API consistent.

[Copilot]

ExecuteCommandWithCallbackAsync returns a Response that can contain Error even when no callbacks were invoked. By discarding the returned response and removing the response.Error check, callers can now receive an empty stream instead of an exception on immediate native failures. Please restore the error propagation (or change ExecuteCommandWithCallbackAsync to throw when Error is set).

[Copilot]

The interface declares createLiveTranscriptionClient() twice with identical signatures/JSDoc. Even if TypeScript merges them, this is redundant and can lead to duplicated docs or confusing hover text; please keep only a single declaration (and consolidate the JSDoc).

Suggested change

	* Creates a LiveAudioTranscriptionClient for real-time audio streaming ASR.
	* @returns A LiveAudioTranscriptionClient instance.
	*/
	createLiveTranscriptionClient(): LiveAudioTranscriptionClient;
	/**

[Copilot]

tryWrite() ignores maxCapacity entirely and can grow the queue without bound even when the queue is meant to be bounded (pushQueueCapacity). If you keep tryWrite, it should return false when at capacity (or implement the same backpressure semantics as write).

Suggested change

	if (this.queue.length >= this.maxCapacity) {
	return false;
	}

[Copilot]

audio_stream_stop return value is ignored. If native core returns a final transcription payload on stop (as the C# implementation does), it should be parsed and delivered to consumers via outputQueue before completing the stream; otherwise callers may miss the final result.

[Copilot]

PR description focuses on the active JS SDK under sdk/js/..., but this change also adds a new sdk_v2/js/src/index.ts entrypoint (and includes substantial C# changes). If sdk_v2/js is still shipped, the description should mention it; if not, consider dropping this file to avoid maintaining two parallel entrypoints.

[Copilot]

audio_stream_push is invoked without sending the actual audioData bytes (only AudioDataLength is passed). With the current JS CoreInterop there is no execute_command_with_binary wrapper, so native core will never receive PCM and the stream will not function. Add a binary-capable CoreInterop method and use it here to pass the PCM buffer, then parse any returned transcription JSON and enqueue it onto outputQueue for getTranscriptionStream() to yield.

Suggested change

	this.coreInterop.executeCommand("audio_stream_push", {
	Params: {
	SessionHandle: this.sessionHandle!,
	AudioDataLength: audioData.length.toString()
	}
	});
	const coreAny = this.coreInterop as any;
	const response = await coreAny.executeCommandWithBinary(
	"audio_stream_push",
	{
	Params: {
	SessionHandle: this.sessionHandle!,
	AudioDataLength: audioData.length.toString()
	}
	},
	audioData
	);
	if (typeof response === 'string' && response.trim().length > 0) {
	try {
	const parsed = JSON.parse(response) as LiveAudioTranscriptionResult;
	await this.outputQueue?.write(parsed);
	} catch (parseError) {
	const parseMsg = parseError instanceof Error ? parseError.message : String(parseError);
	console.error('Failed to parse transcription response from audio_stream_push:', parseMsg);
	}
	}

[Copilot]

AsyncQueue.write() backpressure is not safe with multiple concurrent writers: when at capacity, each writer overwrites backpressureResolve, so earlier writers can wait forever. Use a FIFO of pending writer resolvers (or reuse the existing streaming queue pattern used elsewhere in the JS SDK) so all blocked writers are eventually released.

[Copilot]

tryParseCoreError(errorMsg) is very unlikely to succeed because CoreInterop.executeCommand() throws messages prefixed with "Command 'X' failed: ...", which is not valid JSON. Either change CoreInterop to throw the raw native error string (without prefix) or make tryParseCoreError robust (e.g., extract the JSON substring after the prefix) before attempting JSON.parse.

Suggested change

	const errorInfo = tryParseCoreError(errorMsg);
	const coreErrorMsg = extractCoreErrorMessage(errorMsg);
	const errorInfo = tryParseCoreError(coreErrorMsg);

[Copilot]

This library code writes directly to console.error/console.warn. The rest of the JS SDK avoids emitting console output; errors should be surfaced via exceptions/stream completion and let the host application decide how to log. Consider removing these calls or injecting a logger via configuration/settings.