feat: livekit cast improvements

[cyaiox]

Pull Request Description

What does this PR change?

Enhances the LiveKit media player to support full Cast scenarios with multiple participants, automatic speaker tracking, presentation bot priority, data messaging between scenes and participants, and self-healing stream recovery.

Video routing

• Active speaker tracking: In CurrentStream mode, video automatically follows the dominant speaker using room.ActiveSpeakers with a 1.5s debounce to prevent flickering
• Presentation bot priority: Participants with identity prefix presentation-bot: are always prioritized — once locked on, speaker tracking is suppressed until the bot leaves
• Video muted state: Tracks TrackPublication.Muted per frame and renders a black texture when muted
• Resolution capping: LiveKit video textures are capped at 1920x1080 to prevent GPU stalls
• Auto-recovery: When a specific user stream dies, falls back to CurrentStream (first available track)

Audio routing

• Multi-audio playback: Plays all audio tracks from the room simultaneously instead of just the paired one — all participants are heard regardless of which video is displayed
• Additive discovery: New participants joining mid-session are picked up without disrupting existing audio sources
• Rescan throttling: Audio rescans every 2s (AUDIO_RESCAN_INTERVAL_SECONDS), or immediately when a dead track is detected

Data messaging API

• PublishData(topic, data) — Send messages to a topic (rate-limited to 10 msg/s, 16KB cap)
• SubscribeToTopic(topic) — Subscribe to receive messages on a topic
• ConsumeMessages(topic) — Consume buffered messages for a topic
• UpdateMetadata(metadata) — Update local participant metadata (broadcast by LiveKit)
• Handles non-protobuf data on the shared DataPipe gracefully (InvalidProtocolBufferException)

Stream metadata

• Enriched getActiveVideoStreams response with displayName, speaking, trackName, width, height
• Display name resolved from participant metadata JSON (displayName field), falling back to Name then Identity

Documentation

• Updated docs/cast.md covering all new features

Test Instructions

Prerequisites

• Deploy the template: Streaming Theatre to .zone/.org

Test Steps — Video routing

1. Open the Admin tools smart item and start a DCL Cast
2. Join the cast as a speaker (cast app) with camera on
3. Verify the video screen streams your camera
4. Have a second speaker join — verify video switches to whoever is talking (with ~1.5s hold)
5. Stop talking — verify video stays on the last speaker (no flickering)

Test Steps — Presentation bot

1. Start a cast session with a presentation bot connected
2. Verify video locks onto the bot's track immediately
3. Have other participants speak — verify video stays on the bot
4. Disconnect the bot — verify video falls back to active speaker tracking

Test Steps — Audio

1. Have multiple speakers join the cast
2. Verify audio plays from all participants simultaneously
3. Have a new participant join mid-session — verify their audio is picked up within ~2s
4. Have a participant leave and rejoin — verify audio recovers

Test Steps — Data messaging

1. From a scene, call SubscribeToTopic("test")
2. From another participant, call PublishData("test", "hello")
3. Call ConsumeMessages("test") — verify the message is returned with sender identity
4. Verify rate limiting: publish >10 messages in 1s, confirm excess are dropped

Test Steps — Screen share recovery

1. Join the cast and share your screen
2. Select the screen option from the Showcase dropdown
3. Stop sharing your screen
4. Verify the video screen falls back to the first available stream (your camera)

Additional Testing Notes

• Verify participant names display correctly for unauthenticated Cast users (castV2)
• Verify no audio leaks (sources properly released) after stopping/restarting a stream
• Verify video renders black when a track is muted

Quality Checklist

• Changes have been tested locally
• Documentation has been updated
• Performance impact has been considered
• For SDK features: Test scene is included

Code Review Reference

Please review our Code Review Standards before submitting.

Screenshots

[github-actions]

Build failed! Check the logs to see what went wrong

[cyaiox]

@claude review

[claude]

Claude encountered an error —— View job

---

I'll analyze this and get back to you.

[Ludmilafantaniella]

Tested! Functionality works as expected ✅

7347-evi.mp4

⚠️ Side note — cosmetic issues found:

• "Showcase List" button label wraps to two lines instead of rendering on a single line.
• Showcase camera dropdown: neither option shows a checkmark by default, even though the camera is already active when Cast is activated. The checkmark only appears after manually selecting an option. Feels a bit counterintuitive visually.
• Horizontal scrollbar visible in the camera dropdown (already flagged — dev will fix by adjusting dropdown width).

None of these block the functionality, just leaving them on record!

11.03.2026_11.08.39_REC.mp4

[m3taphysics]

@claude review

[claude]

Claude finished @m3taphysics's task in 2m 53s —— View job

---

Code Review

• Gather context
• Read changed files
• Analyze for bugs, security, and performance issues
• Post review findings

---

Issues Found

🔴 Performance — EnsureAudioIsPlaying polls participants lock every frame when room has no audio (line 100)

When audioSources.Count == 0 (room has no audio tracks, or after initial open), anyDied is false and the guard condition !anyDied && audioSources.Count > 0 evaluates to false. The method falls through every frame to OpenAllAudioStreams(), which acquires lock(room.Participants) and iterates all participants. For any scene where audio is temporarily unavailable, this is a per-frame lock + allocation in a hot path. Fix this →

🔴 Audio disruption on every video recovery (line 71)

When a user stream's video dies, EnsureVideoIsPlaying() calls OpenMedia(LivekitAddress.CurrentStream()). OpenMedia always calls CloseCurrentStream() first, which calls ReleaseAllAudioSources() — pooling all audio MonoBehaviours — then immediately re-acquires them. This causes a brief audio dropout on every video fallback, even when audio streams are healthy. The video and audio reset paths need to be separated. Fix this →

🟡 Dead code — FindPairedAudio never called (line 255)

Private method with no callers. Violates the project's code standards: "Don't design for hypothetical future requirements." Remove until actually needed. Fix this →

🟡 Silent JsonException swallowing (line 39)

catch (JsonException) { } in ResolveDisplayName silently discards all metadata parse failures. Malformed participant metadata would silently fall back to Name/Identity with no trace in logs, making it impossible to diagnose in production. Log a warning via ReportHub. Fix this →

🟡 asCurrent values written but never read in CommsApiWrap.cs

asCurrent is used only as a null-check sentinel — the stored (identity, track) tuple is never read back. This is misleading; it should be a plain bool hasWrittenCurrent = false. Fix this →

---

REVIEW_RESULT: FAIL ❌

[claude]

Performance bug — participants lock acquired every frame when room has no audio

When audioSources.Count == 0 (e.g. room has no audio tracks yet, or after Stop()), anyDied stays false, so !anyDied && audioSources.Count > 0 evaluates to false and the method always falls through to ReleaseAllAudioSources() + OpenAllAudioStreams(). OpenAllAudioStreams acquires lock(room.Participants) and iterates all participants — every frame, for the lifetime of the player when no audio is present.

Fix: track a separate flag for "attempted to open audio, found nothing".

Suggested change

	if (!anyDied && audioSources.Count > 0) return;
	if (!anyDied && audioSources.Count > 0) return;
	if (audioSources.Count == 0 && !anyDied)
	{
	// No sources and nothing died — room has no audio tracks.
	// Only re-check when the frame budget allows (or when first opened).
	OpenAllAudioStreams();
	return;
	}

Simpler fix: add an audioInitialized bool that is set to true in OpenMedia/OpenAllAudioStreams, and only skip the redundant re-poll when audioInitialized && audioSources.Count == 0.

[claude]

Dead code — FindPairedAudio is never called

This private method has no callers. Per the project's code standards: "Don't create helpers, utilities, or abstractions for one-time operations. Don't design for hypothetical future requirements."

The docs note it's reserved for "future targeted audio scenarios", but shipping dead code adds maintenance burden and cognitive noise. Remove it until it's actually needed.

[cyaiox]

I kept it in case it will be required in the future, as livekit sends Video and Audio in separate tracks, this utility helps to match the participant Video with their respective Audio.

[claude]

Audio disruption on every video fallback

OpenMedia(LivekitAddress.CurrentStream()) calls CloseCurrentStream() first, which calls ReleaseAllAudioSources() — releasing and re-pooling all audio MonoBehaviour objects — then immediately re-acquires them in OpenAllAudioStreams(). This causes a brief audio dropout on every video recovery cycle, even when the audio streams are perfectly healthy.

Consider splitting the video and audio reset paths so a dying video track doesn't interrupt audio that is still alive:

Suggested change

	OpenMedia(LivekitAddress.CurrentStream());
	// Video stream died — swap to first available track without resetting audio
	if (playingAddress.Value.IsUserStream(out _))
	{
	currentVideoIdentity = null;
	currentVideoStream = FirstVideoTrackingIdentity();
	playingAddress = LivekitAddress.CurrentStream();
	videoSwitchedAtTime = UnityEngine.Time.realtimeSinceStartup;
	playerState = PlayerState.PLAYING;
	return;
	}

[claude]

Silent exception swallowing — invalid metadata is silently ignored

catch (JsonException) { } discards all parse failures without any trace. If a participant sends malformed metadata this will silently fall through to the Name/Identity fallback, making it very hard to diagnose in the field.

;

Suggested change

	catch (JsonException) { }
	catch (JsonException e) { ReportHub.LogWarning(ReportCategory.COMMS, $"Failed to parse participant metadata for display name: {e.Message}"); }

[NickKhalow]

@NickKhalow - replying to #7347 (comment)

With new routing CommsData = 3, // CommsApi publish/subscribe topic-based data, I believe the catch is excessive. Let's remove it

the catch handles unexpected participants on the shared IDataPipe that send arbitrary non-protobuf bytes into scene data, this is purely a defensive fallback for unexpected senders.

Hey @cyaiox, I see the intention. It can be useful for prod, but I would exclude it for development for sure, otherwise that catch silently swallows the data that could be important for the context and traceability.

[NickKhalow]

Thank you for your updates! Please address the comments

[NickKhalow]

I got the idea, makes a sense. Would be great to enforce the invariant in the code, because there is always a chance that new method may forget to reset the JTW.

WDYT?

UPD: addressed

[NickKhalow]

second LivekitAddress is redundant due the following:

FromIdentity is used only in LiveKitPlayer. LiveKitPlayer is NOT responsible for issuing the LiveKitAddress by design. It only executes the address like a command and never creates that internally

Please remove FromIdentity method. IsPresentationBotIdentity should be placed in LivekitAddress.New method, it will handle the union resolution and LivekitPlayer never needs to create a value.

[NickKhalow]

It would be better to avoid such defensive programming in concurrent scenarios and solve the root cause in place. Would it be possible to address the issue here?

UPD: addressed