Skip to content

Commit 8596f21

Browse files
authored
add image tracking to v25 (#232)
## Description Describe your changes in detail
1 parent 2809191 commit 8596f21

1 file changed

Lines changed: 72 additions & 0 deletions

File tree

versioned_docs/version-0.25.0/tutorials/agents.mdx

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -266,6 +266,78 @@ You can interrupt the currently played audio chunk. See the example below.
266266

267267
</Tabs>
268268

269+
### Making the Agent see
270+
271+
Agents can also request video frames (JPEG images) from peers' video tracks.
272+
Unlike audio, which streams continuously, video frames must be explicitly requested and arrive asynchronously.
273+
274+
:::important
275+
Video frame capture is rate-limited to one frame per second per track.
276+
:::
277+
278+
<Tabs groupId="language">
279+
<TabItem value="ts" label="TypeScript">
280+
281+
```ts
282+
// @noErrors
283+
import { RoomId, FishjamClient, TrackId } from '@fishjam-cloud/js-server-sdk';
284+
285+
const fishjamId = '';
286+
const managementToken = '';
287+
const fishjamClient = new FishjamClient({ fishjamId, managementToken });
288+
const room = await fishjamClient.createRoom();
289+
const { agent } = await fishjamClient.createAgent(room.id, {});
290+
const trackId: TrackId = '' as TrackId;
291+
292+
// ---cut---
293+
import type { IncomingTrackImage } from '@fishjam-cloud/js-server-sdk';
294+
295+
// Listen for incoming video frames
296+
agent.on('trackImage', (message: IncomingTrackImage) => {
297+
const { contentType, data } = message;
298+
// process the image data
299+
});
300+
301+
// Request a frame periodically
302+
setInterval(() => {
303+
// [!code highlight:1]
304+
agent.captureImage(trackId);
305+
}, 1000);
306+
307+
```
308+
309+
</TabItem>
310+
311+
<TabItem value="python" label="Python">
312+
313+
```python
314+
import asyncio
315+
316+
from fishjam import FishjamClient
317+
from fishjam.agent import IncomingTrackImage
318+
319+
fishjam_client = FishjamClient(fishjam_id, management_token)
320+
321+
agent = fishjam_client.create_agent(room_id)
322+
323+
async with agent.connect() as session:
324+
# Request a frame
325+
# [!code highlight:1]
326+
await session.capture_image(track_id)
327+
328+
# Captured frames arrive as IncomingTrackImage messages
329+
async for message in session.receive():
330+
match message:
331+
case IncomingTrackImage() as msg if msg.track_id == track_id:
332+
data = msg.data
333+
# process the image data
334+
pass
335+
```
336+
337+
</TabItem>
338+
339+
</Tabs>
340+
269341
### Disconnecting
270342

271343
After you're done using an agent, you can disconnect it from the room.

0 commit comments

Comments
 (0)