Record outbound phone calls + 2-channel socket by mdmohsin7 · Pull Request #4416 · BasedHardware/omi

mdmohsin7 · 2026-01-27T18:39:34Z

End-to-End Flow

1. Phone Number Verification

User enters phone number → Backend validates E.164 format
Twilio places automated verification call with 6-digit code
User answers, enters code → Number stored encrypted in Firestore (users/{uid}/phone_numbers/)
Security: SHA256 hash for lookups, AES-GCM encryption for storage, 5-min pending verification TTL

2. Call Initiation

App requests Twilio access token (JWT with user's UID as identity)
Native SDK (TwilioVoice) initialized with token
Microphone permission requested, call placed with unique callId

3. Call Routing (TwiML Webhook)

Twilio invokes POST /v1/phone/twiml with request signature
Backend validates:
- Twilio signature
- User's verified caller ID exists
- Destination number E.164 format
- Caller ID still valid in Twilio
Returns TwiML: <Dial callerId="+15551234567">+15559876543</Dial>
Twilio bridges call: User ↔ Twilio ↔ Recipient

4. Recording & Transcription

Audio Capture

Native SDK captures 2-channel PCM16 @ 48kHz:
- Channel 0x01: User microphone
- Channel 0x02: Remote party audio
Each audio chunk prefixed with channel byte: [0x01|0x02][audio_data]

Streaming

WebSocket to wss://api/v4/listen?source=phone_call&channels=2
Multi-channel support integrated directly into the existing /v4/listen route
Firebase auth via get_current_user_uid dependency
Auto-reconnect with exponential backoff (1s → 16s, max 5 attempts)

Backend Processing

When channels >= 2, activates multi-channel mode within the existing stream handler
Routes each channel to its own STT connection (Deepgram/Soniox/Speechmatics)
Real-time transcription with speaker labels:
- Channel 1 → SPEAKER_00 (is_user: true)
- Channel 2 → SPEAKER_01 (is_user: false)
Uses standard segment array format (same as all other sources)
One conversation per call session (no silence-based splitting)

Storage

Raw PCM16 uploaded to GCS every 5s
Encrypted with user-specific AES-GCM keys
Channels saved separately for diarization

5. Call Termination

User/remote ends call → Native SDK disconnects
WebSocket closes → Backend finalizes conversation
Structured output: title, overview, category, action items
Saved with source: phone_call, standard UUID conversation ID

Required Environment Variables

The following env vars must be set on the backend for phone call functionality:

Variable	Description
`TWILIO_ACCOUNT_SID`	Twilio account SID
`TWILIO_AUTH_TOKEN`	Twilio auth token (used for request signature validation)
`TWILIO_API_KEY_SID`	Twilio API key SID (used for generating access tokens)
`TWILIO_API_KEY_SECRET`	Twilio API key secret
`TWILIO_TWIML_APP_SID`	Twilio TwiML app SID (routes calls to the `/v1/phone/twiml` webhook)
`ENCRYPTION_SECRET`	Secret for AES-GCM encryption of stored phone numbers

External.Device-trimmed.mp4

gemini-code-assist

Code Review

This pull request introduces a new multi-channel audio transcription WebSocket endpoint, along with supporting utilities for conversation management, Pusher integration, and translation. The changes include creating new files for streaming utilities and modifying the backend router to handle the new endpoint. A critical issue related to error handling and fallback strategy has been identified and needs to be addressed.

mdmohsin7 · 2026-01-27T19:05:45Z

/gemini summary

gemini-code-assist · 2026-01-27T19:08:19Z

Summary of Changes

This pull request adds phone call functionality to the application, leveraging the Twilio Voice SDK for call management and real-time transcription. It incorporates native platform features like CallKit on iOS for a seamless user experience and introduces a new multi-channel audio processing pipeline. The changes include backend endpoints for token generation and phone number verification, as well as Flutter plugins for handling call initiation, control, and audio streaming.

Highlights

Twilio Voice SDK Integration: This PR introduces the Twilio Voice SDK to enable outbound phone call functionality within the app.
Real-time Transcription: The implementation includes real-time transcription of phone calls, providing users with live transcripts of their conversations.
CallKit Integration (iOS): For iOS, the PR integrates with CallKit to provide a native call experience, including handling incoming and outgoing calls through the system's call interface.
Multi-Channel Audio Support: The PR implements multi-channel audio support, allowing for the capture and processing of both the user's microphone input and the remote audio stream.
Phone Number Verification: The PR includes phone number verification to ensure that users can only make calls from verified numbers.

Changelog

app/android/app/build.gradle
- Added Twilio Voice SDK dependency.
app/android/app/src/main/kotlin/com/example/my_project/MainActivity.kt
- Registered the PhoneCallsPlugin with the Flutter engine.
app/android/app/src/main/kotlin/com/friend/ios/PhoneCallsPlugin.kt
- Added a new Flutter plugin for handling phone call functionality using the Twilio Voice SDK on Android.
- Implements method channel communication for call initialization, making calls, ending calls, muting, and toggling the speaker.
- Uses an event channel to stream call state changes to Flutter.
app/ios/Podfile
- Added TwilioVoice pod dependency.
app/ios/Podfile.lock
- Updated to include TwilioVoice and pasteboard dependencies, along with associated checksums and specifications.
app/ios/Runner/AppDelegate.swift
- Registered the PhoneCallsPlugin with the Flutter engine.
app/ios/Runner/Info.plist
- Added voip to the list of UIBackgroundModes.
app/ios/Runner/OmiRecordingAudioDevice.swift
- Added a custom audio device to capture both local and remote audio streams using Twilio Voice SDK and Core Audio.
- Implements audio unit setup, rendering, and capturing for real-time audio processing.
app/ios/Runner/PhoneCallsPlugin.swift
- Added a Flutter plugin for handling phone call functionality using the Twilio Voice SDK on iOS.
- Integrates with CallKit for native call management and user interface.
- Implements method channel communication for call initialization, making calls, ending calls, muting, and toggling the speaker.
- Uses an event channel to stream call state changes and audio data to Flutter.
app/lib/backend/http/api/phone_calls.dart
- Added API functions for phone number verification, checking verification status, retrieving verified phone numbers, deleting verified phone numbers, and generating Twilio access tokens.
app/lib/backend/schema/phone_call.dart
- Added data models for phone call-related data, including PhoneTranscriptSegment, VerifiedPhoneNumber, and PhoneCallToken.
app/lib/backend/schema/schema.dart
- Exported the new phone_call.dart schema.
app/lib/main.dart
- Added PhoneCallProvider to the list of ChangeNotifierProviders.
app/lib/pages/home/page.dart
- Added a floating action button to navigate to the PhoneCallsPage or PhoneSetupIntroPage based on the user's verification status.
app/lib/pages/phone_calls/active_call_page.dart
- Added a new page for displaying the active phone call UI, including call info, live transcript, and call controls.
app/lib/pages/phone_calls/phone_calls_page.dart
- Added a new page for initiating phone calls, with tabs for contacts and a dialpad.
app/lib/pages/phone_calls/phone_setup_intro_page.dart
- Added a new page to guide users through the phone number verification process.
app/lib/pages/phone_calls/phone_setup_number_page.dart
- Added a new page for entering and verifying the user's phone number.
app/lib/pages/phone_calls/phone_setup_verify_page.dart
- Added a new page for verifying the user's phone number using a validation code.
app/lib/pages/settings/phone_call_settings_page.dart
- Added a new page for managing verified phone numbers.
app/lib/pages/settings/settings_drawer.dart
- Added a navigation item for phone call settings.
app/lib/providers/phone_call_provider.dart
- Added a new provider to manage phone call state, Twilio SDK initialization, call control, and transcription.
app/lib/services/phone_call_service.dart
- Added a new service for handling native method channel calls related to phone call functionality.
backend/database/phone_calls.py
- Added database functions for managing verified phone numbers, including upsert, retrieval, and deletion.
backend/main.py
- Included the phone_calls router in the main FastAPI application.
backend/requirements.txt
- Added the twilio package as a dependency.
backend/routers/multichannel.py
- Added a new router for handling multi-channel audio streams, including phone calls and desktop audio.
- Implements WebSocket endpoints for receiving audio, processing transcriptions, and managing conversation state.
backend/utils/streaming/init.py
- Added streaming utility modules.
backend/utils/streaming/conversation_manager.py
- Added conversation management utilities.
backend/utils/streaming/pusher_handler.py
- Added Pusher handler utilities.
backend/utils/streaming/translator.py
- Added translation utilities.
backend/utils/streaming/usage_tracker.py
- Added usage tracking utilities.
backend/utils/twilio_service.py
- Added a new module for interacting with the Twilio API, including token generation, caller ID verification, and call management.

Activity

mdmohsin7 requested a Gemini summary.
gemini-code-assist[bot] suggested a more informative error message or a more intelligent fallback strategy for the Deepgram STT service in backend/routers/multichannel.py.

beastoin · 2026-01-28T04:12:47Z

@mdmohsin7 The TwiML webhook in phone_calls.py:200 lacks Twilio signature validation so anyone can POST and trigger calls, phone verification is account-level not user-bound in phone_calls.py:120 which lets users claim/delete each other's numbers, /verify/check has no rate limiting (1s polling hits Twilio API), multichannel WS accepts unbounded channels in multichannel.py:117 (DoS risk), TwiML To is not validated to E.164 in phone_calls.py:209, PII is logged in plaintext in phone_calls.py:214, and iOS OmiRecordingAudioDevice has a buffer alignment issue with potential leaks; plus there are no tests for the new endpoints or mobile plugins, though the separation of streaming utils (pusher_handler/usage_tracker/conversation_manager) is solid—can you address these and add coverage, and let me know when it's ready for re-review?

Reviewed by @kenji

mdmohsin7 · 2026-01-28T17:50:35Z

/gemini summary

gemini-code-assist · 2026-01-28T17:56:55Z

Summary of Changes

This pull request introduces phone call functionality to the application, leveraging the Twilio Voice SDK for call management and real-time transcription. It incorporates native platform features like CallKit on iOS for a seamless user experience and introduces a new multi-channel audio processing pipeline. The changes include backend endpoints for token generation and phone number verification, as well as Flutter plugins for handling call initiation, control, and audio streaming.

Highlights

Twilio Voice SDK Integration: This PR introduces the Twilio Voice SDK to enable outbound phone call functionality within the app.
Real-time Transcription: The implementation includes real-time transcription of phone calls, providing users with live transcripts of their conversations.
CallKit Integration (iOS): For iOS, the PR integrates with CallKit to provide a native call experience, including handling incoming and outgoing calls through the system's call interface.
Multi-Channel Audio Support: The PR implements multi-channel audio support, allowing for the capture and processing of both the user's microphone input and the remote audio stream.
Phone Number Verification: The PR includes phone number verification to ensure that users can only make calls from verified numbers.

Changelog

app/android/app/build.gradle
- Added Twilio Voice SDK dependency.
app/android/app/src/main/kotlin/com/example/my_project/MainActivity.kt
- Registered the PhoneCallsPlugin with the Flutter engine.
app/android/app/src/main/kotlin/com/friend/ios/PhoneCallsPlugin.kt
- Added a new Flutter plugin for handling phone call functionality using the Twilio Voice SDK on Android.
- Implements method channel communication for call initialization, making calls, ending calls, muting, and toggling the speaker.
- Uses an event channel to stream call state changes to Flutter.
app/ios/Podfile
- Added TwilioVoice pod dependency.
app/ios/Podfile.lock
- Updated to include TwilioVoice and pasteboard dependencies, along with associated checksums and specifications.
app/ios/Runner/AppDelegate.swift
- Registered the PhoneCallsPlugin with the Flutter engine.
app/ios/Runner/Info.plist
- Added voip to the list of UIBackgroundModes.
app/ios/Runner/OmiRecordingAudioDevice.swift
- Added a custom audio device to capture both local and remote audio streams using Twilio Voice SDK and Core Audio.
- Implements audio unit setup, rendering, and capturing for real-time audio processing.
app/ios/Runner/PhoneCallsPlugin.swift
- Added a Flutter plugin for handling phone call functionality using the Twilio Voice SDK on iOS.
- Integrates with CallKit for native call management and user interface.
- Implements method channel communication for call initialization, making calls, ending calls, muting, and toggling the speaker.
- Uses an event channel to stream call state changes and audio data to Flutter.
app/lib/backend/http/api/phone_calls.dart
- Added API functions for phone number verification, checking verification status, retrieving verified phone numbers, deleting verified phone numbers, and generating Twilio access tokens.
app/lib/backend/schema/phone_call.dart
- Added data models for phone call-related data, including PhoneTranscriptSegment, VerifiedPhoneNumber, and PhoneCallToken.
app/lib/backend/schema/schema.dart
- Exported the new phone_call.dart schema.
app/lib/main.dart
- Added PhoneCallProvider to the list of ChangeNotifierProviders.
app/lib/pages/home/page.dart
- Added a floating action button to navigate to the PhoneCallsPage or PhoneSetupIntroPage based on the user's verification status.
app/lib/pages/phone_calls/active_call_page.dart
- Added a new page for displaying the active phone call UI, including call info, live transcript, and call controls.
app/lib/pages/phone_calls/phone_calls_page.dart
- Added a new page for initiating phone calls, with tabs for contacts and a dialpad.
app/lib/pages/phone_calls/phone_setup_intro_page.dart
- Added a new page to guide users through the phone number verification process.
app/lib/pages/phone_calls/phone_setup_number_page.dart
- Added a new page for entering and verifying the user's phone number.
app/lib/pages/phone_calls/phone_setup_verify_page.dart
- Added a new page for verifying the user's phone number using a validation code.
app/lib/pages/settings/phone_call_settings_page.dart
- Added a new page for managing verified phone numbers.
app/lib/pages/settings/settings_drawer.dart
- Added a navigation item for phone call settings.
app/lib/providers/phone_call_provider.dart
- Added a new provider to manage phone call state, Twilio SDK initialization, call control, and transcription.
app/lib/services/phone_call_service.dart
- Added a new service for handling native method channel calls related to phone call functionality.
backend/database/phone_calls.py
- Added database functions for managing verified phone numbers, including upsert, retrieval, and deletion.
backend/main.py
- Included the phone_calls router in the main FastAPI application.
backend/requirements.txt
- Added the twilio package as a dependency.
backend/routers/multichannel.py
- Added a new router for handling multi-channel audio streams, including phone calls and desktop audio.
- Implements WebSocket endpoints for receiving audio, processing transcriptions, and managing conversation state.
backend/utils/streaming/init.py
- Added streaming utility modules.
backend/utils/streaming/conversation_manager.py
- Added conversation management utilities.
backend/utils/streaming/pusher_handler.py
- Added Pusher handler utilities.
backend/utils/streaming/translator.py
- Added translation utilities.
backend/utils/streaming/usage_tracker.py
- Added usage tracking utilities.
backend/utils/twilio_service.py
- Added a new module for interacting with the Twilio API, including token generation, caller ID verification, and call management.

Activity

mdmohsin7 requested a Gemini summary.
gemini-code-assist[bot] suggested a more informative error message or a more intelligent fallback strategy for the Deepgram STT service in backend/routers/multichannel.py.
beastoin noted several potential issues, including lack of Twilio signature validation, account-level phone verification, missing rate limiting, unbounded channels, lack of E.164 validation, PII logging, and a buffer alignment issue, and requested that these be addressed and that tests be added.
mdmohsin7 requested a Gemini summary.

mdmohsin7 · 2026-01-28T18:06:20Z

@mdmohsin7 The TwiML webhook in phone_calls.py:200 lacks Twilio signature validation so anyone can POST and trigger calls, phone verification is account-level not user-bound in phone_calls.py:120 which lets users claim/delete each other's numbers, /verify/check has no rate limiting (1s polling hits Twilio API), multichannel WS accepts unbounded channels in multichannel.py:117 (DoS risk), TwiML To is not validated to E.164 in phone_calls.py:209, PII is logged in plaintext in phone_calls.py:214, and iOS OmiRecordingAudioDevice has a buffer alignment issue with potential leaks; plus there are no tests for the new endpoints or mobile plugins, though the separation of streaming utils (pusher_handler/usage_tracker/conversation_manager) is solid—can you address these and add coverage, and let me know when it's ready for re-review?

Reviewed by @kenji

Done, pls check now @beastoin

beastoin · 2026-01-29T02:08:19Z

PR Review: Record outbound phone calls + 2-channel socket

Substantial feature adding Twilio-based phone call recording with dual-channel audio streaming. Good test coverage included.

Issues to Address

1. Twilio signature validation bypass when auth_token missing (backend/utils/twilio_service.py:175-177)

if not auth_token:
    return True  # <-- Security risk: bypasses validation

This silently allows unauthenticated requests when TWILIO_AUTH_TOKEN is not set. The webhook at /v1/phone/twiml would be exploitable. Fix: Return False or raise an exception when auth_token is missing.

2. Inconsistent package path (app/android/app/src/main/kotlin/com/friend/ios/)
Android Kotlin files are under com/friend/ios/ package path. This is confusing and should be com/friend/android/ or similar.

3. Missing rate limiting on verification endpoints
/v1/phone/numbers/verify could be abused for toll fraud or phone verification spam. Consider adding rate limiting per-user and per-phone-number.

4. Phone number validation could be stricter (backend/routers/phone_calls.py)
The E164 regex allows very long numbers. Consider adding a max length check (E.164 max is 15 digits).

Minor Suggestions

multichannel.py:115: Channel count validation rejects >2 channels but build_channel_config has generic N-channel logic. Either support N channels or remove the unused generic code.
Consider adding call duration limits to prevent runaway charges.
OmiRecordingAudioDevice (both platforms): No explicit resource cleanup on exceptions during capture/render runnables.

Positive Notes

Phone number encryption with hash-based lookup is well designed
Twilio signature validation on TwiML webhook is correct pattern (when auth_token present)
Good unit test coverage for router endpoints
Proper pending verification TTL prevents stale verification records

Verdict: Address item #1 (security issue) before merging. Items #2-4 can be follow-up tasks.

by AI for @beastoin

mdmohsin7 · 2026-01-29T06:23:15Z

Inconsistent package path (app/android/app/src/main/kotlin/com/friend/ios/)
Android Kotlin files are under com/friend/ios/ package path. This is confusing and should be com/friend/android/ or similar.

It's com/friend/ios coz the package name is com.friend.ios

Rest all fixed, pls check now @beastoin

beastoin · 2026-01-29T06:41:05Z

@mdmohsin7 In backend/routers/phone_calls.py:141-166, /v1/phone/numbers/verify/check only blocks when pending_uid is set for another user, so a caller can claim any already-verified Twilio caller ID by hitting the check endpoint when there is no pending record; please require pending_uid == uid (return verified=false when missing/expired) before calling check_caller_id_verified and add a regression test for the no-pending case. Also in backend/routers/phone_calls.py:88-91, validation only checks startswith('+') even though E164_PATTERN exists, which will accept invalid numbers and persist bad data—please validate against E164_PATTERN (or normalize) before starting verification.

Can you update the verification flow and add the missing test?

by AI for @beastoin

mdmohsin7 · 2026-01-29T08:57:38Z

final fixes done, pls check again @beastoin's AI 🤖

aaravgarg · 2026-02-01T08:18:49Z

conflicts @mdmohsin7

beastoin · 2026-02-02T08:35:32Z

@mdmohsin7 Before review, quick questions:

Why is this feature important? What user problem does it solve?
Quick checklist for review:
- iOS working?
- Android working?
- User consent flow included?
- Tested with real calls?

Thanks!

by AI for @beastoin

aaravgarg · 2026-02-05T20:07:49Z

@mdmohsin7 lets roll this out soon pls

Update transcript parsing to handle standard segment array format. Reset call state to idle on setup failures so retries work. Move mic permission request before SDK initialization. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Required for permission_handler to show the mic permission prompt. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Mix both channels sample-by-sample before sending to pusher so the stored audio is a proper mono stream. Connect pusher at TARGET_SAMPLE_RATE (16kHz) to match the resampled audio. Use standard UUID for conversation IDs instead of call_id.

Use openAppSettings() from permission_handler instead of unreliable platform-specific URL schemes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

This reverts commit 6d97fc7.

When contacts permission was previously denied, tapping Allow did nothing because iOS won't re-show the prompt. Now checks permission status and opens app settings if denied/permanently denied. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

The delete button was always targeting the first verified number. Moved delete icon into per-row widget so each row deletes the correct number. Also converted all strings to use l10n.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mdmohsin7 · 2026-03-07T10:33:18Z

Deployment Steps

1. Twilio Account Setup

Create a Twilio account (or use existing) and complete KYC verification
Same account works for both prod and dev

2. Create Twilio API Key

Go to Twilio Console → Account → API keys & tokens → Create API Key
Save the API Key SID (TWILIO_API_KEY_SID) and API Key Secret (TWILIO_API_KEY_SECRET)
One key can be shared across prod/dev, or create separate keys per environment

3. Create TwiML Apps (one per environment)

Go to Twilio Console → Voice → TwiML Apps → Create TwiML App
Dev TwiML App:
- Voice Request URL: https://api.omiapi.com/v1/phone/twiml (POST)
Prod TwiML App:
- Voice Request URL: https://api.omi.me/v1/phone/twiml (POST)
Save each app's SID as TWILIO_TWIML_APP_SID

4. Add Kubernetes Secrets

Add these 5 secrets to both clusters:

dev-omi-backend-secrets:

TWILIO_ACCOUNT_SID=<account_sid>
TWILIO_AUTH_TOKEN=<auth_token>
TWILIO_API_KEY_SID=<api_key_sid>
TWILIO_API_KEY_SECRET=<api_key_secret>
TWILIO_TWIML_APP_SID=<dev_twiml_app_sid>

prod-omi-backend-secrets: (same values except TWILIO_TWIML_APP_SID points to prod TwiML app)

5. Deploy

Merge this PR — dev auto-deploys via gcp_backend_auto_dev.yml
For prod: gh workflow run gcp_backend.yml -f environment=prod -f branch=main

# Conflicts: # app/lib/l10n/app_ar.arb # app/lib/l10n/app_bg.arb # app/lib/l10n/app_ca.arb # app/lib/l10n/app_cs.arb # app/lib/l10n/app_da.arb # app/lib/l10n/app_de.arb # app/lib/l10n/app_el.arb # app/lib/l10n/app_en.arb # app/lib/l10n/app_es.arb # app/lib/l10n/app_et.arb # app/lib/l10n/app_fi.arb # app/lib/l10n/app_fr.arb # app/lib/l10n/app_hi.arb # app/lib/l10n/app_hu.arb # app/lib/l10n/app_id.arb # app/lib/l10n/app_it.arb # app/lib/l10n/app_ja.arb # app/lib/l10n/app_ko.arb # app/lib/l10n/app_localizations.dart # app/lib/l10n/app_localizations_ar.dart # app/lib/l10n/app_localizations_bg.dart # app/lib/l10n/app_localizations_ca.dart # app/lib/l10n/app_localizations_cs.dart # app/lib/l10n/app_localizations_da.dart # app/lib/l10n/app_localizations_de.dart # app/lib/l10n/app_localizations_el.dart # app/lib/l10n/app_localizations_en.dart # app/lib/l10n/app_localizations_es.dart # app/lib/l10n/app_localizations_et.dart # app/lib/l10n/app_localizations_fi.dart # app/lib/l10n/app_localizations_fr.dart # app/lib/l10n/app_localizations_hi.dart # app/lib/l10n/app_localizations_hu.dart # app/lib/l10n/app_localizations_id.dart # app/lib/l10n/app_localizations_it.dart # app/lib/l10n/app_localizations_ja.dart # app/lib/l10n/app_localizations_ko.dart # app/lib/l10n/app_localizations_lt.dart # app/lib/l10n/app_localizations_lv.dart # app/lib/l10n/app_localizations_ms.dart # app/lib/l10n/app_localizations_nl.dart # app/lib/l10n/app_localizations_no.dart # app/lib/l10n/app_localizations_pl.dart # app/lib/l10n/app_localizations_pt.dart # app/lib/l10n/app_localizations_ro.dart # app/lib/l10n/app_localizations_ru.dart # app/lib/l10n/app_localizations_sk.dart # app/lib/l10n/app_localizations_sv.dart # app/lib/l10n/app_localizations_th.dart # app/lib/l10n/app_localizations_tr.dart # app/lib/l10n/app_localizations_uk.dart # app/lib/l10n/app_localizations_vi.dart # app/lib/l10n/app_localizations_zh.dart # app/lib/l10n/app_lt.arb # app/lib/l10n/app_lv.arb # app/lib/l10n/app_ms.arb # app/lib/l10n/app_nl.arb # app/lib/l10n/app_no.arb # app/lib/l10n/app_pl.arb # app/lib/l10n/app_pt.arb # app/lib/l10n/app_ro.arb # app/lib/l10n/app_ru.arb # app/lib/l10n/app_sk.arb # app/lib/l10n/app_sv.arb # app/lib/l10n/app_th.arb # app/lib/l10n/app_tr.arb # app/lib/l10n/app_uk.arb # app/lib/l10n/app_vi.arb # app/lib/l10n/app_zh.arb

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

## End-to-End Flow ### 1. Phone Number Verification - User enters phone number → Backend validates E.164 format - Twilio places automated verification call with 6-digit code - User answers, enters code → Number stored encrypted in Firestore (`users/{uid}/phone_numbers/`) - Security: SHA256 hash for lookups, AES-GCM encryption for storage, 5-min pending verification TTL ### 2. Call Initiation - App requests Twilio access token (JWT with user's UID as identity) - Native SDK (TwilioVoice) initialized with token - Microphone permission requested, call placed with unique `callId` ### 3. Call Routing (TwiML Webhook) - Twilio invokes `POST /v1/phone/twiml` with request signature - Backend validates: - Twilio signature - User's verified caller ID exists - Destination number E.164 format - Caller ID still valid in Twilio - Returns TwiML: `<Dial callerId="+15551234567">+15559876543</Dial>` - Twilio bridges call: User ↔ Twilio ↔ Recipient ### 4. Recording & Transcription #### Audio Capture - Native SDK captures 2-channel PCM16 @ 48kHz: - Channel 0x01: User microphone - Channel 0x02: Remote party audio - Each audio chunk prefixed with channel byte: `[0x01|0x02][audio_data]` #### Streaming - WebSocket to `wss://api/v4/listen?source=phone_call&channels=2` - Multi-channel support integrated directly into the existing `/v4/listen` route - Firebase auth via `get_current_user_uid` dependency - Auto-reconnect with exponential backoff (1s → 16s, max 5 attempts) #### Backend Processing - When `channels >= 2`, activates multi-channel mode within the existing stream handler - Routes each channel to its own STT connection (Deepgram/Soniox/Speechmatics) - Real-time transcription with speaker labels: - Channel 1 → `SPEAKER_00` (is_user: true) - Channel 2 → `SPEAKER_01` (is_user: false) - Uses standard segment array format (same as all other sources) - One conversation per call session (no silence-based splitting) #### Storage - Raw PCM16 uploaded to GCS every 5s - Encrypted with user-specific AES-GCM keys - Channels saved separately for diarization ### 5. Call Termination - User/remote ends call → Native SDK disconnects - WebSocket closes → Backend finalizes conversation - Structured output: title, overview, category, action items - Saved with `source: phone_call`, standard UUID conversation ID ## Required Environment Variables The following env vars must be set on the backend for phone call functionality: | Variable | Description | |----------|-------------| | `TWILIO_ACCOUNT_SID` | Twilio account SID | | `TWILIO_AUTH_TOKEN` | Twilio auth token (used for request signature validation) | | `TWILIO_API_KEY_SID` | Twilio API key SID (used for generating access tokens) | | `TWILIO_API_KEY_SECRET` | Twilio API key secret | | `TWILIO_TWIML_APP_SID` | Twilio TwiML app SID (routes calls to the `/v1/phone/twiml` webhook) | | `ENCRYPTION_SECRET` | Secret for AES-GCM encryption of stored phone numbers | https://github.com/user-attachments/assets/853ddf22-32ed-4c53-aeb4-7f2ce950f6ac

mdmohsin7 added 2 commits January 28, 2026 00:06

create utils from transcribe

1ba62d6

n-channel socket

a4a2c17

gemini-code-assist Bot reviewed Jan 27, 2026

View reviewed changes

Comment thread backend/routers/multichannel.py Outdated

mdmohsin7 added 3 commits January 28, 2026 00:23

twilio setup

ed1c010

phone calls on app side (swift and kotlin)

e48a428

misc

30851ad

mdmohsin7 marked this pull request as ready for review January 27, 2026 19:56

mdmohsin7 requested a review from beastoin January 27, 2026 19:56

mdmohsin7 and others added 4 commits January 28, 2026 23:04

android + bug fixes

44baf39

wbhook validation + fix channel + other fixes

544ee72

Merge branch 'main' into phone-calls

9a3e2b3

add tests

fe54a5d

mdmohsin7 changed the title ~~Record outbound phone calls + n-channel socket~~ Record outbound phone calls + 2-channel socket Jan 28, 2026

bug fixes

2699747

bug fixes

c01a4e3

mdmohsin7 and others added 2 commits February 5, 2026 22:32

Merge branch 'main' into phone-calls

ccdaccb

misc

bb56198

mdmohsin7 and others added 22 commits February 28, 2026 19:34

fix: add PERMISSION_MICROPHONE macro to iOS Podfile

805dd1b

Required for permission_handler to show the mic permission prompt. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into phone-calls

1149678

Fix contacts permission denied button to open app settings

6d97fc7

Use openAppSettings() from permission_handler instead of unreliable platform-specific URL schemes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Revert "Fix contacts permission denied button to open app settings"

6a30ce6

This reverts commit 6d97fc7.

Fix phone call settings delete targeting wrong number

b994211

The delete button was always targeting the first verified number. Moved delete icon into per-row widget so each row deletes the correct number. Also converted all strings to use l10n.

Add call_id field to Conversation model for phone calls

073d021

Store call_id on conversation for multi-channel phone calls

dbac3aa

Add l10n to active call page

7c16ffb

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add l10n to phone calls page

5ef4706

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add l10n to phone setup intro page

6645109

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add l10n to phone setup number page

aed0d0a

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add l10n to phone setup verify page

1dcf498

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add phone call l10n keys to English ARB

7b95e58

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add phone call l10n translations for all 33 locales

ba854b5

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Regenerate l10n Dart files for phone call translations

7842665

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/main' into phone-calls

585cb85

Add Twilio env vars to .env.template

62bfb21

Add Twilio secrets to prod backend-listen Helm values

856a5fb

Add Twilio secrets to dev backend-listen Helm values

6e5e059

mdmohsin7 and others added 3 commits March 9, 2026 16:41

Add Twilio secrets to dev ESO external secret values

ce042fb

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Add Twilio secrets to prod ESO external secret values

a37bba6

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

mdmohsin7 merged commit cff2c53 into main Mar 9, 2026
1 check passed

mdmohsin7 deleted the phone-calls branch March 9, 2026 12:13

beastoin mentioned this pull request Mar 30, 2026

Multi-channel STT: use single Deepgram connection with multichannel=true instead of N connections #6167

Open

Conversation

mdmohsin7 commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

End-to-End Flow

1. Phone Number Verification

2. Call Initiation

3. Call Routing (TwiML Webhook)

4. Recording & Transcription

Audio Capture

Streaming

Backend Processing

Storage

5. Call Termination

Required Environment Variables

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

mdmohsin7 commented Jan 27, 2026

Uh oh!

gemini-code-assist Bot commented Jan 27, 2026

Summary of Changes

Highlights

Uh oh!

beastoin commented Jan 28, 2026

Uh oh!

mdmohsin7 commented Jan 28, 2026

Uh oh!

gemini-code-assist Bot commented Jan 28, 2026

Summary of Changes

Highlights

Uh oh!

mdmohsin7 commented Jan 28, 2026

Uh oh!

beastoin commented Jan 29, 2026

PR Review: Record outbound phone calls + 2-channel socket

Issues to Address

Minor Suggestions

Positive Notes

Uh oh!

mdmohsin7 commented Jan 29, 2026

Uh oh!

beastoin commented Jan 29, 2026

Uh oh!

mdmohsin7 commented Jan 29, 2026

Uh oh!

aaravgarg commented Feb 1, 2026

Uh oh!

beastoin commented Feb 2, 2026 • edited by mdmohsin7 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aaravgarg commented Feb 5, 2026

Uh oh!

mdmohsin7 commented Mar 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Deployment Steps

1. Twilio Account Setup

2. Create Twilio API Key

3. Create TwiML Apps (one per environment)

4. Add Kubernetes Secrets

5. Deploy

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mdmohsin7 commented Jan 27, 2026 •

edited

Loading

beastoin commented Feb 2, 2026 •

edited by mdmohsin7

Loading

mdmohsin7 commented Mar 7, 2026 •

edited

Loading