feat(native): add useVideoStream hook — native camera streaming for Android & iOS#239
Open
mayankmahavar1mg wants to merge 13 commits into
Open
feat(native): add useVideoStream hook — native camera streaming for Android & iOS#239mayankmahavar1mg wants to merge 13 commits into
mayankmahavar1mg wants to merge 13 commits into
Conversation
…n (Android POC)
Implements a native video streaming hook for Catalyst-Core that renders a CameraX
preview behind a transparent WebView — no getUserMedia required.
Native (Android):
- NativeCameraManager: CameraX session lifecycle, ML Kit barcode scanning,
pinch-to-zoom via ScaleGestureDetector, FILL_CENTER coordinate mapping for
accurate QR-to-screen projection, viewfinder region filtering (full box containment)
- startVideoStream / stopVideoStream bridge commands wired into NativeBridge
- Camera permission flow handled inside start(), result delegated via onPermissionResult
- WebView zoom disabled to prevent conflict with native pinch gesture
- Debug overlays: red viewfinder border + green barcode box + inside/outside status label
- PreviewView added to activity_main.xml (INVISIBLE by default, shown on stream start)
JS Bridge:
- START_VIDEO_STREAM / STOP_VIDEO_STREAM commands and ON_VIDEO_STREAM_READY /
ON_VIDEO_STREAM_STOPPED / ON_QR_DETECTED callbacks added to NativeInterfaces.js
- nativeBridge.videoStream.start/stop added to NativeBridge.js util
- useVideoStream hook: accepts { onQRDetected } callback, exposes isStreaming,
viewfinderRef (auto-computes physical px rect with dpr scaling), start, stop
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…multiplier, camera/ package
- Refactored NativeCameraManager into camera/ package (8 files):
VideoStreamStateMachine, CameraSessionManager, BarcodeDetector,
HoldController, ZoomController, TorchController, ViewfinderMapper,
VideoStreamState
- Added 1080p ResolutionSelector (ImageAnalysis only), ML Kit auto-zoom
with zoom-only-up guard
- Fixed BridgeUtils transport layer (notifyWeb/notifyWebJson)
- Implemented QR hold state machine (suppress flag, 200ms, no flicker)
- Replaced setZoom/setTorch/onTorchChanged with streamState + sendCommand API
- Migrated zoom API from percentage (0-100) to multiplier (1.0x/2.0x);
ON_ZOOM_CHANGED payload: { zoomLevel, minZoom, maxZoom }
- Added setTorch/flip/notifyTorchChanged bridge commands
- Updated hooks.js useVideoStream and NativeInterfaces.js constants
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ay, flicker fix
- Navigation auto-stop: CustomWebview onPageStarted lambda → cameraManager.stop()
- Format filter: qr/barcode/all wired through start() options JS→NativeBridge→NativeCameraManager
- FPS via sendCommand('fps', {min, max}) — Camera2Interop on Preview.Builder, triggers rebind
- Flip/reveal animations removed — CameraSessionManager stripped to bare bind/unbind
- viewfinderRef disabled (useRef declared, logic commented out pending re-enable)
- showQrDetected: debugBarcodeOverlay repurposed — green=QR in frame, orange=ON_QR_DETECTED
200ms debounce hide, state machine resets on HOLD→STREAMING/STOPPING/IDLE
- Overlay flicker fix: overlayVisible flag skips redundant layout updates on normal frames,
overlayPaintedOrange tracks painted color for green→orange repaint, lastBarcodeScreenRect
cached for zoom-triggered repaints, WebView offset subtraction removed
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…view resolution ZoomController: ValueAnimator 300ms DecelerateInterpolator, 2-frame debounce (SUGGESTION_CONFIRM_FRAMES=2, SUGGESTION_TOLERANCE=0.1f) before committing to a zoom target. Cancel-and-restart on new mid-animation suggestion. cancelZoomAnimation() called on detachCamera(). CameraSessionManager: switched Preview from ResolutionStrategy to ResolutionFilter sorting by descending pixel count — HAL picks true maximum (1920x1080 on OnePlus 11 with dual-stream, vs 1640x1232 with ResolutionStrategy). Log.d reports negotiated resolution on every bind. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… JSON.parse
JSON.parse("GRANTED") fails with SyntaxError because the outer double-quotes
are JS string delimiters, leaving the parser with bare `GRANTED` (not valid JSON).
JSONObject.quote() already produces a valid JS string literal — pass it directly.
Fixes requestNotificationPermission, checkNotificationPermissionStatus, and any
other plain-string bridge callback (CAMERA_PERMISSION_STATUS, etc.).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add 9 Camera/ Swift files: VideoStreamState/StateMachine, HoldController, TorchController, ZoomController, BarcodeDetector, CameraSessionManager, NativeCameraManager, CameraPreviewView - Wire NativeBridge.swift with full command dispatch (start/stop/setZoom/ setTorch/flip), JSON param parsing via jsonStringToDict() - ZStack layout in ContentView: CameraPreviewView behind WebViewContainer - WebViewNavigationDelegate auto-stops camera on navigation - CatalystConstants: add all video stream event/command constants - Fix isActive data race: reads _state under NSLock in VideoStreamStateMachine - Fix WeakListenerBox: weak listener array + removeListener() + nil-compact - Fix EventBox/ErrorBox: NSLock + call()/replace() thread-safe API - Fix rewireEvents/rewireError: use .replace() instead of direct assignment - Fix current* data races in CameraSessionManager: paramsLock NSLock - Fix ZoomController: KVO on videoZoomFactor fires onZoomChanged during ramp - Fix pinch-to-zoom: user-scalable=no in viewport meta (NativeBridge.js) - Fix FPS change: wrap in beginConfiguration/commitConfiguration to avoid session graph rebuild - Fix bridge params: jsonStringToDict() parses JSON.stringify'd string from JS Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Thread-safety: CopyOnWriteArrayList + @synchronized transition() in VideoStreamStateMachine; @volatile on HoldController.lastDetectedValue, resumeRunnable, BarcodeDetector.suppressResults; NativeCameraManager overlay/viewfinder fields all @volatile; runOnUiThread for setZoom/setTorch/flip bridge calls; stateMachine.transition + onStopped moved into runOnUiThread in stop(). Resource/memory safety: BarcodeDetector closes existing scanner before rebuild + addOnFailureListener; TorchController fires onTorchChanged via ListenableFuture callback (not speculatively); NativeCameraManager cancels overlayHideHandler in cleanup(); CameraSessionManager guards start() against double-call while cameraProvider != null. Null/encoding safety: NativeBridge saves and restores original WebView background color instead of hardcoding Color.WHITE; BridgeUtils escapes U+2028/U+2029 in JSON before evaluateJavascript. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…atalyst-core into feature/video_hook_android_ios
- VideoStreamStateMachine: notify listeners outside lock (deadlock fix) + duplicate listener guard - BridgeUtils: fix double-JSON encoding — detect JSON strings, inject raw (with U+2028/U+2029 escaping) - ViewfinderMapper: guard imageWidth/imageHeight == 0 + fix KDoc - TorchController: use getMainExecutor via Context, fire onTorchChanged only on Future success - BarcodeDetector: fix stale activeScanner capture using local newScanner var - HoldController: fix TOCTOU race with AtomicReference on lastDetectedValue - CameraSessionManager: null bindToLifecycle guard + cleanupForRebind helper + fpsMin<=fpsMax validation - NativeCameraManager: safe-cast layoutParams - NativeBridge.kt: null nativeCameraManager guard in permission handler - NativeBridge.js: setTorch !!on coerce + setZoom input validation - hooks.js: fps min<=max validation in sendCommand Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- ZoomController: deinit KVO observer + duplicate attachDevice guard + #keyPath + super.observeValue + main-thread dispatch for onZoomChanged - CameraSessionManager: clarify FPS assignment comments + fix previewLayer race (capture to oldPreview local) + fix misleading nil-session log - HoldController: serial DispatchQueue for thread safety + rename HOLD_DURATION to holdDuration - ContentView: cameraManager as @StateObject (NativeCameraManager now ObservableObject) - CameraPreviewView: remove async dispatch from updateUIView - NativeBridge: nil cameraManager guard logs for stop/flip/handleVideoStreamCommand - BarcodeDetector: .qr fallback safety check against supported formats list - NativeCameraManager: fix optional-chaining side-effect in detectionHandler - TorchController: call onTorchChanged(false) on lockForConfiguration failure Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- mcp_v2/setup.js: remove dead syncCatalystDocs function and its unused helpers (fetchUrl, parseSitemapUrls, stripHtml, contentHash, SITEMAP_URL, https, crypto imports) - mcp_v2/mcp.js: convert inner function declaration to const arrow to fix no-inner-declarations - mcp_v2/tools/config.js: remove unused config param from _buildResult destructure - mcp_v2/tools/debug.js: fix _score destructure rename in fallback map - mcp_v2/tools/knowledge.js: suppress intentional github_files omit-destructure, fix empty catch blocks - mcp_v2/tools/tasks.js: remove unused fetcherFiles assignment, suppress projectRoot param on deriveFilesTouched - mcp_v2/tools/conversion.js: add /* ignore */ to intentional empty catch blocks - mcp_v2/lib/helpers.js: add /* ignore */ to intentional empty catch block - src/native/bridge/useBaseHook.js: add isNative to useCallback dependency array Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
DeputyDev will no longer review pull requests automatically.To request a review, simply comment #review on your pull request—this will trigger an on-demand review whenever you need it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
useVideoStream, a native camera streaming hook for Catalyst-Core. Native CameraX (Android) and AVCaptureSession (iOS) render behind a transparent WebView — native owns all hardware and detection, web owns all UI and business logic. The bridge is a minimal event/command contract.What's included
Android
NativeCameraManager— orchestrates CameraX session lifecycle, overlay, and viewfinderCameraSessionManager— binds use cases (preview + image analysis), handles FPS, flip, format changes via cleanupForRebind patternBarcodeDetector— ML Kit QR/barcode scanning with auto-zoom, format filter (qr/barcode/all), and close-before-rebuild resource safetyZoomController— smooth ratio-based zoom (multiplier API: 1x/2x), emitsON_ZOOM_CHANGEDwith{zoomLevel, minZoom, maxZoom}TorchController—ListenableFuture-based torch toggle, firesonTorchChangedonly on hardware confirmationHoldController— QR hold state machine withAtomicReferenceTOCTOU fixVideoStreamStateMachine—@Synchronizedstate transitions with out-of-lock listener notification (deadlock safe)BridgeUtils— JSON detection heuristic to prevent double-encoding; U+2028/U+2029 escaping for V8 safetyNativeBridge.kt/.js—startVideoStream/stopVideoStream,setZoom,setTorch,flip,sendCommandResolutionSelector, debug viewfinder drawable,activity_main.xmloverlay layoutiOS
CameraSessionManager— AVCaptureSession management withparamsLockfor cross-thread property safety,beginConfiguration/commitConfigurationFPS batching, previewLayer race fixBarcodeDetector— AVFoundation barcode scanning with.qrfallback safetyZoomController— KVO-based zoom with#keyPath, deinit observer cleanup, duplicate-attach guard, main-thread dispatchTorchController—lockForConfigurationtorch withonTorchChanged(false)on failureHoldController— serialDispatchQueuefor thread-safe hold stateNativeCameraManager—ObservableObjectconformance,@StateObjectownership inContentViewNativeBridge.swift— full command bridge withjsonStringToDict()for Android-compatible param encodingCameraPreviewView— synchronousupdateUIView(no async dispatch)WebView/WebViewNavigationDelegate— navigation auto-stop,user-scalable=noviewport (blocks WebKit pinch-zoom)VideoStreamState+VideoStreamStateMachine— shared state enum with SwiftUI-safe transitionsJS Bridge
hooks.js—useVideoStreamhook:start()/stop()/sendCommand(),streamState, event subscriptions (onQrDetected,onZoomChanged,onTorchChanged)NativeBridge.js—setZoominput validation,setTorchboolean coerceNativeInterfaces.js—VIDEO_STREAMinterface constantsOther
mcp_v2: removed deadsyncCatalystDocs+ 5 helper functions; fixed 20 pre-existing ESLint errors across 9 filesKey design decisions
JSON.stringifyparams; Swift parses string; Android reads stringcameraManageras@StateObject, notweak varonTorchChangedinsideListenableFuturecallback, not speculativelyVideoStreamStateMachinenotifies listeners outside@SynchronizedlockBridgeUtils.notifyWebdetects JSON strings, injects rawTest coverage
Commits
db0d12cb3b7d56f4879b63960084b3ae81d41b5cbd587ccf11bc37b47dd49d5ba73bc6