Incremental Smile Detection to improve hosted web detection pass rates

[Kwame Yeboah (Yeboahmedia)]

User description

• progressive smile implementation
• clean ups to bring the smile parity up to how the android sdk does capture

---

PR Type

Enhancement

---

Description

• Implement incremental smile detection with progressive thresholds

• Add neutral expression enforcement during non-smile capture zones

• Track smile baseline samples and consecutive frame confirmation

• Adjust smile zone boundaries to align with Android SDK behavior

---

Diagram Walkthrough

flowchart LR
  A["Face Detection"] -- "scores" --> B["Baseline Sampling"]
  B -- "neutral zone" --> C["Neutral Expression Check"]
  B -- "smile zone" --> D["Progressive Smile Tracking"]
  D -- "delta thresholds" --> E["Consecutive Frame Validation"]
  E -- "confirmed" --> F["Resume / Capture"]
  C -- "light neutral" --> F

Loading

File Walkthrough

Relevant files

Enhancement

SmartSelfieCapture.tsx Add incremental smile detection configuration parameters  packages/web-components/lib/components/selfie/src/smartselfie-capture/SmartSelfieCapture.tsx Added new configuration constants: neutralSmileMaxThreshold, smileProgressDelta, mouthDeformationDelta, smileProgressConsecutiveFrames, smileBaselineSampleSize Passed new parameters to the useFaceCapture hook +10/-0    useFaceCapture.ts Implement progressive smile tracking and neutral expression gating packages/web-components/lib/components/selfie/src/smartselfie-capture/hooks/useFaceCapture.ts Added trackSmileProgress function implementing incremental smile detection with baseline sampling, progressive delta thresholds, and consecutive frame confirmation Added hasLightNeutralExpression check to enforce neutral expression during non-smile capture zones Changed smile zone boundary from >= to > (capturesTaken > smileCheckpoint) and recalculated smileCheckpoint as half of liveness capture count Added refs for smile/mouth baseline samples, consecutive frames counter, and zone minimum scores Updated pauseCapture to accept a messageKey parameter, defaulting to 'smile-required' Added neutral expression gating in startCaptureInterval and resumeCapture to pause/block capture when expression is not neutral Reset smile progress tracking state on startCapture and resetCapture In the non-smile zone, resume capture is now triggered when a light neutral expression is detected instead of requiring a smile +152/-9

---

---

Need help?

Type /help how to ... in the comments thread for any questions about PR-Agent usage.

Check out the documentation for more information.

[prfectionist]

PR Reviewer Guide 🔍

(Review updated until commit ced4edd)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🏅 Score: 72

🧪 No relevant tests

🔒 No security concerns identified

🔀 No multiple PR themes

⚡ Recommended focus areas for review Baseline Tracking Bug The trackSmileProgress function tracks the minimum smile score and mouth open values ever seen in the smile zone. Because it uses Math.min to continuously lower the baseline, the baseline will monotonically decrease over time. This means the required delta thresholds (smileProgressionBaseline + smileProgressDelta) will also decrease, making it progressively easier to pass the smile check rather than harder. If the intent is to detect a smile progression (increase from a neutral baseline), the baseline should be captured once at the start of the smile zone or from a fixed window, not continuously updated to the global minimum. if (smileZoneMinScoreRef.current === null) { smileZoneMinScoreRef.current = smileScore; } else { smileZoneMinScoreRef.current = Math.min( smileZoneMinScoreRef.current, smileScore, ); } if (smileZoneMinMouthOpenRef.current === null) { smileZoneMinMouthOpenRef.current = mouthOpen; } else { smileZoneMinMouthOpenRef.current = Math.min( smileZoneMinMouthOpenRef.current, mouthOpen, ); } Missing Reset on Zone Transition smileZoneMinScoreRef and smileZoneMinMouthOpenRef are only reset when trackSmileProgress is called outside the smile zone or when no face is detected. However, if the user transitions from neutral zone to smile zone mid-capture, the baseline refs may already be null (correct) or may have stale values from a previous detection loop iteration. There is no explicit reset at the exact point of zone transition (capturesTaken crossing smileCheckpoint), which could lead to inconsistent baseline initialization depending on frame timing. if (!isCapturing.value || !isSmileZone) { smileConsecutiveFramesRef.current = 0; smileZoneMinScoreRef.current = null; smileZoneMinMouthOpenRef.current = null; return false; } Race Condition The resumeCapture logic in the neutral zone (lines 406-425) and the smile zone (lines 389-405) both use setTimeout(() => ..., 0) with stale closure checks on isPaused.value, isCapturing.value, and resumeCaptureRef.current. Between the time the setTimeout is scheduled and when it executes, multiple detection frames could fire, potentially scheduling multiple deferred resumeCaptureRef.current() calls. This could lead to double-resuming or capturing extra frames if the resume callback is invoked more than once before the next pause. setTimeout(() => { if ( isPaused.value && isCapturing.value && resumeCaptureRef.current ) { resumeCaptureRef.current(); } }, 0); Possible Issue hasLightNeutralExpression reads from currentSmileScore.value and currentMouthOpen.value signals, but in startCaptureInterval (line 566) and resumeCapture (line 591), these signals may not yet reflect the latest frame's values if the detection loop hasn't run since the last capture tick. This could cause the neutral expression check to use stale signal values, leading to incorrect pause/resume decisions. const hasLightNeutralExpression = () => currentSmileScore.value <= neutralSmileMaxThreshold && currentMouthOpen.value < mouthOpenThreshold; Edge Case When totalCaptures is 1, livenessCaptureCount is 0 and smileCheckpoint is 0. This means capturesTaken.value > smileCheckpoint.value is true as soon as capturesTaken reaches 1, but by then the capture may already be complete. Verify that the single-capture case is handled correctly and doesn't require a smile that can never be detected in time. const livenessCaptureCount = useComputed(() => Math.max(totalCaptures.value - 1, 0), ); const smileCheckpoint = useComputed(() => Math.floor(livenessCaptureCount.value / 2), );

[Copilot]

Pull request overview

This PR updates the hosted Smart Selfie capture flow to use incremental (progress-based) smile detection and tighter neutral-expression gating, aiming to improve detection pass rates and align behavior more closely with the Android SDK.

Changes:

• Introduces progressive smile detection logic (delta + consecutive-frame requirements) instead of a single-frame threshold check.
• Adds neutral-expression enforcement during the pre-smile phase (pauses capture until a light neutral expression is detected).
• Adjusts zone checkpoint logic to account for the reference photo and changes pause/resume alert handling.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
packages/web-components/lib/components/selfie/src/smartselfie-capture/hooks/useFaceCapture.ts	Implements progressive smile tracking, neutral-expression gating, and updated pause/resume + checkpoint logic.
packages/web-components/lib/components/selfie/src/smartselfie-capture/SmartSelfieCapture.tsx	Wires new smile/neutral tuning parameters into useFaceCapture.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Kwame Yeboah (Yeboahmedia)]

/review

[prfectionist]

Persistent review updated to latest commit ced4edd

[Chris DeLauder (beaglebets)]

Tell the machine to write some tests for you

[Kwame Yeboah (Yeboahmedia)]

Tell the machine to write some tests for you

I hate you too Chris DeLauder (@beaglebets) ..

[Chris DeLauder (beaglebets)]

Tell the machine to write some tests for you

I hate you too Chris DeLauder (Chris DeLauder (@beaglebets)) ..

Seriously! I tried it and that is the smoothest capture I've ever experienced on the web app. Feels like a huge improvement. I'd smash that approve button if we had some tests for it.

[Chris DeLauder (beaglebets)]

thank you

[github-actions]

This branch has been deployed to s3 / cloudfront.

✅ Preview URL for Smart Camera Web:

https://cdn.smileidentity.com/js/preview-feat/incremental-smile-detection/smart-camera-web.js

✅ Preview URL for Embed:

https://cdn.smileidentity.com/inline/preview-feat/incremental-smile-detection/js/script.min.js

✅ Preview URL for Web Client (Sandbox):

✅ Preview URL for Web Client (Production):

https://d2du6m53jhtq7s.cloudfront.net

[github-actions]

This branch has been deployed to s3 / cloudfront.

✅ Preview URL for Smart Camera Web:

https://cdn.smileidentity.com/js/preview-feat/incremental-smile-detection/smart-camera-web.js

✅ Preview URL for Embed:

https://cdn.smileidentity.com/inline/preview-feat/incremental-smile-detection/js/script.min.js

✅ Preview URL for Web Client (Sandbox):

https://d1uactou1n2j28.cloudfront.net

✅ Preview URL for Web Client (Production):

https://d2du6m53jhtq7s.cloudfront.net

[github-actions]

This PR is stale because it has been open 14 days with no activity. Remove stale label or comment or this will be closed in 7 days.

[github-actions]

This PR was closed because it has been stalled for 7 days with no activity.