An advanced, full-screen 3D robotic kinematics engine featuring real-time analytical Inverse Kinematics (IK) driven by computer vision gesture control.
-
Computer Vision Controller: Integrates Google MediaPipe Hands to track index fingertip
$(X, Y, Z)$ coordinates via a standard webcam. - Holographic Gesture Logic: Implements specialized hand states—Open Hand for Robot IK manipulation, Pinched Fingers for 360° environment orbit, and variable finger distance for dynamic camera zooming.
- Analytical Inverse Kinematics: A high-speed FastAPI Python backend utilizing geometric laws of cosines to resolve 3-DOF joint configurations instantly.
- Interactive 3D Workspace: Rendered smoothly using React Three Fiber (Three.js) with real-time end-effector telemetry and visual safety boundaries.
- Frontend: React.js, Vite, Three.js,
@react-three/fiber,@react-three/drei, React-Webcam - Backend: Python, FastAPI, Uvicorn, NumPy, Pydantic
- AI Engine: Google MediaPipe (Hand Tracking Solution)
The system utilizes a dual-engine full-stack pipeline:
- The camera feed streams to the client-side MediaPipe neural network.
- 2D image coordinates are mapped into a localized 3D coordinate frame and filtered using a Low-Pass Lerp filter to remove high-frequency webcam noise.
- Target vectors are dispatched to the Python FastAPI server where a geometric IK model calculates precise matrix transformations.
- The resolved angular array overrides the 3D joint transformations in real-time.