RGB and depth data correspondance

[peppersaltyy]

Thanks for your great project. I have the following questions,

1. Do the processed depth maps come already registered (aligned) to the RGB camera frame, or are they raw depth sensor frames?

2. Are the output RGB images undistorted (pinhole model) or do they retain the original fisheye distortion?

3. Synchronization: Does the script handle timestamp interpolation to strictly align RGB, Depth, and Headset Pose?

4. Is the hmd_poses.csv in Stage Space (fixed relative to Guardian)?

[samuelm2]

Thanks for the kind words!

To answer your questions:

1. Do the processed depth maps come already registered or are they raw sensor frames? The depth maps exported by the app (.raw) are straight from the sensor and are not aligned to the RGB camera. The Python script convert_depth_to_linear.py creates "linear" maps, but this only normalizes values—it does not reproject them. However, the full reconstruction pipeline does generate a Color Aligned Depth folder (with .npy depth maps) as a final artifact. These maps are created by raycasting the into final reconstructed 3D mesh from the RGB camera's perspective. Currently, the pipeline doesn't use these files internally, but they good for experiments like training Gaussian Splats with depth supervision (e.g., Splatfacto-W) or other RGB-D tasks.

2. Are the output RGB images undistorted? No, they are not undistorted by this pipeline. The application dumps the raw YUV frames directly from the sensor. The Android API reports an empty list of distortion coefficients for the camera stream. I think it is highly likely that the "raw" camera stream exposed by the Quest's Android implementation has already been rectified (undistorted) by the underlying hardware ISP or driver before it even reaches the Camera2 API layer. Consequently, the pipeline defaults to a PINHOLE camera model in COLMAP.

3. Synchronization: Yes. The post-processing pipeline strictly handles synchronization. Specifically, see pose_interpolator.py. It performs interpolation (Linear for position, SLERP for rotation) to align the headset poses to the exact timestamps of the image frames.

4. Is the hmd_poses.csv in Stage Space? No, it is in Local Space. Because the app is configured as a Boundaryless App so it can be used anywhere, there is no fixed Guardian/Stage origin. I think this means the origin is set to the headset position when the app is started. This also means that if you reset the origin while recording, you'll get bad data.

[peppersaltyy]

Hi. Thank you for such a quick response!

My goal is to use meta quest 3 as a stereo camera and run the recorded rgbd video through foundationpose to get 6D pose of objects. Do you have any thoughts on this? I think the rgb should be fine. The depth is a problem to deal with.
‘the full reconstruction pipeline does generate a Color Aligned Depth folder (with .npy depth maps) as a final artifact’ Does this refer to 'OpenQuestCapture 3D Reconstruction' referred in this repo?

[samuelm2]

Sounds like a very cool use case! I hadn't heard of FoundationPose. I'll have to give that paper a read.

Re: Clarification on "OpenQuestCapture 3D Reconstruction": Yes, I'm referring to the Python processing pipeline quest-3d-reconstruction that is a submodule of this repository. Here is the exact place where the color-aligned depth maps are generated: https://github.com/samuelm2/quest-3d-reconstruction/blob/f5f3130eb67e9a8e36dc2c3f9e128b56b41e2419/scripts/processing/reconstruction/reconstruct_scene.py#L140

That code generates RGB camera-aligned depth maps by raycasting the reconstructed 3D mesh from the RGB camera's perspective. This is probably the easiest way to get exactly aligned depth maps, though it might not be as accurate since the reconstructed triangulated mesh doesn't have super fine details and may have some gaps. You could try to increase the density of the mesh (and therefore the quality of the aligned depth maps) by setting some parameters of pipeline_config.yml to be more detailed. Set voxel_size to 0.01 and color_optimization.interval to 1 to process every frame.

Alternative approaches:
- Use raw depth maps directly: The raw depth maps are typically only offset by about 50ms temporally from the RGB frames, which might be acceptable depending on how fast objects are moving in your scene.
- Reproject the raw depth maps (most accurate but more involved): You might be able to geometrically reproject the raw depth maps from the depth sensor's viewpoint to the RGB camera's viewpoint using the camera extrinsics and intrinsics from the depth_descriptors.csv files that OpenQuestCapture outputs. This would involve some non-trivial 3D geometry transformations (applying the relative pose between cameras and reprojecting depth values).