[Bug Report] Data Colletion Warehouse problem

[pietrosig]

We installed IsaacSim 4.2.0 and IsaacLab 1.2.0 as required. The demo works fine on Warehouse, but when running code for Data Collection we had

File "/media/ssd/usr/stefano/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/terrain_analysis.py", line 27, in <module>
    from skimage.draw import line
ModuleNotFoundError: No module named 'skimage'

so we tried installing scikit-image. After this step everything ran fine until a numpy vstack failed (further info below).

Steps to reproduce

After running

./isaaclab.sh -p <path-to-viplanner-repo>/omniverse/standalone/data_collect.py --scene warehouse --num_samples 100

we get

Traceback (most recent call last):
  File "/media/ssd/usr/stefano/IsaacLab/./../viplanner/omniverse/standalone/data_collect.py", line 141, in <module>
    main()
  File "/media/ssd/usr/stefano/IsaacLab/./../viplanner/omniverse/standalone/data_collect.py", line 123, in main
    samples = explorer.sample_viewpoints(args_cli.num_samples)
  File "/media/ssd/usr/stefano/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/viewpoint_sampling.py", line 66, in sample_viewpoints
    self.terrain_analyser.analyse()
  File "/media/ssd/usr/stefano/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/terrain_analysis.py", line 72, in analyse
    self._setup_raycaster()
  File "/media/ssd/usr/stefano/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/terrain_analysis.py", line 719, in _setup_raycaster
    self._mesh_dimensions = self._get_usd_stage_dimensions()
  File "/media/ssd/usr/stefano/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/terrain_analysis.py", line 306, in _get_usd_stage_dimensions
    prim_max = np.vstack([list(prim_range.GetMax()) for prim_range in bbox])
  File "/media/ssd/usr/stefano/isaacsim/extscache/omni.kit.pip_archive-0.0.0+10a4b5c0.lx64.cp310/pip_prebundle/numpy/core/shape_base.py", line 289, in vstack
    return _nx.concatenate(arrs, 0, dtype=dtype, casting=casting)
ValueError: need at least one array to concatenate

The problem is in terrain_analysis.py, because the variable bbox in the function _get_usd_stage_dimensions is empty. We even tried returning 0.0, 0.0, 0.0, 0.0 if bbox is empty, but we got more errors.

System Info

• Commit: 0bccd886c19a9891c0b6bdd37e13f338eacc0bba
• Isaac Sim Version: 4.2.0-rc.18+release.16044.3b2ed111.gl
• OS: Ubuntu 24.04.2
• GPU: NVIDIA GeForce RTX 3090 x2
• CUDA: 12.4
• GPU Driver: 550.144.03

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

• Being able to succesfully collect data

[pascal-roth]

@pietrosig I uploaded a new asset in PR #98 , could you please check if that fixes your issues.

[pietrosig]

@pascal-roth Thanks for the fix! I can confirm that the original issue has been resolved. However, when we ran the script again we got a new error:

Traceback (most recent call last):
  File "/media/ssd/usr/stefano/IsaacLab/../viplanner/omniverse/standalone/data_collect.py", line 141, in <module>
    main()
  File "/media/ssd/usr/stefano/IsaacLab/../viplanner/omniverse/standalone/data_collect.py", line 123, in main
    samples = explorer.sample_viewpoints(args_cli.num_samples)
  File "/media/ssd/usr/stefano/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/viewpoint_sampling.py", line 66, in sample_viewpoints
    self.terrain_analyser.analyse()
  File "/media/ssd/usr/stefano/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/terrain_analysis.py", line 76, in analyse
    self._sample_points()
  File "/media/ssd/usr/stefano/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/terrain_analysis.py", line 130, in _sample_points
    ray_origins, heights = self._point_filter_semantic_cost(ray_origins, heights)
  File "/media/ssd/usr/stefano/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/terrain_analysis.py", line 438, in _point_filter_semantic_cost
    ray_classes = self._raycast_usd_stage(
  File "/media/ssd/usr/stefano/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/terrain_analysis.py", line 930, in _raycast_usd_stage
    ray_class = [
  File "/media/ssd/usr/stefano/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/terrain_analysis.py", line 932, in <listcomp>
    get_semantics(prims_utils.get_prim_at_path(single_hit["collision"]))["Semantics"][1]
KeyError: 'Semantics'

[pascal-roth]

Can you double check that you are using the right versions of IsaacLab and IsaacSim. The semantic API was updated inbetween releases

[pietrosig]

Yes, when doing cat VERSION I get 1.2.0 for Isaac Lab and 4.2.0-rc.18+release.16044.3b2ed111.gl(isaaclab) for Isaac Sim

[pascal-roth]

could you check what is in the output of

get_semantics(prims_utils.get_prim_at_path(single_hit["collision"]))

then in your case?

[pietrosig]

Hi, this is the output of get_semantics(prims_utils.get_prim_at_path(single_hit["collision"])) for each single_hit in hits:

{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'furniture')}
{'Semantics': ('class', 'static')}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{}
{'Semantics': ('class', 'static')}
{}
{'Semantics': ('class', 'furniture')}
{}
{}
{}

[pascal-roth]

Ah it seems like the uploaded mesh does not have the classes assigned to most meshes. I currently don't have the time to address this issue, it will be caused somewhere in the TerrainImporter which does not succeed in the keyword assignment.

[Ruihyw]

Ah it seems like the uploaded mesh does not have the classes assigned to most meshes. I currently don't have the time to address this issue, it will be caused somewhere in the TerrainImporter which does not succeed in the keyword assignment.

Hi @pascal-roth
I made some changes in the new version of isaaclab and isaacsim, and now I can run the data_collect.py function in matterport.
But my problem is that I got the data set from the official matterport, which contains four files:
.dlb .house .navmesh semantic.ply

1. I used the isaaclab mesh_convert function to convert the .dlb file into a usd file, but when it was loaded in isaacsim, it was a pure white model without any texture.
2. I set

height_scanner = MatterportRayCasterCfg(
prim_path="{ENV_REGEX_NS}/Robot/base",
offset=MatterportRayCasterCfg.OffsetCfg(pos=(0.0, 0.0, 0.5)),
attach_yaw_only=True,
pattern_cfg=patterns.GridPatternCfg(resolution=0.1, size=[1.6, 1.0]),
debug_vis=True,
mesh_prim_paths=["/home/xxx/vlmaps/data/HM3D/5LpN3gDmAk7/5LpN3gDmAk7_semantic.ply"],
)

to the semantic.ply file in the dataset. I don't know if it is correct.
3. The final collected RGB image is only white and black, without any texture information
4. ` # filter points based on semantic cost
# if self.cfg.semantic_cost_mapping is not None:
# ray_origins, heights = self._point_filter_semantic_cost(ray_origins, heights)

        # set z height of samples to be at the robot's height above the terrain.

`
This part of the code performs a semantic filtering operation. After this step, only 0 points are retained, so I commented this part of the operation.
5、Image collection is correct in the warehouse scene，so I suspect that the error is caused by the update of the matterport dataset.

[Ruihyw]

[Ruihyw]

The strange thing is that the semantic graph actually exists, even though it doesn’t seem right...

[pascal-roth]

@Ruihyw this seems to be a different issue than discussed here, please move it to its own issue

Just as a quick answer, we only test it with the matterport dataset downloaded from https://github.com/niessner/Matterport. All other formats are not supported in the current code and we don't have experience with them.