python scripts/train.py +experiment=GaussianLSS_map data.version=v1.0-mini
3 | metrics.iou_walkway | IoUMetric | 0
8.9 M Trainable params
0 Non-trainable params
8.9 M Total params
35.645 Total estimated model params size (MB)
Initializing nuScenes map...
Initializing nuScenes map...
Epoch 0: 0%| | 0/81 [00:00<?, ?it/s]/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/functional.py:539: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3637.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/utilities/data.py:77: Trying to infer the batch_size from an ambiguous collection. The batch size we found is 4. To avoid any miscalculations, use self.log(..., batch_size=batch_size).
Epoch 0: 99%|█████████████████████████████████████████████████████████████▏| 80/81 [00:39<00:00, 2.03it/s, v_num=4706]/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/utilities/data.py:77: Trying to infer the batch_size from an ambiguous collection. The batch size we found is 3. To avoid any miscalculations, use self.log(..., batch_size=batch_size).
Epoch 0: 100%|██████████████████████████████████████████████████████████████| 81/81 [00:39<00:00, 2.03it/s, v_num=4706Error executing job with overrides: ['+experiment=GaussianLSS_map', 'data.version=v1.0-mini']/21 [00:07<00:01, 2.26it/s]
Traceback (most recent call last):
File "/root/GaussianLSS-main/scripts/train.py", line 81, in main
trainer.fit(model_module, datamodule=data_module, ckpt_path=ckpt_path)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 545, in fit
call._call_and_handle_interrupt(
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 102, in launch
return function(*args, **kwargs)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 581, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 990, in _run
results = self._run_stage()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1036, in _run_stage
self.fit_loop.run()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 202, in run
self.advance()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 359, in advance
self.epoch_loop.run(self._data_fetcher)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 137, in run
self.on_advance_end(data_fetcher)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 285, in on_advance_end
self.val_loop.run()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/utilities.py", line 181, in _decorator
return loop_run(self, *args, **kwargs)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 127, in run
batch, batch_idx, dataloader_idx = next(data_fetcher)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/fetchers.py", line 127, in next
batch = super().next()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/fetchers.py", line 56, in next
batch = next(self.iterator)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/utilities/combined_loader.py", line 326, in next
out = next(self._iterator)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/utilities/combined_loader.py", line 132, in next
out = next(self.iterators[0])
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 708, in next
data = self._next_data()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1455, in _next_data
return self._process_data(data)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1505, in _process_data
data.reraise()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/_utils.py", line 733, in reraise
raise exception
TypeError: Caught TypeError in DataLoader worker process 2.
Original Traceback (most recent call last):
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 349, in _worker_loop
data = fetcher.fetch(index) # type: ignore[possibly-undefined]
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 52, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/dataset.py", line 350, in getitem
return self.datasets[dataset_idx][sample_idx]
File "/root/GaussianLSS-main/GaussianLSS/data/nuscenes_dataset_generated.py", line 56, in getitem
data = self.transform(data)
File "/root/GaussianLSS-main/GaussianLSS/data/transforms.py", line 388, in call
gt_map = self.get_map(batch, bev_augm)
File "/root/GaussianLSS-main/GaussianLSS/data/transforms.py", line 350, in get_map
map_mask = self.nusc_map[sample['map_name']].get_map_mask((pose[0][-1], pose[1][-1], 100, 100), angle, self.map_layers, (h,w))
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 392, in get_map_mask
return self.explorer.get_map_mask(patch_box, patch_angle, layer_names=layer_names, canvas_size=canvas_size)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 870, in get_map_mask
map_mask = self.map_geom_to_mask(map_geom, local_box, canvas_size)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 819, in map_geom_to_mask
layer_mask = self._layer_geom_to_mask(layer_name, layer_geom, local_box, canvas_size)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 1805, in _layer_geom_to_mask
return self._line_geom_to_mask(layer_geom, local_box, layer_name, canvas_size)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 1935, in _line_geom_to_mask
map_mask = self.mask_for_lines(new_line, map_mask)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 1838, in mask_for_lines
for line in lines:
TypeError: 'MultiLineString' object is not iterable
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /root/GaussianLSS-main/logs/wandb/offline-run-20260415_164709-2026_0415_164706
wandb: Find logs at: logs/wandb/offline-run-20260415_164709-2026_0415_164706/logs
Epoch 0: 100%|██████████████████████████████████████████████████████████████| 81/81 [00:51<00:00, 1.58it/s, v_num=4706]
[rank0]:[W415 16:48:04.034806851 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(tdmpc2) root@hz50t59:~/GaussianLSS-main# z
python scripts/train.py +experiment=GaussianLSS_map data.version=v1.0-mini
3 | metrics.iou_walkway | IoUMetric | 0
8.9 M Trainable params
0 Non-trainable params
8.9 M Total params
35.645 Total estimated model params size (MB)
Initializing nuScenes map...
Initializing nuScenes map...
Epoch 0: 0%| | 0/81 [00:00<?, ?it/s]/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/functional.py:539: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at /pytorch/aten/src/ATen/native/TensorShape.cpp:3637.)
return _VF.meshgrid(tensors, **kwargs) # type: ignore[attr-defined]
/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/utilities/data.py:77: Trying to infer the
batch_sizefrom an ambiguous collection. The batch size we found is 4. To avoid any miscalculations, useself.log(..., batch_size=batch_size).Epoch 0: 99%|█████████████████████████████████████████████████████████████▏| 80/81 [00:39<00:00, 2.03it/s, v_num=4706]/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/utilities/data.py:77: Trying to infer the
batch_sizefrom an ambiguous collection. The batch size we found is 3. To avoid any miscalculations, useself.log(..., batch_size=batch_size).Epoch 0: 100%|██████████████████████████████████████████████████████████████| 81/81 [00:39<00:00, 2.03it/s, v_num=4706Error executing job with overrides: ['+experiment=GaussianLSS_map', 'data.version=v1.0-mini']/21 [00:07<00:01, 2.26it/s]
Traceback (most recent call last):
File "/root/GaussianLSS-main/scripts/train.py", line 81, in main
trainer.fit(model_module, datamodule=data_module, ckpt_path=ckpt_path)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 545, in fit
call._call_and_handle_interrupt(
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/trainer/call.py", line 43, in _call_and_handle_interrupt
return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 102, in launch
return function(*args, **kwargs)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 581, in _fit_impl
self._run(model, ckpt_path=ckpt_path)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 990, in _run
results = self._run_stage()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1036, in _run_stage
self.fit_loop.run()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 202, in run
self.advance()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/fit_loop.py", line 359, in advance
self.epoch_loop.run(self._data_fetcher)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 137, in run
self.on_advance_end(data_fetcher)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/training_epoch_loop.py", line 285, in on_advance_end
self.val_loop.run()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/utilities.py", line 181, in _decorator
return loop_run(self, *args, **kwargs)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/evaluation_loop.py", line 127, in run
batch, batch_idx, dataloader_idx = next(data_fetcher)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/fetchers.py", line 127, in next
batch = super().next()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/loops/fetchers.py", line 56, in next
batch = next(self.iterator)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/utilities/combined_loader.py", line 326, in next
out = next(self._iterator)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/pytorch_lightning/utilities/combined_loader.py", line 132, in next
out = next(self.iterators[0])
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 708, in next
data = self._next_data()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1455, in _next_data
return self._process_data(data)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1505, in _process_data
data.reraise()
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/_utils.py", line 733, in reraise
raise exception
TypeError: Caught TypeError in DataLoader worker process 2.
Original Traceback (most recent call last):
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 349, in _worker_loop
data = fetcher.fetch(index) # type: ignore[possibly-undefined]
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 52, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 52, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/torch/utils/data/dataset.py", line 350, in getitem
return self.datasets[dataset_idx][sample_idx]
File "/root/GaussianLSS-main/GaussianLSS/data/nuscenes_dataset_generated.py", line 56, in getitem
data = self.transform(data)
File "/root/GaussianLSS-main/GaussianLSS/data/transforms.py", line 388, in call
gt_map = self.get_map(batch, bev_augm)
File "/root/GaussianLSS-main/GaussianLSS/data/transforms.py", line 350, in get_map
map_mask = self.nusc_map[sample['map_name']].get_map_mask((pose[0][-1], pose[1][-1], 100, 100), angle, self.map_layers, (h,w))
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 392, in get_map_mask
return self.explorer.get_map_mask(patch_box, patch_angle, layer_names=layer_names, canvas_size=canvas_size)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 870, in get_map_mask
map_mask = self.map_geom_to_mask(map_geom, local_box, canvas_size)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 819, in map_geom_to_mask
layer_mask = self._layer_geom_to_mask(layer_name, layer_geom, local_box, canvas_size)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 1805, in _layer_geom_to_mask
return self._line_geom_to_mask(layer_geom, local_box, layer_name, canvas_size)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 1935, in _line_geom_to_mask
map_mask = self.mask_for_lines(new_line, map_mask)
File "/usr/local/anaconda3/envs/tdmpc2/lib/python3.9/site-packages/nuscenes/map_expansion/map_api.py", line 1838, in mask_for_lines
for line in lines:
TypeError: 'MultiLineString' object is not iterable
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
wandb:
wandb: You can sync this run to the cloud by running:
wandb: wandb sync /root/GaussianLSS-main/logs/wandb/offline-run-20260415_164709-2026_0415_164706
wandb: Find logs at: logs/wandb/offline-run-20260415_164709-2026_0415_164706/logs
Epoch 0: 100%|██████████████████████████████████████████████████████████████| 81/81 [00:51<00:00, 1.58it/s, v_num=4706]
[rank0]:[W415 16:48:04.034806851 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
(tdmpc2) root@hz50t59:~/GaussianLSS-main# z