Command to generate RegionPLC captions

[junha-l]

Hello,

Thank you for your time and for sharing your work!

I'm interested in reproducing the generated caption dataset provided in here. However, I couldn't find specific instructions on how to generate this dataset.

Could you please provide the exact command(s) or script used to produce the generated captions?
Especially, the one used for scannet - spconv_clip_base15 experiment, i.e. caption_detic-template_and_kosmos_125k_iou0.2.json and scannet_caption_idx_detic-template_and_kosmos_125k_iou0.2.pkl?
This would be extremely helpful for reproducing your results and understanding the generation process.

[jihanyang]

Hi, the guideline for generating regional captions in RegionPLC.

• Step 1: Generate captions. You can just check

  PLA/tools/process_tools/generate_caption.py

  Line 932 in 648b7d7

  elif args.caption_mode == 'kosmos2_dense_caption':

  . You should generate kosmos2 and detic_crop_caption in this case.

• Step 2: Generate caption index. This is to build the mapping between captions and points in 3D. Please check this file:
  https://github.com/CVMI-Lab/PLA/blob/regionplc/tools/process_tools/generate_caption_idx.py.

• Step 3: SFusion. You can run this file to merge different captions with SFusion:

  PLA/tools/process_tools/concate_captions_with_iou.py

  Lines 106 to 123 in 648b7d7

  	if __name__ == '__main__':
  	parser = argparse.ArgumentParser('')
  	parser.add_argument('--caption_path_list',
  	default=['data/scannetv2/text_embed/caption_dense_scannet_kosmos2_25k.json',
  	'data/scannetv2/text_embed/caption_basic_crop_scannet_ofa_image-caption_coco_large_en_ofa_w400-500_over0.3.json'],
  	nargs='+', help='')
  	parser.add_argument('--caption_idx_path_list',
  	default=['data/scannetv2/scannetv2_caption_idx_kosmos2_densecap_25k.pkl',
  	'data/scannetv2/scannet_caption_idx_basic_crop.pkl'],
  	nargs='+', help='')
  	parser.add_argument('--caption_save_path', required=True, type=str, help='')
  	parser.add_argument('--caption_idx_save_path', required=True, type=str, help='')
  	parser.add_argument('--iou_high_thresh', default=[0.2, 0.2], nargs='+', type=float, help='iou below such threshold can be merge')
  	parser.add_argument('--iou_low_thresh', default=[0.0, 0.0], nargs='+', type=float, help='iou higher such threshold can be merge')
  	parser.add_argument('--no_cascade', action='store_true', help='if True, the caption idx will not be cascade merged')

[junha-l]

Thanks for the prompt and thorough instruction!

[junha-l]

Could you also share the arguments for generate_caption_idx.py for ScanNet dataset? I've checked the script but there are only examples for nuscenes dataset