Some problems in model reproduction

[Skullface9512]

Recently I tried to reproduce the training process according to code，but after i run the script adapt_detnet_dual.py, I came across some problems:

/home/goertek/anaconda3/envs/py38/bin/python3.8 /media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/adapt_detnet_dual.py -trs ah -tes ah --root_idx 0 --pic 1024 --resume -eid 37 --epochs 10 --start_epoch 1 --gpus 0 --checkpoint in_dataset_adapt --setup 0 --pair 1,2 
               ===============  Options  ===============
                         alpha  :  0.7
                           avg  :  max
                    checkpoint  :  in_dataset_adapt
                         clean  :  False
                     data_root  :  data
                 datasets_test  :  ['ah']
                datasets_train  :  ['ah']
                      det_loss  :  True
                        epochs  :  10
                      evaluate  :  False
                   evaluate_id  :  37
                         gamma  :  0.1
                          gpus  :  0
                         initR  :  pred
                 learning_rate  :  0.0005
                 lr_decay_step  :  250
                       outpath  :  out_loss_auc
                          pair  :  1,2
                           pic  :  1024
                        resume  :  True
                      root_idx  :  0
                  saved_prefix  :  ckp_detnet
                         setup  :  0
                      snapshot  :  1
                   start_epoch  :  1
                    test_batch  :  512
                   train_batch  :  16
                       workers  :  8
               ===============  Options  ===============

CREATE NETWORK
Load annotation from  data/assemblyhands/annotations/, mode: val
loading annotations into memory...
Done (t=2.45s)
creating index...
index created!
Get bbox and root depth from groundtruth annotation
100%|██████████| 32179/32179 [00:02<00:00, 15357.11it/s] 
Number of annotations in single hand sequences: 0
Number of annotations in interacting hand sequences: 0
Number of skipped annotations: 1
<datasets.assembly_hands.AssemblyHandsDataset object at 0x7fed8079d4c0>
Total test set size: 384

LOAD CHECKPOINT in_dataset_adapt/ckp_detnet_37.pth
CHECKPOINT NOT EXIST, TRY pretrain/ckp_detnet_37.pth
Load annotation from  data/assemblyhands/annotations/, mode: val
loading annotations into memory...
Done (t=2.40s)
creating index...
index created!
  0%|          | 0/32179 [00:00<?, ?it/s]Get bbox and root depth from groundtruth annotation
100%|██████████| 32179/32179 [00:02<00:00, 12901.14it/s] 
Number of annotations in single hand sequences: 0
Number of annotations in interacting hand sequences: 0
Number of skipped annotations: 1
<datasets.assembly_hands.AssemblyHandsDataset object at 0x7fed751e7e20>
Total train dataset size: 384

USING 1 GPUs

Epoch: 1/10
group 0 lr: 0.0005
Epoch 1 [0/24] lH:0.00000 lD:0.47252 lL:0.41813 11.94s/batch
Traceback (most recent call last):
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/adapt_detnet_dual.py", line 653, in <module>
    main(parse_args)
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/adapt_detnet_dual.py", line 264, in main
    train(
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/adapt_detnet_dual.py", line 544, in train
    for i, (metas1, metas2) in enumerate(train_loader):
  File "/home/goertek/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 681, in __next__
    data = self._next_data()
  File "/home/goertek/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1356, in _next_data
    return self._process_data(data)
  File "/home/goertek/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1402, in _process_data
    data.reraise()
  File "/home/goertek/anaconda3/envs/py38/lib/python3.8/site-packages/torch/_utils.py", line 461, in reraise
    raise exception
AssertionError: Caught AssertionError in DataLoader worker process 4.
Original Traceback (most recent call last):
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/datasets/handataset.py", line 413, in __getitem__
    (sample1, sample2), ds = self._get_sample(index)
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/datasets/handataset.py", line 428, in _get_sample
    sample1, sample2 = ds.get_sample(index - base)
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/datasets/assembly_hands.py", line 510, in get_sample
    (inputs1, targets1, meta_info1), (inputs2, targets2, meta_info2) = self.__getitem__(index)
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/datasets/assembly_hands.py", line 411, in __getitem__
    inputs1, targets1, meta_info1 = self._data_from_datalist(data1, idx)
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/datasets/assembly_hands.py", line 324, in _data_from_datalist
    img, bbox, joint_img, joint_cam, joint_valid, retval_camera = load_crop_img(
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/datasets/assemblyhands/utils/preprocessing.py", line 78, in load_crop_img
    assert sum(joint_valid) >= 10, f"sum(joint_valid): {sum(joint_valid)}, path: {path}, joint_valid: {joint_valid}"
AssertionError: sum(joint_valid): 9, path: data/assemblyhands/images/ego_images_rectified/val/nusar-2021_action_both_9081-c11b_9081_user_id_2021-02-12_161433/HMC_21176623_mono10bit/003645.jpg, joint_valid: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 1 1 1 0 1 1 1 0 1 1 0 0
 0 0 0 0 0]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/goertek/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 302, in _worker_loop
    data = fetcher.fetch(index)
  File "/home/goertek/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/home/goertek/anaconda3/envs/py38/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 49, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/datasets/handataset.py", line 416, in __getitem__
    (sample1, sample2), ds = self._get_sample(index)
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/datasets/handataset.py", line 428, in _get_sample
    sample1, sample2 = ds.get_sample(index - base)
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/datasets/assembly_hands.py", line 510, in get_sample
    (inputs1, targets1, meta_info1), (inputs2, targets2, meta_info2) = self.__getitem__(index)
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/datasets/assembly_hands.py", line 411, in __getitem__
    inputs1, targets1, meta_info1 = self._data_from_datalist(data1, idx)
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/datasets/assembly_hands.py", line 324, in _data_from_datalist
    img, bbox, joint_img, joint_cam, joint_valid, retval_camera = load_crop_img(
  File "/media/goertek/5f79ed5e-a5cd-4fdc-8223-2f15a0baa8c5/goertek/workspace/S2DHand-main/datasets/assemblyhands/utils/preprocessing.py", line 78, in load_crop_img
    assert sum(joint_valid) >= 10, f"sum(joint_valid): {sum(joint_valid)}, path: {path}, joint_valid: {joint_valid}"
AssertionError: sum(joint_valid): 6, path: data/assemblyhands/images/ego_images_rectified/val/nusar-2021_action_both_9081-c11b_9081_user_id_2021-02-12_161433/HMC_21176623_mono10bit/004787.jpg, joint_valid: [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 1 0 0 0 1 0 0 0 0
 0 0 0 0 1]
Process finished with exit code 1

Is there labels have problems or ignore relative settings, thank you

[MickeyLLG]

Hi. thanks for being interested in our work.
It seems that your "Number of skipped annotations" is different from mine, where yours is 1 while mine is 120. This may include some invalid samples into training and thus cause exceptions.
Have you downloaded the pre-generated invalid.txt here and move it to data/assemblyhands/annotations/? In your case, please copy invalid_val_ego.txt to data/assemblyhands/annotations/val.
Kindly reply once having any further concerns.

[Skullface9512]

Yes, I have put this file in data/assemblyhands/annotations/val and if i don't do this, the program will hint this file doesn't exist

[MickeyLLG]

I think there should be something wrong with the val annotations or invalid.txt, following is my val directory, please have a check if there is something different.
https://drive.google.com/file/d/1zjrnJ6_f1uEwOz8eRgLZVf-u0o4VtTgy/view?usp=sharing

[Skullface9512]

Wow it worked, thank you very much.

[MickeyLLG]

Great! Have you found where the issue is, for example, which file is different from my version? I would like to know any potential problem, thank you!

[Skullface9512]

I tried previous file to run again, it also works, and found no difference from two files, maybe i ignore some setting values before.

[Skullface9512]

Hi Mickey, recently i want to test model on other multi-view datasets, such as FreiHand and DexYCB, i tried to replace the images in assemblyhands_crop folder, but the detected 2D joints is incorrect, maybe the rotation matrix between two cameras has some problems, what should i notice or confirm before this test.

[1a2a3x]

Hi Mickey, recently i want to test model on other multi-view datasets, such as FreiHand and DexYCB, i tried to replace the images in assemblyhands_crop folder, but the detected 2D joints is incorrect, maybe the rotation matrix between two cameras has some problems, what should i notice or confirm before this test.

Have you resolved the rotation matrix problem? If so, could you share how you obtained or calibrated the matrices between cameras?