invalid file pattern error when training the network

[noushinha]

Hi,

I would like to train the model. For the time being I have 3 tomograms in train folder and 1 in validation. Here is the command that I am running:

synapse_net.run_supervised_training -n vesicles_cryo --train_folder /scratch/htc/synapse-net/data/tem-arctiva/training_data/tomograms/Train --image_file_pattern "*.mrc" --label_folder /scratch/htc/synapse-net/data/tem-arctiva/training_data/labels/Train --label_file_pattern "*.mrc" --val_folder /scratch/htc/synapse-net/data/tem-arctiva/training_data/tomograms/Val --val_label_folder /scratch/htc/synapse-net/data/tem-arctiva/training_data/labels/Val --patch_shape 64 64 64 --batch_size 4 --n_samples_train 2000 --n_samples_val 500

I can see that datapath and labelpath variables include the following:

**data path:** ['/scratch/htc/synapse-net/data/tem-arctiva/training_data/tomograms/Train/Tomo_09_Pos_02.mrc', '/scratch/htc/synapse-net/data/tem-arctiva/training_data/tomograms/Train/Tomo_09_Pos_03.mrc', '/scratch/htc/synapse-net/data/tem-arctiva/training_data/tomograms/Train/Tomo_09_Pos_05.mrc']
**label path:** ['/scratch/htc/synapse-net/data/tem-arctiva/training_data/labels/Train/Tomo_09_Pos_02_SynapseNet.mrc', '/scratch/htc/synapse-net/data/tem-arctiva/training_data/labels/Train/Tomo_09_Pos_03_SynapseNet.mrc', '/scratch/htc/synapse-net/data/tem-arctiva/training_data/labels/Train/Tomo_09_Pos_05_SynapseNet.mrc']

However, the code throws the following error when it tries to get through torch_em.default_segmentation_loader

Traceback (most recent call last):
  File "/scratch/htc/miniconda3/envs/synapse-net/bin/synapse_net.run_supervised_training", line 7, in <module>
    sys.exit(main())
             ~~~~^^
  File "/scratch/htc/synapse-net/synapse_net/training/supervised_training.py", line 412, in main
    supervised_training(
    ~~~~~~~~~~~~~~~~~~~^
        name=args.name, train_paths=train_image_paths, val_paths=val_image_paths,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<3 lines>...
        n_iterations=args.n_iterations
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/scratch/htc/synapse-net/synapse_net/training/supervised_training.py", line 255, in supervised_training
    val_loader = get_supervised_loader(val_paths, raw_key, label_key, patch_shape, batch_size,
                                       n_samples=n_samples_val, rois=val_rois, sampler=sampler,
                                       ignore_label=ignore_label, label_transform=label_transform,
                                       label_paths=val_label_paths, **loader_kwargs)
  File "/scratch/htc/synapse-net/synapse_net/training/supervised_training.py", line 170, in get_supervised_loader
    loader = torch_em.default_segmentation_loader(
        raw_paths=data_paths, raw_key=raw_key,
    ...<4 lines>...
        rois=rois, **loader_kwargs,
    )
  File "/scratch/htc/miniconda3/envs/synapse-net/lib/python3.13/site-packages/torch_em/segmentation.py", line 268, in default_segmentation_loader
    ds = default_segmentation_dataset(
        raw_paths=raw_paths,
    ...<18 lines>...
        verify_paths=verify_paths,
    )
  File "/scratch/htc/miniconda3/envs/synapse-net/lib/python3.13/site-packages/torch_em/segmentation.py", line 371, in default_segmentation_dataset
    ds = _load_segmentation_dataset(
        raw_paths,
    ...<16 lines>...
        z_ext=z_ext,
    )
  File "/scratch/htc/miniconda3/envs/synapse-net/lib/python3.13/site-packages/torch_em/segmentation.py", line 123, in _load_segmentation_dataset
    dset = SegmentationDataset(
        raw_path, raw_key, label_path, label_key, roi=roi, n_samples=samples_per_ds[i], **kwargs
    )
  File "/scratch/htc/miniconda3/envs/synapse-net/lib/python3.13/site-packages/torch_em/data/segmentation_dataset.py", line 86, in __init__
    self.raw = load_data(raw_path, raw_key)
               ~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/scratch/htc/miniconda3/envs/synapse-net/lib/python3.13/site-packages/torch_em/util/image.py", line 101, in load_data
    return open_file(path, mode=mode)[key]
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/scratch/htc/miniconda3/envs/synapse-net/lib/python3.13/site-packages/elf/io/image_stack_wrapper.py", line 44, in __getitem__
    raise ValueError(f"Invalid file pattern {pattern}")
ValueError: Invalid file pattern data/data

Only when I change the raw_paths=data_paths to raw_paths=data_paths[0] and label_paths=label_paths to label_paths=label_paths[0], it can be trained. However, this trains the model only on the first training tomogram.
Can you let me know if I should use a specific folder structure for training? Also, it would be nice to add an example of a working training command. Although there is a -h flag, the explanations are not clear enough.

My current folder structure is as follows:

synapse-net
      ├── data
      │ ├── training_data
      │ │ ├── tomograms
      │ │ │ ├── train
      │ │ │ | ├── tomo1.mrc
      │ │ │ | ├── ...
      │ │ │ ├── val
      │ │ │ | ├── tomo4.mrc
      │ │ ├── labels
      │ │ │ ├── train
      │ │ │ | ├── tomo1.mrc
      │ │ │ | ├── ...
      │ │ │ ├── val
      │ │ │ | ├── tomo4.mrc
      ├── ...

I will appreciate the help. I really would like to use the synapse-net.

[constantinpape]

Hi @noushinha ,

Sorry for the late response, this issue fell under my radar after new year's. In general, please feel free to ping again here if you don't receive an answer within a week.

Regarding the specific issue: there was a bug in the code when providing an explicit val_folder. I have fixed it in #153. The command you provide on top should work once you update your synapse_net repo to the latest main branch. Please let me know if this fixes the issue.

A few more points:

• The patch shape of 64 x 64 x 64 seems rather small. You should choose a patch shape that contains several vesicles and some context around it.
• To check if the patch shape is large enough and if the data processing worked in general you can use the --check argument. It will show a couple of examples from the train and val data loader in napari.
• Regarding examples for running the training: that's a good point, I will add it later.

[constantinpape]

I have updated the training CLI a bit more. It's now also possible to initialize the model with pretrained weights, e.g. for the vesicles. See https://computational-cell-analytics.github.io/synapse-net/synapse_net.html#supervised-training for an example on how to call the CLI, including how to use the weight initiaization.