Meeting notes

[JLrumberger]

Agenda:

• What do we want to build?

• student teacher augmentation consistency for domain adaptation

• Which datasets and models will be used for domain adaptation?

• 1 or 2 channel datasets (i.e. tissuenet v1, tissuenet v2, cellpose, live cell)

• start with cellpose

• MVP: cellpose on one dataset, better after self-supervised adaptation

How to organize?

• weekly check-ins at tuesday 12.00 - 13.00, book the room.
• 4 week sprint meeting will be the first tuesday of the months

[JLrumberger]

Actually ship once the tool works, then work on novel science.
Make it work on categorical predictions
Bioarxiv as soon as it exists, together with usable tool.

[JLrumberger]

Jannik:

• Set up interfaces for all classes and functions that we want to have for MVP.

[fabianreith]

Meeting, 21dec23:

TODO MVP:

• Cellpose weights to Unet form
  • should be quite easy (https://cellpose.readthedocs.io/en/latest/api.html#cellpose.models.CellposeModel)
• Trainer class
  • do pull request, get feedback
• Evaluation function
  • maybe valid loss is good enough?
• Loss function
  • use uncertainty
• Barebones wandb
• Create Train.py -> also have example notebook for detailed documentation

[JLrumberger]

Meeting 11jan24

1. Fix loss function to make it work for teacher outputs as targets <- Lorenz
2. Jannik works on logging
3. Dinesh works on livecell
4. Fabian works on trainer

[JLrumberger]

Mini Hackathon Agenda:

• What needs to be done?
  • write loss function that consists of CE and MSE loss for the two outputs (lorenz)
  • write train.py for semi-supervised adaptation (part of it is already implement here) (fabian)
    • dataloader for cellpose
  • write train.py for un-supervised adaptation
    • elastic regularization
    • elastic regularization on the predictions not on the weights
  • trainer validation method (jannik)

If stuff doesn't work in the framework, just adapt it.

Experiments

• Semi-supervised learning (tissuenet wenige labeled und viele unlabeled samples)
• Domain adaptation with source domain (cellpose labeled und tissuenet unlabeled)
• Domain adaptation without source domain (tissuenet unlabeled und elastic regularization)
• Domain adaptation without source domain (predictions are not allowed to deviate too much from initial predictions)
• SOUPS: assemble model from multiple checkpoints, how to select best checkpoints?
• Bin-bucketing: put the flow maps of cellpose into buckets and individually optimize the buckets
• Find best augmentations, checkout if masking helps.
• Make everything efficient and add a UI?
• Mixture of Expert models for cell segmentation
• Semi-supervised learning on tissuenet and Lorenz' 800GB multiplexed imaging data.
• Try domain adaptation loss function from TVT in this setting.
• try child-tuning to retain source domain performance (https://github.com/pkunlp-icler/ChildTuning)

[fabianreith]

Meeting - 25jan24

• Fabian übernimmt train.py
• • Bringt es zum laufen, aber ignoriert wandb
• Jannik bringt wandb logger zum Laufen
• • logger in einer Klasse, welche dann kann:
• • • logger_obj.log_metrics()
• • • logger_obj.log_image()

[JLrumberger]

• Jannik adds validation function to trainer class, somewhere here: https://github.com/Kainmueller-Lab/self_adapt/blob/main/src/trainer.py#L170 with by calling self.student_teacher.predict_teacher(instance_mask=True) or self.student_teacher.predict_student(instance_mask=True).
• Fabian merges train.py
• let it run, and see if validation loss decreases, scratch head if it doesn't
  • shouldn't work because test-time augmentations on teacher mix up flow maps, Fabian looks into this
• Fabian: see if flow map generation with augmentations afterwards produces correct results, implement after augmentations if necessary
• Jannik: Wandb PR
• Jannik: bin-bucketing of flow maps and turn it into a classification problem
• Multi-gpu training pipeline

[JLrumberger]

• Everyone: Look at Cellpose original paper, 2.0 and 3.0, write down results for comparison
• Everyone: write down experiment ideas
• Fabian: Look into multiple augmentation paper (https://arxiv.org/pdf/2203.03304.pdf), communicate with Dinesh who wants to implement this
• Fabian: save model weights and prepare eval_notebook or eval_script. Put notebooks into top level folder "notebooks".
• Future: once we have nice results, write carsen stringer and ask if she wants to be on the paper

[fabianreith]

• Looking at runs, validation loss goes down, other metrics (e.g. AP50) get better in the beginning, but get worse later on.

• • Loss might need to be improved
• • • Try label smoothing?

• Jannik: Visualize the uncertainty map and full prediction (not only flows & fgbg)

• Fabian: do more runs, look into loss ideas - FUTURE also maybe uncertainty metric

• Also log training set accuracy

[JLrumberger]

It's weird that the validation loss and validation ap50 score are correlated instead of anti-correlated

Suspicious stuff:

• merging of the test-time-augmented teacher predictions could be somehow faulty, especially for the flow maps or uncertainties of the flow maps.

Ideas:

• visualize true/false positives and negatives
• logging the filter maps from the uncertainty-based filtering
• exclude uncertainty from loss calculation i.e. no loss masking
• weight pixel-wise ce loss by confidence prediction as in EDAPS
• add elastic regularization to the student to limit weight divergence from initialization or regularize distance of predictions from completely frozen model as in EDAPS

Experiments to run:

1. Optimize hyperparameters of the post-processing (diameter, cellprob_threshold, flow_threshold) of cellpose on the tissuenet validation dataset and see if no test-time-augmentations vs. flips only vs. 90° rotations vs. full test-time-augmentations yields best results. Check if flows and uncertainties of the flow maps look better with/out test-time-augmentations. Should one do mean/max/median to combine flows? <- fabian (keep us updated)
2. Visualize how individual teacher tta predictions look like pre-transformation, post-transformation and visualize differences and means.

Code to implement:

1. elastic regularization
2. uncertainty-based loss weighting
3. visualize true/false positives and negatives and log the filter maps from the uncertainty-based filtering
4. split validation loss into mse and ce loss for wandb logging

[fabianreith]

Meeting - 28mar24

Goal: Structure things to do

Fabian & Jannik

Goal

Optimize hyperparameters of the post-processing (diameter, cellprob_threshold, flow_threshold) of cellpose on the tissuenet validation dataset and see if no test-time-augmentations vs. flips only vs. 90° rotations vs. full test-time-augmentations yields best results.
Check if flows and uncertainties of the flow maps look better with/out test-time-augmentations. Should one do mean/max/median to combine flows? <- fabian (keep us updated)
RESTRUCTURE ABOVE GOAL

• Create a analyze_zero_shot_cellpose.py script which:

  • Creates a trainer object but just runs validation once without any training
  • Allows for a config file to set
    • TTA augmentations
    • Cellpose Instance Segmentation postprocessing parameters

• Cerate a script which trys a lot of parameters (optimally on the MDC cluster)

  • For FUTURE TRAINING RUNS - we might want to look into W&B functionality, but overkill for our validation analysis for now

• Adjust StudentTeacher & Trainer Classs (maybe use the Cellpose Wrapper Classes?) to allow for these parameters to be set (if necessary)

  • Cellpose Instance Segmentation postprocessing parameters
    • For this, one needs to adapt cellpose_model_wrapper.py
      • in def get_instance_mask(self, x): the line - mask, _, _ = self.cellpose_model.eval(x_numpy_i)
        • the .evel should allow to set diameter=None, flow_threshold=0.4, cellprob_threshold=0.0

• During validation, predict Pseudo label on Valid Set

  • Adjust in student_teacher.py the predict_teacher method to allow generate_pseudo_mask to be run instead of the current prediction = self.teacher_model(input)
  • Adjust generate_pseudo_mask to allow for different teacher TTA pred merging options (currently, it takes the mean: output = torch.mean(output_stack, dim=0))

Minium viable goal - Fabian does this until Tuesday (hopefully) - then meets with Jannik to assign the subsequent tasks
Creates a trainer object but just runs validation once without any training
Allows for a config file to set
TTA augmentations

Dinesh

Goal
elastic regularization implemented, PR created

MVG

• Create a analyze_zero_shot_cellpose.py script which:
  • Creates a trainer object but just runs validation once without any training
  • Allows for a config file to set
    • TTA augmentations

Jannik

• Find the elastic regularization paper, send to Dinesh

[JLrumberger]

• Problem of domain adaptation is under-detection not overdetection
• Try multi-scale inputs (0.5x, 1x, 2x) and resize outputs, merge via mean/max or FG/BG as confidence estimate.
• Use FG/BG prediction as confidence estimate and merge flows based on that.

[fabianreith]

Meeting 5apr24

PR Fabian

Need to still add docstrings in some cases (have a look). Otherwise fine
We might need to optimize validation at some point (e.g. predit_teacher also returns [besides mask] flow prediction and uncertainty metric)

W&B - CP eval is best (even better than TTA)

Check cellpose.eval and compare to our predict routine, looks like the only difference is padding and tile&stitch but needs confirmation

Experiments

Most experiments should be run on another Project name or locally or deleted, to not crowd W&B unncecessarily.

Lorenz

• Try multi-scale inputs (0.5x, 1x, 2x) and resize outputs, merge via mean/max or FG/BG as confidence estimate.
  • modify adjust_flow_vectors & get_flow_vector_adjustments_list in cellpose_student_teacher.py
• Add all around 8px mirror padding and then crop again

Jannik

• How to merge TTA Flows
  • Merge via mean/max or FG/BG as confidence estimate.
  • Use FG/BG prediction as confidence estimate and merge flows based on that.

Fabian

• Run CP eval on various parameters:
  • diameter
  • compute masks params
    • niter=200?
    • cellprob_threshold=0.0,
    • flow_threshold=0.4,
    • min_size=15

Future

• Think about how to merge FG/BG confidence estimate

MDC Cluster Training

• Folder at /fast/AG_Kainmueller/self_adapt/cluster_training with script run_self_adapt_train_cellpose.sh as potential example on how to do it
  • the bash script now has -l h_rt time of 136 hours, maybe, just using 5 hours would help with scheduling priority
• git folder is at /fast/AG_Kainmueller/self_adapt/self_adapt

[fabianreith]

11apr24

We decided to do the meeting on Friday 14 o'clock.

Results we talked about

• Diameter of 20 works significantly better. Also, lower Cellprob threshold.
  • Other parameters didn't matter that much
• Multiscale resizing TTA works better; maybe it's just due to modified diameter.
  • Mirror padding didn't have any effect (weird, as we expected that this should help detect boundary cells better)

TODO until our meeting tomorrow

• Fabian and Lorenz create a PR based on their code. Merge it in the evening.
• Fabian modifies Tissuenet to increase image size by 50%
  • reruns all parameters with the resized images

[JLrumberger]

23par24

Recap

• Fabian did hparam tuning on the validation dataset of flow threshold, diameter, niter, min_size
  • some things only changed results in a miniscule way
  • validation loss generally didn't change much, while AP50 & F1 scores varied quite a lot
• Lorenz implemented center padding and the resize_tta, padding didn't change anything, resize_tta improved the scores slightly
  • resizing might only work due to non-optimized diameter size (30px, vs the better-working 20px)

Points raised during the meeting

• Core research question: How do we make TTA work best? -> By getting the best possible pseudo label (right?)
  • Step 1: Optimize Cellpose hyperparameters (in progress)
  • Step 2: Optimize the TTA hyperparameters (we start this now)
    • This boils down to three questions:
      • Question 1: Which augmentations do we want to use/combine?
      • Question 2: How do we want to combine multiple teacher predictions into one pseudo label?
      • Question 3: How do we want to assess the teacher model's uncertainty based on the multiple teacher predictions?
• Trade-off between implementation of analysis in Jupyter notebook (not as replicable) versus code base implementation (can make our code too chaotic)
• Scoring well on this leaderboard (https://neurips22-cellseg.grand-challenge.org/evaluation/testing/leaderboard/) would significantly improve our paper's impact.
  • Maybe try semi-supervised training with the multi-modality dataset

TODO until next Thursday

To everyone who makes plots, make them pretty, accessible and reusable; we might want to use them in the paper

Fabian (Question 1)

• Run training with the rescaled dataset
  • run the resize_tta experiment again with the resized dataset
• Evaluate TTA predictions for various augmentations on the validation set ground truth
• write a function that pickles all predictions, inputs and GT data to make it accessible for the others to run their experiments (write design doc to get a consens on where to implement this)**

Jannik (Question 2)

• Try different ways to merge the TTA predictions (i.e. mean, median, fg/bg confidence) <- maybe Jannik
• Experiment: teacher predicted fg/bg & masks -> instance masks -> zurück zu flows und fg/bg for pseudo labels -> either plot difference to gt fg/bg and flows for both configurations or just run a training experiment. <- maybe Jannik

Lorenz (Question 2 & 3)

• look into the correlation of fg/bg confidence score and MSE(gt_flows, pred_flows)
• Plot uncertainty metrics (diffusion coefficient, variance, stddev, entropy) x cell-wise loss or 'accuracy'

Dinesh?

• Make the datasets from the multi-modality cell segmentation challenge available (https://neurips22-cellseg.grand-challenge.org/dataset/)

Everyone

• Find out how the flows are represented (are they unit-vectors?)

Feedback from Dagmar, 17apr24:

Question 1

• TTA augmentations - why use different augmentations from Cellpose?
  • Which augmentations does Cellpose use during training?
  • What are the standard Tissuenet augmentations? (see Cellpose2)

Question 2

• Create instance masks for each prediction -> average -> re-generate flow/cellprob masks
  • Is it possible to average the instance mask predictions?

Generally

• Try to mostly use Cellpose defaults to further reduce the questions we need to tackle