Reproduce thunder main leaderboard results

[afilt]

Hi @PierreMarza !

I would like to test a custom model: I first pre-compute the embeddings then run the benchmark.
Is that correct ? How are computed the leaderboard's results between online_loading, image_pre_loading and embedding_pre_loading ? Also, I don't see reference to mixed precision, what's the policy to run this benchmark - full precision ?

I've tried online loading but it takes 1 hour for a ViT-B on BACH compared to 2 minutes feature extraction + 2 minutes linear probing.

Lastly, is there somewhere I can find the per-dataset results for models not benchmarked in the paper (for instance H0-mini) ?

Thank you.
Alexadre