🐛 Bug Description
The Gaussian Linear task in the mini sbibm seems to be bugged.
🔄 Steps to Reproduce
Python version: Python 3.11.2
SBI version: 0.25.0
To reproduce, run
pytest tests/bm_test.py --bm --bm-num-simulations XXX
at various simulations budgets.
E.g. below, I did it for 100, 2000 (default), and 10000 with NPE-PFN and NPE using NSF.
Both methods are very bad, and standard NPE even gets worse with more simulations.
NPE with NSF:
100 simulations: 0.787
2000 simulations: 0.889
10000 simulations: 0.927
NPE-PFN is usually super good at this task, but is at around 0.95 at all three budgets.
✅ Expected Behavior
With more simulations performance should improve. And NPE-PFN should be very good at this in terms of C2ST with even very few simulations, and not around 0.95.
🐛 Bug Description
The Gaussian Linear task in the mini sbibm seems to be bugged.
🔄 Steps to Reproduce
Python version: Python 3.11.2
SBI version: 0.25.0
To reproduce, run
pytest tests/bm_test.py --bm --bm-num-simulations XXXat various simulations budgets.
E.g. below, I did it for 100, 2000 (default), and 10000 with NPE-PFN and NPE using NSF.
Both methods are very bad, and standard NPE even gets worse with more simulations.
NPE with NSF:
100 simulations: 0.787
2000 simulations: 0.889
10000 simulations: 0.927
NPE-PFN is usually super good at this task, but is at around 0.95 at all three budgets.
✅ Expected Behavior
With more simulations performance should improve. And NPE-PFN should be very good at this in terms of C2ST with even very few simulations, and not around 0.95.