Skip to content

todo.md

Latest commit

 

History

History
52 lines (40 loc) · 2.39 KB

todo.md

File metadata and controls

52 lines (40 loc) · 2.39 KB

TODO

Product: Python library Bensemble

Deadline: May 10

Resources: 4 developers

Task list

  1. Implement methods and algorithms
    • Implement base library functionality including base interfaces, calibration, uncertainty estimation and metrics
    • Create common interface for all ensembling methods
    • Implement MC Dropout algorithm in corresponding class
    • Implement Deep Neural Ensemble Search algorithm in corresponding class
    • Implement Neural Ensemble Search via Bayesian Sampling algorithm in corresponding class
  2. Library deployment
    • Write tests for all algorithms with >90% coverage
    • Debug methods after running tests until tests are all passed
    • Implement benchmark for methods to
    • Write documentation
    • Build library (write pyproject.toml)
    • Deploy library to pypi

Detailed task overview:

Build the library's core architecture and analytical tools (Dmitrii Vasilenko)

Create a unified interface for all methods and a naive wrapper for ensembles, implement algorithms to separate predictive uncertainty into aleatoric and epistemic implement ECE (Expected Calibration Error), Brier Score, and NLL to evaluate model trustworthiness and add post-hoc Temperature Scaling to fix overconfident predictions.

Implement MC Dropout (Fedor Sobolevsky)

Implement Monte-Carlo Dropout algorithm in its own class as described in Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning, using the common bayesian ensembling method interface. Immediately do a sanity check for the code by performing basic tests.

Implement Deep Neural Ensemble Search algorithm (Muhammadsharif Nabiev)

Write two pool-construction strategies: random sampling (NES-RS) and regularized evolution (NES-RE). Architecture search be connected to the rest of library functionality.

Task Durations

Task Min Time Median Time Max Time
Implement base library functionality 2 3 3
Create method interface 2 2 2
Implement MC Dropout 2 3 5
Implement Deep Neural Ensemble Search 3 4 8
Implement Neural Ensemble Search via Bayesian Sampling 3 4 5
Write tests 5 5 5
Debug 3 3 5
Implement benchmark 2 3 3
Write documentation 2 3 5
Build library 0.5 1 1
Deploy to pypi 0.5 0.5 0.5

Note: time is measured in students' evenings