LLC (Local leaning coefficient, Real Log Canoical Threshold) estimation

from RLCT repository

code

• estimate_softmaxDNN.py

compare exact RCLT(Learning Coeciffient) of linear NN and nonlinear NN (softmax/Relu)

• llt_rlct_estimator_for_softmax_DNN.py

  RLCT of small nonlinear coefficient to compare linear matrix decomposition network(exact solutoin).

• llc_rlct_estimator_for_softmax_networks_via_power_posteriors_sgld.py

  outor interpolation of SGLD LLC extimation with NUTS/HMC SGLD extimaiton with pyto.

• llc_training_trajectory.py

  LLT selection obserbation during SGD update, small LLC saddlepoint is selected or not.

how to run

python estimate_softmaxDNN.py

Reference

• The Local Learning Coefficient: A Singularity-Aware Complexity Measure
• Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient
• Quantifying degeneracy in singular models via the learning coefficient
• Estimating the Local Learning Coefficient
• Singular Learning Theory with Daniel Murfet
• ReLUネットワークにおける局所学習係数推定手法のモデル選択への応用
• Stochastic natural gradient descent draws posterior samples in function space
• Stochastic Gradient Descent as Approximate Bayesian Inference
• Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architectures
• https://github.com/suswei/RLCT
• https://github.com/edmundlth/scalable_learning_coefficient_with_sgld/tree/v1.0