Lightning-uq-box

What's Changed

The 0.2 release includes both new methods, as well as bug fixes. It is especially nice that we have three new contributors who have fixed bugs and improved the documentation. The release notes below mention some of the new highlights as well as central improvements to the library. Thank you to all contributors!

If you come across these release notes and are interested in becoming involved, see [this issue](https://github.com/lightning-uq-box/lightning-uq-box/issues/43) for an overview of other UQ methods, or just get in touch with any other comments or questions via a new issue.

New Methods

We have added new UQ methods as well as extended some methods to new tasks. See the table below for an update of the currently supported UQ method and task combinations.

* Add Img2Img Conformal Prediction by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/118
* Extend other available methods for Pixelwise Regression by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/119
* Add Variational Bayesian Last Layer by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/137
* Add UQ-Method Masked Ensemble (Masksembles) by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/157
* Add ZigZag UQ Method by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/161
* Add VAE UQ Method by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/163
* Add Mixture Density Network by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/179
* Add Density Layer as UQ Method by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/181

Bug Fixes / Improvements

* Remove n_ensemble_members in DeepEnsemble signatures by codingS3b in https://github.com/lightning-uq-box/lightning-uq-box/pull/151
* Improve Toy Heteroscedastic Datamodule and Tutorials by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/147
* Fix GMM equations for `DeepEnsembleRegression` by elcorto in https://github.com/lightning-uq-box/lightning-uq-box/pull/156
* Update Segmentation models to save predictions by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/120
* Fix the loss function in probabilistic UNet.js by qqaazz800624 in https://github.com/lightning-uq-box/lightning-uq-box/pull/135
* Remove unused figure in plot_calibration_uq_toolbox() by elcorto in https://github.com/lightning-uq-box/lightning-uq-box/pull/141
* Add EMA support for CARD by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/187
* Deep Ensemble Eval Mode Fix by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/172
* Laplace improvements by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/138
* Refactor Posthoc by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/12

Installation

Lightning UQ Box is now also available for installation via conda and spack, thanks to adamjstewart.

* Add conda and spack install instructions by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/241

Linters and CI

We have switched to ruff (replacing balck, flake8, isort, pydocstyle, and pyupgrade) as our main linting tool, added dependabot to manage dependencies, and began work on static type hint testing, thanks adamjstewart.

* Improve Test Coverage by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/121
* Use ruff by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/131
* Add dependabot by nilsleh in https://github.com/lightning-uq-box/lightning-uq-box/pull/189
* mypy: fix test type hints by adamjstewart in https://github.com/lightning-uq-box/lightning-uq-box/pull/224
* mypy: fix viz type hints by adamjstewart in https://github.com/lightning-uq-box/lightning-uq-box/pull/225
* mypy: fix more test type hints by adamjstewart in https://github.com/lightning-uq-box/lightning-uq-box/pull/226
* mypy: fix dataset type hints by adamjstewart in https://github.com/lightning-uq-box/lightning-uq-box/pull/227
* mypy: fix eval_utils type hints by adamjstewart in https://github.com/lightning-uq-box/lightning-uq-box/pull/228

Citation
If you find this library useful for your work, you can now cite our [preprint](https://arxiv.org/abs/2410.03390).

* Add citation information by adamjstewart in https://github.com/lightning-uq-box/lightning-uq-box/pull/252

New Contributors
* qqaazz800624 made their first contribution in https://github.com/lightning-uq-box/lightning-uq-box/pull/135
* elcorto made their first contribution in https://github.com/lightning-uq-box/lightning-uq-box/pull/141
* codingS3b made their first contribution in https://github.com/lightning-uq-box/lightning-uq-box/pull/151

Other Contributors
* adamjstewart
* nilsleh


**Full Changelog**: https://github.com/lightning-uq-box/lightning-uq-box/compare/v0.1.0...v0.2.0

This is the first release of the Lightning-UQ-Box.

The following table shows the current status of implemented methods.


Single Forward Pass Methods

| Uncertainty Quantification Method (UQ-Method) | Regression | Classification | Segmentation | Pixel Wise Regression |
|-----------------------------------------------|:----------:|:--------------:|:------------:|:---------------------:|
| Quantile Regression (QR) | ✅ | ❌ | ❌ | ⏳ |
| Deep Evidential (DE) | ✅ | ⏳ | ⏳ | ⏳ |
| Mean Variance Estimation (MVE) | ✅ | ❌ | ❌ | ⏳ |

Approximate Bayesian Methods

| Uncertainty Quantification Method (UQ-Method) | Regression | Classification | Segmentation | Pixel Wise Regression |
|-----------------------------------------------|:----------:|:--------------:|:------------:|:---------------------:|
| Bayesian Neural Network VI ELBO (BNN_VI_ELBO) | ✅ | ✅ | ✅ | ⏳ |
| Bayesian Neural Network VI (BNN_VI) | ✅ | ⏳ | ⏳ | ⏳ |
| Deep Kernel Learning (DKL) | ✅ | ✅ | ❌ | ❌ |
| Deterministic Uncertainty Estimation (DUE) | ✅ | ✅ | ❌ | ❌ |
| Laplace Approximation (Laplace) | ✅ | ✅ | ❌ | ❌ |
| Monte Carlo Dropout (MC-Dropout) | ✅ | ✅ | ✅ | ⏳ |
| Stochastic Gradient Langevin Dynamics (SGLD) | ✅ | ✅ | ⏳ | ⏳ |
| Spectral Normalized Gaussian Process (SNGP) | ✅ | ✅ | ❌ | ❌ |
| Stochastic Weight Averaging Gaussian (SWAG) | ✅ | ✅ | ✅ | ⏳ |
| Deep Ensemble | ✅ | ✅ | ✅ | ⏳ |

Generative Models

| Uncertainty Quantification Method (UQ-Method) | Regression | Classification | Segmentation | Pixel Wise Regression |
|-----------------------------------------------|:----------:|:--------------:|:------------:|:---------------------:|
| Classification And Regression Diffusion (CARD)| ✅ | ✅ | ❌ | ❌ |
| Probabilistic UNet | ❌ | ❌ | ✅ | ❌ |
| Hierarchical Probabilistic UNet | ❌ | ❌ | ✅ | ❌ |

Post-Hoc methods

| Uncertainty Quantification Method (UQ-Method) | Regression | Classification | Segmentation | Pixel Wise Regression |
|-----------------------------------------------|:----------:|:--------------:|:------------:|:---------------------:|
| Test Time Augmentation (TTA) | ✅ | ✅ | ⏳ | ⏳ |
| Temperature Scaling | ❌ | ✅ | ⏳ | ❌ |
| Conformal Quantile Regression (Conformal QR) | ✅ | ❌ | ❌ | ⏳ |
| Regularized Adaptive Prediction Sets (RAPS) | ❌ | ✅ | ❌ | ❌ |


Contributors
nilsleh
adamjstewart
stde
orbitfold
nm19000
JakobCode