Torchmetrics

Added

- Added **Text-related (NLP) metrics**:
- Word Error Rate (WER) ([383](https://github.com/Lightning-AI/torchmetrics/pull/383))
- ROUGE ([399](https://github.com/Lightning-AI/torchmetrics/pull/399))
- BERT score ([424](https://github.com/Lightning-AI/torchmetrics/pull/424))
- BLUE score ([360](https://github.com/Lightning-AI/torchmetrics/pull/360))
- Added `MetricTracker` wrapper metric for keeping track of the same metric over multiple epochs ([238](https://github.com/Lightning-AI/torchmetrics/pull/238))
- Added other metrics:
- Symmetric Mean Absolute Percentage error (SMAPE) ([375](https://github.com/Lightning-AI/torchmetrics/pull/375))
- Calibration error ([394](https://github.com/Lightning-AI/torchmetrics/pull/394))
- Permutation Invariant Training (PIT) ([384](https://github.com/Lightning-AI/torchmetrics/pull/384))
- Added support in `nDCG` metric for target with values larger than 1 ([349](https://github.com/Lightning-AI/torchmetrics/pull/349))
- Added support for negative targets in `nDCG` metric ([378](https://github.com/Lightning-AI/torchmetrics/pull/378))
- Added `None` as reduction option in `CosineSimilarity` metric ([400](https://github.com/Lightning-AI/torchmetrics/pull/400))
- Allowed passing labels in (n_samples, n_classes) to `AveragePrecision` ([386](https://github.com/Lightning-AI/torchmetrics/pull/386))

Changed

- Moved `psnr` and `ssim` from `functional.regression.*` to `functional.image.*` ([382](https://github.com/Lightning-AI/torchmetrics/pull/382))
- Moved `image_gradient` from `functional.image_gradients` to `functional.image.gradients` ([381](https://github.com/Lightning-AI/torchmetrics/pull/381))
- Moved `R2Score` from `regression.r2score` to `regression.r2` ([371](https://github.com/Lightning-AI/torchmetrics/pull/371))
- Pearson metric now only store 6 statistics instead of all predictions and targets ([380](https://github.com/Lightning-AI/torchmetrics/pull/380))
- Use `torch.argmax` instead of `torch.topk` when `k=1` for better performance ([419](https://github.com/Lightning-AI/torchmetrics/pull/419))
- Moved check for number of samples in R2 score to support single sample updating ([426](https://github.com/Lightning-AI/torchmetrics/pull/426))

Deprecated

- Rename `r2score` >> `r2_score` and `kldivergence` >> `kl_divergence` in `functional` ([371](https://github.com/Lightning-AI/torchmetrics/pull/371))
- Moved `bleu_score` from `functional.nlp` to `functional.text.bleu` ([360](https://github.com/Lightning-AI/torchmetrics/pull/360))

Removed

- Removed restriction that `threshold` has to be in (0,1) range to support logit input (
[351](https://github.com/Lightning-AI/torchmetrics/pull/351)
[401](https://github.com/Lightning-AI/torchmetrics/pull/401))
- Removed restriction that `preds` could not be bigger than `num_classes` to support logit input ([357](https://github.com/Lightning-AI/torchmetrics/pull/357))
- Removed module `regression.psnr` and `regression.ssim` ([382](https://github.com/Lightning-AI/torchmetrics/pull/382)):
- Removed ([379](https://github.com/Lightning-AI/torchmetrics/pull/379)):
* function `functional.mean_relative_error`
* `num_thresholds` argument in `BinnedPrecisionRecallCurve`

Fixed

- Fixed bug where classification metrics with `average='macro'` would lead to wrong result if a class was missing ([303](https://github.com/Lightning-AI/torchmetrics/pull/303))
- Fixed `weighted`, `multi-class` AUROC computation to allow for 0 observations of some class, as contribution to final AUROC is 0 ([376](https://github.com/Lightning-AI/torchmetrics/pull/376))
- Fixed that `_forward_cache` and `_computed` attributes are also moved to the correct device if metric is moved ([413](https://github.com/Lightning-AI/torchmetrics/pull/413))
- Fixed calculation in `IoU` metric when using `ignore_index` argument ([328](https://github.com/Lightning-AI/torchmetrics/pull/328))

---

Changed

- Extend typing ([330](https://github.com/Lightning-AI/torchmetrics/pull/330),
[332](https://github.com/Lightning-AI/torchmetrics/pull/332),
[333](https://github.com/Lightning-AI/torchmetrics/pull/333),
[335](https://github.com/Lightning-AI/torchmetrics/pull/335),
[314](https://github.com/Lightning-AI/torchmetrics/pull/314))

Fixed

- Fixed DDP by `is_sync` logic to `Metric` ([339](https://github.com/Lightning-AI/torchmetrics/pull/339))

Added

- Added **Image-related metrics**:
- Fréchet inception distance (FID) ([213](https://github.com/Lightning-AI/torchmetrics/pull/213))
- Kernel Inception Distance (KID) ([301](https://github.com/Lightning-AI/torchmetrics/pull/301))
- Inception Score ([299](https://github.com/Lightning-AI/torchmetrics/pull/299))
- KL divergence ([247](https://github.com/Lightning-AI/torchmetrics/pull/247))
- Added **Audio metrics**: SNR, SI_SDR, SI_SNR ([292](https://github.com/Lightning-AI/torchmetrics/pull/292))
- Added other metrics:
- Cosine Similarity ([305](https://github.com/Lightning-AI/torchmetrics/pull/305))
- Specificity ([210](https://github.com/Lightning-AI/torchmetrics/pull/210))
- Mean Absolute Percentage error (MAPE) ([248](https://github.com/Lightning-AI/torchmetrics/pull/248))
- Added `add_metrics` method to `MetricCollection` for adding additional metrics after initialization ([221](https://github.com/Lightning-AI/torchmetrics/pull/221))
- Added pre-gather reduction in the case of `dist_reduce_fx="cat"` to reduce communication cost ([217](https://github.com/Lightning-AI/torchmetrics/pull/217))
- Added better error message for `AUROC` when `num_classes` is not provided for multiclass input ([244](https://github.com/Lightning-AI/torchmetrics/pull/244))
- Added support for unnormalized scores (e.g. logits) in `Accuracy`, `Precision`, `Recall`, `FBeta`, `F1`, `StatScore`, `Hamming`, `ConfusionMatrix` metrics ([200](https://github.com/Lightning-AI/torchmetrics/pull/200))
- Added `squared` argument to `MeanSquaredError` for computing `RMSE` ([249](https://github.com/Lightning-AI/torchmetrics/pull/249))
- Added `is_differentiable` property to `ConfusionMatrix`, `F1`, `FBeta`, `Hamming`, `Hinge`, `IOU`, `MatthewsCorrcoef`, `Precision`, `Recall`, `PrecisionRecallCurve`, `ROC`, `StatScores` ([253](https://github.com/Lightning-AI/torchmetrics/pull/253))
- Added `sync` and `sync_context` methods for manually controlling when metric states are synced ([302](https://github.com/Lightning-AI/torchmetrics/pull/302))

Changed

- Forward cache is reset when `reset` method is called ([260](https://github.com/Lightning-AI/torchmetrics/pull/260))
- Improved per-class metric handling for imbalanced datasets for `precision`, `recall`, `precision_recall`, `fbeta`, `f1`, `accuracy`, and `specificity` ([204](https://github.com/Lightning-AI/torchmetrics/pull/204))
- Decorated `torch.jit.unused` to `MetricCollection` forward ([307](https://github.com/Lightning-AI/torchmetrics/pull/307))
- Renamed `thresholds` argument to binned metrics for manually controlling the thresholds ([322](https://github.com/Lightning-AI/torchmetrics/pull/322))
- Extend typing ([324](https://github.com/Lightning-AI/torchmetrics/pull/324),
[326](https://github.com/Lightning-AI/torchmetrics/pull/326),
[327](https://github.com/Lightning-AI/torchmetrics/pull/327))

Deprecated

- Deprecated `functional.mean_relative_error`, use `functional.mean_absolute_percentage_error` ([248](https://github.com/Lightning-AI/torchmetrics/pull/248))
- Deprecated `num_thresholds` argument in `BinnedPrecisionRecallCurve` ([322](https://github.com/Lightning-AI/torchmetrics/pull/322))

Removed

- Removed argument `is_multiclass` ([319](https://github.com/Lightning-AI/torchmetrics/pull/319))

Fixed

- AUC can also support more dimensional inputs when all but one dimension are of size 1 ([242](https://github.com/Lightning-AI/torchmetrics/pull/242))
- Fixed `dtype` of modular metrics after reset has been called ([243](https://github.com/Lightning-AI/torchmetrics/pull/243))
- Fixed calculation in `matthews_corrcoef` to correctly match formula ([321](https://github.com/Lightning-AI/torchmetrics/pull/321))

---

Added

- Added `is_differentiable` property:
* To `AUC`, `AUROC`, `CohenKappa` and `AveragePrecision` ([178](https://github.com/Lightning-AI/torchmetrics/pull/178))
* To `PearsonCorrCoef`, `SpearmanCorrcoef`, `R2Score` and `ExplainedVariance` ([225](https://github.com/Lightning-AI/torchmetrics/pull/225))

Changed

- `MetricCollection` should return metrics with prefix on `items()`, `keys()` ([209](https://github.com/Lightning-AI/torchmetrics/pull/209))
- Calling `compute` before `update` will now give warning ([164](https://github.com/Lightning-AI/torchmetrics/pull/164))

Removed

- Removed `numpy` as direct dependency ([212](https://github.com/Lightning-AI/torchmetrics/pull/212))

Fixed

- Fixed auc calculation and add tests ([197](https://github.com/Lightning-AI/torchmetrics/pull/197))
- Fixed loading persisted metric states using `load_state_dict()` ([202](https://github.com/Lightning-AI/torchmetrics/pull/202))
- Fixed `PSNR` not working with `DDP` ([214](https://github.com/Lightning-AI/torchmetrics/pull/214))
- Fixed metric calculation with unequal batch sizes ([220](https://github.com/Lightning-AI/torchmetrics/pull/220))
- Fixed metric concatenation for list states for zero-dim input ([229](https://github.com/Lightning-AI/torchmetrics/pull/229))
- Fixed numerical instability in `AUROC` metric for large input ([230](https://github.com/Lightning-AI/torchmetrics/pull/230))

- Cleaning remaining inconsistency and fix PL develop integration (
[191](https://github.com/Lightning-AI/torchmetrics/pull/191),
[192](https://github.com/Lightning-AI/torchmetrics/pull/192),
[193](https://github.com/Lightning-AI/torchmetrics/pull/193),
[194](https://github.com/Lightning-AI/torchmetrics/pull/194)
)

Added

- Added `BootStrapper` to easily calculate confidence intervals for metrics ([101](https://github.com/Lightning-AI/torchmetrics/pull/101))
- Added Binned metrics ([128](https://github.com/Lightning-AI/torchmetrics/pull/128))
- Added metrics for Information Retrieval ([(PL^5032)](https://github.com/Lightning-AI/lightning/pull/5032)):
* `RetrievalMAP` ([PL^5032](https://github.com/Lightning-AI/lightning/pull/5032))
* `RetrievalMRR` ([119](https://github.com/Lightning-AI/torchmetrics/pull/119))
* `RetrievalPrecision` ([139](https://github.com/Lightning-AI/torchmetrics/pull/139))
* `RetrievalRecall` ([146](https://github.com/Lightning-AI/torchmetrics/pull/146))
* `RetrievalNormalizedDCG` ([160](https://github.com/Lightning-AI/torchmetrics/pull/160))
* `RetrievalFallOut` ([161](https://github.com/Lightning-AI/torchmetrics/pull/161))
- Added other metrics:
* `CohenKappa` ([69](https://github.com/Lightning-AI/torchmetrics/pull/69))
* `MatthewsCorrcoef` ([98](https://github.com/Lightning-AI/torchmetrics/pull/98))
* `PearsonCorrcoef` ([157](https://github.com/Lightning-AI/torchmetrics/pull/157))
* `SpearmanCorrcoef` ([158](https://github.com/Lightning-AI/torchmetrics/pull/158))
* `Hinge` ([120](https://github.com/Lightning-AI/torchmetrics/pull/120))
- Added `average='micro'` as an option in AUROC for multilabel problems ([110](https://github.com/Lightning-AI/torchmetrics/pull/110))
- Added multilabel support to `ROC` metric ([114](https://github.com/Lightning-AI/torchmetrics/pull/114))
- Added testing for `half` precision ([77](https://github.com/Lightning-AI/torchmetrics/pull/77),
[135](https://github.com/Lightning-AI/torchmetrics/pull/135)
)
- Added `AverageMeter` for ad-hoc averages of values ([138](https://github.com/Lightning-AI/torchmetrics/pull/138))
- Added `prefix` argument to `MetricCollection` ([70](https://github.com/Lightning-AI/torchmetrics/pull/70))
- Added `__getitem__` as metric arithmetic operation ([142](https://github.com/Lightning-AI/torchmetrics/pull/142))
- Added property `is_differentiable` to metrics and test for differentiability ([154](https://github.com/Lightning-AI/torchmetrics/pull/154))
- Added support for `average`, `ignore_index` and `mdmc_average` in `Accuracy` metric ([166](https://github.com/Lightning-AI/torchmetrics/pull/166))
- Added `postfix` arg to `MetricCollection` ([188](https://github.com/Lightning-AI/torchmetrics/pull/188))

Changed

- Changed `ExplainedVariance` from storing all preds/targets to tracking 5 statistics ([68](https://github.com/Lightning-AI/torchmetrics/pull/68))
- Changed behaviour of `confusionmatrix` for multilabel data to better match `multilabel_confusion_matrix` from sklearn ([134](https://github.com/Lightning-AI/torchmetrics/pull/134))
- Updated FBeta arguments ([111](https://github.com/Lightning-AI/torchmetrics/pull/111))
- Changed `reset` method to use `detach.clone()` instead of `deepcopy` when resetting to default ([163](https://github.com/Lightning-AI/torchmetrics/pull/163))
- Metrics passed as dict to `MetricCollection` will now always be in deterministic order ([173](https://github.com/Lightning-AI/torchmetrics/pull/173))
- Allowed `MetricCollection` pass metrics as arguments ([176](https://github.com/Lightning-AI/torchmetrics/pull/176))

Deprecated

- Rename argument `is_multiclass` -> `multiclass` ([162](https://github.com/Lightning-AI/torchmetrics/pull/162))

Removed

- Prune remaining deprecated ([92](https://github.com/Lightning-AI/torchmetrics/pull/92))

Fixed

- Fixed when `_stable_1d_sort` to work when `n>=N` ([PL^6177](https://github.com/Lightning-AI/lightning/pull/6177))
- Fixed `_computed` attribute not being correctly reset ([147](https://github.com/Lightning-AI/torchmetrics/pull/147))
- Fixed to Blau score ([165](https://github.com/Lightning-AI/torchmetrics/pull/165))
- Fixed backwards compatibility for logging with older version of pytorch-lightning ([182](https://github.com/Lightning-AI/torchmetrics/pull/182))

---