Nannyml

Changed

- Updated API docs for the `nannyml.io` package, thanks [maciejbalawejder](https://github.com/maciejbalawejder) [(#286)](https://github.com/NannyML/nannyml/issues/286)
- Restricted versions of `numpy` to be `<1.25`, since there seems to be a change in the `roc_auc` calculation somehow [(301)](https://github.com/NannyML/nannyml/issues/301)

Added

- Support for Data Quality calculators in the CLI runner
- Support for Data Quality results in `Ranker` implementations [(297)](https://github.com/NannyML/nannyml/issues/297)
- Support `mendable` in the docs [(295)](https://github.com/NannyML/nannyml/issues/295)
- Documentation landing page [(303)](https://github.com/NannyML/nannyml/issues/303)
- Support for calculations with delayed targets [(306)](https://github.com/NannyML/nannyml/issues/306)

Fixed

- Small changes to quickstart, thanks [NeoKish](https://github.com/NeoKish) [(#291)](https://github.com/NannyML/nannyml/issues/291)
- Fix an issue passing `*args` and `**kwargs` in `Result.filter()` and subclasses [(298)](https://github.com/NannyML/nannyml/issues/298)
- Double listing of the binary dataset documentation page
- Add missing thresholds to `roc_auc` in `CBPE` [(294)](https://github.com/NannyML/nannyml/issues/294)
- Fix plotting issue due to introduction of additional values in the 'display names tuple' [(305)](https://github.com/NannyML/nannyml/issues/305)
- Fix broken exception handling due to inheriting from `BaseException` and not `Exception` [(307)](https://github.com/NannyML/nannyml/issues/307)

Changed

- Significant QA work on all the documentation, thanks [santiviquez](https://github.com/santiviquez) and
[maciejbalawejder](https://github.com/maciejbalawejder)
- Reworked the [`nannyml.runner`](nannyml/runner.py) and the accompanying configuration format to improve flexibility (e.g. setting
custom initialization parameters, running a calculator multiple times, excluding a calculator, ...).
- Added support for custom thresholds to the [`nannyml.runner`](nannyml/runner.py)
- Simplified some of the `nannyml.io` interfaces, especially the [`nannyml.io.RawFilesWriter`](nannyml/io/raw_files_writer.py)
- Reworked the [`nannyml.base.Result`](nannyml/base.py)
- Totally revamped [quickstart documentation](docs/quick.rst) based on a real life dataset, thanks [jakubnml](https://github.com/jakubnml)

Added

- Added new calculators to support simple data quality metrics such as counting missing or unseen values.
For more information, check out the [data quality tutorials](https://nannyml.readthedocs.io/en/main/tutorials/data_quality.html).

Fixed

- Fixed an issue where x-axis titles would appear on top of plots
- Removed erroneous checks during calculation of realized regression performance metrics. [(279)](https://github.com/NannyML/nannyml/issues/279)
- Fixed an issue dealing with `az://` URLs in the CLI, thanks [michael-nml](https://github.com/michael-nml) [(#283)](https://github.com/NannyML/nannyml/issues/283)

Changed

- Applied new rules for visualizations. Estimated values will be the color indigo and represented with a dashed line.
Calculated values will be blue and have a solid line. This color coding might be overridden in comparison plots.
Data periods will no longer have different colors, we've added some additional text fields to the plot to indicate the data period.
- Cleaned up legends in plots, since there will no longer be a different entry for reference and analysis periods of metrics.
- Removed the lower threshold for default thresholds of the KS and Wasserstein drift detection methods.

Added

- We've added the `business_value` metric for both estimated and realized binary classification performance. It allows
you to assign a value (or cost) to true positive, true negative, false positive and false negative occurrences.
This can help you track something like a monetary value or business impact of a model as a metric. Read more in the
business value tutorials ([estimated](https://nannyml.readthedocs.io/en/latest/tutorials/performance_estimation/binary_performance_estimation/business_value_estimation.html)
or [realized](https://nannyml.readthedocs.io/en/latest/tutorials/performance_calculation/binary_performance_calculation/business_value_calculation.html))
or the [how it works](https://nannyml.readthedocs.io/en/latest/how_it_works/business_value.html) page.

Fixed

- Sync quickstart of the README with the dedicated quickstart page. [(256)](https://github.com/NannyML/nannyml/issues/256)
Thanks [NeoKish](https://github.com/NeoKish)!
- Fixed incorrect code snippet order in the thresholding tutorial. [(258)](https://github.com/NannyML/nannyml/issues/258)
Thanks once more to the one and only [NeoKish](https://github.com/NeoKish)!
- Fixed broken container build that had sneakily been going on for a while
- Fixed incorrect confidence band color in comparison plots [(259)](https://github.com/NannyML/nannyml/issues/259)
- Fixed incorrect titles and missing legends in comparison plots [(264)](https://github.com/NannyML/nannyml/issues/264)
- Fixed an issue where numerical series marked as category would cause issues during Chi2 calculation

Changed

- Updated univariate drift methods to no longer store all reference data by default [(182)](https://github.com/NannyML/nannyml/issues/182)
- Updated univariate drift methods to deal better with missing data [(202)](https://github.com/NannyML/nannyml/issues/202)
- Updated the included example datasets
- Critical security updates for dependencies
- Updated visualization of multi-level table headers in the docs [(242)](https://github.com/NannyML/nannyml/issues/242)
- Improved typing support for Result classes using generics

Added

- Support for estimating the confusion matrix for binary classification [(191)](https://github.com/NannyML/nannyml/issues/191)
- Added `treat_as_categorical` parameter to univariate drift calculator [(239)](https://github.com/NannyML/nannyml/issues/239)
- Added comparison plots to help visualize two different metrics at once

Fixed

- Fix missing confidence boundaries in some plots [(193)](https://github.com/NannyML/nannyml/issues/193)
- Fix incorrect metric names on plot y-axes [(195)](https://github.com/NannyML/nannyml/issues/195)
- Fix broken links to external docs [(196)](https://github.com/NannyML/nannyml/issues/196)
- Fix missing display name to performance calculation and estimation charts [(200)](https://github.com/NannyML/nannyml/issues/200)
- Fix missing confidence boundaries for single metric plots [(203)](https://github.com/NannyML/nannyml/issues/203)
- Fix incorrect code in example notebook for ranking
- Fix result corruption when re-using calculators [(206)](https://github.com/NannyML/nannyml/issues/206)
- Fix unintentional period filtering [(199)](https://github.com/NannyML/nannyml/issues/199)
- Fixed some typing issues [(213)](https://github.com/NannyML/nannyml/issues/213)
- Fixed missing data requirements documentation on regression [(215)](https://github.com/NannyML/nannyml/issues/215)
- Corrections in the glossary [(214)](https://github.com/NannyML/nannyml/issues/214), thanks [sebasmos](https://github.com/sebasmos)!
- Fix missing treshold in plotting legend [(219)](https://github.com/NannyML/nannyml/issues/219)
- Fix missing annotation in single row & column charts [(221)](https://github.com/NannyML/nannyml/issues/221)
- Fix outdated performance estimation and calculation docs [(223)](https://github.com/NannyML/nannyml/issues/223)
- Fix categorical encoding of unseen values for DLE [(224)](https://github.com/NannyML/nannyml/issues/224)
- Fix incorrect legend for None timeseries [(235)](https://github.com/NannyML/nannyml/issues/235)

Added

- Added some extra semantic methods on results for easy property access. No dealing with multilevel indexes required.
- Added functionality to compare results and plot that comparison. Early release version.

Fixed

- Pinned Sphinx version to 4.5.0 in the [documentation requirements](docs/requirements.txt).
Version selector, copy toggle buttons and some styling were broken on RTD due to unintended usage of Sphinx 6 which
treats jQuery in a different way.

Changed

- Log Ranker usage logging
- Remove some redundant parameters in `plot()` function calls for data reconstruction results, univariate drift results,
CBPE results and DLE results.
- Support "single metric/column" arguments in addition to lists in class creation [(165)](https://github.com/NannyML/nannyml/issues/165)
- Fix incorrect 'None' checks when dealing with defaults in univariate drift calculator
- Multiple updates and corrections to the docs (thanks [nikml](https://github.com/nikml)!), including:
- Updating univariate drift tutorial
- Updating README
- Update PCA: How it works
- Fix incorrect plots
- Fix quickstart [(171)](https://github.com/NannyML/nannyml/issues/171)
- Update chunker docstrings to match parameter names, thanks [mrggementiza](https://github.com/jrggementiza)!
- Make sequence 'None' checks more readable, thanks [mrggementiza](https://github.com/jrggementiza)!
- Ensure error handling in usage logging does not cause errors...
- Start using `OrdinalEncoder` instead of `LabelEncorder` in DLE. This allows us to deal with "unseen" values in the
analysis period.

Added

- Added a Store to provide persistence for objects. Main use case for now is storing fitted calculators to be reused
later without needing to fit on reference again. Current store implementation uses a local or remote filesystem as a
persistence layer. Check out the documentation on [persisting calculators](https://nannyml.readthedocs.io/en/latest/tutorials/persisting_calculators.html).

Fixed

- Fix incorrect interpretation of `y_pred` column as continuous values for the included sample binary classification data.
Converting the column explicitly to "category" data type for now, update of the dataset to follow soon.
[(171)](https://github.com/NannyML/nannyml/issues/171)
- Fix broken image link in README, thanks [mrggementiza](https://github.com/jrggementiza)!
- Fix missing key in the CLI section on raw files output, thanks [CoffiDev](https://github.com/CoffiDev)!
- Fix upper and lower thresholds for data reconstruction being swapped [(179)](https://github.com/NannyML/nannyml/issues/179)
- Fix stacked bar chart plots (missing bars + too many categories shown)