Crepes

Features

- Semi-online inductive conformal classifiers, regressors and predictive systems, which iteratively update the calibration set after making each prediction, have been incorporated. For `WrapClassifier`, the methods `predict_p`, `predict_set` and `evaluate`, now include the argument `online` together with the argument `warm_start`, where the former enables online calibration (disabled by default) and the latter enables extending the original calibration set during online calibration (enabled by default). Similarly, for `WrapRegressor`, the methods `predict_p`, `predict_int`, `predict_percentiles`, `predict_cpds` and `evaluate` include the arguments `online` and `warm_start`. Semi-online conformal predictors are also enabled by the methods `predict_p_online` (for `ConformalClassifier`, `ConformalRegressor`, and `ConformalPredictiveSystem`), `predict_set_online` (for `ConformalClassifier`), `predict_int_online` (for `ConformalRegressor` and `ConformalPredictiveSystem`), `predict_percentiles_online` and `predict_cpds_online` (for `ConformalPredictiveSystem`). Moreover, the `evaluate` method for all three types of conformal predictor now includes an additional argument `online`, which if set to true enables online calibration.

- The method `predict_p` for generating (smoothed or non-smoothed) p-values using conformal regressors has been added.

- New methods for conformal predictive systems have been added; `predict_p` for obtaining p-values, `predict_int` for obtaining prediction intervals, `predict_percentiles` for obtaining percentiles, and `predict_cpds` for obtaining conformal predictive distributions. These outputs can alternatively (as before) be generated by the methods `predict` (for `ConformalPredictiveSystem`) and `predict_cps` (for `WrapRegressor`).

- Both smoothed and non-smoothed p-values may now be output for conformal predictive systems; smoothing is the default but can be disabled by setting `smoothing=False` for the methods `predict_p` and `predict`.

- The `evaluate` method of `ConformalClassifier`, `ConformalRegressor`, and `ConformalPredictiveSystem`, as well as of `WrapClassifier` and `WrapRegressor`, now includes the metric `ks_test`, which provides the p-value for the Kolmogorov-Smirnov test of uniformity of predicted p-values. Thanks to egonmedhatten for the suggestion!

Fixes

- Fitted objects now contain the attribute `fitted_` to allow for proper handling in scikit-learn pipelines. Thanks to lukethomrichardson for suggesting the fix.

Features

- The `calibrate` methods of the classes `WrapClassifier` and `WrapRegressor` now take an additional argument `seed`, for setting the state of the random number generator. This allows for `predict_p` and `predict_set` of the former class and `predict_int` and `predict_cps` of the latter class as well as `evaluate` for both classes to become deterministic. The methods `predict_p`, `predict_set`, `predict_int` ,`predict_cps` and `evaluate` of these classes also have an argument `seed`, which can be used to over-ride any setting by the `calibrate` method. In addition, the corresponding methods of the classes `ConformalClassifier` and `ConformalPredictiveSystem` also include the argument `seed` for the same purpose. (The methods of `ConformalRegressor` currently contains no stochastic components and there is hence no need for a seed.) Thanks to egonmedhatten and tuvelofstrom for suggesting this extension.

- The `predict_p` methods of the classes `ConformalClassifier` and `WrapClassifier` now takes an optional argument `smoothing` to allow for generating both smoothed and non-smoothed p-values (default: `smoothing=True`).

- The default value for the parameter `smoothing` of the `predict_set` and `evaluate` methods has been changed to `True`.

- The class `DifficultyEstimator` in `crepes.extras` now includes a parameter `f` for providing a function to compute the difficulty estimates.

Fixes

- The documentation for the class `MondrianCategorizer` in `crepes.extras` has been corrected.

Features

- The class `MondrianCategorizer` was added to `crepes.extras`, for generating categories to be used by Mondrian conformal classifiers, regressors and predictive systems. See [the documentation](https://crepes.readthedocs.io/en/latest/crepes.extras.html) for the interface to objects of the class through the `fit` and `apply` methods.

- The class `WrapRegressor` has been updated so that the method `calibrate` allows for specifying an (optional) difficulty estimator and (optional) Mondrian categorizer, which are used both for calibration and making predictions, instead of requiring that difficulty estimates and Mondrian categories are provided separately for these tasks. The methods `predict_int`, `predict_cps` and `evaluate` no longer require `sigmas` and `bins` to be provided for normalized and Mondrian conformal regressors and predictive systems. Thanks to tuvelofstrom for suggestions along these lines.

- The class `WrapClassifier` has been updated so that the method `calibrate` allows for specifying an (optional) Mondrian categorizer, which is used both for calibration and making predictions. The methods `predict_p`, `predict_set` and `evaluate` no longer require `bins` to be provided for Mondrian conformal classifiers.

Fix

- Label vectors represented by pandas.Series are converted to NumPy arrays to avoid indexing issues. Thanks to valeman for pointing this out.

Fixes

- Fixed deprecated code for checking if an array is non-empty in the `ConformalPredictiveSystem` class. Thanks to tuvelofstrom for pointing this out.

Features

- The function `margin` for computing non-conformity scores for conformal classifiers has been added to `crepes.extras`.

Fixes

- Fixed a bug in the `DifficultyEstimator` class (in `crepes.extras`), which caused an error when trying to display a non-fitted object. Thanks to tuvelofstrom for pointing this out.

- Fixed an error in the documentation for the function `hinge`.

- The Jupyter notebooks `crepes_nb_wrap.ipynb` and `crepes_nb.ipynb` have been updated to illustrate the new `margin` function.

Features

- The classes `ConformalClassifier` and `WrapClassifier` have been added to `crepes`, allowing for generation of standard and Mondrian conformal classifiers, which produce p-values and prediction sets. The `calibrate` method of `WrapClassifier` allows for easily generating class-conditional conformal classifiers and using out-of-bag calibration. See [the documentation](https://crepes.readthedocs.io/en/latest/crepes.html) for the interface to objects of the class through the `calibrate`, `predict_p` and `predict_set` methods, in addition to the `fit`, `predict` and `predict_proba` methods of the wrapped learner. The method `evaluate` allows for evaluating the predictive performance using a set of standard metrics.

- The function `hinge` for computing non-conformity scores for conformal classifiers has been added to `crepes.extras`.

Fixes

- The class `Wrap` has changed name to `WrapRegressor` and the arguments to the `calibrate` method of this class have been changed to be in line with the `calibrate` method of `WrapClassifier`.

- The Jupyter notebooks `crepes_nb_wrap.ipynb` and `crepes_nb.ipynb` have been updated and extended