Xplique

Fix issues:
- 150 [Bug]: - Causal fidelity problem with `steps=-1`
- 151 [Bug]: - MuFidelity does not work for Tabular data and time series

Now `CausalFidelity` metric for attribution methods works with `steps=-1` for `Insertion` and `Deletion` to be done feature by feature.

Now `MuFidelity` works as intended for tabular data and time series.

Patch
Fix issue 143 : all VRAM was allocated when Xplique was imported, this is not the case anymore.

Data type coverage

The first part of this release concerns Xplique data type coverage extension. It first modifies some methods to extend the coverage.




Non-square images

`SobolAttributionMethod` and `HsicAttributionImage` now support non-square images.




Image explanation shape harmonization

For image explanation, depending on the method, the explanation shape could be either $(n, h, w)$, $(n, h, w, 1)$, or $(n, h, w, 3)$. It was decided to harmonize it to $(n, h, w, 1)$.

Reducer for gradient-based methods

For images, most gradient-based provide a value for each channel, however, for consistency, it was decided that for images, explanations will have the shape $(n, h, w, 1)$. Therefore, gradient-based methods need to reduce the channel dimension of their image explanations and the `reducer` parameter chooses how to do it among {`"mean"`, `"min"`, `"max"`, `"sum"`, `None`}. In the case `None` is given, the channel dimension is not reduced. The default value is `"mean"` for methods except `Saliency` which is `"max"` to comply with the paper and `GradCAM` and `GradCAMPP` which are not concerned.




Time series

Xplique was initially designed for images but it also supports attribution methods for tabular data and now time series data.

Xplique conciders data with:
- 4 dimensions as images.
- 3 dimensions as time series.
- 2 dimensions as tabular data.

Tutorial

To show how to use Xplique on time series a new tutorial was designed: [Attributions: Time Series and Regression](https://colab.research.google.com/drive/1h0lThbcP5d2VKtRxwLG8z7KC8PExcVIA).

Plot

The function `xplique.plots.plot_timeseries_attributions` was modified to match `xplique.plots.plot_attributions` API. Here is an example from the tutorial on temperature forecasting for the next 24 hours based on weather data from the last 48 hours:

<img width="521" alt="image" src="https://github.com/deel-ai/xplique/assets/90199266/f4ad2060-378a-4138-8609-33ce95078215">


Methods

`Rise`, `Lime`, and `Kernelshap` now support time series natively.


Overview of covered data types and tasks

| **Attribution Method** | Type of Model | Images | Time Series and Tabular Data |
| :--------------------- | :----------------------- | :------------: | :--------------------------: |
| Deconvolution | TF | C✔️ OD❌ SS❌ | C✔️ R✔️ |
| Grad-CAM | TF | C✔️ OD❌ SS❌ | ❌ |
| Grad-CAM++ | TF | C✔️ OD❌ SS❌ | ❌ |
| Gradient Input | TF, PyTorch** | C✔️ OD✔️ SS✔️ | C✔️ R✔️ |
| Guided Backprop | TF | C✔️ OD❌ SS❌ | C✔️ R✔️ |
| Integrated Gradients | TF, PyTorch** | C✔️ OD✔️ SS✔️ | C✔️ R✔️ |
| Kernel SHAP | TF, PyTorch**, Callable* | C✔️ OD✔️ SS✔️ | C✔️ R✔️ |
| Lime | TF, PyTorch**, Callable* | C✔️ OD✔️ SS✔️ | C✔️ R✔️ |
| Occlusion | TF, PyTorch**, Callable* | C✔️ OD✔️ SS✔️ | C✔️ R✔️ |
| Rise | TF, PyTorch**, Callable* | C✔️ OD✔️ SS✔️ | C✔️ R✔️ |
| Saliency | TF, PyTorch** | C✔️ OD✔️ SS✔️ | C✔️ R✔️ |
| SmoothGrad | TF, PyTorch** | C✔️ OD✔️ SS✔️ | C✔️ R✔️ |
| SquareGrad | TF, PyTorch** | C✔️ OD✔️ SS✔️ | C✔️ R✔️ |
| VarGrad | TF, PyTorch** | C✔️ OD✔️ SS✔️ | C✔️ R✔️ |
| Sobol Attribution | TF, PyTorch** | C✔️ OD✔️ SS✔️ | 🔵 |
| Hsic Attribution | TF, PyTorch** | C✔️ OD✔️ SS✔️ | 🔵 |
| FORGrad enhancement | TF, PyTorch** | C✔️ OD✔️ SS✔️ | ❌ |

TF : Tensorflow compatible
C : [Classification](https://deel-ai.github.io/xplique/latest/api/attributions/classification/) | R : [Regression](https://deel-ai.github.io/xplique/latest/api/attributions/regression/) |
OD : [Object Detection](https://deel-ai.github.io/xplique/latest/api/attributions/object_detection/) | SS : [Semantic Segmentation (SS)](https://deel-ai.github.io/xplique/latest/api/attributions/semantic_segmentation/)

\* : See the [Callable documentation](https://deel-ai.github.io/xplique/latest/api/attributions/callable/)

** : See the [Xplique for PyTorch documentation](https://deel-ai.github.io/xplique/latest/api/attributions/pytorch/), and the [**PyTorch models**: Getting started](https://colab.research.google.com/drive/1bMlO29_0K3YnTQBbbyKQyRfo8YjvDbhe) notebook.

✔️ : Supported by Xplique | ❌ : Not applicable | 🔵 : Work in Progress


Metrics

Naturally, metrics now support Time series too.






---

Bugs correction

The second part of this release is to solve pending issues: 102, 123, 127, 128, 131, and 137.




Memories problem

Indeed, among the reported issues several concerned memory management.


SmoothGrad, VarGrad, and SquareGrad issue 137

`SmoothGrad`, `VarGrad`, and `SquareGrad` now use online statistics to compute explanations, which allows to make batch inferences. Furthermore, their implementation was refactorized with a `GradientStatistic` abstraction. It does not modify usage.


MuFidelity issue 137

The metric MuFidelity had the same problem as the three previous methods, it was also solved.


HsicAttributionMethod

This method had a different memory problem the `batch_size` for the model was used correctly, however, when computing the estimator a tensor of size `grid_size**2 * nb_design**2` was created. However, for big images and/or small objects in images, the `grid_size` needs to be increased, furthermore, for the estimator to converge, `nb_design` should also be increased accordingly. Which creates out-of-memory errors.

Thus an `estimator_batch_size` (different from the initial `batch_size`) was introduced to batch over the `grid_size**2` dimension. The default value is `None`, thus conserving the default behavior of the method, but when an out-of-memory occurs, setting an `estimator_batch_size` smaller than `grid_size**2` will reduce the memory cost of the method.




Other issues

Metrics input types issues 102 and 128

Now inputs and targets are sanitized to numpy arrays


Feature visualization latent dtype issue 131

In issue 131, there was a conflict in dtype between the model internal dtype and Xplique dtype. We made sure that the dtype used for the conflicting computation was the model's internal dtype.




Other corrections

Naturally, other problems were reported to us outside of issues or discovered by the team, we also addressed these.


Some refactorization

`Lime` was refactorized but it does not impact usage.


Small fixes

In `HsicAttributionMethod` and `SobolAttributionMethod` there was a difference between the documentation of the `perturbation_function` and the actual code.

For Craft, there were some remaining prints, but they may be useful, thus Craft's methods with print now take a `verbose` parameter.

New Features

CRAFT or Concept Recursive Activation FacTorization for Explainability

Introduction of the CRAFT method (see the [Paper](https://arxiv.org/pdf/2211.10154)) for both frameworks: PyTorch and Tensorflow. CRAFT is a method for automatically extracting human-interpretable concepts from deep networks.

python
from xplique.concepts import CraftTf as Craft

Cut the model in two parts (as explained in the paper)
first part is g(.) our 'input_to_latent' model returning positive activations,
second part is h(.) our 'latent_to_logit' model

g = tf.keras.Model(model.input, model.layers[-3].output)
h = tf.keras.Model(model.layers[-2].input, model.layers[-1].output)

Create a Craft concept extractor from these 2 models
craft = Craft(input_to_latent_model = g,
latent_to_logit_model = h,
number_of_concepts = 10,
patch_size = 80,
batch_size = 64)

Use Craft to get the crops (crops), the embedding of the crops (crops_u),
and the concept bank (w)
crops, crops_u, w = craft.fit(images_preprocessed, class_id=rabbit_class_id)

Compute Sobol indices to understand which concept matters
importances = craft.estimate_importance()

Display those concepts by showing the 10 best crops for each concept
craft.plot_concepts_crops(nb_crops=10)


See related [documentation](https://deel-ai.github.io/xplique/latest/api/concepts/craft), [Tensorflow tutorials](https://colab.research.google.com/drive/1jmyhb89Bdz7H4G2KfK8uEVbSC-C_aht_) and [PyTorch tutorial](https://colab.research.google.com/drive/16Jn2pQy4gi2qQYZFnuW6ZNtVAYiNyJHO)

Minor fix

Dead links

There were several dead links modified in 88165b3d996b139cbdbf054079b1de1aff104473.


Update methods table data types coverage

Lime does not work for now for semantic segmentation, hence the table was updated from "supported by xplique" to "work in progress". c307bb55dd1524ec246d2c3fbd59260a242f19e3.


Add tutorial links for feature visualization

The tutorials for feature visualization were not visible, thus the links were added in several places in 0547dbb62d1fb8ac3937d9019247fb6769111756.

Modify `setup.cfg`

In the setup, a tag was created when calling `bump2version`. However, this tag is created on the current branch and not master, which cannot be used. Thus this behavior was removed in 113974095120e48410d36e579e6fd6a0d0ee7ade.

New features

Semantic Segmentation

See related [documentation](https://deel-ai.github.io/xplique/latest/api/attributions/semantic_segmentation/) and [tutorial](https://colab.research.google.com/drive/1AHg7KO1fCOX5nZLGZfxkZ2-DLPPdSfbX).

python
explainer = Method(model, operator=xplique.Tasks.SEMANTIC_SEGMENTATION)


A new operator was designed to treat the semantic segmentation task, with the relative documentation and tutorial. It is used similarly to classification and regression, as shown in the example above. But the `model` specification changes and `targets` parameter definition differs (to design them, the user should use `xplique.utils_functions.segmentation` set of functions).


Object Detection

See related [documentation](https://deel-ai.github.io/xplique/latest/api/attributions/object_detection/) and [tutorial](https://colab.research.google.com/drive/1X3Yq7BduMKqTA0XEheoVIpOo3IvOrzWL).

python
explainer = Method(model, operator=xplique.Tasks.OBJECT_DETECTION)


The object detection API was adapted to the operator API, hence an object detection operator was designed to enable white box methods for object detection. Furthermore, the relative documentation and tutorials were introduced. Here also, `targets` and `model` specifications differ from classification ones.

Therefore, the `BoundingBoxExplainer` is now deprecated.



Documentation

Merge model, operator, and API description page into one

As fel-thomas highlighted in 132 remarks, the documentation was too divided, furthermore, a lot of information was redundant between those pages and they were interdependent. Hence the choice was made to merge the model, operator, and API description page into one. We believe it will simplify the use of the library.

Create task-related pages

As aforementioned, two tasks (Object Detection and Semantic Segmentation) were introduced in the documentation, their complexity induced a specific documentation page. However, it was not consistent to have documentation pages only for those two tasks. Therefore information about Classification and Regression was extracted from the common API page to create two other new task-specific pages. Finally, four task-specific were introduced to the documentation


Bug fixes

Regression

The regression operator was set to the MAE function in the previous release to allow the explanation of multi-output regression. However, such a function is not differentiable in zero, thus gradient-based methods were not working.

Hence, the behavior was set back to the previous behavior (a sum of the targeted outputs). Nonetheless, this operator is limited to single-output explanations, hence for multi-output regression, each output should be explained individually.