This release extends CAM methods with Layer-CAM, greatly improves the core features (CAM computation for multiple layers at once, CAM fusion, support of `torch.nn.Module`), while improving accessibility for entry users.
**Note**: TorchCAM 0.3.0 requires PyTorch 1.5.1 or higher.
Highlights
Enters Layer-CAM
The previous release saw the introduction of Score-CAM variants, and this one introduces you to [Layer-CAM](http://mftp.mmcheng.net/Papers/21TIP_LayerCAM.pdf), which is meant to be considerably faster, while offering very competitive localization cues!
Just like any other CAM methods, you can now use it as follows:
python
from torchcam.cams import LayerCAM
model = ....
Hook the model
cam_extractor = LayerCAM(model)
Consequently, the illustration of visual outputs for all CAM methods has been updated so that you can better choose the option that suits you:
Computing CAMs for multiple layers & CAM fusion
A class activation map is specific to a given layer in a model. To fully capture the influence of visual traits on your classification output, you might want to explore the CAMs for multiple layers.
For instance, here are the CAMs on the layers "layer2", "layer3" and "layer4" of a `resnet18`:
python
from torchvision.io.image import read_image
from torchvision.models import resnet18
from torchvision.transforms.functional import normalize, resize, to_pil_image
import matplotlib.pyplot as plt
from torchcam.cams import LayerCAM
from torchcam.utils import overlay_mask
Download an image
!wget https://www.woopets.fr/assets/races/000/066/big-portrait/border-collie.jpg
Set this to your image path if you wish to run it on your own data
img_path = "border-collie.jpg"
Get your input
img = read_image(img_path)
Preprocess it for your chosen model
input_tensor = normalize(resize(img, (224, 224)) / 255., [0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
Get your model
model = resnet18(pretrained=True).eval()
Hook the model
cam_extractor = LayerCAM(model, ["layer2", "layer3", "layer4"])
out = model(input_tensor.unsqueeze(0))
cams = cam_extractor(out.squeeze(0).argmax().item(), out)
Plot the CAMs
_, axes = plt.subplots(1, len(cam_extractor.target_names))
for idx, name, cam in zip(range(len(cam_extractor.target_names)), cam_extractor.target_names, cams):
axes[idx].imshow(cam.numpy()); axes[idx].axis('off'); axes[idx].set_title(name);
plt.show()
Now, the way you would combine those together is up to you. By default, most approaches use an element-wise maximum. But, LayerCAM has its own fusion method:
python
Let's fuse them
fused_cam = cam_extractor.fuse_cams(cams)
Plot the raw version
plt.imshow(fused_cam.numpy()); plt.axis('off'); plt.title(" + ".join(cam_extractor.target_names)); plt.show()
python
Overlay it on the image
result = overlay_mask(to_pil_image(img), to_pil_image(fused_cam, mode='F'), alpha=0.5)
Plot the result
plt.imshow(result); plt.axis('off'); plt.title(" + ".join(cam_extractor.target_names)); plt.show()
Support of `torch.nn.Module` as `target_layer`
While making the API more robust, CAM constructors now also accept `torch.nn.Module` as `target_layer`. Previously, you had to pass the name of the layer as string, but you can now pass the object reference directly if you prefer:
python
from torchcam.cams import LayerCAM
model = ....
Hook the model
cam_extractor = LayerCAM(model, model.layer4)
:zap: Latency benchmark :zap:
Since CAMs can be used from localization or production pipelines, it is important to consider latency along with pure visual output quality. For this reason, a latency evaluation script has been included in this release along with a full [benchmark table](https://github.com/frgfm/torch-cam#latency-benchmark).
Should you wish to have latency metrics on your dedicated hardware, you can run the script on your own:
shell
python scripts/eval_latency.py SmoothGradCAMpp --size 224
Notebooks :play_or_pause_button:
Do you prefer to only run code rather than write it? Perhaps you only want to tweak a few things?
Then enjoy the brand new [Jupyter notebooks](https://github.com/frgfm/torch-cam/tree/master/notebooks) than you can either run locally or on [Google Colab](https://colab.research.google.com/)!
:hugs: Live demo :hugs:
The ML community was recently blessed by HuggingFace with their beta of [Spaces](https://huggingface.co/spaces), which let you host free-of-charge your ML demos!
Previously, you were able to run the demo locally on deploy it on your own, but now, you can enjoy the [live demo of TorchCAM](https://huggingface.co/spaces/frgfm/torch-cam) :art:
Breaking changes
Multiple CAM output
Since CAM extractor can now compute the resulting maps for multiple layer at a time, the return type of all CAMs has been changed from `torch.Tensor` to `List[torch.Tensor]` with N elements, where N is the number of target layers.