Slideflow

Faster slide reading with cuCIM

This release adds [cuCIM](https://github.com/rapidsai/cucim) as a slide reading backend, which is now used by default if available. cuCIM is 2-3x faster for whole-slide image reading, with lighter CPU utilization and about half the memory footprint. It is also much easier to install, requiring only pip:


pip install slideflow[cucim] cupy-cuda11x


cuCIM only supports SVS and TIFF images, however, so users working with other slide formats may opt to continue using Libvips. The Libvips backend can be manually enabled by setting the environmental variable `SF_SLIDE_BACKEND=libvips`.

New QC functionality
Slide quality control (QC) methods are now customizable, and can be used during training by passing any Callable or list of Callables to `sf.WSI.qc()`. Example QC functions (Otsu and Gaussian) are available in `sf.slide.qc`, and would be applied as so:

python
from slideflow.slide import qc

wsi = sf.WSI(...)
wsi.qc([qc.Otsu(), qc.Gaussian(sigma=3)]


Additionally, QC masks can now be saved to disk for reproducibility and efficiency. To automatically save a QC mask after generation, use the `sf.slide.qc.Save` and `sf.slide.qc.Load` functions. The former accepts an optional path argument in which the masks will be saved; if not supplied, masks will be saved in the same directory as the slides.

python
from slideflow.slide import qc

Save QC masks after generation
wsi = sf.WSI(...)
wsi.qc([qc.Otsu(), qc.Save('/path/to/directory')])

...

Load any previously saved QC masks
wsi = sf.WSI(...)
wsi.qc([qc.Load('/path/to/directory')])


This same interface can be used when using the project-level function `Project.extract_tiles()`:

python
P.extract_tiles(..., qc=[qc.Otsu()])


Easier dataset reproduction
It is now much easier to setup and reproduce an existing project, particularly if the slides are available at the [Genomic Data Commons](https://portal.gdc.cancer.gov/) (GDC). The new `sf.project.create()` function uses a project configuration file and an internal GDC manifest to automatically setup a project from a given annotations file, download slides, and verify integrity with MD5 hashing.

Several example datasets are now provided in `datasets/` to assist with easy initial setup. Setting up a new project, including automatic slide downloads, is as easy as:

python
import slideflow as sf

P = sf.project.create(
'/home/project',
cfg='slideflow/datasets/thyroid_brs/thyroid_brs.json')
download=True


Custom training loss
Models can now be trained with custom loss functions by supplying a dictionary to the `loss` argument in `sf.ModelParams`, with the keys `type` and `fn`. `type` must be either `'categorical'`, `'linear'`, or `'cph'`, and 'fn' is a callable loss function.

Tensorflow/Keras
For Tensorflow/Keras, the loss function must accept arguments `y_true, y_pred`. `y_true` will not be one-hot encoded. For linear losses, `y_true` may need to be cast to `tf.float32`. An example custom linear loss is given below:

python
def custom_linear_loss(y_true, y_pred):
y_true = tf.cast(y_true, tf.float32)
squared_difference = tf.square(y_true - y_pred)
return tf.reduce_mean(squared_difference, axis=-1)


This loss would be applied as follows:

python
hp = sf.ModelParams(..., loss={'type': 'linear', 'fn': custom_linear_loss})


PyTorch
For PyTorch, the loss function must return a nested loss function with arguments `output, target`. An example linear loss is given below:

python
def custom_linear_loss():
def loss_fn(output, target):
return torch.mean((target - output) ** 2)
return loss_fn


This would be applied as follow:

python
hp = sf.ModelParams(..., loss={'type': 'linear', 'fn': custom_linear_loss})


Optimizations
- Tile extraction now saves a CSV report of all extraction parameters, QC methods, image format, and backend/version in the same folder as the PDF report.
- Decreased memory utilization when training from whole-slide images (`from_wsi=True`)
- Change default `roi_method` from `'inside'` to `'auto'` for `Project.generate_heatmaps()`

Bug fixes
- Fixes `numeric_only` futurewarning for Pandas
- Fixes `StopIteration` error when training models using `from_wsi=True` in the PyTorch backend
- Specifies supported `tensorflow_probability` version in setup.py to match Tensorflow compatibility
- Renames `sf.header()` -> `sf.about()`
- Add `requests` to package requirements

Workbench: interactive visualization

This release contains a new interactive whole-slide visualization tool, **Slideflow Workbench**. Workbench offers an interactive, real-time preview of **whole-slide image processing** (background filtering, stain normalization, and ROIs); **model predictions**, **heatmaps**, **uncertainty**, and **saliency**. In addition to whole-slide images, Workbench supports visualization of **StyleGAN**-generated images, and includes experimental support for generating predictions from a real-time camera feed.

The goal of Workbench is to improve transparency of whole-slide image processing, easily and rapidly generate whole-slide predictions, and provide tools for model troubleshooting. Workbench is both fast and flexible, with an Imgui interface and OpenGL rendering that runs on Linux (x86 and aarch64), MacOS, and Windows. It also runs on Raspberry Pi (tested on Pi 4, 4 GB model).

Run workbench using the ``workbench.py`` script in the root repository directory:

bash
python3 workbench.py


or by instancing a Workbench object:

python
from slideflow.workbench import Workbench

bench = Workbench()
bench.run()


See the [full documentation](https://slideflow.dev/workbench_tools.html) for examples and more information.

StyleGAN3 support
StyleGAN3 models can now be trained in addition to StyleGAN2 models, with the new submodule `slideflow/gan/stylegan3`. Training is executed as with StyleGAN2, but with `model='stylegan3'`. Additional StyleGAN3-specific arguments can be passed to the `Project.gan_train()` function. For example, to train StyleGAN3 with rotational equivariance:

python
P.gan_train(
...,
model='stylegan3',
cfg='stylegan3-r'
)


See [the documentation](https://slideflow.dev/stylegan.html) for more information.

Train models directly from slides, without TFRecords
Models can now be trained directly from slides, without extracting tiles into TFRecords, using the argument `P.train(..., from_wsi=True)`. Additional related changes include:
- The `sf.io.tensorflow.interleave()` function allows interleaving of WSI tensorflow datasets, without intermediate extraction to TFRecords. the first argument was renamed from `tfrecords` to `paths`. WSIs undergo background filtering with Otsu's thresholding only (no grayspace filtering). Dataset balancing/clipping not supported.
- New `sf.WSI.tensorflow()` function which returns a Tensorflow Dataset from the WSI generator.
- New `Dataset.slide_manifest()` function estimates number of non-background tiles among slides in the dataset.
- New argument `lazy_iter` to WSI/TMA build_generator, where tile extraction is processed to the Pool only in batches to prevent high memory usage when multiple slides are extracting tiles simultaneously.

Heatmap updates
Heatmaps have undergone extensive optimization and feature expansion, with updates including:
- New `sf.heatmap.ModelHeatmap` allows generating heatmaps from models (tf.keras.Model or torch.nn.Module) already loaded in memory
- Heatmap `slide` argument now accepts either path to slide (str) or WSI object.
- Adds `device` argument to `sf.Heatmap`, for PyTorch backend
- More efficient tile processing during whole-slide predictions/heatmaps for PNG-trained models
- New `num_threads` and `num_processes` arguments for `sf.Heatmap` which allow specifying the type and amount of multithreading/multiprocessing to use during tile extraction from slides. If `num_threads` is non-zero, a ThreadPool will be used to parallelize tile extraction. If `num_processes` is non-zero, a multiprocessing Pool will be used.
- Heatmaps can now be generated asynchronously by setting `Heatmap(generate=False)` then manually calling `Heatmap.generate(asynchronous=True)`. The latter will return the grid (updated asynchronously with predictions) and the heatmap thread.
- Heatmap performance improvement by using vectorized normalizers, when available
- Heatmap logits/uncertainty can be saved with `heatmap.save_npz()`. `heatmap.save()` will also automatically export logits/uncertainty in npz format with the filename `[slidename].npz`.
- Heatmap logits/uncertainty can be loaded with either `heatmap.load()` or `heatmap.load_npz()`

Normalizer optimizations
Several stain normalization improvements were made, including:
- New PyTorch-native Macenko implementation
- Efficiency improvements in Macenko (numpy) normalizer
- Efficiency improvements in Macenko (Tensorflow) normalizer
- Efficiency improvements in Vahadane normalizer
- Default Vahadane normalizer ('vahadane') switched back to SPAMS. Both are still accessible via 'vahadane_sklearn' and 'vahadane_spams'
- Skip Vahadane normalizer throughput testing during functional testing

Other important updates
- The `'r_squared'` metric has been changed from r^2 (square of Pearson correlation coefficient) to R^2 (coefficient of determination), which is the more appropriate metric for determining model strength
- Tensorflow-32 now disabled by default. TF32 can be enabled by passing `allow_tf32=True` to any function which also accepts the argument `mixed_precision`
- New `Dataset.summary()` function provides an informative overview of a Dataset object
- Bespoke QC methods can now be applied to slides with `sf.WSI.apply_qc_mask`.
- New `load_method` argument for any project function that loads Tensorflow models allows specifying how models are loaded (either the full model is loaded, or just weights are loaded; see function docstrings for more information).

Smaller changes
- `SlideMap` now supports `ParametricUMAP` for dimension reduction
- Slight tweak to the ROI filtering algorithm provides slightly more accurate tile extraction from within ROIs
- Allows 'jpeg' slide extension (in addition to 'jpg')
- Can now generate saliency maps by ID, with the following syntax: `SaliencyMap.get(img, method=sf.grad.VANILLA)`
- New function `_VIPSWrapper.read_from_pyramid()` allows for efficient reading from a VIPS image using the best available downsample layer
- New `sf.util.get_gan_config()` utility function
- Error checks for mismatched img_format for `evaluate()` and `train()`
- Allow configuring a `Project` with `sources` equal to a string (rather than only list of str)
- Disable UQ during mid-training validation (Tensorflow)
- Change `silent` argument to `verbose` in `sf.WSI`
- Decrease size of threadpool from 16 -> 8 for `Dataset.load_indices()`
- Decrease default chunk_size from 16 -> 1 for PyTorch interleave
- Rename `preload_factor` argument to `prefetch_factor` for PyTorch interleave_dataloader, to match PyTorch syntax
- `pin_memory` now defaults to False for PyTorch dataloaders
- `persistent_workers` now defaults to False for PyTorch dataloaders
- TestSuite accepts `tile_px` argument to manually site tile size
- Improvements to tile extraction progress bar
- Rename `sf.stats.eval_from_dataset` -> `.eval_dataset()`
- Rename `sf.stats.predict_from_dataset` -> `.predict_dataset()`
- Make `sf.model.tensorflow.eval_from_model()` public, as with `...torch.eval_from_model()`, and `.predict_from_model()` for both backends
- Overhaul of Tensorflow evaluate/predict metrics functions with removal of unused arguments (e.g. pred_args) and removal of redundant code.
- Tensorflow training no long instantiates multiple redundant validation datasets, and now instead only uses one validation dataset.
- Multiprocessing/pickling support for `sf.WSI`

Bug fixes
- Fixes for `UncertaintyInterface` for Tensorflow when using classmethod `.from_model()`.
- Fixes tile extraction for JPEG slides.
- Fixes overflow when calculating loss during evaluation (Tensorflow)
- Fixes PyTorch models saving without `.zip` extension
- Fixes overflow error when calculating loss during evaluation/predictions in Tensorflow backend
- Fixes bug where preserved site cross validation sometimes hangs when using promo/bonmin

Known issues
- Progress bars sometimes prevent exiting via `KeyboardInterrupt`
- `from_wsi` not implement for PyTorch backend
- Heatmaps do not work for multi-outcome models