Slideflow

_Multimodal inputs:_
- Multiple and linear variable inputs can be used to train models in addition to images, using the input_header parameter. Multiple inputs are merged as a concatenated vector with post-convolutional layers prior to hidden dense layers
- Models can also be trained without images with the hyperparameter drop_images

_CPH models:_
- Added negative_log_likelihood loss for time to event models. The outcome must be set to the time to event. The presence or absence of event numerical variable should be included as the first element in the input_header. This replicates DeepSurv i.e. neural network log loss cox proportional hazards (CPH) model. Also included custom metric c_index for tracking training and for evaluation of CPH models.
- Included permutation feature importance as an option for evaluate. When set to true, will run permutation feature importance to define the relative benefit of histology and each input in input_header.

_CLAM support:_
- Added CLAM training via new pipeline function SFP.train_clam().
- Added ability to export whole-slide, tile-level feature activations to torch format (for use with CLAM): AV.export_to_torch()
- Gave ActivationsVisualizer a min_tiles_per_slide (necessary for CLAM)
- Preliminary support for CLAM evaluation (SFP.evaluate_clam())

_QOL improvements / new features:_
- sf.SlideflowProject -> sf.Project
- Cleaned up method of instantiating a project; interactive project creation can be done by calling sf.Project.from_prompt(), but regular initialization now acts as expected without prompting for user input
- Moved several project creation functions out of the Project class and into project_utils
- Multiple categorical outcomes can now be used as model output by supplying a list of headers to outcome_label_headers() in SFP.train() or evaluate()
- Removed multi_outcome argument from SFP.train()
- Added slide PIL image error handling (with corrupted SVS files)
- Models no longer stored in .h5 format; saved using TF SavedModel format
- update.py now includes ability to update old models from .h5 to newer format
- Default image storage format for tfrecords is now lossless PNG
- Can now specify PNG or JPG tile extraction with img_format argument
- Unified & automatic tfrecord parsing, auto-detects internal tfrecord format for improved compatibility
- Training augmentation now includes random compression with JPG encoding at random quality levels
- Models now copy hyperparameters.json & slide_manifest.log into the model folder
- Adds roi_method argument to generate_heatmaps()
- added support for manual k-folds with k-fold-manual strategy and k_fold_header
- Can randomize origin during tile extraction with randomize_origin argument
- Standardized color output with heatmaps to display results 0 -> 1
- Added predict() functionality to SlideReader, which generates a slide-level grid of predictions / activations from a model
- Added SFP.predict_wsi() which generates whole-slide grid predictions from a model
- New argument for SFP.train(): skip_metrics. If True, will skip the validation metrics generation during training.
- For generate_activations, can now specify include_logits = True|False.
- Can now manually specify which predictions to save during evaluation - 'tile', 'patient', 'slide', or True for all
- relative paths now used by default in projects; specified with $ROOT/ prefix in settings.json
- Updated test suite & added new predict_wsi test
- TFRecords will now store tile location. Some pipeline functions (e.g. CLAM attention heatmaps) will require new TFRecords in this new format
- Added learning_rate_decay and dropout hyperparameters
- Streamlined ModelActivationsInterface; activation layers can now be manually specified and legacy compatibility is maintained
- Added support for NOT passing a slide manifest to SlideflowModel
- Commented-out code for preliminary RNA-seq support (as outcome)
- Adds sf.experimental for features not (yet) fully fleshed-out
- Removed use_fp16 references and replaced with project-wide mixed_precision setting
- Documentation updates
- Bug fix in ModelActivationsInterface with legacy models
- Removed GPU pools & autoselection
- Removed multithreading for predict_wsi()
- Replaces internal logging functions with official python logging module
- Logging improvements -> moved many messages to DEBUG logging level, which prompt with dim text
- Overhaul of logging, progress bars, and user experience
- New sf.io.reader submodule includes tensorflow-agnostic tfrecord reader, using dareblopy backend, for pytorch compatibility
- New optional tfrecord sharding
- SlideReader, when manually called, no longer automatically skips slides without ROI
- Heatmap now reads tile_px/tile_um from model, does not require explicitly setting

_Performance improvements:_
- 100x+ improvement in evaluation and validation speed
- 10-fold improvement in tile extraction speed with multiprocessing extraction
- Multi-GPU training can be enabled by passing multi_gpu=True to SFP.train() (SFP must be configured to have multiple GPUs visible via CUDA)
- Validation histogram generation & CSV predictions disabled during training
- Heatmaps can auto-buffer slides for faster generation
- Improved heatmap thumbnail resolution & speed of generation

_Breaking changes:_
- Annotation 'outcomes' are now referred to as annotation 'labels', since slide level features may be used as either input or outcomes
- All instances of "outcome_header" arguments replaced with "outcome_label_headers"
- Renamed AV.export_activations_to_csv() -> export_to_csv()
- Renamed SFP.export_activations_analytics() -> generate_activations()
- sf.SlideflowProject -> sf.Project
- Pipeline functions no long accept specific validation variables as arguments. To create and use a different validation plan than default, use sf.project.get_validation_settings() and pass the returned namespace to SFP.train() via `validation_settings`
- Removed validation settings from project settings
- Removed "use_fp16" from project-wide settings, replaced with "mixed_precision"

_General code cleanup:_
- Removed unused dependencies
- Removed old commented-out or redundant sections
- Re-ordered SFP functions alphabetically
- Improved indentations, arguments formatting
- Added error logging instead of sys.exit
- Moved SlideflowProject -> sf.project
- Moved multiprocessing SFP utilities into sf.project_utils
- Optimized import speed; tensorflow only imported when needed
- Reworked statistics module to reduce redundant code
- get_labels_from_annotations() now accepts a 'key' variable to customize the returned dictionary
- get_training_and_validation_tfrecords() now requires 'outcome_key' variable to specify the dictionary key that includes the outcome variables, provided in slide_labels_dict

_Bug fixes:_
- Bug fix in heatmaps (tiles were being shuffled prior to display)
- Bug fix for SFP.extract_tiles_from_tfrecords()
- fixed Dataset() global variables bug
- fixed bug where get_manifest() from dataset was not using filters
- fixed bug with PDF image error handling
- Bug fix with large datasets in Dataset.get_tfrecords()
- Bug fix with realtime normalization
- Fixed bug where evaluate() was not loading already-assigned int labels for multi-categorical outcomes.
- Bug fix where patient-level predictions were overwriting slide-level predictions
- Bug fix with permutation_feature_importance
- Fixes in interleave_tfrecords
- Progress bar bug fixes
- NASNetLarge bug fix
- Removes 'auto' num_thread option for Heatmap, which was unstable
- Mosaic logging fix
- Activations fix for handling empty tfrecords
- Other various fixes

New augmentation function - JPEG and/or flipping or rotation
- Augmentations can be specified by passing string to augment argument
- If 'j' is in string, applies random JPEG compression augmentation
- If 'x' is in string, applies random horizontal flip
- 'y' applies random vertical flip
- 'r' applies random rotation
- True (or 'xyrj') applies all augmentations

_New dependencies:_
- torch
- torchvision
- lifelines

Includes support for Ampere GPUs, test suite improvements, site-wise segmentation, and various other fixes and improvements.

*New features:*
- Adds support for new k-fold site-wise separation by passing validation_strategy="k-fold-preserved-site" [Fred]

*Revamped Test Suite:*
- Test suite: uses TaskWrapper to monitor pass/fail and time to completion
- Test suite: much cleaner low-verbosity output
- Test suite: improved TCGA download speed with larger chunk size
- Test suite: includes GPU availability check

*Enhancements:*
- Improves speed of TFRecord reading by allowing parallel reads
- Allows specifying UMAP dimensions
- Slide reading now fails at first sign of data corruption
- Adds support for Microns-Per-Pixel (MPP) data stored in TIF EXIF (Tag 65326)
- Adds logging levels to SlideflowProject.FLAGS for cleaner logging/verbosity settings

*Bug fixes:*
- Tensorboard training callback disabled due to bug with CUDA 11 / TF 2.4
- Bug fix in tfrecords.example_to_image() (dtype conversion was happening before image standardization)
- Fixes broken normalizer (tf.int8 -> tf.int32)
- Updates QuPath script to support version 0.2.X (standard script). retains 0.1.X version as legacy script
- Fixes bug in Seaborn plot, which now no longer supports display of R2 on the plot
- Logging level configurations should now follow across processes
- Fixes some improper TFRecord and slide warnings when filters areu sed
- Fixes Tkinter/matplotlib canvas error during save_fig()
- Fixed improper TFRecord warning when a sf.io.Dataset object is used with tile_px/tile_um = None
- Fixed segmentation fault during tile extraction
- Fixed bug in tile extraction when isntalled as wheel

*Requirements:*
- Increased Tensorflow version requirements to >= 2.3, with support for TF 2.4
- Requires CPLEX binary, and cplex and cvxpy python modules for k-fold-preserved-site separation

_New features:_
- Can specify project directory in summarize script
- in get_training_and_validation_tfrecords, new argument "read_only" prevents accidental k-fold plans from being written if the validations plans are merely being queried
- new tfrecords.get_tfrecords_from_model_manifest() function that returns list of either training or validation tfrecords from a given model manifest file
- New force_gpu argument allows manual specification of which GPU to use in SlideflowProject
- In SFP.generate_activations_analytics() and SFP.generate_mosaic(), the umap_cache and activations_cache may be explicitly set
- SFP.generate_mosaic() can now display predictions as the output class on the 2D umap
- SFP.generate_mosaic() now accepts mosaic_filename and umap_filename as arguments to explicitly name the generated mosaic map & UMAPs
- SFP.generate_mosaic() now accepts map_centroid as an argument; if True, only the centroid tiles for each slide will be mapped on the UMAP & corresponding mosaic.
- SFP.generate_mosaic() now accepts a colormap for the 2D UMAP via argument cmap
- SFP.generate_mosaic_from_predictions now requires "x" and "y" arguments, to specify which outcome categories should correspond to UMAP-x and UMAP-y
- ActivationsVisualizer can have caching disabled through argument use_activations_cache
- New function AV.slide_tile_dict() which returns dictionary mapping slide names to a list of node activations (one list of node activations corresponding to each tile within the slide)
- Mosaic now accepts argument relative_size; if true, will size each tile relative to the number of points within that grid space
- New argument tile_select helps specify behavior of how to choose which tile for display, when multiple image tiles map to a given Mosaic grid:
- If 'nearest' (default), the tile/point nearest to the Mosaic grid center will be chosen for display
- If 'centroid', then the argument 'tile_meta' -- a dictionary mapping slides to a a list of some variables (list of length = number of tiles) -- is used to calculcate the centroid tile for all eligible tiles in the given Mosaic grid space. This could be used, for example, to pass penultimate activations via tile_meta, and for each grid space, to display centroid tiles with respect to these activations
- Mosaic.save() now saves to the specified filename
- New function TFRecordUMAP.calculate_from_centroid() allows for the display of centroid tiles for each slide using a given dictionary ("slide_activations") mapping slide names to a list of activations for the centroid tile. "tile_indices" is a dict mapping slide names to the index of the centroid tile.
- TFRecordUMAP.save_2d_plot() now accepts a colormap via arg cmap
- statistics.calculate_centroid allows for calculating centroid indices for each slide via the dictionary slide_node_dict (as provided by ActivationsVisualizer)
- statistics.get_centroid_index() finds the centroid index of a given array of arrays
- SFP.generate_activations_analytics() replaces boolean argument "export_csv" with "activations_export" string; if provided, will export CSV activations to the provided filename
- Same functionality as above to SFP.generate_mosaic(); also with argument "umap_export"
- In ActivationsVisualizer, there is now no default location for activations CSV export; it must be explicitly provided
- New TFRecordUMAP.export_to_csv()
- sf.models.HyperParameters object is now used to detect model_type, rather than using what is stored in hyperparameters.json under "model_type"
- Hyperparameter validation checks now happens within the HP object, instead of the SlideflowProject
- Heatmap generator will now skip already-completed heatmaps instead of overwriting
- Manifest automatically updates after extracting tiles
- Mosaic can now show predictions and restrict predictions to a subset of outcomes
- Mosaic/UMAP can now show linear predictions
- umaps are now created from either TFRecord.from_precalculated() or TFRecord.from_activations()
- New SFP.generate_thumbnails() function
- New AV.get_slide_level_predictions() function
- calculate_umap() removed from AV; now these functions are performed in TFRecord.from_activations()
- Heatmaps can now skip thumbnail generation to save time
- Heatmaps can now be provided a colormap function for manually specifying display parameters
- Mosaic now saves report of which tfrecords/tiles were used
- New sfutil.read_predictions_from_csv()
- Added support for NASNetLarge
- Added grayscale normalization for mosaic maps

_Code improvements:_
- Reduced verbosity: getting a TFRecord/slide dataset within a given SlideflowProject can now be accomplished with SFP.get_dataset()
- Better error handling in get_tfrecord_by_index, in case a TFRecord is queried for an index it does not have
- In ActivationsVisualizer: logits_dict and activations_dict are now both stored in the same pickle cache file
- AV.generate_mosaic() function now removed; Mosaics are now generated directly from TFRecordUMAP
- Mosaic generator overhaul
- TFRecordUMAP overhaul
- Improved error handling
- Mosaic error handling improvements

_Bug fixes:_
- Fix in automatic slide annotations association
- Fix in map caching
- Fix in UMAP 2D plotting where colors would be reused if >8 categories
- Fix in Mosaic.focus()
- Fix in linear model training
- Fix in 3D node plot generation

_New Features_
- ProgressBar can display text at beginning of bar
- New merge_split_tfrecords function can be used to merge TFRecords with the same name in different folders (e.g. if TFRecords were split for training/validation)
- can manually specify how many steps of validation to perform during training with validation_steps argument
- can resume training at a given starting_epoch
- summarize.py can now show tile-level or slide-level ROCs
- option in extract_tiles() to skip slides with missing ROIs
- added JPG support for slides
- TFRecordUMAP can now show predicted categories (must provide sldie_logits_dict to calculate_from_nodes() and show_prediction=True to save_2d_plot())
- Tiles can now be extracted directly into TFRecords, saving on disk writes (defaults to this behavior unless "save_tiles" is true in SFP.extract_tiles)

_Enhancements_
- Tile extraction will now skip slides that already have TFRecords/tiles generated
- extract_tiles() now accepts "roi_method" argument; if 'inside', extracts tiles within ROIs; if 'outside', will only extract tiles outside ROIs
- Logging improvements
- Streamlined dataset manifests
- Tile extraction from slides now occurs using quality=100% to avoid compression artifacts
- Can provide dataset name to extract_tiles
- Slides can now cache thumbnails in specified folder (default is project root)
- can run heatmap generator in single thread mode
- heatmap_generator() now accepts interpolation, whole_slide, show_roi, and thumb_folder as arguments
- improved heatmap logging & progress bar
- improved Heatmap performance (thumbnail calculation now happens asynchronously during model predictions)
- better logging of validation checking strategy during model training
- Overhaul of ProgressBar with new support for ETA
- Improved tile extraction progress bar monitoring
- Auto detection and removal of corrupt TFRecords during validation at project initiation

_Code cleanup_
- Improvements to Slide tile extraction and generators, with improved readability/comprehension and support for tfrecord splitting
- Removed redundant "generate_tfrecords" argument in extract_tiles
- Improved/cleaned up tile extraction code in SlideReader class
- simplification of sfutil.get_slide_paths
- GPU autoselection occurs at project initialization; no need to manually call SFP.autoselect_gpu()
- Mosaic generator is no longer process encapsulated
- Updated requirements.txt, specified pyvips version requirements

_Bug fixes_
- Fixed false GPU selection warnings
- Fixed bugs in datasets and tile extraction
- Fixed bug in automatic slide annotations assocation
- Fixed bug during training when linear models are attempted to be used with accuracy-based early stopping
- Fixed rare bug in Mosaic if only one point is in a single grid