Tomotwin-cryoet

With TomoTwin v0.9.0 we introduce a new target manager for napari. The documentation of the clustering workflow has been updated accordingly. Also, locate is now parallelized using dask and is now much faster :-)

What's Changed
* Update rapids to 23.10 by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/48
* Update pytorch-metric-learning by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/87
* Locate is now using dask by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/90
* Remove tomotwin from env file by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/91
* Switch to SPDX-License-Identifiers by kephale in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/96
* Fast locate, new rapids and updated tutorial by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/89

New Contributors
* kephale made their first contribution in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/96

**Full Changelog**: https://github.com/MPI-Dortmund/tomotwin-cryoet/compare/v0.8.0...v0.9.0

What's Changed

This release mainly improves the user experience when working with the clustering workflow. The changes you will notice are:

- BREAKING CHANGE: When calculating a umap, no label mask is created anymore.Instead, this map is now computed in the background within napari and is transparent to the user. This also means that you don't have to select a label layer anymore when loading a umap (PR 82 MPI-Dortmund/napari-tomotwin22). However, this is a breaking change, so you need to recalculate your umaps to take advantage of it.
- Napari no longer freezes when loading umaps and shows a progress bar while loading (MPI-Dortmund/napari-tomotwin20)
- When selecting clusters in the umap, there is a time delay between the selection and the actual highlighting. Now this process is 2x-3x faster (PR BiAPoL/napari-clusters-plotter283 BiAPoL/napari-clusters-plotter282)
- When saving cluster targets, the appropriate embedding file is automatically selected if possible (PR MPI-Dortmund/napari-tomotwin26). This means that the user does not have to remember which embedding belongs to which umap.
- Saving cluster targets (medoids) is now 5x faster (https://github.com/MPI-Dortmund/napari-tomotwin/pull/23).

Minor changes

* Add median filter strategy by GavinR1 in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/76 and https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/79
* Stop making `tomotwin_tools` crash on non gpu machines. by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/78
* Add short training tutorial by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/81


**Full Changelog**: https://github.com/MPI-Dortmund/tomotwin-cryoet/compare/0.7.0...v0.8.0

What's Changed

Breaking changes:

* Write embeddings metadata + positions into umap attributes thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/82
* Remove calculation of label mask by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/82

The above change have the effect, that already calculated umaps to not work anymore. However, you can easily recreate them with `tomotwin_tools.py umap`. With argument `-m` you can even provide the old fitted model. In that way you will get the same umap as before.

Minor changes

* Add median filter strategy by GavinR1 in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/76 and https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/79
* Stop making `tomotwin_tools` crash on non gpu machines. by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/78
* Add short training tutorial by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/81



**Full Changelog**: https://github.com/MPI-Dortmund/tomotwin-cryoet/compare/0.7.0...v0.8.0b1

What's Changed

With this release we add new tools for advanced [strategies](https://tomotwin-cryoet.readthedocs.io/en/stable/strategies/strategy_overview.html) that describe how to speed up the calculation of embeddings, potentially improve their umaps, and refine references/targets!

Improve reference/targets with umaps ([Strategy 1](https://tomotwin-cryoet.readthedocs.io/en/stable/strategies/strategy_overview.html#strategy-1-refinement-of-references-targets-using-umaps))

Add new umap and filter options by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/64

<img src='https://github.com/MPI-Dortmund/tomotwin-cryoet/assets/1113977/45ab662c-b187-4c8b-a88d-0ea4bf93b0f5' width='400'>

Faster embeddings with median filtering ([Strategy 2](https://tomotwin-cryoet.readthedocs.io/en/stable/strategies/strategy_overview.html#strategy-2-speedup-and-improvement-of-embeddings-using-median-filtering))

Add fast median mask estimation by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/66

<img src='https://github.com/MPI-Dortmund/tomotwin-cryoet/assets/1113977/b7cf33cb-fc4c-48bc-b37a-1b835e2fdfa7' width='400'>

It can dramatically speed up the embedding step compared to using no masking, and it also outperforms the previous method based on intensity heuristics:

<img src='https://github.com/MPI-Dortmund/tomotwin-cryoet/assets/1113977/d7d26225-d80e-4289-be0d-bd9b582b6f73' width='400'>



Other

* increase timeout of the dataloader to 180 seconds. by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/57
* `--save_after_improvement` saves models after each epoch by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/60
* Check & correct for duplicate embedded subvolumes by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/61
* Fix padding when creating heatmaps during locate with `--write_heatmaps` by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/65

Documentation
* Add strategies pages to the documentation and link them by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/67 https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/68 https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/69 https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/70 https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/71


**Full Changelog**: https://github.com/MPI-Dortmund/tomotwin-cryoet/compare/v0.6.1...0.7.0

What's Changed
* Use random port for distributed processing by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/54
* Use stable pytorch 2.1 by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/52
* Fix read the docs building by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/55
* Fix 'too many open files' problem by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/56


**Full Changelog**: https://github.com/MPI-Dortmund/tomotwin-cryoet/compare/v0.6.0.3...v0.6.1

What's Changed

* `tomotwim_embed.py` is now 1.6x faster and linearly scales across multiple GPUs by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/40
* Exploiting new the compile option of the latest pytorch 2.1 nightly build.
* Internally DistributedDataParallel is used instead of DataParallel
* Unpin pytorch version by thorstenwagner in https://github.com/MPI-Dortmund/tomotwin-cryoet/pull/46

**Full Changelog**: https://github.com/MPI-Dortmund/tomotwin-cryoet/compare/v0.5.1...v0.6.0.1