Cryodrgn

This is **a patch release to address several bugs and issues** that have come to our attention:

- adding `--micrograph-files` argument to `filter_star` to create separate output files for each `_rlmMicroGraphName` encountered in the file
- `--ind` with `--encode-mode=tilt` wasn’t working in the case where all particles had the same number of tilts due to `dtype=object` patch introduced earlier
- fixed by storing particle→tilt index produced by `TiltSeriesData.parse_particle_tilt()` as a list instead of an array; this is more robust in general and all downstream cases are agnostic (see tests below)
- `backproject_voxel` was producing errors when trying to calculate threshold FSC values due to deprecated code used to parse FSC matrix (371)
- fixed by copying over code already used in `commands/fsc`
- `train_nn` and `train_vae` would error out if inputs were not divisible by 8 when using AMP optimization (e.g. 353)
- a warning here suffices as AMP optimization is the default and this is frustrating for many users
- better error message when CTF file is missing from `write_star` inputs
- better error message when `backproject_voxel` output is not `.mrc`
- bug in `ET_viz` notebook when `--ind` not specified caused by inconsistent definition of `ind0`
- bug in filtering notebook caused by using `ind=ind_orig` when loading dataset and then trying to filter again (363)
- `ZeroDivisionError` bugs in all notebooks when using small training datasets
- updating template analysis notebooks to use the given `kmeans` value in the copied-over notebook, similarly to out auto-updating of notebook epoch numbers

In addition to making the required fixes, we have **expanded and improved our deployment tests** to cover these cases and close some gaps in our testing coverage:

- adding a stand-alone test of backprojection under `test_reconstruct` applying both `.mrcs` and `.star` inputs
- more testing of `train_nn` cases with different `--amp`, `--batch-size`, `--poses` values
- fixing `check=True` issue in `utils.run_command()` that was allowing tests of backprojection to fail silently
- new deployment task schedule
- the `main` deployment task has been split into `tests` and `style` for tests of code integrity and code linting respectively
- run `tests` and `style` along with `beta-release` any time a patch version tag `[0-9]+\.[0-9]+\.[0-9]+-*` is pushed to any branch to trigger a verified upload to TestPyPI
- also run `tests` and `style` for any push to `develop` branch to allow for testing before beta release
- update `release` to only run when a stable version tag (`^[0-9]+\.[0-9]+\.[0-9]+$`) is pushed to `main`
- `tests` and `style` run on any push to `main` to allow for testing prior to release

**Other changes include:**

- applying `tmpdir_factory` to improve the `train_dir` and `AbinitioDir` fixtures used in tests with more robust setup and teardowns
- CodeFactor badge and nicer TestPyPI installation command in `README`
- dynamic update of plotted point sizes in `cryoDRGN_filtering.ipynb` interactive filtering widget, useful for smaller datasets for which the default is too small for points to be seen
- using `plt.close()` after `analyze` plotting for better memory management

New Features
- `cryodrgn direct_traversal`, a tool for interpolating a path in the latent conformation space connecting two points in a direct line
- making the package available for installation using the TestPyPI distribution service:
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ "cryodrgn<=3.3.0" --pre
- this will let us to make both the development and stable versions of the package available for easy download using `pip`, as opposed to having to use `git clone` for the former.

Improving Existing Features
- updated interfaces for `cryodrgn graph_traversal` and `cryodrgn pc_traversal` so that the arguments, argument formats, and help docstrings between all three traversal methods are as clear and consistent as possible
- `--ind` in `direct_traversal` is replaced with `--anchors` as in `graph_traversal`, allowing both to take a list of integers as well as files containing lists of integers
- `-o` now also has a more verbose alias `--outtxt` in `graph_traversal` and `direct_traversal`; updating its behavior in `graph` to save the latent space co-ordinates and updating `--outind` to save path indices; similarly verbose alias `--outdir` in `pc_traversal`
- `-o` now also has a default value that is used when the flag is given with no argument across all three traversal commands to mean that we want to save output but don't have a file name
- when `-o` is *not* given, all three commands display a prettier log message to screen with traversal output
- epoch numbers are automatically updated to the epoch used in `cryodrgn analyze` in copied-over demo notebooks
- improving package status badges shown in GitHub README: available versions, PyPI downloads

Addressing Issues and Bugs
- adding the `--datadir` flag to `cryodrgn abinit_homo`, addressing an oversight that complicated using `.star` files with this command (343)
- fixing bugs and other issues found in our demonstration Jupyter notebooks (363)
- `analysis.plot_projections()` doesn't fail if of imgs is two or one
- makeover of GitHub deployment workflow actions to fix errors and simplify release infrastructure
- `master`->`main` branch names
- removing remaining errors in continuous integration testing action so that it is again a useful tool for checking pull requests and protecting our `main` branch, especially with the now expanded coverage of notebooks, traversal, etc.
- last `pytest` bug fixed (`n=tilts` in `eval_images`)
- switching off `pyright` for now as type checks are not essential
- leftover `pre-commit` formatting issue in `commands.filter`
- more lightweight Docs action by only releasing new Sphinx autodocs version when a new version tag is pushed — not nuking these docs for now (350)
- new Beta Release action for automatically deploying a release to TestPyPI whenever a new version tag is pushed
- existing Release action still not working (needs updated credentials) but is now also only deployed automatically when a new version tag is pushed
- fixing `ntilts=10` default behaviour bug in `eval_images` which was activating tilt mode
- officially removing support for outdated Python versions 3.7 and 3.8 (already implicitly not supported)

Testing
- renaming `test_quick` to `test_integration` and improving the coverage of the reconstruction pipeline integration tests contained therein
- adding integration tests for Jupyter demonstration notebooks to check that they execute successfully upon running `cryodrgn analyze` after `cryodrgn train_vae` with different types of inputs and parameters
- expanded fidelity and unit tests for all three traversal commands
- adding `CODEOWNERS` letting michal-g be e.g. automatically added to new issues

New Features

- introducing `cryodrgn_utils clean`, a new tool for removing extraneous output files from completed experiments (297)
- `backproject_voxel` now produces half-maps and a half-map FSC by default (329)
- creating `cryodrgn_utils fsc` from the `fsc` analysis script for calculating Fourier shell correlations between two `.mrc` volume files, and likewise `cryodrgn_utils plot_fsc` based on `plotfsc`; making the latter available through the former using `-p`
- creating `cryodrgn_utils gen_mask` based on `cryoem_tools.gen_mask.py`, now with reparametrization in Angstroms

Addressing Issues and Bugs

- fixing 358 and improving the I/O interface in both `cryodrgn_utils flip_hand` and `cryodrgn_utils invert_contrast` so that the name of the output file and any parent directories are created automatically, with more unit tests for each
- making `write_star` use RELION 3.1 format by default with optics groups generated from image size, pixel size, voltage, spherical aberration, and amplitude contrast; `-relion30` to use old format (324)
- updating install setup to prevent use of Python 3.11 (306)
- `abinit_homo` now saves a `config.yaml` with a summary of parameters used, like `abinit_het`, `train_vae`, and `train_nn`
- fixing `filter_star` to accept tilt series as well (335)
- fixing `affinity` bug in `analyze_landscape` (345)
- fixing beta value bug in `train_vae` (356)
- removing references to `scipy.ndimage.morphology` which is deprecated
- fixing `dtype=object` warning message in `TiltSeries.parse_particle_tilt()`

User Interface

- cleaner implementation of command-line interface, defining both `cryodrgn` and `cryodrgn_utils` commands in one file `cryodrgn/command_line.py`, and removing e.g. manually defined lists of modules with commands in them
- better doc strings with some usage examples for commands (e.g. `cryodrgn abinit_homo -h`), with module-level doc strings being included explicitly in the automatically generated help screen

Testing

- using `conftest.py` to define a new setup/teardown routine for experiment output directories created by tests
- writing new tests for `abinit` and `train` methods by applying these routines
- fixing `test_dataset` to account for changes within `make_dataloader`
- updating unit tests that use `argparse.ArgumentParser()` directly for commands in which the `__main__` method was removed
- updating tests for new and updated commands `fsc`, `clean`, `gen_mask`, etc.

We have introduced a number of small fixes and feature updates since our last release `v3.0.1-beta`:

- creating a new interactive command-line interface `cryodrgn filter` as an alternative to the buggy interface in the Jupyter filtering notebook (https://github.com/ml-struct-bio/cryodrgn/issues/323)
- making `cryodrgn analyze` produce a plot of the learning curve (https://github.com/ml-struct-bio/cryodrgn/issues/304)
- adding cell in `cryoDRGN_filtering` jupyter notebook returned by `cryodrgn analyze` for filtering by UMAP/PC values (https://github.com/ml-struct-bio/cryodrgn/pull/313)
- fixing bugs with deprecated signatures in plotting functions (https://github.com/ml-struct-bio/cryodrgn/issues/322) and numpy dependency versioning (https://github.com/ml-struct-bio/cryodrgn/issues/318)

This is a patch release to address 312 and 310 found in v3.0.0-beta by updating the list of dependencies and making some updates to the analysis Jupyter notebooks produced by experiments.

The official [cryoDRGN-ET](https://www.biorxiv.org/content/10.1101/2023.08.18.553799v1) release for heterogeneous subtomogram analysis.

- [NEW] Heterogeneous reconstruction of subtomograms. See documentation [on gitbook](https://ez-lab.gitbook.io/cryodrgn/)
- [NEW] cryodrgn `direct_traversal` for making movies
- Updated `cryodrgn backproject_voxel` for voxel-based homogeneous reconstruction
- Major refactor of dataset loading for handling large datasets