Cryodrgn

New Features
- `cryodrgn direct_traversal`, a tool for interpolating a path in the latent conformation space connecting two points in a direct line
- making the package available for installation using the TestPyPI distribution service:
pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ "cryodrgn<=3.3.0" --pre
- this will let us to make both the development and stable versions of the package available for easy download using `pip`, as opposed to having to use `git clone` for the former.

Improving Existing Features
- updated interfaces for `cryodrgn graph_traversal` and `cryodrgn pc_traversal` so that the arguments, argument formats, and help docstrings between all three traversal methods are as clear and consistent as possible
- `--ind` in `direct_traversal` is replaced with `--anchors` as in `graph_traversal`, allowing both to take a list of integers as well as files containing lists of integers
- `-o` now also has a more verbose alias `--outtxt` in `graph_traversal` and `direct_traversal`; updating its behavior in `graph` to save the latent space co-ordinates and updating `--outind` to save path indices; similarly verbose alias `--outdir` in `pc_traversal`
- `-o` now also has a default value that is used when the flag is given with no argument across all three traversal commands to mean that we want to save output but don't have a file name
- when `-o` is *not* given, all three commands display a prettier log message to screen with traversal output
- epoch numbers are automatically updated to the epoch used in `cryodrgn analyze` in copied-over demo notebooks
- improving package status badges shown in GitHub README: available versions, PyPI downloads

Addressing Issues and Bugs
- adding the `--datadir` flag to `cryodrgn abinit_homo`, addressing an oversight that complicated using `.star` files with this command (343)
- fixing bugs and other issues found in our demonstration Jupyter notebooks (363)
- `analysis.plot_projections()` doesn't fail if of imgs is two or one
- makeover of GitHub deployment workflow actions to fix errors and simplify release infrastructure
- `master`->`main` branch names
- removing remaining errors in continuous integration testing action so that it is again a useful tool for checking pull requests and protecting our `main` branch, especially with the now expanded coverage of notebooks, traversal, etc.
- last `pytest` bug fixed (`n=tilts` in `eval_images`)
- switching off `pyright` for now as type checks are not essential
- leftover `pre-commit` formatting issue in `commands.filter`
- more lightweight Docs action by only releasing new Sphinx autodocs version when a new version tag is pushed — not nuking these docs for now (350)
- new Beta Release action for automatically deploying a release to TestPyPI whenever a new version tag is pushed
- existing Release action still not working (needs updated credentials) but is now also only deployed automatically when a new version tag is pushed
- fixing `ntilts=10` default behaviour bug in `eval_images` which was activating tilt mode
- officially removing support for outdated Python versions 3.7 and 3.8 (already implicitly not supported)

Testing
- renaming `test_quick` to `test_integration` and improving the coverage of the reconstruction pipeline integration tests contained therein
- adding integration tests for Jupyter demonstration notebooks to check that they execute successfully upon running `cryodrgn analyze` after `cryodrgn train_vae` with different types of inputs and parameters
- expanded fidelity and unit tests for all three traversal commands
- adding `CODEOWNERS` letting michal-g be e.g. automatically added to new issues

New Features

- introducing `cryodrgn_utils clean`, a new tool for removing extraneous output files from completed experiments (297)
- `backproject_voxel` now produces half-maps and a half-map FSC by default (329)
- creating `cryodrgn_utils fsc` from the `fsc` analysis script for calculating Fourier shell correlations between two `.mrc` volume files, and likewise `cryodrgn_utils plot_fsc` based on `plotfsc`; making the latter available through the former using `-p`
- creating `cryodrgn_utils gen_mask` based on `cryoem_tools.gen_mask.py`, now with reparametrization in Angstroms

Addressing Issues and Bugs

- fixing 358 and improving the I/O interface in both `cryodrgn_utils flip_hand` and `cryodrgn_utils invert_contrast` so that the name of the output file and any parent directories are created automatically, with more unit tests for each
- making `write_star` use RELION 3.1 format by default with optics groups generated from image size, pixel size, voltage, spherical aberration, and amplitude contrast; `-relion30` to use old format (324)
- updating install setup to prevent use of Python 3.11 (306)
- `abinit_homo` now saves a `config.yaml` with a summary of parameters used, like `abinit_het`, `train_vae`, and `train_nn`
- fixing `filter_star` to accept tilt series as well (335)
- fixing `affinity` bug in `analyze_landscape` (345)
- fixing beta value bug in `train_vae` (356)
- removing references to `scipy.ndimage.morphology` which is deprecated
- fixing `dtype=object` warning message in `TiltSeries.parse_particle_tilt()`

User Interface

- cleaner implementation of command-line interface, defining both `cryodrgn` and `cryodrgn_utils` commands in one file `cryodrgn/command_line.py`, and removing e.g. manually defined lists of modules with commands in them
- better doc strings with some usage examples for commands (e.g. `cryodrgn abinit_homo -h`), with module-level doc strings being included explicitly in the automatically generated help screen

Testing

- using `conftest.py` to define a new setup/teardown routine for experiment output directories created by tests
- writing new tests for `abinit` and `train` methods by applying these routines
- fixing `test_dataset` to account for changes within `make_dataloader`
- updating unit tests that use `argparse.ArgumentParser()` directly for commands in which the `__main__` method was removed
- updating tests for new and updated commands `fsc`, `clean`, `gen_mask`, etc.

We have introduced a number of small fixes and feature updates since our last release `v3.0.1-beta`:

- creating a new interactive command-line interface `cryodrgn filter` as an alternative to the buggy interface in the Jupyter filtering notebook (https://github.com/ml-struct-bio/cryodrgn/issues/323)
- making `cryodrgn analyze` produce a plot of the learning curve (https://github.com/ml-struct-bio/cryodrgn/issues/304)
- adding cell in `cryoDRGN_filtering` jupyter notebook returned by `cryodrgn analyze` for filtering by UMAP/PC values (https://github.com/ml-struct-bio/cryodrgn/pull/313)
- fixing bugs with deprecated signatures in plotting functions (https://github.com/ml-struct-bio/cryodrgn/issues/322) and numpy dependency versioning (https://github.com/ml-struct-bio/cryodrgn/issues/318)

This is a patch release to address 312 and 310 found in v3.0.0-beta by updating the list of dependencies and making some updates to the analysis Jupyter notebooks produced by experiments.

The official [cryoDRGN-ET](https://www.biorxiv.org/content/10.1101/2023.08.18.553799v1) release for heterogeneous subtomogram analysis.

- [NEW] Heterogeneous reconstruction of subtomograms. See documentation [on gitbook](https://ez-lab.gitbook.io/cryodrgn/)
- [NEW] cryodrgn `direct_traversal` for making movies
- Updated `cryodrgn backproject_voxel` for voxel-based homogeneous reconstruction
- Major refactor of dataset loading for handling large datasets

- Model configuration files are now saved as human-readable config.yaml files (https://github.com/zhonge/cryodrgn/issues/235)
- Fix machine stamp in output .mrc files for better compatibility with downstream tools (https://github.com/zhonge/cryodrgn/pull/260)
- Better documentation of help flags in ab initio reconstruction tools (https://github.com/zhonge/cryodrgn/issues/258)
- [FIX] By default, window images in cryodrgn abinit_homo (now consistent with other reconstruction tools) (https://github.com/zhonge/cryodrgn/issues/258)
- [FIX] Reduce memory usage when using --preprocessed and --ind (https://github.com/zhonge/cryodrgn/pull/272)
- Updated functionality in `cryodrgn_utils filter_star`
- Upcoming in the next minor version (v2.4):

- We are working on a major refactor of data loading for handling large datasets. This will entail an API change for the `mrc.py` library module