Ark-analysis

You may download the latest release with:

shell
git clone -b v0.7.1 https://github.com/angelolab/ark-analysis.git


What's new?


Pixie updates

In certain datasets, it is impossible to assign all unique SOM clusters. This was previously prohibitive because error checks required all unique SOM clusters to have at least 1 pixel/cell assigned to it. With this update, this is no longer the case, and users may continue in spite of this.

To prevent users from generating duplicate `cluster_id` columns by accidentally running the mask generation function multiple times, a safeguard has been added.

An ongoing issue with visualizing unordered cluster IDs using the existing plotting functions has been resolved. Regardless of the spacing and ordering of the IDs internally, they are mapped to the correct color in the overlays.

Update the Mantis directory conversion process to turn off erosion and use an updated `cluster_id` column to generate correct and clearer overlays.

Lab members often have multiple resolutions of clustering analysis that need to be resolved. The post clustering notebook has been updated to allow users to define new cluster groups based on user-set mappings.

Spatial Analysis Updates

Our repo is gradually transitioning our internal single-cell data representation format to AnnData. We need a way to convert our existing format to be scverse compatible. The AnnData conversion notebook is the first step of this large-scale conversion process.

Prevent incorrect ordering of kmeans clusters by adding control params.

Correct a bug in fiber segmentation that causes metric averages for a fiber to be the same at the first metric value.

ezSegmenter Updates

`merge_masks.py` took a while to run on very long datasets with thousands of FOVs. This has been optimized by only merging cells that are in the areas surrounding objects.

Due to requests from lab members, we've added a generalized masking script to create masks from single or composite channel tiffs. This is implemented using the channel masking function from the TNBC repo.

Update image loading process to use `np.int32` for compatibility of composite array addition and subtraction.

With the addition of various types of masks, it's important for the cell table generation notebook to differentiate only `_whole_cell.tiff` and `_nuclear.tiff` files if desired. Support for this has been added. Also prevents FOVs with duplicate substrings from breaking the cell table generation process by requiring specific names.

Optimize the generation of nuclear masks to prevent repetitive computation.

Deployment + Environment Updates

Our deployment stack requires enhanced security protocols, including OIDC and 2FA. These have been updated to allow for future releases.

Downgraded Python to 3.10 while waiting for `scikit-image` support for 3.11.

---

🚀 Features

- Generalized masking camisowers (1092)
- Don't require user to have all pixel SOM clusters represented during assignment alex-l-kong (1124)
- 1119 reducing time to merge in merge maskspy wi ez segmenter bryjcannon (1123)
- Add multiple clustering resolution functionality camisowers (1095)
- `AnnData` Conversion Notebook srivarra (1079)

🐛 Bug Fixes

- Fix kmeans notebook mask generation camisowers (1130)
- Manual mask names camisowers (1126)
- Fiber stats bugs camisowers (1128)
- Fix color masking in Pixie camisowers (1127)
- Changes process lists search for matching fov and mask names to use a… bryjcannon (1104)
- 1100 empty composites masks in ez segmenter notebook bryjcannon (1101)
- Don't generate duplicate `cluster_id` columns if `generate_and_save_cluster_masks` run twice alex-l-kong (1110)
- Only load whole cell masks if `nuclear_counts=False` in `generate_cell_table` alex-l-kong (1107)
- General CI Updates srivarra (1114)
- Fix cell cluster visualization in Mantis camisowers (1083)

🧰 Maintenance

- Change python version in conda env file srivarra (1134)
- Next Release 0.7.1 srivarra (1144)
- Updated GitHub Workflows srivarra (1139)


alex-l-kong, bryjcannon, camisowers and srivarra

You may download the latest release with:

shell
git clone -b v0.7.0 https://github.com/angelolab/ark-analysis.git


What's new?


ezSegmenter support

Starting with `v0.7.0`, support for ezSegmenter masks has been added. The Jupyter notebook for this can be found in `templates/ez_segmenter.ipynb`.

**Please note that this feature may experience frequent updates. Updates may be required for Pixie integration and further downstream analysis.**

Updated mask visualization

`v0.7.0` introduces significant updates to the visualization process of masks. Shared functionality (ex. colormap generation, plotting clusters) has been abstracted away into internal functions and dataclasses. Erosion support has been added to decrease the size of the borders between cells, and `interpolation` has been turned off to prevent interpolation of cell borders.

In the past, users reported problems of incorrectly sorted colors in the cmap file. This has been addressed. Additionally, users may now specify a `dpi` and `fig_size`, as well as multi-compatibility support for saving `matplotlib` figures.

Mantis saving updates

Starting with `v0.7.0`, only a single top-level Mantis directory will be created, and support for subfolders (via the `img_sub_folder` param) has been added.

Pixie updates

During Pixie SOM assignment for pixel clustering, some users reported it taking unusually long. This was caused by the batch size for cluster assignment being too small. With this update, the batch size can now be set by the user to speed the process up, and defaults to a more reasonable value.

Additionally, in some Pixie applications, certain cells won't have any pixel clusters associated with them. These cells now get dropped prior to running the cell clustering portion.

Previously, the `overwrite` flag did not work for certain functions because it wasn't propagated correctly. This has also been addressed by `v0.7.0`.

Versioning Updates

`ark-analysis` now uses `scikit-image==0.19.3` and `alpineer==v0.1.10`.

Licensing

`ark-analysis` has migrated from the Modified Apache 2.0 License to the MIT license. This was made due to the cell segmentation algorithm, Mesmer, having a more restrictive license.

---

🚀 Features

- ezSegmentation srivarra ([958](https://github.com/angelolab/ark-analysis/pull/958))
- Mask Generation Refactoring srivarra ([1030](https://github.com/angelolab/ark-analysis/pull/1030))
- Include `img_sub_folder` as parameter in Mantis calls alex-l-kong ([1050](https://github.com/angelolab/ark-analysis/pull/1050))
- Remove cells that don't have any pixel clusters expressed prior to Pixie alex-l-kong ([1051](https://github.com/angelolab/ark-analysis/pull/1051))
- Single mantis directory camisowers ([1061](https://github.com/angelolab/ark-analysis/pull/1061))
- Allow user to dynamically set batch size of pixel SOM assignments alex-l-kong ([1069](https://github.com/angelolab/ark-analysis/pull/1069))

🐛 Bug Fixes

- Make sure overwrite functionality for pixel clustering propagates into helper functions alex-l-kong ([1058](https://github.com/angelolab/ark-analysis/pull/1058))
- Adjustments to Mass Plotting Workflow srivarra ([1067](https://github.com/angelolab/ark-analysis/pull/1067))

🧰 Maintenance

- Update README from v0.6.5 to v0.6.6 alex-l-kong ([1048](https://github.com/angelolab/ark-analysis/pull/1048))
- Update HuggingFace to main alex-l-kong ([1053](https://github.com/angelolab/ark-analysis/pull/1053))
- Pin to scikit-image to v0.19.3 camisowers ([1060](https://github.com/angelolab/ark-analysis/pull/1060))
- Alpineer v0.1.10 camisowers ([1057](https://github.com/angelolab/ark-analysis/pull/1057))
alex-l-kong, camisowers and srivarra
- MIT license srivarra ([1080](https://github.com/angelolab/ark-analysis/pull/1080))
- Bump to v0.7.0 alex-l-kong ([1093](https://github.com/angelolab/ark-analysis/pull/1093))

🧰 Maintenance

- Update README with Pixie reference cliu72 ([1042](https://github.com/angelolab/ark-analysis/pull/1042))

🚀 Features

- Update progress bars camisowers ([1031](https://github.com/angelolab/ark-analysis/pull/1031))
- Optional normalization for generic\_cell\_clustering.ipynb camisowers ([1027](https://github.com/angelolab/ark-analysis/pull/1027))
- Deepcell upload loop cleaning camisowers ([1023](https://github.com/angelolab/ark-analysis/pull/1023))
- Add channel normalization and pixel thresholding back alex-l-kong ([985](https://github.com/angelolab/ark-analysis/pull/985))

🐛 Bug Fixes

- Update progress bars camisowers ([1031](https://github.com/angelolab/ark-analysis/pull/1031))
alex-l-kong and camisowers

You may download the latest release with:

shell
git clone -b v0.6.4 https://github.com/angelolab/ark-analysis.git


What's new in Ark Analysis?


Python Compatibility

**The minimum supported Python is now version 3.9.**

We have an Anaconda `environment.yml` file so users do not need to use the Docker Image if they wish. This will create a Python 3.11 environment. Here is how to use it.

shell
cd ark-analysis
conda env create -f environment.yml
conda activate ark_env


Pixie Improvements

Pixie now features improved fallback protection during the clustering step, preventing duplicate cluster names and the ability to save cluster masks as colored images.

K-Means Improvements

K-means now has improved plots, default parameters, and has received minor improvements.

Cell Neighbors Analysis

We have a new analysis pipeline containing two independent features: analyzing the homogeneity/diversity of the neighbors surrounding each of the cells and analyzing the proximity/distance between cell phenotypes in samples.

Fiber Segmentation

Fiber segmentation acquires more properties and has received optimizations regarding extracting distinct fiber objects.

---

🚀 Features

- Add progress bars in Cell Neighbors Analysis notebook camisowers ([1022](https://github.com/angelolab/ark-analysis/pull/1022))
- Drop 3rd dimension in segmentation masks camisowers ([1018](https://github.com/angelolab/ark-analysis/pull/1018))
- Mixing score default thresholds camisowers ([1015](https://github.com/angelolab/ark-analysis/pull/1015))
- Include all fiber properties in stats generation camisowers ([1014](https://github.com/angelolab/ark-analysis/pull/1014))
- Save Colored Pixie Masks srivarra ([1007](https://github.com/angelolab/ark-analysis/pull/1007))
- Initial commit of phenotype by neighbour analysis khoulahan ([988](https://github.com/angelolab/ark-analysis/pull/988))
- Cell neighborhood diversity camisowers ([989](https://github.com/angelolab/ark-analysis/pull/989))
- Downstream fiber analysis updates camisowers ([974](https://github.com/angelolab/ark-analysis/pull/974))

🐛 Bug Fixes

- Maximize Available GitHub Storage Space srivarra ([1025](https://github.com/angelolab/ark-analysis/pull/1025))
- Allow FOVs to contain various maximum clusters srivarra ([1021](https://github.com/angelolab/ark-analysis/pull/1021))
- Pixie percentile saving camisowers ([1020](https://github.com/angelolab/ark-analysis/pull/1020))
- Fix error from zero fibers in an image camisowers ([1011](https://github.com/angelolab/ark-analysis/pull/1011))
- Fix segmentation rescale\_factor camisowers ([1008](https://github.com/angelolab/ark-analysis/pull/1008))
- Add fallback protection to Pixie cell clustering alex-l-kong ([999](https://github.com/angelolab/ark-analysis/pull/999))
- Prevent duplicate renamed metaclusters in Pixie remapping alex-l-kong ([993](https://github.com/angelolab/ark-analysis/pull/993))
- Pixel Clustering - Run without segmentaiton files srivarra ([998](https://github.com/angelolab/ark-analysis/pull/998))
- Mantis Directory with no segmentation files srivarra ([994](https://github.com/angelolab/ark-analysis/pull/994))
- Extended Attributes on macOS srivarra ([973](https://github.com/angelolab/ark-analysis/pull/973))
- Mutiple tiled stitching tweaks camisowers ([981](https://github.com/angelolab/ark-analysis/pull/981))
- Fix pixel and cell cluster mask generation index for rectangular images alex-l-kong ([983](https://github.com/angelolab/ark-analysis/pull/983))
- Kmeans plotting cell border fix camisowers ([978](https://github.com/angelolab/ark-analysis/pull/978))
- Set Kmeans parameter to prior default camisowers ([977](https://github.com/angelolab/ark-analysis/pull/977))
- Kmeans notebook minor fixes camisowers ([966](https://github.com/angelolab/ark-analysis/pull/966))
- Stitching Large Images srivarra ([939](https://github.com/angelolab/ark-analysis/pull/939))

🧰 Maintenance

- GitHub Actions, Python 3.9+ Support, Jupyterlab Extensions srivarra ([991](https://github.com/angelolab/ark-analysis/pull/991))
- Added Conda environment file srivarra ([1006](https://github.com/angelolab/ark-analysis/pull/1006))
- Skip spatial LDA visualization tests due to Numpy error alex-l-kong ([984](https://github.com/angelolab/ark-analysis/pull/984))
alex-l-kong, camisowers, khoulahan and srivarra

- Update citation information ngreenwald ([961](https://github.com/angelolab/ark-analysis/pull/961))
ngreenwald