Loompy

Any layer can be created from sparse matrix

This now works:

python
G = 1000
C = 100
S = sparse.eye(G, C)
with loompy.connect("test.loom") as ds:
ds["layer"] = S


Fixes 66.

Create empty file

`loompy.new()` creates an empty loom file, and returns it as a context manager. The file can then be populated with data. This is especially useful when you're building a dataset incrementally, e.g. by pooling subsets of other datasets:

python
with loompy.new("outfile.loom") as dsout:
for sample in samples:
with loompy.connect(sample) as dsin:
logging.info(f"Appending {sample}.")
dsout.add_columns(ds.layers, col_attrs=dsin.col_attrs, row_attrs=dsin.row_attrs)


As a consequence, `create_append()` is now deprecated.

Fixes 42.

Experimental plotting features

`ds.pandas()` to return a Pandas DataFrame for the whole matrix, or selected parts. The interface is intended to simplify plotting, since many plotting libraries take Pandas as input. The interface is experimental, and e.g. lacks support for layers.

`ds.embedding()` and `ds.embeddings()` to find attributes that are >1-dimensional. Again intended to support plotting, by making it easy to find the X/Y coordinates without knowing if they are stored as `TSNE`, `PCA` or something else. The interface is liable to change (in particular, I'd like to find a shorter name than "embedding").

Two useful colormaps: `loompy.zviridis` is a zero-inflated version of `viridis`, good for plotting zero-inflated data. `loompy.cat_colors()` is a function that generates N distinct colors, in pleasing and distinguishable hues, for large N.

Contributes to 62.

Compatibility with Seurat & loomR

In some cases, Seurat would create loom files with attributes being variable-length ascii. This technically violates the loom specification, but what's worse is that loompy would read them as byte arrays. We now handle such strings gracefully and they are returned as arrays of string objects (supporting unicode).

Create empty layers

Previously, there was no way to create an empty layer without supplying a dense matrix. This would cause problems when you wanted to add a larger-than-RAM layer to an existing file. We now support an elegant syntax for creating an empty layer, directly on disk, by assigning a data type to the layer name. For example:

python
with loompy.connect("filename.loom") as ds:
ds.layers["intronic"] = "int16"


Or, using the shorthand syntax directly on the connection object:

python
with loompy.connect("filename.loom") as ds:
ds["intronic"] = "int16"


Once the layer has been created, you can assign values to (parts of) the layer, building it up incrementally.

**Note:** If you're using sparse matrices to create loom files, this update fixes a nasty performance bug (see 48)

Bugfixes

* Exception when assigning a layer directly on the connection object:

python
with loompy.connect("filename.loom") as ds:
ds["layername"] = m


* Severe performance bug when creating from large sparse matrix (issue 48)
* Error in example of valid file format (issue 44)
* Improved handling of connections opened in readonly mode (see commit d92bac9f04880bffcf00c1655d98d8fdc84a67d1)

Bugfixes

* Bug in sparse() caused it to load only the first 1000 cells


New features

* `scan()` method now accepts boolean mask arrays to select items (rows/cols)
* Opening a file that uses old-style `row_edges` and `col_edges` automatically adds `row_graphs` and `col_graphs`

Bugfixes and minor new features:

* `loompy.__version___` attribute (gives version of loompy package)
* Better handling of global attributes when combining files
* Make it possible to delete global attributes

Experimental new file feature: file spec version stamp

`LOOM_SPEC_VERSION` HDF5 attribute on the root group. Use it like this:

python
with loompy.connect(filename) as lc:
if "LOOM_SPEC_VERSION" in lc.attrs:
print("file version: " + lc.attrs.LOOM_SPEC_VERSION)
else:
print("file version: less than 2.0.1")


**Note**: this is experimental, and is a proposed new feature of the loom file format specification. As such, the attribute is not guaranteed to exist, and you must always check for it before trying to read it. If it doesn't exist, then the loom file spec version is assumed to be less than "2.0.1" by default.

**Note**: the LoomSpecVersion is not the same thing as the loompy package version.

Finally fixed sparse matrix support, so that this code is verified to work:

python
import numpy as np
import loompy
import scipy.sparse as sparse
filename = "test.loom"
matrix = sparse.coo_matrix((100, 100))
row_attrs = { "SomeRowAttr": np.arange(100) }
col_attrs = { "SomeColAttr": np.arange(100) }
loompy.create(filename, matrix, row_attrs, col_attrs)


(CSR, CSC and COO matrices are all supported and tested).

Also updated the docs with the example above.