Announcement
I'm happy to announce Loom 2.0, an almost complete rewrite of loompy. v2.0 implements many recently **requested features** such as Unicode support, multidimensional attributes, and multidimensional global attributes.
It also supports a powerful new concept of **in-memory views** (essentially a slice through the file, including all layers, attributes, and graphs) which works great with the new ``scan()`` method for out-of-memory algorithms.
It is much **more Pythonic**, with a uniform API for reading/writing/deleting attributes, graphs and layers.
It is **more generous** (allowing almost any kind of list, tuple or array to be assigned to attributes, or any kind of sparse or dense adjacency matrix for graphs). At the same time, it normalizes everything to conform to the file spec.
Finally, I have updated the file format specification to be **more specific** about the allowed datatypes and shapes.
All this, and it remains **fully backwards compatible with your old loom files**. I have used 2.0 for a couple of months already and it's a significant step forward. Your code will be simpler, more expressive and more capable.
Note: your current code should mostly work without problems. You'll get log messages for deprecated methods and attributes, but they can be safely ignored. There are **two breaking changes**, both explained below. After fixing instances of these two issues, our current analysis pipeline runs without error and uses nearly every feature of loompy 2.0, so I would consider this a reasonably stable release. Out-of-memory manifold learning and clustering of a 500k cells by 27k genes dataset with all the bells and whistles runs just fine on my MacBook with 16 GB RAM (admittedly, it takes a few hours)!
I've **updated the docs**, which are now (for technical reasons) hosted at http://linnarssonlab.org/loompy.
What's new
*Note: in the short examples below, `ds` is a `LoomConnection` object.*
1. More expressive and pythonic attribute management
* Attributes can be accessed via `ds.ca`(column attributes) and `ds.ra` properties.
* Both named access and dictionary-style access: `ds.ca.CellID` and `ds.ca["CellID"]`
* Set attribute values by assignment: `ds.ra.Chromosome = <values>`
* Delete attributes directly: `del ds.ra.Gene`
* Old-style attributes `col_attrs` and `row_attrs` remain, but are considered deprecated
* **Breaking change**: Old-style direct named attributes such as `ds.Gene` are no longer supported
* Attribute values are lazy-loaded. Nothing is loaded when you connect to a file, so connecting to files of any size is instantaneous and consumes almost no resources.
2. More expressive and pythonic graph management
* Graphs can be accessed via `ds.col_graphs` and `ds.row_graphs`
* Both by name and as dictionaries, e.g. `ds.col_graphs["KNN"]` or `ds.col_graphs.KNN`
* Graphs are stored and accessed as `scipy.sparse` graphs
* Store graphs directly from sparse matrices: `ds.col_graphs.MySparseGraph = sparse.coo_matrix(...)`
* Delete graphs: `del ds.col_graphs.KNN`
* Graphs are lazy loaded
* Old-style graph methods remain, but are considered deprecated
3. New in-memory view object
* Call `ds.view[..., ...]` to slice along rows and columns
* Views automatically slice through all layers, attributes and graphs
* New `scan()` method returns view objects representing partial views of the underlying file
4. Improved support for sparse matrices
* Fixed bugs in loompy.create() when supplying a sparse matrix
* Attribute values can now be sparse
* Store graphs from sparse matrices: `ds.col_graphs.MySparseGraph = sparse.coo_matrix(...)`
5. More flexible assignment of attributes
* Automatic conversion from tuples, lists, np.ndarray, np.matrix or scipy.sparse
* Automatic conversion of elements that are (unicode) strings, byte arrays, string objects and all kinds of numbers
* More accurate validation of inputs
* Whatever the values are, normalize them to HDF5 numbers or fixed-length ascii, per spec
6. New attribute data types
* Support for numeric types `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, `uint64`, `float16`, `float32` and `float64`.
* Support for multi-dimensional attributes (i.e. each value is an N-dimensional array)
7. New layer data types
* Support for numeric types `int8`, `int16`, `int32`, `int64`, `uint8`, `uint16`, `uint32`, `uint64`, `float16`, `float32` and `float64`.
8. Better handling of file lifecycle
* Support for context manager (`with loompy.connect(...) as ds:`)
* Better handling of errors on create() and close()
* **Breaking change:** `create()` and `create_from_cellranger()` do not return a value (they used to return an open `LoomConnection`, but this usually would leak a file handle).
9. Unicode support
* Roundtrip support for Unicode strings on the Python side
* ASCII encoding and decoding using [XML entitites](https://en.wikipedia.org/wiki/List_of_XML_and_HTML_character_entity_references), to ensure maximum compatibility
10. Improved scanning functionality
* New `scan` method replaces `batch_scan` and `batch_scan_layers`
* Scan columns while ordering rows based on key attribute (and vice versa)
* Iterator returns a view object, which is a slice through all layers, attributes and graphs
* Together with the new `create_append` function, makes it trivial to scan and combine files
11. Sample datasets
* We provide published datasets for browsing and download at http://loom.linnarssonlab.org (**Note:** broken in Safari; please use Chrome instead for now).
12. Changes to the loom file format specification
* More precise definition of allowed data types
* Multidimensional attributes
* Unicode support