Kosh

Description

This release is a patch release.

New in this release

* store can open a dataset based on a Sina record (`store.open(sina_record)`).

Improvements

* Copyright for `compute_hopkins_statistic`
* Uses sinas `exist` function rather than a `try`/`except` to decide if we update or insert new records into the store.

Bug fixes

* None

Description

This release introduces clustering capabilities into Kosh. It also drops support for Python 2.

New in this release

* Support for Clustering (via operators).
* Dropped Python 2 support.
* loaders can access the dataset requesting the data
* Operators now have a `describe_entries` function to help them understand what's coming to them.

Improvements

* `find` function accepts `id` as an alias for `id_pool` to restrict search to some ids
* The store can now be used within a context manager
* Curves can be added/removed to a dataset
* Ensembles can be created from the command line
* passing `type=None` when searching the store will return Kosh specific objects as well as regular datasets (e.g associated files objects)
* Added a `verbose` mode to `dataset.list_features()` to let users know when a loader failed to load a uri. Mostly useful for debugging


Bug fixes

* Fixed an issue where associating a file multiple time was not reflected into the store and a subsequent dissociation or an async association would not be caught. Dissociation would cause the object to be removed from the store.
* Deleting a dataset attribute and re-adding it would cause a crash
* Loaders can be removed from store

Description

This is a maintenace release, with added support for Windows systems.

New in this release

* Support for Windows systems (note that `kosh cp` and `kosh mv` are not currently supported on Windows)
* While importing sina `json` files you can now skip over some sections, such as `curve_sets`.

Improvements

* `find` function accepts `id` as an alias for `id_pool` to restrict search to some ids

Bug fixes

* Fixed an issue where associating a file multiple time was not reflected into the store and a subsequent dissociation or an async association would not be caught. Dissociation would cause the object to be removed from the store.

Description

This is a mainly maintenance release introducing a few new features.

New in this release

* Decorators for transformers and operators
* Regular `def foo(...):` can now be converted to transformers or operators via decorators. e.g for a numpy transformer: `kosh.numpy_transformer`. See the [transformer](examples/Example_05a_Transformers.ipynb) and [operators](examples/Example_06_Operators.ipynb) notebooks for more details.
* Introducing a loader for text and column-based data, based on top `numpy.loadtxt`. See [this](examples/Example_column_based_text_files.ipynb) notebook.
* When cloning a dataset one can choose to preserve ensemble memberships (`preserve_ensembles_memberships=True`), simply copy over these attributes (`preserve_ensembles_memberships=False`) (**default**) or ignore information related to ensembles (`preserve_ensembles_memberships=-1`)
* Dataset objects now have a `is_ensemble_attribute()` function to know if an attribute belongs to an ensemble.
* Added `list_attributes()` function to ensemble objects.

Improvements

* When printing a dataset, the attributes coming from ensembles the dataset belongs to are listed in a separate section.

Bug fixes

* When cloning a dataset, an artificial `id` attribute was created with the original dataset `id` in it.
* Setting an attribute to an invalid value would cause the dataset to disappear from the store.

Description

This release aligns Kosh with Sina and makes it the only backend. Kosh and Sina (1.11) API's have been mostly aligned.
Sina curve and file section can now be recognized and taken advantage by Kosh.

New in this release

* Sina alignment:
* Sina is only supported backend, no more code to potentially support other backends
* curves appear as associated
* files with `mimetype` appear as virtual associated files
* Kosh exported json files are sina-compatible and Kosh can ingest Sina's json files
* Stores are now opened via: `kosh.connect(...)`
* `search(...)` is now `find(...)`
* any non Kosh-reserved record type is considered a dataset
* `find` functions return generators (used to be lists)
* Support for ensembles
* Kosh stores can be associated with other Kosh stores.
* `kosh` command line:
* can create stores
* can add datasets
* can use htar to tar up data
* datasets can be cloned
* While importing a dataset into a store, there are now options to handle conflicts.
* Loader for file saved by numpy (`.npy`)
* Store can fix changed/updated fast_sha

Improvements

* Do not try to import external Python packages until needed -> some loader might appear as valid even though python packages are missing.
* versioning is now pip compatible
* `import_dataset(...)` can import list of datasets
* Added `verbose` argument to transformers and operators -> this will let the user know when retrieving data from cache
* set user name to "default" if can't get it from USER env var

Bug fixes

* `matplotlib` import would crash if no DISPLAY environment variable
* hdf5 leading / fix
* No more error if a known mime type points to missing file (`list_features` will not show it)
* dissociate files from store after moving them

Description

This release is fully backward compatible but introduces new concepts.
Operators are introduced allowing the composition of features from one or many sources.
Feature selection without extraction is now possible via the new execution graphs introduced in this release.
Execution graphs are the recommended way to use Kosh going forward, as reflected in the updated notebooks.


New in this release

* Operators: Compose multiple features (and their transformers).
* Execution graph concept (select and compose features before executing).
* New Conduit's Sidre Mesh Blueprint field loader.

Deprecation Warning

* In future versions (not 1.2) the `search` functions will return a generator (they are currently returning a list). In this version a warning is issued when you use the `search` function.

Improvements

* Multiple speed/caching optimizations.
* A cleanup function helps you clean your store from files that no longer exist.
* Cleaned up tables in documentation.
* Transformers get a `parent` attribute, allowing you to access its caller in the `transform` function.

Bug fixes

* In some case where a feature was available from the loader but not listed, Kosh would let the user access it. With this bug fix, the loader's `list_features` must be fixed first in order to access the feature. This bug affected groups in HDF5 files. Groups were not listed but still accessible, HDF5 loader now lists `groups` as actual features.
* If a loader needed matplotlib and no X connection was available the import of `matplotlib.pyplot` would lead to an uncatchable error. We now check for a valid backend first (via environment's `DISPLAY` variable).