Muninn

~~~~~~~~~~~~~~

* Added support for oauth2 (Resource Owner Password Credentials Grant) for
muninn-pull and added oauth2 support to the auth_file credentials file.

* url prefixes can now be used as keys in the auth_file credentials file
(instead of only hostnames).

* muninn-pull will now auto-unzip a downloaded file if the downloaded filename
equals core.physical_name + '.zip' (or '.ZIP').

* Improved memory efficiency of muninn-pull. Files are no longer read fully
in memory when downloading.

* Added new plugin attribute 'hash_type', to specify hash algorithm
(matching hashlib module). The 'use_hash' attribute has been deprecated.
core.hash will now contain '<type>:<hash>' as content. Existing core.hash
values will still be treated as sha1 hashes for backward compatibility.

* Fixed issue where enabling indices on Geometry fields did not work.

* Archive.retrieve() and Archive.retrieve_by_name() now return a list of
product paths. Archive.retrieve_by_uuid() now returns a single path.

* Added Archive.delete_properties_by_uuid().

* Added download_args, upload_args, copy_args, and transfer_config options
to S3 storage backend.

* Added (optional) product type plugin attribute 'namespaces', which
contains a list of names of non-core namespace names of properties that
plugin.analyze may return. Those non-core properties are now also passed to
the other plugin methods.

* Removed `muninn-update --namespaces` option, as this should now work
automatically.

* Added product plugin method 'post_remove_hook' that is called after a
successful remove.

* Changed `muninn-update -a <action>` command to `munnin-update <action>`.

* Changed `muninn-ingest -c` command to mean 'catalogue only'.

* Added basic 'in' and 'not in' operators (so that e.g. the following works:
`physical_name not in ["a", "b"]`)

* Namespace records can now be removed with Archive.update_properties() by
passing: `Struct({'mynamespace': None})`

~~~~~~~~~~~~~~

* Added a storage backend framework, including S3 and Swift backends.

* Changed configuration file format due to new storage backends:
- renamed `backend` parameter to `database` in `archive` section
- added `storage` parameter to `archive` section
- added new sections for `fs`, `s3`, and 'swift`
- moved `root` and `use_symlinks` parameters from `archive` section
to `fs` section
- added `tmp_root` parameter to `s3` and `swift` sections

* Added support for pg8000 as postgresql driver. Added 'library'
parameter to 'postgresql' section to specify library to use.

* Added a basic test/coverage framework.

* Changed the syntax for defining namespace properties, allowing
one to specify if a property should be indexed in the database backend.
The old approach is still supported for backward compatibility.

* Added -c/--catalogue-only option to muninn-remove.

* Added support for sub-queries in is_source_of and is_derived_from
(specify a sub-query instead of a uuid).

* Added support for searching on UUID directly (without uuid==).

* Several fixes for NOT expressions in queries.

* Improved querying on NULL properties. For example, for 'property != b',
we now also return products for which b is NULL.

* Added support for `is_defined(mynamespace)` to check whether a namespace
record exists.

* Added property_names argument to Archive.retrieve_properties().

* Added Archive.delete_properties().

* Fixed issue with intra-archive symlinks without enclosing dir (resulting
in broken symlinks).

* Fixed issue where product hashes were not actually verified.

* Made product hashing compatible with Python 3 (and using UTF-8 encoding
before hashing, instead of the current filesystem encoding).

~~~~~~~~~~~~~~

* Added optional progress bars to muninn-ingest (requires 'tqdm' package).

* Added --parallel and --processes options to muninn-ingest.

* Added --processes option to muninn-update.

* Added property_names argument to Archive.search().

* Cleaned up terminology: we now use 'properties' instead of 'attributes'
everywhere when refering to product metadata.

* Fixed issue in sqlite backend where geometry covers()/intersects()
operations returned true if geometry property was not defined.

~~~~~~~~~~~~~~

* Added muninn-summary to display aggregate statistics.

* Removed muninn-search '-s' option in favor of muninn-summary.

* Fixed initialization of database when using sqlite backend.

* Strip operation now automatically restricts itself to products with a
non-empty archive_path.

~~~~~~~~~~~~~~

* Added -f/--force option to muninn-ingest.

* Improved formatting of duration in muninn-search summary output.

* Added optional progress bars to muninn-update (requires 'tqdm' package).

* Added --parallel and --keep options to muninn-update.

* Several fixes for muninn-search --paths.

* Archive.export() now returns the list of paths of the exported products. The
`export_` functions in the product type plugins must return the path of the
exported product.

* Fixed issue where the database connection was not properly cleaned up when
an error was raised during a database commit/rollback.

* Fixed issue where muninn-prepare --dryrun would not produce output for
sqlite if the database file did not exist yet.

* Fixed issue where 'active' state was not always reset when a muninn-pull
failed.

* Improve handling of creating tags/links that already exist
(postgresql will now only raise an exception for contention cases).

* Added check on startup to verify that all mandatory attributes/methods are
available for each product type plugin.

* Several smaller fixes (including Python3 compatibility).

~~~~~~~~~~~~~~

* Added support for Python 3.

* Dropped support for spatialite 3.1 and earlier. spatialite 4.0.0 or higher
is now required (which requires a version of sqlite3 that support
extensions). Support for pyspatialite, which only worked with spatialite
3.1, has been dropped. pysqlite2 is now the preferred module for sqlite
support.

* Muninn now supports custom remote backends (used for muninn-pull).
The ecmwfapi pull support has now been removed from the core muninn code and
will become available as a separate plugin.

* Added muninn-info to display generic archive information (currently, the
availables namespaces and product types)

* Added archive maintenance tool muninn-update that allows updating properties
for a set of products (for when e.g. a product type plugin has changed).

* Added mod_spatialite_path option for sqlite backend.

* Command line tools now have a --verbose mode, displaying debug-level logging.

* Added 'paths' option to muninn-search that prints the physical path of the
products found instead of the normal output.

* Added --keep option to muninn-ingest to ingest products that are already in
the archive, maintaining its current relative path.

* Allow calling `muninn-search -p '*'` to fetch all fields of all namespaces.

* muninn.Archive.product_path() argument can be a product properties Struct.

* Added support for tabulate package to format the output.

* muninn.Archive.update_properties() takes an optional 'create_namespaces'
parameter that, if set, will create any undefined namespaces for a product.

* Added product plugin method 'post_pull_hook' that is called after a
successful pull.

* Close SQL connection after each operation (ingestion/search/etc.).

* Added support for separate auth_file that can contain credentials for
downloads performed using muninn-pull.

* Added -f option to muninn-strip.

* Fixed issue where geometry columns in namespace extensions were not
created when using the sqlite backend.