Muninn

~~~~~~~~~~~~~~~~

* Fixed issue where a force ingest on a catalogue-only archive would fail.

~~~~~~~~~~~~~~

* Added generic hooking mechanism (not coupled to a product type).

* Added 'muninn-update retype' to allow correction of product_type values.

* Added 'day', 'hour', 'minute', 'second', and 'time' subscripts for
grouping by timestamps in muninn-summary.

* UUID values can now be used everywhere as boolean in query expressions.

* Add support for new type conversions of pg8000 >= 1.16.

* Allow filename in http result header for muninn-pull to be unquoted.

~~~~~~~~~~~~~~

* Added support for oauth2 (Resource Owner Password Credentials Grant) for
muninn-pull and added oauth2 support to the auth_file credentials file.

* url prefixes can now be used as keys in the auth_file credentials file
(instead of only hostnames).

* muninn-pull will now auto-unzip a downloaded file if the downloaded filename
equals core.physical_name + '.zip' (or '.ZIP').

* Improved memory efficiency of muninn-pull. Files are no longer read fully
in memory when downloading.

* Added new plugin attribute 'hash_type', to specify hash algorithm
(matching hashlib module). The 'use_hash' attribute has been deprecated.
core.hash will now contain '<type>:<hash>' as content. Existing core.hash
values will still be treated as sha1 hashes for backward compatibility.

* Fixed issue where enabling indices on Geometry fields did not work.

* Archive.retrieve() and Archive.retrieve_by_name() now return a list of
product paths. Archive.retrieve_by_uuid() now returns a single path.

* Added Archive.delete_properties_by_uuid().

* Added download_args, upload_args, copy_args, and transfer_config options
to S3 storage backend.

* Added (optional) product type plugin attribute 'namespaces', which
contains a list of names of non-core namespace names of properties that
plugin.analyze may return. Those non-core properties are now also passed to
the other plugin methods.

* Removed `muninn-update --namespaces` option, as this should now work
automatically.

* Added product plugin method 'post_remove_hook' that is called after a
successful remove.

* Changed `muninn-update -a <action>` command to `munnin-update <action>`.

* Changed `muninn-ingest -c` command to mean 'catalogue only'.

* Added basic 'in' and 'not in' operators (so that e.g. the following works:
`physical_name not in ["a", "b"]`)

* Namespace records can now be removed with Archive.update_properties() by
passing: `Struct({'mynamespace': None})`

~~~~~~~~~~~~~~

* Added a storage backend framework, including S3 and Swift backends.

* Changed configuration file format due to new storage backends:
- renamed `backend` parameter to `database` in `archive` section
- added `storage` parameter to `archive` section
- added new sections for `fs`, `s3`, and 'swift`
- moved `root` and `use_symlinks` parameters from `archive` section
to `fs` section
- added `tmp_root` parameter to `s3` and `swift` sections

* Added support for pg8000 as postgresql driver. Added 'library'
parameter to 'postgresql' section to specify library to use.

* Added a basic test/coverage framework.

* Changed the syntax for defining namespace properties, allowing
one to specify if a property should be indexed in the database backend.
The old approach is still supported for backward compatibility.

* Added -c/--catalogue-only option to muninn-remove.

* Added support for sub-queries in is_source_of and is_derived_from
(specify a sub-query instead of a uuid).

* Added support for searching on UUID directly (without uuid==).

* Several fixes for NOT expressions in queries.

* Improved querying on NULL properties. For example, for 'property != b',
we now also return products for which b is NULL.

* Added support for `is_defined(mynamespace)` to check whether a namespace
record exists.

* Added property_names argument to Archive.retrieve_properties().

* Added Archive.delete_properties().

* Fixed issue with intra-archive symlinks without enclosing dir (resulting
in broken symlinks).

* Fixed issue where product hashes were not actually verified.

* Made product hashing compatible with Python 3 (and using UTF-8 encoding
before hashing, instead of the current filesystem encoding).

~~~~~~~~~~~~~~

* Added optional progress bars to muninn-ingest (requires 'tqdm' package).

* Added --parallel and --processes options to muninn-ingest.

* Added --processes option to muninn-update.

* Added property_names argument to Archive.search().

* Cleaned up terminology: we now use 'properties' instead of 'attributes'
everywhere when refering to product metadata.

* Fixed issue in sqlite backend where geometry covers()/intersects()
operations returned true if geometry property was not defined.

~~~~~~~~~~~~~~

* Added muninn-summary to display aggregate statistics.

* Removed muninn-search '-s' option in favor of muninn-summary.

* Fixed initialization of database when using sqlite backend.

* Strip operation now automatically restricts itself to products with a
non-empty archive_path.