Muninn

~~~~~~~~~~~~~~~~

* Fixed python 2 issues and updated documentation that python>=2.7 is required.

~~~~~~~~~~~~~~

* Introduced a public python API for muninn.

* muninn-retrieve and muninn-export will now download a product based on the
core.remote_url setting if core.archive_path was not set (i.e. the product
was not stored in the archive).
To disable this and use the old behaviour, make sure to explictly add an
'is_defined(archive_path)' filter when performing a retrieve/export.

* Introduced storage=none option for archives. This works as a pure catalogue:
archive_path/archive_date should never be set, attach/strip/pull won't work,
ingest will set remote_url to a file:// url pointing to the ingested product,
retrieve and export work purely based on remote_url (see previous point).

* S3 backend 'host' configuration parameter can now be a full endpoint url and
'port' parameter is now optional.

* S3/Swift storage credentials can now be taken from auth_file credentials
file and don't have to be included in the archive configuration file itself.

* Added JSON datatype for metadata properties (this uses JSONB storage when
using the postgresql database backend).

* Added --force and --verify-hash-before options to muninn-attach and added
additional improvements to keep hash and size in line with attached product.

* Improved (re)setting metadata state after storage failure for attach, pull,
and ingest.

* Added support for alternative number bases in queries (e.g. '==0xffff').

* Added --parallel option to all remaining muninn command-line tools.

* Added --force and --verify-hash-before options to muninn-attach.

* Allow 'muninn-strip --force' to reactivate a product even if archive_path is
empty.

* Removed muninn-update post_ingest and post_pull commands since these were no
longer working with generic hooks and the S3/Swift storage backends.

* Added built-in remote backend for sftp (requires paramiko package).

* Fixed issue in zip auto-extraction for muninn-pull.

* Added support for auto-extraction of tar(/gz) files for muninn-pull.

* Run hooks after core.active is set to True instead of before (regression
introduced in 5.4).

* Creating a table will now set COLLATE of TEXT properties to 'C' for the
postgres backend to guarantee consistent sorting. Fields of existing tables
can be manually updated using the following sql command template:
ALTER TABLE table ALTER COLUMN property SET DATA TYPE TEXT COLLATE "C"

* Removed support for all deprecated archive configuration options.

~~~~~~~~~~~~~~

* Added muninn-attach/Archive.attach() which acts as the inverse of
muninn-strip/Archive.strip().

* Fixed grand_type -> grant_type keyword typo for oauth2 entries in auth_file.

* Allow explicit setting of region for S3 backend.

* Added (optional) 'paths' argument to plugin export methods.

* Fix pg8000 >= 1.22 compatibility issue with filtering on timestamps.

~~~~~~~~~~~~~~

* Added verify-hash-download option to muninn-pull.

* Use Bi/Ki/Gi/... as unit for human readable sizes instead of K/G/...

* When using the S3/Swift backend, a post ingest/pull hook will now use a
locally kept temporary version of the product (instead of retrieving it
again from the bucket). The path(s) to the ingested/pulled product files
is passed on using a new 'paths' argument. Old hook functions without this
argument will still work, but will not be able to access the files.

* muninn-pull for http(s) now retrieves data in blocks instead of reading the
whole file in memory.

* Fixed issue when trying to ingest empty files or directories into S3/Swift
storage.

* Fix compatibility issue with pg8000 >= 1.22.

~~~~~~~~~~~~~~~~

* Fixed issue where a force ingest on a catalogue-only archive would fail.

~~~~~~~~~~~~~~

* Added generic hooking mechanism (not coupled to a product type).

* Added 'muninn-update retype' to allow correction of product_type values.

* Added 'day', 'hour', 'minute', 'second', and 'time' subscripts for
grouping by timestamps in muninn-summary.

* UUID values can now be used everywhere as boolean in query expressions.

* Add support for new type conversions of pg8000 >= 1.16.

* Allow filename in http result header for muninn-pull to be unquoted.