Muninn

~~~~~~~~~~~~~~

* Added -f/--force option to muninn-ingest.

* Improved formatting of duration in muninn-search summary output.

* Added optional progress bars to muninn-update (requires 'tqdm' package).

* Added --parallel and --keep options to muninn-update.

* Several fixes for muninn-search --paths.

* Archive.export() now returns the list of paths of the exported products. The
`export_` functions in the product type plugins must return the path of the
exported product.

* Fixed issue where the database connection was not properly cleaned up when
an error was raised during a database commit/rollback.

* Fixed issue where muninn-prepare --dryrun would not produce output for
sqlite if the database file did not exist yet.

* Fixed issue where 'active' state was not always reset when a muninn-pull
failed.

* Improve handling of creating tags/links that already exist
(postgresql will now only raise an exception for contention cases).

* Added check on startup to verify that all mandatory attributes/methods are
available for each product type plugin.

* Several smaller fixes (including Python3 compatibility).

~~~~~~~~~~~~~~

* Added support for Python 3.

* Dropped support for spatialite 3.1 and earlier. spatialite 4.0.0 or higher
is now required (which requires a version of sqlite3 that support
extensions). Support for pyspatialite, which only worked with spatialite
3.1, has been dropped. pysqlite2 is now the preferred module for sqlite
support.

* Muninn now supports custom remote backends (used for muninn-pull).
The ecmwfapi pull support has now been removed from the core muninn code and
will become available as a separate plugin.

* Added muninn-info to display generic archive information (currently, the
availables namespaces and product types)

* Added archive maintenance tool muninn-update that allows updating properties
for a set of products (for when e.g. a product type plugin has changed).

* Added mod_spatialite_path option for sqlite backend.

* Command line tools now have a --verbose mode, displaying debug-level logging.

* Added 'paths' option to muninn-search that prints the physical path of the
products found instead of the normal output.

* Added --keep option to muninn-ingest to ingest products that are already in
the archive, maintaining its current relative path.

* Allow calling `muninn-search -p '*'` to fetch all fields of all namespaces.

* muninn.Archive.product_path() argument can be a product properties Struct.

* Added support for tabulate package to format the output.

* muninn.Archive.update_properties() takes an optional 'create_namespaces'
parameter that, if set, will create any undefined namespaces for a product.

* Added product plugin method 'post_pull_hook' that is called after a
successful pull.

* Close SQL connection after each operation (ingestion/search/etc.).

* Added support for separate auth_file that can contain credentials for
downloads performed using muninn-pull.

* Added -f option to muninn-strip.

* Fixed issue where geometry columns in namespace extensions were not
created when using the sqlite backend.

~~~~~~~~~~~~~~

* Changed license of muninn to BSD.

* Muninn can now be used as a pure catalogue server.
An archive root directory is still needed, but can remain empty.

* Changed the core schema:
- core.archive_date is now optional and is only set if core.archive_path is
set.
It will reflect the time at which the product was stored in the archive.
- added core.metadata_date which represents the last modification time of
the metadata record. It will automatically be updated if the metadata is
changed via the muninn interfaces.
- added optional core.remote_url field which can contain urls of the form
file://, http://, ftp://, etc. that point to a remote location of the
product.

* The schema_name option for the postgresql backend has been removed.
This is now replaced by a table_prefix option that is applicable for
both the postgresql and sqlite backends.

* The sqlite backend will no longer remove the database file for a 'destroy'
(it will only remove the tables). Both 'prepare' and 'destroy' can be
performed on an existing sqlite database file.

* Added 'pull' command that allows downloading of products that are not yet
in the archive. It will use the core.remote_url value to perform the
download. A 'retrieve' still only works on products that are already in the
archive.
The pull method supports the default http/https/file/ftp protocols but also
supports a custom ecmwfapi protocol that will use the ecmwf-api-client
python package to download data from the ECMWF MARS archive.

* The muninn Struct type that is used in the Python interface to store
metadata can now be initialized with a Python dict.

* Added create_properties() and generate_uuid() functions to the Python
muninn.Archive interface. These are functions that should be used to create
pure metadata records (without archived product) in a muninn archive.

* Added verify_hash() function to the Python muninn.Archive interface.

* muninn.Archive.generate_uuid() is now a static method.

* Improved preparing and destroying of muninn archives. This will now better
distinguish between the archive directory and the catalogue.

* Verifying of the hash after an ingest of a product in the archive is now
optional (--verify-hash argument to muninn-ingest).

* The muninn sqlite backend now adds a spatialindex on footprint.
The indices for both backends have all been renamed to idx_<table>_<field>.

* muninn-prepare has a --dry-run mode, that just dumps the SQL that would be
executed.

* The muninn sqlite backend now supports the escape character '\' when using
the ~= operator.

* Moved change history from README to separate CHANGELOG file.
Renamed README.developer to EXTENSIONS and added DEVELOPER file.
Include all documentation files with the source package.

~~~~~~~~~~~~~~~~

* Fixed parsing of negative coordinates in literal geometry values.

~~~~~~~~~~~~~~

* Added sqlite backend.

* Added arithmetic operators "+", "-", "*", and "/" for all numeric types.

* Added binary "-" operator for timestamps. This operator returns the length
of the time interval between two timestamps as a fractional number of
seconds.

* Updated comparison operators to support comparisons between integer and real
types.

* Changed the muninn expression language to use a sub-set of the Well Known
Text (WKT) markup language to represent literal geometry values.

* Added -c/--catalogue-only arguments to prepare/destroy commands to only
create/remove the catalogue database (while keeping the product archive
intact).

* If a product that is ingested is already in the right target location in
the archive on disk, muninn will just leave the product at its location
without trying to move or symlink it.

* Added 'id' primary key to links and tags tables, which is to support other
front-ends (which don't support multi-field primary keys) on top of the
same muninn database.
This change should not require a migration of existing databases (as long
as no other front-ends are planned to be used).

* Tags no longer have the constraint that they should be an 'identifier'.
Added ',' as tag separator to output of muninn-list-tags.

* Changed the Postgresql backend to create indices on most columns of the
'core', 'tag', and 'link' tables when creating these tables.

* MUNINN_CONFIG_PATH can now also refer directly to archive configuration
files.

~~~~~~~~~~~~~~~~

* Changed the tag() function of the Archive class such that setting a tag on
a product that already has that tag is not treated as an error.

* Changed the link() function of the Archive class such that adding a link
to a product that already has that link is not treated as an error.

* Changed the tags() function of the Archive class to return a list of tags
ordered by tag name.

* Improved error reporting for errors related to the creation or destruction
of an archive.

* Improved error reporting by the Postgresql backend.