Openghg

Added

- Ability to convert from an old style NetCDF object store to the new Zarr based store format - [PR 967](https://github.com/openghg/openghg/pull/967)
- Updated `parse_edgar` function to handle processing of v8.0 Edgar datasets. [PR 965](https://github.com/openghg/openghg/pull/965)
- Argument `time_resolved` is added as phase 1 change for `high_time_resolution`, also metadata is updated and added deprecation warning. - [PR 968](https://github.com/openghg/openghg/pull/968)
- Added ability to pass additional tags as optional metadata through `standardising_*` and `transform_flux_data functions`. [PR 981](https://github.com/openghg/openghg/pull/981)
- Renaming `high_time_resolution` argument to `time_resolved` in metadata as more appropriate description for footprints going forward and added deprecation warning. - [PR 968](https://github.com/openghg/openghg/pull/968)
- Added explicit backwards compatability when searching previous object stores containing the `high_time_resolution` keyword rather than `time_resolved` - [PR 990](https://github.com/openghg/openghg/pull/990)
- Added ability to pass additional tags as optional metadata through `standardise_*` and `transform_flux_data functions`. [PR 981](https://github.com/openghg/openghg/pull/981)
- Check added object stores listed in the configuration which are in the previous format with a warning raised for this [PR 962](https://github.com/openghg/openghg/pull/962)

Fixed

- `source` can be passed to `transform_flux_data` with the EDGAR parser; `date` isn't stored with the transformed EDGAR data, since this is used for choosing what data to add, but doesn't describe all of the data in the object store. Fixed bug due to string split over two lines in logging message - [PR 1010](https://github.com/openghg/openghg/pull/1010)
- Fixed problem where the zarr store check raised an error for empty stores, preventing new zarr stores from being created - [PR 993](https://github.com/openghg/openghg/pull/993)
- Retrieval of level 1 data from the ICOS Carbon Portal now no longer tries to retrieve a large number of CSV files - [PR 868](https://github.com/openghg/openghg/pull/868)
- Added check for duplicate object store path being added under different store name, if detected raises `ValueError`. - [PR 904](https://github.com/openghg/openghg/pull/904)
- Added check to verify if `obs` and `footprint` have overlapping time coordinates when creating a `ModelScenario` object, if not then raise `ValueError` - [PR 954](https://github.com/openghg/openghg/pull/954)
- Added fix to make sure data could be returned within a date range when the data had been added non-sequentially to an object store - [PR 997](https://github.com/openghg/openghg/pull/997)
- Replace references to old `supplementary_data` repository with `openghg_defs` - [PR 999](https://github.com/openghg/openghg/pull/999)
- Added call to synonyms for species while standardising - [PR 984](https://github.com/openghg/openghg/pull/984)

This version brings a breaking change with the move to use the [Zarr](https://zarr.dev/) file format for data storage. This means that object stores created with previous versions of OpenGHG will need to be repopulated to use this version. You should notice improvements in time taken for standardisation, memory consumption and disk usage. With the use of Zarr comes the ability for the user to control the way which data is processed and stored. Please see [the documentation](https://docs.openghg.org/tutorials/local/Adding_data/Adding_ancillary_data.html#chunking) for more on this.

Added

- Added option to pass `store` argument to `ModelScenario` init method. [PR 928](https://github.com/openghg/openghg/pull/928)

Fixed

- Issue caused when passing a list of files to be processed. If OpenGHG had seen some of the files before it would refuse to process any of them - [PR 890](https://github.com/openghg/openghg/pull/890)

Changed

- Moved to store data in [Zarr](https://github.com/zarr-developers/zarr-python) stores, this should reduce both the size of the object store and memory consumption whilst processing and retrieving data - [PR #803](https://github.com/openghg/openghg/pull/803)
- standardise_footprint was updated to allow a source_format input to be specified. This currently only supports "acrg_org" type but can be expanded upon [PR 914](https://github.com/openghg/openghg/pull/914).
- Internal format for "footprint" data type was updated to rename meteorological variable names to standard names [PR 918](https://github.com/openghg/openghg/pull/918).
- Standardise_footprint now uses the meterological model input as a distinguishing keyword when adding data. [PR 955](https://github.com/openghg/openghg/pull/955).
- Meterological model input renamed from `metmodel` to `met_model` [PR 957](https://github.com/openghg/openghg/pull/957).
- Updated internal naming and input data_type to use "flux" rather than "emissions" consistently. - [PR 827](https://github.com/openghg/openghg/pull/827)

Added

- More more explanation regarding use of `search_*` and `get_*` function in tutorial 1 [PR 952](https://github.com/openghg/openghg/pull/952)

Fixed

- Bug fix for conversion of species parameter with its synonym value inside get_obs_surface_local. [PR 871](https://github.com/openghg/openghg/pull/871)
- Missing requirement for filelock package added to conda environment file [PR 857](https://github.com/openghg/openghg/pull/857)
- Missing store argument adding to search function allow searching within specific object stores [PR 859](https://github.com/openghg/openghg/pull/859)
- Bug fix for allowing a period to be specified when this cannot be inferred from the input data [PR 899](https://github.com/openghg/openghg/pull/899)

Fixed

- Bug fix for passing calibration_scale as optional parameter to the parser function. [PR 872](https://github.com/openghg/openghg/pull/872)

Added

- Added `DeprecationWarning` to the functions `parse_cranfield` and `parse_btt`. - [PR 792](https://github.com/openghg/openghg/pull/792)
- Added `environment-dev.yaml` file for developer conda environment - [PR 769](https://github.com/openghg/openghg/pull/769)
- Added generic `standardise` function that accepts a bucket as an argument, and used this to refactor `standardise_surface` etc, and tests that standardise data - [PR 760](https://github.com/openghg/openghg/pull/760)
- Added `MetaStore` abstract base class as interface for metastore classes, and a `ClassicMetaStore` subclass implements the same bucket/key structure as the previous metastore.
All references to TinyDB are now in the `objectstore` module, meaning that there is only one place where code needs to change to use a different backend with the metastore - [PR 771](https://github.com/openghg/openghg/pull/771)
- Added compression to `Datasource.save` and modified `Datasource.load` to take advantage of lazy loading via `xarray.open_dataset` - [PR 755](https://github.com/openghg/openghg/pull/755)
- Added progress bars using `rich` package - [PR 718](https://github.com/openghg/openghg/pull/718)
- Added config for Black to `pyproject.toml` - [PR 822](https://github.com/openghg/openghg/pull/822)
- Added `force` option to `retrieve_atmospheric` and `ObsSurface.store_data` so that retrieved hashes can be ignored - [PR 819](https://github.com/openghg/openghg/pull/819)
- Added `SafetyCachingMiddleware` to metastore, which caches writes and only saves them to disk if the underlying file
has not changed. This is to prevent errors when concurrent writes are made to the metastore. [PR 836](https://github.com/openghg/openghg/pull/836)

Fixed

- Bug fix for sampling period attribute having a value of "NOT_SET" and combining the observation and footprint data. Previously this was raising a ValueError. [PR 808](https://github.com/openghg/openghg/pull/808)
- Bug where `radon` was not fetched using `retrieve_atmospheric` from icos data. - [PR 794](https://github.com/openghg/openghg/pull/794)
- Bug with CRDS parse function where data for all species was being dropped if only one species was missing - [PR 829](https://github.com/openghg/openghg/pull/829)
- Datetime processing has been updated to be compatible with Pandas 2.0: the `date_parser` argument of `read_csv` was deprecated in favour of `date_format`. [PR 816](https://github.com/openghg/openghg/pull/816)
- Updated ICOS retrieval functionality to match new metadata retrieved from ICOS Carbon Portal - [PR 806](https://github.com/openghg/openghg/pull/806)
- Added "parse_intem" function to parse intem emissions files - [PR 804](https://github.com/openghg/openghg/pull/804)

Changed

- Datasource UUIDs are no longer stored in the storage class and are now only stored in the metadata store - [PR 752](https://github.com/openghg/openghg/pull/752)
- Support dropped for Python 3.8 - [PR 818](https://github.com/openghg/openghg/pull/818). OpenGHG now supports Python >= 3.9.

Fixed

- Bug where the object store path being written to JSON led to an invalid path being given to some users - [PR 741](https://github.com/openghg/openghg/pull/741)

Changed

- Added read-only opening of the metadata store of each storage class when searching. This is done using a `mode` argument pased to the `load_metastore` function - [PR 763](https://github.com/openghg/openghg/pull/763)

Added

- Added `rich` package to printing out SearchResults object in a table format. If using an editable install please update your environment to match requirements.txt / environment.yml - [PR 696](https://github.com/openghg/openghg/pull/696)

Fixed

- Bug in `get_readable_buckets`: missing check for tutorial store - [PR 729](https://github.com/openghg/openghg/pull/729)
- Bug when adding high time resolution footprints to object store: they were not being distinguished from low resolution footprints - [PR 720](https://github.com/openghg/openghg/pull/720)
- Bug due to `object_store` key not being present in `Datasource` metadata - [PR 725](https://github.com/openghg/openghg/pull/725)
- Bug in `DataManager` where a string was interpreted as a list when processing metadata keys to be deleted - [PR 713](https://github.com/openghg/openghg/pull/713)