Openghg

Fixed

- Added `align_metadata_attributes` to retrieve_remote and shifted function defination to standardise/meta. [PR 1197](https://github.com/openghg/openghg/pull/1197)
- Added `icos flags` to handle data that is flagged bad for remote icos data.[PR 1200](https://github.com/openghg/openghg/pull/1200)
- Pinned Zarr to `2.18.3` as github runners are picking `zarr 3.0` which is still in significant development state.[PR 1205](https://github.com/openghg/openghg/pull/1205)
- Added exist_ok = true argument to create_config_folder and removed ObjectStoreError call.[PR 1198](https://github.com/openghg/openghg/pull/1198)

Fixed

- Fixed options used with `xr.Dataset.to_zarr` in reponse to updates in xarray. [PR 1160](https://github.com/openghg/openghg/pull/1160)
- Added xfail for `cfchecker` tests due to broken link. [PR 1178](https://github.com/openghg/openghg/pull/1178)
- Removed duplicate code from `read_file` method in `BoundaryConditions` and `EulerianModel`. [PR 1192](https://github.com/openghg/openghg/pull/1192)

Added

- Check for file lock permissions with helpful error message. File locks are now created with rw permissions for user and group. [PR 1168](https://github.com/openghg/openghg/pull/1168)
- Removed parsers that are unused. [PR 1129](https://github.com/openghg/openghg/pull/1129)
- Added `data_owner` and `inlet_height_magl` as attributes to parse_icos. [PR 1147](https://github.com/openghg/openghg/pull/1147)
- Align the dataset(s) when opening the data while standardising footprints data to prevent error due to misalign coordinates. [PR 1164](https://github.com/openghg/openghg/pull/1164)
- Moved `sync_surface_metadata` to ObsSurface.read_file function so this is applied for all input data regardless of source_format. [PR 1138](https://github.com/openghg/openghg/pull/1138)
- Added `parse_co2_games` parser splitting multiple model from one file into separate datasources. [PR 1170](https://github.com/openghg/openghg/pull/1170)
- Added `dobj_url` as attribute to icos `retrieve_atmospheric` function. [PR 1174](https://github.com/openghg/openghg/pull/1174)
- Added parser for BoundaryConditions class. [PR 1180](https://github.com/openghg/openghg/pull/1180)
- Added parser for Eulerian Model class. [PR 1181](https://github.com/openghg/openghg/pull/1181)

Updated

- Removed serverless/cloud code since it is not being used. [PR 1177](https://github.com/openghg/openghg/pull/1177)
- Minimum version of python to 3.10. [PR 1175](https://github.com/openghg/openghg/pull/1175)
- Updated ICOS standardise function to reflect changes in ASCII file format. [PR 1140](https://github.com/openghg/openghg/pull/1140)
- Added `rename_vars` option to `get_obs_surface` to allow variable names based around species to be returned. [PR 1130](https://github.com/openghg/openghg/pull/1130)
- Added option to `get_obs_column` to return the column data directly rather than converting to mole fractions. [PR 1131](https://github.com/openghg/openghg/pull/1131)
- Made calculations in `ModelScenario._calc_modelled_obs_HiTRes` more efficient. [PR 1062](https://github.com/openghg/openghg/pull/1062)
- Updated type hints and `typing` imports for Python 3.10 [PR 1193](https://github.com/openghg/openghg/pull/1193)

Fixed

- Bug where `search_surface` couldn't accept a dictionary argument for `data_level`. [PR 1133](https://github.com/openghg/openghg/pull/1133)
- GIT_TAG variable passed to build step of release_conda and also environment activation is executed for publishing to conda step. [PR 1135](https://github.com/openghg/openghg/pull/1135)

Updated

- Updated parse_* functions for surface data type to accept `filepath` rather than `data_filepath`. This maps better to the `standardise_surface` input and makes this consistent with the other data types. [PR 1101](https://github.com/openghg/openghg/pull/1101)
- Separated data variable formatting from assign attributes function into dataset_formatter function.- [PR 1102](https://github.com/openghg/openghg/pull/1102)
- Required and optional keys for "column" data type were updated to reflect the two sources of this data (site, satellite) [PR 1104](https://github.com/openghg/openghg/pull/1104)
- Ensure metadata keywords for NOAA obspack are consistent with wider definitions including renaming data_source to dataset_source (data_source would be "internal"). [PR 1110](https://github.com/openghg/openghg/pull/1110)
- Updated data type classes to dynamically select inputs to pass to parse function and to include any required/optional keys not passed to the parse function within the metadata. [PR 1111](https://github.com/openghg/openghg/pull/1111)
- When adding new data sources, updated how lookup keys add optional keys. This used to only extract these from the optional_metadata input but this now allows keys to be added through any metadata. [PR 1112](https://github.com/openghg/openghg/pull/1112)
- Formalising metadata data merging logic within new util.metadata_util functions. [PR 1113](https://github.com/openghg/openghg/pull/1113)
- Splited build and publish steps in the workflow and check for `-` and `.` in the tags for build and publish. [PR 759](https://github.com/openghg/openghg/pull/759)

Fixed

- Bug where a datasource's folder in the `data` directory was not deleted by `Datasource.delete_all_data()`. This was causing `check_zarr_store` in `util/_user.py` to give a false negative. [PR 1126](https://github.com/openghg/openghg/pull/1126)
- Bug where an input filepath list to standardise_surface was only storing the last file hash. This allowed for some files to bypass the check for the same files depending on where they were in the original filepath list. [PR 1100](https://github.com/openghg/openghg/pull/1100)
- Bug where filepath needed to be a Path object when storing the file hash values. [PR 1108](https://github.com/openghg/openghg/pull/1108)
- Catch an `AttributeError` when trying synchronise attributes and metadata and a user passes a `bool` - [PR 1029](https://github.com/openghg/openghg/pull/1029)
- Mypy issue fixed for `util.download_data()` function based on updates described [requests Issue 465](https://github.com/psf/requests/issues/465) and included in [urllib3 PR 159](https://github.com/urllib3/urllib3/pull/159/files). This allowed the `decode_content` flag to be set directly rather than needing to patch the method. [PR #1118](https://github.com/openghg/openghg/pull/1118)

Added

- In `get_obs_surface`, if `inlet` is passed a slice and multiple search results are found, they will be combined into a single `ObsData` object with a "inlet" data variable. [PR 1066](https://github.com/openghg/openghg/pull/1066)
- Packaging and release documentation. [PR 961](https://github.com/openghg/openghg/pull/961)
- Options to search metastore by "negative lookup" and by "test functions"; the latter is used to implement
searching by a `slice` object to find a range of values - [PR 1064](https://github.com/openghg/openghg/pull/1064)
- Code to combine multiple data objects - [PR 1063](https://github.com/openghg/openghg/pull/1063)
- A new object store config file to allow customisation of metadata keys used to store data in unique Datasources - [PR 1041](https://github.com/openghg/openghg/pull/1041)
- Adds `data_level` and `data_sublevel` as additional keys which can be used to distinguish observation surface data - [PR 1051](https://github.com/openghg/openghg/pull/1051)
- New keywords `data_level` and `data_sublevel` as additional keys which can be used to distinguish observation surface data - [PR 1051](https://github.com/openghg/openghg/pull/1051)
- The `dataset_source` keyword previously only used for retrieved data is now available when using `standardise_surface` as well. This allows an origin key for the dataset to be included e.g. "InGOS", "European ObsPack". [PR 1083](https://github.com/openghg/openghg/pull/1083)
- Footprint parser for "paris" footprint format which includes the new format used for both the NAME and FLEXPART LPDM models - [PR 1070](https://github.com/openghg/openghg/pull/1070)
- Added `AGAGE` as a source format in the `standardise_surface` function, and associated parser functions and tests. Reads in output files from Matt Rigby's agage-archive - [PR 912](https://github.com/openghg/openghg/pull/912)
- Added feature of file sorting at standardise level before processing and remove `filepaths` input option - [PR 1074](https://github.com/openghg/openghg/pull/1074)
- Utility functions to combine multiple "data objects" (e.g. `ObsData`, or anything with `.data` and `.metadata` attributes) - [PR 1063](https://github.com/openghg/openghg/pull/1063)

Updated

- Updated `base` to `offset` in `resample` due to xarray deprecation. [PR 1073](https://github.com/openghg/openghg/pull/1073)
- Updated `get_obs_column` to output mole fraction details. This involves using the apriori level data above a maximum level and applying a correction to the column data (aligned with this process within [acrg code](https://github.com/ACRG-Bristol/acrg)). [PR #1050](https://github.com/openghg/openghg/pull/1050)
- The `data_source` keyword is now included as "internal" when using `standardise_surface` to distinguish this from data retrieved from external sources (e.g. "icos", "noaa_obspack"). [PR 1083](https://github.com/openghg/openghg/pull/1083)
- Added interactive timeseries plots in Search and Plotting tutorial. [PR 953](https://github.com/openghg/openghg/pull/953)
- Pinned the `icoscp` version within requirements to 0.1.17 based on new authentication requirements. [PR 1084](https://github.com/openghg/openghg/pull/1084)
- The `icos_data_level` metadata keyword is now retired and replaced with `data_level` when using the `retrieve.icos.retrieve_atmopsheric` workflow to access data from the ICOS Carbon Portal. [PR 1087](https://github.com/openghg/openghg/pull/1087)
- Removing 'station_long_name' and 'data_type' as required keys from the metadata config file as these do not need to be used as keys to distinguish datasources when adding new data. [PR 1088](https://github.com/openghg/openghg/pull/1088)
- The sort flag can now be passed via the SearchResults.retrieve interfaces to choose whether the data is returned sorted along the time axis. [PR 1090](https://github.com/openghg/openghg/pull/1090)

Fixed

- Bug in test when checking customised chunks were stored correctly in the zarr store. dask v2024.8 now changed the chunk shape after this was sorted was test was updated to ensure this didn't sort when retrieving the data. [PR 1090](https://github.com/openghg/openghg/pull/1090)
- Error reporting for `BaseStore` context manager. [PR 1059](https://github.com/openghg/openghg/pull/1059)
- Formatting of `inlet` (and related keys) in search, so that float values of inlet can be retrieved - [PR 1057](https://github.com/openghg/openghg/pull/1057)
- Test for zarr compression `test_bytes_stored_compression` that was failing due to a slight mismatch between actual and expected values. The test now uses a bound on relative error - [PR 1065](https://github.com/openghg/openghg/pull/1065)
- Typo and possible performance issue in `analysis._scenario.combine_datasets` - [PR 1047](https://github.com/openghg/openghg/pull/1047)
- Pinned numpy to < 2.0 and netcdf4 to <= 1.6.5. Numpy 2.0 release caused some minor bugs in OpenGHG, and netCDF4's updates to numpy 2.0 were also causing tests to fail - [PR 1043](https://github.com/openghg/openghg/pull/1043)
- Fixed bug where slightly different latitude and longitude values were being standardised and not aligned later down the line. These are now all fixed to the openghg_defs domain definitions where applicable upon standardisation. [PR 1049](https://github.com/openghg/openghg/pull/1049)

Fixed

- Updated incorrect import for data_manager within tutorial. This now shows the import from `openghg.dataobjects` not `openghg.store` - [PR 1007](https://github.com/openghg/openghg/pull/1007)
- Issue causing missing data when standardising multiple files in a loop - [PR 1032](https://github.com/openghg/openghg/pull/1032)

Added

- Added ability to process CRF data as `flux_timeseries` datatype (one dimensional data) - [PR 870](https://github.com/openghg/openghg/pull/870)