Openghg

Added

- New `datapack` submodule to allow output obspacks to be created. This includes the `create_obspack` function which takes an input search file and produces an obspack within a defined structure from this. [PR 1117](https://github.com/openghg/openghg/pull/1117)

Updated

- Unpinned numpy so that we can now use numpy 2.0. [PR 1235](https://github.com/openghg/openghg/pull/1235)
- When combining obs and footprint data in ModelScenario, allow resample_to to be set to None so that the data is aligned but not resampled. This is also turned on by default by passing the `platform` keyword and setting to any name which contains "flask" to the relevant ModelScenario methods. [PR 1236](https://github.com/openghg/openghg/pull/1236)
- Extracted `align_obs_and_other` from `ModelScenario.align_obs_footprint` into `analyse._alignment`. [PR 1234](https://github.com/openghg/openghg/pull/1234)

Fixed

- Bug where attributes were not preserved during some resampling operations. [PR 1233](https://github.com/openghg/openghg/pull/1233)

Updated
- Update `standardise_column` inputs to include more explicit keywords around selection of satellite points. This includes adding the `obs_region` keyword to describe an area selected for satellite points (not necessarily the same as `domain`) and updating the definition of `selection` to be linked to any additional selection filters included for the satellite data. [PR 1217](https://github.com/openghg/openghg/pull/1217/)
- Update `standardise_footprint` inputs to include more explicit keywords around selection of satellite points. This includes adding the `obs_region` keyword to describe an area selected for satellite points (not necessarily the same as `domain`) and updating the definition of `selection` to be linked to any additional selection filters included for the satellite data. [PR 1218](https://github.com/openghg/openghg/pull/1218/)
- Output of parsers changed from nested dictionary to list of `MetadataAndData` objects. [PR 1199](https://github.com/openghg/openghg/pull/1199)

Added

- Added parser to process and add "NIWA" network data to the object store. [PR 1208](https://github.com/openghg/openghg/pull/1208)
- Improved resampling of variability when number of observations is present. Also added methods for customising resampling, and a `Registry` class to "register" functions. [PR 1156](https://github.com/openghg/openghg/pull/1156)
- Allow parsers to return a list of `MetadataAndData` directly. [PR 1222](https://github.com/openghg/openghg/pull/1222)

Fixed

- Changed `icos_data_level` to `data_level` in `ObsSurface.store_data` to fix bug where ICOS data was not distinguished by data level. [PR 1211](https://github.com/openghg/openghg/pull/1211)
- Fixed permissions for file locks [PR 1221](https://github.com/openghg/openghg/pull/1221)

Fixed

- Added `align_metadata_attributes` to retrieve_remote and shifted function defination to standardise/meta. [PR 1197](https://github.com/openghg/openghg/pull/1197)
- Added `icos flags` to handle data that is flagged bad for remote icos data.[PR 1200](https://github.com/openghg/openghg/pull/1200)
- Pinned Zarr to `2.18.3` as github runners are picking `zarr 3.0` which is still in significant development state.[PR 1205](https://github.com/openghg/openghg/pull/1205)
- Added exist_ok = true argument to create_config_folder and removed ObjectStoreError call.[PR 1198](https://github.com/openghg/openghg/pull/1198)

Fixed

- Fixed options used with `xr.Dataset.to_zarr` in reponse to updates in xarray. [PR 1160](https://github.com/openghg/openghg/pull/1160)
- Added xfail for `cfchecker` tests due to broken link. [PR 1178](https://github.com/openghg/openghg/pull/1178)
- Removed duplicate code from `read_file` method in `BoundaryConditions` and `EulerianModel`. [PR 1192](https://github.com/openghg/openghg/pull/1192)

Added

- Check for file lock permissions with helpful error message. File locks are now created with rw permissions for user and group. [PR 1168](https://github.com/openghg/openghg/pull/1168)
- Removed parsers that are unused. [PR 1129](https://github.com/openghg/openghg/pull/1129)
- Added `data_owner` and `inlet_height_magl` as attributes to parse_icos. [PR 1147](https://github.com/openghg/openghg/pull/1147)
- Align the dataset(s) when opening the data while standardising footprints data to prevent error due to misalign coordinates. [PR 1164](https://github.com/openghg/openghg/pull/1164)
- Moved `sync_surface_metadata` to ObsSurface.read_file function so this is applied for all input data regardless of source_format. [PR 1138](https://github.com/openghg/openghg/pull/1138)
- Added `parse_co2_games` parser splitting multiple model from one file into separate datasources. [PR 1170](https://github.com/openghg/openghg/pull/1170)
- Added `dobj_url` as attribute to icos `retrieve_atmospheric` function. [PR 1174](https://github.com/openghg/openghg/pull/1174)
- Added parser for BoundaryConditions class. [PR 1180](https://github.com/openghg/openghg/pull/1180)
- Added parser for Eulerian Model class. [PR 1181](https://github.com/openghg/openghg/pull/1181)

Updated

- Removed serverless/cloud code since it is not being used. [PR 1177](https://github.com/openghg/openghg/pull/1177)
- Minimum version of python to 3.10. [PR 1175](https://github.com/openghg/openghg/pull/1175)
- Updated ICOS standardise function to reflect changes in ASCII file format. [PR 1140](https://github.com/openghg/openghg/pull/1140)
- Added `rename_vars` option to `get_obs_surface` to allow variable names based around species to be returned. [PR 1130](https://github.com/openghg/openghg/pull/1130)
- Added option to `get_obs_column` to return the column data directly rather than converting to mole fractions. [PR 1131](https://github.com/openghg/openghg/pull/1131)
- Made calculations in `ModelScenario._calc_modelled_obs_HiTRes` more efficient. [PR 1062](https://github.com/openghg/openghg/pull/1062)
- Updated type hints and `typing` imports for Python 3.10 [PR 1193](https://github.com/openghg/openghg/pull/1193)

Fixed

- Bug where `search_surface` couldn't accept a dictionary argument for `data_level`. [PR 1133](https://github.com/openghg/openghg/pull/1133)
- GIT_TAG variable passed to build step of release_conda and also environment activation is executed for publishing to conda step. [PR 1135](https://github.com/openghg/openghg/pull/1135)

Updated

- Updated parse_* functions for surface data type to accept `filepath` rather than `data_filepath`. This maps better to the `standardise_surface` input and makes this consistent with the other data types. [PR 1101](https://github.com/openghg/openghg/pull/1101)
- Separated data variable formatting from assign attributes function into dataset_formatter function.- [PR 1102](https://github.com/openghg/openghg/pull/1102)
- Required and optional keys for "column" data type were updated to reflect the two sources of this data (site, satellite) [PR 1104](https://github.com/openghg/openghg/pull/1104)
- Ensure metadata keywords for NOAA obspack are consistent with wider definitions including renaming data_source to dataset_source (data_source would be "internal"). [PR 1110](https://github.com/openghg/openghg/pull/1110)
- Updated data type classes to dynamically select inputs to pass to parse function and to include any required/optional keys not passed to the parse function within the metadata. [PR 1111](https://github.com/openghg/openghg/pull/1111)
- When adding new data sources, updated how lookup keys add optional keys. This used to only extract these from the optional_metadata input but this now allows keys to be added through any metadata. [PR 1112](https://github.com/openghg/openghg/pull/1112)
- Formalising metadata data merging logic within new util.metadata_util functions. [PR 1113](https://github.com/openghg/openghg/pull/1113)
- Splited build and publish steps in the workflow and check for `-` and `.` in the tags for build and publish. [PR 759](https://github.com/openghg/openghg/pull/759)

Fixed

- Bug where a datasource's folder in the `data` directory was not deleted by `Datasource.delete_all_data()`. This was causing `check_zarr_store` in `util/_user.py` to give a false negative. [PR 1126](https://github.com/openghg/openghg/pull/1126)
- Bug where an input filepath list to standardise_surface was only storing the last file hash. This allowed for some files to bypass the check for the same files depending on where they were in the original filepath list. [PR 1100](https://github.com/openghg/openghg/pull/1100)
- Bug where filepath needed to be a Path object when storing the file hash values. [PR 1108](https://github.com/openghg/openghg/pull/1108)
- Catch an `AttributeError` when trying synchronise attributes and metadata and a user passes a `bool` - [PR 1029](https://github.com/openghg/openghg/pull/1029)
- Mypy issue fixed for `util.download_data()` function based on updates described [requests Issue 465](https://github.com/psf/requests/issues/465) and included in [urllib3 PR 159](https://github.com/urllib3/urllib3/pull/159/files). This allowed the `decode_content` flag to be set directly rather than needing to patch the method. [PR #1118](https://github.com/openghg/openghg/pull/1118)