Echopype

Overview

This is a minor release that includes an important performance enhancement for combining large volumes of data residing in individual files into a single entity, a number of bug fixes, and other smaller improvements.

New features
- Overhaul `combine_echodata` function
- Allow combine a large number of `EchoData` objects exceeding memory limits (808, 824, 830)
- Remove reversed time check from combine_echodata (835)
- Add minimal ZarrCombine test (826)
- Order the channel coordinate in file conversion to ensure consistent combination across files (818)
- Revise outdated data combination behavior (797, 799)
- Clean up all coordinate and attribute details under `combine_echodata` function (848, 849)
- Track provanance for filenames of raw data files and auxiliary files
- Propagate `xml_path` as `meta_source_filenames` to combined echodata (814, 852)
- Standardize handling of `source_filenames` in individual and combined echodata as well as downstream datasets (804, 806)

Under the hood enhancements
- Clean up functions for setting encoding in the converted files (851)
- Add `requests` and `aiohttp` to dependency (844)
- Pin `netcdf4` to be <1.6 for pypi package due to ongoing `netcdf-c` problem (843)
- Write `Parsed2Zarr` generated files to `temp_echopype_output/parsed2zarr_temp_files` (832)
- Change `isel` to `sel` to fix `compute_Sv` to allow working with dask array (828)
- Add `open_raw(offload_to_zarr=True)` integration tests (794)

Bug fixes
- Fix regression bug with interpolating environmental variables to `ping_time` grid (837, 856)
- Fix WindowsPath error with compute_Sv when run on Windows (829)
- Fix logic problem in `open_raw(offload_to_zarr=True)` that sometimes cause problems (794, 853)

Infrastructure
- Update CI to use micromamba (805)
- Fix version string on CI (804, 820)
- Rename `ci.yaml` to `build.yaml` for clarity (807)

Overview

This is a minor release that includes a few new features and memory efficiency-related changes that make echopype better.

New features
- Add a new subpackage `consolidate` that contains functions to consolidate data across the calibrated Sv dataset and the corresponding raw-converted file
- Add function `consolidate.add_location` to interpolate location to calibrated dataset (749)
- Add function `consolidate.add_depth` to convert `range_meter` to `depth` with information on transducer tilt and depth (738)
- Move function that swaps the channel with frequency dimension (now as `consolidate.swap_dims_channel_frequency`) (738)
- Add new functionality to allow control of logging outputs (772)

Under the hood enhancements
- Improve memory usage while converting files that require significant NaN-padding and previously would incur very large memory expansion (774)
- This is achieved by directly writing variables that may incur a large memory expansion into a temporary zarr store
- Beta function that will benefit from user feedback
- Overhaul access pattern for EchoData (762)
- Remove previous access pattern for different groups in the raw-converted file
- Starting from this release all groups are accessed with `echodata["GROUP_PATH"]`, e.g., `echodata["Platform"]`, `echodata["Sonar/Beam_group1"]`, etc.
- Make long_name in ds_power for EK80 consistent with other sonar model (771)
- Modify `set_beam()` so it returns a list (780)
- Change the order in `_save_groups_to_file` so the order of groups is preserved when opening a converted netcdf file (779)
- Remove the user option to select NMEA sentences in `open_raw` to ensure raw data is preserved (778)

Infrastructure
- Update CI set up to avoid exceeding GitHub actions memory limitation (761)
- decrease the number of workers from 4 to 2

Overview

This is a minor release that includes important bug fixes, a number of new features, and some leftover data format changes specific to parsed AD2CP data format.

Bug fixes
- We use `datatree` under the hood for the `EchoData` object, but `datatree` v0.0.4 had a bug in accessing subgroups in netCDF data model in Windows OS. This was fixed in `datatree` v0.0.6, and hence we updated our dependency and made other associated changes to support cross platform users (732, 748)
- Fix empty Sv problem related to renaming time coordinate associated with environmental parameters used for calibration; this is a regression bug from v0.6.0 changes (755)
- Fix a bug in `compute_MVBS` in selecting `echo_range` for specific frequency. This is a regression bug from v0.6.0 changes (736)
- Allow parsing data from EK60 with split-beam transducers but without phase/angle data (491, 718)
- Check filter coeffs existence in `SetGroupsEK80.set_vendor` before saving (720, 724)
- Fix the check in `compute_MVBS` for handling different variations of NaN entries in the Sv dataset (753)
- Fix invalid timestamp issue in AD2CP data conversion (733)

New features
- Enhance `update_platform` to support a new use case (location data from fixed location) and add more consistency (741)
- Ability to parse and store RAW4 datagram for EK80 data (714)
- Add utility function for swapping `channel` coordinate with `frequency_nominal` (710)
- Add ES70, ES80, EA640 to allowed data type for calibration (759)

Changes of netCDF data model
- Reorganize AD2CP data variables into different `Sonar/Beam_groupX`s and different first-level groups in a form consistent with v0.6.0 changes for all other sonar models (731); some variables remain to be discussed and may change in future releases (719)

Enhancements
- Refactor AD2CP conversion to improve speed and memory usage through removal of xr.merge (505)
- Update Python requirements in docs to >=3.8 (744)

Infrastructure
- Update PR action to use PR title [all tests ci] to run the entire suite of tests and [skip ci] to skip all tests (721)

Overview

This is a major release that contains changes that enhances the compliance of echopype data model (and hence generated file structure) to the [SONAR-netCDF4 convention Version 1.0 ](http://www.ices.dk/sites/pub/Publication%20Reports/Cooperative%20Research%20Report%20(CRR)/CRR341.pdf). In addition, some variables were renamed to improve intuitive understanding of sonar data, provenance and standardized attributes are added to the processed dataset (e.g. Sv), the deprecated old API (<0.5.0) was removed, and some bugs were fixed.

Changes of netCDF data model

- Move and rename the original `Beam` and `Beam_power` group to be subgroups under the `Sonar` group, in order to comply with the structure defined in the convention (567, 574, 605, 606, 611)
- `Beam` --> `Sonar/Beam_group1`: contains either raw power or power/angle data for all sonar models other than EK80. For EK80, if only complex or power/angle data exist, all data are in this group; if _both_ complex and power/angle data exist, the complex data are in this group.
- `Beam_power` --> `Sonar/Beam_group2`: contains power/angle when complex data occupies `Sonar/Beam_group1`; only exists for EK80 data when _both_ power/angle data and complex data bothexist in the file
- Rename the coordinate `range_bin` to `range_sample` to make it obvious that this coordinate indicates the digitization sample number for the sound waveform or intensity time series, and hence it takes a form of sequential integers 0, 1, 2, 3, ... (595)
- Rename the data variable `range` in the calibrated Sv or TS dataset to `echo_range`, so that it is not confused with the python built-in function (590)
- Rename the coordinate `quadrant` for EK80 data to `beam` (619)
- Add coordinate `beam` with length 1 for all sonar models, except for AD2CP (638, 646)
- Rename the data variable `Sp` to `TS` since "point backscattering strength" is a Simrad terminology and target strength (TS) is clearly defined and widely used. (615)
- Rename time dimensions in the `Platform` group (`location_time`: `time1`, `mru_time`: `time2`) (518, 631, 647)
- Rename the coordinate `frequency` to `channel` for all groups, to be more flexible (can accommodate channels with identical frequencies 490) and reasonable (since for broadband data the channel frequency is only nominal 566) (657)
- Rename the data variable `heave` to `vertical_offset` in the Platform group (592, 623)
- Change `src_filenames` string attribute to `source_filenames` list-of-strings variable (620, 621)
- Bring consistency to the names of the time coordinates for the `Platform` and `Environment groups` (656, 672)

Changes of `EchoData` group access pattern

- The groups can now be accessed via a path in the form `echodata["Sonar/Platform"]`, `echodata["Sonar/Platform"]`, `echodata["Sonar/Beam_groupX"]`, etc. using [DataTree v0.0.4](https://github.com/xarray-contrib/datatree/tree/0.0.4) functionalities. (#611)
- The previous access pattern `echodata.platform`, `echodata.sonar`, `echodata.beam`, etc. is deprecated and will be removed in v0.6.1.

Addition of attributes and variables in raw-converted and processed data

- Add indexing info for `Beam_groupX` as data variable under the `Sonar` group (658)
- Add missing coordinate and variable attributes in the processed datasets Sv, MVBS, TS(594)
- Add `water_level` to processed datasets (Sv, MVBS, TS) for convenient downstream conversion to depth (259, 583, 615)
- Add additional environment variables for EK80 data (616)
- Add missing platform data variables (592, 631, 649, 654)

New features and other enhancements

- Add parser for Echoview ECS file (510)
- Add provenance to raw-converted and processed datasets (621)
- Consolidate convention specs into a single yml file for pre-loading when creating `EchoData` objects (565)
- Extra visualization module can now handle both `frequency` and `channel` filtering, as well as files with duplicated frequencies (660)
- Improve selection of Vendor-specific calibration parameters for narrowband EK data (697)

CI improvements

- Upgrade python minimum to 3.8 and tidy up packaging (604, 608, 609)
- Upgrade echopype development status to Beta (610)
- Update `setup-services.py` to include images & volumes subtleties (651)

Other changes

- Remove the deprecated old (<0.5.0) API (506, 601)
- Update README in the `echopype/test_data` folder (584)
- Add documentation for visualization (655)
- Add development roadmap to documentation (636, 688)
- Restructure and expand data format section (635)

This is a minor release that contains an experimental new feature and a number of enhancements, clean-up and bug fixes, which pave the way for the next major release.

New feature
- (beta) Allow interpolating CTD data in calibration (464)
- Interpolation currently allowed along the `ping_time` dimension (the `"stationary"` case) and across `latitude` and `longitude` (the `"mobile"` case).
- This mechanism is enabled via a new `EnvParams` class at input of calibration functions.

Enhancements
- Make visualize module fully optional with `matplotlib`, `cmocean` being optional dependency (526, 559)
- Set range entries with no backscatter data to NaN in output of `echodata.compute_range()` (547) and still allows quick visualization (555)
- Add `codespell` GitHub action to ensure correct spellings of words (557)
- Allow `sonar_model="EA640"` for `open_raw` (before it had to be "EK80") (539)

Bug fixes
- Allow using `sonar_model="EA640"` (538, 539)
- Allow flexible and empty environment variables in EA640/EK80 files (537)
- Docstring overhaul and fix bugs in `utils.uwa` (525)

Documentation
- Upgrade echopype docs to use jupyter book (543)
- Change the RTD `latest` to point to the `dev` branch (467)

Testing
- Update convert tests to enable parallel testing (556)
- Overhaul tests (523, 498)
- use `pytest.fixture` for testing
- add ES70/ES80/EA640 test files
- add new EK80 small test files with parameter combinations
- reduce size for a subset of large EK80 test data files
- Add packaging testing for the `dev` branch (554)

This is a minor release that includes new features, enhancements, bug fixes, and linking to an echopype preprint.

New features
- Allow converting ES60/70/80 files and handle various datagram anomaly (409)
- Add simple echogram plotting functionality (beta) (436)

Enhancements
- `update_platform` method for `EchoData` now include proper variable attributes and correctly selects time range of platform data variables corresponding to those of the acoustic data (476, 492, 493, 488)
- Improve testing for `preprocess.compute_MVBS` by running through real data for all supported sonar models (454)
- Generalize handling of Beam group coordinate attributes and a subset of variable attributes (480, 493)
- Allow optional kwargs when loading `EchoData` groups to enable delaying operations (456)

Bug fixes
- The gain factor for band-integrated Sv is now computed from broadband calibration data stored in the Vendor group (when available) or use nominal narrowband values (446, 477)
- Fix time variable encoding for `combine_echodata` (486)
- Fix missing `ping_time` dimension in AZFP Sv dataset to enable MVBS computation (453)
- Fix bugs re path when writing to cloud (462)

Documentation
- Improvements to the "Contributing to echopype" page: Elaborate on the git branch workflow. Add description of PR squash and merge vs merge commit. Add instructions for running only a subset of tests locally (482)
- Add documentation about `output_storage_options` for writing to cloud storage (482)
- Add documentation and docstring for `sonar_model` in `open_raw` (475)
- Improve documentation of EchoData object by adding a sample of the xarray Dataset HTML browser (503)

Others
- Zenodo badge update (469)
- Add github citation file (496), linking to [echopype preprint on arXiv](https://arxiv.org/abs/2111.00187)