Echopype

Overview

This release includes a number of new features and performance enhancements, as well as fixes for bugs and warnings. We have fully switched to using the [trunk-based](https://www.atlassian.com/continuous-delivery/continuous-integration/trunk-based-development) development workflow in this release.


New features
* Add noise and attenuation functions from Ryan et. al. 2015 and echopy by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1316
* Add `.bot` and `.idx` files to EK60/EK80 parsing by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1318
* Add index aggregation option for cleaning mask functions by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1326
* Added alpha-version code for new AZFP6 format by dash-uvic in https://github.com/OSOceanAcoustics/echopype/pull/1323
* Use latest MRU parser version and enable `ep.consolidate.add_location` can be based off datagram type by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1359

Enhancements
* Support lazy-loaded EK80 broadband-complex data by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1311
* Allow `MVBS` into `ep.mask.apply_mask`, and add alignment check for target variable and mask by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1345
* Enhance `ep.calibrate.compute_Sv` scalability by enabling EchoData chunking and removing eager computation by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1331
* Use Echodata `Platform` and `Beam_group1` variables to compute depth for EK60/EK80 by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1319
* Allow waveform mode "FM" in calibrate by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1363
* Allow external arrays in `add_depth` and add an universal ping time alignment function by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1369

Bug and warning fixes
* Fix duck array `RuntimeWarning` for EK60 `ep.open_raw` by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1337
* Coding: `channel_mode` dtype from `float16` to `float32` by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1350 and https://github.com/OSOceanAcoustics/echopype/pull/1351
* Fix problems in applying `compute_Sv` on combined `EchoData` object by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1328
* Fix data type check for padding short complex pings by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1353
* Update `xr.drop` to `xr.drop_vars` by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1364

Documentation
* Update docs for visualization by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1347
* Merge data processing level changes from `stable` by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1355
* Add the Ping acoustic library to other resources by gavinmacaulay in https://github.com/OSOceanAcoustics/echopype/pull/1362
* Update contributing guide and README/index by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1356 and https://github.com/OSOceanAcoustics/echopype/pull/1373

Others
* Remove the deprecated `visualize` subpackage by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1336
* Temporarily skip datagrams generated by EC150 by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1357
* Modify dask image in requirement.txt by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1361

New Contributors
* dash-uvic made their first contribution in https://github.com/OSOceanAcoustics/echopype/pull/1323

**Full Changelog**: https://github.com/OSOceanAcoustics/echopype/compare/v0.8.4...v0.9.0

Overview

This release includes a few performance enhancements, bug fixes, and improving warnings/error messages. We are also preparing to switch to using the [trunk-based](https://www.atlassian.com/continuous-delivery/continuous-integration/trunk-based-development) development workflow in the next release to streamline new contributions.

Enhancements
* Upgrade `compress_pulse` to use parallelized convolution by anantmittal in https://github.com/OSOceanAcoustics/echopype/pull/1208
* Optimize `harmonize_env_param_time` by anantmittal in https://github.com/OSOceanAcoustics/echopype/pull/1235
* Optimize frequency differencing with dask by anantmittal in https://github.com/OSOceanAcoustics/echopype/pull/1198
* Optimize `get_vend_cal params_power` by anujsinha3 in https://github.com/OSOceanAcoustics/echopype/pull/1285
* `apply_mask` changes: Allow multi-channel masks, allow the `depth` dimension, and simplify fill value by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1230
* Update `_calc_Sv_offset` for AZFP parsing by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1304
* Move `compute_Sv_offset` to inside of AZFP parser by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1307
* Support calibration of EK80 broadband complex data from sequential pinging by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1302
* Add `skipna` argument to `compute_MVBS/NASC` to control mean behavior over NaN elements by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1269

Bug fixes and warnings
* Fix error parsing integer in frequency-differencing equation by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1278
* Fix time encoding to use int64 and "nanoseconds since 1970-01-01 00:00:00Z" by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1299
* Add warnings and raised errors for `add_location` by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1296

Others
* One time `dev` merge to `main` to facilitate switching to trunk-based workflow by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1297
* Engaging in Google Summer of Code 2024 program by leewujung https://github.com/OSOceanAcoustics/echopype/pull/1266, https://github.com/OSOceanAcoustics/echopype/pull/1267, https://github.com/OSOceanAcoustics/echopype/pull/1274
* Update Development roadmap by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1262
* Deprecating `echopype.visualize` by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1310

New Contributors
* anantmittal made their first contribution in https://github.com/OSOceanAcoustics/echopype/pull/1208
* anujsinha3 made their first contribution in https://github.com/OSOceanAcoustics/echopype/pull/1285
* ctuguinay made their first contribution in https://github.com/OSOceanAcoustics/echopype/pull/1230

**Full Changelog**: https://github.com/OSOceanAcoustics/echopype/compare/v0.8.3...v0.8.4

Overview

This release includes a bug fix for changes from the previous release and a few functionality enhancements.

Enhancements
- Add parser support for EK80 MRU1 datagram (1242)
- Add support for `consolidate` subpackage functions to accept both in-memory or stored datasets (1216)
- Add test for ES60 spare field decoding issue (1233)
- Add test for EK80 missing `receiver_sampling_freq` error (1234)

Bug fixes
- Fixed reshape bug in `pad_shorter_ping` that was remnant from `use_swap` full refactoring (1234)

Overview

This release includes a number of important performance enhancements, bug fixes, and under-the-hood refactoring to improve code readability.

Enhancements
- Full refactor of the functionality to directly storing parsed data to zarr before set_groups_* (1185, 1218, 1070)
- This avoids memory expansion during NaN padding across channel
- Improve `compute_MVBS` using flox (1124)
- Revive and improve `compute_NASC` using flox (1167)
- Refactor AZFP XML parser
- Improved XML parsing by using `xml.etree.ElementTree` to be consistent with other XML parsers in echopype (1135)
- Parse more parameters and store in the resulting `EchoData` objects (1135)
- Add partial support for AZFP multiple phase settings (1182)
- Overhaul AZFP Environment group
- Set mandatory variables not in data to NaN (`sounds_speed_indicative`, `absorption_indicative`) (1226)
- Parse and store AZFP pressure data when exist (1189, 1226)
- Add `utils.misc.depth_from_pressure` function for converting pressure to depth (1207)
- Remove unused mechanism to selectively parse some EK datagrams (1214)
- Enhancements of `EchoData.update_platform` method
- Factor out utility functions to a submodule (1209)
- Auto-assign timestamp for fixed-location external data using first `ping_time`(1196)
- Add support for `consolidate` subpackage functions to accept both in-memory or stored datasets (1216) [NOT MERGED]

Bug fixes
- Fix `EchoData.to_zarr` encoding (1128)
- Eliminate preferred chunks
- Ensuring chunk alignment and encoding only when handling dask arrays
- Fix scaling bugs in `compute_NASC` (1167)
- Fix `UnicodeDecodeError` for ES60 files (1215)
- Handle missing `receiver_sampling_frequency` for EK80 data (1219)

Infrastructure and refactoring
- Add support for running individual test files (1166)
- Add module based testing (1180)
- Update CI to barebone python, without conda (1192)

Overview

This release includes a bug fix and a few infrastructure upgrades and enhancements.

Enhancements and bug fix
- More intuitive input syntax for `frequency_differencing` (1106)
- Set EK60/80 Platform and NMEA nan timestamp to first `ping_time` value (1154)
- This avoids warnings due to xarray's new behavior with nan timestamp in coordinate
- Remove automatic version conversion in `open_converted` (1143)

Infrastructure
- Bump testing minimum Python version to 3.9 (1146)
- Experiment with Python 3.11 in testing (1148, 1156)
- Some tests related to `parsed2zarr` fail due to the pinned `more_itertools` version in python 3.11. `parsed2zarr` is being overhauled in 1070 so these are deferred to the next release.
- Remove logging during testing (1141)
- Remove NASC import causing failure (1145)

Overview

This release includes important updates to the raw-converted data format based on adaptation of convention, an efficiency boost for combining multiple `EchoData` objects, packaging and infrastructure upgrades, and other updates necessitated by these changes.

Enhancement
- Overhaul the `combine_echodata` function (1042, 1116)
- No longer require spinning up dask client under the hood during combine
- Use `xr.concat` directly compatible with delayed computation
- No longer require destination path as input argument
- Overhaul `EchoData.update_platform` (1060)
- Allow adding external variables with arbitrary number of time dimensions, with new time dimensions following an increasing order (e.g., `time3`, `time4`, etc)
- Only allow updating standardized, sonar model-dependent Platform group variables

Raw-converted data format changes
- Drop the `beam` and `ping_time` dimensions for some parameters (1056, 1083)
- The dimensions of these parameters were previously expanded to conform with convention requirements, but this inflated the data volume, especially for in memory operations, and made the calibration code confusing
- Standardize `backscatter_r/i` long_name in the `Sonar/Beam_groupX` group, and correct units (1047)
- Standardize use of `transmit_frequency_start/stop` for all echosounder models (1091)
- Move filter coefficients and decimation factor to variables in EK80 `Vendor_specific` group (1044, 1046, 1105)
- Add new `Provenance` group `combination_*` attributes to combined `EchoData` object, mirroring the `conversion_*` attributes (1113)
- Bring more consistency in the `Platform` group across sensors on conversion (1058, 1061)
- Add missing mandatory variables and other data format changes (1094, 1099, 1101, 1102, 1103, 1104, 1107, 1114)
- Set time arrays explicitly to datetime64[ns] in parsers and remove duplicated timestamp stuff (1117)

Packaging and infrastructure
- Remove Pandas<2 pinning (1080)
- Pin netcdf to >1.6 and add explicit encoding (1112)
- Update CI to prepare for python 3.11 (1108)
- Switch github action for get-changed-files (1118)
- Fix discrepancy on test log output (1119)

Others
- Set and fix encoding for variable encoding (1072)
- Specify handling for expected string type
- Sanitize and set encoding as part of EchoData
- Handling problematic NMEA messages when setting lat/lon in Platform group (1067)
- Update permission check filename to be UUID to make it thread safe (1110)
- use `pulse_form` for EK80 transmit type checks (1091)
- Factor out a normalization factor from pulse compression function for flexible use (1105)
- Add existence checking for angle parameters in `consolidate.add_splitbeam_angle` (1105)
- Revise qc function to clean up reversed time (1065)
- Update matplotlib call to register EK500 colormap (1068)
- Temporarily remove `compute_NASC` for debugging and overhaul (1136)
- Remove the deprecated `preprocess` subpackage (1077, 1082)
- Remove the `ZarrCombine` module since it is no longer used (1117)
- Remove outdated `ui_param` (1071)

Documentation
- Add rendering for example `EchoData` objects for all supported echosounders to show the updated format (1089)
- Restructure TOC to have separate Raw data formats and Data Processing sections, each with new subpages (1093, 1132, 1134)
- Add discussion on z-axis vertical coordinate system and variables (1133)
- Add bibliography page linked from inline citations (1137)