Echopype

Overview

This release includes critical updates to migration to support python 3.12, use the new `xr.DataTree` functionality, and many other changes to resolve deprecation warnings and small bugs. There are a number of pinned dependencies we are continuing to work on.

Enhancements
* Drop ping time duplicates by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1382
* Support Python 3.12 and partial migration to Xarray DataTree by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1419
* Fix tests that fail from new Xarray variable and attribute assignment updates by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1433
* Add backward compatibility of raw-converted dataset with xr.DataTree by oftfrfbf leewujung ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1447
* Add assign actual range utility function by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1435

Bug and warning fixes
* Assemble AD2CP timestamp with nanosecond precision by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1436
* Use `import_resources.files` instead of the legacy `open_text` by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1434
* Replace `pkg_resources.resource_string` with `importlib.resources.files` by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1468
* Fix invalid `\d` warning by using raw string by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1448
* Check if there exist any swap files before cleaning them up by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1451
* Remove `__setattr__` from EchoData by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1457
* Compute dask array before np array equal by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1452
* `Dataset.dims` to `Dataset.sizes` by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1453
* Chunks as dictionaries in `_get_auto_chunk` by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1454
* Set decode timedelta to False since it will default to this in later xarray version by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1462
* Fixed logic bug in AZFP6 parser in validating parameters by ctuguinay leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1450
* Fill in NaN for missing EK80 coefficients by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1458

Infrastructure
* Update workflows to use python 3.12 and ubuntu 22.04 by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1425 and https://github.com/OSOceanAcoustics/echopype/pull/1466
* Bump codespell version, add exceptions by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1438
* Pin `zarr` and `netcdf4` temporarily by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1429
* Add `type-extensions` to requirements.txt by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1440
* Add manual trigger to pypi workflow by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1442
* Pin `scipy` to temporarily ensure that rotation matrix calculation does not fail by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1460

Others
* Update cff citation file by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1443
* Replace previous arXiv citation with recent ICES paper citation by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1455


**Full Changelog**: https://github.com/OSOceanAcoustics/echopype/compare/v0.9.1...v0.10.0

Overview

This release includes a number of small fixes and enhancements. An important component is to set the max supported python version to <3.12, before we fully migrate to using the new [`xarray.DataTree`](https://github.com/pydata/xarray/blob/main/DATATREE_MIGRATION_GUIDE.md) in the next release with breaking changes.

Enhancements
* Remove min max attributes for `compute_MVBS` by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1380
* Update setup-services.py by spacetimeengineer in https://github.com/OSOceanAcoustics/echopype/pull/1398
* Add support for AZFP 130kHz by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1412
* Update calibrate_base.py to include EA640 in backscatter size check. by torsteinnh in https://github.com/OSOceanAcoustics/echopype/pull/1407

Bug and warning fixes
* Handle `NaN` in `apply_mask` input mask by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1376
* EK60 Conversion: Drop channels that don't have power data by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1383
* Skip Parsing Invalid EK80 Environment Datagram(s) by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1387
* Fix 'Division by Zero' error when converting EK80 files without Sound Velocity Profile Depth value(s) by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1381
* Add error warning for all NaN lat/lon in `get_distance_from_latlon` by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1403
* Updated paths in `test_align_to_ping_time_glider_azfp` to use pytest fixture by oftfrfbf in https://github.com/OSOceanAcoustics/echopype/pull/1414

Documentation
* Update installation doc page by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1389

Others
* Set max python version <3.12 [all tests ci] by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1422

New Contributors
* spacetimeengineer made their first contribution in https://github.com/OSOceanAcoustics/echopype/pull/1398
* oftfrfbf made their first contribution in https://github.com/OSOceanAcoustics/echopype/pull/1414
* torsteinnh made their first contribution in https://github.com/OSOceanAcoustics/echopype/pull/1407

**Full Changelog**: https://github.com/OSOceanAcoustics/echopype/compare/v0.9.0...v0.9.1

Overview

This release includes a number of new features and performance enhancements, as well as fixes for bugs and warnings. We have fully switched to using the [trunk-based](https://www.atlassian.com/continuous-delivery/continuous-integration/trunk-based-development) development workflow in this release.


New features
* Add noise and attenuation functions from Ryan et. al. 2015 and echopy by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1316
* Add `.bot` and `.idx` files to EK60/EK80 parsing by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1318
* Add index aggregation option for cleaning mask functions by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1326
* Added alpha-version code for new AZFP6 format by dash-uvic in https://github.com/OSOceanAcoustics/echopype/pull/1323
* Use latest MRU parser version and enable `ep.consolidate.add_location` can be based off datagram type by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1359

Enhancements
* Support lazy-loaded EK80 broadband-complex data by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1311
* Allow `MVBS` into `ep.mask.apply_mask`, and add alignment check for target variable and mask by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1345
* Enhance `ep.calibrate.compute_Sv` scalability by enabling EchoData chunking and removing eager computation by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1331
* Use Echodata `Platform` and `Beam_group1` variables to compute depth for EK60/EK80 by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1319
* Allow waveform mode "FM" in calibrate by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1363
* Allow external arrays in `add_depth` and add an universal ping time alignment function by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1369

Bug and warning fixes
* Fix duck array `RuntimeWarning` for EK60 `ep.open_raw` by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1337
* Coding: `channel_mode` dtype from `float16` to `float32` by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1350 and https://github.com/OSOceanAcoustics/echopype/pull/1351
* Fix problems in applying `compute_Sv` on combined `EchoData` object by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1328
* Fix data type check for padding short complex pings by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1353
* Update `xr.drop` to `xr.drop_vars` by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1364

Documentation
* Update docs for visualization by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1347
* Merge data processing level changes from `stable` by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1355
* Add the Ping acoustic library to other resources by gavinmacaulay in https://github.com/OSOceanAcoustics/echopype/pull/1362
* Update contributing guide and README/index by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1356 and https://github.com/OSOceanAcoustics/echopype/pull/1373

Others
* Remove the deprecated `visualize` subpackage by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1336
* Temporarily skip datagrams generated by EC150 by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1357
* Modify dask image in requirement.txt by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1361

New Contributors
* dash-uvic made their first contribution in https://github.com/OSOceanAcoustics/echopype/pull/1323

**Full Changelog**: https://github.com/OSOceanAcoustics/echopype/compare/v0.8.4...v0.9.0

Overview

This release includes a few performance enhancements, bug fixes, and improving warnings/error messages. We are also preparing to switch to using the [trunk-based](https://www.atlassian.com/continuous-delivery/continuous-integration/trunk-based-development) development workflow in the next release to streamline new contributions.

Enhancements
* Upgrade `compress_pulse` to use parallelized convolution by anantmittal in https://github.com/OSOceanAcoustics/echopype/pull/1208
* Optimize `harmonize_env_param_time` by anantmittal in https://github.com/OSOceanAcoustics/echopype/pull/1235
* Optimize frequency differencing with dask by anantmittal in https://github.com/OSOceanAcoustics/echopype/pull/1198
* Optimize `get_vend_cal params_power` by anujsinha3 in https://github.com/OSOceanAcoustics/echopype/pull/1285
* `apply_mask` changes: Allow multi-channel masks, allow the `depth` dimension, and simplify fill value by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1230
* Update `_calc_Sv_offset` for AZFP parsing by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1304
* Move `compute_Sv_offset` to inside of AZFP parser by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1307
* Support calibration of EK80 broadband complex data from sequential pinging by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1302
* Add `skipna` argument to `compute_MVBS/NASC` to control mean behavior over NaN elements by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1269

Bug fixes and warnings
* Fix error parsing integer in frequency-differencing equation by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1278
* Fix time encoding to use int64 and "nanoseconds since 1970-01-01 00:00:00Z" by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1299
* Add warnings and raised errors for `add_location` by ctuguinay in https://github.com/OSOceanAcoustics/echopype/pull/1296

Others
* One time `dev` merge to `main` to facilitate switching to trunk-based workflow by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1297
* Engaging in Google Summer of Code 2024 program by leewujung https://github.com/OSOceanAcoustics/echopype/pull/1266, https://github.com/OSOceanAcoustics/echopype/pull/1267, https://github.com/OSOceanAcoustics/echopype/pull/1274
* Update Development roadmap by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1262
* Deprecating `echopype.visualize` by leewujung in https://github.com/OSOceanAcoustics/echopype/pull/1310

New Contributors
* anantmittal made their first contribution in https://github.com/OSOceanAcoustics/echopype/pull/1208
* anujsinha3 made their first contribution in https://github.com/OSOceanAcoustics/echopype/pull/1285
* ctuguinay made their first contribution in https://github.com/OSOceanAcoustics/echopype/pull/1230

**Full Changelog**: https://github.com/OSOceanAcoustics/echopype/compare/v0.8.3...v0.8.4

Overview

This release includes a bug fix for changes from the previous release and a few functionality enhancements.

Enhancements
- Add parser support for EK80 MRU1 datagram (1242)
- Add support for `consolidate` subpackage functions to accept both in-memory or stored datasets (1216)
- Add test for ES60 spare field decoding issue (1233)
- Add test for EK80 missing `receiver_sampling_freq` error (1234)

Bug fixes
- Fixed reshape bug in `pad_shorter_ping` that was remnant from `use_swap` full refactoring (1234)

Overview

This release includes a number of important performance enhancements, bug fixes, and under-the-hood refactoring to improve code readability.

Enhancements
- Full refactor of the functionality to directly storing parsed data to zarr before set_groups_* (1185, 1218, 1070)
- This avoids memory expansion during NaN padding across channel
- Improve `compute_MVBS` using flox (1124)
- Revive and improve `compute_NASC` using flox (1167)
- Refactor AZFP XML parser
- Improved XML parsing by using `xml.etree.ElementTree` to be consistent with other XML parsers in echopype (1135)
- Parse more parameters and store in the resulting `EchoData` objects (1135)
- Add partial support for AZFP multiple phase settings (1182)
- Overhaul AZFP Environment group
- Set mandatory variables not in data to NaN (`sounds_speed_indicative`, `absorption_indicative`) (1226)
- Parse and store AZFP pressure data when exist (1189, 1226)
- Add `utils.misc.depth_from_pressure` function for converting pressure to depth (1207)
- Remove unused mechanism to selectively parse some EK datagrams (1214)
- Enhancements of `EchoData.update_platform` method
- Factor out utility functions to a submodule (1209)
- Auto-assign timestamp for fixed-location external data using first `ping_time`(1196)
- Add support for `consolidate` subpackage functions to accept both in-memory or stored datasets (1216) [NOT MERGED]

Bug fixes
- Fix `EchoData.to_zarr` encoding (1128)
- Eliminate preferred chunks
- Ensuring chunk alignment and encoding only when handling dask arrays
- Fix scaling bugs in `compute_NASC` (1167)
- Fix `UnicodeDecodeError` for ES60 files (1215)
- Handle missing `receiver_sampling_frequency` for EK80 data (1219)

Infrastructure and refactoring
- Add support for running individual test files (1166)
- Add module based testing (1180)
- Update CI to barebone python, without conda (1192)