Echopype

Overview

This release includes a bug fix and a few infrastructure upgrades and enhancements.

Enhancements and bug fix
- More intuitive input syntax for `frequency_differencing` (1106)
- Set EK60/80 Platform and NMEA nan timestamp to first `ping_time` value (1154)
- This avoids warnings due to xarray's new behavior with nan timestamp in coordinate
- Remove automatic version conversion in `open_converted` (1143)

Infrastructure
- Bump testing minimum Python version to 3.9 (1146)
- Experiment with Python 3.11 in testing (1148, 1156)
- Some tests related to `parsed2zarr` fail due to the pinned `more_itertools` version in python 3.11. `parsed2zarr` is being overhauled in 1070 so these are deferred to the next release.
- Remove logging during testing (1141)
- Remove NASC import causing failure (1145)

Overview

This release includes important updates to the raw-converted data format based on adaptation of convention, an efficiency boost for combining multiple `EchoData` objects, packaging and infrastructure upgrades, and other updates necessitated by these changes.

Enhancement
- Overhaul the `combine_echodata` function (1042, 1116)
- No longer require spinning up dask client under the hood during combine
- Use `xr.concat` directly compatible with delayed computation
- No longer require destination path as input argument
- Overhaul `EchoData.update_platform` (1060)
- Allow adding external variables with arbitrary number of time dimensions, with new time dimensions following an increasing order (e.g., `time3`, `time4`, etc)
- Only allow updating standardized, sonar model-dependent Platform group variables

Raw-converted data format changes
- Drop the `beam` and `ping_time` dimensions for some parameters (1056, 1083)
- The dimensions of these parameters were previously expanded to conform with convention requirements, but this inflated the data volume, especially for in memory operations, and made the calibration code confusing
- Standardize `backscatter_r/i` long_name in the `Sonar/Beam_groupX` group, and correct units (1047)
- Standardize use of `transmit_frequency_start/stop` for all echosounder models (1091)
- Move filter coefficients and decimation factor to variables in EK80 `Vendor_specific` group (1044, 1046, 1105)
- Add new `Provenance` group `combination_*` attributes to combined `EchoData` object, mirroring the `conversion_*` attributes (1113)
- Bring more consistency in the `Platform` group across sensors on conversion (1058, 1061)
- Add missing mandatory variables and other data format changes (1094, 1099, 1101, 1102, 1103, 1104, 1107, 1114)
- Set time arrays explicitly to datetime64[ns] in parsers and remove duplicated timestamp stuff (1117)

Packaging and infrastructure
- Remove Pandas<2 pinning (1080)
- Pin netcdf to >1.6 and add explicit encoding (1112)
- Update CI to prepare for python 3.11 (1108)
- Switch github action for get-changed-files (1118)
- Fix discrepancy on test log output (1119)

Others
- Set and fix encoding for variable encoding (1072)
- Specify handling for expected string type
- Sanitize and set encoding as part of EchoData
- Handling problematic NMEA messages when setting lat/lon in Platform group (1067)
- Update permission check filename to be UUID to make it thread safe (1110)
- use `pulse_form` for EK80 transmit type checks (1091)
- Factor out a normalization factor from pulse compression function for flexible use (1105)
- Add existence checking for angle parameters in `consolidate.add_splitbeam_angle` (1105)
- Revise qc function to clean up reversed time (1065)
- Update matplotlib call to register EK500 colormap (1068)
- Temporarily remove `compute_NASC` for debugging and overhaul (1136)
- Remove the deprecated `preprocess` subpackage (1077, 1082)
- Remove the `ZarrCombine` module since it is no longer used (1117)
- Remove outdated `ui_param` (1071)

Documentation
- Add rendering for example `EchoData` objects for all supported echosounders to show the updated format (1089)
- Restructure TOC to have separate Raw data formats and Data Processing sections, each with new subpages (1093, 1132, 1134)
- Add discussion on z-axis vertical coordinate system and variables (1133)
- Add bibliography page linked from inline citations (1137)

Overview

This release includes important updates to the requirements, small bug fixes, and refactoring of AZFP parser code to handle files from glider deployment and improve code coherence.

Enhancement and bug fixes
- AZFP conversion enhancements
- Add error catch to handle AZFP files with no temperature or tilt, such as those from glider deployments (1020)
- Refactor AZFP parser code to improve code coherence (1024)
- Correct and add missing SONAR-netCDF4 variable attributes for AZFP (1023)
- Improve impedance parameter handling for EK80 files (1019)
- allow impedance data type to be float
- unify naming of impedance parameters
- Allow only one chirp generation option for EK80 broadband processing (1019)
- previously there was an option to use the implementation from Matlab echolab
- starting from this version only the implementation identical to those from Lars Andersen ([repo](https://github.com/CRIMAC-WP4-Machine-learning/CRIMAC-Raw-To-Svf-TSf))
- Remove unneeded return self in processing level decorator wrapper-inner function for class method (1037)

Tests and infrastructure
- Unpin xarray version and pin pandas < 2 (981)
- allow v0.7.x in `map_ep_version` checks

Overview

This release includes new features to interface with Echoview ECS files for computing Sv, reorganization of computing functions into new subpackages, addition of data processing level attributes to data products, and other improvements and bug fixes.

New features and major changes
- Allow using ECS for calibrating Simrad echosounders (996, 1004)
- This functionality is in a **beta** testing stage
- Details of implementation may change and bugs are possible
- Expand ECS parser to accept frequency-dependent values in EK80 ECS files
- Overhaul `env_params` to ensure correct intake for calibration (985)
- Now allows using `env_params` entries that are xr.DataArrays
- Move functions previously in `preprocess` subpackage to new subpackages (993). Calling these functions from `preprocess` is deprecated and will be removed in v0.7.1.
- `clean`: `remove_noise`, `estimate_noise`
- `commongrid`: `compute_MVBS`, `compute_MVBS_index_binning`
- Add `commongrid.compute_NASC` (1005)
- The current implementation uses brute force looping for mean Sv computation, this will be refactored and optimized together with other functions requiring the same pattern in an upcoming release
- Add global attributes for data processing levels (1001).
- This functionality is in a **beta** testing stage
- See [data processing level specifications](https://echopype.readthedocs.io/en/stable/processing-levels.html) for functions and conditions under which such attributes are added
- Expand `mask.apply_mask` to handle multi-channel Sv datasets (1010)
- Standardize sonar metadata for EK80 data (992)
- `sonar_serial_number` is now an empty global attribute, no longer a variable, as in the EK60 case
- `transducer_name`, `transducer_serial_number`, `transceiver_serial_number` based on parser parameter `transducer_name`, `transducer_serial_number`, and `serial_number`, respectively

Bug fixes
- Fix scaling bug for `beamwidth_alongship` and `beamwidth_athwartship` from CW-based parameters to values corresponding to center frequency of broadband transmit signals (998)

Tests and infrastructure
- Add more comprehensive tests for `add_location` (1000)
- Add test for splitbeam angle `ek80_CW_power` case (994)
- Add unit and integration tests for `env_params` intake for calibration (985)
- Exclude `test_data` folder in `codespell` pre-commit hook (1016)

Overview

This is a minor release that includes a small bug fix and an enhancement, which removes excessive warnings due to changes in v0.6.4.

Bug fix
- Fix a bug that prevented passing in `env_params` for EK60 calibration (987)

Enhancement
- Handling non-positive values that causes log10 warnings on EK calibration (986)

Overview

This is a release that includes important performance enhancements that allow user-provided calibration parameters for broadband and narrowband calibration, new functionalities to compute and attach split-beam angles to calibrated Sv dataset, perform frequency-differencing and masking, as well as a number of bug fixes and other improvements.

New features
- Allow passing in `cal_params` as a dictionary for narrowband and broadband calibration (955)
- Add default chunk encoding for zarr output (939)
- Add `add_splitbeam_angle` function to the `consolidate` subpackage (916, 971)
- Add a new `mask` subpackage
- Add `apply_mask` function to the `mask` subpackage (912)
- Add `frequency-differencing` function to the `mask` subpackage (901)
- Allow selection of a subset of channels when combining multiple echodata objects (892)
- This is done via the added `channel_selection` input argument to `combine_echodata`
- Add default consolidated flag for `echodata.to_zarr` (855)
- Generalize and improve efficiency of `compute_MVBS` (878)
- Allow `echo_range` that vary with `ping_time`
- Allow `Sv` data that are dask arrays
- Add `.nbytes` to obtain that data size of an `echodata` object (874)
- Add new data variables from the raw data files to facilitate EK80 calibration (944)
- transceiver type, transmit impedance, receive impedance, receiver sampling frequency

Enhancements and other changes
- Unify the order of dimensions of `echo_range` across all sonar models (968)
- Create and use the default echopype home directory (896, 954)
- This default directory is at `~/.echopype`
- Refactor the `calibrate` subpackage (904)
- routines for `env_params` and `cal_params` intake and range computation are now in different modules rather than methods in each classes
- Tidy up `cal_params` related routines (953)
- Revise `env_params` related routines (952)
- Improving pulse compressed broadband Sv computation and echo range computation (944)
- The resulting values are tested against pyEcholab and Echoview outputs
- Note there is unresolved discrepancy between pyEcholab/Echopype outputs with Echoview outputs for the first section of Sv values
- Change input argument names for `open_raw` (962)
- Change `offload_to_zarr` to `use_swap`
- Change `max_zarr_mb` to `max_mb`

Bug fixes
- Handling of provenance attributes in apply-mask and add-depth, especially for testing (930)
- Fix bugs in combining multiple echodata objects (the `combine_echodata` function)
- Fix meta_source_filenames bug and enable (meta)source_filenames appending of path and list (913)
- Fix bugs in `env_params` intake for calibration (952)

Infrastructure
- Update docker build for arm64 mac silicon chips (964)
- Fix windows github actions workflow (947)
- Remove mamba dependency (946)
- Fix pre-commit ci and update RTD config for jupyter-book (934)