Echopop

Overview
This release adds important functions (e.g., NASC data ingestion, semivariogram fitting) and substantial documentation expansions, along with a few bug fixes.

What's Changed

New features, bug fixes, and enhancements :sparkles:

Software :desktop_computer:
- Raw Echoview export ingestion and batch processing by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/257, https://github.com/OSOceanAcoustics/echopop/pull/274
- This deprecates the need for the haul-to-transect key `*.xlsx` files
- New semivariogram fitting and optimization methods by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/251
- This includes a notebook GUI by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/258, https://github.com/OSOceanAcoustics/echopop/pull/267
- Adjustments to the kriging mesh cropping method by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/269, https://github.com/OSOceanAcoustics/echopop/pull/273
- Addressed a bug preventing INPFC from being used instead of KS for `transect_analysis` by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/272
- Upgrades to summary console printouts of analysis results brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/255

Documentation :open_book:
- **[Differences between `EchoPro` and `Echopop`](https://echopop.readthedocs.io/en/latest/echopro_vs_echopop.html)** by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/277
- Updates to the **[example workbook in the software documentation](https://echopop.readthedocs.io/en/latest/example_notebooks/example_echopop_workflow.html)** which reflect updated methods by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/258, https://github.com/OSOceanAcoustics/echopop/pull/267
- **[Updates to the description of the dataset columns in 'Input Files'](https://echopop.readthedocs.io/en/latest/input_files.html)** by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/283
- **[Description of how data files are loaded and preprocessed within `Echopop`](https://echopop.readthedocs.io/en/latest/implementation/preprocessing_data.html)** by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/274
- **[Theoretical description of how kriged biomass results can be converted into abundance and _NASC_](https://echopop.readthedocs.io/en/latest/theory/apportion_abundance.html)** by brandynlucca, leewujung in https://github.com/OSOceanAcoustics/echopop/pull/216
- Improvements to the equations in both the **[kriged abundance](https://echopop.readthedocs.io/en/latest/theory/apportion_abundance.html)** and **[kriged biomass](https://echopop.readthedocs.io/en/latest/theory/apportion_biomass.html)** apportionment document pages by leewujung in https://github.com/OSOceanAcoustics/echopop/pull/280

Infrastructure :building_construction:
- New internal checks for validating configuration and data file inputs have been added by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/248, https://github.com/OSOceanAcoustics/echopop/pull/271, https://github.com/OSOceanAcoustics/echopop/pull/275
- Methods for translating various column names into those required in `Echopop` by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/268
- Additional testing methods via `tox` by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/251


**Full Changelog**: https://github.com/OSOceanAcoustics/echopop/compare/v0.4.0...v.0.4.1

Overview
This is a major release that overhauls and renames the entire package. The changes include many bug fixes from the previous implementation, and a new set of documentation that continue to be built in subsequent releases.

What's new

New features, bug fixes, and enhancements
* Overhauled codebase with a new underlying data structure to `Survey`-class objects by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/241
* See the [updated documentation](https://echopop.readthedocs.io/) for more information on the updated data structure and example workflow
* Overhauled data loader and `core` API that defines the expected data structure with built-in validators by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/161
* Overhauled the kriging algorithm and biomass conversions by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/206
* Added new population-level calculations, stratified statistics, and other metrics by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/176
* New initialization and user-specific configuration `yaml` files to specify data files by emiliom in https://github.com/OSOceanAcoustics/echopop/pull/121

Infrastruture
* Added `pytest` tests by leewujung in https://github.com/OSOceanAcoustics/echopop/pull/174
* Configured tests to run on python 3.9, 3.10, 3.11 by leewujung in https://github.com/OSOceanAcoustics/echopop/pull/231
* Renamed package from `EchoPro` to `Echopop` by leewujung in https://github.com/OSOceanAcoustics/echopop/pull/200
* Consolidated unused files, modules, notebooks, tests by brandynlucca in https://github.com/OSOceanAcoustics/echopop/pull/227
* `pre-commit` formatting implemented by leewujung in https://github.com/OSOceanAcoustics/echopop/pull/232
* Added `Read the Docs` [documentation](https://echopop.readthedocs.io/) auto build by leewujung in https://github.com/OSOceanAcoustics/echopop/pull/182

New Contributors
* brandynlucca made their first contribution in https://github.com/OSOceanAcoustics/echopop/pull/161
* pre-commit-ci made their first contribution in https://github.com/OSOceanAcoustics/echopop/pull/238

**Full Changelog**: https://github.com/OSOceanAcoustics/echopop/compare/v0.3.1...v0.4.0

Overview

The major new feature in this release is the generation of "report" Excel files, the primary quantitative export from EchoPro. A new notebook was added to illustrate the report generation workflow. In addition, the transect subset selection scheme was upgraded to reflect the Matlab implementation (but see 95); the test infrastructure was improved, including new tests; and the documentation was enhanced with the addition of auto-generated API documentation in the Jupyter Book and an update to the repository README file.

The Python EchoPro GitHub repository was made public in April. It was previously private while initial development was taking place.

New computations and functionality

- Python EchoPro can now generate the Excel file "reports" found in Matlab EchoPro (18, PR 84). [15 reports](https://github.com/uw-echospace/EchoPro/issues/18#issuecomment-1739675989) are generated. This development entailed significant new computations to generate binned results before exporting to files, plus other computations.
- The transect subset selection scheme was updated from the previous interim scheme, to reflect the one found in Matlab EchoPro (33, PR 89). But see 95 for a remaining issue.

Tests, packaging and documentation

- New test modules, for report generation and transect subset selection (PR's 84, 89, 99).
- Generalized and simplified most of the test path generation functionality in `conftest.py` and the test modules, with better handling of base paths, including a new `HERE` path accessible from `_testing.py`. One hard-wired path still remains, to be addresses in the next release (PR's 94, 84).
- Updated installation requirements, and conda environment file after testing with the latest dependencies. Included a couple of new version pinnings that will be addressed in a future release (PR 98).
- Updated installation instructions and other information in `README.md` (PR 98).
- New `reports_workflow.ipynb` Jupyter notebook (PR's 84).
- Sanitized input file names (in Google Drive and in `survey_year_2019_config.yml`), replacing `&` with `_` to avoid issue of Google Drive downloads automatically renaming file names during zip-and-download of input files (96, PR 97).
- Added auto-generated API documentation in the Jupyter Book (PR 90).

Overview

This release includes major additions of functionality that largely complete the expected computational capabilities of Python EchoPro. Some of these additions necessitated changes in the API, such as creation and renaming of functions. Additional improvements include integration tests, packaging refinements and documentation.

New computations and functionality

- Comprehensive generation of biomass, biomass density, abundance and numerical density variables, with these variables being defined for the male and female populations over transect ("unkriged") and kriging mesh points. This work included the addition of abundance and numerical density calculations, and a harmonization of variable names related to those four core variables. (61, 70, PR 60, PR 64, PR 67)
- Generation of binned distributions of those four core variables by age and length (and sex), including distribution over transect ("unkriged") and kriging mesh points. (63, 77, PR 74, PR 78)
- API Changes.
- The API call `survey.compute_biomass_density()` was renamed to `survey.compute_transect_results()`
- Column names in some DataFrame and GeoDataFrame results were modified.
- Several new API calls that produce DataFrame and GeoDataFrame results were added. See the [Python EchoPro Workflow notebook](https://uw-echospace.github.io/EchoPro/example_notebooks/echopro_workflow.html) for a complete demonstration.

Tests, packaging and documentation

- A new testing infrastructure was added, with three sets of tests: Tests for biomass, biomass density, abundance and numerical density (by sex) over transect points; similar tests over kriging mesh points; and tests for length-age-sex distributions over both transects and kriging mesh points. (PR 67, PR 70, PR 74, PR 80)
- The packaging setup was overhauled to ensure error-free installation in a user mode, in addition to development mode. A Python wheel file is being made available for such usage, and dependency version pinning was made less strict (75, PR 68, PR 69, PR 76)
- Documentation was improved, particularly with the addition of a new Jupyter Book (https://uw-echospace.github.io/EchoPro/) hosting executed and rendered versions of the example notebooks and an overhaul of the README.md file (#75, PR 65, PR 76)

Overview

This release is focused on three key improvements:
1. Revision of input data handling, particularly the names of columns in the data files and in corresponding data frames.
2. API refinements, especially a reorganization of modules into a couple of subpackages (folders) and root-level modules; and harmonization of a core set of variable names corresponding to EchoPro results.
3. Improvements to code consistency and error checking.

All notebooks were updated to reflect these changes.

No new functionality per se was added in this release.

Input data handling

- Previous decisions on column names in input data files were revisited and implemented in both the 2019 sample input data files and the code base. Columns (variables) not being used in EchoPro computations are now excluded more clearly from the results (19, 44, PR 43, PR 47)
- A new input data file containing only the haul-to-transect mapping was introduced. It's intended to replace the use of the mapping found in the gear file (45, PR 46)

API refinements and variable harmonization

- Reorganized modules into root-level modules, subpackages (folders) `computation` and `data_loader`, and utilities in `utils`. Includes breaking up some modules into more logical groups based on functionality (48, 14, PR 54)
- Harmonized names associated with kriging and associated GeoDataFrames that hold final results (52, PR 57)
- Harmonized key variable and function names involved in the computation of biomass density (34, PR 51)
- Added arguments to map plotting functions to provide option to produce lighter-weight (fewer points) plots (49, PR 53)

Code consistency and error checking.

- Added pre-commit automatic code checking and style enforcement bots on GitHub repository (PR 58)
- Improved scheme for input dataframe column checks and associated error output messsages (28, PR 36)
- Extended cross-platform capability by allowing input configuration file paths to be pathlib Paths, converting input string paths to pathlib Paths, and usage of pathlib Paths for all input data paths created at runtime (37, PR 36)

Overview

This is the first pre-release of Python EchoPro. This release marks the completion of all core routines contained within the Matlab version of EchoPro and construction of the initial API. Specifically, the following core items have been completed:

- Creation of easily viewable DataFrames containing data related to the biological, NASC, and stratification files
- Computation of the normalized biomass density using the biological files
- Implementation of the Jolly-Hampton algorithm for both Kriged data and data without Kriging
- Routines to construct, transform, subselect, and plot the mesh associated with Kriging and the semi-variogram calculation
- Computation of the normalized semi-variogram and fitting of the semi-variogram model using ipywidgets
- A routine to perform Ordinary Kriging of the calculated biomass density over a provided mesh and plotting functions for the results
- The ability to select a subset of transects and perform any of the above core routines
- Implementation of bootstrapping (for Kriged data and data without Kriging) based on a reduction percentage of the total number of transects

2019 input files and package installation

Development and testing so far has been based on input files and corresponding Matlab EchoPro runs from the 2019 survey. The `README.md` file contains a link to download (if granted permission) the 2019 input files that can be used with this release, as well as installation instructions.