Ripple1d

Summary
This release of `ripple1d` fixes several bugs that were identified in a large production run. Those bugs were detailed and summarized in 268.

Bug Fixes
- When a new terrain is downloaded but contains only nodata values, `NullTerrainError` is raised, and the endpoint process will fail.
- When a new terrain is downloaded but the HEC-RAS CreateTerrain subprocess fails silently, the output is checked, and a `RasTerrainFailure` error is raised. Additionally, bad_projection_test.py has been added to test that the error will raise.
- When a source model is conflated such that the most downstream XS for a NWM reach a has higher river station than the most upstream reach, a `BadConflation` error is raised.
- When sub model extraction is attempted on a reach that would have one or zero cross-sections but is not tagged as eclipsed, a `SingleXSModel` error is raised. The endpoint response was also cleaned up such that eclipsed reaches no longer return a path to a nonexistent 123.ripple1d.json. Logging has been improved for all these cases.
- flow indexing in the create_flow_depth_combinations function has been improved to avoid rounding values. A test has been added for one of the models that was causing problems.
- Handling of interpolated cross-sections has been improved such that they may have sub models created, may be run, and may be used in the create FIM lib steps. A new test model -- winooski -- has been added with interpolated cross-sections.
- Null (blank) downstream reach values are now loaded as 0. This is consistent with how HEC-RAS handles them. The new winooski model contains null downstream reach length at the most downstream cross-section.

Closes 281
Closes 282
Closes 291
Closes 292
Closes 295
Closes 298

> [!IMPORTANT]
> Version 0.8.0 was released without having incorporated bugfixes added after v0.8rc1. This update adds in the necessary bugfixes to reflect the most up-to-date version of the software. The text below is a copy of the 0.8.0 release notes.

Summary

This release adds a suite of diagnostic metrics on model quality and terrain agreement to the ripple1d outputs. Model quality metrics reflect how well a HEC-RAS model conforms to standards of practice for developing HEC-RAS models and provide easy access to many model attributes that would be checked during an engineering model QC. Terrain agreement metrics quantify how well the terrain generated in the create_ras_terrain endpoint matches the source model terrain. Access to these metrics give ripple1d users actionable information to inform which HEC-RAS models should be preferred when models overlap. Furthermore, elevation-specific terrain agreement metrics can be used to alert users when a FIM product substantially disagrees with a source model geometry (indicative of lower FIM quality).

Additional features have been added to handle HEC-RAS warnings about hydraulic table start points and cross-section naming.

Features Added

Hydraulic Table Start Elevation

When a hydraulic table for a cross-section has a minimum elevation below the section invert, HEC-RAS generates a pop up warning as demonstrated in https://github.com/Dewberry/ripple1d/issues/249. In the warning, the software specifies that any subsequent saves to the geometry file will automatically adjust the hydraulic table minimum elevation to be above the section invert. To mimic this functionality, a new function call has been added to the RasGeomText constructor. The call will check for any occurrences of the htab issue, adjust the minimum elevation, and overwrite the geometry file with the updated value.

HEC-RAS implements some convoluted process to update both the htab start value and the increments. We elected to simply update the htab start elevation to the cross-section invert and preserve the original engineer-determined increments.

Sub Model Cross-Section Names

In earlier versions of ripple1d, the software attempted to carry cross-section names from source model to sub model. When that naming would have resulted in decreasing values in the upstream direction, the stationing was adjusted to preserve increasing values in the names. The renaming process led to a couple of bugs in downstream processes (ex 225 and 266). Since cross-section names do not affect hydraulic computation, the software will now rename all sub model cross-sections to an autoincremented value (ex 1, 2, 3, etc).

Additionally, the extract_submodel endpoint has been refactored to fix a potential bug in the junction walking step. For cases where more than one junction separated the u/s and d/s cross-sections or cases where the junctions were not reported from upstream to downstream in the HEC-RAS geometry file, the extract_submodel endpoint would only grab a single intermediate reach. No examples of this behavior have been found in production, but the bug was tested in development with a testing HEC-RAS model.

Terrain Agreement Metrics

A new step has been added to the create_ras_terrain endpoint that will generate a json of terrain agreement metrics. The json is saved to a file with suffix ".terrain_agreement.json" that is saved to the sub model directory. The JSON contains a suite of metrics that have been documented in the Technical Summary of the readthedocs page (https://ripple1d.readthedocs.io/en/latest/tech_summary.html) and the create_ras_terrain endpoint page (https://ripple1d.readthedocs.io/en/latest/endpoints/create_ras_terrain.html).

Model Quality Metrics

A new step has been added to the extract source model endpoint which extracts source model information to determine the quality of the source model. These metrics are populated in the cross section layer located in the source model geopackage layer for every cross section. The submodel will inherit these metrics during the subset gpkg phase.

Bug Fixes
- A feature was added in v0.7.0 to allow html rendering of the jobs endpoint, however, the endpoint was not updated to accept a flag toggling between json and html responses. Additionally, an oversight in the project manifest meant that the html and css templates for the jobs page were not transferred to ripple1d when installed via pip. This has been resolved so that the jobs endpoint returns a json response by default, and an html response will be returned when the key-value pair f=html is in the query string.
- readthedocs was failing to build our documentation on their server. The docs folder has been updated such that it will build via their service. Additionally, the changelog will autogenerate now. Some old windows dependencies and imports have been removed since they are not used to run HEC-RAS any more and needed unnecessary logic to avoid breaking the readthedocs build service.

Closes 118
Closes 207
Closes 208
Closes 209
Closes 225
Closes 249
Closes 266
Closes 267
Closes 268
Closes 269
Closes 270

Summary

This release adds a suite of diagnostic metrics on model quality and terrain agreement to the ripple1d outputs. Model quality metrics reflect how well a HEC-RAS model conforms to standards of practice for developing HEC-RAS models and provide easy access to many model attributes that would be checked during an engineering model QC. Terrain agreement metrics quantify how well the terrain generated in the create_ras_terrain endpoint matches the source model terrain. Access to these metrics give ripple1d users actionable information to inform which HEC-RAS models should be preferred when models overlap. Furthermore, elevation-specific terrain agreement metrics can be used to alert users when a FIM product substantially disagrees with a source model geometry (indicative of lower FIM quality).

Additional features have been added to handle HEC-RAS warnings about hydraulic table start points and cross-section naming.

Features Added

Hydraulic Table Start Elevation

When a hydraulic table for a cross-section has a minimum elevation below the section invert, HEC-RAS generates a pop up warning as demonstrated in https://github.com/Dewberry/ripple1d/issues/249. In the warning, the software specifies that any subsequent saves to the geometry file will automatically adjust the hydraulic table minimum elevation to be above the section invert. To mimic this functionality, a new function call has been added to the RasGeomText constructor. The call will check for any occurrences of the htab issue, adjust the minimum elevation, and overwrite the geometry file with the updated value.

HEC-RAS implements some convoluted process to update both the htab start value and the increments. We elected to simply update the htab start elevation to the cross-section invert and preserve the original engineer-determined increments.

Sub Model Cross-Section Names

In earlier versions of ripple1d, the software attempted to carry cross-section names from source model to sub model. When that naming would have resulted in decreasing values in the upstream direction, the stationing was adjusted to preserve increasing values in the names. The renaming process led to a couple of bugs in downstream processes (ex 225 and 266). Since cross-section names do not affect hydraulic computation, the software will now rename all sub model cross-sections to an autoincremented value (ex 1, 2, 3, etc).

Additionally, the extract_submodel endpoint has been refactored to fix a potential bug in the junction walking step. For cases where more than one junction separated the u/s and d/s cross-sections or cases where the junctions were not reported from upstream to downstream in the HEC-RAS geometry file, the extract_submodel endpoint would only grab a single intermediate reach. No examples of this behavior have been found in production, but the bug was tested in development with a testing HEC-RAS model.

Terrain Agreement Metrics

A new step has been added to the create_ras_terrain endpoint that will generate a json of terrain agreement metrics. The json is saved to a file with suffix ".terrain_agreement.json" that is saved to the sub model directory. The JSON contains a suite of metrics that have been documented in the Technical Summary of the readthedocs page (https://ripple1d.readthedocs.io/en/latest/tech_summary.html) and the create_ras_terrain endpoint page (https://ripple1d.readthedocs.io/en/latest/endpoints/create_ras_terrain.html).

Model Quality Metrics

A new step has been added to the extract source model endpoint which extracts source model information to determine the quality of the source model. These metrics are populated in the cross section layer located in the source model geopackage layer for every cross section. The submodel will inherit these metrics during the subset gpkg phase.

Bug Fixes
- A feature was added in v0.7.0 to allow html rendering of the jobs endpoint, however, the endpoint was not updated to accept a flag toggling between json and html responses. Additionally, an oversight in the project manifest meant that the html and css templates for the jobs page were not transferred to ripple1d when installed via pip. This has been resolved so that the jobs endpoint returns a json response by default, and an html response will be returned when the key-value pair f=html is in the query string.
- readthedocs was failing to build our documentation on their server. The docs folder has been updated such that it will build via their service. Additionally, the changelog will autogenerate now. Some old windows dependencies and imports have been removed since they are not used to run HEC-RAS any more and needed unnecessary logic to avoid breaking the readthedocs build service.

Closes 118
Closes 207
Closes 208
Closes 209
Closes 225
Closes 249
Closes 266
Closes 267
Closes 268
Closes 269
Closes 270

Users Changelog

This release of `ripple1d` attempts to speed up the known water surface elevation (kwse) computes by parallelizing the creation of the depth grids. To do this, all kwse runs are ran without producing depth grids. From these initial kwse runs the rating curve database are created and used to inform the boundary condition for the next reach upstream. Once all of the boundary conditions are known, a second kwse run is ran to produce the depth grids in parallel. To make this happen some changes were necessary to the endpoints.

In addition to adding/modifying the endpoints, this release of `ripple1d` makes significant updates to both ripple1d execution and logging. This includes running each endpoint as a separate subprocess which allows the user to have the ability to dismiss running jobs. This should be handy for when jobs appear to have hung up. Dismissing these hung up jobs will free up cpu for new jobs.

Features Added

Write depth grids argument
A new argument boolean "write_depth_grids" was added to the "run_incremental_normal_depth" and "run_known_wse" endpoints. This allows the user to specify whether ripple1d should compute raw RAS depth grids or not.

Create rating curves database
A new endpoint called "create_rating_curves_db" was added. This endpoint is a post processing step that creates a rating curve database from RAS plan hdf results. This endpoint only requires 2 args: "submodel_directory" and "plans. The location of the rating curve database is inferred from the submodel directory. It will be located directly in the submodel directory and will be named as the network reach name; e.g., "2820002/2820002.db".

The "create_rating_curves_db" endpoint checks for the presence of the depth grid for each profile and records its existence or absence along with the plan suffix in the database table. The columns are "plan_suffix" and "map_exist" and are of type string and boolean, respectively.

Create FIM library update
The "create_fim_lib" endpoint no longer produces the rating curve database.

Update to the Ripple1d workflow
The user should take care with the args when calling the new/modified endpoints; specifically the plan names. The recommended order for calling these endpoints is:

1. run_known_wse : with "write_depth_grids" set to false and "plan_suffix" set to ["ikwse"]
2. create_rating_curves_db: with "plans" set to ["ikwse"]
3. run_known_wse: with "write_depth_grids" set to true and "plan_suffix" set to ["nd","kwse"]
4. create_rating_curves_db: with "plans" set to ["nd","kwse"]
5. create_fim_lib: with "plans" set to ["nd","kwse"]

Subprocess encapsulation
The execution of all endpoints are now encapsulated within subprocesses

Logs and database

- Huey and flask logs have been combined into a single log file (server-logs.jsonld).
- Huey.db has been renamed to jobs.db
- The process id for each endpoint are now tracked in a p_id field of the task_status table in jobs.db
- A huey_status field has been added to the task_status table. This field tracks the execution status of the endpoint subprocess.
- A new table called task_logs has been added to jobs.db. This table contains stdout, stderr, and results stemming from endpoint subprocesses.
- A proof of concept graphical (html-based) job status page has been added

Closes 187
Closes 188
Closes 237
Closes 238
Closes 239
Closes 240

Users Changelog
This release of `ripple1d` fixes several bugs identified during testing.

Bug Fixes
- Technical Documentation has been updated with high level summary of package functionality.
- The ID column was removed from geopackage layers. All code dependencies on the ID column have been removed.
- Now only reaches that are connected via the "to_id" are considered eclipsed reaches.
- Precision has been added to the rating curves used to inform the incremental normal depth runs by reducing the amount of rounding.
- CRS is now stored as WKT instead of EPSG in the ripple.json file to more robustly represent all possible CRSs; e.g., ESRI.

Closes 59
Closes 133
Closes 192
Closes 219
Closes 221