Kamu

Added
- Added database migration & scripting to create an application user with restricted permissions
- `kamu delete` command will respect dependency graph ordering allowing to delete multiple datasets without encountering dangling reference

Added
- Support `List` and `Struct` arrow types in `json` and `json-aoa` encodings

Fixed
- Open Telemetry integration fixes

Changed
- Major dependency upgrades:
- DataFusion 42
- HTTP stack v.1
- Axum 0.7
- latest AWS SDK
- latest versions of all remaining libs we depend on
- Outbox refactoring towards true parallelism via Tokio spaned tasks instead of futures
Fixed
- Failed flows should still propagate `finishedAt` time
- Eliminate `span.enter`, replaced with instrument everywhere

Added
- REST API: New `/verify` endpoint allows verification of query commitment as per [documentation](https://docs.kamu.dev/node/commitments/#dispute-resolution) (831)
Changed
- Outbox main loop was revised to minimize the number of transactions:
- split outbox into planner and consumption jobs components
- planner analyzes current state and loads a bunch of unprocessed messages within a 1 transaction only
- consumption jobs invoke consumers and detect their failures
- Detecting concurrent modifications in flow and task event stores
- Improved and cleaned handling of flow abortions at different stages of processing
- Revised implementation of flow scheduling to avoid in-memory time wheel:
- recording `FlowEventScheduledForActivation` event (previously, placement moment into the time wheel)
- replaced binary heap based time wheel operations with event store queries
- Postgres/SQLite event stores additionally track activation time for the waiting flows
- in-memory event store keeps prepared map-based lookup structures for activation time

Added
- Added first integration of Prometheus metrics starting with Outbox
- Added `--metrics` CLI flag that will dump metrics into a file after command execution
Changed
- Telemetry improvements:
- Improved data collected around transactional code
- Revised associating span objects with large JSON structures such as messages
- Suppressed several noisy, but not very useful events
- Improved Outbox stability when message consumers fail
- Similarly, Task Executor keeps executing next tasks in case running a task results in an internal error