Kamu

Fixed
- Associating correct input dataset that was hard compacted with the error during transformation of derived dataset

Added
- REST API: The `/query` endpoint now supports response proofs via reproducibility and signing (816)
- REST API: New `/{dataset}/metadata` endpoint for retrieving schema, description, attachments etc. (816)
Fixed
- Fixed unguaranteed ordering of events when restoring event sourcing aggregates
- Enqueuing and cancelling future flows should be done with transactions taken into account (via Outbox)

Fixed
- Fixed crash when a derived dataset is manually forced to update while an existing flow
for this dataset is already waiting for a batching condition

Added
- Persistency has been enabled for Task and Flow domains.
Both `TaskExecutor` and `FlowExecutor` now fully support transactional processing mode,
and save state in Postgres or Sqlite database.
- Tasks now support attaching metadata properties. Storing task->flow association as this type of metadata.
- Flows and Tasks now properly recover the unfinished requests after server restart
Changed
- Simplified database schema for flow configurations and minimized number of migrations
(breaking change of the database schema)
- Introduced `pre_run()` phase in flow executor, task executor & outbox processor to avoid startup races
- Explicit in-memory task queue has been eliminated and replaced with event store queries
- Get Data Panel: use SmTP for pull & push links
- GQL api method `setConfigCompaction` allows to set `metadataOnly` configuration for both root and derived datasets
- GQL api `triggerFlow` allows to trigger `HARD_COMPACTION` flow in `metadataOnly` mode for both root and derived datasets

Added
- Container sources allow string interpolation in env vars and command
- Private Datasets, changes related to Smart Transfer Protocol:
- `kamu push`: added `--visibility private|public` argument to specify the created dataset visibility
- Send the visibility attribute in the initial request of the push flow
Changed
- Schema propagation improvements:
- Dataset schema will be defined upon first ingest, even if no records were returned by the source
- Schema will also be defined for derivative datasets even if no records produced by the transformation
- Above ensures that datasets that for a long time don't produce any data will not block data pipelines
- Smart Transfer Protocol:
- Use `CreateDatasetUseCase` in case of creation at the time of the dataset pulling
- Now requires the `x-odf-smtp-version` header, which is used to compare client and server versions to prevent issues with outdated clients

Added
- Private Datasets, ReBAC integration:
- ReBAC properties update based on `DatasetLifecycleMessage`'s:
- `kamu add`: added hidden `--visibility private|public` argument, assumed to be used in multi-tenant case
- GQL: `DatasetsMut`:
- `createEmpty()`: added optional `datasetVisibility` argument
- `createFromSnapshot()`: added optional `datasetVisibility` argument