Kamu

Changed
- If a polling/push source does not declare a `read` schema or a `preprocess` step (which is the case when ingesting data from a file upload) we apply the following new inference rules:
- If `event_time` column is present - we will try to coerce it into a timestamp:
- strings will be parsed as RFC3339 date-times
- integers will be treated as UNIX timestamps in seconds
- Columns with names that conflict with system columns will get renamed
- All tests related to databases use the `database_transactional_test` macro
- Some skipped tests will now also be run
- Access token with duplicate names can be created if such name exists but was revoked (now for MySQL as well)
- Updated `sqlx` crate to address [RUSTSEC-2024-0363](https://rustsec.org/advisories/RUSTSEC-2024-0363)
Fixed
- Derivative transform crash when input datasets have `AddData` events but don't have any Parquet files yet

Changed
- **Breaking:** Using DataFusion's [`enable_ident_normalization = false`](https://datafusion.apache.org/user-guide/configs.html) setting to work with upper case identifiers without needing to put quotes everywhere. This may impact your root and derivative datasets.
- Datafusion transform engine was updated to latest version and includes JSON extensions
- **Breaking:** Push ingest from `csv` format will default to `header: true` in case schema was not explicitly provided
- Access token with duplicate names can be created if such name exists but was revoked
- Many examples were simplified due to ident normalization changes
Fixed
- Crash in `kamu login` command on 5XX server responses
- The push smart protocol now delivers internal errors to the client
Added
- HTTP sources now include `User-Agent` header that defaults to `kamu-cli/{major}.{minor}.{patch}`
- Externalized configuration of HTTP source parameters like timeouts and redirects
- CI: build `sqlx-cli` image if it is missing

Added
- The `/ingest` endpoint will try to infer the media type of file by extension if not specified explicitly during upload.
This resolves the problem with `415 Unsupported Media Type` errors when uploading `.ndjson` files from the Web UI.
- Private Datasets, preparation work:
- Added SQLite-specific implementation of ReBAC repository
- Added SQLite-specific implementation of `DatasetEntryRepository`
- `internal-error` crate:
- Added `InternalError::reason()` to get the cause of an error
- Added methods to `ResultIntoInternal`:
- `map_int_err()` - shortcut for `result.int_err().map_err(...)` combination
- `context_int_err()` - ability to add a context message to an error
- Added macro `database_transactional_test!()` to minimize boilerplate code
Changed
- Upgraded `sqlx` crate to v0.8
- Renamed `setConfigSchedule` GQL api to `setConfigIngest`. Also extended
`setConfigIngest` with new field `fetchUncacheable` which indicates to ignore cache
during ingest step

Fixed
- Add `reset` ENUM variant to `dataset_flow_type` in postgres migration

Added
- Reliable transaction-based internal cross-domain message passing component (`MessageOutbox`), replacing `EventBus`
- Metadata-driven producer/consumer annotations
- Immediate and transaction-backed message delivery
- Background transactional message processor, respecting client idempotence
- Persistent storage for flow configuration events
Changed
- Upgraded to `datafusion v41` (713)
- Introduced use case layer, encapsulating authorization checks and action validations, for first 6 basic dataset scenarios
(creating, creating from snapshot, deleting, renaming, committing an event, syncing a batch of events),
- Separated `DatasetRepository` on read-only and read-write parts
- Isolated `time-source` library
Fixed
- E2E: added additional force off colors to exclude sometimes occurring ANSI color sequences
- E2E: modify a workaround for MySQL tests

Fixed
- Add `recursive` field to `Reset` flow configurations in GQL Api which triggers `HardCompaction` in `KeepMetadataOnly` mode flow for each owned downstream dependency