Kamu

Added
- Support for multi-tenant workspaces in Jupyter Notebook extension
- Support for GraphQL multi-tenant mode:
- Added new endpoint for querying engine supported login methods
- Added `AuthenticationService` and `AuthenticationProvider` concepts that implement login functionality
- CLI authentication provider: login same as password with preconfigured accounts in `.kamucliconfig` files
- Login issues and interprets Kamu-specific JWT tokens
- GraphQL queries are expected to attach JWT tokens as Bearer authentication header
- Modeling anonymous account sessions
- Login guards in GraphQL write operations
- Simple model for dataset permission queries
- Login instructions and feature flags are sent as configuration in `kamu ui` mode
- Implemented previously mocked account resolution API
Fixed
- Failing transform operations in multi-tenant workspaces due to invalid propagation of dataset aliases
Changed
- Updated WEB UI image to latest release 0.10.0
- GitHub OAuth functionality isolated in a separate component `kamu-adapter-oauth`
- GraphQL: filtering datasets based on logged account
- Unified and clarified namings in account-related data structures

Fixed
- `RecordsFormat` Utf8 issue when truncating strings
Changed
- Updated `kamu-base:latest-with-data` image to use new DF-ingest-based datasets

Fixed
- Ignoring the downloads cache when `--fetch-uncacheable` flag is used
- Restored pre-sorting of events by `event_time` within one data slice in DataFusion-based ingest
- Performance degradation in local file copying due to async
- Race condition in Zip decompress step that caused file truncation

Fixed
- `datafusion` ingest will not crash on empty inputs when schema inference is enabled
Added
- Added a warning when fetch cache is used to resume ingest: `Using cached data from X minutes ago (use kamu system gc to clear cache)`

Added
- **Experimental:** Data ingest using `DataFusion` engine
- It's an entirely new implementation of data readers and merge strategies
- It's often over **100x faster** than the `Spark`-based ingest as it has near-instant startup time (even avoids container overhead)
- New merge strategies can work directly over S3 without downloading all data locally
- It supports all existing data formats (Parquet, CSV, NdJson, GeoJson, Shapefile)
- Some advanced CSV / Json reader options are not yet implemented, most notably `timestampFormat`
- `Spark` is still used by default for compatibility. To start using `DataFusion` declare (a potentially no-op) `preprocess` step in your root dataset manifest ([see example](examples/currency_conversion/ca.bankofcanada.exchange-rates.daily.yaml))
- `Spark`-based ingest will be removed in future versions with `DataFusion` becoming the default, however we are planning to support `Spark` and all other engines in the `preprocess` step, while `DataFusion` will still be handling the initial reading of data and merging of results
Changed
- All examples where possible are now using `DataFusion` ingest

Added
- Prototyped authorization checks for CLI functionality based on OSO-framework:
- for now the assumption is that all datasets are public
- public datasets can be read by anyone, but written only by owner
- authorization checks (Read, Write) integrated into every known CLI command or underlying service