What's Changed
PRs:
* Allow multiple locations for source import/export by AndrewNepogoda in https://github.com/RelationalAI/rai-workflow-manager/pull/23
1. Deleted CLI arguments:
- `--run-mode`
- `--dev-data-dir`
- `--output-root`
2. CLI Arguments replacement with `loader.toml`:
CLI arguments `--run-mode local`, `--dev-data-dir`, `--output-root` now replaced with local containers in `loader.toml`:
Arguments
`python main.py --run-mode local --dev-data-dir ./data --output-root ./output ...`
`loader.toml`
[[container]]
name="input"
type="local"
data_path="./data"
[[container]]
name="export"
type="local"
data_path="./output"
3. Renamed properties in `loader.toml`:
- `RAI_PROFILE` -> `rai_profile`
- `RAI_PROFILE_PATH` -> `rai_profile_path`
- `RAI_SDK_HTTP_RETRIES` ->`rai_sdk_http_retries`
4. Azure container configuration in `loader.toml`
Azure blob
AZURE_EXPORT_ACCOUNT="account"
AZURE_EXPORT_CONTAINER="dev"
AZURE_EXPORT_DATA_PATH="output"
AZURE_EXPORT_SAS="token"
Azure IMPORT blob
AZURE_IMPORT_ACCOUNT="account"
AZURE_IMPORT_CONTAINER="dev"
AZURE_IMPORT_DATA_PATH="data"
AZURE_IMPORT_SAS="token"
Replaced by:
[[container]]
name="export"
type="azure"
account="account"
container="dev"
data_path="output"
sas="token"
[[container]]
name="input"
type="azure"
account="account"
container="dev"
data_path="data"
sas="token"
5. `defaultContainer` parameter for `ConfigureSources` and `Export` steps in batch config
A new `defaultContainer` is required for `ConfigureSources` and `Export` steps and name a container which will be used as default container for import/export.
6. `container` parameter for sources and exports
`container` for sources in `ConfigureSources` and exports in `Export` steps allow user specify the name of the container which will be used for the source for import/export. This feature allow user specify multiple datasources (`local`, `azure`, `snowflake`(upcoming), `s3`(in road map)) as inputs and outputs and mix different types of datasources.
**Full Changelog**: https://github.com/RelationalAI/rai-workflow-manager/compare/v0.0.12...v0.0.13