Orquestra-sdk

What's Changed
* fix: Ensure decimal types are converted by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/82
* chore: Update Changelog for release by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/83


**Full Changelog**: https://github.com/zapatacomputing/orquestra-workflow-sdk/compare/v0.45.0...v0.45.1

What's Changed
* chore: shift changelog by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/72
* fix: Switch to `cloudpickle` for pickling by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/71
* Chore: remove skipping service integration tests by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/73
* feat: Scale up CE workflows by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/76
* feat: Apply resources to Ray remote functions by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/77
* feat: Version metadata added to IR by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/78
* docs: All dependencies in quotes by VolodyaCO in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/80
* chore: Update Changelog for release by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/81

New Contributors
* VolodyaCO made their first contribution in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/80

**Full Changelog**: https://github.com/zapatacomputing/orquestra-workflow-sdk/compare/v0.44.0...v0.45.0

Changelog

🚨 *Breaking Changes*
* Removed FluentBit-related CLI options: `orq {up,down} --fluentbit` flag. Logs produced by the local Ray runtime are read directly by the SDK now. This only affects users who used the experimental integration with FluentBit docker container.
* `GitImport` will no longer be downloaded automatically when using Ray locally. This reverts behavior to `v0.42.0`.
* Internal configuration environment variables have changed.


🔥 *Features*
* Secrets can now be used inside workflow functions
* `sdk.secrets.get("name")` will now use passport-based authorization if `ORQUESTRA_PASSPORT_FILE` environment variable is set. Otherwise, passing a valid `config_name="..."` is required.
* Bump Ray version to 2.3
* `GithubImport` can be used with a username and a secret referring to a "personal access token" to enable private GitHub repositories on Compute Engine. Server side support coming soon!


🐛 *Bug Fixes*
* Getting full logs produced by Ray workflows. Previously, the dictionary returned by `logs_dict = wf_run.get_logs()` had just a single entry: `{"logs": ["task 1 log", "task 1 log", "task 2 log", "task 2 log"]}`. Now, the dictionary has a correct shape: `{"task_invocation_id1": ["task 1 log", "task 1 log"], "task_invocation_id2": ["task 2 log", "task 2 log"]}`.
* Getting single task logs. Previously `orq task logs` would raise an unhandled exception. Now, it prints the log lines.
* Workflow run IDs inside logs on CE now match the expected run ID.


💅 *Improvements*
* `orq wf view` now shows `TaskInvocationID`s instead of `TaskRunID`s. This improves usage of `orq wf view` with other CLI commands that require passing invocation ID, like `orq task {logs,results}`.
* `sdk.WorkflowRun.wait_until_finished()` will now print workflow status every now and then.


*Internal*
* Git URL model changed inside the IR
* `orq up` will now configure Ray's Plasma directory


PRs
* chore: update changelog after the release by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/45
* docs: describe workaround for Ray & M1 by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/46
* feat: dorq auto login by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/41
* internal: remove obsolete iter_logs by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/49
* fix: tag Ray logs with proper task IDs by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/50
* feat: Allow secrets in workflow functions and IR by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/52
* feat: IR Secrets in runtimes by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/53
* chore: Update ctrl-c handling in orq login by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/54
* chore: Pin dill to a release by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/55
* docs: remove recipes by max-radin in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/48
* fix: reading single-task logs on Ray by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/51
* refactor: split API module by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/58
* chore: add missing changelog entry by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/57
* feat: use passport auth whenever possible by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/56
* refactor: harden mypy typechecks by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/59
* chore: update Compute Engine error handling by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/61
* fix!: remove fluentbit by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/63
* fix: showing task inv IDs by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/60
* fix: Make downloading Gitimports optional in Ray by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/62
* chore: Configure Ray plasma storage by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/66
* chore: Expand Git URL model by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/65
* feat: Private Github repo support by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/67
* fix: Ray: Set workflow run ID from environment by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/68
* imp: print when polling when awaiting wf completion by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/69
* Feat: bump ray to 2.3 by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/64
* fix: Make Ray dirs before starting Ray by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/70


**Full Changelog**: https://github.com/zapatacomputing/orquestra-workflow-sdk/compare/v0.43.0...v0.44.0

What's Changed

🚨 Breaking Changes
* Brand-new `orq` CLI with simplified command tree and interactive prompts when a required argument isn't passed. New commands:
* `orq workflow submit`
* `orq workflow view`
* `orq workflow list`
* `orq workflow stop`
* `orq workflow logs`
* `orq workflow results`
* `orq wf` as a shorthand for `orq workflow`
* `orq task logs`
* `orq task results`
* `orq up`
* `orq down`
* `orq status`
* `orq login`
* `sdk.WorkflowRun.get_logs()` doesn't accept any arguments any more. Now, it returns all the logs produced by the tasks in the workflow. If you're interested in only a subset of your workflow's logs, please consider using one of the following filtering options:
python
from orquestra import sdk
from orquestra.sdk.schema.workflow_run import State

wf_run = sdk.WorkflowRun.by_id("foo")

logs = wf_run.get_logs()
Option 1
single_task_logs = logs["my_inv_id"]

Option 2
logs_subset = {id: lines for id, lines in logs.items() if id in ["foo", "bar", "baz"]}

Option 3
for task in wf_run.get_tasks():
if task.get_status() == State.FAILED:
print(task.get_logs())

* `sdk.WorkflowRun.get_artifacts()` doesn't accept any arguments any more. Now, it returns all the artifacts produced by the tasks in the workflow.
* `sdk.TaskRun.get_logs()` returns a list of log lines produced by this task. Previously, it returned a dictionary with one entry.
* Executing a workflow on Ray with Git imports will now install them. A known limitation is that this will only work for Git repositories that are Python packages and will fail for Git repositories that are not Python packages.
* The API will no longer accept `config_save_file` as optional parameters, from now on if you want to use a different config file use the `ORQ_CONFIG_PATH` environment variable.


🔥 Features

* `list_workflow_runs` added to the Public API. This lets you list the workflows for a given config, for example `sdk.list_workflow_runs("ray")` or `sdk.list_workflow_runs("prod-d")`.


🐛 Bug Fixes

* Fixed broken link on docs landing page.
* Internal logs from Ray are no longer displayed.
* Fixed the docstrings for `sdk.WorkflowRun.get_artifacts()`. It returns a dictionary with `TaskInvocationID` as keys and whatever the task returns as values.
* Fixed bug where some log line from Ray may be duplicated when viewing logs
* Tasks with duplicate imports will no longer fail when running on QE
* AST parser will no longer print a lot of "Info" messages
* `sdk.WorkflowRun.get_logs()` now only returns logs produced by the user. Previously, it included internal debug messages produced by Ray.
* Logs from workflows submitted to Ray are now always returned as JSONL lines


New Contributors
* max-radin made their first contribution in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/12
* BenjaminMummery made their first contribution in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/14

**Full Changelog**: https://github.com/zapatacomputing/orquestra-workflow-sdk/compare/v0.42.0...v0.43.0

What's Changed
* ci: Add missing makefile by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/2
* Fix: Fix logging scenarios when no config was passed by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/1

New Contributors
* alexjuda made their first contribution in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/2
* SebastianMorawiec made their first contribution in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/1

**Full Changelog**: https://github.com/zapatacomputing/orquestra-workflow-sdk/commits/v0.42.0