Orquestra-sdk

:rotating_light: *Breaking Changes*
* `WorkflowRun.get_logs()` now returns a data structure with logs split into categories.
* Local workflow database has changed format: downgrading from v0.50.0 will require user intervention.

:fire: *Features*
* Add .project property to WorkflowRun to get the info about workspace and project of running workflow
* `VersionMismatch` warnings are shown only when interacting with specific workflow runs, not while listing workflows.
* Add `--qe` flag to `orq login`, this is the default so there is no change in behavior.
* Bump Ray version to 2.4.0
* New API method `WorkflowRun.start_from_ir()` that allows to start workflow run having only IR object
* The WorkflowLogs object returned from `WorkflowRun.get_logs()` now includes Compute Engine system logs for workflow runs using CE.

:bug: *Bug Fixes*
* Secrets with workspaces now work inside workflow functions and for personal access tokens in `GithubImport`.
* `list_workspaces` and `list_projects` work inside Studio with `auto` config
* Listing workflows on QE no longer fails if there was a CE workflow in the DB

:nail_care: *Improvements*
* Add prompters to `orq wf submit` command for CE runtime if workspace and project weren’t passed explicitly
* Auto-select and highlight current workspace and project when using `auto` config in CLI in studio
* `orquestra-sdk-base` CPU container image has a 20% size reduction.
* Added `State` enum to the base `orquestra.sdk` package for easier filtering task runs.
* Logs fetched from CE are now split into “task” and “env setup” categories.


:page_with_curl: *Docs*
* Update resource management guide for Compute Engine
* Add section for using custom images on Compute Engine

What's Changed
* chore: Update Changelog by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/179
* Feat: Add project property to workflow run to return workspace and project by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/173
* Imp: Prompt for workspace and project while submitting a workflow via CLI by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/181
* chore: Add QE CLI flag by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/168
* feat: suppress VersionMismatch warnings when listing wf runs by BenjaminMummery in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/174
* feat: secret with workspaces in IR and GithubImport by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/185
* ci: add `imp`/`improvement` prefix to the PR template by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/172
* chore: Add ruff to github workflows by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/180
* ci: run pyright by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/177
* Internal: Use external URI when using auto config on cluster instead of internal one by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/190
* docs: some .rst typos and missing link by 1ethanhansen in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/184
* refactor!: logs datastructure by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/187
* chore: Bump Ray to 2.4.0 by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/137
* fix: Reduce CPU base image size by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/160
* Feat: New API to start workflow runs from IR by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/175
* fix: Skip CE workflows when listing workflows in QE by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/192
* imp: re-export State in the base package by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/193
* docs: Update for custom images and Compute Engine by emre-aydin in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/186
* Imp: Improve workspace and project prompters in studio showing and auto-selecting current context by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/182
* docs: Update documentation as we disable single node runtime by emre-aydin in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/197

New Contributors
* 1ethanhansen made their first contribution in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/184

**Full Changelog**: https://github.com/zapatacomputing/orquestra-workflow-sdk/compare/v0.49.1...v0.50.0

What's Changed
* chore: Update CHANGELOG.md by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/176
* Fix: Fix prompters for workspaces and projects by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/178


**Full Changelog**: https://github.com/zapatacomputing/orquestra-workflow-sdk/compare/v0.49.0...v0.49.1

What's Changed
* Feat: Prompt for workspace and project in all CLI commands that prompted for WF id by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/149
* internal: Fix CUDA image publishing workflow failure when no Docker tag is given by emre-aydin in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/152
* Install SSH Client by emre-aydin in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/155
* chore: roll changelog by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/156
* feat: add sdk.get_backend_ids() function. by BenjaminMummery in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/150
* Internal: Fix random failures due to the warning of socket not being closed by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/161
* Feat: cli easy re login by BenjaminMummery in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/154
* feat: Check tokens before saving in orq by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/158
* fix: Tasks that set n_outputs=1 in QE and in-process by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/164
* feat: include workspace and project as parameters for listing workflows on CE by BenjaminMummery in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/153
* Feat: add support for Workspaces to all Secret-related functions by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/157
* feat: give more informative error message when ray cluster isn't running by BenjaminMummery in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/163
* internal: Add a makefile for building the sdk images, tweak for building on M1 by roryschadler in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/159
* Feat: Inherit WS and Project from Studio if used inside portal by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/162
* feat: Custom Image Support by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/166
* fix: Use default image in task resources by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/169
* impr: show subprocess error for `orq up` by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/165
* fix: Bump minimum Pydantic version by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/171
* Chore!: Removed `WorkflowDef.prepare()` and `WorkflowRun.start()` functions from public API by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/170
* Docs: Update secrets guide to take workspaces into account by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/167

New Contributors
* roryschadler made their first contribution in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/159

**Full Changelog**: https://github.com/zapatacomputing/orquestra-workflow-sdk/compare/v0.48.0...v0.49.0

What's Changed
* internal: Placeholder GitHub Actions workflow to publish orquestra-sdk-base Docker image by emre-aydin in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/138
* chore: Update changelog for 0.47.0 by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/136
* Feat: Add workspace and project parameters to "orq wf submit" CLI command by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/134
* Feat!: cleanup config api code by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/130
* Feat: New API functions: list_workspaces and list_projects by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/139
* chore: pass for spelling, imports, copyright, style, etc. by BenjaminMummery in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/121
* feat: don't return single packed values as sequences by BenjaminMummery in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/141
* fix: Set max retries in Ray by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/140
* feat: auto-resolve choice when there is only one. by BenjaminMummery in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/135
* feat: use new login endpoint by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/143
* [OP-39] orquestra-sdk-base images for CUDA and non-CUDA workflows by emre-aydin in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/142
* Internal, Fix: rewrite tests to use sockets instead of FS by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/146
* fix: `orq submit` with dirty repo by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/147
* internal: Tagging Docker images via optional parameter [OP-39] by emre-aydin in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/145
* feat: support for custom images on CE by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/148
* chore: revert 148 by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/151


**Full Changelog**: https://github.com/zapatacomputing/orquestra-workflow-sdk/compare/v0.47.0...v0.48.0

What's Changed
* chore: update changelog for 0.46.0 by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/114
* Feat: workspaceId and projectID support for wf submit by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/109
* fix: Correct changelog ordering after merge by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/116
* Fix: pin polars version used in our tests by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/118
* Feat: Add new config "self" to submit to CE from within a cluster by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/117
* fix: Retry getting results from CE after a workflow has succeeded by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/115
* fix: Do not modify arg when building Ray workflow by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/122
* docs: fix formatting in the Resource Management guide by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/119
* Fix: Fix WorkflowDef.graph function to properly generate graph by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/124
* chore: add reminder about linking JIRA tickets by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/125
* Feat: Change "self" config name to "auto". "auto" config acts as "local" if not on studio environment by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/129
* chore!: Merge breaking changes to main by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/100
* fix: Custom Image default to None by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/131


**Full Changelog**: https://github.com/zapatacomputing/orquestra-workflow-sdk/compare/v0.46.0...v0.47.0

What's Changed
* docs: guide for dependency imports by BenjaminMummery in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/74
* feat: raise an exception if workflows have zero tasks. by BenjaminMummery in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/86
* tests: Fix test isolation in docs examples by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/87
* fix: Misc QE fixes by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/84
* fix: Remove in_process runtime from CLI options by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/88
* Feat: Add start time value and sort by it to CLI commands and prompts that list WFs by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/85
* Fix: Move constant deserialization inside ray remote function by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/89
* chore: limit github action workflows by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/90
* feat: parse binary response into logs. by BenjaminMummery in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/79
* chore: Remove corq by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/93
* fix!: multi-output tasks when some outputs are unused by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/75
* refactor: remove obsolete RuntimeInterface methods by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/92
* internal: skip slow tests by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/94
* Revert "fix!: multi-output tasks when some outputs are unused (75)" by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/97
* Feat!: Remove task.model and task.import_models interfaces by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/101
* Feat: Allow setting default source imports and dependency imports for all tasks in given WF by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/102
* feat: Add workflow resources support by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/103
* fix: Allow single import as dependency_imports by jamesclark-Zapata in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/104
* Fix: default_dependency_imports as a single import allowed. Fix __main__ reference issue when default_source_import was Inline by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/106
* docs: guide on workarounds for resource congestion by BenjaminMummery in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/96
* feat: limit CE runs list. by BenjaminMummery in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/111
* feat: print debug information from orq by alexjuda in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/112
* Feat!: Remove env variable that disabled setting resources for local ray by SebastianMorawiec in https://github.com/zapatacomputing/orquestra-workflow-sdk/pull/113


**Full Changelog**: https://github.com/zapatacomputing/orquestra-workflow-sdk/compare/v0.45.1...v0.46.0