Octue

Summary
Add further improvements and fixes to the SDK to facilitate and improve the experience and speed of digital twin development and deployment.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([427](https://github.com/octue/octue-sdk-python/pull/427))
**IMPORTANT:** There is 1 breaking change.

New features
- Allow specification of cloud output location to upload output datasets to in app configuration

Enhancements
- Add shorter name `--rm` to optional argument `--delete-topic-and-subscription-on-exit` in CLI `start` command
- Log deletion of topic and subscription in `Service.serve`
- Demote datafile upload and download messages to debug level

Fixes
- **BREAKING CHANGE:** Stop storing file paths in dataset metadata
- Ensure generating signed URLs works on Google Cloud Run as well as with a service account key
- Set default signed URL expiration to the maximum of 7 days
- Refresh credentials to get access token for blob signing
- Ensure signed URL datasets can be uploaded and downloaded
- Ensure `Datafile` instances can be copied and pickled
- Exclude query parameters from dataset names where the dataset was instantiated from a signed URL
- Ensure topics and subscriptions are always deleted in `Service.serve` and `Service.wait_for_answer` if requested
- Ensure datasets' `path` parameter doesn't change after instantiation if the default is used
- Unify local and cloud dataset metadata file formats
- Ignore signed metadata directory when calling `Dataset.from_cloud`
- Fix `Subscription.update`
- Minimise resending of questions via Pub/Sub while they are processing
- Ensure consecutive child analysis log message sections are always coloured differently
- Avoid colourising logs in Google Cloud
- Strip ANSI escape sequences from log messages handled by `GooglePubSubHandler`

Dependencies
- Update to `twined=^0.4` to allow validation of the new output location strand
- Increase the allowed range of `click` to `>=7,<9`
- Allow freedom of most dependencies within their major versions
- Update Dataflow `setup.py`

Refactoring
- Move signed URL generator mock so it's importable externally
- Rename `LOCAL_METADATA_FILENAME` to `METADATA_FILENAME`

Testing
- Test getting output dataset files from the example service
- Use `Child` instead of `Service` in cloud run deployment test

<!--- END AUTOGENERATED NOTES --->

Summary
Add colour to the SDK's log messages that differentiate the following three things from each other:
- The basic log attributes (e.g. module name and log level)
- The analysis ID
- Log messages from any number of levels of children

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([430](https://github.com/octue/octue-sdk-python/pull/430))

New features
- Add service-differentiating colours to logs using the `seaborn` colourblind palette

Enhancements
- Separate analysis ID into its own section in the logs

Fixes
- Avoid adding `None` as a log handler in `Runner`
- Only allow one `StreamHandler` to be applied to the logger at once in `apply_log_handler`

Dependencies
- Only attempt `DataflowDeployer` import inside `dataflow deploy` CLI command

Other
- Add an `ATTRIBUTION.md` file

<!--- END AUTOGENERATED NOTES --->

Summary
Improve the interactions of datafiles with datasets and make output datasets available in the cloud via expiring signed URLs after `Analysis.finalise` has been called.

Contents (https://github.com/octue/octue-sdk-python/pull/416)
IMPORTANT: There are 3 breaking changes.

New features
- Add the ability to generate expiring signed URLs for datasets and their datafiles
- Add the ability to instantiate datasets and datafiles from signed URLs instead of cloud paths

Enhancements
- **BREAKING CHANGE:** Re-engineer `Dataset.add` so that datafiles are actually copied/uploaded/downloaded to the dataset when adding them
- **BREAKING CHANGE:** Re-engineer `Analysis.finalise` to:
- Provide signed URLs for uploaded datasets if a cloud path to a directory is provided
- Not support local data persistence
- **BREAKING CHANGE:** Remove redundant concepts of `output_dir` and output manifest path
- Add `Datafile.cloud_hash` and `Datafile.extension` properties
- Add `Dataset.bucket_name` and `Dataset.path_in_bucket` properties
- Add `Dataset.__contains__` method
- Avoid downloading cloud datafiles to determine their cloud hash values
- Multithread dataset and datafile instantiations/metadata-gathering in `Manifest` and `Dataset`
- Copy local datafiles to their new paths when setting their `local_path` properties
- Add `is_url` and `is_cloud_path` functions to `octue.cloud.storage.path`

Fixes
- Set default `Dataset` path to the current working directory
- When calling `Datafile.to_cloud`, allow uploading of the file if nothing exists at the given cloud path
- Calculate relative path of datafile to dataset properly in `Dataset.to_cloud`
- Raise errors from threads used for concurrent download/upload of data/metadata from the cloud
- Call `Analysis.finalise` inside the template apps
- Create directories if missing when setting datafile local path

Dependencies
- Use the latest `twined` version, allowing datasets to be optional if specified in `twine.json`
- Loosen the dependency on `twined` to allow freedom in the patch versions of the current minor version

Refactoring
- Reduce repetition and improve getting of buckets in `GoogleCloudStorageClient`
- Replace `GoogleCloudStorageClient._create_intermediate_local_directories` with proper usage of `os.makedirs`
- Simplify `Manifest._instantiate_dataset` and remove support for datasets with no explicit path

Testing
- Test calculation/getting of datafile hash values
- Create and use a mock for generating signed URLs to avoid the limitations of workload identity federation in CI testing

<!--- SKIP AUTOGENERATED NOTES --->

<!--- START AUTOGENERATED NOTES --->
Contents ([423](https://github.com/octue/octue-sdk-python/pull/423))

Fixes
- Update parameter name to remove typo

<!--- END AUTOGENERATED NOTES --->

Summary

Allows installation alongside django-gcp, to enable us to improve integration of octue with django

<!--- START AUTOGENERATED NOTES --->
Contents ([422](https://github.com/octue/octue-sdk-python/pull/422))

Operations
- Add dockerfile to allow correct use of poetry in devcontainer
- Update version

Dependencies
- Update google-cloud-storage range

<!--- END AUTOGENERATED NOTES --->

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([419](https://github.com/octue/octue-sdk-python/pull/419))

New features
- Support push subscriptions for Google Pub/Sub services, allowing answers to be pushed to an HTTP endpoint

Enhancements
- Add `push_endpoint` parameter to `Subscription` constructor
- Add `is_pull_subscription` and `is_push_subscription` properties to `Subscription`

Testing
- Switch to parent/child language in `Service` tests for greater clarity/consistency

<!--- END AUTOGENERATED NOTES --->