Octue

Summary
Add colour to the SDK's log messages that differentiate the following three things from each other:
- The basic log attributes (e.g. module name and log level)
- The analysis ID
- Log messages from any number of levels of children

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([430](https://github.com/octue/octue-sdk-python/pull/430))

New features
- Add service-differentiating colours to logs using the `seaborn` colourblind palette

Enhancements
- Separate analysis ID into its own section in the logs

Fixes
- Avoid adding `None` as a log handler in `Runner`
- Only allow one `StreamHandler` to be applied to the logger at once in `apply_log_handler`

Dependencies
- Only attempt `DataflowDeployer` import inside `dataflow deploy` CLI command

Other
- Add an `ATTRIBUTION.md` file

<!--- END AUTOGENERATED NOTES --->

Summary
Improve the interactions of datafiles with datasets and make output datasets available in the cloud via expiring signed URLs after `Analysis.finalise` has been called.

Contents (https://github.com/octue/octue-sdk-python/pull/416)
IMPORTANT: There are 3 breaking changes.

New features
- Add the ability to generate expiring signed URLs for datasets and their datafiles
- Add the ability to instantiate datasets and datafiles from signed URLs instead of cloud paths

Enhancements
- **BREAKING CHANGE:** Re-engineer `Dataset.add` so that datafiles are actually copied/uploaded/downloaded to the dataset when adding them
- **BREAKING CHANGE:** Re-engineer `Analysis.finalise` to:
- Provide signed URLs for uploaded datasets if a cloud path to a directory is provided
- Not support local data persistence
- **BREAKING CHANGE:** Remove redundant concepts of `output_dir` and output manifest path
- Add `Datafile.cloud_hash` and `Datafile.extension` properties
- Add `Dataset.bucket_name` and `Dataset.path_in_bucket` properties
- Add `Dataset.__contains__` method
- Avoid downloading cloud datafiles to determine their cloud hash values
- Multithread dataset and datafile instantiations/metadata-gathering in `Manifest` and `Dataset`
- Copy local datafiles to their new paths when setting their `local_path` properties
- Add `is_url` and `is_cloud_path` functions to `octue.cloud.storage.path`

Fixes
- Set default `Dataset` path to the current working directory
- When calling `Datafile.to_cloud`, allow uploading of the file if nothing exists at the given cloud path
- Calculate relative path of datafile to dataset properly in `Dataset.to_cloud`
- Raise errors from threads used for concurrent download/upload of data/metadata from the cloud
- Call `Analysis.finalise` inside the template apps
- Create directories if missing when setting datafile local path

Dependencies
- Use the latest `twined` version, allowing datasets to be optional if specified in `twine.json`
- Loosen the dependency on `twined` to allow freedom in the patch versions of the current minor version

Refactoring
- Reduce repetition and improve getting of buckets in `GoogleCloudStorageClient`
- Replace `GoogleCloudStorageClient._create_intermediate_local_directories` with proper usage of `os.makedirs`
- Simplify `Manifest._instantiate_dataset` and remove support for datasets with no explicit path

Testing
- Test calculation/getting of datafile hash values
- Create and use a mock for generating signed URLs to avoid the limitations of workload identity federation in CI testing

<!--- SKIP AUTOGENERATED NOTES --->

<!--- START AUTOGENERATED NOTES --->
Contents ([423](https://github.com/octue/octue-sdk-python/pull/423))

Fixes
- Update parameter name to remove typo

<!--- END AUTOGENERATED NOTES --->

Summary

Allows installation alongside django-gcp, to enable us to improve integration of octue with django

<!--- START AUTOGENERATED NOTES --->
Contents ([422](https://github.com/octue/octue-sdk-python/pull/422))

Operations
- Add dockerfile to allow correct use of poetry in devcontainer
- Update version

Dependencies
- Update google-cloud-storage range

<!--- END AUTOGENERATED NOTES --->

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([419](https://github.com/octue/octue-sdk-python/pull/419))

New features
- Support push subscriptions for Google Pub/Sub services, allowing answers to be pushed to an HTTP endpoint

Enhancements
- Add `push_endpoint` parameter to `Subscription` constructor
- Add `is_pull_subscription` and `is_push_subscription` properties to `Subscription`

Testing
- Switch to parent/child language in `Service` tests for greater clarity/consistency

<!--- END AUTOGENERATED NOTES --->

Summary
Simplify manifest and dataset serialisation so their datasets' and files' metadata is gathered dynamically at instantiation instead of from hard-coded values in the serialisation. To allow this properly on both cloud and local platforms, the ability to store metadata locally has been introduced. This has also required the definition of a dataset to be tightened so that each dataset is either cloud-based or local, but not both.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([402](https://github.com/octue/octue-sdk-python/pull/402))
**IMPORTANT:** There are 5 breaking changes.

New features
- Store local datafile metadata locally in a `.octue` JSON file in the same directory

Enhancements

Dataset:
- **BREAKING CHANGE:** Change `Dataset` serialisation to only include the paths to its files and its own metadata
- **BREAKING CHANGE:** Store local dataset metadata locally in a `.octue` JSON file in its root directory instead of `datafile_metadata.json`
- Allow datasets to be instantiated from iterables of paths to files
- Remove `Pathable` mixin
- Unify `Dataset.path` and `Dataset.cloud_path`
- Raise error if trying to download files from a local dataset
- **BREAKING CHANGE:** Remove confusing way of adding files to datasets
- **BREAKING CHANGE:** Only allow one file to be added to a dataset at a time in `Dataset.add`
- Allow specification of path within dataset when adding a file to a dataset

Manifest:
- **BREAKING CHANGE:** Change `Manifest` serialisation to only include the paths to its datasets and its own metadata
- Remove `Pathable` mixin
- Remove `path` attribute

Other:
- Use `Serialisable.to_primitive` as the basis for `Serialisable.serialise` to simplify and speed up conversion to primitives
- Use `Manifest.to_primitive` in `Analysis.finalise`
- Update remaining JSON metaschema references to use the latest metaschema

Fixes
- Stop setting any given kwarg as attributes in `Dataset` and `Manifest`
- Ignore `.octue` files when constructing datasets

Dependencies
- Change minimum python version supported from `3.7` to `3.7.1`
- Use `pandas=^1.3` to avoid `numpy` array size errors
- Use `twined=0.3.0`

Refactoring
- Reorder and rename methods in `Datafile`, `Dataset`, and `Manifest`
- Move dataset files tag checking from `twined` into `Runner._validate_dataset_file_tags`
- Move datafile instantiation in `Dataset` into separate method

<!--- END AUTOGENERATED NOTES --->