Octue

Summary
Add the ability to deploy services as Google Dataflow streaming jobs to allow greater freedom in machine specifications and analysis complexities/demands. Parents do not need to know whether a child is a Cloud Run service or a Dataflow service.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([291](https://github.com/octue/octue-sdk-python/pull/291))

New features
- Add function and CLI command for deploying apps as Google Dataflow streaming pipeline services

Enhancements
- Add static methods for topic and subscription path generation without requiring `Topic` or `Subscription` instantiation

Fixes
- Allow response subscriptions to already exist in `Service.answer` so that questions can be re-asked if no delivery acknowledgement is received

Refactoring
- Move Cloud Run deployment module and Dockerfile into `cloud_run` subpackage
- Factor out Pub/Sub question answering function from Cloud Run `flask` app

Dependencies
- Temporarily downgrade to `google-cloud-pubsub==1.7.0`. This can be reverted when version `2.36.0` of `apache-beam` is released as it provides support for `google-cloud-pubsub>=2`

Style
- Ignore missing public docstring warning during pre-commit checks to allow docstrings to be added at the end of a pull request once new code is stable

Testing
- Move Cloud Run deployment tests into their own test module
- Change `RUN_DEPLOYMENT_TESTS` environment variable expected values to integers 0 and 1

<!--- END AUTOGENERATED NOTES --->

Summary
Remove two `Dataset` methods that were deprecated 13 months ago. Also remove the `Loggable` mixin, which was causing confusion on how log messages should be emitted.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([305](https://github.com/octue/octue-sdk-python/pull/305))
**IMPORTANT:** There is 1 breaking change.

Refactoring
- Remove `Loggable` mixin
- **BREAKING CHANGE:** Remove deprecated `Dataset.append` and `Dataset.get_files` methods

<!--- END AUTOGENERATED NOTES --->

Summary
Ensure the root logger includes the analysis ID in the logging context for the duration of the analysis.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([304](https://github.com/octue/octue-sdk-python/pull/304))

Enhancements
- Swap root log handlers for handlers that include the analysis ID in the logging context for duration of analysis
- Allow easy addition of line number, process name, and thread name to logging context via new environment variables

Fixes
- Ensure analysis logging includes log messages from any packages or modules used during the analysis
- Ensure the name of the module from which the log message is emitted is correct for log messages emitted during the analysis
- Fix Pub/Sub delivery attempt key error

Refactoring
- Rename `create_formatter` to `create_octue_formatter`

Testing
- Update example service ID

<!--- END AUTOGENERATED NOTES --->

Summary
Multi-thread uploads of multiple files from dataset to cloud storage.

When uploading 100 files, the time taken was:
- 15.5s with the new approach
- 50.9s with the old approach

This is a 3.3x speed up that is likely to increase with the number of files in the dataset and the number of threads available.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([303](https://github.com/octue/octue-sdk-python/pull/303))

Enhancements
- Use `ThreadPool` to upload files from a dataset to the cloud
- Lift 10-worker limit for dataset file downloads

Fixes
- Ensure datafile paths relative to their dataset are constructed correctly on non-Linux-based OSs
<!--- END AUTOGENERATED NOTES --->

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([301](https://github.com/octue/octue-sdk-python/pull/301), [#302](https://github.com/octue/octue-sdk-python/pull/302))

Fixes
- Preserve directory structure on dataset cloud upload
<!--- END AUTOGENERATED NOTES --->

<!--- START AUTOGENERATED NOTES --->
Contents ([296](https://github.com/octue/octue-sdk-python/pull/296))

Enhancements
- Allow instantiation of `Dataset`s from a local directory recursively and non-recursively

<!--- END AUTOGENERATED NOTES --->