Octue

Summary
Allow recording of:
- **Sent messages by children** if they crash (if allowed by the parent on a per-question basis). This will aid child debugging and allow crashes to be replayed from the parent's perspective during testing.
- **Received messages by the parent** for the purpose of creating test fixtures of different child operation modes.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([511](https://github.com/octue/octue-sdk-python/pull/511))

New features
- Allow child services to record messages sent to their parent
- Allow parents to record messages received from their children

Enhancements
- Save all messages sent by a child to its parent on crash instead of just its log messages

Fixes
- Only try saving crash diagnostics if a crash diagnostics cloud path has been set
- Unify `ChildEmulator` message format with `Service` message format
- Remove ability to pass `Exception` instances into `ChildEmulator`

Refactoring
- Factor out message sending in `Service`
- Factor out patching of classes used by `Service` to enable `MockService` operation
- Make `pub_sub` emulators module private

Testing
- Test the child services template app using emulated children
- Clean up `Service` tests

Documentation
- Add documentation on how to test services with `ChildEmulator`
- Add documentation on how to get crash diagnostics for a service

<!--- END AUTOGENERATED NOTES --->

Summary
Add the ability to store crash diagnostics (the configuration, input, and logs) for a question when a child service fails. This requires:
- The permission of the parent on a per-question basis
- For the child to be configured to store this data

The crash diagnostics can be retrieved via the new `octue get-crash-diagnostics` command.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([507](https://github.com/octue/octue-sdk-python/pull/507))

New features
- Allow inputs and configurations to be saved in the cloud and retrieved via the new `get-crash-diagnostics` CLI command if an app crashes

Enhancements
- Add crash diagnostics cloud path to `ServiceConfiguration`
- Add crash diagnostics attribute to questions
- Add ability to copy files within Google Cloud
- Log if the path of a `CloudPathable` is changed
- Avoid need to download datafiles to move them to a new cloud path
- Add ability to choose whether colour is used by log formatter
- Add `GoogleCloudStorageClient.download_all_files` method

Fixes
- Fix configuration manifest validation in `Runner`
- Ensure existing cloud datafiles and datasets can be uploaded to a new cloud location
- Avoid misleading warning when writing to new cloud datafiles

Refactoring
- Rename `CloudPathable._get_cloud_location` to `_set_cloud_location`
- Simplify datafile upload/download in datasets

<!--- END AUTOGENERATED NOTES --->

Summary
Add a child emulator that can be used to test a service without connecting to its real/production children. The messages the child returns are specified during instantiation in python or via a JSON file. All messages that a real child can produce are supported - log messages, monitor messages, exceptions, and results.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([510](https://github.com/octue/octue-sdk-python/pull/510))

New features
- Add `ChildEmulator` class

Enhancements
- Improve `OrderedMessageHandler` log messages

Fixes
- Ensure `KeyError`s are properly raised by `OrderedMessageHandler`

Refactoring
- Convert `emulators` module into subpackage
- Move Pub/Sub mocks from tests package into `octue.cloud.emulators` subpackage
- Factor out exceptions mapping into `octue.cloud` subpackage

Testing
- Allow Cloud Run and Dataflow deployment tests to be independently skipped
- Update test structure and imports

<!--- END AUTOGENERATED NOTES --->

Summary
Lazily load Google Cloud credentials for `Service` and `Child` instances, allowing apps that don't always contact their children in an analysis to work without the credentials available when they don't need them (instead of failing as currently happens).

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([509](https://github.com/octue/octue-sdk-python/pull/509))

Fixes
- Allow `Service` to be instantiated without Google Cloud credentials

<!--- END AUTOGENERATED NOTES --->

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([506](https://github.com/octue/octue-sdk-python/pull/506))

Enhancements
- Give specifics in the error raised when a datafile fails dataset file tag template validation

Fixes
- Ignore trailing slashes when splitting/parsing cloud paths
- Ensure the `Dataset.name` property works for datasets with a cloud path ending in a forward slash
- Ensure `GoogleCloudStorageClient.scandir` works on the root directory of a bucket

<!--- END AUTOGENERATED NOTES --->

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([500](https://github.com/octue/octue-sdk-python/pull/500))

Enhancements
- Set default question timeout to 1 day
- Allow dataset upload without metadata change
- Cache dataset cloud metadata privately on instance
- Only update dataset cloud metadata if it's changed

Testing
- Update dataflow deployment test

<!--- END AUTOGENERATED NOTES --->