Octue

Summary
Remove confusing warnings about instantiation metadata vs. stored metadata when instantiating datafiles and datasets. Clarify the resolution order when both instantiation metadata and stored metadata are present.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([518](https://github.com/octue/octue-sdk-python/pull/518))

**IMPORTANT:** There are 2 breaking changes.

Fixes
- Remove misleading metadata override warnings from `Datafile` and `Dataset`

Operations
- Use newer poetry syntax for installing extras

Refactoring
- 💥 **BREAKING CHANGE:** Rename Datafile `hypothetical` parameter to `ignore_stored_metadata`
- 💥 **BREAKING CHANGE:** Rename Dataset `hypothetical` parameter to `ignore_stored_metadata`

Operations
- Use latest commit message checker

---
Upgrade instructions
<details>
<summary>💥 <b>Rename Datafile `hypothetical` parameter</b></summary>

Replace `hypothetical` with `ignore_stored_metadata`.
</details>

<details>
<summary>💥 <b>Rename Dataset `hypothetical` parameter</b></summary>

Replace `hypothetical` with `ignore_stored_metadata`.
</details>

<!--- END AUTOGENERATED NOTES --->

Summary

<!--- START AUTOGENERATED NOTES --->
Contents ([517](https://github.com/octue/octue-sdk-python/pull/517))

Fixes
- Combine duplicate installations in Docker container

Operations
- Use new poetry installation script in Docker containers

<!--- END AUTOGENERATED NOTES --->

Summary
This release upgrades children to send heartbeats to parents. This allows the parent to know that a child it's asked a question to is still alive and hasn't crashed. In the event of a crash, heartbeats stop being received and the parent aborts the questions, avoiding large waits for nothing.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([514](https://github.com/octue/octue-sdk-python/pull/514))
**IMPORTANT:** There are 2 breaking changes.

New features
- Send heartbeat messages to parents from children
- Raise error in parent if child heartbeats aren't received for too long an interval

Enhancements
- Tell user how to question a service after using `octue start` CLI command

Refactoring
- 💥 **BREAKING CHANGE:** Rename `run_subprocess_and_log_stdout_and_stderr` to `run_logged_subprocess`
- Split thread and process utilities into separate modules
- Remove redundant extra `while` loop in `OrderedMessageHandler`
- Remove redundant `unwrap` function and `AppFrom.run` method from `runner` module

Operations
- Use non-deprecated/latest Codecov action in workflows

Testing
- Increase `ChildEmulator` test coverage


---
Upgrade instructions
<details>
<summary>💥 <b>Rename run_subprocess_and_log_stdout_and_stderr to run_logged_subprocess</b></summary>

Use `octue.utils.threads.run_logged_subprocess` instead.
</details>

<!--- END AUTOGENERATED NOTES --->

Summary
Add the ability to ask a child any number of questions in parallel instead of in serial.

<!--- START AUTOGENERATED NOTES --->
Contents ([513](https://github.com/octue/octue-sdk-python/pull/513))

New features
- Add ability to ask children questions in parallel

Other
- Document parallelised question asking

<!--- END AUTOGENERATED NOTES --->

Summary
Allow recording of:
- **Sent messages by children** if they crash (if allowed by the parent on a per-question basis). This will aid child debugging and allow crashes to be replayed from the parent's perspective during testing.
- **Received messages by the parent** for the purpose of creating test fixtures of different child operation modes.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([511](https://github.com/octue/octue-sdk-python/pull/511))

New features
- Allow child services to record messages sent to their parent
- Allow parents to record messages received from their children

Enhancements
- Save all messages sent by a child to its parent on crash instead of just its log messages

Fixes
- Only try saving crash diagnostics if a crash diagnostics cloud path has been set
- Unify `ChildEmulator` message format with `Service` message format
- Remove ability to pass `Exception` instances into `ChildEmulator`

Refactoring
- Factor out message sending in `Service`
- Factor out patching of classes used by `Service` to enable `MockService` operation
- Make `pub_sub` emulators module private

Testing
- Test the child services template app using emulated children
- Clean up `Service` tests

Documentation
- Add documentation on how to test services with `ChildEmulator`
- Add documentation on how to get crash diagnostics for a service

<!--- END AUTOGENERATED NOTES --->

Summary
Add the ability to store crash diagnostics (the configuration, input, and logs) for a question when a child service fails. This requires:
- The permission of the parent on a per-question basis
- For the child to be configured to store this data

The crash diagnostics can be retrieved via the new `octue get-crash-diagnostics` command.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([507](https://github.com/octue/octue-sdk-python/pull/507))

New features
- Allow inputs and configurations to be saved in the cloud and retrieved via the new `get-crash-diagnostics` CLI command if an app crashes

Enhancements
- Add crash diagnostics cloud path to `ServiceConfiguration`
- Add crash diagnostics attribute to questions
- Add ability to copy files within Google Cloud
- Log if the path of a `CloudPathable` is changed
- Avoid need to download datafiles to move them to a new cloud path
- Add ability to choose whether colour is used by log formatter
- Add `GoogleCloudStorageClient.download_all_files` method

Fixes
- Fix configuration manifest validation in `Runner`
- Ensure existing cloud datafiles and datasets can be uploaded to a new cloud location
- Avoid misleading warning when writing to new cloud datafiles

Refactoring
- Rename `CloudPathable._get_cloud_location` to `_set_cloud_location`
- Simplify datafile upload/download in datasets

<!--- END AUTOGENERATED NOTES --->