Octue

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([647](https://github.com/octue/octue-sdk-python/pull/647))

**IMPORTANT:** There are 2 breaking changes.

Enhancements
- 💥 **BREAKING CHANGE:** Return question UUID from `Child.ask`
- Deserialise manifests from events in `get_event`
- Raise error if no events found when calling `get_events`

Fixes
- Use correct base image for `python3.11` dockerfile
- Return schema-compliant events and attributes from `get_events`

Operations
- Import missing APIs into terraform config
- Deploy version `0.5.0` of event handler cloud function and update event store schema
- Update `actions/setup-python` to version 5

Dependencies
- 💥 **BREAKING CHANGE:** Make `db-dtypes` and `google-cloud-bigquery` optional
- Upgrade `gunicorn` to avoid vulnerability
- Loosen `numpy` dependency

Testing
- Test retrieving results from real asynchronous question
- Run tests with `python3.10` (`python3.9` isn't available on `macos-latest` for `arm64`)

Other
- Add DOI badge to readme

---
Upgrade instructions
<details>
<summary>💥 <b>Return question UUID from `Child.ask`</b></summary>

Instead of writing `answer = Child.ask(...)`, write `answer, question_uuid = Child.ask(...)` (and the same for `ChildEmulator`)
</details>

<details>
<summary>💥 <b>Make `db-dtypes` and `google-cloud-bigquery` optional</b></summary>

To keep using the `get_events` function, add the `bigquery` optional extra to your installation command e.g. `poetry install -E bigquery`.
</details>

<!--- END AUTOGENERATED NOTES --->

Summary
This pull request:
- Makes the SDK fully event-driven by using a single topic to emit/consume events
- Majorly refactors the event handler to facilitate asynchronous event retrieval
- Adds the ability to get and replay events from a BigQuery store

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([632](https://github.com/octue/octue-sdk-python/pull/632))

**IMPORTANT:** There are 4 breaking changes.

New features
- 💥 **BREAKING CHANGE**: Use single topic per workspace (https://github.com/octue/octue-sdk-python/pull/639)
- Add `get_events` function for retrieving events asynchronously from BigQuery
- Add `EventReplayer` class to replay asynchronously-retrieved events
- Add `Manifest.download` method

Enhancements
- Get subscription project name from topic by default
- Improve asking of asynchronous questions via `Child.ask`
- Return download path from `Dataset.download`
- Include question UUID in delivery acknowledgement log message
- Improve handling of invalid events
- Add `datetime` and `uuid` attributes to all events

Fixes
- Await successful publishing of question messages
- Fix `api_access_endpoint` usage in `mock_generate_signed_url`

Operations
- Add test BigQuery dataset, cloud function, and IAM roles to terraform config
- Switch to reusable workflows where possible

Dependencies
- Add `google-cloud-bigquery`
- Upgrade `coolname`
- Add `db-dtypes` for converting bigquery rows to dataframes

Refactoring
- 💥 **BREAKING CHANGE**: Rename `x.received_messages` to `x.received_events`
- 💥 **BREAKING CHANGE**: Rename `record_messages` parameters to `record_events`
- 💥 **BREAKING CHANGE**: Update `ChildEmulator` to use `event*` instead of `message*`
- Factor out making minimal dictionary
- Factor out creating push subscription
- Factor out emitting question event in `Service.ask`
- Factor out event handlers and related logic from `OrderedMessageHandler` into new `AbstractEventHandler`
- Move `validation` module into `octue.cloud.events` subpackage
- Rename `OrderedMessageHandler` to `GoogleCloudPubSubEventHandler`
- Rename "message" to "event" in event handler classes
- Rename `GooglePubSubHandler` to `GoogleCloudPubSubHandler`

Chores
- Update licence year to 2024

Testing
- Simplify various tests

Upgrade instructions
- Update all services in your services network to this version of `octue` or later (`0.53.0`+)
- Replace any usages of the `received_messages` methods with `received_events`
- Replace any usages of the `record_messages` parameters with `record_events`
- Replace the word `message` with `event` in usages of `ChildEmulator` methods (apart from in the case of `monitor_message`)

<!--- END AUTOGENERATED NOTES --->

<!--- START AUTOGENERATED NOTES --->
Contents ([627](https://github.com/octue/octue-sdk-python/pull/627))

Enhancements
- Warn about messages with duplicate message numbers

<!--- END AUTOGENERATED NOTES --->

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([625](https://github.com/octue/octue-sdk-python/pull/625))

**IMPORTANT:** There is 1 breaking change.

Enhancements
- Allow setting of maximum number of workers for parallel questions in `Child.ask_multiple`
- Pull up to 50 messages from answer subscriptions at once instead of 1
- Allow skipping of any missing message after a 10s delay in `OrderedMessageHandler`
- Suppress name/namespace override warning if the value is the same in the environment and service configuration file
- Speed up event validation by caching service communication JSON schema
- 💥 **BREAKING CHANGE:** Extract SRUID for child logs context from subscription in message handler

Fixes
- Exit early from message pulling if heartbeat check fails
- Make `Manifest.update_dataset_paths` method thread-safe

Refactoring
- Factor out multiple checks of package version in message handler

Testing
- Improve message handling tests by not mocking `_pull_and_enqueue_available_messages` method and removing `MockMessagePuller`

---
Upgrade instructions
<details>
<summary>💥 <b>Extract SRUID for child logs from subscription in message handler</b></summary>

This removes the `service_name` argument from `Service.wait_for_answer`. If you were using this argument, simply remove it; logs from children shown in a parent will now have the full and correct SRUID automatically.
</details>

<!--- END AUTOGENERATED NOTES --->

Summary
This pull request removes the use of answer topics by publishing answer messages to the service revision (formerly known as question) topic and filtering subscriptions to only receive a) questions or b) response messages to a specific question. This speeds up the question asking process, reduces cloud infrastructure requirements and the permissions surface, and allows us to avoid topic number limits.

Also added is validation of all messages and their attributes against a new [publicly available schema](https://strands.octue.com/octue/service-communication). This ensures services are communicating as they should and opens up the possibility of writing services in other languages and creating emulators.

As this, by itself, constitutes an inter-service communication breaking change, we've taken the opportunity to reduce the complexity of the codebase by removing backwards compatibility patches for service communication (i.e. we've grouped multiple breaking changes together into one).

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([603](https://github.com/octue/octue-sdk-python/pull/603))

**IMPORTANT:** There are 7 breaking changes.

New features
- Validate messages and their attributes against new service communication schema (see [614](https://github.com/octue/octue-sdk-python/pull/614) for changelog - it was merged into this branch)
- Allow diagnostics (formerly known as crash diagnostics) to always be switched on for a service

Enhancements
- 💥 **BREAKING CHANGE:** Publish responses to questions to the service revision (question) topic instead of creating a separate answer topic
- 💥 **BREAKING CHANGE:** Store message number in message attributes instead of in message data
- 💥 **BREAKING CHANGE:** Remove question UUID from log record message body
- 💥 **BREAKING CHANGE:** Remove inter-service communication backwards compatibility code
- 💥 **BREAKING CHANGE:** Make input and output values and manifest optional
- 💥 **BREAKING CHANGE:** Replace boolean `allow_save_diagnostics_data_on_crash` argument with string/enum `save_diagnostics` argument in `Service.ask` and related methods
- Add ability to filter subscriptions
- Add question UUID attribute to all messages
- Send more possible errors to parent in `Service.answer`
- Add `kind` field to question messages
- Add `sender_type` attribute to all messages
- Add ability to instantiate `Runner` from service/app configurations

Fixes
- Stop double-JSON-encoding output manifests

Dependencies
- Update `octue` version in template apps' dependencies

Refactoring
- 💥 **BREAKING CHANGE:** Rename crash diagnostics to diagnostics
- Group message attributes in `Service._send_message` and `MockMessage` under explicit `attributes` argument
- Make `OrderedMessageHandler._waiting_messages` attribute public
- Rename various message attributes

Testing
- Store mock pub/sub messages against subscriptions instead of topics
- Add missing `type` field to emulated Pub/Sub questions

Operations
- Fix `add-issues-to-octue-board` workflow
- Stop automatically building docker images for registry in `release` workflow
- Add ReadTheDocs config file to fix documentation building

---
Upgrade instructions
💥 Update all Octue services in your network to use this version of `octue` so they're still able to communicate. Postpone upgrading until you can upgrade all services simultaneously.

💥 Replace `allow_save_diagnostics_data_on_crash` with `save_diagnostics` set to one of these values: "SAVE_DIAGNOSTICS_OFF", "SAVE_DIAGNOSTICS_ON_CRASH", or "SAVE_DIAGNOSTICS_ON"

💥 Crash diagnostics rename:
- Use the `octue get-diagnostics` CLI command instead of the `octue get-crash-diagnostics` command
- Rename `crash_diagnostics_cloud_path` in your service configurations to `diagnostics_cloud_path`
<!--- END AUTOGENERATED NOTES --->