Octue

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([650](https://github.com/octue/octue-sdk-python/pull/650))

**IMPORTANT:** There is 1 breaking change.

Enhancements
- Include received invalid data in flask app error messages
- Allow any iterable for `Dataset` `files` argument

Fixes
- Ensure `order` argument is given in `Service.send_exception`
- Add workaround for apparent bug in getting local metadata file's absolute path
- Remove (now-) unnecessary json decoding in `get_events`

Dependencies
- 💥 **BREAKING CHANGE:** Drop support for python3.7
- Use `twined==0.5.5`
- Update to `black==24.4.2`

Testing
- Remove unnecessary test
- Update asynchronous deployment test to accept numpy array as output values

---
Upgrade instructions
<details>
<summary>💥 <b>Drop support for python3.7</b></summary>

Upgrade to `python>=3.8` to keep using `octue`.
</details>

<!--- END AUTOGENERATED NOTES --->

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([647](https://github.com/octue/octue-sdk-python/pull/647))

**IMPORTANT:** There are 2 breaking changes.

Enhancements
- 💥 **BREAKING CHANGE:** Return question UUID from `Child.ask`
- Deserialise manifests from events in `get_event`
- Raise error if no events found when calling `get_events`

Fixes
- Use correct base image for `python3.11` dockerfile
- Return schema-compliant events and attributes from `get_events`

Operations
- Import missing APIs into terraform config
- Deploy version `0.5.0` of event handler cloud function and update event store schema
- Update `actions/setup-python` to version 5

Dependencies
- 💥 **BREAKING CHANGE:** Make `db-dtypes` and `google-cloud-bigquery` optional
- Upgrade `gunicorn` to avoid vulnerability
- Loosen `numpy` dependency

Testing
- Test retrieving results from real asynchronous question
- Run tests with `python3.10` (`python3.9` isn't available on `macos-latest` for `arm64`)

Other
- Add DOI badge to readme

---
Upgrade instructions
<details>
<summary>💥 <b>Return question UUID from `Child.ask`</b></summary>

Instead of writing `answer = Child.ask(...)`, write `answer, question_uuid = Child.ask(...)` (and the same for `ChildEmulator`)
</details>

<details>
<summary>💥 <b>Make `db-dtypes` and `google-cloud-bigquery` optional</b></summary>

To keep using the `get_events` function, add the `bigquery` optional extra to your installation command e.g. `poetry install -E bigquery`.
</details>

<!--- END AUTOGENERATED NOTES --->

Summary
This pull request:
- Makes the SDK fully event-driven by using a single topic to emit/consume events
- Majorly refactors the event handler to facilitate asynchronous event retrieval
- Adds the ability to get and replay events from a BigQuery store

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([632](https://github.com/octue/octue-sdk-python/pull/632))

**IMPORTANT:** There are 4 breaking changes.

New features
- 💥 **BREAKING CHANGE**: Use single topic per workspace (https://github.com/octue/octue-sdk-python/pull/639)
- Add `get_events` function for retrieving events asynchronously from BigQuery
- Add `EventReplayer` class to replay asynchronously-retrieved events
- Add `Manifest.download` method

Enhancements
- Get subscription project name from topic by default
- Improve asking of asynchronous questions via `Child.ask`
- Return download path from `Dataset.download`
- Include question UUID in delivery acknowledgement log message
- Improve handling of invalid events
- Add `datetime` and `uuid` attributes to all events

Fixes
- Await successful publishing of question messages
- Fix `api_access_endpoint` usage in `mock_generate_signed_url`

Operations
- Add test BigQuery dataset, cloud function, and IAM roles to terraform config
- Switch to reusable workflows where possible

Dependencies
- Add `google-cloud-bigquery`
- Upgrade `coolname`
- Add `db-dtypes` for converting bigquery rows to dataframes

Refactoring
- 💥 **BREAKING CHANGE**: Rename `x.received_messages` to `x.received_events`
- 💥 **BREAKING CHANGE**: Rename `record_messages` parameters to `record_events`
- 💥 **BREAKING CHANGE**: Update `ChildEmulator` to use `event*` instead of `message*`
- Factor out making minimal dictionary
- Factor out creating push subscription
- Factor out emitting question event in `Service.ask`
- Factor out event handlers and related logic from `OrderedMessageHandler` into new `AbstractEventHandler`
- Move `validation` module into `octue.cloud.events` subpackage
- Rename `OrderedMessageHandler` to `GoogleCloudPubSubEventHandler`
- Rename "message" to "event" in event handler classes
- Rename `GooglePubSubHandler` to `GoogleCloudPubSubHandler`

Chores
- Update licence year to 2024

Testing
- Simplify various tests

Upgrade instructions
- Update all services in your services network to this version of `octue` or later (`0.53.0`+)
- Replace any usages of the `received_messages` methods with `received_events`
- Replace any usages of the `record_messages` parameters with `record_events`
- Replace the word `message` with `event` in usages of `ChildEmulator` methods (apart from in the case of `monitor_message`)

<!--- END AUTOGENERATED NOTES --->

<!--- START AUTOGENERATED NOTES --->
Contents ([627](https://github.com/octue/octue-sdk-python/pull/627))

Enhancements
- Warn about messages with duplicate message numbers

<!--- END AUTOGENERATED NOTES --->

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([625](https://github.com/octue/octue-sdk-python/pull/625))

**IMPORTANT:** There is 1 breaking change.

Enhancements
- Allow setting of maximum number of workers for parallel questions in `Child.ask_multiple`
- Pull up to 50 messages from answer subscriptions at once instead of 1
- Allow skipping of any missing message after a 10s delay in `OrderedMessageHandler`
- Suppress name/namespace override warning if the value is the same in the environment and service configuration file
- Speed up event validation by caching service communication JSON schema
- 💥 **BREAKING CHANGE:** Extract SRUID for child logs context from subscription in message handler

Fixes
- Exit early from message pulling if heartbeat check fails
- Make `Manifest.update_dataset_paths` method thread-safe

Refactoring
- Factor out multiple checks of package version in message handler

Testing
- Improve message handling tests by not mocking `_pull_and_enqueue_available_messages` method and removing `MockMessagePuller`

---
Upgrade instructions
<details>
<summary>💥 <b>Extract SRUID for child logs from subscription in message handler</b></summary>

This removes the `service_name` argument from `Service.wait_for_answer`. If you were using this argument, simply remove it; logs from children shown in a parent will now have the full and correct SRUID automatically.
</details>

<!--- END AUTOGENERATED NOTES --->