Octue

Summary
This release add support and requirement for service namespaces and service revision tags. This allows services to be versioned and provides a more familiar format for service IDs that's, for example, similar to Docker image names (e.g. `octue/example-service:latest`). Full validation of service IDs comes as part of this.

The new `octue deploy create-push-subscription` has also been added as requested by users and will be accompanied by a GitHub action in another repository.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([529](https://github.com/octue/octue-sdk-python/pull/529))

**IMPORTANT:** There are 4 breaking changes.

New features
- 💥 **BREAKING CHANGE:** Add support for and require revision tags for services
- 💥 **BREAKING CHANGE:** Validate service IDs before creating or questioning services
- Add ability to set service namespace, name, and revision tag by environment variable
- Add `octue deploy create-push-subscription` CLI command

Enhancements
- 💥 **BREAKING CHANGE:** Require services to have a namespace and replace use of `organisation` with `namespace` in service configurations
- 💥 **BREAKING CHANGE:** Remove `namespace` parameter from `Topic` and `Subscription` and apply `octue.services` Pub/Sub namespace to all topic/subscription paths
- Remove application of `octue.services` Pub/Sub namespace from `Service`
- Use a `coolname` revision tag to allow a service revision with a non-unique SRUID to start as a unique service revision when using the `octue start` CLI command
- Set the `Service.id` attribute to the user-friendly SRUID instead of the Pub/Sub-friendly SRUID
- Return SRUID from `DataflowDeployer.deploy`

Fixes
- Handle inability to find/load version compatibility data file
- Use latest Apache Beam base image in Dataflow `Dockerfile`
- Fix some log and error messages

Operations
- Speed up and simplify `release` workflow
- Use new Octue GitHub actions in workflows
- Improve installation process in main `Dockerfile`
- Use latest commit message checker

Dependencies
- Update Dataflow `setup.py` file

Refactoring
- Rename `OCTUE_NAMESPACE` to `OCTUE_SERVICES_NAMESPACE`
- Remove cool-naming of services when instantiated without `id` and simplify the `name` argument

---
Upgrade instructions
<details>
<summary>💥 <b>Add support for and require revision tags for services</b></summary>

The `ServiceConfiguration.service_id` attribute and the `SERVICE_ID` environment variable have been removed.

- **`ServiceConfiguration`:** Use `octue.cloud.service_id.create_service_id` to create IDs from `ServiceConfiguration.namespace` and `ServiceConfiguration.name`

- **`SERVICE_ID` environment variable:** Use the new `OCTUE_SERVICE_NAMESPACE`, `OCTUE_SERVICE_NAME`, and `OCTUE_SERVICE_REVISION_TAG` environment variables.
</details>

<details>
<summary>💥 <b>Validate service IDs before creating or questioning services</b></summary>

Update your services' namespaces, names, and revision tags to follow the requirements set out in the `Creating services` doc.
</details>

<details>
<summary>💥 <b>Require services to have a namespace and replace use of organisation with namespace in service configurations</b></summary>

Provide the `namespace` key in the service configuration (the `octue.yaml` file). If you were using the `organisation` key before, the `namespace` key now replaces it. Examples of a `namespace` are your organisation's name or your GitHub username.
</details>

<details>
<summary>💥 <b>Remove namespace parameter from Topic and Subscription and apply octue.services Pub/Sub namespace to all topic/subscription paths</b></summary>

The `octue.services` namespace is now mandatory and applied automatically in topic and subscription paths. Please stop providing the `namespace` argument if you were providing it before.
</details>

<!--- END AUTOGENERATED NOTES --->

Summary
Allow parents to override the default children its child uses (i.e. the parent's grandchildren). This allows services that produce the same type of results but use, for example, different versions of a model to be swapped in and out for each other without modifying the child.

Note that the child has to be running version `0.39.0` to be able to accept the override (otherwise it will just use its default children).

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([524](https://github.com/octue/octue-sdk-python/pull/524))

New features
- Add ability to dynamically override a child's children when asking it a question

Enhancements
- Use raw service IDs in logs and errors

Fixes
- Use full service ID in `MockService` when answering question

Refactoring
- Refactor `OrderedMessageHandler` methods into smaller methods

Testing
- Remove middle-man `ask_question_and_wait_for_answer` method in `TestService`

<!--- END AUTOGENERATED NOTES --->

Summary
Ensure that datasets constructed using the `files` argument are successfully serialised/deserialised without losing file information.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([525](https://github.com/octue/octue-sdk-python/pull/525))

Fixes
- Fix serialisation/deserialisation of datasets in `Manifest` when the datasets have been constructed using the `files` argument
- Use datafile cloud path in `files` field if available in `Dataset.to_primitive`
- Use `Manifest.serialise` instead of `Manifest.to_primitive` in `Analysis.finalise`

Dependencies
- Use `twined=^0.5.1`

<!--- END AUTOGENERATED NOTES --->

Summary
Make the `Topic` and `Subscription` classes standalone instead of relying on a `Service` instance and a related subscriber.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([523](https://github.com/octue/octue-sdk-python/pull/523))

**IMPORTANT:** There are 2 breaking changes.

Enhancements
- 💥 **BREAKING CHANGE:** Remove `service` argument from `Topic` constructor
- 💥 **BREAKING CHANGE:** Remove `subscriber` argument from `Subscription` constructor
- Implicitly get `Service` credentials from environment and remove `credentials` attribute

Fixes
- Move `finally` clause to correct nesting level in `OrderedMessageHandler.handle_messages`

---
Upgrade instructions
<details>
<summary>💥 <b>Remove `service` argument from `Topic` constructor</b></summary>

Provide the `project_name` argument instead of the old `service` argument.
</details>

<details>
<summary>💥 <b>Remove `subscriber` argument from `Subscription` constructor</b></summary>

Remove the old `subscriber` argument as it's no longer needed.
</details>

<!--- END AUTOGENERATED NOTES --->

Summary
Only allow services with unique names to be started via the `octue start` CLI command. This ensures that, when using the command:
- A new service can't intercept/receive questions meant for an existing service of the same name
- When running a service locally that's already deployed, a modified service name is generated and used, preventing the deletion of the deployed service's topic/subscription on exit

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([515](https://github.com/octue/octue-sdk-python/pull/515))

**IMPORTANT:** There is 1 breaking change.

Enhancements
- 💥 **BREAKING CHANGE:** Remove the `--rm` option and instead delete the service topic and subscription by default on exit of the `octue start` CLI command
- Add `--no-rm` flag to `octue start` to avoid topic and subscription deletion
- Allow `octue start` users to create a service with a variant of the ID in `octue.yaml` if the service already exists
- Add `allow_existing` parameter to `Service.serve` and default it to `False` (it was always `True` within the method before)
- Add `creation_triggered_locally` property to `Topic` and `Subscription` to facilitate avoiding deletion of subscriptions and topics that weren't created by the local service

Fixes
- Don't allow answer subscriptions to already exist
- Don't allow a service to be started by `octue start` if its ID is the same as an existing service's ID
- Ensure "what next" log message for starting a service is displayed at the right time

Testing
- Speed up `Topic.exists` test
- Simplify `Topic` tests

Operations
- Improve table generated by version compatibility table printing script

---
Upgrade instructions
<details>
<summary>💥 <b>Delete topic/subscription at end of octue start command by default</b></summary>

Stop providing the `--rm` / `--delete-topic-and-subscription-on-exit` option to the `octue start` CLI command. Use the `--no-rm` flag or deploy your service if you want the service topic and subscription to persist after exiting this command.
</details>

<!--- END AUTOGENERATED NOTES --->

Summary
This PR adds the use of empirical data (instead of intuition) on compatible versions of `octue` for inter-service communication to warn about version compatibilities when errors are raised during communication.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([498](https://github.com/octue/octue-sdk-python/pull/498))

New features
- Add `octue` version compatibility checker that uses empirical compatibility data

Enhancements
- Warn if incompatible SDK version detected if an error is raised while handling messages in parent or answering questions in child
- Add recorded questions and empirical version compatibility data for versions since `0.16.0` inclusively

Operations
- Add workflow that checks version compatibility testing has been carried out for the current version
- Add script that prints a table of version compatibilities

<!--- END AUTOGENERATED NOTES --->