Octue

Summary
Improve the creation experience of Octue services.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([548](https://github.com/octue/octue-sdk-python/pull/548))

New features
- Add ability to set up periodic monitor messages from `Analysis`
- Add poetry installation option to Cloud Run Dockerfile

Enhancements
- Allow revision tag to be set in `octue deploy` commands
- Tell user SRUID of service being deployed by `octue deploy`
- Change default internal service name to `local/local:local`
- Allow Cloud Run execution environment to be set by deployer

Fixes
- Ensure service revision tag is set as environment variable in deployers

Dependencies
- Update `flake8` used for pre-commit checks
- Update lockfile to address certificate removal in `certifi`
- Use new poetry format for dev dependencies in `pyproject.toml`

Testing
- Update `tox.ini` to new format

<!--- END AUTOGENERATED NOTES --->

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([545](https://github.com/octue/octue-sdk-python/pull/545))

Fixes
- Avoid double-JSON-encoding input manifests in `Service.ask`

Dependencies
- Update lock file

<!--- END AUTOGENERATED NOTES --->

Summary
This release fixes and adds new features to service crash diagnostics, making it much easier to locally reconstruct errors in a service. This especially true if it has children - crash diagnostics now include a child emulator for every question asked to a child by the service before it failed.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([541](https://github.com/octue/octue-sdk-python/pull/541))

**IMPORTANT:** There are 3 breaking changes.

New features
- Add ability to load crash diagnostics data as a test fixture
- Record messages from each of a service's children when it crashes

Enhancements
- 💥 **BREAKING CHANGE:** Avoid downloading datasets by default in `get-crash-diagnostics`
- 💥 **BREAKING CHANGE:** Save recorded messages to attribute instead of file
- Log a warning if trying to download files from empty directory
- Add `Serialisable.from_file` method
- Add `Manifest.update_dataset_paths` method
- Add representation for `Runner`
- Avoid uploading null strings when values or manifests not present
- Add option to include `.octue` files when instantiating datasets
- Add ability to filter the files downloaded in `GoogleCloudStorageClient.download_all_files`

Fixes
- Save original unmutated input and configuration data for crash diagnostics
- Update manifests with local paths if datasets downloaded locally
- Ensure crash diagnostics upload works without manifests
- Ensure empty values or manifests are uploaded for crash diagnostics
- Fix splitting bucket name from signed cloud storage URLs

Refactoring
- 💥 **BREAKING CHANGE:** Rename `filename` parameter to `path` in `Serialisable.to_file`
- Factor out child instantiation in `Runner` into a method
- Move `AnalysisLogHandlerSwitcher` into `log_handlers` module
- Move `AppFrom` into new `app_loading` module
- Remove unused `Service._sent_messages` attribute
- Rename `recorded_messages` attributes to `received_messages` properties
- Factor out manifest/dataset and values uploading for crash diagnostics
- Reduce nesting in `Dataset._instantiate_from_cloud`
- Factor out app loading and running in `Runner`

Dependencies
- Update `poetry.lock` to avoid usage of [insecure `py` package](https://github.com/octue/octue-sdk-python/security/dependabot/11) by `pytest`

Testing
- Make testing downloading crash diagnostics more efficient
- Test crash diagnostics with service that has its own children
- Use test fixture to speed up `octue get-crash-diagnostics` tests
- Test that dataset/file metadata is uploaded with crash diagnostics
- Test metadata is preserved with `octue get-crash-diagnostics`

Other
- Clarify service troubleshooting doc

---
Upgrade instructions
<details>
<summary>💥 <b>Avoid downloading datasets by default in `get-crash-diagnostics`</b></summary>

To keep the previous behaviour, add the `--download-datasets` flag when using the `octue get-crash-diagnostics` CLI command
</details>

<details>
<summary>💥 <b>Save recorded messages to attribute instead of file</b></summary>

Instead of using the `record_messages_to` parameter, set the `record_messages` parameter to `True` when using `Child.ask` or `Service.wait_for_answer`. Retrieve messages from the `received_messages` property of either class instance.
</details>

<details>
<summary>💥 <b>Rename filename parameter to path in Serialisable.to_file</b></summary>

If using the positional argument `filename` in the `to_file` method of an instance of a `Serialisable` mixed-in class as a keyword argument, replace it with `path`.
</details>

<!--- END AUTOGENERATED NOTES --->

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([536](https://github.com/octue/octue-sdk-python/pull/536))

Fixes
- Always use `OctueJSONEncoder` when encoding JSON
- Log analysis errors before attempting to save crash diagnostics in `Runner`

Operations
- Update author email address

Refactoring
- Use `Manifest.serialise` instead of `Manifest.to_primitive` in `Manifest.to_cloud`

<!--- END AUTOGENERATED NOTES --->

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([533](https://github.com/octue/octue-sdk-python/pull/533))

Operations
- Use new version of `octue/check-semantic-version` action and remove `mkver.conf` file
- Use latest `octue/generate-pull-request-description` action
- Add `prettier` to `pre-commit` checks

Other
- Fix `octue.yaml` code block in docs
- Improve troubleshooting services doc
- Update service ID usage in docs

<!--- END AUTOGENERATED NOTES --->

Summary
This release add support and requirement for service namespaces and service revision tags. This allows services to be versioned and provides a more familiar format for service IDs that's, for example, similar to Docker image names (e.g. `octue/example-service:latest`). Full validation of service IDs comes as part of this.

The new `octue deploy create-push-subscription` has also been added as requested by users and will be accompanied by a GitHub action in another repository.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([529](https://github.com/octue/octue-sdk-python/pull/529))

**IMPORTANT:** There are 4 breaking changes.

New features
- 💥 **BREAKING CHANGE:** Add support for and require revision tags for services
- 💥 **BREAKING CHANGE:** Validate service IDs before creating or questioning services
- Add ability to set service namespace, name, and revision tag by environment variable
- Add `octue deploy create-push-subscription` CLI command

Enhancements
- 💥 **BREAKING CHANGE:** Require services to have a namespace and replace use of `organisation` with `namespace` in service configurations
- 💥 **BREAKING CHANGE:** Remove `namespace` parameter from `Topic` and `Subscription` and apply `octue.services` Pub/Sub namespace to all topic/subscription paths
- Remove application of `octue.services` Pub/Sub namespace from `Service`
- Use a `coolname` revision tag to allow a service revision with a non-unique SRUID to start as a unique service revision when using the `octue start` CLI command
- Set the `Service.id` attribute to the user-friendly SRUID instead of the Pub/Sub-friendly SRUID
- Return SRUID from `DataflowDeployer.deploy`

Fixes
- Handle inability to find/load version compatibility data file
- Use latest Apache Beam base image in Dataflow `Dockerfile`
- Fix some log and error messages

Operations
- Speed up and simplify `release` workflow
- Use new Octue GitHub actions in workflows
- Improve installation process in main `Dockerfile`
- Use latest commit message checker

Dependencies
- Update Dataflow `setup.py` file

Refactoring
- Rename `OCTUE_NAMESPACE` to `OCTUE_SERVICES_NAMESPACE`
- Remove cool-naming of services when instantiated without `id` and simplify the `name` argument

---
Upgrade instructions
<details>
<summary>💥 <b>Add support for and require revision tags for services</b></summary>

The `ServiceConfiguration.service_id` attribute and the `SERVICE_ID` environment variable have been removed.

- **`ServiceConfiguration`:** Use `octue.cloud.service_id.create_service_id` to create IDs from `ServiceConfiguration.namespace` and `ServiceConfiguration.name`

- **`SERVICE_ID` environment variable:** Use the new `OCTUE_SERVICE_NAMESPACE`, `OCTUE_SERVICE_NAME`, and `OCTUE_SERVICE_REVISION_TAG` environment variables.
</details>

<details>
<summary>💥 <b>Validate service IDs before creating or questioning services</b></summary>

Update your services' namespaces, names, and revision tags to follow the requirements set out in the `Creating services` doc.
</details>

<details>
<summary>💥 <b>Require services to have a namespace and replace use of organisation with namespace in service configurations</b></summary>

Provide the `namespace` key in the service configuration (the `octue.yaml` file). If you were using the `organisation` key before, the `namespace` key now replaces it. Examples of a `namespace` are your organisation's name or your GitHub username.
</details>

<details>
<summary>💥 <b>Remove namespace parameter from Topic and Subscription and apply octue.services Pub/Sub namespace to all topic/subscription paths</b></summary>

The `octue.services` namespace is now mandatory and applied automatically in topic and subscription paths. Please stop providing the `namespace` argument if you were providing it before.
</details>

<!--- END AUTOGENERATED NOTES --->