Octue

Summary
Add some performance and reliability improvements, including a local metadata file memory cache to speed up local metadata access for datafiles and datasets. Also set push subscriptions made via the CLI to never expire by default.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([567](https://github.com/octue/octue-sdk-python/pull/567))

Enhancements
- Cache local metadata files' contents in memory to avoid wasted time repeatedly access the disk
- Break from datafile finding loop earlier if not in recursive mode

Fixes
- Set push subscriptions created via CLI to never expire by default

Refactoring
- Add and use local metadata context manager when updating local metadata files

<!--- END AUTOGENERATED NOTES --->

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([563](https://github.com/octue/octue-sdk-python/pull/563))

Enhancements
- Allow ignoring stored dataset and datafile metadata when instantiating manifests

<!--- END AUTOGENERATED NOTES --->

Summary
After completion of an analysis, delete any files downloaded over its duration so the storage available to services isn't used up unnecessarily. This update frees service authors from having to deal with this themselves.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([562](https://github.com/octue/octue-sdk-python/pull/562))

Fixes
- After completion of an analysis, delete any files downloaded over its duration

Testing
- Avoid test logging issue
- Move runner tests into correct test class

<!--- END AUTOGENERATED NOTES --->

Summary
Make a number of improvements and fixes to message handling when using pull subscriptions. Also update the small amount of testing that interacts with GCP to use a dedicated separate GCP project.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([558](https://github.com/octue/octue-sdk-python/pull/558))

Enhancements
- Increase number of questions that can be asked concurrently in `Child.ask_multiple` to 32
- Make delivery acknowledgement and maximum hearbeat interval kwargs available in `Child.ask`
- Allow parents to start handling child responses from the first non-missed message (`n + 1`) if the first `n` were missed
- Add question UUID to heartbeat log messages
- Improve `PushSubscriptionCannotBePulled` error message

Fixes
- Mark question as delivered on receipt of first response from child in case the delivery acknowledgement message is missed
- Stop loss of delivered question UUIDs if local metadata file does not yet exist
- Avoid message gap greater than the delivery acknowledgement timeout causing failure to receive child messages
- Allow a start time of zero in message handler

Dependencies
- Update to latest versions of `protobuf` and `werkzeug` to avoid security issues

Operations
- Add terraform configuration for new test project

Refactoring
- Simplify nested conditional
- Minimise code within try/except block in `OrderedMessageHandler`
- Move message recording into `OrderedMessageHandler._handle_message`
- Factor out raising message handling error in message handler
- Rename `OrderedMessageHandler.received_messages` to `handled_messages`

Testing
- Use new URI for Strands JSON schemas in tests
- Use dedicated GCP project for testing services
<!--- END AUTOGENERATED NOTES --->

Summary
Improve the Cloud Run and Dataflow deployment dockerfiles and enable `poetry`-based packages on Dataflow.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([554](https://github.com/octue/octue-sdk-python/pull/554))

Enhancements
- Use `windpioneers/gdal-python` as base in Cloud Run default deployment `Dockerfile`
- Use latest `apache-beam` base for Dataflow default deployment dockerfile
- In both the Cloud Run and Dataflow default deployment dockerfiles:
- Improve dependency caching
- Remove unnecessary installation of `curl`
- Only install from one type of dependency file while building
- Use `/workspace` as `WORKDIR` in Dockerfile
- Use `/root/.poetry` for `POETRY_HOME`

Fixes
- Always install `poetry` in deployment dockerfiles

Operations
- Use latest `octue/check-semantic-version` GitHub action
<!--- END AUTOGENERATED NOTES --->

Summary
Enable automated code security scanning via GitHub's CodeQL.

<!--- SKIP AUTOGENERATED NOTES --->
Contents ([553](https://github.com/octue/octue-sdk-python/pull/553))

Operations
- Add `codeql.yml` workflow to enable code scanning

<!--- END AUTOGENERATED NOTES --->