Octue

Contents

Fixes
- [x] Add `from_string` option to `Serialisable.deserialise`

Testing
- [x] Mock Google Pub/Sub `Service`, `Topic`, `Subscription`, `Publisher` and `Subscriber` in tests
- [x] Remove unneeded cleanup code from `Service` tests

Contents

Breaking changes
- [x] Remove `TagSet.__str__`

Fixes
- [x] Use TagSet to deserialise tags in `Datafile.from_cloud`
- [x] Add custom (de)serialise methods to `TagSet`
- [x] Return subtags of a `Tag` in order using a `FilterList`
- [x] Remove separate dependencies copy/cache steps in Google Cloud Run Dockerfile so that it works for older versions of `docker`

Minor improvements
- [x] Remove absolute path from `Dataset` and `Manifest` serialisation
- [x] Add `Serialisable.deserialise` method
- [x] Add `filter` method to `TagSet` to avoid e.g. `taggable.tags.tags.filter`

Operations
- [x] Improve description of release workflow

Contents

New features
- [x] Support `setup.py` and `requirements-dev.txt` in Cloud Run Dockerfile
- [x] Retrieve credentials from Google Cloud Secret Manager and inject into environment in `Runner.run`
- [x] Add ability to retrieve and update cloud files via the `Datafile.download` or `Datafile.open` methods
- [x] Allow cloud file attributes to be updated via `Datafile.to_cloud` method
- [x] Allow instantiation of `TagSet`s from JSON-encoded lists

Breaking changes
- [x] Raise error if the datasets of the input manifest passed to `Service.ask` aren't all cloud-based

Fixes
- [x] Fix `Dataset` construction from serialised form in `Manifest`
- [x] Fix `Datafile` construction from serialised form in `Dataset`
- [x] Fix `Datafile.deserialise`
- [x] Adjust usages of `tempfile.NamedTemporaryFile` to also work on Windows
- [x] Add timeout and retry to `Service.answer`
- [x] Add retry to `Service.wait_for_answer`
- [x] Add 60 second timeout for answering question in Cloud Run deployment
- [x] Use correct environment variable for service ID in Cloud Run Dockerfile
- [x] Set `_last_modified`, `size_bytes`, and `_hash_value` to null values if a `Datafile` representing a cloud file is instantiated for a hypothetical cloud location (i.e. not synced to a cloud file at that point in time)
- [x] Allow `Dataset.get_file_sequence` use with no filter

Dependencies
- [x] Use new `twined` version that supports validation of `credentials` strand
- [x] Use newest version of `gcp-storage-emulator`

Minor improvements
- [x] Make `path` a positional argument of `Datafile`
- [x] Move `gunicorn` requirement into `octue` requirements
- [x] Raise warning instead of error if Google Cloud credentials environment variable is not found and return `None` as credentials
- [x] Move cloud code into new `cloud` subpackage
- [x] Raise `TimeoutError` in `Service.wait_for_answer` if no response is received by end of retries
- [x] Only look for `deployment_configuration.json` file in docker container `/app` directory
- [x] Ensure `deployment_configuration.json` file is always loaded correctly in docker container
- [x] Pass credentials strand into `Runner` instance in Cloud Run deployment
- [x] Add `name` attribute to `Identifiable` mixin
- [x] Add Google Cloud metadata to `Datafile` serialisation
- [x] Add `deserialise` method to `Datafile`
- [x] Add ability to add metadata to a `Datafile` instantiated from a regular cloud file
- [x] Use CRC32C hash value from Google Cloud when instantiating a `Datafile` from the cloud
- [x] Add ability to name `Datafile`s
- [x] Add ability to check whether a `Datafile`, all `Datafile`s in a `Dataset`, or all `Dataset`s in a `Manifest` are located in Google Cloud
- [x] Use `Datafile.deserialise` when instantiating a `Dataset` from a dictionary
- [x] Add representation to `GCPPubSubBackend`
- [x] Load credentials strand JSON in `Runner` initialisation
- [x] Add location searched to message of error raised when `app` module can't be found in `Runner.run`
- [x] Ignore `E203` flake8 warning

Testing
- [x] Remove subjective `Service` test `test_serve_with_timeout`
- [x] Use temporary file rather than temporary directory for tests where possible
- [x] Test `Dataset.deserialise`

Quality Checklist
- [x] New features are fully tested (No matter how much Coverage Karma you have)

Coverage Karma
- [x] If your PR decreases test coverage, do you feel you have built enough `Coverage Karma`* to justify it?

Contents

New Features

- [x] Add Google Cloud Run deployment for services

Breaking changes

- [x] Move most parameters from `Runner.run` to `Runner.__init__` (this avoids the need for partial functions)
- [x] Split `Service.answer` into two methods
- [x] Return question UUID from `Service.ask`

Minor fixes and improvements

- [x] Use CRC32C hash function instead of Blake3 (due to extra requirements of Blake3 and the fact that Google Cloud uses CRC32C)
- [x] Use default Google credentials in Pub/Sub service if `GCPPubSubBackend.credentials_environment_variable` is `None`
- [x] Add representations to `Topic` and `Subscription`
- [x] Ensure all topic/subscription names start with their provided namespace (and ensure the namespace appears only once)
- [x] Give `Service`s a random UUID as an ID if none is provided
- [x] Give `GCPPubSubBackend` a default value for the credentials environment variable
- [x] Ensure GCP Storage paths always have the correct path separator
- [x] Fix other Windows path issues
- [x] Remove unused `copy_template` function

Testing
- [x] Add automated testing for Windows and MacOS (in addition to Ubuntu)
- [x] Use `tox` for cross-platform testing
- [x] Use `sys.executable` instead of `python` in `subprocess.Popen` calls to ensure the virtual environment's python executable is used
- [x] Ensure test paths are agnostic of operating system

Quality Checklist

- [x] New features are fully tested (No matter how much Coverage Karma you have)

Coverage Karma

- [x] If your PR decreases test coverage, do you feel you have built enough `Coverage Karma`* to justify it?

Contents

Minor fixes and improvements

- [x] Remove test bucket environment variable
- [x] Remove environment variable default argument from `GoogleCloudStorageEmulator`
- [x] Add installation, usage, and testing instructions to README

Testing
- [x] Test ability to start more than one Google Cloud Storage emulator at once

Contents

New Features
- [x] Move Google Cloud Storage emulator into octue package, making it importable

Minor fixes and improvements

- [x] Allow storage emulator to find and use a free port
- [x] Remove need for `STORAGE_EMULATOR_HOST` environment variable for tests
- [x] Avoid assuming custom metadata is set in storage client
- [x] Move `unittest.TestResult` method replacements into Google Cloud emulators module
- [x] Remove `tox` from CI tests, using just GitHub actions instead