Octue

Contents

New features
- [x] Support `setup.py` and `requirements-dev.txt` in Cloud Run Dockerfile
- [x] Retrieve credentials from Google Cloud Secret Manager and inject into environment in `Runner.run`
- [x] Add ability to retrieve and update cloud files via the `Datafile.download` or `Datafile.open` methods
- [x] Allow cloud file attributes to be updated via `Datafile.to_cloud` method
- [x] Allow instantiation of `TagSet`s from JSON-encoded lists

Breaking changes
- [x] Raise error if the datasets of the input manifest passed to `Service.ask` aren't all cloud-based

Fixes
- [x] Fix `Dataset` construction from serialised form in `Manifest`
- [x] Fix `Datafile` construction from serialised form in `Dataset`
- [x] Fix `Datafile.deserialise`
- [x] Adjust usages of `tempfile.NamedTemporaryFile` to also work on Windows
- [x] Add timeout and retry to `Service.answer`
- [x] Add retry to `Service.wait_for_answer`
- [x] Add 60 second timeout for answering question in Cloud Run deployment
- [x] Use correct environment variable for service ID in Cloud Run Dockerfile
- [x] Set `_last_modified`, `size_bytes`, and `_hash_value` to null values if a `Datafile` representing a cloud file is instantiated for a hypothetical cloud location (i.e. not synced to a cloud file at that point in time)
- [x] Allow `Dataset.get_file_sequence` use with no filter

Dependencies
- [x] Use new `twined` version that supports validation of `credentials` strand
- [x] Use newest version of `gcp-storage-emulator`

Minor improvements
- [x] Make `path` a positional argument of `Datafile`
- [x] Move `gunicorn` requirement into `octue` requirements
- [x] Raise warning instead of error if Google Cloud credentials environment variable is not found and return `None` as credentials
- [x] Move cloud code into new `cloud` subpackage
- [x] Raise `TimeoutError` in `Service.wait_for_answer` if no response is received by end of retries
- [x] Only look for `deployment_configuration.json` file in docker container `/app` directory
- [x] Ensure `deployment_configuration.json` file is always loaded correctly in docker container
- [x] Pass credentials strand into `Runner` instance in Cloud Run deployment
- [x] Add `name` attribute to `Identifiable` mixin
- [x] Add Google Cloud metadata to `Datafile` serialisation
- [x] Add `deserialise` method to `Datafile`
- [x] Add ability to add metadata to a `Datafile` instantiated from a regular cloud file
- [x] Use CRC32C hash value from Google Cloud when instantiating a `Datafile` from the cloud
- [x] Add ability to name `Datafile`s
- [x] Add ability to check whether a `Datafile`, all `Datafile`s in a `Dataset`, or all `Dataset`s in a `Manifest` are located in Google Cloud
- [x] Use `Datafile.deserialise` when instantiating a `Dataset` from a dictionary
- [x] Add representation to `GCPPubSubBackend`
- [x] Load credentials strand JSON in `Runner` initialisation
- [x] Add location searched to message of error raised when `app` module can't be found in `Runner.run`
- [x] Ignore `E203` flake8 warning

Testing
- [x] Remove subjective `Service` test `test_serve_with_timeout`
- [x] Use temporary file rather than temporary directory for tests where possible
- [x] Test `Dataset.deserialise`

Quality Checklist
- [x] New features are fully tested (No matter how much Coverage Karma you have)

Coverage Karma
- [x] If your PR decreases test coverage, do you feel you have built enough `Coverage Karma`* to justify it?

Contents

New Features

- [x] Add Google Cloud Run deployment for services

Breaking changes

- [x] Move most parameters from `Runner.run` to `Runner.__init__` (this avoids the need for partial functions)
- [x] Split `Service.answer` into two methods
- [x] Return question UUID from `Service.ask`

Minor fixes and improvements

- [x] Use CRC32C hash function instead of Blake3 (due to extra requirements of Blake3 and the fact that Google Cloud uses CRC32C)
- [x] Use default Google credentials in Pub/Sub service if `GCPPubSubBackend.credentials_environment_variable` is `None`
- [x] Add representations to `Topic` and `Subscription`
- [x] Ensure all topic/subscription names start with their provided namespace (and ensure the namespace appears only once)
- [x] Give `Service`s a random UUID as an ID if none is provided
- [x] Give `GCPPubSubBackend` a default value for the credentials environment variable
- [x] Ensure GCP Storage paths always have the correct path separator
- [x] Fix other Windows path issues
- [x] Remove unused `copy_template` function

Testing
- [x] Add automated testing for Windows and MacOS (in addition to Ubuntu)
- [x] Use `tox` for cross-platform testing
- [x] Use `sys.executable` instead of `python` in `subprocess.Popen` calls to ensure the virtual environment's python executable is used
- [x] Ensure test paths are agnostic of operating system

Quality Checklist

- [x] New features are fully tested (No matter how much Coverage Karma you have)

Coverage Karma

- [x] If your PR decreases test coverage, do you feel you have built enough `Coverage Karma`* to justify it?

Contents

Minor fixes and improvements

- [x] Remove test bucket environment variable
- [x] Remove environment variable default argument from `GoogleCloudStorageEmulator`
- [x] Add installation, usage, and testing instructions to README

Testing
- [x] Test ability to start more than one Google Cloud Storage emulator at once

Contents

New Features
- [x] Move Google Cloud Storage emulator into octue package, making it importable

Minor fixes and improvements

- [x] Allow storage emulator to find and use a free port
- [x] Remove need for `STORAGE_EMULATOR_HOST` environment variable for tests
- [x] Avoid assuming custom metadata is set in storage client
- [x] Move `unittest.TestResult` method replacements into Google Cloud emulators module
- [x] Remove `tox` from CI tests, using just GitHub actions instead

Contents

New Features
- [x] Add `GoogleCloudStorageClient`
- [x] Write manifest, its datasets, and its datafiles to cloud in `Analysis.finalise` (96)
- [x] Closes 84 - add auto tag and release workflow
- [x] Allow Google Cloud storage blobs to be represented by `Pathable`
- [x] Add `Datafile`, `Dataset`, and `Manifest` `to_cloud` and `from_cloud` methods
- [x] Allow regular GCP files to be represented as `Datafile`s

Minor fixes and improvements

- [x] Add cloud storage emulator once for all tests
- [x] Add disk usage and file age utilities
- [x] Allow `Dataset`s to have custom names
- [x] Add `storage.path` module akin to `os.path` but for Google Cloud Storage paths
- [x] Allow `Hashable`s' hash values to be set
- [x] Pass GCP project and bucket names to tests from environment (93)
- [x] Add ability to delete topic and subscription when a `Service` has finished serving
- [x] Facilitate graceful exit for serving `Service`s on `KeyboardInterrupt`
- [x] Use latest versions of flake8, isort, and black in pre-commit and across all files (87)
- [x] Fix CI test skipping flag
- [x] Fix documentation links (92)

Breaking changes
- [x] Remove testing and explicit support for `python3.6` and `python3.7`
- [x] Remove `base_from` from Pathable and replace with more transparent method
- [x] Rename `Datafile.posix_timestamp` to `Datafile.timestamp` and remove default value
- [x] Make `Datafile.last_modified` private
- [x] Rename `persistence` subpackage to `storage`

Testing
- [x] Test that children can question their own children as part of answering a question
- [x] Close 94 - delete topics and subscriptions at the end of each test
- [x] Remove timeouts from tests and replace with thread executor shutdown upon test pass, meaning that tests that connect to Google Pub/Sub won't fail because the connection is slower than expected

Contents

New Features

- [x] Enable use of child services - solving 46.
- To solve 57 we need to be able to define and run local children as well as remote ones.
- This means we must allow multiple services to run locally and independently...
- Which probably means we can also solve https://github.com/octue/twined-server/issues/2 at the same time
- [x] Enable Documentation Build and Serve, Update README 70
- Ultimately we wish to unify documentation between twined and octue-sdk-python, but this is best done at the time of refactoring large chunks of octue-sdk-python into twined ( See 69 ) but at the moment we wish to just serve what we've got so we can at least link to it.
- [x] Add option to handle developer logs separately from Scientist logs (78)
- [x] Allow skipping of CI tests if skip_ci_tests is in the commit body - the use case is to reduce unnecessary computation when knowing the tests will fail for a commit but still wanting to commit.

Minor fixes and improvements

- [x] Implement a proper issue template, either derived from `.github` repo or applied directly (c.f. https://github.com/octue/twined/issues/60 )
- [x] Close 32 - stop CLI tests leaving output files in working area.