Octue

Contents

New Features

- [x] Add Google Cloud Run deployment for services

Breaking changes

- [x] Move most parameters from `Runner.run` to `Runner.__init__` (this avoids the need for partial functions)
- [x] Split `Service.answer` into two methods
- [x] Return question UUID from `Service.ask`

Minor fixes and improvements

- [x] Use CRC32C hash function instead of Blake3 (due to extra requirements of Blake3 and the fact that Google Cloud uses CRC32C)
- [x] Use default Google credentials in Pub/Sub service if `GCPPubSubBackend.credentials_environment_variable` is `None`
- [x] Add representations to `Topic` and `Subscription`
- [x] Ensure all topic/subscription names start with their provided namespace (and ensure the namespace appears only once)
- [x] Give `Service`s a random UUID as an ID if none is provided
- [x] Give `GCPPubSubBackend` a default value for the credentials environment variable
- [x] Ensure GCP Storage paths always have the correct path separator
- [x] Fix other Windows path issues
- [x] Remove unused `copy_template` function

Testing
- [x] Add automated testing for Windows and MacOS (in addition to Ubuntu)
- [x] Use `tox` for cross-platform testing
- [x] Use `sys.executable` instead of `python` in `subprocess.Popen` calls to ensure the virtual environment's python executable is used
- [x] Ensure test paths are agnostic of operating system

Quality Checklist

- [x] New features are fully tested (No matter how much Coverage Karma you have)

Coverage Karma

- [x] If your PR decreases test coverage, do you feel you have built enough `Coverage Karma`* to justify it?

Contents

Minor fixes and improvements

- [x] Remove test bucket environment variable
- [x] Remove environment variable default argument from `GoogleCloudStorageEmulator`
- [x] Add installation, usage, and testing instructions to README

Testing
- [x] Test ability to start more than one Google Cloud Storage emulator at once

Contents

New Features
- [x] Move Google Cloud Storage emulator into octue package, making it importable

Minor fixes and improvements

- [x] Allow storage emulator to find and use a free port
- [x] Remove need for `STORAGE_EMULATOR_HOST` environment variable for tests
- [x] Avoid assuming custom metadata is set in storage client
- [x] Move `unittest.TestResult` method replacements into Google Cloud emulators module
- [x] Remove `tox` from CI tests, using just GitHub actions instead

Contents

New Features
- [x] Add `GoogleCloudStorageClient`
- [x] Write manifest, its datasets, and its datafiles to cloud in `Analysis.finalise` (96)
- [x] Closes 84 - add auto tag and release workflow
- [x] Allow Google Cloud storage blobs to be represented by `Pathable`
- [x] Add `Datafile`, `Dataset`, and `Manifest` `to_cloud` and `from_cloud` methods
- [x] Allow regular GCP files to be represented as `Datafile`s

Minor fixes and improvements

- [x] Add cloud storage emulator once for all tests
- [x] Add disk usage and file age utilities
- [x] Allow `Dataset`s to have custom names
- [x] Add `storage.path` module akin to `os.path` but for Google Cloud Storage paths
- [x] Allow `Hashable`s' hash values to be set
- [x] Pass GCP project and bucket names to tests from environment (93)
- [x] Add ability to delete topic and subscription when a `Service` has finished serving
- [x] Facilitate graceful exit for serving `Service`s on `KeyboardInterrupt`
- [x] Use latest versions of flake8, isort, and black in pre-commit and across all files (87)
- [x] Fix CI test skipping flag
- [x] Fix documentation links (92)

Breaking changes
- [x] Remove testing and explicit support for `python3.6` and `python3.7`
- [x] Remove `base_from` from Pathable and replace with more transparent method
- [x] Rename `Datafile.posix_timestamp` to `Datafile.timestamp` and remove default value
- [x] Make `Datafile.last_modified` private
- [x] Rename `persistence` subpackage to `storage`

Testing
- [x] Test that children can question their own children as part of answering a question
- [x] Close 94 - delete topics and subscriptions at the end of each test
- [x] Remove timeouts from tests and replace with thread executor shutdown upon test pass, meaning that tests that connect to Google Pub/Sub won't fail because the connection is slower than expected

Contents

New Features

- [x] Enable use of child services - solving 46.
- To solve 57 we need to be able to define and run local children as well as remote ones.
- This means we must allow multiple services to run locally and independently...
- Which probably means we can also solve https://github.com/octue/twined-server/issues/2 at the same time
- [x] Enable Documentation Build and Serve, Update README 70
- Ultimately we wish to unify documentation between twined and octue-sdk-python, but this is best done at the time of refactoring large chunks of octue-sdk-python into twined ( See 69 ) but at the moment we wish to just serve what we've got so we can at least link to it.
- [x] Add option to handle developer logs separately from Scientist logs (78)
- [x] Allow skipping of CI tests if skip_ci_tests is in the commit body - the use case is to reduce unnecessary computation when knowing the tests will fail for a commit but still wanting to commit.

Minor fixes and improvements

- [x] Implement a proper issue template, either derived from `.github` repo or applied directly (c.f. https://github.com/octue/twined/issues/60 )
- [x] Close 32 - stop CLI tests leaving output files in working area.

New Features

- [x] Add `Hashable` mixin to create data hashes of `Datafile`s and their containers, including metadata
- [x] Add `Filterable` mixin to make `Datafiles` in a `Dataset` and `Tag`s in a `TagSet` filterable
- [x] Add `FilterSet` class for storing and facilitating filtering of `Datafiles` in `Dataset` and `Tag`s in `TagSet`
- [x] Add `FilterList` for the output of ordering `FilterSet`s
- [x] Add a plethora of type- and interface-based filters for use on `FilterSet`s and `FilterList`s

Breaking changes

- [x] Any `sha256` properties have been replaced with `blake3_hash` properties
- [x] Replace `TagSet.has_tag` with native `__contains__ method
- [x] Rename `TagSet.starts_with` to `TagSet.any_tag_starts_with`
- [x] Rename `TagSet.ends_with` to `TagSet.any_tag_ends_with`

Minor fixes and improvements

- [x] Replace SHA256 hashing with [BLAKE3 hashing](https://github.com/BLAKE3-team/BLAKE3) - this is reportedly around 10 times as fast!
- [x] Neaten up 40/42 by combining the two workflow files while still keeping separate checks on GitHub. This also makes the version check display less verbosely on GitHub
- [x] Add `Tag` class, neatening up the retrieval and filtering of subtags
- [x] Rename `TagGroup` to `TagSet` to reflect that it's set-based