Octue

Contents

New Features
- [x] Make `Datafile`s and `Dataset`s labelable
- [x] Use new version of tags in all `Taggable`s
- [x] Replace string tags in a `TagSet` with key-value pairs in a `TagDict`
- [x] Add new `Taggable` mixin for providing the new tags interface
- [x] Add `FilterDict`, allowing filtering of key-value pairs by their values
- [x] Allow nested attribute/dictionary filtering in the filter containers `FilterSet`, `FilterList` and `FilterDict`
- [x] Allow any number of filters to be specified when filtering in filter containers
- [x] Allow ignoring of filterables missing the filtered-for attribute in a filter container instead of raising an error
- [x] Add filter container `one` method?
- [x] Allow ordering by nested attributes in all `FilterContainer`s
- [x] Allow `gs://` paths to be used in `Datafile`, `Dataset`, and `Manifest`
- [x] Allow `gs://` paths to be used in storage client
- [x] Add `datetime` filters
- [x] Add in-range filters to `str`, `datetime`, and `Number` filters

Breaking changes
- [x] Use new format for manifests' datasets in `twine.json` files
- [x] Convert old `Taggable` mixin to `Labelable` mixin
- [x] Convert old `Tag` class to `Label` class
- [x] Convert `TagSet` to `LabelSet`
- [x] Use key-value pairs for filter names and values when filtering `Filterable`s
- [x] Stop logging in `Serialisable`
- [x] Always exclude `logger` field in `Serialisable`
- [x] Simplify tag name pattern to `^[a-z0-9][a-z0-9_]*(?<!_)$`
- [x] Simplify label pattern to `^[a-z0-9][a-z0-9-]*(?<!-)$`
- [x] Store tags as key-value pairs in GCS custom metadata
- [x] Unbase `TagDict` and `LabelSet` from filter containers
- [x] JSON-encode cloud storage custom metadata again
- [x] Store tags in `tags` field of cloud metadata again
- [x] Close 165: prefix GCS custom metadata fields with "octue__"

Minor improvements
- [x] Remove `filters` field from manifest strand in twines
- [x] Allow tags to be added via `kwargs` in `Taggable.add_tags`
- [x] Remove unused `_FILTERSET_ATTRIBUTE` class variables
- [x] Base `Label` on `str`
- [x] Support non-English characters in case-insensitive filtering
- [x] Add `octue-sdk-python` version to datafile metadata
- [x] Base filter containers on new `FilterContainer` abstract class
- [x] Move `filter` and `order` methods into `FilterContainer`
- [x] Use `OctueJSONDecoder` in `Serialisable` and `GoogleCloudStorageClient`
- [x] Add de/serialisation of `datetime` objects to de/encoders
- [x] Clarify name of some `GoogleCloudStorageClient` methods
- [x] Add `set` and `UserString` encoding to `OctueJSONEncoder`
- [x] Use `OctueJSONDecoder`
- [x] Add `set` and `datetime` decoding to `OctueJSONDecoder`
- [x] Remove unnecessary methods from `LabelSet`
- [x] Rename `add_labels` method and add `add` method to `LabelSet`
- [x] Automatically generate complementary (`not`) filters from other filters
- [x] Remove a line of duplicated code in `Datafile`

Fixes
- [x] Handle timestamps from cloud with/without timezone information
- [x] Fix `OctueJSONDecoder`
- [x] Make it harder to add invalid labels to `LabelSet`

Dependencies
- [x] Use new version of `twined` that distinguishes tags from labels

Testing
- [x] Use latest GCS emulator
- [x] Only run deployment test if `RUN_DEPLOYMENT_TESTS` envvar is `True`

Quality Checklist
- [x] New features are fully tested (No matter how much Coverage Karma you have)
- [x] **[v0.2 onward]** New features are included in the documentation

Contents

New features
- [x] Allow decimal points in tags

Minor improvements
- [x] Close 162: make timestamp an optional parameter for Datafile


Quality Checklist

- [x] New features are fully tested (No matter how much Coverage Karma you have)

Contents

New Features
- [x] Allow `Datafile` to be used as a context manager for changes to local datafiles
- [x] Allow `Datafile.from_cloud` to be used as a context manager for changes to cloud datafiles
- [x] Allow `Datafile` to remember where in the cloud it came from
- [x] Add the following methods to `Datafile`:
- `get_cloud_metadata`
- `update_cloud_metadata`
- `clear_from_file_cache`
- `_get_cloud_location`
- `_store_cloud_location`
- `_check_for_attribute_conflict`
- [x] Avoid re-uploading `Datafile` file or metadata if they haven't changed
- [x] Raise error if implicit cloud location is missing from `Datafile`
- [x] Add `GoogleCloudStorageClient.update_metadata` method
- [x] Allow option to not update cloud metadata in `Datafile` cloud methods
- [x] Allow tags to contain capitals and forward slashes (but not start or end in a forward slash)
- [x] Allow `datetime` and posix timestamps for `Datafile.timestamp`
- [x] Add `Datafile.posix_timestamp` property

Breaking changes
- [x] Close 148: remove `hash_value` from `Datafile` GCS metadata
- [x] When hashing `Datafile`s, only hash represented file (i.e. stop hashing metadata)
- [x] When hashing `Dataset`s and `Manifest`s, only hash the files contained (i.e. stop hashing metadata)
- [x] Make hash of `Hashable` instance with `_ATTRIBUTES_TO_HASH=None` the empty string hash value `"AAAAAA=="`

Minor improvements
- [x] Simplify output of `GoogleCloudStorageClient.get_metadata`
- [x] Make `Hashable` instances re-calculate their `hash_value` every time unless an `immutable_hash_value` is explicitly provided (e.g. for cloud datafiles where you don't have the file locally to hash)
- [x] Add private `Identifiable._set_id` method
- [x] Close 147: pull metadata gathering for `Datafile` into method
- [x] Get `datetime` objects directly from GCS blob instead of parsing string serialisations
- [x] Add `time` utils module
- [x] Add hash preparation function to `Hashable` for `datetime` instances
- [x] Use the empty string hash value for `Datafile` if GCS `crc32c` metadata isn't present
- [x] Stop serialising hash value of `Manifest`, `Dataset`, and `Datafile`

Fixes
- [x] Close 146: Stop serialising GCS metadata as JSON. This avoids strings in the metadata appearing in two sets of quotation marks on Google Cloud Storage. This is a breaking change for any files already persisted with JSON-encoded metadata.
- [x] Remove ability to set custom hash value via `kwargs` when using `Datafile.from_cloud`

Testing
- [x] Factor out cloud datafile creation in datafile tests

Quality Checklist

- [x] New features are fully tested (No matter how much Coverage Karma you have)

Contents

Breaking changes
- [x] Rename `Service.__init__` parameter `id` to `service_id` to avoid built-in name clash
- [x] Move `deployment` package into `cloud` package

Dependencies
- [x] Use newest version of `twined` to support python>=3.6

Minor improvements
- [x] Remove duplicate code and unnecessary comments from `Runner`
- [x] Raise error if `SERVICE_ID` envvar is missing from deployment environment
- [x] Disallow non-None empty values as `Service` IDs
- [x] Add base class for service backends; update docstrings

Fixes
- [x] Use `OctueJSONEncoder` in JSON serialisation inside `Service.answer` to ensure `numpy` arrays are serialised

Testing
- [x] Add tests for `Topic` and `Subscription`
- [x] Add extra test for `Service`
- [x] Shorten runtime of `cli.start` test

Contents

Fixes
- [x] Add `from_string` option to `Serialisable.deserialise`

Testing
- [x] Mock Google Pub/Sub `Service`, `Topic`, `Subscription`, `Publisher` and `Subscriber` in tests
- [x] Remove unneeded cleanup code from `Service` tests

Contents

Breaking changes
- [x] Remove `TagSet.__str__`

Fixes
- [x] Use TagSet to deserialise tags in `Datafile.from_cloud`
- [x] Add custom (de)serialise methods to `TagSet`
- [x] Return subtags of a `Tag` in order using a `FilterList`
- [x] Remove separate dependencies copy/cache steps in Google Cloud Run Dockerfile so that it works for older versions of `docker`

Minor improvements
- [x] Remove absolute path from `Dataset` and `Manifest` serialisation
- [x] Add `Serialisable.deserialise` method
- [x] Add `filter` method to `TagSet` to avoid e.g. `taggable.tags.tags.filter`

Operations
- [x] Improve description of release workflow