Summary
This release makes major improvements to event handling and question auditing. Some of the main changes are:
- Questions are now automatically associated with their parent question and the question that originated them, however deep they are in a question tree
- Events are ordered by datetime by the event backend, not the SDK
- Better feedback is provided when asking questions in parallel
- You can specify the event store to use
- Log message contexts have been slimmed down without losing any information, and events are replayable with no context (good for smaller screens)
- Various public classes and functions are faster and easier to use
- Question retries have the same question UUID
<!--- SKIP AUTOGENERATED NOTES --->
Contents ([660](https://github.com/octue/octue-sdk-python/pull/660))
**IMPORTANT:** There are 6 breaking changes.
New features
Events
- 💥 **BREAKING CHANGE:** Add `parent_question_uuid`, `originator_question_uuid`, `originator` and `retry_count` event attributes
- Avoid redelivery of questions by checking the event store on delivery
Event handlers
- Add ability to not include service metadata in logs in even handlers
- Enable `EventReplayer` to handle question events
- Add `RegisteredTemporaryDirectory` class, use it when downloading datasets, and add ability to delete them at end of analysis
Enhancements
Resources
- 💥 **BREAKING CHANGE:** Make datasets recursive by default in `Dataset`
- Log a warning if a dataset is empty at instantiation
Services
- 💥 **BREAKING CHANGE:** Remove `name` argument from `Service` and provide an SRUID to `Child` internal service instead of a name
- Improve logging of errors, retries, and threading in `Child.ask_multiple`
- Order pub/sub messages by datetime using ordering key and remove `order` event attribute
- Set question UUIDs in advance in `Child.ask_multiple`
Subscriptions
- Allow existing subscriptions in `create_push_subscription`
- Give feedback on (un)successful push subscription creation in CLI
Questions and events
- Remove unnecessary `sender` argument from `get_events` and make getting the tail of events the default
- Allow retried questions to have the same UUID
- Allow explicit question retries by using `retry_count` attribute
- Return empty list from `get_events` if no events for question
Service configuration
- Allow setting of event store table ID and `delete_local_files` in service configuration
- Use envvar to specify service configuration location by default
- Add `overrides` option to `Runner.from_configuration`
Other
- Log warning when `PYTHONUNBUFFERED` envvar is unset
- Remove "analysis-" from start of question UUIDs in log context
Fixes
- 💥 **BREAKING CHANGE:** Return question UUID alongside error from `Child.ask_multiple` for failed questions
- Set analysis ID at start of `Runner.run`
- Emit correct logs when no diagnostics available with `octue get-diagnostics`
- Fix deserialisation of events in `get_events`
- Use (meta-)generation agnostic retry strategy with cloud storage
- Return correct question UUIDs with failed questions from `Child.ask_multiple`
- Avoid logging that app failed when it didn't when uploading diagnostics
- Allow setting of `max_workers` when CPU count is indeterminate
- Disable `delete_local_files` by default
Operations
- Update event handler and its bigquery table
Dependencies
- Loosen `Sphinx` and other docs package ranges
- Remove unneeded `db-dtypes` package
- Make `google-cloud-bigquery` a mandatory dependency
- Upgrade `google-cloud-secret-manager`
Refactoring
Event handlers
- 💥 **BREAKING CHANGE:** Remove redundant datetime from delivery ack and heartbeat events
- 💥 **BREAKING CHANGE:** Rename `originator` event attribute to `parent`
- Factor out finalising and cleaning up in `Runner`
- Move service accounts into separate terraform file
- Cache metadata against datafile/dataset instead of path
- Rename python3.9 dockerfile to reflect its python version
---
Upgrade instructions
- Add `recursive=False` to `Dataset` instantiations
- Update all services in your service network to use `octue>=0.56.0`
- Use version `0.6.1` of the event handler or above and a correspondingly up-to-date BigQuery table.
- Swap the `internal_service_name` argument for `internal_sruid` argument to `Child.__init__` and provide a valid SRUID
- Instances of `Service` can no longer be given names. Please give them a valid SRUID instead.
- To get the unraised exception from a failed answer returned by `Child.ask_multiple`, access the zeroth element e.g. if the third question failed:
answers = Child.ask_multiple(*questions)
exception, question_uuid = answers[3]
- `Service.received_events`, `AbstractEventHandler.handled_events`, and `Child.received_events` now include event attributes instead of just the event. These attributes/properties now return a list of dictionaries with the keys {"event", "attributes"}, where what was previously returned is now mapped to the "event" key.
- Stop providing the `recipient` argument to `EventReplayer` and `GoogleCloudPubSubEventHandler` - it's now automatically acquired from each event's attributes
- Stop passing the `skip_missing_events_after` argument to `EventReplayer` and `GoogleCloudPubSubEventHandler`
- Stop using the `awaiting_missing_event` and `time_since_missing_event` properties on the event handlers
<!--- END AUTOGENERATED NOTES --->