With the introduction of the `RedisTaskQueue`, this is another important step towards version 1.0.0. This release also includes various improvements and introduces the deprecation of the `NamespaceLock`.
New in this version
1. You can now set an expiry on a namespace
Starting with this version, you can set an expiry on an entire namespace. This means that when the expiry is reached, the value you get back is `None` instead of the value you had stored previously. The expiry is managed by Redis and has a precision of about 1 ms.
You can set the expiry in two flavours:
1. `cache.set_expiry(seconds: float)` sets an expiry `seconds` in the future. The highest precision is in ms; if you provide more decimal places, they will be truncated.
2. `cache.set_expiry_at(timestamp: float | datetime.datetime)` sets an expiry at the specified timestamp. If you provide a timestamp in seconds, it will be interpreted at seconds since the Unix Epoch (max. precision: 1 ms; actual resolution depends on the platform running Redis). You can also provide a `datetime.datetime` instance, which will be converted into a Unix timestamp. The general recommendation for `datetime.datetime` objects is to use one that is timezone aware or a naive instance in a timezone matching the timezone settings of the current machine.
2. Compound operations are now atomic from a Redis perspective
Some operations, like `RedisCache.pop`, require the execution of multiple Redis commands. As this creates sensitivity to race conditions, namespaces were locked to a single operation at a time. However, this still meant that race conditions could exist in a multi-client setup. This is obviously not ideal.
To prevent those kind of race conditions, compound actions are now executed as a Redis Lua script. This means that their execution is atomic within Redis itself, eliminating potential race conditions between clients.
Another advantage is that the `namespace_lock` decorator was susceptible to introducing deadlock conditions: If one locked method called another method that required the same lock, the namespace would be locked "forever". Using the new scripting approach, such a lock is no longer necessary.
3. Consume Queue items using asynchronous iteration
The `RedisCache` class now supports asynchronous iteration. By default, a `async for task in queue` will consume all tasks in the queue before stopping. Alternatively, the `iter_tasks` methods can be used to start an iteration that waits for new tasks to become available, optionally waiting only for the set `timeout`.
4. New datatype: `RedisTaskQueue`
A `RedisTaskQueue` is very similar to a `TaskQueue` except that instead of consuming elements from the queue directly, it tracks tasks until they're completed. The way this works is that tasks are moved to a "pending" queue from which they are only removed when a task is marked as done.
To make working with this new tasks easier, they are wrapped in a `RedisTask` instance, with methods to `finalize` (mark as done) and `reschedule` (move back to the main queue) the task, if needed.
In general, clients should reschedule all tasks in the "pending" queue at start up using the `reschedule_all_client_tasks` method to clean up lingering tasks from the previous client application run.
To support a multi-client setup, each client can set a unique ID, meaning that they will get their own, unique "pending" queue. Do make sure that client IDs persist through restarts of the client, otherwise the previous "pending" queue would become inaccessible.
Deprecation of `NamespaceLock`
Related to making RedisCache operations atomic, the `NamespaceLock` has been deprecated and will be removed in version 1.0.0. While the `NamespaceLock` class and `namespace_lock` decorator will continue to function until version 1.0.0, using them is discouraged, as it's easy to get into a deadlock situation.
If you're running a single-client setup and want to make two of your own functions/methods atomic/mutually exclusive, consider using the `RedisObject.atomic_transaction` decorator, which available on all Redis types.