Funcx-common

Added

- A new subpackage, `funcx_common.messagepack` provides an implementation of
serialization and deserialization of message objects meant to be compatible
with `funcx_endpoint.executors.high_throughput.messages` in its on-the-wire
representation of messages. Changes between the two implementations are
noted in a README for `messagepack`.

- `messagepack` defines protocol versions, starting with the current and
unversioned v0, for handling changes to the protocol over time

- only v0 is implemented, but a suggested plan for v1 of the protocol can be
seen in the `messagepack` readme

- Add the `funcx_common.task_storage.get_default_task_storage()` method, which
reads the `FUNCX_REDIS_STORAGE_THRESHOLD` and `FUNCX_S3_BUCKET_NAME` environment
variables and constructs the appropriate TaskStorage object

- Detection of S3/Redis vs Redis-only storage is done by the presence/absence
of the `FUNCX_S3_BUCKET_NAME` variable. It can be forced to redis-only by
setting `FUNCX_REDIS_STORAGE_THRESHOLD="-1"`

- The storage threshold has a default value of 20,000 if not set

- Add `funcx_common.task_storage.ImplicitRedisStorage` as a storage class which
only reads and writes the task object, on the assumption that it is a
RedisTask

Changed

- RedisS3Storage now contains an ImplicitRedisStorage object which is used to
implement read/write to redis

- RedisS3Storage now requires `bucket_name` and `redis_threshold` as
keyword-only arguments, if constructed directly

Added

- A new TaskStorage class that is designed to abstract the storage systems for result or payload blobs
- A RedisS3Storage implementation which is ready for use for storing results.

- This will store small results in Redis and large results in S3 using a
configurable threshold.
- The write to redis is implicit, assuming that task objects define `result`
as a `RedisField`

- TaskProtocol now defines a new dict attribute: `result_reference`, which is
written and read by the TaskStorage component. Tasks must add an
implementation of this field to comply with the protocol.

Removed

- The `HasRedisConnection` class has been removed

Added

- Redis connections created by `default_redis_connection_factory` now default
to loading a redis URL from an environment variable, `FUNCX_COMMON_REDIS_URL`,
with a default of `redis://localhost:6379`

Changed

- The redis logging decorator has been moved. It is no longer attached to a
class, but is now available as `funcx_common.redis.redis_connection_error_logging`
- The signatures for creating `FuncxRedisPubSub` and `FuncxEndpointTaskQueue`
have changed. They can now be passed a `redis.Redis` object, and default to
calling `default_redis_connection_factory`

Removed

- `funcx_common.redis.HasRedisConnection` no longer provides the
`log_connection_errors` decorator

- `funcx_common.redis.FuncxRedisPubSub` and `funcx_common.redis.FuncxEndpointTaskQueue`
no longer automatically log redis connection errors to the `funcx_common` logger. Users
of these classes should either handle these errors themselves or make use of the new
error logging decorator

Added

- A `funcx_common.redis.HasRedisConnection` object now accepts a
`redis_connection_factory` callable which is used to instantiate the `redis.Redis`
object, to allow overrides to the connection construction process. The callable
must have the signature `(str, int) -> redis.Redis`, and the default
is a function visible under the name
`funcx_common.redis.default_redis_connection_factory`.

- A new context-manager is available as a method of the `HasRedisConnection` class,
`HasRedisConnection.connection_error_logging`. This can be used to capture
and log (at exception-level) redis connection errors to the `funcx_common` logger

- The default connection construction for `HasRedisConnection` now sets
`health_check_interval=30`. To override this setting, use
`redis_connection_factory`

Changed

- Begin using `scriv` to manage the changelog
-
- The `FuncxRedisConnection` class has been renamed to
`HasRedisConnection`. The name change better indicates that this class
is not a connection itself -- it's just an inheritable way of constructing
a connection.

- Add `FuncxRedisPubSub`, which provides a pubsub wrapper over the
`redis` library's `PubSub` client

- Fix mypy type inference on RedisField descriptors