Daskqueue

Latest version: v0.2.0

Safety actively analyzes 622867 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

0.2.0

Durable queues:
- This is a disk-backed queue for persisting the messages.
- The tasks are served in a FIFO manner.
- Durable queues append serialized messages to a fixed-sized file called `LogSegment`.
- The durable queues also append queue operations to an `IndexSegment`.
- The index segment serves as both a :
- [Bitcask](https://riak.com/assets/bitcask-intro.pdf) index for storing the messages offsets
- A WAL file: it is a Log where we append message status after each queue operation (ready, delivered, acked, and failed) and an offset to the message in one of the `LogSegments`. At init time, we just scan this file to rebuild the in-memory data structure.

Consumers
- Implement an acking mechanism for consumers.
- Changes the consumer `join()` to wait for the queue to be empty before canceling the consumers.

0.1.6

Fixes :
- Batch submit when len(items) < nb queues
- Progress print prettier

0.1.5

Changed scheduling to improve performance :

PR : 12

Key changes :
- Faster submission using `batch_submit`
- Consumer can be start before submission and will not exit until first item
- Items are fairly distributed across queue_pool
- Consumer pulls directly from predefined queue ( before consumer was getting a new queue from queue_pool;) : This may change in the future

0.1.4

Links

Releases

Has known vulnerabilities

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.