Curated-tokenizers

Latest version: v2.0.0

Safety actively analyzes 622918 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 1 of 2

* Improve Byte BPE encoding speed (57)

* Make piece processors picklable (53, 54).

* Support loading the ByteBPEProcessor and SentencePieceProcessor from files (51).

* Make piece processors picklable (53, 54).

The `cutlery` package is renamed to `curated-tokenizers`.

* Add docstrings, `piece_to_id` and `id_to_piece` to `SentencePieceProcessor` (https://github.com/explosion/cutlery/pull/37).

Has known vulnerabilities