Curated-tokenizers

Latest version: v2.0.0

Safety actively analyzes 688724 Python packages for vulnerabilities to keep your Python projects secure.

Page 2 of 2

* Add new `WordPieceProcessor` methods, including `decode` and `id_to_piece` (https://github.com/explosion/cutlery/pull/35).

* Add `ByteBPEProcessor.decode_from_ids`, which returns the decoded string from piece ids (33).

* Add missing `regex` install dependency (25).

* Add `WordPieceProcessor` (18).
* Add `ByteBPEProcessor` (21).

Has known vulnerabilities