Tokenizer

Latest version: v3.4.5

Safety actively analyzes 685525 Python packages for vulnerabilities to keep your Python projects secure.

Page 1 of 8

* Compatibility with Python 3.13
* Now requires Python 3.9 or later

**Full Changelog**: https://github.com/mideind/Tokenizer/compare/3.4.4...3.4.5

* Better handling of abbreviations

**Full Changelog**: https://github.com/mideind/Tokenizer/compare/3.4.3...3.4.4

* Various minor fixes.
* Now requires Python 3.8 or later.

**Full Changelog**: https://github.com/mideind/Tokenizer/compare/3.4.2...3.4.3

- Some abbreviations and phrases added
- META_BEGIN token added to help users distinguish between metatokens and regular tokens

* Improved performance on large input chunks

* Improved handling and normalization of punctuation

Has known vulnerabilities