Autotiktokenizer

Latest version: v0.2.2

Safety actively analyzes 710445 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 1 of 2

0.2.2

What's Changed
* [BUG] Fix path to support windows machines by not-lain in https://github.com/chonkie-ai/autotiktokenizer/pull/12
* support local tiktokenizer loading by not-lain in https://github.com/chonkie-ai/autotiktokenizer/pull/13
* Fix 10: Add local tokenizer support by bhavnicksm in https://github.com/chonkie-ai/autotiktokenizer/pull/16
* Bump version to 0.2.2 and update __init__.py to reflect the new version by bhavnicksm in https://github.com/chonkie-ai/autotiktokenizer/pull/17

New Contributors
* not-lain made their first contribution in https://github.com/chonkie-ai/autotiktokenizer/pull/12

**Full Changelog**: https://github.com/chonkie-ai/autotiktokenizer/compare/v0.2.1...v0.2.2

0.2.1

What's Changed
* Add support for WordPiece Tokenizers + Aya Family by bhavnicksm in https://github.com/bhavnicksm/autotiktokenizer/pull/9


**Full Changelog**: https://github.com/bhavnicksm/autotiktokenizer/compare/v0.2.0...v0.2.1

0.2.0

What's Changed
* Remove tokenizers dependency from AutoTikTokenizers by bhavnicksm in https://github.com/bhavnicksm/autotiktokenizer/pull/8


**Full Changelog**: https://github.com/bhavnicksm/autotiktokenizer/compare/v0.1.2...v0.2.0

0.1.2

What's Changed
* Update README.md by bhavnicksm in https://github.com/bhavnicksm/autotiktokenizer/pull/5
* Update README.md by bhavnicksm in https://github.com/bhavnicksm/autotiktokenizer/pull/6
* Bump version to 0.1.2 and update package metadata by bhavnicksm in https://github.com/bhavnicksm/autotiktokenizer/pull/7


**Full Changelog**: https://github.com/bhavnicksm/autotiktokenizer/compare/v0.1.1...v0.1.2

0.1.1

What's Changed
* Bump version to 0.1.1 and enhance _AutoTikTokenizer functionality by bhavnicksm in https://github.com/bhavnicksm/autotiktokenizer/pull/4


**Full Changelog**: https://github.com/bhavnicksm/autotiktokenizer/compare/v0.1.0...v0.1.1

0.1.0

What's Changed
* Add logo + Bump version to v0.1.0 by bhavnicksm in https://github.com/bhavnicksm/autotiktokenizer/pull/3


**Full Changelog**: https://github.com/bhavnicksm/autotiktokenizer/compare/v0.0.1a0...v0.1.0

Page 1 of 2

© 2025 Safety CLI Cybersecurity Inc. All Rights Reserved.