Tiktoken

Latest version: v0.9.0

Safety actively analyzes 723158 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 2 of 3

0.5.0

- Add `tiktoken._educational` submodule to better document how byte pair encoding works
- Ensure `encoding_for_model` knows about several new models
- Add `decode_with_offets`
- Better error for failures with the plugin mechanism
- Make more tests public
- Update versions of dependencies

0.4.0

- Add `decode_batch` and `decode_bytes_batch`
- Improve error messages and handling

0.3.3

- `tiktoken` will now make a best effort attempt to replace surrogate pairs with the corresponding
Unicode character and will replace lone surrogates with the Unicode replacement character.

0.3.2

- Add encoding for GPT-4

0.3.1

- Build aarch64 wheels
- Make `blobfile` an optional dependency

Thank you to messense for the environment variable that makes cargo not OOM under emulation!

0.3.0

- Improve performance by 5-20%; thank you to nistath!
- Add `gpt-3.5-turbo` models to `encoding_for_model`
- Add prefix matching to `encoding_for_model` to better support future model versions
- Fix a bug in the README instructions on extending tiktoken
- Update the set of available encodings
- Add packaging metadata

Page 2 of 3

© 2025 Safety CLI Cybersecurity Inc. All Rights Reserved.