Bpetokenizer

Latest version: v1.2.1

Safety actively analyzes 665899 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

Page 1 of 2

1.2.1

What's Changed
* feat: starttime-endtime added with the throughput on verbose by Hk669 in https://github.com/Hk669/bpetokenizer/pull/10
* Updates for the pretrained tokenizers. by Hk669 in https://github.com/Hk669/bpetokenizer/pull/11


**Full Changelog**: https://github.com/Hk669/bpetokenizer/compare/v1.2.0...v1.2.1

1.2.0

What's Changed
* deprecated the __version__ check when loading by Hk669 in https://github.com/Hk669/bpetokenizer/pull/8


**Full Changelog**: https://github.com/Hk669/bpetokenizer/compare/v1.0.4...v1.2.0

1.0.32

**Full Changelog**: https://github.com/Hk669/bpetokenizer/compare/v1.0.31...v1.0.32

- added hyperparameter `min_frequency` to adjust the merge pairs to avoid extra vocab.
- default is set to 2.
- made some changes in the tests.

1.0.31

**Full Changelog**: https://github.com/Hk669/bpetokenizer/compare/v1.0.3...v1.0.31

- added a tokens visibilty feature to the developers to view their splitting of the tokens and as well as the text chunks split using the pattern.
- added more [samples](https://github.com/Hk669/bpetokenizer/tree/main/sample)

1.0.21

build working correctly, ensuring the upload to pypi working.

1.0.10

testing the pypi package auto upload

Page 1 of 2

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.