Bpetokenizer

Description

A Byte Pair Encoding (BPE) tokenizer, which algorithmically follows along the GPT tokenizer(tiktoken), allows you to train your own tokenizer. The tokenizer is capable of handling special tokens and uses a customizable regex pattern for tokenization(includes the gpt4 regex pattern). supports `save` and `load` tokenizers in the `json` and `file` format. The `bpetokenizer` also supports [pretrained](bpetokenizer/pretrained/) tokenizers.

Resources

• GitHub
• PyPi

• 1.2.1
• 1.2.0
• 1.0.32
• 1.0.31
• 1.0.4
• 1.0.3
• 1.0.2
• 1.0.1
• 1.0.0