- New `--allow-special` option for allowing special tokens: `ttok '<|endoftext|>' --encode --allow-special` [13](https://github.com/simonw/ttok/issues/13)
0.2
- `--encode` now encodes text to integer tokens and displays them. This has been renamed from `--tokens`. - `--decode` against a sequence of integer tokens now turns those back into text. [7](https://github.com/simonw/ttok/issues/7) - `--tokens` for either of those options outputs the raw tokens in a format that helps show if they have a leading space or similar. [4](https://github.com/simonw/ttok/issues/4)
0.1
- Initial release. `echo input.txt | ttok` to count tokens, use `-t 100` to truncate to 100 tokens, or `--tokens` to see what those integer tokens would be. [1](https://github.com/simonw/ttok/issues/1)