Ai21

Chore

* chore(release): v0.8.1 [skip ci] ([`fcacbf8`](https://github.com/AI21Labs/ai21-tokenizer/commit/fcacbf89a590e47d6ac3b8d385c9a6628a3ef4b2))

Fix

* fix: re-ordered parameters in ctor to avoid a breaking change (79) ([`6c1b608`](https://github.com/AI21Labs/ai21-tokenizer/commit/6c1b6088c0914ffc77b53613047606c398e0557c))

Chore

* chore(release): v0.8.0 [skip ci] ([`c8b54df`](https://github.com/AI21Labs/ai21-tokenizer/commit/c8b54dff67c13587943f03198ec5a4e1dca7be88))

* chore(deps-dev): bump pytest from 7.2.1 to 7.4.4 (75)

Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.2.1 to 7.4.4.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/7.2.1...7.4.4)

---
updated-dependencies:
- dependency-name: pytest
dependency-type: direct:development
update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com>
Co-authored-by: asafgardin <147075902+asafgardinusers.noreply.github.com> ([`081dda3`](https://github.com/AI21Labs/ai21-tokenizer/commit/081dda305ebc33af78ad433d511bdef3d63e1307))

Feature

* feat: Add start_of_line to decode (77)

* feat: Add start_of_line param to decode

* test: added unittest with start_of_line=True and False ([`182a8d1`](https://github.com/AI21Labs/ai21-tokenizer/commit/182a8d10020862c233f7f67cddb965eee2398b98))

Chore

* chore(release): v0.7.0 [skip ci] ([`26f34b2`](https://github.com/AI21Labs/ai21-tokenizer/commit/26f34b290cdc6b5166872bd6b8af5ca53d736936))

Feature

* feat: Init tokenizer from filehandle (76)

* feat: allow creating JurassicTokenizer from model file handle

* fix: Add default for model_path and model_file_handle

* feat: Add JurassicTokenizer.from_file_path classmethod

* fix: remove model_path=None in JurassicTokenizer.from_file_handle

* fix: rename _assert_exactly_one to _validate_init and make it not static

* refactor: semantics

* test: Added tests

---------

Co-authored-by: Asaf Gardin <asafgai21.com> ([`dcb73a7`](https://github.com/AI21Labs/ai21-tokenizer/commit/dcb73a72348e576b06cd4a066e06141ceae37a44))

Chore

* chore(release): v0.6.0 [skip ci] ([`7b8348d`](https://github.com/AI21Labs/ai21-tokenizer/commit/7b8348d303eb54c4a75ca1c58be5c08c35ec3de8))

* chore: add test case for encode with is_start=False (74)

* chore: add test case for encode with is_start=False

* fix: split is_start=False to a different testcase ([`77c0a39`](https://github.com/AI21Labs/ai21-tokenizer/commit/77c0a39d1bcde81cc0166a512eb454dad6d3c569))

Feature

* feat: Add decode with offsets (73)

* feat: Add decode_with_offsets() to JurassicTokenizer

* refactor: remove kwargs from decode_with_offsets since it&39;s not used

* chore: Add unittest for decode and for offsets

* fix: test only decode_with_offsets

* fix: dummy for returned offsets in decode_with_offsets ([`a5a7bb4`](https://github.com/AI21Labs/ai21-tokenizer/commit/a5a7bb4b27fa4f74a0b1a1d6874599556a35c1c5))

* feat: Add the is_start parameter to JurassicTokenizer.encode() (72)

* feat: Add the is_start parameter to JurassicTokenizer.encode()

* refactor: take &39;is_start&39; from kwargs ([`296bda5`](https://github.com/AI21Labs/ai21-tokenizer/commit/296bda5578edd57ff58d6763b3ccd5b9ba709795))

Chore

* chore(release): v0.5.0 [skip ci] ([`96f384f`](https://github.com/AI21Labs/ai21-tokenizer/commit/96f384f2873ec363016c8def483f59ac535acba9))

Feature

* feat: Add more special tokens (71)

* fix: commitizen tag starts with &34;v&34;

* feat: add eos_id

* feat: Add newline_id

* fix: typo &34;_newline_piece&34; instead of &34;newline_piece&34;

* fix: newline_id already existed as &34;private&34;. Just make it &34;public&34;

* fix: forgot to rename everywhere ([`9a9e1a8`](https://github.com/AI21Labs/ai21-tokenizer/commit/9a9e1a8a21dbfd4d3f983ce8ec1f97335470e2c6))

Fix

* fix: commitizen tag starts with &34;v&34; (70) ([`cf495ad`](https://github.com/AI21Labs/ai21-tokenizer/commit/cf495ada8341131c0d022f02d2c9e86cee8723ba))

Chore

* chore(release): v0.4.0 [skip ci] ([`b761edc`](https://github.com/AI21Labs/ai21-tokenizer/commit/b761edc0947b132192d4f963dd8d8704df4ebac4))

Feature

* feat: add pad_id and bos_id to jurassic_tokenizer (69) ([`ffb2ce3`](https://github.com/AI21Labs/ai21-tokenizer/commit/ffb2ce38aa59eec03305c4fdc6bfedd99c7b5255))