Ai21-tokenizer

Chore

* chore(release): v0.6.0 [skip ci] ([`7b8348d`](https://github.com/AI21Labs/ai21-tokenizer/commit/7b8348d303eb54c4a75ca1c58be5c08c35ec3de8))

* chore: add test case for encode with is_start=False (74)

* chore: add test case for encode with is_start=False

* fix: split is_start=False to a different testcase ([`77c0a39`](https://github.com/AI21Labs/ai21-tokenizer/commit/77c0a39d1bcde81cc0166a512eb454dad6d3c569))

Feature

* feat: Add decode with offsets (73)

* feat: Add decode_with_offsets() to JurassicTokenizer

* refactor: remove kwargs from decode_with_offsets since it&39;s not used

* chore: Add unittest for decode and for offsets

* fix: test only decode_with_offsets

* fix: dummy for returned offsets in decode_with_offsets ([`a5a7bb4`](https://github.com/AI21Labs/ai21-tokenizer/commit/a5a7bb4b27fa4f74a0b1a1d6874599556a35c1c5))

* feat: Add the is_start parameter to JurassicTokenizer.encode() (72)

* feat: Add the is_start parameter to JurassicTokenizer.encode()

* refactor: take &39;is_start&39; from kwargs ([`296bda5`](https://github.com/AI21Labs/ai21-tokenizer/commit/296bda5578edd57ff58d6763b3ccd5b9ba709795))

Chore

* chore(release): v0.5.0 [skip ci] ([`96f384f`](https://github.com/AI21Labs/ai21-tokenizer/commit/96f384f2873ec363016c8def483f59ac535acba9))

Feature

* feat: Add more special tokens (71)

* fix: commitizen tag starts with &34;v&34;

* feat: add eos_id

* feat: Add newline_id

* fix: typo &34;_newline_piece&34; instead of &34;newline_piece&34;

* fix: newline_id already existed as &34;private&34;. Just make it &34;public&34;

* fix: forgot to rename everywhere ([`9a9e1a8`](https://github.com/AI21Labs/ai21-tokenizer/commit/9a9e1a8a21dbfd4d3f983ce8ec1f97335470e2c6))

Fix

* fix: commitizen tag starts with &34;v&34; (70) ([`cf495ad`](https://github.com/AI21Labs/ai21-tokenizer/commit/cf495ada8341131c0d022f02d2c9e86cee8723ba))

Chore

* chore(release): v0.4.0 [skip ci] ([`b761edc`](https://github.com/AI21Labs/ai21-tokenizer/commit/b761edc0947b132192d4f963dd8d8704df4ebac4))

Feature

* feat: add pad_id and bos_id to jurassic_tokenizer (69) ([`ffb2ce3`](https://github.com/AI21Labs/ai21-tokenizer/commit/ffb2ce38aa59eec03305c4fdc6bfedd99c7b5255))

Chore

* chore(release): v0.3.11 [skip ci] ([`5280149`](https://github.com/AI21Labs/ai21-tokenizer/commit/528014946c5d35bbd9a5c8fa8c6eed7754e1f2f1))

Fix

* fix: BaseTokenizer in init (68) ([`3cc71e7`](https://github.com/AI21Labs/ai21-tokenizer/commit/3cc71e747a0b5d5450dbe70618456d415171ae23))

Chore

* chore(release): v0.3.10 [skip ci] ([`1601535`](https://github.com/AI21Labs/ai21-tokenizer/commit/16015359022592a058a48893076fd4c34ab85b74))

* chore(deps-dev): bump safety from 2.3.4 to 2.3.5 (64)

Bumps [safety](https://github.com/pyupio/safety) from 2.3.4 to 2.3.5.
- [Release notes](https://github.com/pyupio/safety/releases)
- [Changelog](https://github.com/pyupio/safety/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pyupio/safety/compare/2.3.4...2.3.5)

---
updated-dependencies:
- dependency-name: safety
dependency-type: direct:development
update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com> ([`95696bb`](https://github.com/AI21Labs/ai21-tokenizer/commit/95696bb2eee8e3ddfff3d4e297ba8d8633c7c112))

* chore(deps-dev): bump ruff from 0.0.285 to 0.1.8 (63)

Bumps [ruff](https://github.com/astral-sh/ruff) from 0.0.285 to 0.1.8.
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](https://github.com/astral-sh/ruff/compare/v0.0.285...v0.1.8)

---
updated-dependencies:
- dependency-name: ruff
dependency-type: direct:development
update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com> ([`81123d3`](https://github.com/AI21Labs/ai21-tokenizer/commit/81123d3a7afc9794dc21657cc5bab9dc94a123b2))

* chore(deps-dev): bump black from 22.12.0 to 23.3.0 (61)

Bumps [black](https://github.com/psf/black) from 22.12.0 to 23.3.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/22.12.0...23.3.0)

---
updated-dependencies:
- dependency-name: black
dependency-type: direct:development
update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com> ([`7190d28`](https://github.com/AI21Labs/ai21-tokenizer/commit/7190d28ab2ee7d8a5bfcf8b1b90606940f4a25ad))

* chore(deps-dev): bump safety from 2.3.4 to 2.3.5 (60)

Bumps [safety](https://github.com/pyupio/safety) from 2.3.4 to 2.3.5.
- [Release notes](https://github.com/pyupio/safety/releases)
- [Changelog](https://github.com/pyupio/safety/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pyupio/safety/compare/2.3.4...2.3.5)

---
updated-dependencies:
- dependency-name: safety
dependency-type: direct:development
update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com> ([`2fa7bef`](https://github.com/AI21Labs/ai21-tokenizer/commit/2fa7bef0b26c891778180970f66932b42126c8a1))

Fix

* fix: JurassicTokenizer in init (67) ([`253ae07`](https://github.com/AI21Labs/ai21-tokenizer/commit/253ae073ba20ebef544cde803c9180975aa70a5d))

Refactor

* refactor: Added __all__ in __init__ (65)

* refactor: Added __all__ in __init__

* fix: tests

* refactor: added __version__ to __all__ ([`c0d9286`](https://github.com/AI21Labs/ai21-tokenizer/commit/c0d9286a2a8d8ae7d7cb418620d709be7b7d8193))

* refactor: sentencepiece version to support all patch versions (66) ([`845008c`](https://github.com/AI21Labs/ai21-tokenizer/commit/845008cf61f86b40516bbc1836e0b4cfd0559192))

Chore

* chore(release): v0.3.9 [skip ci] ([`84f17da`](https://github.com/AI21Labs/ai21-tokenizer/commit/84f17dafee88962ead47d19a1ea2eb3998de3c18))

* chore: add github badges (58) ([`821455c`](https://github.com/AI21Labs/ai21-tokenizer/commit/821455c6bf2e897a3bf3619d2e078e2daf3d4653))

* chore(deps-dev): bump urllib3 from 2.0.4 to 2.0.7 (57)

Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.0.4 to 2.0.7.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.0.4...2.0.7)

---
updated-dependencies:
- dependency-name: urllib3
dependency-type: indirect
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com> ([`93ef6d6`](https://github.com/AI21Labs/ai21-tokenizer/commit/93ef6d6a18ca7ce7fb65fdf4cbc09e7105bcd8c3))

Fix

* fix: Modify badges (59)

* docs: fixed url

* fix: inline

* fix: README.md ([`20b7090`](https://github.com/AI21Labs/ai21-tokenizer/commit/20b709012059719a5084b7abed745f8705f0e8a1))