Ai21-tokenizer

Feature

* feat: add jamba 1.5 tokenizers (mini and large) (96)

* feat: add jamba 1.5 tokenizers (mini and large)

* test: fix value error on conftest.py

* refactor: rename file name

* chore: fix path

* fix: enter ([`e052b22`](https://github.com/AI21Labs/ai21-tokenizer/commit/e052b22e69f03bb03992129ea34b91c98bac0fbb))

Chore

* chore(release): v0.11.3 [skip ci] ([`5be5b59`](https://github.com/AI21Labs/ai21-tokenizer/commit/5be5b59820cefe61ab4c6b4f273f9296fabb2983))

* chore(deps-dev): bump authlib from 1.2.1 to 1.3.1 (91)

Bumps [authlib](https://github.com/lepture/authlib) from 1.2.1 to 1.3.1.
- [Release notes](https://github.com/lepture/authlib/releases)
- [Changelog](https://github.com/lepture/authlib/blob/master/docs/changelog.rst)
- [Commits](https://github.com/lepture/authlib/compare/v1.2.1...v1.3.1)

---
updated-dependencies:
- dependency-name: authlib
dependency-type: indirect
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com> ([`fe24518`](https://github.com/AI21Labs/ai21-tokenizer/commit/fe24518a0c4cb15ac48f8d246fe22887ffcc3216))

* chore(deps-dev): bump black from 23.3.0 to 24.3.0 (90)

Bumps [black](https://github.com/psf/black) from 23.3.0 to 24.3.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/23.3.0...24.3.0)

---
updated-dependencies:
- dependency-name: black
dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com> ([`5257f3d`](https://github.com/AI21Labs/ai21-tokenizer/commit/5257f3d5c712380d857c7a575ffe3ac9966b4427))

* chore(deps-dev): bump python-semantic-release from 8.3.0 to 8.7.0 (80)

Bumps [python-semantic-release](https://github.com/python-semantic-release/python-semantic-release) from 8.3.0 to 8.7.0.
- [Release notes](https://github.com/python-semantic-release/python-semantic-release/releases)
- [Changelog](https://github.com/python-semantic-release/python-semantic-release/blob/master/CHANGELOG.md)
- [Commits](https://github.com/python-semantic-release/python-semantic-release/compare/v8.3.0...v8.7.0)

---
updated-dependencies:
- dependency-name: python-semantic-release
dependency-type: direct:development
update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com>
Co-authored-by: Asaf Gardin <147075902+asafgardinusers.noreply.github.com>
Co-authored-by: Asaf Joseph Gardin <39553475+Josephasafgusers.noreply.github.com> ([`376894f`](https://github.com/AI21Labs/ai21-tokenizer/commit/376894f7e548e96445234ccf1bde3df331b928cf))

Ci

* ci: allow running tests and semantic checks on external prs (93) ([`c08604d`](https://github.com/AI21Labs/ai21-tokenizer/commit/c08604d84deffd911718f568f11d4a4aa37395b3))

Fix

* fix(deps): bump requests from 2.31.0 to 2.32.2 (92)

Bumps [requests](https://github.com/psf/requests) from 2.31.0 to 2.32.2.
- [Release notes](https://github.com/psf/requests/releases)
- [Changelog](https://github.com/psf/requests/blob/main/HISTORY.md)
- [Commits](https://github.com/psf/requests/compare/v2.31.0...v2.32.2)

---
updated-dependencies:
- dependency-name: requests
dependency-type: indirect
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com>
Co-authored-by: Asaf Joseph Gardin <39553475+Josephasafgusers.noreply.github.com> ([`3216898`](https://github.com/AI21Labs/ai21-tokenizer/commit/321689884f7342f650293eb512aa6f728322b33f))

Chore

* chore(release): v0.11.2 [skip ci] ([`57e712e`](https://github.com/AI21Labs/ai21-tokenizer/commit/57e712efa7baba2d3ca3742b15794127f08455e9))

Fix

* fix: add missing space tokens init on jursssic (89)

* fix: missing space tokens init on jursssic, fix import in example

* test: add test for space_tokens init ([`4ec43a0`](https://github.com/AI21Labs/ai21-tokenizer/commit/4ec43a04e875d0fcb4dc557607c17a6ff8734176))

Chore

* chore(release): v0.11.1 [skip ci] ([`07bf0cd`](https://github.com/AI21Labs/ai21-tokenizer/commit/07bf0cd237d566a43fac94513ce30d8d20668fd7))

Fix

* fix: Added backwards compatibility for jamba tokenizer (88)

* fix: Added backwards compatibility for jamba tokenizer

* fix: Added to async

---------

Co-authored-by: asafg <asafgai21.com> ([`9ad3475`](https://github.com/AI21Labs/ai21-tokenizer/commit/9ad34759295dc8ccde30fe273c508b617ea81175))

Chore

* chore(release): v0.11.0 [skip ci] ([`0df321c`](https://github.com/AI21Labs/ai21-tokenizer/commit/0df321c8491404c595f9a30cc413ccebdcf24551))

Feature

* feat: Async tokenizer (86)

* feat: support async, wip

* feat: fix and add tests, examples, update readme

* fix: poetry lock

* fix: anyio -> aiofiles

* fix: try 3.8

* fix: remove 3.7 from tests

* fix: poetry lock

* fix: add 3.7 back

* fix: poetry lock

* fix: poetry.lock

* ci: pipenv

* fix: pipenv

* fix: pipenv

* fix: pyproject

* fix: lock

* fix: version

* fix: Removed aiofiles

* ci: update python version,

* fix: switch from aiofiles to anyio, remove redundant comments

* chore: poetry lock

* fix: disable initializing async classes directly, cr comments

* test: fix import

* ci: add asyncio-mode to test workflow

* fix: to_thread -> run_in_executor

* ci: add asyncio

* fix: cr comments

* fix: cr comments

---------

Co-authored-by: asafg <asafgai21.com> ([`3006cda`](https://github.com/AI21Labs/ai21-tokenizer/commit/3006cda2305a5ad31d705d3f6e3fe5605820e2f5))

Chore

* chore(release): v0.10.0 [skip ci] ([`1178ba7`](https://github.com/AI21Labs/ai21-tokenizer/commit/1178ba7fd1d980681cf143292848dacd51cdc2d5))

Feature

* feat: remove python 3.7 support (87) ([`58482ab`](https://github.com/AI21Labs/ai21-tokenizer/commit/58482abbda7767055345274b38c58ba9fd33b42f))