Ai21

Chore

* chore(release): v0.3.0 [skip ci] ([`0c4ada0`](https://github.com/AI21Labs/ai21-tokenizer/commit/0c4ada02851933763da28bdeb726380930b7fb46))

Documentation

* docs: Release md update before publish (36)

* fix: Added support for both str and path

* fix: rename package

* fix: updated pre commits and added new one

* docs: Updated docs

* ci: down grade

* docs: Added another example ([`18ccbeb`](https://github.com/AI21Labs/ai21-tokenizer/commit/18ccbeb89745491fa2d5ac92b8e017ee2af4ca88))

* docs: CONTRIBUTING.md (35)

* docs: CONTRIBUTING.md

* ci: end_of_line fix

* docs: inv test ([`e282440`](https://github.com/AI21Labs/ai21-tokenizer/commit/e2824402e375aa4e4714fb1afa3c212abd276c2f))

Feature

* feat: Added char for testing (37) ([`40d3feb`](https://github.com/AI21Labs/ai21-tokenizer/commit/40d3febf9a4df9f54b91e50981326b71ce362c89))

Fix

* fix: string example (38) ([`833038c`](https://github.com/AI21Labs/ai21-tokenizer/commit/833038c0ff348cfa240764346e2faffea09ed6ac))

Chore

* chore(release): v0.2.0 [skip ci] ([`8988faa`](https://github.com/AI21Labs/ai21-tokenizer/commit/8988faa1419069916aedcb317e7c813a57a7d03d))

* chore(deps-dev): bump black from 22.12.0 to 23.3.0 (32)

Bumps [black](https://github.com/psf/black) from 22.12.0 to 23.3.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/22.12.0...23.3.0)

---
updated-dependencies:
- dependency-name: black
dependency-type: direct:development
update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com>
Co-authored-by: asafgardin <147075902+asafgardinusers.noreply.github.com> ([`bb4986e`](https://github.com/AI21Labs/ai21-tokenizer/commit/bb4986e880f24eebc7be7eee050d85c4d56a1aa9))

Feature

* feat: Tokenizer factory (31)

* feat: Added tokenizer abc and factory

* fix: api to receive default and none

* fix: example

* fix: factory and tests

* fix: rename base

* fix: rename base class

* fix: rename package

* fix: example

* fix: readme and tasks

* docs: factory class

* docs: renames

* fix: directory hierarchy in tests

* fix: rename package

* chore(release): v0.1.2 [skip ci]

* fix: rename package

* ci: example

* fix: assert in example

* fix: src_path

---------

Co-authored-by: github-actions <github-actionsgithub.com> ([`e55cd1d`](https://github.com/AI21Labs/ai21-tokenizer/commit/e55cd1dac5ad501a0a48c369a28d061f37950f5f))

Fix

* fix: token name (34) ([`2b229b2`](https://github.com/AI21Labs/ai21-tokenizer/commit/2b229b28ace8ac72dcc9bc727d187350020a2e12))

Chore

* chore(release): v0.1.2 [skip ci] ([`5b1dc14`](https://github.com/AI21Labs/ai21-tokenizer/commit/5b1dc140213615bbe8a6a65caea8eea2ec7d3cbc))

* chore(deps-dev): bump safety from 2.3.4 to 2.3.5 (28)

Bumps [safety](https://github.com/pyupio/safety) from 2.3.4 to 2.3.5.
- [Release notes](https://github.com/pyupio/safety/releases)
- [Changelog](https://github.com/pyupio/safety/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pyupio/safety/compare/2.3.4...2.3.5)

---
updated-dependencies:
- dependency-name: safety
dependency-type: direct:development
update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com> ([`28118ac`](https://github.com/AI21Labs/ai21-tokenizer/commit/28118acd7477a128eecf36119d60d50fa1fbb169))

* chore(deps-dev): bump pytest-mock from 3.10.0 to 3.11.1 (24)

Bumps [pytest-mock](https://github.com/pytest-dev/pytest-mock) from 3.10.0 to 3.11.1.
- [Release notes](https://github.com/pytest-dev/pytest-mock/releases)
- [Changelog](https://github.com/pytest-dev/pytest-mock/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest-mock/compare/v3.10.0...v3.11.1)

---
updated-dependencies:
- dependency-name: pytest-mock
dependency-type: direct:development
update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com>
Co-authored-by: asafgardin <147075902+asafgardinusers.noreply.github.com> ([`0a36f21`](https://github.com/AI21Labs/ai21-tokenizer/commit/0a36f21e0c375b12f743e4c89c90b5391502d2b3))

Ci

* ci: Remove install from publish (29)

* ci: Removed install dependency

* ci: changlog changes ([`f8e8392`](https://github.com/AI21Labs/ai21-tokenizer/commit/f8e8392c717d93d1a103d1eec84b7580beaa1b14))

* ci: dependabot pr limit (27)

* ci: dependabot pr limit

* ci: dependabot pr limit ([`9b2c4f8`](https://github.com/AI21Labs/ai21-tokenizer/commit/9b2c4f885a5f2be3076f3c4563de33efb3f7af9a))

Fix

* fix: workflow dispatch for release action (33) ([`f81b4ab`](https://github.com/AI21Labs/ai21-tokenizer/commit/f81b4abbe4fe45163dec45dfce89677fe673e46b))

Chore

* chore(release): v0.1.1 [skip ci] ([`f5150f5`](https://github.com/AI21Labs/ai21-tokenizer/commit/f5150f5473bb95759cf389db2033789f69e3ac38))

Fix

* fix: used PAT (26)

* test: write to main

* fix: token

* test: debug

* test: debugging tokens

* test: uncomment

* test: write to main

* fix: token

* test: debug

* test: debugging tokens

* test: uncomment

* fix: Changed to main ([`2ff12c9`](https://github.com/AI21Labs/ai21-tokenizer/commit/2ff12c9b4d490eec0d089b18a6bdabda7e23e952))

Chore

* chore(release): v0.1.0 [skip ci] ([`48607f9`](https://github.com/AI21Labs/ai21-tokenizer/commit/48607f92d977ce4d725879d5ac1306c8efd3255c))

Ci

* ci: Create dependabot.yml (19)

* ci: Create dependabot.yml

* fix: commit-message prefix

* fix: Added more config to dependabot action ([`23faaa8`](https://github.com/AI21Labs/ai21-tokenizer/commit/23faaa81d37a673d56542b367470aab6013d2b39))

Feature

* feat: Pypi publish (18)

* feast: Added setup.py

* feast: Added publish.yaml ([`77ee751`](https://github.com/AI21Labs/ai21-tokenizer/commit/77ee751b6d7610c4b4955ad488579a88f47d4f8b))

* feat: test PAT (9) ([`a10b6a4`](https://github.com/AI21Labs/ai21-tokenizer/commit/a10b6a494c20e8929dc3ba355527f8290f27afbf))

* feat: Added semantic prs actions (8) ([`afab5ff`](https://github.com/AI21Labs/ai21-tokenizer/commit/afab5ff05154da698120d67f7a018907c6ed1ecc))

Fix

* fix: Added permissions (25)

* fix: Added permissions

* fix: permissions location

* fix: verbose

* fix: Removed bad input &34;root_options&34; ([`d684575`](https://github.com/AI21Labs/ai21-tokenizer/commit/d684575675e85e70af96639cc3c33f125770e7da))

* fix: Change token (17)

* fix: token key

* fix: token github

* fix: token github cls

* fix: token github cls ([`ec4f35b`](https://github.com/AI21Labs/ai21-tokenizer/commit/ec4f35bf477a602e648f1d6d212c2a388b9c97e0))

* fix: Change token (16)

* fix: token key

* fix: token github ([`d876a43`](https://github.com/AI21Labs/ai21-tokenizer/commit/d876a43b4bde91f202f93fb22411df088af49293))

* fix: token key (15) ([`0b76344`](https://github.com/AI21Labs/ai21-tokenizer/commit/0b76344bfcad9c03cc50d19ad2ecdf5d2127b3e2))

* fix: keys (14) ([`b064ea6`](https://github.com/AI21Labs/ai21-tokenizer/commit/b064ea657945e7e3201d9b7678e2551ad0c75f89))

* fix: Test token (11)

* feat: test PAT

* feat: test github token

* fix: PAT ([`94f64b6`](https://github.com/AI21Labs/ai21-tokenizer/commit/94f64b6bded3bba3e6f616cfd37315bacf68b3f4))

* fix: Test token (10)

* feat: test PAT

* feat: test github token ([`52484fe`](https://github.com/AI21Labs/ai21-tokenizer/commit/52484fe2a0951652ed4d34c17cf48ca47669ab11))

* fix: root_options verbose (6) ([`220ba5b`](https://github.com/AI21Labs/ai21-tokenizer/commit/220ba5b47e9f2c4b3db0c38291abe4a983b35c2e))

* fix: Release action test (5)

* fix: Added release step

* fix: branch name for testing

* fix: Removed comment

* chore(release): v0.0.1 [skip ci]

* fix: branch rename

* fix: Removed CHANGELOG.md

---------

Co-authored-by: github-actions <github-actionsgithub.com> ([`fae5423`](https://github.com/AI21Labs/ai21-tokenizer/commit/fae5423193754d4e60826e1a74dfb70e6e463c47))

Test

* test: Test token (13)

* feat: test PAT

* feat: test github token

* fix: PAT

* test: Added test step

* test: Added test step

* test: Added token to use ([`d445e36`](https://github.com/AI21Labs/ai21-tokenizer/commit/d445e3688412ca4142c9309b28de6400a1f54220))

* test: Test token (12)

* feat: test PAT

* feat: test github token

* fix: PAT

* test: Added test step

* test: Added test step ([`9619c3e`](https://github.com/AI21Labs/ai21-tokenizer/commit/9619c3e064e0014d2dbf776a7bb18cd7a914c817))

Unknown

* Add kwargs to functions (7)

* feat: added kwargs

* test: Added tests ([`efecff9`](https://github.com/AI21Labs/ai21-tokenizer/commit/efecff97f0291a0f155208ba09c83dfdcfd9247c))

* Release action (4)

* feat: Added release action

* fix: Removed unnecessary code

* fix: testing on branch

* fix: removed node install

* fix: Removed unnecessary step

* fix: base-branch

* fix: removed code

* 0.0.2

* feat: python-semantic-release test

* fix: branch name

* fix: branch name in .toml

* fix: change from branch to match

* fix: Added release_action part

* chore(release): v0.1.0 [skip ci]

* refactor: removed CHANGELOG.md

* fix: branch to main

* feat: Added version.py to version_variable

* feat: Upgraded python-semantic-release

* feat: Added python-semantic-release

* fix: Removed unnecessary file

* fix: Changed version

---------

Co-authored-by: github-action <41898282+github-actions[bot]users.noreply.github.com>
Co-authored-by: github-actions <github-actionsgithub.com> ([`694683a`](https://github.com/AI21Labs/ai21-tokenizer/commit/694683ac39330944368a1e6ca3370857804bb960))

* Add code (2)

* feat: Jurassic tokenizer

* fix: remove is_start

* fix: add types

* fix: add types

* chore: extracted utils

* fix: simplified tokenizer even more

* fix: simplified tokenizer even more

* feat: Added tests

* feat: exposed prop

---------

Co-authored-by: Asaf Gardin <asafgai21.com> ([`6b80a05`](https://github.com/AI21Labs/ai21-tokenizer/commit/6b80a05549267e44e59f8ae40a92e4de68df979c))

* First commit (1)

* feat: init project ([`f50565e`](https://github.com/AI21Labs/ai21-tokenizer/commit/f50565eeb8d7259dec565f1292592d66b479f962))

* Initial commit ([`556d3e6`](https://github.com/AI21Labs/ai21-tokenizer/commit/556d3e64af018cf7269d9face370099900aec8eb))


CHANGELOG