Ai21-tokenizer

Chore

* chore(release): v0.3.2 [skip ci] ([`f3b2e73`](https://github.com/AI21Labs/ai21-tokenizer/commit/f3b2e737c0dbf8c29524e83ffdebe95d3a12c1de))

Ci

* ci: Added newline sequence (41) ([`63a1898`](https://github.com/AI21Labs/ai21-tokenizer/commit/63a1898f49cd6d60bf2fbf98ea62915c55b91cb6))

Fix

* fix: Version path (42)

* fix: version

* fix: version variables

* fix: name ([`ee4d744`](https://github.com/AI21Labs/ai21-tokenizer/commit/ee4d744bc8a5565f91e888b1ab81d9fd7011c0a2))

Chore

* chore(release): v0.3.1 [skip ci] ([`2eebeb8`](https://github.com/AI21Labs/ai21-tokenizer/commit/2eebeb8418bd7a915eb2a82142dc73962b4a5fbb))

Fix

* fix: test 2 (40) ([`2028613`](https://github.com/AI21Labs/ai21-tokenizer/commit/20286135bbdc5f3193bdf378ab000145d95c3c2c))

* fix: Test bump 1 (39)

* fix: crlf forbid

* fix: test 1 ([`26da29e`](https://github.com/AI21Labs/ai21-tokenizer/commit/26da29e921dd8dd546a127c76214f20a0718f17d))

Chore

* chore(release): v0.3.0 [skip ci] ([`0c4ada0`](https://github.com/AI21Labs/ai21-tokenizer/commit/0c4ada02851933763da28bdeb726380930b7fb46))

Documentation

* docs: Release md update before publish (36)

* fix: Added support for both str and path

* fix: rename package

* fix: updated pre commits and added new one

* docs: Updated docs

* ci: down grade

* docs: Added another example ([`18ccbeb`](https://github.com/AI21Labs/ai21-tokenizer/commit/18ccbeb89745491fa2d5ac92b8e017ee2af4ca88))

* docs: CONTRIBUTING.md (35)

* docs: CONTRIBUTING.md

* ci: end_of_line fix

* docs: inv test ([`e282440`](https://github.com/AI21Labs/ai21-tokenizer/commit/e2824402e375aa4e4714fb1afa3c212abd276c2f))

Feature

* feat: Added char for testing (37) ([`40d3feb`](https://github.com/AI21Labs/ai21-tokenizer/commit/40d3febf9a4df9f54b91e50981326b71ce362c89))

Fix

* fix: string example (38) ([`833038c`](https://github.com/AI21Labs/ai21-tokenizer/commit/833038c0ff348cfa240764346e2faffea09ed6ac))

Chore

* chore(release): v0.2.0 [skip ci] ([`8988faa`](https://github.com/AI21Labs/ai21-tokenizer/commit/8988faa1419069916aedcb317e7c813a57a7d03d))

* chore(deps-dev): bump black from 22.12.0 to 23.3.0 (32)

Bumps [black](https://github.com/psf/black) from 22.12.0 to 23.3.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/22.12.0...23.3.0)

---
updated-dependencies:
- dependency-name: black
dependency-type: direct:development
update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com>
Co-authored-by: asafgardin <147075902+asafgardinusers.noreply.github.com> ([`bb4986e`](https://github.com/AI21Labs/ai21-tokenizer/commit/bb4986e880f24eebc7be7eee050d85c4d56a1aa9))

Feature

* feat: Tokenizer factory (31)

* feat: Added tokenizer abc and factory

* fix: api to receive default and none

* fix: example

* fix: factory and tests

* fix: rename base

* fix: rename base class

* fix: rename package

* fix: example

* fix: readme and tasks

* docs: factory class

* docs: renames

* fix: directory hierarchy in tests

* fix: rename package

* chore(release): v0.1.2 [skip ci]

* fix: rename package

* ci: example

* fix: assert in example

* fix: src_path

---------

Co-authored-by: github-actions <github-actionsgithub.com> ([`e55cd1d`](https://github.com/AI21Labs/ai21-tokenizer/commit/e55cd1dac5ad501a0a48c369a28d061f37950f5f))

Fix

* fix: token name (34) ([`2b229b2`](https://github.com/AI21Labs/ai21-tokenizer/commit/2b229b28ace8ac72dcc9bc727d187350020a2e12))

Chore

* chore(release): v0.1.2 [skip ci] ([`5b1dc14`](https://github.com/AI21Labs/ai21-tokenizer/commit/5b1dc140213615bbe8a6a65caea8eea2ec7d3cbc))

* chore(deps-dev): bump safety from 2.3.4 to 2.3.5 (28)

Bumps [safety](https://github.com/pyupio/safety) from 2.3.4 to 2.3.5.
- [Release notes](https://github.com/pyupio/safety/releases)
- [Changelog](https://github.com/pyupio/safety/blob/main/CHANGELOG.md)
- [Commits](https://github.com/pyupio/safety/compare/2.3.4...2.3.5)

---
updated-dependencies:
- dependency-name: safety
dependency-type: direct:development
update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com> ([`28118ac`](https://github.com/AI21Labs/ai21-tokenizer/commit/28118acd7477a128eecf36119d60d50fa1fbb169))

* chore(deps-dev): bump pytest-mock from 3.10.0 to 3.11.1 (24)

Bumps [pytest-mock](https://github.com/pytest-dev/pytest-mock) from 3.10.0 to 3.11.1.
- [Release notes](https://github.com/pytest-dev/pytest-mock/releases)
- [Changelog](https://github.com/pytest-dev/pytest-mock/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest-mock/compare/v3.10.0...v3.11.1)

---
updated-dependencies:
- dependency-name: pytest-mock
dependency-type: direct:development
update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <supportgithub.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]users.noreply.github.com>
Co-authored-by: asafgardin <147075902+asafgardinusers.noreply.github.com> ([`0a36f21`](https://github.com/AI21Labs/ai21-tokenizer/commit/0a36f21e0c375b12f743e4c89c90b5391502d2b3))

Ci

* ci: Remove install from publish (29)

* ci: Removed install dependency

* ci: changlog changes ([`f8e8392`](https://github.com/AI21Labs/ai21-tokenizer/commit/f8e8392c717d93d1a103d1eec84b7580beaa1b14))

* ci: dependabot pr limit (27)

* ci: dependabot pr limit

* ci: dependabot pr limit ([`9b2c4f8`](https://github.com/AI21Labs/ai21-tokenizer/commit/9b2c4f885a5f2be3076f3c4563de33efb3f7af9a))

Fix

* fix: workflow dispatch for release action (33) ([`f81b4ab`](https://github.com/AI21Labs/ai21-tokenizer/commit/f81b4abbe4fe45163dec45dfce89677fe673e46b))

Chore

* chore(release): v0.1.1 [skip ci] ([`f5150f5`](https://github.com/AI21Labs/ai21-tokenizer/commit/f5150f5473bb95759cf389db2033789f69e3ac38))

Fix

* fix: used PAT (26)

* test: write to main

* fix: token

* test: debug

* test: debugging tokens

* test: uncomment

* test: write to main

* fix: token

* test: debug

* test: debugging tokens

* test: uncomment

* fix: Changed to main ([`2ff12c9`](https://github.com/AI21Labs/ai21-tokenizer/commit/2ff12c9b4d490eec0d089b18a6bdabda7e23e952))