Mltb2

What's Changed
* Option for OpenAI special token removal. by PhilipMay in https://github.com/telekom/mltb2/pull/164
* Add support for Azure authentication using Azure AD Tokens by alaeddine-13 in https://github.com/telekom/mltb2/pull/165
* Update Python Versions by PhilipMay in https://github.com/telekom/mltb2/pull/166

New Contributors
* alaeddine-13 made their first contribution in https://github.com/telekom/mltb2/pull/165

**Full Changelog**: https://github.com/telekom/mltb2/compare/1.0.0...1.0.1

What's Changed
* Option for OpenAI special token removal. by PhilipMay in https://github.com/telekom/mltb2/pull/164
* Add support for Azure authentication using Azure AD Tokens by alaeddine-13 in https://github.com/telekom/mltb2/pull/165

New Contributors
* alaeddine-13 made their first contribution in https://github.com/telekom/mltb2/pull/165

**Full Changelog**: https://github.com/telekom/mltb2/compare/1.0.0...1.0.1rc3

What's Changed
* Add FileBasedRestartableBatchDataProcessor. by PhilipMay in https://github.com/telekom/mltb2/pull/154
* Add option to pass OpenAI key with os env. by PhilipMay in https://github.com/telekom/mltb2/pull/156
* Change data format to pkl / joblib for FileBasedRestartableBatchDataProcessor. by PhilipMay in https://github.com/telekom/mltb2/pull/157
* Fix unknown file issue in FileBasedRestartableBatchDataProcessor. by PhilipMay in https://github.com/telekom/mltb2/pull/158
* Avoid duplicate results in FileBasedRestartableBatchDataProcessor. by PhilipMay in https://github.com/telekom/mltb2/pull/159
* Add openai async function. by PhilipMay in https://github.com/telekom/mltb2/pull/160
* Add len to FileBasedRestartableBatchDataProcessor by PhilipMay in https://github.com/telekom/mltb2/pull/161
* Handle orphaned lock files for FileBasedRestartableBatchDataProcessor. by PhilipMay in https://github.com/telekom/mltb2/pull/162
* Add ignore error in FileBasedRestartableBatchDataProcessor data loading. by PhilipMay in https://github.com/telekom/mltb2/pull/163


**Full Changelog**: https://github.com/telekom/mltb2/compare/0.12.3...1.0.0

What's Changed
* Add clean function `clean_all_invisible_chars_and_strip`. by PhilipMay in https://github.com/telekom/mltb2/pull/153


**Full Changelog**: https://github.com/telekom/mltb2/compare/0.12.2...0.12.3

What's Changed
* Add SoMaJo token extractor. by PhilipMay in https://github.com/telekom/mltb2/pull/152


**Full Changelog**: https://github.com/telekom/mltb2/compare/0.12.0...0.12.2

What's Changed
* Fix Somajo md link extraction. by PhilipMay in https://github.com/telekom/mltb2/pull/150


**Full Changelog**: https://github.com/telekom/mltb2/compare/0.11.1...0.12.0