Tensorflow-text

Major Features and Improvements

* Added UnicodeCharacterTokenizer

Bug Fixes and Other Changes

* oss-segmenter:
* fix end-offset error in split_merge_tokenizer_kernel.
* TensorFlow text python ops wordshape:
* More comprehensive emoji handling
* Other:
* Unref lookup_table in wordpiece_kernel fixing a possible memory leak.
* Add missing LICENSE file for third_party/tensorflow_text/core/kernels.
* add normalize kernals test
* Add some metric logs to tokenizers.
* Fix documentation formatting for SplitMergeTokenizer
* Bug fix: make sure tokenize() method does not ignore itself.
* Improve logging efficiency.
* Update tf.text's regression test model for model server. Without the asserts, errors are erroneously swallowed by tensorflow. I also added tf.unicode_script test just to ensure that ICU is working correctly from within model server.
* Add the ability to define a user-defined destination directory to make testing easier.
* Fix typo in documentation of BertTokenizer
* Clarify docstring of UnicodeScriptTokenizer about splitting on space
* Add executable flag to the run_build.sh script.
* Clarify docstring of WordpieceTokenizer on unknown_token:
* Update protobuf library and point HEAD to build on tf 2.3.0-rc0

Thanks to our Contributors

Major Features and Improvements

Breaking Changes

Bug Fixes and Other Changes

* Update version

Thanks to our Contributors

Bug fixes

* Force MacOS builds to build for OSX 10.9 so they can be installed to a wider range of MacOS versions.

Major Features and Improvements

* Add op for solving max-spanning-tree (MST) problems. The code here is intended for NLP applications, but attempts to remain agnostic to particular NLP tasks (such as dependency parsing).
* Add max_spanning_tree_gradient.
* Add support for 'preserve_unused_tokens' options in BertTokenizer.

Bug Fixes and Other Changes

* Documentation updates.
* Reorganize the BUILD file for keras layers.
* Update model server testing. The test script now generates a model that integrates into tf serving's testing infra.
* Remove unneeded heavy dependencies in regex_split library.
* Turn TF text's ConstrainedSequence implementations into standalone callable functions.
* Fix bug in ViterbiAnalysis computation triggered when not using transition_weights.
* Removing testing_utils run_tf_function which is enabled by default now.
* Update patch params to work with Bazel >=1.0.0
* Remove circular dependencies by removing submodule imports from ragged package.
* Prevent lack of ragged_ops.py being released in TF from breaking tf.Text

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Hyunwoo Cho