Tensorflow-text

Major Features and Improvements

* New ByteSplitter which tokenizes strings into bytes.
* New tutorial: Fine tune BERT with Orbit [will be added to tensorflow.org/text soon].
* Fixed an issue where dynamic TF Lite tensors were not getting resized correctly.

Bug Fixes and Other Changes

* Fix typo error in subwords_tokenizer guide with text.WordpieceTokenizer
* Fixes prepare_tf_dep.sh for OSX.
* Add cross-links to tensorflow_models.nlp API reference.
* (Generated change) Update tf.Text versions and/or docs.
* Update shape inference of kernel template for fast wordpiece and activate the op test.
* Update configure.sh for Apple Silicon.
* Export `Trimmer` ABC to be usable as tf_text.Trimmer
* Fix TensorFlow checkpoint and trackable imports.
* Correct tutorial explanation: meaning of attention weights
* Modernize `fine_tune_bert`.
* Lint and update the Fine-tuning a BERT model tutorial
* Use pointer for pointer math instead of iterator. Fixes c++17 compilation for regex_split on windows.
* Add install_bazel.sh script to make it easy to install the correctly needed version of Bazel. (946)
* Make install_bazel.sh script executable.
* Prevent runtime errors from happening due to invalid regular expressions using regex_split & RegexSplitter.
* Centralize tensorflow-models docs into a top-level docs/ directory.
* Remove link to non-existant section on tf.org.
* Move fine_tune_bert guide.
* Updated the spelling mistakes in subwords_tokenizer.ipynb
* Fixes a bug caused by passing an empty tensor into SentencepieceTokenizer's detokenize method.
* Update build for Sentencepiece. Darts was not properly being depended on.
* Improve Sentencepiece build by adding missing dependency - str_format.
* Fix typos and lint Neural machine translation with attention tutorial
* Fix external link formatting, lint NMT with attention tutorial

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

gadagashwini, mnahinkhan, Steve R. Sun, synandi

Major Features and Improvements

* New ByteSplitter which tokenizes strings into bytes.
* New tutorial: Fine tune BERT with Orbit [will be added to tensorflow.org/text soon].
* Fixed an issue where dynamic TF Lite tensors were not getting resized correctly.

Bug Fixes and Other Changes

* Fix typo error in subwords_tokenizer guide with text.WordpieceTokenizer
* Fixes prepare_tf_dep.sh for OSX.
* Add cross-links to tensorflow_models.nlp API reference.
* (Generated change) Update tf.Text versions and/or docs.
* Update shape inference of kernel template for fast wordpiece and activate the op test.
* Update configure.sh for Apple Silicon.
* Export `Trimmer` ABC to be usable as tf_text.Trimmer
* Fix TensorFlow checkpoint and trackable imports.
* Correct tutorial explanation: meaning of attention weights
* Modernize `fine_tune_bert`.
* Lint and update the Fine-tuning a BERT model tutorial
* Use pointer for pointer math instead of iterator. Fixes c++17 compilation for regex_split on windows.
* Add install_bazel.sh script to make it easy to install the correctly needed version of Bazel. (946)
* Make install_bazel.sh script executable.
* Prevent runtime errors from happening due to invalid regular expressions using regex_split & RegexSplitter.
* Centralize tensorflow-models docs into a top-level docs/ directory.
* Remove link to non-existant section on tf.org.
* Move fine_tune_bert guide.
* Updated the spelling mistakes in subwords_tokenizer.ipynb
* Fixes a bug caused by passing an empty tensor into SentencepieceTokenizer's detokenize method.
* Update build for Sentencepiece. Darts was not properly being depended on.
* Improve Sentencepiece build by adding missing dependency - str_format.
* Fix typos and lint Neural machine translation with attention tutorial
* Fix external link formatting, lint NMT with attention tutorial

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

gadagashwini, mnahinkhan, Steve R. Sun, synandi

Major Features and Improvements

* Added FastSentencepieceTokenizer which is convertible to TF Lite. Please note the op name in the graph will change, so any models trained with this version will need to be retrained when the release candidate for 2.10 is released.

Important Notes

* This beta release is outside the normal release cycle and is meant to work with TF versions 2.8.x.
* Again, the op name for FSP will change in future releases.

Major Features and Improvements

* New FastBertNormalizer that improves speed for BERT normalization and is convertible to TF Lite.
* New FastBertTokenizer that combines FastBertNormalizer and FastWordpieceTokenizer.
* New ngrams kernel for handling STRING_JOIN reductions.

Bug Fixes and Other Changes

* NgramsStringJoin shape inference fixed to handle unranked tensors
* Upgrade pybind11 and reenable tests that were broken.
* Rename a couple files to match the naming of the other tflite kernels. Also adds some deps to tflite_ops that were missing and causing an error when testing `:all`.
* Add to TF Lite documentation that ngrams is a convertible op.
* Fix public access and missing ICU data to build_fast_bert_normalizer_model and enable the disabled tests.
* Update the doc for FastWordpieceTokenizer.
* Refine the doc for FastWordpieceTokenizer.
* Bug fix: make BertTokenizer work for RaggedTensors with row_splits_dtype=int32
* Fix typo error text.WordpieceTokenizer
* Added comma at missing places in emoticons for normalizer
* Refactor build and test scripts to use prepare_tf_dep.sh
* Fixes prepare_tf_dep.sh for OSX.
* Fixed bug in setup.py that was requiring the wrong version.
* Updated package with the correct versions of Python we release on.
* Update documentation on TF Lite convertible ops.
* Transition to use TF's version of bazel.
* Transition to use TF's bazel configuration.
* Add missing symbols for tokenization layers
* Fix typo in text_generation.ipynb
* Fix grammar typo
* Allow fast wordpiece tokenizer to take in external wordpiece model.
* Internal change
* Improvement to guide where mean call is redundant. See https://github.com/tensorflow/text/issues/810 for more info.
* Update broken link and fix typo in BERT-SNGP demo notebook
* Consolidate disparate test-related files into a single testing_infra folder.
* Pin tf-text version to guides & tutorials.
* Fix bug in constrained sequence op. Added a check on an edge case where num_steps = 0 should do nothing and prevent it from SIGSEV crashes.
* Remove outdated Keras tests due to them no longer making the testing utilities available.
* Update bert preprocessing by padding correct tensors
* Update tensorflow-text notebooks from 2.7 to 2.8
* Optimize FastWordPiece to only generate requested outputs.
* Add a note about byte-indexing vs character indexing.
* Add a MAX_TOKENS to the transformer tutorial.
* Only export tensorflow symbols from shared libs.
* (Generated change) Update tf.Text versions and/or docs.
* Do not run the prepare_tf_dep script for Apple M1 macs.
* Update text_classification_rnn.ipynb
* Fix the exported symbols for the linker test. By adding it to the share objects instead of the c++ code, it allows for the code to be compiled together in one large shared lib.
* Implement FastBertNormalizer based on codepoint-wise mappings.
* Add pybind for fast_bert_normalizer_model_builder.
* Remove unused comments related to Python 2 compatibility.
* update transformer.ipynb
* Update toolchain & temporarily disable tf lite tests.
* Define manylinux2014 for the new toolchain target, and have presubmits use it.
* Move tflite build deps to custom target.
* Add FastBertTokenizer.
* Update bazel version to 5.1.0
* Update TF Text to use new Ngrams kernel.
* Don't try to set dimension if shape is unknown for ngrams.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Aflah, Connor Brinton, devnev39, Janak Ramakrishnan, Martin, Nathan Luehr, Pierre Dulac, Rabin Adhikari, gadagashwini, mohantym, rtg0795

Major Features and Improvements

* New FastBertNormalizer that improves speed for BERT normalization and is convertible to TF Lite.
* New FastBertTokenizer that combines FastBertNormalizer and FastWordpieceTokenizer.
* New ngrams kernel for handling STRING_JOIN reductions.

Bug Fixes and Other Changes

* Fixed bug in setup.py that was requiring the wrong version.
* Updated package with the correct versions of Python we release on.
* Update documentation on TF Lite convertible ops.
* Transition to use TF's version of bazel.
* Transition to use TF's bazel configuration.
* Add missing symbols for tokenization layers
* Fix typo in text_generation.ipynb
* Fix grammar typo
* Allow fast wordpiece tokenizer to take in external wordpiece model.
* Internal change
* Improvement to guide where mean call is redundant. See https://github.com/tensorflow/text/issues/810 for more info.
* Update broken link and fix typo in BERT-SNGP demo notebook
* Consolidate disparate test-related files into a single testing_infra folder.
* Pin tf-text version to guides & tutorials.
* Fix bug in constrained sequence op. Added a check on an edge case where num_steps = 0 should do nothing and prevent it from SIGSEV crashes.
* Remove outdated Keras tests due to them no longer making the testing utilities available.
* Update bert preprocessing by padding correct tensors
* Update tensorflow-text notebooks from 2.7 to 2.8
* Optimize FastWordPiece to only generate requested outputs.
* Add a note about byte-indexing vs character indexing.
* Add a MAX_TOKENS to the transformer tutorial.
* Only export tensorflow symbols from shared libs.
* (Generated change) Update tf.Text versions and/or docs.
* Do not run the prepare_tf_dep script for Apple M1 macs.
* Update text_classification_rnn.ipynb
* Fix the exported symbols for the linker test. By adding it to the share objects instead of the c++ code, it allows for the code to be compiled together in one large shared lib.
* Implement FastBertNormalizer based on codepoint-wise mappings.
* Add pybind for fast_bert_normalizer_model_builder.
* Remove unused comments related to Python 2 compatibility.
* update transformer.ipynb
* Update toolchain & temporarily disable tf lite tests.
* Define manylinux2014 for the new toolchain target, and have presubmits use it.
* Move tflite build deps to custom target.
* Add FastBertTokenizer.
* Update bazel version to 5.1.0
* Update TF Text to use new Ngrams kernel.
* Don't try to set dimension if shape is unknown for ngrams.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Aflah, Connor Brinton, devnev39, Janak Ramakrishnan, Martin, Nathan Luehr, Pierre Dulac, Rabin Adhikari