Tensorflow-text

Major Features and Improvements

Breaking Changes

Bug Fixes and Other Changes

* Replace usage of the tsl::Status constructor with a tsl::{error, errors}::Code.
* Replace usage of the tsl::Status(tsl::error::Code, ...) constructor.
* Update version
* Pin tensorflow-datasets version to 4.8.3

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Raviteja Gorijala

Major Features and Improvements

* New PhraseTokenizer.
* New ByteSplitter.split_by_offsets which splits a string using byte offsets.
* New `concatenate_segments` op.

Bug Fixes and Other Changes

* Updated kernel code and Python API for BoiseTagsToOffsets op
* Fix the bug that we should not re-build the config in the create function.
* Register kernel and ops for phrase tokenizer.
* fix the issue of conversion.
* Fix typos in nmt_with_attention.ipynb
* MacOS TF library was renamed. Update build configuration.
* Update tokenization_layers_test.py
* (Generated change) Update tf.Text versions and/or docs.
* Update TF Text's TF Lite guide with ops that are convertible to TF Lite.
* Update transformer test size.
* Fix typos in uncertainty_quantification_with_sngp_bert.ipynb
* (Generated change) Update tf.Text versions and/or docs.
* Adds LastNItemSelector an ItemSelector that selects the last n items in the batch.
* Temporarily remove tests for EOS offset since this is being changed in SP.
* Update test files for new ICU version.
* New helper function in the Op Kernel Shim for writing out data to the output tensors.
* Adds configuration flags to enable switch to Fast Wordpiece Tokenizer implementation alternative for on device
* New kernels to enable TF Lite conversion for SentenceFragmenterV2 op.
* Fix possible heap overflow bug in sentence fragmenter op.
* Deprecate PY37 support for TF-Text
* Fix BUILD file by moving tf dep in the appropriate place for FBN to prevent conflicts when building on mobile.
* Clean up a couple dependencies in the kernel BUILD file.
* C++ API for new kernel for the RoundRobinTrimmer which fixes a bug and makes it available for conversion to TF Lite.
* New kernels for the RoundRobinTrimmer which fixes a bug and makes it available for conversion to TF Lite.
* Add two functions to implementations of the OpKernelShim for accessing the name & doc string. Accessing internals directly causes problems when trying to use techniques like Object composition as the op template. In particular, this change is needed for improvements to the polymorphic wrapper.
* Allow int32 or int64 as types for RoundRobinTrimmer ops' splits.
* Extend RoundRobinTrimmer kernels to allow any type as the value.
* Return empty results if get_offsets is false.
* Skip-uncompressing of bazel to try and locate error for mac ci tests.
* Fix scraping full commit from short commit sha
* Update tensorflow-text notebooks from 2.8 to 2.11
* Fix bazel version scrapping logic for .bazelversion in install_bazel.sh
* Fix conditional so it works better with Apple silicon. See issue 1077 for more details.
* Force osname check to always be in lower-case. See 1077

**Thanks to our Contributors**
This release contains contributions from many people at Google, as well as:
synandi, tilakrayal

Bug Fixes and Other Changes

* BOISE TF op:
* Add kernel code and Python API for `BoiseTagsTpOffsets` op
* Other:
* Internal change
* Add model builder for phrase tokenizer.
* Fix the bug that we should not re-build the config in the create function.
* Register kernel and ops for phrase tokenizer.
* fix the issue of conversion.
* Fix typos in nmt_with_attention.ipynb
* Fix broken link in transformer.ipynb
* MacOS TF library was renamed. Update build configuration.
* Update tokenization_layers_test.py
* (Generated change) Update tf.Text versions and/or docs.
* Update TF Text's TF Lite guide with ops that are convertible to TF Lite.
* Update transformer test size.
* Fix typos in uncertainty_quantification_with_sngp_bert.ipynb
* (Generated change) Update tf.Text versions and/or docs.
* Adds LastNItemSelector an ItemSelector that selects the last n items in the batch.
* Add split_by_offsets method to ByteSplitter.
* Add split_by_offsets method to ByteSplitter.
* Temporarily remove tests for EOS offset since this is being changed in SP.
* Update test files for new ICU version.
* New helper function in the Op Kernel Shim for writing out data to the output tensors.
* Add split_by_offsets method to ByteSplitter.
* Adds configuration flags to enable switch to Fast Wordpiece Tokenizer implementation alternative for on device
* New kernels to enable TF Lite conversion for SentenceFragmenterV2 op.
* Fix possible heap overflow bug in sentence fragmenter op.
* Deprecate PY37 support for TF-Text
* Fix BUILD file by moving tf dep in the appropriate place for FBN to prevent conflicts when building on mobile.
* Clean up a couple dependencies in the kernel BUILD file.
* C++ API for new kernel for the RoundRobinTrimmer which fixes a bug and makes it available for conversion to TF Lite.
* New kernels for the RoundRobinTrimmer which fixes a bug and makes it available for conversion to TF Lite.
* Add two functions to implementations of the OpKernelShim for accessing the name & doc string. Accessing internals directly causes problems when trying to use techniques like Object composition as the op template. In particular, this change is needed for improvements to the polymorphic wrapper.
* Allow int32 or int64 as types for RoundRobinTrimmer ops' splits.
* Extend RoundRobinTrimmer kernels to allow any type as the value.
* internal
* license rules update
* Remove license changes for now since it has broken the builds.
* Return empty results if get_offsets is false.
* tf_text: Add a "concatenate_segments" function.
* Skip-uncompressing of bazel to try and locate error for mac ci tests.
* Fix scraping full commit from short commit sha
* Update tensorflow-text notebooks from 2.8 to 2.11
* Fix bazel version scrapping logic for .bazelversion in install_bazel.sh

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

synandi, tilakrayal

Major Features and Improvements

* Added op for converting to/from BOISE labels to offsets

Bug Fixes and Other Changes

* tensorflow:
* Moving logging.h and bitmap from tf/core to tf/tsl.
* BOISE TF op:
* Add main C++ functions for converting to/from BOISE labels to offsets
* Add main C++ functions for converting to/from BOISE labels to offsets
* Add kernel code and Python API for `OffsetsToBoiseTags` op
* Other:
* Add link to KPLs, fix typo in Neural machine translation with attention tutorial
* Update README.md
* Publish the tensorflow_models.nlp guide docs to tensorflow.org
* Add missing dependency to constrained sequence kernel.
* Add missing absl status dependency to sentence breaking utils.
* Another missing absl status dependency. this time for sentence fragmenter.
* Add absl status to sentence fragmenter v2.
* Update pybind11 to 2.10.0 to match tensorflow.
* Better error message for WordPiece when the vocabulary file has unicode issues.
* Update Transformer tutorial with Keras MultiHeadAttention
* transformers.ipynb: fix length filter and target slicing
* transformers.ipynb: cleanup wording, create a `PositionalEmbedding` layer.
* Replace `tensorflow::Status::OK()` with` tensorflow::OkStatus()`.
* Update README with note about various OS releases.
* Cast the step type.
* Reactivate TFLite ByteSplitter test.
* Modify tokenizer to process pt_examples to tokenizers.pt
* fix words alignment in documentation
* Update nmt_with_attention:
* transformers.ipynb: Factor out CrossAttention, GlobalSelfAttention, and CausalSelfAttention layers.
* Switch the transformer to train with `Model.fit`.
* Whitespace changes to force republish.
* Fix tutorial display, again.
* Update version

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

satojkovic

Bug Fixes and Other Changes

* tensorflow:
* Moving logging.h and bitmap from tf/core to tf/tsl.
* BOISE TF op:
* Add main C++ functions for converting to/from BOISE labels to offsets
* Add main C++ functions for converting to/from BOISE labels to offsets
* Add main C++ functions for converting to/from BOISE labels to offsets
* Add kernel code and Python API for `OffsetsToBoiseTags` op
* Other:
* Add link to KPLs, fix typo in Neural machine translation with attention tutorial
* Update README.md
* Publish the tensorflow_models.nlp guide docs to tensorflow.org
* Add missing dependency to constrained sequence kernel.
* Add missing absl status dependency to sentence breaking utils.
* Another missing absl status dependency. this time for sentence fragmenter.
* Add absl status to sentence fragmenter v2.
* Update pybind11 to 2.10.0 to match tensorflow.
* Better error message for WordPiece when the vocabulary file has unicode issues.
* Update Transformer tutorial with Keras MultiHeadAttention
* transformers.ipynb: fix length filter and target slicing
* transformers.ipynb: cleanup wording, create a `PositionalEmbedding` layer.
* Replace `tensorflow::Status::OK()` with` tensorflow::OkStatus()`.
* Update README with note about various OS releases.
* Cast the step type.
* Reactivate TFLite ByteSplitter test.
* Modify tokenizer to process pt_examples to tokenizers.pt
* fix words alignment in documentation
* Update nmt_with_attention:
* transformers.ipynb: Factor out CrossAttention, GlobalSelfAttention, and CausalSelfAttention layers.
* Switch the transformer to train with `Model.fit`.
* Whitespace changes to force republish.
* Add a phrase based tokenzier
* Fix tutorial display, again.
* Update version

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

satojkovic

Major Features and Improvements

* New ByteSplitter which tokenizes strings into bytes.
* New tutorial: Fine tune BERT with Orbit [will be added to tensorflow.org/text soon].
* Fixed an issue where dynamic TF Lite tensors were not getting resized correctly.

Bug Fixes and Other Changes

* Fix typo error in subwords_tokenizer guide with text.WordpieceTokenizer
* Fixes prepare_tf_dep.sh for OSX.
* Add cross-links to tensorflow_models.nlp API reference.
* (Generated change) Update tf.Text versions and/or docs.
* Update shape inference of kernel template for fast wordpiece and activate the op test.
* Update configure.sh for Apple Silicon.
* Export `Trimmer` ABC to be usable as tf_text.Trimmer
* Fix TensorFlow checkpoint and trackable imports.
* Correct tutorial explanation: meaning of attention weights
* Modernize `fine_tune_bert`.
* Lint and update the Fine-tuning a BERT model tutorial
* Use pointer for pointer math instead of iterator. Fixes c++17 compilation for regex_split on windows.
* Add install_bazel.sh script to make it easy to install the correctly needed version of Bazel. (946)
* Make install_bazel.sh script executable.
* Prevent runtime errors from happening due to invalid regular expressions using regex_split & RegexSplitter.
* Centralize tensorflow-models docs into a top-level docs/ directory.
* Remove link to non-existant section on tf.org.
* Move fine_tune_bert guide.
* Updated the spelling mistakes in subwords_tokenizer.ipynb
* Fixes a bug caused by passing an empty tensor into SentencepieceTokenizer's detokenize method.
* Update build for Sentencepiece. Darts was not properly being depended on.
* Improve Sentencepiece build by adding missing dependency - str_format.
* Fix typos and lint Neural machine translation with attention tutorial
* Fix external link formatting, lint NMT with attention tutorial

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

gadagashwini, mnahinkhan, Steve R. Sun, synandi