Tensorflow-text

Bug Fixes and Other Changes

* Update word_embeddings.ipynb with Time Series as default dashboard.
* Python ops for new RoundRobinTrimmer kernels.
* Fix bug where rank 1 max sequence lengths were breaking round robin trimmer.
* Fix roundrobintrimmer not being linked in correctly.
* Move control_flow_ops.Assert into its own file, control_flow_assert.py, as a first step in removing circular dependencies with control_flow_ops.cond.
* Move control_flow_ops.while_loop into its own file, while_loop.py.
* Redirect references to stack and unstack to their new location in array_ops_stack.py.
* Prevent crashes with new trimmer op when max_sequence_length is set to a negative value.
* Fix run_build not getting tensorflow bazel version correctly. Also removed some "set -x" that were added for debugging.
* Add RetVec-style UTF-8 binarization
* Allow pad_model_inputs to work with Tensors as well.
* Move usages of python/util:util to the newly split up targets.
* (Generated change) Update tf.Text versions and/or docs.
* Fix typo in Transformer tutorial
* Pin protobuf version to prevent failure. See https://github.com/tensorflow/text/issues/1115 for more info.
* Avoid an expensive temporary std::string.
* Callout the differences compared to the reference paper.
* Remove decoding_api.ipynb from tf-text docs (this belongs to TF-Models)
* Altered build scripts to use python3 before python.
* Removes un-used tensorflow/core/platform:status dependency from round_robin_trimmer.
* Remove usages of tsl::Status::error_message.
* Use Github API to fetch full commit hash from short
* Avoid using jq in prepare_tf_dep.sh since it breaks macos builds
* Update version

Major Features and Improvements

Breaking Changes

Bug Fixes and Other Changes

* Replace usage of the tsl::Status constructor with a tsl::{error, errors}::Code.
* Replace usage of the tsl::Status(tsl::error::Code, ...) constructor.
* Update version
* Pin tensorflow-datasets version to 4.8.3

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Raviteja Gorijala

Major Features and Improvements

* New PhraseTokenizer.
* New ByteSplitter.split_by_offsets which splits a string using byte offsets.
* New `concatenate_segments` op.

Bug Fixes and Other Changes

* Updated kernel code and Python API for BoiseTagsToOffsets op
* Fix the bug that we should not re-build the config in the create function.
* Register kernel and ops for phrase tokenizer.
* fix the issue of conversion.
* Fix typos in nmt_with_attention.ipynb
* MacOS TF library was renamed. Update build configuration.
* Update tokenization_layers_test.py
* (Generated change) Update tf.Text versions and/or docs.
* Update TF Text's TF Lite guide with ops that are convertible to TF Lite.
* Update transformer test size.
* Fix typos in uncertainty_quantification_with_sngp_bert.ipynb
* (Generated change) Update tf.Text versions and/or docs.
* Adds LastNItemSelector an ItemSelector that selects the last n items in the batch.
* Temporarily remove tests for EOS offset since this is being changed in SP.
* Update test files for new ICU version.
* New helper function in the Op Kernel Shim for writing out data to the output tensors.
* Adds configuration flags to enable switch to Fast Wordpiece Tokenizer implementation alternative for on device
* New kernels to enable TF Lite conversion for SentenceFragmenterV2 op.
* Fix possible heap overflow bug in sentence fragmenter op.
* Deprecate PY37 support for TF-Text
* Fix BUILD file by moving tf dep in the appropriate place for FBN to prevent conflicts when building on mobile.
* Clean up a couple dependencies in the kernel BUILD file.
* C++ API for new kernel for the RoundRobinTrimmer which fixes a bug and makes it available for conversion to TF Lite.
* New kernels for the RoundRobinTrimmer which fixes a bug and makes it available for conversion to TF Lite.
* Add two functions to implementations of the OpKernelShim for accessing the name & doc string. Accessing internals directly causes problems when trying to use techniques like Object composition as the op template. In particular, this change is needed for improvements to the polymorphic wrapper.
* Allow int32 or int64 as types for RoundRobinTrimmer ops' splits.
* Extend RoundRobinTrimmer kernels to allow any type as the value.
* Return empty results if get_offsets is false.
* Skip-uncompressing of bazel to try and locate error for mac ci tests.
* Fix scraping full commit from short commit sha
* Update tensorflow-text notebooks from 2.8 to 2.11
* Fix bazel version scrapping logic for .bazelversion in install_bazel.sh
* Fix conditional so it works better with Apple silicon. See issue 1077 for more details.
* Force osname check to always be in lower-case. See 1077

**Thanks to our Contributors**
This release contains contributions from many people at Google, as well as:
synandi, tilakrayal

Bug Fixes and Other Changes

* BOISE TF op:
* Add kernel code and Python API for `BoiseTagsTpOffsets` op
* Other:
* Internal change
* Add model builder for phrase tokenizer.
* Fix the bug that we should not re-build the config in the create function.
* Register kernel and ops for phrase tokenizer.
* fix the issue of conversion.
* Fix typos in nmt_with_attention.ipynb
* Fix broken link in transformer.ipynb
* MacOS TF library was renamed. Update build configuration.
* Update tokenization_layers_test.py
* (Generated change) Update tf.Text versions and/or docs.
* Update TF Text's TF Lite guide with ops that are convertible to TF Lite.
* Update transformer test size.
* Fix typos in uncertainty_quantification_with_sngp_bert.ipynb
* (Generated change) Update tf.Text versions and/or docs.
* Adds LastNItemSelector an ItemSelector that selects the last n items in the batch.
* Add split_by_offsets method to ByteSplitter.
* Add split_by_offsets method to ByteSplitter.
* Temporarily remove tests for EOS offset since this is being changed in SP.
* Update test files for new ICU version.
* New helper function in the Op Kernel Shim for writing out data to the output tensors.
* Add split_by_offsets method to ByteSplitter.
* Adds configuration flags to enable switch to Fast Wordpiece Tokenizer implementation alternative for on device
* New kernels to enable TF Lite conversion for SentenceFragmenterV2 op.
* Fix possible heap overflow bug in sentence fragmenter op.
* Deprecate PY37 support for TF-Text
* Fix BUILD file by moving tf dep in the appropriate place for FBN to prevent conflicts when building on mobile.
* Clean up a couple dependencies in the kernel BUILD file.
* C++ API for new kernel for the RoundRobinTrimmer which fixes a bug and makes it available for conversion to TF Lite.
* New kernels for the RoundRobinTrimmer which fixes a bug and makes it available for conversion to TF Lite.
* Add two functions to implementations of the OpKernelShim for accessing the name & doc string. Accessing internals directly causes problems when trying to use techniques like Object composition as the op template. In particular, this change is needed for improvements to the polymorphic wrapper.
* Allow int32 or int64 as types for RoundRobinTrimmer ops' splits.
* Extend RoundRobinTrimmer kernels to allow any type as the value.
* internal
* license rules update
* Remove license changes for now since it has broken the builds.
* Return empty results if get_offsets is false.
* tf_text: Add a "concatenate_segments" function.
* Skip-uncompressing of bazel to try and locate error for mac ci tests.
* Fix scraping full commit from short commit sha
* Update tensorflow-text notebooks from 2.8 to 2.11
* Fix bazel version scrapping logic for .bazelversion in install_bazel.sh

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

synandi, tilakrayal

Major Features and Improvements

* Added op for converting to/from BOISE labels to offsets

Bug Fixes and Other Changes

* tensorflow:
* Moving logging.h and bitmap from tf/core to tf/tsl.
* BOISE TF op:
* Add main C++ functions for converting to/from BOISE labels to offsets
* Add main C++ functions for converting to/from BOISE labels to offsets
* Add kernel code and Python API for `OffsetsToBoiseTags` op
* Other:
* Add link to KPLs, fix typo in Neural machine translation with attention tutorial
* Update README.md
* Publish the tensorflow_models.nlp guide docs to tensorflow.org
* Add missing dependency to constrained sequence kernel.
* Add missing absl status dependency to sentence breaking utils.
* Another missing absl status dependency. this time for sentence fragmenter.
* Add absl status to sentence fragmenter v2.
* Update pybind11 to 2.10.0 to match tensorflow.
* Better error message for WordPiece when the vocabulary file has unicode issues.
* Update Transformer tutorial with Keras MultiHeadAttention
* transformers.ipynb: fix length filter and target slicing
* transformers.ipynb: cleanup wording, create a `PositionalEmbedding` layer.
* Replace `tensorflow::Status::OK()` with` tensorflow::OkStatus()`.
* Update README with note about various OS releases.
* Cast the step type.
* Reactivate TFLite ByteSplitter test.
* Modify tokenizer to process pt_examples to tokenizers.pt
* fix words alignment in documentation
* Update nmt_with_attention:
* transformers.ipynb: Factor out CrossAttention, GlobalSelfAttention, and CausalSelfAttention layers.
* Switch the transformer to train with `Model.fit`.
* Whitespace changes to force republish.
* Fix tutorial display, again.
* Update version

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

satojkovic

Bug Fixes and Other Changes

* tensorflow:
* Moving logging.h and bitmap from tf/core to tf/tsl.
* BOISE TF op:
* Add main C++ functions for converting to/from BOISE labels to offsets
* Add main C++ functions for converting to/from BOISE labels to offsets
* Add main C++ functions for converting to/from BOISE labels to offsets
* Add kernel code and Python API for `OffsetsToBoiseTags` op
* Other:
* Add link to KPLs, fix typo in Neural machine translation with attention tutorial
* Update README.md
* Publish the tensorflow_models.nlp guide docs to tensorflow.org
* Add missing dependency to constrained sequence kernel.
* Add missing absl status dependency to sentence breaking utils.
* Another missing absl status dependency. this time for sentence fragmenter.
* Add absl status to sentence fragmenter v2.
* Update pybind11 to 2.10.0 to match tensorflow.
* Better error message for WordPiece when the vocabulary file has unicode issues.
* Update Transformer tutorial with Keras MultiHeadAttention
* transformers.ipynb: fix length filter and target slicing
* transformers.ipynb: cleanup wording, create a `PositionalEmbedding` layer.
* Replace `tensorflow::Status::OK()` with` tensorflow::OkStatus()`.
* Update README with note about various OS releases.
* Cast the step type.
* Reactivate TFLite ByteSplitter test.
* Modify tokenizer to process pt_examples to tokenizers.pt
* fix words alignment in documentation
* Update nmt_with_attention:
* transformers.ipynb: Factor out CrossAttention, GlobalSelfAttention, and CausalSelfAttention layers.
* Switch the transformer to train with `Model.fit`.
* Whitespace changes to force republish.
* Add a phrase based tokenzier
* Fix tutorial display, again.
* Update version

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

satojkovic