Tensorflow-text

Major Features and Improvements

* Added new tokenizer: FastWordpieceTokenizer that is considerably faster than the original WordpieceTokenizer
* WhitespaceTokenizer was rewritten to increase speed and smaller kernel size
* Ability to convert WhitespaceTokenizer & FastWordpieceTokenizer to TF Lite
* Added Keras layers for tokenizers: UnicodeScript, Whitespace, & Wordpiece

Bug Fixes and Other Changes

* (Generated change) Update tf.Text versions and/or docs.
* tiny change for variable name in transformer tutorial
* Update nmt_with_attention.ipynb
* Add vocab_size for wordpiece tokenizer to have consistency with sentence piece.
* This is a general clean up to the build files. The previous tf_deps paradigm was confusing. By encapsulating everything into a single call lib, I'm hoping this makes it easier to understand and follow.
* This adds the builder for the new WhitespaceTokenizer config cache. This is the first in a series of changes to update the WST for mobile.
* C++ API for new WhitespaceTokenizer. The updated API is more useful (accepts strings instead of ints), faster, and smaller in size.
* Adds pywrap for WhitespaceTokenizer config builder.
* Simplify the configure.bzl. Since for each platform we build with C++14, let's just make it easier to default to it across the board. This should be easier to understand and maintain.
* Remove most of the default oss deps for kernels as they are no longer required for building.
* Updating this BERT tutorial to use model subclassing (easier for students to hack on it this way).
* Adds kernels for TF & TFLite for the new WhitespaceTokenizer.
* Fix a problem with the WST template that was causing members to be exported as undefined symbols. After this change they become a unique global symbol in the shared object file.
* Update whitespace op to use new kernel. This change still allows for building the old kernel as well so current users can continue to use it, even though we cannot make new calls to it.
* Update whitespace op to use new kernel. This change still allows for building the old kernel as well so current users can continue to use it, even though we cannot make new calls to it.
* Convert the TFLite kernel for ngram with STRING_JOIN mode to use tfshim so the same code is now used for TF and TFLite kernels.
* fix: masked_ids -> masked_lm_ids
* Save the transformer.
* Remove the sentencepiece patch in OSS
* fix vocab_table arg is not used in bert_pretrain_preprocess()
* Disable TSAN for one more tutorial test that may run for >900sec when TSAN is
* Remove the sentencepiece patch in OSS
* internal
* (Generated change) Update tf.Text versions and/or docs.
* Update deps to fix broken build.
* Remove --gen_report flag.
* Small typo fixed
* Explain that all heads are handled with a single Dense layer
* internal change, should be a noop in github.
* Update whitespace op to use new kernel. This change still allows for building the old kernel as well so current users can continue to use it, even though we cannot make new calls to it.
* Creates tf Lite registrar and adds TF Lite tests for mobile ops.
* Fix nmt_with_attention start_index
* Export LD_LIBRARY_PATH when configuring for build.
* Update tf lite test to use the function rather than having to globally share the linked library symbols so the interpreter can find the name since this is only available on linux.
* Temporarily switch to the definition of REGISTER_TF_OP_SHIM while it updates.
* Update REGISTER_TF_OP_SHIM macro to remove unnecessary parameter.
* Remove temporary code and set back to using the op shim macro.
* Updated import statement
* Internal change
* pushed back forward compatibility date for tf_text.WhitespaceTokenizer.
* Add .gitignore
* The --keep_going flag will make bazel run all tests instead of stopping
* Add missing blank line between test and doctest.
* Adds a regression test for model server for the replaced WST op. This ensures that current models using the old kernel will continue to work.
* Fix the build by adding a new dependency required by TF to kernel targets.
* Add sentenepiece detokenize op to stateful allowlist.
* Fix broken build. This occurred because of a change on TF that updated the compiler infra version (https://github.com/tensorflow/tensorflow/commit/e0940f269a10f409466b6fef4ef531aec81f9afa).
* Clean up code now that the build horizon has passed.
* Add pywrap dependency for tflite ops.
* Update TextVectorization layer
* Allows overridden get_selectable to be used.
* Update version

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Aaron Siddhartha Mondal, Abhijeet Manhas, Dominik Schlösser, jaymessina3, Mao, Xiaoquan Kong, Yasir Modak

Major Features and Improvements

* WhitespaceTokenizer was rewritten to increase speed and smaller kernel size
* Ability to convert some ops to TF Lite

Bug Fixes and Other Changes

* (Generated change) Update tf.Text versions and/or docs.
* tiny change for variable name in transformer tutorial
* Update nmt_with_attention.ipynb
* Add vocab_size for wordpiece tokenizer to have consistency with sentence piece.
* This is a general clean up to the build files. The previous tf_deps paradigm was confusing. By encapsulating everything into a single call lib, I'm hoping this makes it easier to understand and follow.
* This adds the builder for the new WhitespaceTokenizer config cache. This is the first in a series of changes to update the WST for mobile.
* C++ API for new WhitespaceTokenizer. The updated API is more useful (accepts strings instead of ints), faster, and smaller in size.
* Adds pywrap for WhitespaceTokenizer config builder.
* Simplify the configure.bzl. Since for each platform we build with C++14, let's just make it easier to default to it across the board. This should be easier to understand and maintain.
* Remove most of the default oss deps for kernels as they are no longer required for building.
* Updating this BERT tutorial to use model subclassing (easier for students to hack on it this way).
* Adds kernels for TF & TFLite for the new WhitespaceTokenizer.
* Fix a problem with the WST template that was causing members to be exported as undefined symbols. After this change they become a unique global symbol in the shared object file.
* Update whitespace op to use new kernel. This change still allows for building the old kernel as well so current users can continue to use it, even though we cannot make new calls to it.
* Update whitespace op to use new kernel. This change still allows for building the old kernel as well so current users can continue to use it, even though we cannot make new calls to it.
* Convert the TFLite kernel for ngram with STRING_JOIN mode to use tfshim so the same code is now used for TF and TFLite kernels.
* fix: masked_ids -> masked_lm_ids
* Save the transformer.
* Remove the sentencepiece patch in OSS
* fix vocab_table arg is not used in bert_pretrain_preprocess()
* Disable TSAN for one more tutorial test that may run for >900sec when TSAN is
* Remove the sentencepiece patch in OSS
* internal
* (Generated change) Update tf.Text versions and/or docs.
* Update deps to fix broken build.
* Remove --gen_report flag.
* Small typo fixed
* Explain that all heads are handled with a single Dense layer
* internal change, should be a noop in github.
* Update whitespace op to use new kernel. This change still allows for building the old kernel as well so current users can continue to use it, even though we cannot make new calls to it.
* Creates tf Lite registrar and adds TF Lite tests for mobile ops.
* Fix nmt_with_attention start_index
* Export LD_LIBRARY_PATH when configuring for build.
* Update tf lite test to use the function rather than having to globally share the linked library symbols so the interpreter can find the name since this is only available on linux.
* Temporarily switch to the definition of REGISTER_TF_OP_SHIM while it updates.
* Update REGISTER_TF_OP_SHIM macro to remove unnecessary parameter.
* Remove temporary code and set back to using the op shim macro.
* Updated import statement
* Internal change
* pushed back forward compatibility date for tf_text.WhitespaceTokenizer.
* Add .gitignore
* The --keep_going flag will make bazel run all tests instead of stopping
* Add missing blank line between test and doctest.
* Adds a regression test for model server for the replaced WST op. This ensures that current models using the old kernel will continue to work.
* Fix the build by adding a new dependency required by TF to kernel targets.
* Add sentenepiece detokenize op to stateful allowlist.
* Fix broken build. This occurred because of a change on TF that updated the compiler infra version (https://github.com/tensorflow/tensorflow/commit/e0940f269a10f409466b6fef4ef531aec81f9afa).
* Clean up code now that the build horizon has passed.
* Add pywrap dependency for tflite ops.
* Update TextVectorization layer
* Allows overridden get_selectable to be used.
* Update version

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

Aaron Siddhartha Mondal, Dominik Schlösser, Xiaoquan Kong, Yasir Modak

Bug Fixes and Other Changes

* Update `__init__.py`: Added a `__version__` variable
* Fixes the benchmark suite for graph mode. While using tf.function prevented caching, it was also causing the graph being tested to rebuild each time. Using placeholder instead fixes this.
* Pin nightly version.
* Remove TF patch as it is not needed anymore. The code is in core TF.
* Typos
* Format and lint NBs, add images
* Add a couple notes to the BertTokenizer docs.
* Narrative docs migration: TF Core -> TF Text
* Update nmt_with_attention
* Moved examples of a few API docs above the args sections to better match other formats.
* Fix NBs
* Update Installation from source instruction.
* Add SplitterWithOffsets as an exported symbol.
* Fix a note to the BertTokenizer docs.
* Remove unused index.md
* Convert tensorflow_text to use public TF if possible.
* Fix failing notebooks.
* Create user_ops BUILD file.
* Remove unnecessary METADATA.
* Replace tf.compat.v2.xxx with tf.xxx, since tf_text is using tf2 only.
* Fix load_data function in nmt tutorial
* Update tf.data.AUTOTUNE in Fine-tuning a BERT model
* Switch TF to OSS keras (1/N).
* added *sub*spaces
* Disable TSAN for tutorial tests that may run for >900sec when TSAN is enabled.
* Adds a short description to the main landing page of our GitHub repo to point users to the tf.org subsite.
* Phrasing fix to TF Transformer tutorial.
* Disable RTTI when building Tf.Text kernels for mobile
* Migrate the references in third_party/toolchains directory as it is going to be deleted soon.
* Fix bug in RoundRobinTrimmer. Previously the stopping condition was merging and combining from across different batches. Instead now the stopping condition is first determined in each batch, then aggregated.
* Set mask_token='' to make it work with TF 2.6.0
* Builds TF Text with C++14 by default. This is already done by TensorFlow, and the TF Lite shim has C++14 features used within; thus, this is needed to build kernels against it.
* This is a general clean up to the build files. The previous tf_deps paradigm was confusing. By encapsulating everything into a single call lib, I'm hoping this makes it easier to understand and follow.
* Update the WORKSPACE to not use the same "workspace" name when initializing TensorFlow.

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

8bitmp3, akiprasad, bongbonglemon, Jules Gagnon-Marchand, Stonepia

Bug Fixes and Other Changes

* Update `__init__.py`: Added a `__version__` variable
* Fixes the benchmark suite for graph mode. While using tf.function prevented caching, it was also causing the graph being tested to rebuild each time. Using placeholder instead fixes this.
* Pin nightly version.
* Remove TF patch as it is not needed anymore. The code is in core TF.
* Typos
* Format and lint NBs, add images
* Add a couple notes to the BertTokenizer docs.
* Narrative docs migration: TF Core -> TF Text
* Update nmt_with_attention
* Moved examples of a few API docs above the args sections to better match other formats.
* Fix NBs
* Update Installation from source instruction.
* Add SplitterWithOffsets as an exported symbol.
* Fix a note to the BertTokenizer docs.
* Remove unused index.md
* Convert tensorflow_text to use public TF if possible.
* Fix failing notebooks.
* Create user_ops BUILD file.
* Remove unnecessary METADATA.
* Replace tf.compat.v2.xxx with tf.xxx, since tf_text is using tf2 only.
* Fix load_data function in nmt tutorial
* Update tf.data.AUTOTUNE in Fine-tuning a BERT model
* Switch TF to OSS keras (1/N).
* added *sub*spaces
* Disable TSAN for tutorial tests that may run for >900sec when TSAN is enabled.
* Adds a short description to the main landing page of our GitHub repo to point users to the tf.org subsite.
* Phrasing fix to TF Transformer tutorial.
* Disable RTTI when building Tf.Text kernels for mobile

Thanks to our Contributors

This release contains contributions from many people at Google, as well as:

8bitmp3, akiprasad, bongbonglemon, Jules Gagnon-Marchand, Stonepia

We want to particularly point out that guides, tutorials, and API docs are currently being published to http://tensorflow.org/text ! This should make it easier for users to find our documentation. We worked hard on improving docs across the board, so feel free to let us know if further clarification is needed.

Major Features and Improvements

* API docs, guides, & tutorial are now available on http://tensorflow.org/text
* New guides & tutorials including: tokenizers, subwords tokenizer, and BERT text preprocessing guide.
* Add RoundRobinTrimmer
* Add a function to generate a BERT vocab from a tf.data.Dataset.
* Add detokenize methods for BertTokenizer and WordpieceTokenizer.
* Enable NFD and NFKD in NormalizeWithOffset op

Bug Fixes and Other Changes

* Many API updates (eg. adding descriptions & examples) to various ops.
* Let SentencePieceTokenizer optionally return the nbest tokenizations instead of sampling from them.
* Fix a bug in split mode tokenizers that caused tests to fail on Windows.
* Fix broadcasting bugs in RoundRobinTrimmer
* Add WordpieceTokenizeWithOffsets with ALLOW_STATEFUL_OP_FOR_DATASET_FUNCTIONS for tf.data
* Remove PersistentTensor from sentencepiece_kernels.cc
* Document examples are now tested.
* Fix benchmarking of graph mode ops through use of tf.function.
* Set the default for mask_token for StringLookup and IntegerLookup to None
* Update the sentence_breaking_ops docstring to indicate that it's deprecated.
* Adding an i18n-friendly BasicTokenizer that can preserve accents
* For Windows, always include ICU data files since they need to be built in statically.
* Rename documentation file WordShape.md to WordShape_cls.md. Fix 361.
* Convert input to tensor to allow for numpy inputs to state based sentence breaker.
* Add classifiers to py packages and fix header image.
* Fix for the model server test.
* Update regression test for break_sentences_with_offsets.
* Add a shape attribute to the ToDense Keras layer.
* Add support for [batch, 1] shaped inputs in StateBasedSentenceBreaker
* Fix for the model server test.
* Refactor saved_model.py to make it easier to comment out blocks of related code to identify problems.
* Add regression test for Find Source Offsets
* Fix unselectable_ids shape check in ItemSelector.
* Switch out architecture image in tf.Text documentation.
* Fix regression test for state_based_sentence_breaker_v2
* Update run_build with enable_runfiles flag.
* Update the version of bazel_skylib to match TF's and fix a possible visibility issue.
* Simplify tf-text WORKSPACE, by relying on tf_workspace().
* Update transformer.ipynb to use a saved text.BertTokenizer
* Update mobile targets to use :mobile rather than separate :android & :ios targets.
* Make tools part of the tensorflow_text pip package.
* Import tools from the tf-text package, instead of cloning the git repo.
* Minor cleanups to make some code compile on the android build system.
* Fix pip install command in readme
* Fix tools pip package inclusion.
* A tensorfow.org compatible docs generator for tf-text.
* Sample random tokens correctly during MLM.
* Treat Sentencepiece ops as stateful in tf.data pipelines.
* Replacing use of TFT's deprecated dataset_schema.from_feature_spec with its replacement schema_utils.schema_from_feature_spec.