Keras-nlp

The 0.8.0 release focuses on generative LLM features in KerasNLP.

Summary
* Added the `Mistral` and `Gemma` models.
* Allow passing `dtype` directly to backbone and task constructors.
* Add a settable `sequence_length` property to all preprocessing layers.
* Added `enable_lora()` to the backbone class for parameter efficient fine-tuning.
* Added layer attributes to backbone models for easier access to model internals.
* Added `AlibiBias` layer.

python
Pass dtype to a model.
causal_lm = keras_nlp.MistralCausalLM.from_preset(
"mistral_instruct_7b_en",
dtype="bfloat16"
)
Settable sequence length property.
causal_lm.preprocessor.sequence_length = 128
Lora API.
causal_lm.enable_lora(rank=4)
Easy layer attributes.
for layer in causal_lm.backbone.transformer_layers:
print(layer.count_params())


What's Changed
* Fix test for recent keras 3 change by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1400
* Pass less state to jax generate function by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1398
* Add llama tokenizer by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1401
* Add Bloom Model by abuelnasr0 in https://github.com/keras-team/keras-nlp/pull/1382
* Try fixing tests by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1411
* Revert "Pass less state to jax generate function (1398)" by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1412
* Bloom tokenizer by abuelnasr0 in https://github.com/keras-team/keras-nlp/pull/1403
* Update black formatting by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1415
* Add Alibi bias layer by abuelnasr0 in https://github.com/keras-team/keras-nlp/pull/1404
* Pin to `tensorflow-hub 0.16.0` to fix CI error by sampathweb in https://github.com/keras-team/keras-nlp/pull/1420
* Update TF Text and remove TF Hub deps by sampathweb in https://github.com/keras-team/keras-nlp/pull/1423
* Pin Jax Version in GPU CI by sampathweb in https://github.com/keras-team/keras-nlp/pull/1430
* Add Bloom preprocessor by abuelnasr0 in https://github.com/keras-team/keras-nlp/pull/1424
* Add layer attributes for all functional models by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1421
* Allow setting dtype per model by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1431
* Add a Causal LM model for Mistral by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1429
* Fix bart by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1434
* Add a settable property for sequence_length by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1437
* Add dependabot to update GH Actions and Python dependencies by pnacht in https://github.com/keras-team/keras-nlp/pull/1380
* Bump the github-actions group with 1 update by dependabot in https://github.com/keras-team/keras-nlp/pull/1438
* Add 7B presets for Mistral by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1436
* Update byte_pair_tokenizer.py to close merges file properly by divyashreepathihalli in https://github.com/keras-team/keras-nlp/pull/1440
* bump version to 0.8 by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1441
* Update our sampler documentation to reflect usage by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1444
* Add Gemma model by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1448
* Version bump for dev release by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1449
* Version bump to 0.8.0 by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1450

New Contributors
* dependabot made their first contribution in https://github.com/keras-team/keras-nlp/pull/1438
* divyashreepathihalli made their first contribution in https://github.com/keras-team/keras-nlp/pull/1440

**Full Changelog**: https://github.com/keras-team/keras-nlp/compare/v0.7.0...v0.8.0

This release integrates KerasNLP and [Kaggle Models](https://www.kaggle.com/models). KerasNLP models will now work in Kaggle offline notebooks and all assets will quickly attach to a notebook rather than needing a slow download.

Summary

KerasNLP pre-trained models are now all made available through [Kaggle Models](https://www.kaggle.com/models). You can see all models currently available in both KerasCV and KerasNLP [here](https://www.kaggle.com/organizations/keras/models). Individual [model pages](https://www.kaggle.com/models/keras/gpt2) will include example usage and a file browser to examine all available assets for a model preset.

This change will not affect the existing usage of `from_preset()`. Statement like `keras_nlp.models.BertClassifier.from_preset("bert_base_en")` will continue to work and download checkpoints from the Kaggle Models hub.

A note on model saving—for saving support across Keras 2 and Keras 3, we recommend using the new Keras saved model format. You can use `model.save('path/to/location.keras')` for a full model and `model.save_weights('path/to/location.weights.h5')` for checkpoints. See the [Keras saving guide](https://keras.io/guides/serialization_and_saving/) for more details.

What's Changed
* Don't export model internals publicly by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1255
* Bump master branch version number to 0.7.0.dev0 by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1254
* Fix/allow different encoder and decoder feature dimensions in transformer decoder layer by ferraric in https://github.com/keras-team/keras-nlp/pull/1260
* Doc updates to switch branding to Keras 3 by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1259
* Remove unused TPU testing for backbones by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1266
* Make gelu a function, not a lambda so it can be loaded without safe_mode=False by calvingiles in https://github.com/keras-team/keras-nlp/pull/1262
* Update requirements and install instructions for multi-backend keras by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1257
* Support Keras 3 installation by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1258
* Remove dtensor by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1268
* Add a lora dense layer by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1263
* Factor out testing routines for models by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1269
* Convert T5 to Keras 3 by nkovela1 in https://github.com/keras-team/keras-nlp/pull/1274
* Fix missing backticks in DistilBertClassifier docstrings by Philmod in https://github.com/keras-team/keras-nlp/pull/1278
* T5 checkpoint conversion with HF by nkovela1 in https://github.com/keras-team/keras-nlp/pull/1277
* Use gelu_approximate directly in t5 presets by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1284
* Add preset tests and weights URLs by nkovela1 in https://github.com/keras-team/keras-nlp/pull/1285
* Support loading keras 3 nightly by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1286
* Remove the use of `SentencePieceTrainer` from tests by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1283
* Fix XLM-RoBERTa detokenize() by abheesht17 in https://github.com/keras-team/keras-nlp/pull/1289
* Correct tie_embedding_weights and add logit checking by nkovela1 in https://github.com/keras-team/keras-nlp/pull/1288
* Add detokenize testing for model tokenizers by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1290
* Fix Whisper by abheesht17 in https://github.com/keras-team/keras-nlp/pull/1287
* Test against Keras 3 by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1273
* Support TF_USE_LEGACY_KERAS by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1295
* Run workflows with read-only tokens by pnacht in https://github.com/keras-team/keras-nlp/pull/1305
* Update CONTRIBUTING.md by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1310
* Add GitHub Action for Nightly by sampathweb in https://github.com/keras-team/keras-nlp/pull/1309
* Fix the publish to pypi action by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1311
* Fix nightly tf failure by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1316
* Switch deberta to use the "int" dtype by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1315
* Add security policy by pnacht in https://github.com/keras-team/keras-nlp/pull/1319
* Fix missing export for reversible embedding by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1327
* Add `version` API to keras_nlp by grasskin in https://github.com/keras-team/keras-nlp/pull/1324
* Fix Keras 3 version check by sampathweb in https://github.com/keras-team/keras-nlp/pull/1328
* Simplify running KerasNLP with Keras 3 by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1308
* Fix issues with version by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1332
* Fix typo in whisper presets files by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1337
* `ELECTRA` backbone implementation in keras by pranavvp16 in https://github.com/keras-team/keras-nlp/pull/1291
* Fix t5 tokenizer expected output by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1348
* Add __init__.py for electra by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1352
* Remove lora dense for now by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1359
* Adds Kokoro Build script for Keras-NLP GPU tests by sampathweb in https://github.com/keras-team/keras-nlp/pull/1355
* Fixes GPU Test failures for Keras 3 by sampathweb in https://github.com/keras-team/keras-nlp/pull/1361
* Change Continuous config to also run only large tests by sampathweb in https://github.com/keras-team/keras-nlp/pull/1362
* ElectraTokenizer by pranavvp16 in https://github.com/keras-team/keras-nlp/pull/1357
* Add MistralAI's 7B Transformer as a backbone in KerasNLP Models by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1314
* changing pooling output by mbrhd in https://github.com/keras-team/keras-nlp/pull/1364
* Add `LlamaBackbone` by shivance in https://github.com/keras-team/keras-nlp/pull/1203
* Align pip_build with keras by sampathweb in https://github.com/keras-team/keras-nlp/pull/1374
* Remove cloudbuild config by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1375
* Fix one last bad preset hash by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1381
* Add a tokenizer for the Mistral backbone by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1383
* Kaggle Presets by sampathweb in https://github.com/keras-team/keras-nlp/pull/1365
* Fix mistral and electra tokenizer to match kaggle changes by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1387
* Align requirments with Keras by sampathweb in https://github.com/keras-team/keras-nlp/pull/1386
* Add a preprocessor for the Mistral backbone by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1385
* Switch to always expect full Kaggle preset handles by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1390

New Contributors
* calvingiles made their first contribution in https://github.com/keras-team/keras-nlp/pull/1262
* tirthasheshpatel made their first contribution in https://github.com/keras-team/keras-nlp/pull/1283
* pnacht made their first contribution in https://github.com/keras-team/keras-nlp/pull/1305
* grasskin made their first contribution in https://github.com/keras-team/keras-nlp/pull/1324
* pranavvp16 made their first contribution in https://github.com/keras-team/keras-nlp/pull/1291
* mbrhd made their first contribution in https://github.com/keras-team/keras-nlp/pull/1364

**Full Changelog**: https://github.com/keras-team/keras-nlp/compare/v0.6.4...v0.7.0

Summary
This point release simplifies our support for Keras 3 and Keras 2.
- If Keras 2 is installed, KerasNLP will use Keras 2 and TensorFlow.
- If Keras 3 is installed, KerasNLP will use Keras 3 and run on any backend.

If you have any issue installing KerasNLP, please open [an issue](https://github.com/keras-team/keras-nlp/issues/new/choose).

What's Changed
* 0.6.4 cherry picks by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1350
* Version bump for 0.6.4.dev0 pre-release by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1351
* Version bump for 0.6.4 release by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1356

**Full Changelog**: https://github.com/keras-team/keras-nlp/compare/v0.6.3...v0.6.4

Summary

This release adds support for running KerasNLP against Keras 3. You can try this today by installing `tf-nightly` and `tensorflow-text-nightly`.

shell
pip install keras-nlp
pip uninstall -y tensorflow-text tensorflow keras
pip install tensorflow-text-nightly tf-nightly


Otherwise, this release should be a no-op for all users. No new features, no change in default behavior.

Upcoming changes

After the release of Keras 3, we will drop support for running KerasNLP against the Keras Core package (no more `import keras_core as keras`), in favor of Keras 3. Keras 3 is the long-term replacement for Keras Core.

What's Changed
* Cherry picks for 0.6.3 by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1297
* Version bump 0.6.3 by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1298
* Bump the version to 0.6.3.dev1 by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1301
* Version bump to 0.6.3 by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1302


**Full Changelog**: https://github.com/keras-team/keras-nlp/compare/v0.6.2...v0.6.3

Summary
* Support mixed precision on keras-core on all of jax, torch and tensorflow.
* Add `keras_nlp.layers.RotaryEmbedding` for rotary embeddings.
* Add `keras_nlp.layers.ReversibleEmbedding` to better support tied or untied weights for logit projections.
* Many bug fixes and improvements.

What's Changed
* Generic `RotaryEmbedding` Layer by shivance in https://github.com/keras-team/keras-nlp/pull/1180
* Raise ValueError when number of dims evaluate to zero by sampathweb in https://github.com/keras-team/keras-nlp/pull/1198
* Add XLNetBackbone by susnato in https://github.com/keras-team/keras-nlp/pull/1084
* Switch from tf.nest to dm-tree by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1199
* Fix CI for keras-core 0.1.4 by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1202
* Fix ModuleNotFoundError `keras_nlp.models.xlnet` by shivance in https://github.com/keras-team/keras-nlp/pull/1204
* Add support for "untied" embedding weights in language models by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1201
* Add start_index argument to all position embedding layers by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1209
* Remove windows line endings by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1210
* Fix Autograph error with perplexity metric by shivance in https://github.com/keras-team/keras-nlp/pull/1211
* [JAX backend]: Fix errors with perplexity by shivance in https://github.com/keras-team/keras-nlp/pull/1213
* Improve layer naming consistency by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1219
* Stop asserting key order in bart preprocessor by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1221
* Remove file level docstrings by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1222
* Fix typos by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1220
* Typo fix by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1223
* Fix RotaryEmbedding import by shivance in https://github.com/keras-team/keras-nlp/pull/1217
* Update transformer_decoder for the proper naming of the sublayers. by qlzh727 in https://github.com/keras-team/keras-nlp/pull/1230
* Replace tf with numpy by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1232
* Update to always using ops.shape by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1231
* Add a test harness based on keras-core's `run_layer_test` by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1238
* fixed token_to_id doc + error msg by jackd in https://github.com/keras-team/keras-nlp/pull/1240
* Changed default TokenAndPositionEmbedding initializer to 'uniform' by jackd in https://github.com/keras-team/keras-nlp/pull/1237
* Add compat shims for the upcoming keras-core release by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1244
* Depend on latest keras-core by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1246
* Removed the undefined self.sequence_length by sahusiddharth in https://github.com/keras-team/keras-nlp/pull/1245
* Bump devcontainer to 3.9 by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1249
* Add a mixed precision test and fix mixed precision errors for layers by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1242
* Quick fix for 0.1.7 keras-core release by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1251
* Small docstring fixes for the upcoming release by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1253

New Contributors
* qlzh727 made their first contribution in https://github.com/keras-team/keras-nlp/pull/1230
* jackd made their first contribution in https://github.com/keras-team/keras-nlp/pull/1240
* sahusiddharth made their first contribution in https://github.com/keras-team/keras-nlp/pull/1245

**Full Changelog**: https://github.com/keras-team/keras-nlp/compare/v0.6.1...v0.6.2

With the 0.6.1. release, all remaining models, metrics and samplers have been ported to [keras-core](https://keras.io/keras_core/). The full KerasNLP API is now available on TensorFlow, PyTorch and Jax ([instructions](https://github.com/keras-team/keras-nlp#using-kerasnlp-with-keras-core)).

Summary
* FNet and DeBERTa are now multi-backend.
* All `keras_nlp.models.FNetXX` and `keras_nlp.models.DebertaV3XX` symbols work on all backends.
* `keras_nlp.samplers.BeamSampler` and `keras_nlp.samplers.ContrastiveSampler` work on all backends.
* All `keras_nlp.metrics` classes work on all backends.
* For Jax and PyTroch, pass python strings to metrics (as tensor strings are strictly tensorflow).
* Restored the `mask_positions` named argument to `MaskedLMHead`.

What's Changed
* Update README for Keras Core by jbischof in https://github.com/keras-team/keras-nlp/pull/1135
* Ignore errors in UTF-8 decoding by abheesht17 in https://github.com/keras-team/keras-nlp/pull/1150
* Ports GPTNeoX to KerasCore by shivance in https://github.com/keras-team/keras-nlp/pull/1137
* Small fix for mixed precision generation on tf by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1153
* Port DeBERTa to multi-backend by abheesht17 in https://github.com/keras-team/keras-nlp/pull/1155
* Change all tensors passed to tf.data.Dataset to numpy by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1161
* Fix broken tests by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1163
* Pin keras-core to 0.1.0 while investigating failures by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1168
* Run GPU tests on Jax + Torch by ianstenbit in https://github.com/keras-team/keras-nlp/pull/1160
* Fix flakes in masked lm testing by removing any indeterminism by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1171
* Always install the correct version with pip_build by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1174
* Remove tests for preprocessing inside a functional model by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1175
* Extend the timeout for large tests by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1103
* Add `GPTNeoXCausalLM` by shivance in https://github.com/keras-team/keras-nlp/pull/1110
* Bump tensorflow to latest stable by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1170
* Add compute_output_shape to tokenizer by shivance in https://github.com/keras-team/keras-nlp/pull/1166
* Stop pinning keras-core by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1178
* Port FNet by abheesht17 in https://github.com/keras-team/keras-nlp/pull/1164
* Automate the update image flow by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1179
* Restore mask_position argument name by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1185
* Port contrastive sampler to multi-backend by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1187
* Port `BeamSampler` to core by shivance in https://github.com/keras-team/keras-nlp/pull/1181
* Port metrics to multi-backend by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1186

New Contributors
* ianstenbit made their first contribution in https://github.com/keras-team/keras-nlp/pull/1160

**Full Changelog**: https://github.com/keras-team/keras-nlp/compare/v0.6.0...v0.6.1