The 0.9.0 release adds new models, hub integrations, and general usability improvements.
Summary
* Added the **Gemma 1.1** release.
* Added the **Llama 2**, **BLOOM** and **ELECTRA** models.
* Expose new base classes. Allow `from_preset()` on base classes.
* `keras_nlp.models.Backbone`
* `keras_nlp.models.Task`
* `keras_nlp.models.Classifier`
* `keras_nlp.models.CausalLM`
* `keras_nlp.models.Seq2SeqLM`
* `keras_nlp.models.MaskedLM`
* Some *initial* features for uploading to model hubs.
* `backbone.save_to_preset`, `tokenizer.save_to_preset`, `keras_nlp.upload_preset`.
* `from_preset` and `upload_preset` now work with the Hugging Face Models Hub.
* More features (task saving, lora saving), and full documentation coming soon.
* Numerical fixes for the Gemma model at mixed_bfloat16 precision. Thanks unsloth [for catching](https://unsloth.ai/blog/gemma-bugs)!
python
Llama 2. Needs Kaggle consent and login, see https://github.com/Kaggle/kagglehub
causal_lm = keras_nlp.models.LlamaCausalLM.from_preset(
"llama2_7b_en",
dtype="bfloat16", Run at half precision for inference.
)
causal_lm.generate("Keras is a", max_length=128)
Base class usage.
keras_nlp.models.Classifier.from_preset("bert_base_en", num_classes=2)
keras_nlp.models.Tokenizer.from_preset("gemma_2b_en")
keras_nlp.models.CausalLM.from_preset("gpt2_base_en", dtype="mixed_bfloat16")
What's Changed
* Add dtype arg to Gemma HF conversion script by nkovela1 in https://github.com/keras-team/keras-nlp/pull/1452
* Fix gemma testing import by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1462
* Add docstring for PyTorch conversion script install instructions by nkovela1 in https://github.com/keras-team/keras-nlp/pull/1471
* Add an annotation to tests that need kaggle auth by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1470
* Fix Mistral memory consumption with JAX and default dtype bug by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1460
* Bump the master version to 0.9 by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1473
* Pin to TF 2.16 RC0 by sampathweb in https://github.com/keras-team/keras-nlp/pull/1478
* Fix gemma rms_normalization's use of epsilon by cpsauer in https://github.com/keras-team/keras-nlp/pull/1472
* Add `FalconBackbone` by SamanehSaadat in https://github.com/keras-team/keras-nlp/pull/1475
* CI - Add kaggle creds to pull model by sampathweb in https://github.com/keras-team/keras-nlp/pull/1459
* bug in example for ReversibleEmbedding by TheCrazyT in https://github.com/keras-team/keras-nlp/pull/1484
* doc fix for constrastive sampler by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1488
* Remove broken link to masking and padding guide by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1487
* Fix a typo in causal_lm_preprocessors by SamanehSaadat in https://github.com/keras-team/keras-nlp/pull/1489
* Fix dtype accessors of tasks/backbones by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1486
* Auto-labels 'gemma' on 'gemma' issues/PRs. by shmishra99 in https://github.com/keras-team/keras-nlp/pull/1490
* Add BloomCausalLM by abuelnasr0 in https://github.com/keras-team/keras-nlp/pull/1467
* Remove the bert jupyter conversion notebooks by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1492
* Add `FalconTokenizer` by SamanehSaadat in https://github.com/keras-team/keras-nlp/pull/1485
* Add `FalconPreprocessor` by SamanehSaadat in https://github.com/keras-team/keras-nlp/pull/1498
* Rename 176B presets & Add other presets into bloom_presets.py by abuelnasr0 in https://github.com/keras-team/keras-nlp/pull/1496
* Add bloom presets by abuelnasr0 in https://github.com/keras-team/keras-nlp/pull/1501
* Create workflow for auto assignment of issues and for stale issues by sachinprasadhs in https://github.com/keras-team/keras-nlp/pull/1495
* Update requirements to TF 2.16 by sampathweb in https://github.com/keras-team/keras-nlp/pull/1503
* Expose Task and Backbone by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1506
* Clean up and add our gemma conversion script by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1493
* Don't auto-update JAX GPU by sampathweb in https://github.com/keras-team/keras-nlp/pull/1507
* Keep rope at float32 precision by grasskin in https://github.com/keras-team/keras-nlp/pull/1497
* Bump the python group with 2 updates by dependabot in https://github.com/keras-team/keras-nlp/pull/1509
* Fixes for the LLaMA backbone + add dropout by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1499
* Add `LlamaPreprocessor` and `LlamaCausalLMPreprocessor` by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1511
* Always run the rotary embedding layer in float32 by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1508
* CI: Fix psutil - Remove install of Python 3.9 and alias of python3 by sampathweb in https://github.com/keras-team/keras-nlp/pull/1514
* Update gemma_backbone.py for sharding config. by qlzh727 in https://github.com/keras-team/keras-nlp/pull/1491
* Docs/modelling layers by mykolaskrynnyk in https://github.com/keras-team/keras-nlp/pull/1502
* Standardize docstring by sachinprasadhs in https://github.com/keras-team/keras-nlp/pull/1516
* Support tokenization of special tokens for word_piece_tokenizer by abuelnasr0 in https://github.com/keras-team/keras-nlp/pull/1397
* Upload Model to Kaggle by SamanehSaadat in https://github.com/keras-team/keras-nlp/pull/1512
* Add scoring mode to MistralCausalLM by RyanMullins in https://github.com/keras-team/keras-nlp/pull/1521
* Add Mistral Instruct V0.2 preset by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1520
* Add Tests for Kaggle Upload Validation by SamanehSaadat in https://github.com/keras-team/keras-nlp/pull/1524
* Add presets for Electra and checkpoint conversion script by pranavvp16 in https://github.com/keras-team/keras-nlp/pull/1384
* Allow saving / loading from Huggingface Hub preset by Wauplin in https://github.com/keras-team/keras-nlp/pull/1510
* Stop on multiple end tokens by grasskin in https://github.com/keras-team/keras-nlp/pull/1518
* Fix doc: `mistral_base_en` -> `mistral_7b_en` by asmith26 in https://github.com/keras-team/keras-nlp/pull/1528
* Add lora example to GemmaCausalLM docstring by SamanehSaadat in https://github.com/keras-team/keras-nlp/pull/1527
* Add LLaMA Causal LM with 7B presets by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1526
* Add task base classes; support out of tree library extensions by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1517
* Doc fixes by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1530
* Run the LLaMA and Mistral RMS Layer Norm in float32 by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1532
* Adds score API to GPT-2 by RyanMullins in https://github.com/keras-team/keras-nlp/pull/1533
* increase pip timeout to 1000s to avoid connection resets by sampathweb in https://github.com/keras-team/keras-nlp/pull/1535
* Adds the score API to LlamaCausalLM by RyanMullins in https://github.com/keras-team/keras-nlp/pull/1534
* Implement compute_output_spec() for tokenizers with vocabulary. by briango28 in https://github.com/keras-team/keras-nlp/pull/1523
* Remove staggler type annotiations by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1536
* Always run SiLU activation in float32 for LLaMA and Mistral by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1540
* Bump the python group with 2 updates by dependabot in https://github.com/keras-team/keras-nlp/pull/1538
* Disallow saving to preset from keras 2 by SamanehSaadat in https://github.com/keras-team/keras-nlp/pull/1545
* Fix the rotary embedding computation in LLaMA by tirthasheshpatel in https://github.com/keras-team/keras-nlp/pull/1544
* Fix re-compilation bugs by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1541
* Fix preprocessor from_preset bug by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1549
* Fix a strange issue with preprocessing layer output types by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1550
* Fix lowercase bug in wordpiece tokenizer by abuelnasr0 in https://github.com/keras-team/keras-nlp/pull/1543
* Small docs updates by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1553
* Add a few new preset for gemma by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1556
* Remove the dev prefix for 0.9.0 release by mattdangerw in https://github.com/keras-team/keras-nlp/pull/1557
New Contributors
* cpsauer made their first contribution in https://github.com/keras-team/keras-nlp/pull/1472
* SamanehSaadat made their first contribution in https://github.com/keras-team/keras-nlp/pull/1475
* TheCrazyT made their first contribution in https://github.com/keras-team/keras-nlp/pull/1484
* shmishra99 made their first contribution in https://github.com/keras-team/keras-nlp/pull/1490
* sachinprasadhs made their first contribution in https://github.com/keras-team/keras-nlp/pull/1495
* mykolaskrynnyk made their first contribution in https://github.com/keras-team/keras-nlp/pull/1502
* RyanMullins made their first contribution in https://github.com/keras-team/keras-nlp/pull/1521
* Wauplin made their first contribution in https://github.com/keras-team/keras-nlp/pull/1510
* asmith26 made their first contribution in https://github.com/keras-team/keras-nlp/pull/1528
* briango28 made their first contribution in https://github.com/keras-team/keras-nlp/pull/1523
**Full Changelog**: https://github.com/keras-team/keras-nlp/compare/v0.8.2...v0.9.0