Optimum-neuron

What's Changed

Training

Distributed Training

- `parallel_cross_entropy` loss support for tensor parallelism (246)
- Support for training the Mistral architecture with tensor parallelism (303)

AWS SDK

- Fix: `neuron_parallel_compile` is compatible with the cache system (352)
- Full support for `neuron_parallel_compile` with the cache system: compilation files produced by `neuron_parallel_compile` will be pushed to the remote cache repo on the Hugging Face Hub at the beginning of the next training job (354)

Documentation

- [Guide](https://huggingface.co/docs/optimum-neuron/guides/distributed_training) explaining how distributed training works in `optimum-neuron` (#339)

Inference

- Data parallelism option for Stable Diffusion - LCM allowing multi-device inference (346)
- Support decoding sequences of byte tokens in TGI (350)

Documentation

- Updated the documentation on LCM (351)

What's Changed

LCM support

* [Stable Diffusion] Add LCM(Latent Consistency Models) support by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/323

Tutorials and doc improvement

* notebooks: add llama2 chatbot example by dacorvo in https://github.com/huggingface/optimum-neuron/pull/300
* Add llama 2 tutorial by dacorvo in https://github.com/huggingface/optimum-neuron/pull/321
* Migrate documentation of Stable Diffusion and add notebooks by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/312

Major bugfixes

* Noisy loss fix by bocchris-aws in https://github.com/huggingface/optimum-neuron/pull/293
* Fix neuron cache starting compilation before fetching by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/280
* fix(pipelines): support passing decoder model + tokenizer by dacorvo in https://github.com/huggingface/optimum-neuron/pull/319

Other changes

* chore: update dev version by dacorvo in https://github.com/huggingface/optimum-neuron/pull/276
* Explicitly mention aws repo extra url in documentation by dacorvo in https://github.com/huggingface/optimum-neuron/pull/277
* Update supported architecture in the doc by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/281
* Fix doc build source code broken links by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/282
* Add revision to push_to_hub by philschmid in https://github.com/huggingface/optimum-neuron/pull/292
* Set default device id for SD and SDXL by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/297
* Add missing decoder model architectures by dacorvo in https://github.com/huggingface/optimum-neuron/pull/298
* Official support for AWS inferentia2 TGI container by dacorvo in https://github.com/huggingface/optimum-neuron/pull/302
* Transformers fix by dacorvo in https://github.com/huggingface/optimum-neuron/pull/320
* Add sagemaker compatible image by dacorvo in https://github.com/huggingface/optimum-neuron/pull/322
* Fix broken tests by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/274
* chore: align with AWS Neuron SDK 2.15.1 by dacorvo in https://github.com/huggingface/optimum-neuron/pull/325
* Deleted the 'maybe_free_model_hooks()' from Diffusers Pipelines by Cerrix in https://github.com/huggingface/optimum-neuron/pull/330
* Bump diffusers version by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/335

New Contributors
* Cerrix made their first contribution in https://github.com/huggingface/optimum-neuron/pull/330

**Full Changelog**: https://github.com/huggingface/optimum-neuron/compare/v0.0.13...v0.0.14

What's Changed

The main change in this release is the alignment with AWS Neuron SDK 2.15.

Text-generation
* add support for `bloom` and `opt` models by dacorvo in 275

Other changes
* Use attention masks for TGI generation by dacorvo in 264
* Various fixes for TP by michaelbenayoun in 260
* Fix neuron pipelines by dacorvo in 265
* Fix 241 by michaelbenayoun in 268
* Fixes generation during the evaluation step by michaelbenayoun in 266
* Save / load from checkpoint TP by michaelbenayoun in 269

**Full Changelog**: https://github.com/huggingface/optimum-neuron/compare/v0.0.12...v0.0.13

Major bugfixes
* Fix https://github.com/huggingface/optimum-neuron/issues/241 by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/268

**Full Changelog**: https://github.com/huggingface/optimum-neuron/compare/v0.0.12...v0.0.12.1

What's Changed

Stable Diffusion: SDXL Refiner, Stable Diffusion Img2Img, Inpaint support
* [Stable Diffusion] Image2image and inpaint pipeline support by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/161
* [SDXL] Add SDXL image to image support by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/239

Distributed Training:
* Sequence parallelism by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/233
* Parallelism support for GPTNeoX by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/244

Text generation updates
* Add text generation pipeline by dacorvo in https://github.com/huggingface/optimum-neuron/pull/258

Other changes
* TGI stability fixes by dacorvo in https://github.com/huggingface/optimum-neuron/pull/226
* Remove experimental compilation flag for text-generation models by dacorvo in https://github.com/huggingface/optimum-neuron/pull/228
* Patch for diffusers 0.21.0 release by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/229
* test_examples uses ExampleRunner by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/227
* Using the real model name instead of hard code "model" by davidshtian in https://github.com/huggingface/optimum-neuron/pull/231
* Replace transformers list of logits warpers by a fused logic warper by dacorvo in https://github.com/huggingface/optimum-neuron/pull/234
* Use AWS Neuron SDK 2.14 by dacorvo in https://github.com/huggingface/optimum-neuron/pull/236
* Weight loading after lazy loading fix by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/238
* Add `debug` attribute to `NeuronPartialState` by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/240
* Update `tests/test_examples.py` for AWS team by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/242
* Rework text-generation example by dacorvo in https://github.com/huggingface/optimum-neuron/pull/245
* Fix evaluation recompilation issue by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/248
* test(generation): specify revision for hub test model by dacorvo in https://github.com/huggingface/optimum-neuron/pull/250
* Add sequence length for generative models and llama tests by dacorvo in https://github.com/huggingface/optimum-neuron/pull/251
* Fix noisy loss for T5 when doing TP by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/257
* Fix bug with transformers 4.34 by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/259

New Contributors
* davidshtian made their first contribution in https://github.com/huggingface/optimum-neuron/pull/231

**Full Changelog**: https://github.com/huggingface/optimum-neuron/compare/v0.0.11...v0.0.12

SDXL Export and Inference
Optimum CLI now supports compiling components in the SDXL pipeline for inference on neuron devices (inf2/trn1).

Below is an example of compiling SDXL models. You can either compile it with an inf2 instance (`inf2.8xlarge` or larger recommended) or a CPU-only instance (disable the validation with `--disable-validation`) :
bash
optimum-cli export neuron --model stabilityai/stable-diffusion-xl-base-1.0 --task stable-diffusion-xl --batch_size 1 --height 1024 --width 1024 --auto_cast matmul --auto_cast_type bf16 sdxl_neuron/

And then run inference with the class `NeuronStableDiffusionXLPipeline`
python
from optimum.neuron import NeuronStableDiffusionXLPipeline

prompt = "Astronaut in a jungle, cold color palette, muted colors, detailed, 8k"
stable_diffusion_xl = NeuronStableDiffusionXLPipeline.from_pretrained(
model_id="sdxl_neuron/", device_ids=[0, 1]
)
image = stable_diffusion_xl(prompt).images[0]


* Add sdxl exporter support by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/203
* Add Stable Diffusion XL inference support by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/212


Llama v1, v2 Inference
* Add support for Llama inference through NeuronModelForCausalLM by dacorvo in https://github.com/huggingface/optimum-neuron/pull/223

Llama v2 Training
* Llama V2 training support by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/211
* LLama V1 training fix by michaelbenayoun in 211

TGI
* AWS Inferentia2 TGI server by dacorvo in https://github.com/huggingface/optimum-neuron/pull/214

Major bugfixes
* `neuron_parallel_compile`, `ParallelLoader` and Zero-1 fixes for torchneuron 8+ by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/200
* flan-t5 fix: `T5Parallelizer`, `NeuronCacheCallback` and `NeuronHash` refactors by michaelbenayoun in https://github.com/huggingface/optimum-neuron/pull/207
* Fix optimum-cli broke by optimum 1.13.0 release by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/217

Other changes
* Bump Inference APIs to Neuron 2.13 by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/206
* Add log for SD when applying optim attn & pipelines lazy loading by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/208
* Cancel concurreny CIs for inference by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/218
* fix(tgi): typer does not support Union types by dacorvo in https://github.com/huggingface/optimum-neuron/pull/219
* Bump neuron-cc version to 1.18.* by JingyaHuang in https://github.com/huggingface/optimum-neuron/pull/224

**Full Changelog**: https://github.com/huggingface/optimum-neuron/compare/v0.0.10...v0.0.11