Text-generation

Features

- **server**: reduce vram requirements of continuous batching (contributed by njhill)
- **server**: Support BLOOMChat-176B (contributed by njhill)
- **server**: add watermarking tests (contributed by ehsanmok)
- **router**: Adding response schema for compat_generate (contributed by gsaivinay)
- **router**: use number of tokins in batch as input for dynamic batching (co-authored by njhill)
- **server**: improve download and decrease conversion to safetensors RAM requirements
- **server**: optimize flash causal lm decode token
- **server**: shard decode token
- **server**: use cuda graph in logits warping
- **server**: support trust_remote_code
- **tests**: add snapshot testing

Fix

- **server**: use float16
- **server**: fix multinomial implem in Sampling
- **server**: do not use device_map auto on single GPU

Misc

- **docker**: use nvidia base image

New Contributors
* ehsanmok made their first contribution in https://github.com/huggingface/text-generation-inference/pull/248
* gsaivinay made their first contribution in https://github.com/huggingface/text-generation-inference/pull/292
* xyang16 made their first contribution in https://github.com/huggingface/text-generation-inference/pull/343
* oOraph made their first contribution in https://github.com/huggingface/text-generation-inference/pull/359

**Full Changelog**: https://github.com/huggingface/text-generation-inference/compare/v0.6.0...v0.7.0

Features

- **server**: flash attention past key values optimization (contributed by njhill)
- **router**: remove requests when client closes the connection (co-authored by njhill)
- **server**: support quantization for flash models
- **router**: add info route
- **server**: optimize token decode
- **server**: support flash sharded santacoder
- **security**: image signing with cosign
- **security**: image analysis with trivy
- **docker**: improve image size

Fix

- **server**: check cuda capability before importing flash attention
- **server**: fix hf_transfer issue with private repositories
- **router**: add auth token for private tokenizers

Misc

- **rust**: update to 1.69

Features

- **server**: add flash-attention based version of Llama
- **server**: add flash-attention based version of Santacoder
- **server**: support OPT models
- **router**: make router input validation optional
- **docker**: improve layer caching

Fix

- **server**: improve token streaming decoding
- **server**: fix escape charcaters in stop sequences
- **router**: fix NCCL desync issues
- **router**: use buckets for metrics histograms

Fix

- **router**: fix OTLP distributed tracing initialization

Features

- **benchmark**: tui based benchmarking tool
- **router**: Clear cache on error
- **server**: Add mypy-protobuf
- **server**: reduce mlp and attn in one op for flash neox
- **image**: aws sagemaker compatible image

Fix

- **server**: avoid try/except to determine the kind of AutoModel
- **server**: fix flash neox rotary embedding

Features

- **server**: New faster GPTNeoX implementation based on flash attention

Fix

- **server**: fix input-length discrepancy between Rust and Python tokenizers