Ctransformers

Changes

- Handle special tokens properly while tokenizing
- Handle incomplete UTF-8 multi-byte characters while generating text
- Increase buffer size in `tokenize()` for BOS token
- Update ggml and llama.cpp

Changes

- Add support for the new k-quantization formats
- Update GGML

Changes

- Add support for StarChat special tokens
- Add `context_length` parameter support for MPT
- Add low-level API to get `eos_token_id`, `vocab_size`, `context_length`

Changes

- Add `local_files_only` option
- Fix utf-8 decode error

Changes

- Use smallest model file in a Hugging Face Hub model repo if `model_file` is not specified

Changes

- Add cuBLAS support for LLaMA models
- Simplify installing and building from source