Installation
$ pip/pip3 install mixtral.py
Usage
Run the model
python
import mixtral
mixtral = Mixtral()
print("Running the model:")
print(mixtral.generate_text("..."))
In half-precision
python
import mixtral
print("Running the model in half precision:")
mixtral_half_precision = Mixtral(use_half_precision=True)
print(mixtral_half_precision.generate_text("..."))
Lower precision using (8-bit & 4-bit) using `bitsandbytes`
python
import mixtral
print("Running the model in lower precision using (8-bit & 4-bit) using bitsandbytes:")
mixtral_4bit = Mixtral(load_in_4bit=True)
print(mixtral_4bit.generate_text("..."))
Load the model with Flash Attention 2
python
import mixtral
print("Load the model with Flash Attention 2:")
mixtral_flash_attention_2 = Mixtral(use_flash_attention_2=True)
print(mixtral_flash_attention_2.generate_text("..."))
Hardware Requirements
- minimum 100GB GPU RAM ([Mistral AI](https://docs.mistral.ai/models/))
- Hardware requirements after fine-tuning can be found in the discussion [here](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1/discussions/3)
Licenses
- mixtral.py is under the [GNU General Public License v3](https://github.com/ibnaleem/mixtral.py/blob/main/LICENSE)
- Mixtral 8x7b is under the [Apache 2.0 License](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1)
Verifying Signatures
Import PGP Key into Keyring
gpg --keyserver 185.125.188.27 --recv-keys 20247EC023F2769E66181C0F581B4A2A862BBADE
or
wget https://github.com/ibnaleem/ibnaleem/blob/main/public_key.asc
Download Signature
The signatures (.asc and .sig) can be found in the [/sig directory.](https://github.com/ibnaleem/mixtral.py/tree/main/sig) Download either of them. Open [an issue](https://github.com/ibnaleem/mixtral.py/issues) with the title "invalid signature/wrong signature/corrupt signature" for any issues regarding my signatures.
Sign My Key
gpg --sign-key 20247EC023F2769E66181C0F581B4A2A862BBADE
gpg --send-keys 20247EC023F2769E66181C0F581B4A2A862BBADE
If you cannot upload your signature, see ["*gpg: keyserver receive failed: No route to host*"](https://stackoverflow.com/questions/54801274/gpg-keyserver-receive-failed-no-route-to-host-stack-overflow)
Verify
gpg --verify mixtral.py.asc mixtral.py
Desired output:
gpg: Signature made Tue 6 Feb 10:27:34 2024 GMT
gpg: using RSA key 20247EC023F2769E66181C0F581B4A2A862BBADE
gpg: Good signature from "Ibn Aleem <ibnaleemoutlook.com>" [ultimate]