Dstack

We've upgraded the default Docker image's CUDA drivers to 12.1 (for better compatibility with modern libraries).

Mixtral 8x7B

Lastly, and most importantly, we've added the [example](https://dstack.ai/examples/mixtral) showing how to deploy Mixtral 8x7B as a service.

env:
- MODEL=NousResearch/Llama-2-7b-chat-hf
commands:
- pip install vllm
- python -m vllm.entrypoints.openai.api_server --model $MODEL --port 8000
port: 8000

resources:
gpu: 24GB

model:
format: openai
type: chat
name: NousResearch/Llama-2-7b-chat-hf


What's changed

* Configuration resources & ranges by Egor-S in https://github.com/dstackai/dstack/pull/844
* Range.__str__ always returns a string by Egor-S in https://github.com/dstackai/dstack/pull/845
* Add infinity example by deep-diver in https://github.com/dstackai/dstack/pull/847
* error in documentation: use --url instead of --server by promsoft in https://github.com/dstackai/dstack/pull/852
* Support authorization on the gateway by Egor-S in https://github.com/dstackai/dstack/pull/851
* Implement Kubernetes backend by r4victor in https://github.com/dstackai/dstack/pull/853
* Add gpu support for kubernetes by r4victor in https://github.com/dstackai/dstack/pull/856
* Resources parse and store by Egor-S in https://github.com/dstackai/dstack/pull/857
* Use python3.11 in generate-json-schema by r4victor in https://github.com/dstackai/dstack/pull/859
* Implement OpenAI to OpenAI adapter for gateway by Egor-S in https://github.com/dstackai/dstack/pull/860

New contributors

* deep-diver made their first contribution in https://github.com/dstackai/dstack/pull/847
* promsoft made their first contribution in https://github.com/dstackai/dstack/pull/852

**Full Changelog**: https://github.com/dstackai/dstack/compare/0.14.0...0.15.0

max_duration: 1d


New examples :fire::fire:

Thanks to the contribution from deep-diver, we got two new examples:

* [Alignment Handbook](https://github.com/dstackai/dstack/blob/master/examples/fine-tuning/alignment-handbook/README.md)
* [Axolotl](https://github.com/dstackai/dstack/blob/master/examples/fine-tuning/axolotl/README.md)

Other

- Configuring VPCs using their IDs (via `vpc_ids` in `server/config.yml`)
- Support for global profiles (via `~/.dstack/profiles.yml`)
- Updated the default environment variables (`DSTACK_RUN_NAME`, `DSTACK_GPUS_NUM`, `DSTACK_NODES_NUM`, `DSTACK_NODE_RANK`, and `DSTACK_MASTER_NODE_IP`)
- It’s now possible to use NVIDIA `A10` GPU on Azure
- More granular permissions for Azure

What's changed

* Fix server freeze on terminate instance by jvstme in https://github.com/dstackai/dstack/pull/1132
* Support profile params in run configurations by r4victor in https://github.com/dstackai/dstack/pull/1131
* Support global `.dstack/profiles.yml` by r4victor in https://github.com/dstackai/dstack/pull/1134
* Fix `No such profile: None` when missing `.dstack/profiles.yml` by r4victor in https://github.com/dstackai/dstack/pull/1135
* Make Azure permissions more granular by r4victor in https://github.com/dstackai/dstack/pull/1139
* Validate min disk size by r4victor in https://github.com/dstackai/dstack/pull/1146
* Fix unexpected error if system Python version is unknown by r4victor in https://github.com/dstackai/dstack/pull/1147
* Add request timeouts to prevent code freezes by jvstme in https://github.com/dstackai/dstack/pull/1140
* Refactor backends to wait for instance IP address outside `run_job/create_instance` by r4victor in https://github.com/dstackai/dstack/pull/1149
* Fix provisioning Azure instances with A10 GPU by jvstme in https://github.com/dstackai/dstack/pull/1150
* [Internal] Move `packer` -> `scripts/packer` by jvstme in https://github.com/dstackai/dstack/pull/1153
* Added the ability of adding own instances by TheBits in https://github.com/dstackai/dstack/pull/1115
* An issue with the `executor_error` check being falsely positive by TheBits in https://github.com/dstackai/dstack/pull/1160
* Make user project quota configurable by r4victor in https://github.com/dstackai/dstack/pull/1161
* Configure CORS headers on gateway by r4victor in https://github.com/dstackai/dstack/pull/1166
* Allow to configure AWS `vpc_ids` by r4victor in https://github.com/dstackai/dstack/pull/1170
* [Internal] Show dstack version in Sentry issues by jvstme in https://github.com/dstackai/dstack/pull/1167
* Fix `KeyError: 'IpPermissions'` when using AWS by jvstme in https://github.com/dstackai/dstack/pull/1169
* Create public ssh key is it not exist in `dstack pool add-ssh` by TheBits in https://github.com/dstackai/dstack/pull/1173
* Fixed is the environment file upload by TheBits in https://github.com/dstackai/dstack/pull/1175
* Updated shim status processing by TheBits in https://github.com/dstackai/dstack/pull/1174
* Fix bugs in `dstack pool add-ssh` by TheBits in https://github.com/dstackai/dstack/pull/1178
* Fix Cudo Create VM response error by Bihan in https://github.com/dstackai/dstack/pull/1179
* Implement API for configuring backends via yaml by r4victor in https://github.com/dstackai/dstack/pull/1181
* Allow running gated models with `HUGGING_FACE_HUB_TOKEN` by r4victor in https://github.com/dstackai/dstack/pull/1184
* Pass all dstack runner envs as `DSTACK_*` by r4victor in https://github.com/dstackai/dstack/pull/1185
* Improve the retries in the get_host_info and get_shim_healthcheck by TheBits in https://github.com/dstackai/dstack/pull/1183
* Example/h4alignment handbook by deep-diver in https://github.com/dstackai/dstack/pull/1180
* The deploy is launched in ThreadPoolExecutor by TheBits in https://github.com/dstackai/dstack/pull/1186

**Full Changelog**: https://github.com/dstackai/dstack/compare/0.18.0...0.18.1rc2

What's changed

* Support multi-node tasks with `dstack pool add-ssh` instances by TheBits in https://github.com/dstackai/dstack/pull/1189
* Fixed the JSON schema errors by r4victor in https://github.com/dstackai/dstack/pull/1193
* Support spot instances with `runpod` by Bihan in https://github.com/dstackai/dstack/pull/1119
* Speed up AWS VPC validation by r4victor in https://github.com/dstackai/dstack/pull/1196
* [Internal] Optimize `ProjectModel` loading by r4victor in https://github.com/dstackai/dstack/pull/1199
* Support provisioning instances without public IPs on AWS by r4victor in https://github.com/dstackai/dstack/pull/1203
* Minor improvements of `dstack pool add-ssh` TheBits in https://github.com/dstackai/dstack/pull/1202
* The instance cannot be reused by another user by TheBits in https://github.com/dstackai/dstack/pull/1204
* Do not create AWS instance profile when launching instances by r4victor in https://github.com/dstackai/dstack/pull/1212
* Add the `axolotl` example by deep-diver in https://github.com/dstackai/dstack/pull/1187

**Full changelog**: https://github.com/dstackai/dstack/compare/0.18.1...0.18.2rc1

On-prem servers

Now you can add your own servers as pool instances:


dstack pool add-ssh -i ~/.ssh/id_rsa ubuntu54.73.155.119


> [!NOTE]
> The server should be pre-installed with CUDA 12.1 and NVIDIA Docker.

Configuration

All `.dstack/profiles.yml` properties now can be specified via run configurations:

yaml
type: dev-environment

ide: vscode

spot_policy: auto
backends: ["aws"]

regions: ["eu-west-1", "eu-west-2"]

instance_types: ["p3.8xlarge", "p3.16xlarge"]

RunPod

The update adds the long-awaited integration with RunPod, a distributed GPU cloud that offers GPUs at affordable prices.

To use RunPod, specify your RunPod API key in `~/.dstack/server/config.yml`:

yaml
projects:
- name: main
backends:
- type: runpod
creds:
type: api_key
api_key: US9XTPDIV8AR42MMINY8TCKRB8S4E7LNRQ6CAUQ9


Once the server is restarted, go ahead and run workloads.

Clusters

Another major change with the update is the ability to run multi-node tasks over an interconnected cluster of instances.

yaml
type: task

nodes: 2

commands:
- git clone https://github.com/r4victor/pytorch-distributed-resnet.git
- cd pytorch-distributed-resnet
- mkdir -p data
- cd data
- wget -c --quiet https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
- tar -xvzf cifar-10-python.tar.gz
- cd ..
- pip3 install -r requirements.txt torch
- mkdir -p saved_models
- torchrun --nproc_per_node=$DSTACK_GPUS_PER_NODE
--node_rank=$DSTACK_NODE_RANK
--nnodes=$DSTACK_NODES_NUM
--master_addr=$DSTACK_MASTER_NODE_IP
--master_port=8008 resnet_ddp.py
--num_epochs 20

resources:
gpu: 1


Currently supported providers for this feature include AWS, GCP, and Azure.

Other

- The `commands` property is now not required for tasks and services if you use an `image` that has a default entrypoint configured.
- The permissions required for using `dstack` with GCP are [more granular](https://dstack.ai/docs/installation/#gcp).

What's changed

* Add `username` filter to `/api/runs/list` by r4victor in https://github.com/dstackai/dstack/pull/1068
* Inherit core models from DualBaseModel by r4victor in https://github.com/dstackai/dstack/pull/967
* Fixed the YAML schema validation for `replicas` by peterschmidt85 in https://github.com/dstackai/dstack/pull/1055
* Improve the `server/config.yml` reference documentation by peterschmidt85 in https://github.com/dstackai/dstack/pull/1077
* Add the `runpod` backend by Bihan in https://github.com/dstackai/dstack/pull/1063
* Support JSON log handler by TheBits in https://github.com/dstackai/dstack/pull/1085
* Added lock to the `terminate_idle_instance` by TheBits in https://github.com/dstackai/dstack/pull/1081
* `dstack init` doesn't work with a remote Git repo by peterschmidt85 in https://github.com/dstackai/dstack/pull/1090
* Minor improvements of `dstack server` output by peterschmidt85 in https://github.com/dstackai/dstack/pull/1088
* Return an error information from `dstack-shim` by TheBits in https://github.com/dstackai/dstack/pull/1061
* Replace `RetryPolicy.limit` to `RetryPolicy.duration` by TheBits in https://github.com/dstackai/dstack/pull/1074
* Make `dstack version` configurable when deploying docs by peterschmidt85 in https://github.com/dstackai/dstack/pull/1095
* `dstack init` doesn't work with a local Git repo by peterschmidt85 in https://github.com/dstackai/dstack/pull/1096
* Fix infinite `create_instance()` on the `cudo` provider by r4victor in https://github.com/dstackai/dstack/pull/1082
* Do not update the `latest` Docker image and YAML scheme for pre-release builds by peterschmidt85 in https://github.com/dstackai/dstack/pull/1099
* Support multi-node tasks by r4victor in https://github.com/dstackai/dstack/pull/1103
* Make `commands` optional in run configurations by jvstme in https://github.com/dstackai/dstack/pull/1104
* Allow the `cudo` backend use non-gpu instances by Bihan in https://github.com/dstackai/dstack/pull/1092
* Make GCP permissions more granular by r4victor in https://github.com/dstackai/dstack/pull/1107

**Full changelog**: https://github.com/dstackai/dstack/compare/0.17.0...0.18.0