Dstack

This is a preview build of the upcoming `0.18.4` release. See below to see what's new.

TPU

One of the major new features in this update is the initial support for Google Cloud TPU.

To request a TPU, you simply need to specify the system architecture of the required TPU prefixed by `tpu-` in `gpu`:


type: task

Oracle Cloud Infrastructure

With the new update, it is now possible to run workloads with your Oracle Cloud Infrastructure (OCI) account. The backend is called `oci` and can be configured as follows:

yaml
projects:
- name: main
backends:
- type: oci
creds:
type: default


The supported credential types include `default` and `client`. In case `default` is used, `dstack` automatically picks the default OCI credentials from `~/.oci/config`.

Just like other backends, `oci` supports dev environments, tasks, and services:

<img src="https://github.com/dstackai/static-assets/blob/main/static-assets/images/oci-vllm-task.gif?raw=true" width="750" />

> [!NOTE]
> Support for spot instances, multi-node tasks, and gateways is coming soon.

Find more documentation on using Oracle Cloud Infrastructure on the [reference page](https://dstack.ai/docs/reference/server/config.yml/#oci_1).

Retry policy

We have reworked how to configure the retry policy and how it is applied to runs. Here's an example:


type: task

commands:
- python train.py

retry:
on_events: [no-capacity]
duration: 2h


Now, if you run such a task, `dstack` will keep trying to find capacity within 2 hours. Once capacity is found, `dstack` will run the task.

The `on_events` property also supports `error` (in case the run fails with an error) and `interruption` (if the run is using a spot instance and it was interrupted).

Previously, `dstack` only allowed retries when spot instances were interrupted.

RunPod

Previously, the `runpod` backend only allowed the use of Docker images with `/bin/bash` or `/bin/sh` as the entrypoint. Thanks to the fix on the RunPod's side, `dstack` now allows the use of any Docker images.

Additionally, the `runpod` backend now also supports spot instances.

GCP

The `gcp` backend now also allows configuring VPCs:

yaml
projects:
- name: main
backends:
- type: gcp

project_id: my-awesome-project
creds:
type: default

vpc_name: my-custom-vpc


The VPC should belong to the same project. If you would like to use a shared VPC from another project, you can also specify `vpc_project_id`.

AWS

Last but not least, for the `aws` backend, it is now possible to configure VPCs for selected regions and use the default VPC in other regions:


projects:
- name: main
backends:
- type: aws
creds:
type: default

vpc_ids:
us-east-1: vpc-0a2b3c4d5e6f7g8h

default_vpcs: true


You just need to set `default_vpcs` to `true`.

Other changes

* Fix reverse server-gateway ssh tunnel by r4victor in https://github.com/dstackai/dstack/pull/1303
* Respect run filters for the `ssh` backend by r4victor in https://github.com/dstackai/dstack/pull/1278
* Support resubmitted runs in `dstack run` attached mode by r4victor in https://github.com/dstackai/dstack/pull/1285
* Do not run jobs on `unreachable` instances by r4victor in https://github.com/dstackai/dstack/pull/1286
* Show job termination reason in `dstack ps -v` by r4victor in https://github.com/dstackai/dstack/pull/1301
* Rename `dstack destroy` to `dstack delete` by r4victor in https://github.com/dstackai/dstack/pull/1275
* Prepare OCI backend for release by jvstme in https://github.com/dstackai/dstack/pull/1308
* [Docs] Improve the documentation of the Pydantic models 1293 by peterschmidt85 in https://github.com/dstackai/dstack/pull/1295
* [Docs] Fix Authorization header by jvstme in https://github.com/dstackai/dstack/pull/1305

OCI

With the new update, it is now possible to run workloads with your Oracle Cloud Infrastructure (OCI) account. The backend is called `oci` and can be configured as follows:

yaml
projects:
- name: main
backends:
- type: oci
creds:
type: default


The supported credential types include `default` and `client`. In case `default` is used, `dstack` automatically picks the default OCI credentials from `~/.oci/config`.

> [!WARNING]
> OCI support does not yet include spot instances, multi-node tasks, and gateways. These features will be added in upcoming updates.

Retry policy

We have reworked how to configure the retry policy and how it is applied to runs. Here's an example:


type: task

commands:
- python train.py

retry:
on_events: [no-capacity]
duration: 2h


Now, if you run such a task, `dstack` will keep trying to find capacity within 2 hours. Once capacity is found, `dstack` will run the task.

The `on_events` property also supports `error` (in case the run fails with an error) and `interruption` (if the run is using a spot instance and it was interrupted).

Previously, `dstack` only allowed retries when spot instances were interrupted.

VPC

GCP

The `gcp` backend now also allows configuring VPCs:

yaml
projects:
- name: main
backends:
- type: gcp

project_id: my-awesome-project
creds:
type: default

vpc_name: my-custom-vpc


The VPC should belong to the same project. If you would like to use a shared VPC from another project, you can also specify `vpc_project_id`.

AWS

Last but not least, for the `aws` backend, it is now possible to configure VPCs for selected regions and use the default VPC in other regions:


projects:
- name: main
backends:
- type: aws
creds:
type: default

vpc_ids:
us-east-1: vpc-0a2b3c4d5e6f7g8h

default_vpcs: true


You just need to set `default_vpcs` to `true`.

Other changes

* Fix reverse server-gateway ssh tunnel by r4victor in https://github.com/dstackai/dstack/pull/1303
* Respect run filters for the `ssh` backend by r4victor in https://github.com/dstackai/dstack/pull/1278
* Support resubmitted runs in `dstack run` attached mode by r4victor in https://github.com/dstackai/dstack/pull/1285
* Do not run jobs on `unreachable` instances by r4victor in https://github.com/dstackai/dstack/pull/1286
* Show job termination reason in `dstack ps -v` by r4victor in https://github.com/dstackai/dstack/pull/1301
* Rename `dstack destroy` to `dstack delete` by r4victor in https://github.com/dstackai/dstack/pull/1275
* Prepare OCI backend for release by jvstme in https://github.com/dstackai/dstack/pull/1308
* [Docs] Improve the documentation of the Pydantic models 1293 by peterschmidt85 in https://github.com/dstackai/dstack/pull/1295

**Full changelog**: https://github.com/dstackai/dstack/compare/0.18.2...0.18.3rc1

> [!WARNING]
> This is an RC build. Please report any bugs to the [issue tracker](https://github.com/dstackai/dstack/issues). The final release is planned for later this week, and the official documentation and examples will be updated then.

On-prem clusters

Network

The `dstack pool add-ssh` command now supports the `--network` argument. Use this argument if you want to use multiple instances that share the same private network as a cluster to run multi-node tasks.

The `--network` argument accepts the IP address range (CIDR) of the private network of the instance.

Example:

shell
dstack pool add-ssh -i ~/.ssh/id_rsa ubuntu141.144.229.104 --network 10.0.0.0/24


Once you've added multiple instances with the same network value, you'll be able to use them as a cluster to run multi-node tasks.

Private subnets

By default, `dstack` uses public IPs for SSH access to running instances, requiring public subnets in the VPC. The new update allows AWS instances to use private subnets instead.

To create instances only in private subnets, set `public_ips` to `false` in the AWS backend settings:


type: aws
creds:
type: default
vpc_ids:
...
public_ips: false


> [!NOTE]
> * Both `dstack server` and the `dstack` CLI should have access to the private subnet to access instances.
> * If you want running instances to access the Internet, the private subnets need to have a NAT gateway.

Gateways

`dstack apply`

Previously, to create or update gateways, one had to use the `dstack gateway create` or `dstack gateway update` commands.
Now, it's possible to define a gateway configuration via YAML and create or update it using the `dstack apply` command.

Example:

yaml
type: gateway
name: example-gateway

backend: gcp
region: europe-west1
domain: example.com


shell
dstack apply -f examples/deployment/gateway.dstack.yml


For now, the `dstack apply` command only supports the `gateway` configuration type. Soon, it will also support `dev-environment`, `task`, and `service`, replacing the `dstack run` command.

The `dstack destroy` command can be used to delete resources.

Private gateways

By default, gateways are deployed using public subnets. Since `0.18.2`, it is now possible to deploy gateways using private subnets. To do this, you need to set `public_ips` to `false` and specify the ARN of a certificate from AWS Certificate Manager.


type: gateway
name: example-gateway

backend: aws
region: eu-west-1
domain: "example.com"

public_ip: false
certificate:
type: acm
arn: "arn:aws:acm:eu-west-1:3515152512515:certificate/3251511125--1241-1224-121251515125"


In this case, `dstack` will deploy the gateway in a private subnet behind a load balancer using the specified certificate.

> [!NOTE]
> Private gateways are currently supported only for AWS.

What's changed

* Support multi-node tasks with `dstack pool add-ssh` instances by TheBits in https://github.com/dstackai/dstack/pull/1189
* Fixed the JSON schema errors by r4victor in https://github.com/dstackai/dstack/pull/1193
* Support spot instances with `runpod` by Bihan in https://github.com/dstackai/dstack/pull/1119
* Speed up AWS VPC validation by r4victor in https://github.com/dstackai/dstack/pull/1196
* [Internal] Optimize `ProjectModel` loading by r4victor in https://github.com/dstackai/dstack/pull/1199
* Support provisioning instances without public IPs on AWS by r4victor in https://github.com/dstackai/dstack/pull/1203
* Minor improvements of `dstack pool add-ssh` by TheBits in https://github.com/dstackai/dstack/pull/1202
* Instances cannot be reused by other users by TheBits in https://github.com/dstackai/dstack/pull/1204
* Do not create AWS instance profile when launching instances by r4victor in https://github.com/dstackai/dstack/pull/1212
* Allow running services without `https` by r4victor in https://github.com/dstackai/dstack/pull/1217
* Implement `dstack apply` for gateways by r4victor in https://github.com/dstackai/dstack/pull/1223
* Support gateways without public IPs on AWS by r4victor in https://github.com/dstackai/dstack/pull/1224
* Support `--network` with `dstack pool add-ssh` by TheBits in https://github.com/dstackai/dstack/pull/1225
* [Internal] Make gateway creation async by r4victor in https://github.com/dstackai/dstack/pull/1236
* Using a more resourceful VM type by default for GCP gateway by r4victor in https://github.com/dstackai/dstack/pull/1237
* Handle properly if the `network` passed to `dstack pool add-ssh` is not correct by TheBits in https://github.com/dstackai/dstack/pull/1233
* Use valid GCP resource names by r4victor in https://github.com/dstackai/dstack/pull/1248
* Always try to restart `dstack-shim.service` with `dstack pool add-ssh` by TheBits in https://github.com/dstackai/dstack/pull/1253
* [Internal] Improve instance processing by r4victor in https://github.com/dstackai/dstack/pull/1251
* Changed `dstack pool remove` to `rm` by muddi900 in https://github.com/dstackai/dstack/pull/1258
* Support gateways behind ALB with ACM certificate by r4victor in https://github.com/dstackai/dstack/pull/1264
* Support IP addresses with `--network` by TheBits in https://github.com/dstackai/dstack/pull/1263
* [Internal] Fix double unlocking when processing runs and instances by r4victor in https://github.com/dstackai/dstack/pull/1268
* Add dstack destroy command and improve dstack apply by r4victor in https://github.com/dstackai/dstack/pull/1271
* Fix instances from pools ignoring regions by r4victor in https://github.com/dstackai/dstack/pull/1272
* Add the `axolotl` example by deep-diver in https://github.com/dstackai/dstack/pull/1187

New Contributors
* muddi900 made their first contribution in https://github.com/dstackai/dstack/pull/1258

**Full Changelog**: https://github.com/dstackai/dstack/compare/0.18.1...0.18.2

On-prem servers

Now you can add your own servers as pool instances:


dstack pool add-ssh -i ~/.ssh/id_rsa ubuntu54.73.155.119


> [!NOTE]
> The server should be pre-installed with CUDA 12.1 and NVIDIA Docker.

Configuration

All `.dstack/profiles.yml` properties now can be specified via run configurations:

yaml
type: dev-environment

ide: vscode

spot_policy: auto
backends: ["aws"]

regions: ["eu-west-1", "eu-west-2"]

instance_types: ["p3.8xlarge", "p3.16xlarge"]

RunPod

The update adds the long-awaited integration with RunPod, a distributed GPU cloud that offers GPUs at affordable prices.

To use RunPod, specify your RunPod API key in `~/.dstack/server/config.yml`:

yaml
projects:
- name: main
backends:
- type: runpod
creds:
type: api_key
api_key: US9XTPDIV8AR42MMINY8TCKRB8S4E7LNRQ6CAUQ9


Once the server is restarted, go ahead and run workloads.

Clusters

Another major change with the update is the ability to run multi-node tasks over an interconnected cluster of instances.

yaml
type: task

nodes: 2

commands:
- git clone https://github.com/r4victor/pytorch-distributed-resnet.git
- cd pytorch-distributed-resnet
- mkdir -p data
- cd data
- wget -c --quiet https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
- tar -xvzf cifar-10-python.tar.gz
- cd ..
- pip3 install -r requirements.txt torch
- mkdir -p saved_models
- torchrun --nproc_per_node=$DSTACK_GPUS_PER_NODE
--node_rank=$DSTACK_NODE_RANK
--nnodes=$DSTACK_NODES_NUM
--master_addr=$DSTACK_MASTER_NODE_IP
--master_port=8008 resnet_ddp.py
--num_epochs 20

resources:
gpu: 1


Currently supported providers for this feature include AWS, GCP, and Azure.

Other

- The `commands` property is now not required for tasks and services if you use an `image` that has a default entrypoint configured.
- The permissions required for using `dstack` with GCP are [more granular](https://dstack.ai/docs/installation/#gcp).

What's changed

* Add `username` filter to `/api/runs/list` by r4victor in https://github.com/dstackai/dstack/pull/1068
* Inherit core models from DualBaseModel by r4victor in https://github.com/dstackai/dstack/pull/967
* Fixed the YAML schema validation for `replicas` by peterschmidt85 in https://github.com/dstackai/dstack/pull/1055
* Improve the `server/config.yml` reference documentation by peterschmidt85 in https://github.com/dstackai/dstack/pull/1077
* Add the `runpod` backend by Bihan in https://github.com/dstackai/dstack/pull/1063
* Support JSON log handler by TheBits in https://github.com/dstackai/dstack/pull/1085
* Added lock to the `terminate_idle_instance` by TheBits in https://github.com/dstackai/dstack/pull/1081
* `dstack init` doesn't work with a remote Git repo by peterschmidt85 in https://github.com/dstackai/dstack/pull/1090
* Minor improvements of `dstack server` output by peterschmidt85 in https://github.com/dstackai/dstack/pull/1088
* Return an error information from `dstack-shim` by TheBits in https://github.com/dstackai/dstack/pull/1061
* Replace `RetryPolicy.limit` to `RetryPolicy.duration` by TheBits in https://github.com/dstackai/dstack/pull/1074
* Make `dstack version` configurable when deploying docs by peterschmidt85 in https://github.com/dstackai/dstack/pull/1095
* `dstack init` doesn't work with a local Git repo by peterschmidt85 in https://github.com/dstackai/dstack/pull/1096
* Fix infinite `create_instance()` on the `cudo` provider by r4victor in https://github.com/dstackai/dstack/pull/1082
* Do not update the `latest` Docker image and YAML scheme for pre-release builds by peterschmidt85 in https://github.com/dstackai/dstack/pull/1099
* Support multi-node tasks by r4victor in https://github.com/dstackai/dstack/pull/1103
* Make `commands` optional in run configurations by jvstme in https://github.com/dstackai/dstack/pull/1104
* Allow the `cudo` backend use non-gpu instances by Bihan in https://github.com/dstackai/dstack/pull/1092
* Make GCP permissions more granular by r4victor in https://github.com/dstackai/dstack/pull/1107

**Full changelog**: https://github.com/dstackai/dstack/compare/0.17.0...0.18.0