Dstack

Python 3.13 support

Following a recent [Python 3.13](https://docs.python.org/3/whatsnew/3.13.html#whatsnew313-better-interactive-interpreter) release on October 7, 2024, `dstack` now supports `python: 3.13` in run configurations. `python: 3.8` is still supported but deprecated.

_Note: the `dstack` package itself does not yet work on Python 3.13 due to some limitations in dependencies. We're looking into supporting it as well._

Custom backend tags

You can now define custom tags that `dstack` will assign to all cloud resources it creates including instances and volumes. The tags are defined in the backend configuration:

yaml
type: aws
tags:
company_department: finance
company_project: dstack
company_user: victor
creds:
type: default


Custom tags are supported for AWS, Azure, and GCP.

Improved support of AWS private subnets

Previously, when configuring an AWS backend to use private subnets (`public_ips: false`), `dstack` would require a NAT Gateway. Now `dstack` supports more networking setups that provide outbound internet traffic including NAT Gateway, Transit Gateway, and VPC Peering Connection.

New required permissions

* `dstack` now sets labels on GCP volumes which requires a `compute.disks.setLabels` permission.

Deprecations

* `python: 3.8` in run configurations is deprecated.

What's Changed
* Add created_at to projects and users by r4victor in https://github.com/dstackai/dstack/pull/1857
* Improvements for model details page in the UI by olgenn in https://github.com/dstackai/dstack/pull/1860
* [Bug]: Users logged out after rotating their tokens without seeing tokens by olgenn in https://github.com/dstackai/dstack/pull/1861
* [dind] Move dind processes to a separate cgroup by un-def in https://github.com/dstackai/dstack/pull/1859
* [Docs] Add Docker protip and Docker Compose example by un-def in https://github.com/dstackai/dstack/pull/1858
* Implement custom backend tags by r4victor in https://github.com/dstackai/dstack/pull/1872
* [shim] Remove anonymous volumes along associated container by un-def in https://github.com/dstackai/dstack/pull/1873
* Allow running services without a gateway by jvstme in https://github.com/dstackai/dstack/pull/1869
* [UX]: Resize chat input field based on content 1562 by olgenn in https://github.com/dstackai/dstack/pull/1875
* Collect AMD GPU metrics by r4victor in https://github.com/dstackai/dstack/pull/1877
* [Blog] Monitoring GPU usage and other container metrics by peterschmidt85 in https://github.com/dstackai/dstack/pull/1874
* [Docs] Rename `HUGGING_FACE_HUB_TOKEN` to `HF_TOKEN` by peterschmidt85 in https://github.com/dstackai/dstack/pull/1871
* Support Python 3.13 and deprecate 3.8 in run configurations by jvstme in https://github.com/dstackai/dstack/pull/1878
* Support AWS private subnets with Transit Gateway by r4victor in https://github.com/dstackai/dstack/pull/1881
* Fix collecting metrics from CPU instances by r4victor in https://github.com/dstackai/dstack/pull/1882


**Full Changelog**: https://github.com/dstackai/dstack/compare/0.18.19...0.18.20

This release contains CLI hotfixes for `0.18.18`, including a fix for client backward compatibility and a fix for reported memory usage in `dstack stats`. It's recommended to update the CLI from `0.18.18` to `0.18.19`. The server update from `0.18.18` is not necessary.

What's Changed
* Fix Trl example by Bihan in https://github.com/dstackai/dstack/pull/1851
* Fix dstack client 0.18.18 compatibility with older servers by un-def in https://github.com/dstackai/dstack/pull/1850
* [Docs]: Fix frontend build instructions by jvstme in https://github.com/dstackai/dstack/pull/1849
* Show working set memory in dstack stats by r4victor in https://github.com/dstackai/dstack/pull/1856


**Full Changelog**: https://github.com/dstackai/dstack/compare/0.18.18...0.18.19

Hardware metrics

The CLI introduces a new command, `dstack stats`, which displays real-time hardware metrics for runs, including CPU,
memory, and GPU usage per replica and job.

shell
$ dstack stats hot-frog-1
NAME CPU MEMORY GPU
hot-frog-1 2% 15307MB/49152MB 0 22764MB/24576MB 0% Util


Use the `-w` option to view stats updating every few seconds in the loop.

You can also retrieve the metrics using the [REST API](https://dstack.ai/docs/reference/api/rest/).

Docker inside dstack

Run configurations now have a new optional `privileged` property (equivalent to the
`--privileged` [Docker CLI flag](https://docs.docker.com/engine/containers/run/#runtime-privilege-and-linux-capabilities)).
When it is set to `true`, the run container gets extended privileges, making it possible to use Docker inside
`dstack`.

To use Docker and Docker Compose within `dstack`, set the `image` property
to `dstackai/dind`. Additionally, you must invoke `start-dockerd` as the first command to start the Docker daemon.

yaml
type: task
name: misc-task-dind

image: dstackai/dind
privileged: true

commands:
- start-dockerd
- docker compose up


<details>
<summary>Dev environment example</summary>

yaml
type: dev-environment
name: vscode-dind

image: dstackai/dind
privileged: true
ide: vscode

init:
- start-dockerd

</details>

See more examples in [`examples/misc/docker-compose`](https://github.com/dstackai/dstack/tree/master/examples/misc/docker-compose).

> [!NOTE]
> The `privileged` property is only supported by VM backends (all backends except `runpod`, `vastai`, and `kubernetes`).

What's changed

* [Feature] Track hardware metrics by r4victor in https://github.com/dstackai/dstack/pull/1827
* [Feature] Add `privileged` run property by un-def in https://github.com/dstackai/dstack/pull/1835
* [Feature] Run container as `--privileged` if `DSTACK_DOCKER_PRIVILEGED` by mtaran in https://github.com/dstackai/dstack/pull/1821
* [Feature] Added the `Code` button on the model page in UI by olgenn in https://github.com/dstackai/dstack/pull/1825
* [Feature] Publish `dstackai/dind` image by un-def in https://github.com/dstackai/dstack/pull/1837
* [Bugfix] Fix running NVIDIA NIM images by jvstme in https://github.com/dstackai/dstack/pull/1843
* [Bugfix] Mitigate concurrent `dstack attach` issues by un-def in https://github.com/dstackai/dstack/pull/1816
* [Bugfix] Fixed runs ordering and pagination in UI by un-def in https://github.com/dstackai/dstack/pull/1812
* [Bugfix] Removed dummy Test fleet in UI by priyanshuverma-dev in https://github.com/dstackai/dstack/pull/1819
* [UX] Display the SSH and vscode prompt to after executing `init` commands by fool1280 in https://github.com/dstackai/dstack/pull/1818
* [Blog] AMD MI300X inference benchmark by peterschmidt85 and Bihan in https://github.com/dstackai/dstack/pull/1808
* [Docs] Add AWS EFA support info by un-def in https://github.com/dstackai/dstack/pull/1803
* [Docs] Fixed typos and added minor enhancements by FarukhS52 in https://github.com/dstackai/dstack/pull/1810 and https://github.com/dstackai/dstack/pull/1811
* [Docs] Fixed typos and added minor enhancements by amantyagiprojects in https://github.com/dstackai/dstack/pull/1800
* [Docs] Use correct discord server invite in readme by sravan1946 in https://github.com/dstackai/dstack/pull/1824
* [Docs] Fixed a minor issue in `README.md` by Hacker24a9 in https://github.com/dstackai/dstack/pull/1823
* [Docs] Fixed typos and added minor enhancements VaibhavWakde52 in https://github.com/dstackai/dstack/pull/1813
* [UX] Show run error on the run page in the UI by olgenn in https://github.com/dstackai/dstack/pull/1829
* [UX] Normalize AMD Instinct accelerator names by un-def in https://github.com/dstackai/dstack/pull/1817
* [Bugfix] Fixed the issue with attaching volume to existing instance by un-def in https://github.com/dstackai/dstack/pull/1836
* [Bugfix] Fix GCP config `public_ips` not allowed via the API by r4victor in https://github.com/dstackai/dstack/pull/1840
* [Internal] Don't cache `GpuVendor` in `global var` by un-def in https://github.com/dstackai/dstack/pull/1845
* [Internal] Fix shim tests binding to 0 port by r4victor in https://github.com/dstackai/dstack/pull/1846

New contributors

* FarukhS52 made their first contribution in https://github.com/dstackai/dstack/pull/1810
* amantyagiprojects made their first contribution in https://github.com/dstackai/dstack/pull/1800
* priyanshuverma-dev made their first contribution in https://github.com/dstackai/dstack/pull/1819
* sravan1946 made their first contribution in https://github.com/dstackai/dstack/pull/1824
* Hacker24a9 made their first contribution in https://github.com/dstackai/dstack/pull/1823
* VaibhavWakde52 made their first contribution in https://github.com/dstackai/dstack/pull/1813
* fool1280 made their first contribution in https://github.com/dstackai/dstack/pull/1818
* mtaran made their first contribution in https://github.com/dstackai/dstack/pull/1821

**Full changelog**: https://github.com/dstackai/dstack/compare/0.18.17...0.18.18

AMD

SSH fleets

`dstack` now supports [SSH fleets](https://dstack.ai/docs/concepts/fleets/#ssh-fleets) with AMD GPUs. Hosts should be pre-installed with Docker and AMDGPU-DKMS kernel driver (e.g. via [native package manager](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/native-install/index.html) or [AMDGPU installer](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/install/amdgpu-install.html)).

AWS

Elastic Fabric Adapter (EFA)

`dstack` now automatically enables [AWS EFA](https://aws.amazon.com/hpc/efa/) if it is supported by the instance type, no extra configuration needed. The following EFA-enabled instance types are supported: `p5.48xlarge`, `p4d.24xlarge`, `g4dn.12xlarge`, `g4dn.16xlarge`, `g4dn.8xlarge`, `g4dn.metal`, `g5.12xlarge`, `g5.16xlarge`, `g5.24xlarge`, `g5.48xlarge`, `g5.8xlarge`, `g6.12xlarge`, `g6.16xlarge`, `g6.24xlarge`, `g6.48xlarge`, `g6.8xlarge`, `gr6.8xlarge`.

CLI

Improved `dstack apply` plan

Previously, `dstack apply` showed a plan only for run configurations. Now it shows a plan for all configuration types including fleets, volumes, and gateways. Here's a fleet plan showing configuration parameters and the offers that will be tried for provisioning:


$ dstack apply -f .dstack/confs/fleet.yaml
Project main
User admin
Configuration .dstack/confs/fleet.yaml
Type fleet
Fleet type cloud
Nodes 2
Placement cluster
Backends aws
Resources 2..xCPU, 8GB.., 100GB.. (disk)
Spot policy on-demand

BACKEND REGION INSTANCE RESOURCES SPOT PRICE
1 aws eu-west-1 m5.large 2xCPU, 8GB, 100.0GB (disk) no $0.107
2 aws eu-central-1 m5.large 2xCPU, 8GB, 100.0GB (disk) no $0.115
3 aws eu-west-1 c5.xlarge 4xCPU, 8GB, 100.0GB (disk) no $0.192
...
Shown 3 of 82 offers, $40.9447 max

Fleet my-cluster-fleet does not exist yet.
Create the fleet? [y/n]:


UI

Volumes

Server administrators and regular users can now see volumes in the UI.

What's changed
* Dstack version on UI by olgenn in https://github.com/dstackai/dstack/pull/1742
* Fix restarting gateway connections by jvstme in https://github.com/dstackai/dstack/pull/1746
* Fix Handle KeyboardInterrupt in CLI when getting run plan 1626 by IshuSinghSE in https://github.com/dstackai/dstack/pull/1756
* Add AMD support on on-prem fleets by un-def in https://github.com/dstackai/dstack/pull/1754
* Implement fleet apply plan by r4victor in https://github.com/dstackai/dstack/pull/1765
* chore: update provisioning.py by eltociear in https://github.com/dstackai/dstack/pull/1768
* Fix use all available runpod regions by default by IshuSinghSE in https://github.com/dstackai/dstack/pull/1757
* Implement apply plan for gateways and volumes by r4victor in https://github.com/dstackai/dstack/pull/1774
* Fix connection to ssh instance on non-standard ssh port by un-def in https://github.com/dstackai/dstack/pull/1766
* Fix docker SSH commands by un-def in https://github.com/dstackai/dstack/pull/1771
* Add Llama3.2 Vision Model Example by Bihan in https://github.com/dstackai/dstack/pull/1770
* Disable backend autoconfig via default creds by r4victor in https://github.com/dstackai/dstack/pull/1778
* Set backends requests timeouts by r4victor in https://github.com/dstackai/dstack/pull/1793
* Add UI for volumes 1683 by olgenn in https://github.com/dstackai/dstack/pull/1785
* UI for volumes 1683 by olgenn in https://github.com/dstackai/dstack/pull/1795
* [Docs] Add AMD GPU info to ssh fleets section by un-def in https://github.com/dstackai/dstack/pull/1779
* [shim] Use DockerRootDir to detect free disk space by un-def in https://github.com/dstackai/dstack/pull/1802
* Add AWS EFA support by un-def in https://github.com/dstackai/dstack/pull/1801

New contributors
* IshuSinghSE made their first contribution in https://github.com/dstackai/dstack/pull/1756
* eltociear made their first contribution in https://github.com/dstackai/dstack/pull/1768

**Full changelog**: https://github.com/dstackai/dstack/compare/0.18.16...0.18.17

New versioning policy

Starting with this release, `dstack` adopts a new versioning policy to provide better server and client backward compatibility and improve the upgrading experience. `dstack` continues to follow semver versioning scheme (`{major}.{minor}.{patch}`) with the following principles:

* The server backward compatibility is maintained across all minor and patch releases. The specific features can be removed but the removal is preceded with deprecation warnings for several minor releases. This means you can use older client versions with newer server versions.
* The client backward compatibility is maintained across patch releases. A new minor release indicates that the release breaks client backward compatibility. This means you don't need to update the server when you update the client to a new patch release. Still, upgrading a client to a new minor version requires upgrading the server too.

Perviously, `dstack` never guaranteed client backward compatibility, so you had to always update the server when updating the client. The new versioning policy makes the client and server upgrading more flexible.

**Note:** The new policy only takes affect after both the clients and the server are upgraded to `0.18.16`. The `0.18.15` server still won't work with newer clients.

dstack attach

The CLI gets a new [`dstack attach` ](https://dstack.ai/docs/reference/cli/#dstack-attach)command that allows attaching to a run. It establishes the SSH tunnel, forwards ports, and streams run logs in real time:

bash
✗ dstack attach silent-panther-1
Attached to run silent-panther-1 (replica=0 job=0)
Forwarded ports (local -> remote):
- localhost:7860 -> 7860
To connect to the run via SSH, use `ssh silent-panther-1`.
Press Ctrl+C to detach...


This command is a replacement for `dstack logs --attach` with major improvements and bugfixes.

CloudWatch-related bugfixes

The releases includes several important bugfixes for `CloudWatchLogStorage`. We strongly recommend upgrading the `dstack` server if it's configured to store logs in CloudWatch.

Deprecations

* `dstack logs --attach` is deprecated in favor of `dstack attach` and may be removed in the following minor releases.

What's Changed
* Check client-server compatibility according to new versioning policy by r4victor in https://github.com/dstackai/dstack/pull/1730
* [runner] fix MonotonicTimestamp by un-def in https://github.com/dstackai/dstack/pull/1728
* Gateway-in-server early prototype by jvstme in https://github.com/dstackai/dstack/pull/1718
* Implement dstack attach command by r4victor in https://github.com/dstackai/dstack/pull/1733
* Respect CloudWatch timestamp constraints by un-def in https://github.com/dstackai/dstack/pull/1732
* Add AMD examples with vLLM, Axolotl and Trl by Bihan in https://github.com/dstackai/dstack/pull/1693
* dstack-proxy naming tweaks by jvstme in https://github.com/dstackai/dstack/pull/1734
* Fix Failed to attach via Python API by r4victor in https://github.com/dstackai/dstack/pull/1739
* Support calling RunCollection.get_plan() without repo by r4victor in https://github.com/dstackai/dstack/pull/1741


**Full Changelog**: https://github.com/dstackai/dstack/compare/0.18.15...0.18.16

Cluster placement groups

Instances of AWS cluster fleets are now provisioned into [cluster placement groups](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/placement-groups.html) for better connectivity. For example, when you create this fleet:

yaml
type: fleet
name: my-cluster-fleet
nodes: 4
placement: cluster
backends: [aws]


`dstack` will automatically create a cluster placement group and use it to provision the instances.

On-prem and VM-based fleets improvements

* All available Nvidia [driver capabilities](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/1.16.0/docker-specialized.html#driver-capabilities) are now requested by default, which makes it possible to run GPU workloads requiring OpenGL/Vulkan/RT/Video Codec SDK libraries. (https://github.com/dstackai/dstack/pull/1714)
* Automatic container cleanup. Previously, when the run completed, either successfully or due to an error, its container was not deleted, which led to ever-increasing storage consumption. Now, only the last stopped container is preserved and is available until the next run is completed. (https://github.com/dstackai/dstack/pull/1706)

Major bug fixes
* Fixed a bug where under some conditions logs wouldn't be uploaded to CloudWatch Logs due to size limits. (https://github.com/dstackai/dstack/pull/1712)
* Fixed a bug that prevented running services on on-prem instances. (https://github.com/dstackai/dstack/pull/1716)

Changelog
* Fix cli connection issue with TPU by Bihan in https://github.com/dstackai/dstack/pull/1705
* Rename `--default` to `--yes` and `no-default` to `--no` in `dstack config` and `dstack server` by peterschmidt85 in https://github.com/dstackai/dstack/pull/1709
* [CI] Fix shim/runner release versions by un-def in https://github.com/dstackai/dstack/pull/1704
* Document run diagnostic logs by r4victor in https://github.com/dstackai/dstack/pull/1710
* [shim] Add old container cleanup routine by un-def in https://github.com/dstackai/dstack/pull/1706
* Write events to CloudWatch in batches by un-def in https://github.com/dstackai/dstack/pull/1712
* [shim] Request all Nvidia driver capabilities by un-def in https://github.com/dstackai/dstack/pull/1714
* Added showing dstack version on the UI by olgenn in https://github.com/dstackai/dstack/pull/1717
* Add missing project SSH key to on-prem instances by un-def in https://github.com/dstackai/dstack/pull/1716
* Simplify handling missing `GatewayConfiguration` by jvstme in https://github.com/dstackai/dstack/pull/1724
* [shim] Fix container logs processing by un-def in https://github.com/dstackai/dstack/pull/1721
* Support AWS placement groups for cluster fleets by r4victor in https://github.com/dstackai/dstack/pull/1725


**Full Changelog**: https://github.com/dstackai/dstack/compare/0.18.14...0.18.15