Dstack

TPU

Trillium (`v6e`)

`dstack` adds support for the latest Trillium TPU (`v6e`), which became [generally available](https://cloud.google.com/blog/products/compute/trillium-tpu-is-ga) in GCP on December 12th. The new TPU generation doubles the TPU memory and bursts performance, supporting larger workloads.

Resources

`dstack` now includes CPU, RAM, and TPU memory in Google Cloud TPU offers:

shell
$ dstack apply --gpu tpu

BACKEND REGION INSTANCE RESOURCES SPOT PRICE
1 gcp europe-west4 v5litepod-1 24xCPU, 48GB, 1xv5litepod-1 (16GB), 100.0GB (disk) no $1.56
2 gcp europe-west4 v6e-1 44xCPU, 176GB, 1xv6e-1 (32GB), 100.0GB (disk) no $2.97
3 gcp europe-west4 v2-8 96xCPU, 334GB, 1xv2-8 (64GB), 100.0GB (disk) no $4.95


Volumes

By default, TPU VMs contain a 100GB boot disk, and its size cannot be changed. Now, you can add more storage using [Volumes](https://dstack.ai/docs/concepts/volumes/).

Gateways

In this update, we've greatly refactored [Gateways](https://dstack.ai/docs/concepts/gateways/), improving their reliability and fixing several bugs.

> [!NOTE]
If you are running [multiple replicas](https://dstack.ai/docs/guides/server-deployment/#can-i-run-multiple-replicas-of-the-dstack-server) of the `dstack` server, ensure all replicas are upgraded promptly. Leaving some replicas on an older version may prevent them from creating or deleting services and could result in minor errors in their logs.

> [!WARNING]
Ensure you update to [0.18.33](https://github.com/dstackai/dstack/releases/tag/0.18.33), which includes critical hot-fixes for important issues.

What's changed
* [`dstack-shim`] Rework resource management by un-def in https://github.com/dstackai/dstack/pull/2093
* [Gateways] Restore `dstack-proxy` state on gateway restarts by jvstme in https://github.com/dstackai/dstack/pull/2119
* [TPU] Support TPU v6e by r4victor in https://github.com/dstackai/dstack/pull/2124
* [UI] Updated `Backend config` Info section by peterschmidt85 in https://github.com/dstackai/dstack/pull/2125
* [UI] It's not possible to manage fleets by olgenn in https://github.com/dstackai/dstack/pull/2126
* [UI] Improvements by olgenn in https://github.com/dstackai/dstack/pull/2127
* [Gateways] Add migration from `state.json` on gateways by jvstme in https://github.com/dstackai/dstack/pull/2128
* [Volumes] Forbid deleting backends with active instances or volumes by r4victor in https://github.com/dstackai/dstack/pull/2131
* [TPU] Fix backward compatibility with new TPUs by r4victor in https://github.com/dstackai/dstack/pull/2138
* Update `gpuhunt` to `0.0.17` by r4victor in https://github.com/dstackai/dstack/pull/2139
* [Docs] Improve docs by r4victor in https://github.com/dstackai/dstack/pull/2135
* [Gateways] Fix certbot process getting stuck in `dstack-proxy` by jvstme in https://github.com/dstackai/dstack/pull/2143
* [Gateways] Run `dstack-proxy` on gateways by jvstme in https://github.com/dstackai/dstack/pull/2136
* [Volumes] Support volumes for TPUs by r4victor in https://github.com/dstackai/dstack/pull/2144
* [Gateways] Optimize `dstack-gateway` installation time by jvstme in https://github.com/dstackai/dstack/pull/2146
* [Gateways] Fix OpenAI endpoint on Kubernetes gateways by jvstme in https://github.com/dstackai/dstack/pull/2147

**Full changelog**: https://github.com/dstackai/dstack/compare/0.18.31...0.18.32

GCP

Running VMs on behalf of a service account

Like all major clouds, GCP supports running a VM on behalf of a managed identity using a service account. Now you can assign a service account to a GCP VM with `dstack` by specifying the `vm_service_account` property in the GCP config:

yaml
type: gcp
project_id: myproject
vm_service_account: samyproject.iam.gserviceaccount.com
creds:
type: default


Assigning a service account to a VM can be used to access GCP resources from within runs. Another use case is using firewall rules that rely on the service account as the target. Such rules are typical for Shared VPC setups when admins of a host project can create firewall rules for service projects based on their service accounts.

Volumes

Creating user home directory automatically

Following support for non-root users in Docker images, `dstack` improves handling of users' home directories. Most importantly, the `HOME` environment variable is set according to `/etc/passwd`, and the home directory is created automatically if it does not exist.

The update opens up new possibilities including the use of an empty volume for `/home`:

yaml
type: dev-environment
ide: vscode
image: ubuntu
user: ubuntu
volumes:
- volume-aws:/home


AWS volumes with non-Nitro instances

`dstack` users previously reported AWS Volumes not working with some instance types. This is now fixed and tested for all instance types supported by `dstack` including older Xen-based instances like the P3 family.

Deprecations

* The `home_dir` and `setup` parameters in run configurations have been deprecated. If you're using `setup`, move `setup` commands to the top of `init`.

What's changed

* [`dstack-shim`] Implement multi-task state by un-def in https://github.com/dstackai/dstack/pull/2078
* [AWS] Support AWS volumes for Xen-based instances by r4victor in https://github.com/dstackai/dstack/pull/2088
* Handle empty user when processing image manifest by un-def in https://github.com/dstackai/dstack/pull/2090
* [Docs] Move Reference to a separate page for more space and better st… by peterschmidt85 in https://github.com/dstackai/dstack/pull/2092
* Init VirtualRepo when `--no-repo` specified by r4victor in https://github.com/dstackai/dstack/pull/2098
* [Docs] Add missing backends docs reference by r4victor in https://github.com/dstackai/dstack/pull/2099
* [gateways] Support gateway features in `dstack-proxy` by jvstme in https://github.com/dstackai/dstack/pull/2087
* [Docs] Add `Repos` page inside `Concepts` to explain how repos work 2096 by peterschmidt85 in https://github.com/dstackai/dstack/pull/2097
* [GCP] Allow specifying `vm_service_account` in GCP config by r4victor in https://github.com/dstackai/dstack/pull/2110
* [`dstack-shim`] Create user home directory if it doesn't exist by un-def in https://github.com/dstackai/dstack/pull/2109
* [Tests] Disallow remote network connections in tests by un-def in https://github.com/dstackai/dstack/pull/2111
* [Docs] Add Developers page featuring community links, ambassador program, contributing links, etc 2103 by peterschmidt85 in https://github.com/dstackai/dstack/pull/2104
* [Docs] Refactor the reference guide 2112 by peterschmidt85 in https://github.com/dstackai/dstack/pull/2113
* [Tests] Support tests that access db from a new thread by r4victor in https://github.com/dstackai/dstack/pull/2116
* [Deprecation] Deprecate `home_dir` and `setup` by un-def in https://github.com/dstackai/dstack/pull/2115


**Full changelog**: https://github.com/dstackai/dstack/compare/0.18.30...0.18.31

AWS Capacity Reservations and Capacity Blocks

`dstack` now allows provisioning AWS instances using [Capacity Reservations and Capacity Blocks](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/capacity-reservation-overview.html). Given a `CapacityReservationId`, you can specify it in a fleet or a run configuration:

yaml
type: fleet
nodes: 1
name: my-cr-fleet
reservation: cr-0f45ab39cd64a1cee


The instance will use the reserved capacity, so as long as you have enough, the provisioning is guaranteed to succeed.

Non-root users in Docker images

Previously, `dstack` always executed the workload as `root`, ignoring the user property set in the image. Now, `dstack` executes the workload with the default image user, and you can override it with a new [`user`](https://dstack.ai/docs/reference/dstack.yml/task/#user) property:

yaml
type: task
image: nvcr.io/nim/meta/llama-3.1-8b-instruct
user: nim


The format of the `user` property is the [same as Docker uses](https://docs.docker.com/engine/containers/run/#user): `username[:groupname]`, `uid[:gid]`, and so on.

Improved `dstack apply` and repos UX

Previously, `dstack apply` used the current directory as the repo that's made available within the run at `/workflow`. The directory had to be initialized with `dstack init` before running `dstack apply`.

Now you can pass `--repo` to `dstack apply`. It can be a path to a local directory or a remote Git repo URL. The specified repo will be available within the run at `/workflow`. You can also specify `--no-repo` if the run doesn't need any repo. With `--repo` or `--no-repo` specified, you don't need to run `dstack init`:

bash
$ dstack apply -f task.dstack.yaml --repo .
$ dstack apply -f task.dstack.yaml --repo ../parent_dir
$ dstack apply -f task.dstack.yaml --repo https://github.com/dstackai/dstack.git
$ dstack apply -f task.dstack.yaml --no-repo


Specifying `--repo` explicitly can be useful when running `dstack apply` from scripts, pipelines, or CI. `dstack init` stays relevant for use cases when you work with `dstack apply` interactively and want to set up the repo to work with once.

Lightweight `pip install dstack`

`pip install dstack` used to install all the `dstack` server dependencies. Now `pip install dstack` installs only the CLI and Python API, which is optimal for use cases when a remote dstack server is used. You can do `pip install "dstack[server]"` to install the server or do `pip install "dstack[all]"` to install the server with all backends supported.

Breaking changes

* `pip install dstack` no longer install the server dependencies. If you relied on it to install the server, ensure you use `pip install "dstack[server]"` or `pip install "dstack[all]"`.

What's Changed
* [chore]: Move `run_async` to `_internal/utils` by jvstme in https://github.com/dstackai/dstack/pull/2057
* Move server deps to dstack[server] extra by r4victor in https://github.com/dstackai/dstack/pull/2058
* Add `user` property to run configurations by un-def in https://github.com/dstackai/dstack/pull/2055
* [Blog] Exploring inference memory saturation effect: H100 vs MI300x by peterschmidt85 in https://github.com/dstackai/dstack/pull/2061
* [Internal]: Fix building docs in CI by jvstme in https://github.com/dstackai/dstack/pull/2063
* [chore]: Drop unused gateway-related runner code by jvstme in https://github.com/dstackai/dstack/pull/2062
* [shim] Clean up and document API by un-def in https://github.com/dstackai/dstack/pull/2060
* Improve RESP API docs by r4victor in https://github.com/dstackai/dstack/pull/2064
* Allow underscores in custom GCP tags by r4victor in https://github.com/dstackai/dstack/pull/2065
* Make repo optional when submitting runs via HTTP API by r4victor in https://github.com/dstackai/dstack/pull/2066
* Fix changing configuration type with dstack apply by r4victor in https://github.com/dstackai/dstack/pull/2070
* Fix instances stuck in busy status by r4victor in https://github.com/dstackai/dstack/pull/2071
* [Minor] If errors should be passed silently, then in pythonic way by dimitriillarionov in https://github.com/dstackai/dstack/pull/2075
* AWS Capacity Reservation support by solovyevt in https://github.com/dstackai/dstack/pull/1977
* [Blog] Beyond Kubernetes: 2024 recap and what's next for AI infra by peterschmidt85 in https://github.com/dstackai/dstack/pull/2074
* Fix `reservation` property backward compatibility by un-def in https://github.com/dstackai/dstack/pull/2077
* Fix ~/.ssh write permissions check by r4victor in https://github.com/dstackai/dstack/pull/2079
* Fix errors exit codes in dstack apply by r4victor in https://github.com/dstackai/dstack/pull/2081
* Fix RESERVATIONS display in fleets table by r4victor in https://github.com/dstackai/dstack/pull/2082
* Support --repo, --no-repo, and autoinit in dstack apply by r4victor in https://github.com/dstackai/dstack/pull/2080
* Support AWS partitioned volumes by r4victor in https://github.com/dstackai/dstack/pull/2084
* [shim] Update OpenAPI doc by un-def in https://github.com/dstackai/dstack/pull/2085

New Contributors
* dimitriillarionov made their first contribution in https://github.com/dstackai/dstack/pull/2075
* solovyevt made their first contribution in https://github.com/dstackai/dstack/pull/1977

**Full Changelog**: https://github.com/dstackai/dstack/compare/0.18.29...0.18.30

Support `internal_ip` for SSH fleet clusters

It's now possible to specify instance IP addresses used for communication inside SSH fleet clusters using the `internal_ip` property:

yaml
type: fleet
name: my-ssh-fleet
placement: cluster
ssh_config:
user: ubuntu
identity_file: ~/.ssh/dstack/key.pem
hosts:
- hostname: "3.79.203.200"
internal_ip: "172.17.0.1"
- hostname: "18.184.67.100"
internal_ip: "172.18.0.2"


If `internal_ip` is not specified, `dstack` automatically detects internal IPs by inspecting network interfaces. This works when all instances have IPs belonging to the same subnet and are accessible on those IPs. The explicitly specified `internal_ip` enables networking configurations when the instances are accessible on IPs that do not belong to the same subnet.

UX enhancements for `dstack apply`

The `dstack apply` command gets many improvements including more concise and consistent output and better error reporting. When applying run configurations, `dstack apply` now prints a table similar to the `dstack ps` output:


✗ dstack apply
Project main
User admin
...

Submit a new run? [y/n]: y
NAME BACKEND RESOURCES PRICE STATUS SUBMITTED
spicy-tiger-1 gcp 2xCPU, 8GB, $0.06701 running 14:52
(us-central1) 100.0GB (disk)

spicy-tiger-1 provisioning completed (running)


What's Changed
* [UX]: live table when provisioning dstack configuration runs 1978 by Tob-iee in https://github.com/dstackai/dstack/pull/2036
* Fix returning metrics from deleted runs by jvstme in https://github.com/dstackai/dstack/pull/2038
* [UI] Migrate the chat components to the new CloudScape chat componets by olgenn in https://github.com/dstackai/dstack/pull/2044
* Recover unreachable instances by un-def in https://github.com/dstackai/dstack/pull/2043
* UX enhancements for `dstack apply` by jvstme in https://github.com/dstackai/dstack/pull/2045
* Implement /api/fleets/list endpoint by r4victor in https://github.com/dstackai/dstack/pull/2050
* Remove padding in `dstack apply` live tables by jvstme in https://github.com/dstackai/dstack/pull/2048
* Fix typo in `dstack attach --help` by jvstme in https://github.com/dstackai/dstack/pull/2054
* Support specifying internal_ip for SSH fleet hosts by r4victor in https://github.com/dstackai/dstack/pull/2056

New Contributors
* Tob-iee made their first contribution in https://github.com/dstackai/dstack/pull/2036

**Full Changelog**: https://github.com/dstackai/dstack/compare/0.18.28...0.18.29

CLI improvements
* Added alias `-R` for `--reuse` with `dstack apply`
* Shorten model URL output
* `dstack apply` and `dstack attach` no longer rely on external tools such as `ps` and `grep` on Unix-like systems and `powershell` on Windows. With this change, it's now possible to use `dstack` CLI client in minimal environments such as Docker containers, including the official [dstackai/dstack](https://hub.docker.com/r/dstackai/dstack) image

What's Changed
* Add `DSTACK_{RUNNER,SHIM}_DOWNLOAD_URL` env vars by un-def in https://github.com/dstackai/dstack/pull/2023
* [Feature] Add alias `-R` for `--reuse` with `dstack apply` by peterschmidt85 in https://github.com/dstackai/dstack/pull/2032
* Replace `ps | grep` with psutil in SSHAttach by un-def in https://github.com/dstackai/dstack/pull/2029
* Shorten model URL output in CLI by jvstme in https://github.com/dstackai/dstack/pull/2035


**Full Changelog**: https://github.com/dstackai/dstack/compare/0.18.27...0.18.28

UI/UX improvements

This release fixes a login issue in the control plane UI and introduces other UI/UX improvements.

What's Changed
* Another batch of many minor improvements to the docs by peterschmidt85 in https://github.com/dstackai/dstack/pull/2016
* Show OpenAI-compatible endpoint URL in CLI by jvstme in https://github.com/dstackai/dstack/pull/2022
* [Bug]: Cannot open UI login screen by olgenn in https://github.com/dstackai/dstack/pull/2025
* [UI] Model page code snippets fixes and improvements by olgenn in https://github.com/dstackai/dstack/pull/2026
* [UI]: Fix curl sample in model code button by jvstme in https://github.com/dstackai/dstack/pull/2027


**Full Changelog**: https://github.com/dstackai/dstack/compare/0.18.26...0.18.27