Dstack

Multiple backends per project

Now, `dstack run` leverages price data from multiple configured cloud providers and regions to automatically suggest the most cost-effective options.

<img src="https://github.com/dstackai/dstack/assets/54148038/1b7eacff-fa99-4ac7-9145-d6ab0602eaf6" width="700"/>

To implement this change, we have modified the way projects are configured. You can now configure multiple clouds and regions within a single project.

<img src="https://github.com/dstackai/dstack/assets/54148038/a7eca63e-a360-41a7-b8f2-0195e1204981" width="700"/>

Custom domains and HTTPS

Also, it is now possible to deploy services using [HTTPS](https://dstack.ai/docs/guides/services/#configure-a-domain-and-enable-https-optional). All you need to do is configure a wildcard domain (e.g., `*.mydomain.com`), point it to the gateway IP address, and then pass the subdomain you want to use (e.g., `myservice.mydomain.com`) to the gateway property in YAML (instead of the gateway IP address).

Other changes

`.dstack/profiles.yml`

- The `project` property is no longer supported.
- You can now use `max_price` to set the maximum price per hour in dollars.

`dstack run`

- Using the `dstack run` command, you are now able to utilize options such as `--gpu`, `--memory`, `--env`, `--max-price`, and several [other arguments](https://dstack.ai/docs/reference/cli/run/) to override the profile settings.

Lastly, the local backend is no longer supported. Now, you can run everything using only a cloud backend.

The [documentation](https://dstack.ai/docs/) is updated to reflect the changes in the release.

Migration

The `dstack` version 0.11 update brings significant changes that break backward compatibility. If you used prior `dstack` versions, after updating to `dstack==0.11`, you'll need to log in to the UI and reconfigure clouds. We apologize for any inconvenience and aim to ensure future updates maintain backward compatibility.

This minor update addresses a crucial issue that was causing services to malfunction. If you're utilizing services, be sure to update.

**Changelog**: https://github.com/dstackai/dstack/compare/0.10.7...0.10.8

Services

Until now, `dstack` has supported `dev-environment` and `task` as configuration types. With the `0.10.7` update, we introduce
`service`, a dedicated configuration type for serving.

Usage example:

yaml
type: service

gateway: ${{ secrets.GATEWAY_ADDRESS }}

image: ghcr.io/huggingface/text-generation-inference:0.9.3

port: 8000

commands:
- text-generation-launcher --hostname 0.0.0.0 --port 8000 --trust-remote-code


The `gateway` property represents the address of a special cloud instance that wraps the running service with a public endpoint.

Gateways

Before you can run a service, you have to configure a gateway.

First, you have to create a gateway in a project of your choice using the `dstack gateway create` command:

shell
dstack gateway create


Once the gateway is up, the command will print its address. Go ahead and create a secret with this address.

shell
dstack secrets add GATEWAY_ADDRESS <gateway address>


That's it! Now you can run your service using the `dstack run` command, which deploys the service and forwards the traffic to the gateway, thereby providing you with a public endpoint.

This initial support for services is the first step towards providing multi-cloud and cost-effective inference. In the near future, we plan to make it more functional and easier to use.

What's changed

* Support restarting local and GCP runs by r4victor in https://github.com/dstackai/dstack/pull/590
* Add API for listing all runs by r4victor in https://github.com/dstackai/dstack/pull/591
* The `dstack init` should handle invalid Git credentials by peterschmidt85 in https://github.com/dstackai/dstack/pull/594
* Support custom run names by r4victor in https://github.com/dstackai/dstack/pull/595
* `dstack` doesn't work if the repo contains a submodule with SSH URL by peterschmidt85 in https://github.com/dstackai/dstack/pull/598
* Always use cuda images for instances with GPU by r4victor in https://github.com/dstackai/dstack/pull/602
* Introduce gateways for services publication by Egor-S in https://github.com/dstackai/dstack/pull/596
* Small ports refactoring by Egor-S in https://github.com/dstackai/dstack/pull/603
* 588 Created All run list page. by olgenn in https://github.com/dstackai/dstack/pull/607
* Improve gateway security, show verbose errors by Egor-S in https://github.com/dstackai/dstack/pull/608
* Do not require sshd in task configuration for custom docker images by Egor-S in https://github.com/dstackai/dstack/pull/609
* 588 all runs list by olgenn in https://github.com/dstackai/dstack/pull/611
* [Bug]: Doesn't run a dev environment if `code` is not configured in `PATH` by peterschmidt85 in https://github.com/dstackai/dstack/pull/613
* Allow to create gateways in AWS and Azure by Egor-S in https://github.com/dstackai/dstack/pull/614
* [Bug]: `dstack init` doesn't work if `.git/config` or `~/.gitconfig` doesn't have the `user` section by peterschmidt85 in https://github.com/dstackai/dstack/pull/617
* [Bug]: `dstack config` doesn't work if `~/.dstack` doesn't exist by peterschmidt85 in https://github.com/dstackai/dstack/pull/618
* Migrate to Gen2 images for Azure, add A100 support by Egor-S in https://github.com/dstackai/dstack/pull/619
* Add time.Sleep() in /logsws handler by r4victor in https://github.com/dstackai/dstack/pull/623
* Services & gateway docs by Egor-S in https://github.com/dstackai/dstack/pull/620
* Replace localhost with gateway hostname in service logs by Egor-S in https://github.com/dstackai/dstack/pull/624


**Changelog**: https://github.com/dstackai/dstack/compare/0.10.6...0.10.7

Port mapping

Any task that is running on `dstack` can expose ports. Here's an example:

yaml
type: task

ports:
- 7860

commands:
- pip install -r requirements.txt
- gradio app.py


When you run it with `dstack run`, by default, `dstack` forwards the traffic from the specified port to the same port on your local machine.

With this update, you now have the option to override the local machine's port for traffic forwarding.

shell
dstack run . -f serve.dstack.yml --port 3000:7860


This command forwards the traffic to port `3000` on your local machine.

If you specify a port on your local machine already taken by another process, dstack will notify you before provisioning cloud resources.

Max duration

Previously, when running a dev environment or task with dstack and forgetting about it, it would continue indefinitely. Now, you can use the `max_duration` property in `.dstack/profiles.yml` to set a maximum time for workloads.

Example:

yaml
profiles:
- name: gcp-t4
project: gcp
resources:
memory: 24GB
gpu:
name: T4
max_duration: 2h


With this profile, `dstack` will automatically stop the workload after 2 hours.

If you don't specify `max_duration`, `dstack` defaults to `6h` for dev environments and `72h` for tasks.

To disable `max duration`, you can set it to `off`.

Imagine the amount of money your team can save with this minor configuration.

More supported GPUs

With the CUDA version updated to 11.8, `dstack` now supports additional GPU types, including `NVIDIA T4` and `NVIDIA L4`. These GPUs are highly efficient for LLM development, offering excellent performance at low costs!

If you are using a custom Docker image, you can now utilize a CUDA version up to 12.2.

Last but not least, the K80 GPU is no longer supported.

Examples

Make sure to check the new page with [examples](https://dstack.ai/examples/).

The [documentation](https://dstack.ai/docs/index.md) is updated to reflect the changes in the release.

What's changed

* Elaborate error message on unmatched requirements for local backend by r4victor in https://github.com/dstackai/dstack/pull/563
* Handle no permissions for listing aws buckets by r4victor in https://github.com/dstackai/dstack/pull/566
* Support max_duration by r4victor in https://github.com/dstackai/dstack/pull/571
* Fix aws not using default region without default creds by r4victor in https://github.com/dstackai/dstack/pull/572
* Add --ports to `dstack run` by Egor-S in https://github.com/dstackai/dstack/pull/573
* Add T4 support for GCP by Egor-S in https://github.com/dstackai/dstack/pull/575
* Close 537 Added field extra_regions for aws backend by olgenn in https://github.com/dstackai/dstack/pull/577
* Reserve ports before creating instance by Egor-S in https://github.com/dstackai/dstack/pull/578
* Upgrade instance images by Egor-S in https://github.com/dstackai/dstack/pull/581
* Update CUDA version to `11.8` by peterschmidt85 in https://github.com/dstackai/dstack/pull/584
* Use identity ports mapping by default by Egor-S in https://github.com/dstackai/dstack/pull/587

**Changelog**: https://github.com/dstackai/dstack/compare/0.10.5...0.10.6

Lambda

The [Lambda Cloud](https://lambdalabs.com) integration has significantly improved with this release. We've added the possibility to create Lambda Cloud projects via the user interface.

<img src="https://dstack.ai/assets/images/dstack-hub-create-lambda-project.png" width="700"/>

All you need to do is provide your Lambda Cloud API key and specify an S3 bucket and AWS credentials for storing state and artifacts.
Check the [docs](https://dstack.ai/docs/reference/backends/lambda/) for more details.

Custom Docker images

By default, dstack uses its own base [Docker images](https://hub.docker.com/r/dstackai/miniforge/tags) to run dev environments and tasks. These base images come pre-configured with Python, Conda, and essential CUDA drivers. However, there may be times when you need additional dependencies that you don't want to install every time you run your dev environment or task.

To address this, dstack now allows specifying custom Docker images. Here's an example:

yaml
type: task

image: ghcr.io/huggingface/text-generation-inference:0.9

env:
- MODEL_ID=tiiuae/falcon-7b

ports:
- 3000

commands:
- text-generation-launcher --hostname 0.0.0.0 --port 3000 --trust-remote-code


> **Note**
> Dev environments require the Docker image to have `openssh-server` pre-installed.

What's Changed

* Refactor providers, add custom docker images support by Egor-S in https://github.com/dstackai/dstack/pull/544
* Use nvidia docker runtime for build if available by Egor-S in https://github.com/dstackai/dstack/pull/552
* Allow env variables interpolation by Egor-S in https://github.com/dstackai/dstack/pull/553
* Display docker pull progress by Egor-S in https://github.com/dstackai/dstack/pull/554
* Handle backend creds becoming invalid by r4victor in https://github.com/dstackai/dstack/pull/556
* Add PATH to .profile by Egor-S in https://github.com/dstackai/dstack/pull/557
* Close 529 Supported lambda backend type in UI by olgenn in https://github.com/dstackai/dstack/pull/558
* 529 Fixed labels by olgenn in https://github.com/dstackai/dstack/pull/560
* 529 fixes after review by olgenn in https://github.com/dstackai/dstack/pull/562

**Changelog**: https://github.com/dstackai/dstack/compare/0.10.4...0.10.5

What's changed

* [Bug]: Provisioning on Lambda Cloud doesn't work 542 by peterschmidt85 in https://github.com/dstackai/dstack/pull/543
* [Bug]: Cannot create an AWS project 545 by peterschmidt85 in https://github.com/dstackai/dstack/pull/546

**Changelog**: https://github.com/dstackai/dstack/compare/0.10.3...0.10.4