AMD
With the latest update, you can now specify an AMD GPU under `resources`. Below is an example.
yaml
type: service
name: amd-service-tgi
image: ghcr.io/huggingface/text-generation-inference:sha-a379d55-rocm
env:
- HUGGING_FACE_HUB_TOKEN
- MODEL_ID=meta-llama/Meta-Llama-3.1-70B-Instruct
- TRUST_REMOTE_CODE=true
- ROCM_USE_FLASH_ATTN_V2_TRITON=true
commands:
- text-generation-launcher --port 8000
port: 8000
resources:
gpu: MI300X
disk: 150GB
spot_policy: auto
model:
type: chat
name: meta-llama/Meta-Llama-3.1-70B-Instruct
format: openai
> [!NOTE]
> AMD accelerators are currently supported only with the `runpod` backend. Support for on-prem fleets and more backends
> is coming soon.
GPU vendors
The `gpu` property now accepts the `vendor` attribute, with supported values: `nvidia`, `tpu`, and `amd`.
Alternatively, you can also prefix the GPU name with the vendor name followed by a colon, for example: `tpu:v2-8` or `amd:192GB`, etc. This change ensures consistency in GPU requirements configuration across vendors.
Encryption
`dstack` now supports encryption of sensitive data, such as backend credentials, user tokens, etc. Learn more on the [reference](https://dstack.ai/docs/reference/server/config.yml/#encryption_1) page.
Storing logs in AWS CloudWatch
By default, the `dstack` server stores run logs in `~/.dstack/server/projects/<project name>/logs`. To store logs in AWS CloudWatch, set the [DSTACK_SERVER_CLOUDWATCH_LOG_GROUP](https://dstack.ai/docs/reference/cli/#environment-variables) environment variable.
Project manager role
With this update, it's now possible to assign any user as a project manager. This role grants permission to manage project users but does not allow management of backends or resources.
Default permissions
By default, all users can create and manage their own projects. If you want only global admins to create projects, add the following to `~/.dstack/server/config.yml`:
yaml
default_permissions:
allow_non_admins_create_projects: false
Other
* [Feature] Allow to store logs in AWS CloudWatch by un-def in https://github.com/dstackai/dstack/pull/1597 and https://github.com/dstackai/dstack/pull/1597
* [Feature] Introduce default permissions 1559 by olgenn in https://github.com/dstackai/dstack/pull/1567
* [Feature] Support the `vendor` property under `resources.gpu` un-def in https://github.com/dstackai/dstack/pull/1558
* [Feature] Implement configurable default permissions by r4victor in https://github.com/dstackai/dstack/pull/1591
* [Bugfix] Provision AWS instances in all eligible availability zones by r4victor in https://github.com/dstackai/dstack/pull/1585
* [Bugfix] Support users without projects olgenn in https://github.com/dstackai/dstack/pull/1578
* [UI] Support `manager` project role olgenn in https://github.com/dstackai/dstack/pull/1566
* [Docs] Mention AMD GPUs, describe `gpu.vendor` property by un-def in https://github.com/dstackai/dstack/pull/1570
* [Bugfix] Fix global admin restricted by manager role by r4victor in https://github.com/dstackai/dstack/pull/1592
* [Bugfix] Fixed defect with incorrect setting project role in the UI by olgenn in https://github.com/dstackai/dstack/pull/1593
* [Bugfix] Abort provisioning fleet when parsing ssh key fails(1442) by swsvc in https://github.com/dstackai/dstack/pull/1589
* [UI] Ensure users can create projects 191 by olgenn in https://github.com/dstackai/dstack/pull/1554
* [UI] Use a toggle button switching themes 190 by olgenn in https://github.com/dstackai/dstack/pull/1556
* [UI] Fix the Logs component appearance for the dark theme by olgenn in https://github.com/dstackai/dstack/pull/1579
* [UI] Minor restyle of the side navigation by olgenn in https://github.com/dstackai/dstack/pull/1580
* [Bugfix] Avoid TGI error `logit_bias: invalid type` by jvstme in https://github.com/dstackai/dstack/pull/1557
* [Docs] Document projects 1547 by peterschmidt85 in https://github.com/dstackai/dstack/pull/1548
* [Docs] Document AMD support on RunPod by peterschmidt85 in https://github.com/dstackai/dstack/pull/1598
* [Internal] Approximate on-prem GPU memory size by jvstme in https://github.com/dstackai/dstack/pull/1588
* [Docs] Fix some of the broken links by jvstme in https://github.com/dstackai/dstack/pull/1602
* [Docs] Fix broken links in README.md by jvstme in https://github.com/dstackai/dstack/pull/1604
* [Docs] Document configuring logs storage in AWS CloudWatch un-def in https://github.com/dstackai/dstack/pull/1606
* [Docs] Publish the blog post and examples about AMD on RunPod by peterschmidt85 in https://github.com/dstackai/dstack/pull/1598
* [Internal] Force `root` in Kubernetes runs by jvstme in https://github.com/dstackai/dstack/pull/1555
* [Internal] Improve gateway auth issues troubleshooting by jvstme in https://github.com/dstackai/dstack/pull/1569
* [Feature] Implement "encryption at rest" by r4victor in https://github.com/dstackai/dstack/pull/1561
* [Feature] Implement project `manager` role by r4victor in https://github.com/dstackai/dstack/pull/1572
* [Feature] Implement user activation/deactivation by r4victor in https://github.com/dstackai/dstack/pull/1575
* [Internal] Reintroduce `tpu-` prefix; add `tpu` vendor alias by un-def in https://github.com/dstackai/dstack/pull/1587
New contributors
* swsvc made their first contribution in https://github.com/dstackai/dstack/pull/1589
**Full changelog**: https://github.com/dstackai/dstack/compare/0.18.10...0.18.11