We are excited to announce the release of SkyPilot v0.7.0! This release brings significant performance improvements and many new features:
* Upto 3x faster provisioning
* Reservation support: AWS Capacity Reservations, AWS Capacity Blocks, GCP reservations, GCP Dynamic Workload Scheduler (DWS), and more
* Observability features
* Admin policy enforcement
* Support for H100 Mega, TPU v6, TPU v5, gVNIC, azure blob storage, faster disks, and more
* New UX for `sky` CLI
and many bug fixes and enhancements!
Release Highlights
Performance
We have made 2-3x performance improvements across cloud providers through optimizations in our provisioning stack and the images we use.
| Cloud | Provisioning Time | Speedup |
|----------------|-------------------|---------|
| AWS | 1 min 10s | 3x |
| GCP | 1 min 15s | 3x |
| Azure | 2 min 16s | 2x |
| Kubernetes | 52s | 2.5x |
Reservations
SkyPilot now supports [short-term and long-term reservations](https://skypilot.readthedocs.io/en/latest/reservations/reservations.html) across clouds:
* AWS Capacity Reservations
* AWS Capacity Blocks
* GCP reservations
* GCP Dynamic Workload Scheduler (DWS)
* Bring your own [VMs](https://skypilot.readthedocs.io/en/latest/reservations/existing-machines.html) or [Kubernetes clusters](https://skypilot.readthedocs.io/en/latest/reference/kubernetes/index.html)
SkyPilot's failover includes these reservations, so they can be combined with spot instances or any other resources/clouds to create a resilient and cost-effective infrastructure.
Observability on Kubernetes
SkyPilot now has two new observability features on Kubernetes:
* `sky status --kubernetes` shows all SkyPilot resources on the cluster. (4040, 4079)
$ sky status --cloud kubernetes
Kubernetes cluster state (context: mycluster)
SkyPilot clusters
USER NAME LAUNCHED RESOURCES STATUS
alice infer-svc-1 23 hrs ago 1x Kubernetes(cpus=1, mem=1, {'L4': 1}) UP
alice sky-jobs-controller-80b50983 2 days ago 1x Kubernetes(cpus=4, mem=4) UP
alice sky-serve-controller-80b50983 23 hrs ago 1x Kubernetes(cpus=4, mem=4) UP
bob dev 1 day ago 1x Kubernetes(cpus=2, mem=8, {'H100': 1}) UP
bob multinode-dev 1 day ago 2x Kubernetes(cpus=2, mem=2) UP
bob sky-jobs-controller-2ea485ea 2 days ago 1x Kubernetes(cpus=4, mem=4) UP
Managed jobs
In progress tasks: 1 STARTING
USER ID TASK NAME RESOURCES SUBMITTED TOT. DURATION JOB DURATION RECOVERIES STATUS
alice 1 - eval 1x[CPU:1+] 2 days ago 49s 8s 0 SUCCEEDED
bob 4 - pretrain 1x[H100:4] 1 day ago 1h 1m 11s 1h 14s 0 SUCCEEDED
bob 3 - bigjob 1x[CPU:16] 1 day ago 1d 21h 11m 4s - 0 STARTING
bob 2 - failjob 1x[CPU:1+] 1 day ago 54s 9s 0 FAILED
bob 1 - shortjob 1x[CPU:1+] 2 days ago 1h 1m 19s 1h 16s 0 SUCCEEDED
* `sky show-gpus --cloud kubernetes` shows detailed GPU availability information on the cluster. (3816, 4085)
$ sky show-gpus --cloud kubernetes
Kubernetes GPUs
GPU REQUESTABLE_QTY_PER_NODE TOTAL_GPUS TOTAL_FREE_GPUS
L4 1, 2, 4 8 8
H100 1, 2, 4, 8 16 16
Kubernetes per node GPU availability
NODE_NAME GPU_NAME TOTAL_GPUS FREE_GPUS
my-cluster-0 L4 4 4
my-cluster-1 L4 4 4
my-cluster-2 H100 8 8
my-cluster-3 H100 8 8
Admin policy enforcement
SkyPilot has a new [admin policy mechanism](https://skypilot.readthedocs.io/en/latest/cloud-setup/policy.html) (#3966) that admins can use to enforce policies on users’ SkyPilot usage. These policies apply custom validation and mutation logic to a user’s tasks and SkyPilot config.
Example policies:
* [Add Labels for all Tasks on Kubernetes](https://skypilot.readthedocs.io/en/latest/cloud-setup/policy.html#kubernetes-labels-policy)
* [Always Disable Public IP for AWS Tasks](https://skypilot.readthedocs.io/en/latest/cloud-setup/policy.html#disable-public-ip-policy)
* [Use Spot for all GPU Tasks](https://skypilot.readthedocs.io/en/latest/cloud-setup/policy.html#use-spot-for-gpu-policy)
* [Enforce Autostop for all Tasks](https://skypilot.readthedocs.io/en/latest/cloud-setup/policy.html#enforce-autostop-policy)
Azure Blob Storage support
In addition to S3, GCS and R2, you can now use Azure Blob Storage as a storage backend for storing and accessing data. (3032)
New AI hardware support
* New accelerators: TPU v6 (4115), TPU v5 (3814), H100 Mega (4099),
* Faster networking on GCP with gVNIC (4095)
* Faster disks: new disk tier `ultra` (3860) for GCP and AWS.
UX revamp
SkyPilot CLI is cleaner, simpler and even easier to parse now (4023)
<img src="https://i.imgur.com/fg8tOYq.gif" width="600"/>
New LLM Recipes
* Llama 3.1 and Llama 3.2 recipes (3990, 3779, 3780)
* llm.c training for GPT 2 (3611)
* Pixtral (3938, 3940)
* Qwen2-VL and Qwen 2.5 support (3961, 3959)
* Yi model family support (3958)
* Nemo GPT (3743)
* Other examples: Airflow (3982), AWS Neuron Accelerator (4020), and Deepspeed with k8s support (4124)
Deprecation Notice
* All `SKY_*` environment variables are deprecated in favor of `SKYPILOT_*` variables.
* All `SKY_*` variables will be removed in v0.9.0.
* See [docs](https://skypilot.readthedocs.io/en/latest/running-jobs/environment-variables.html#skypilot-environment-variables) for list of currently supported variables.
Backend
New Features
* Managed jobs can now recover from job-level failures (e.g., GPU errors, non-zero exit codes, etc.) (3919)
* Set `max_restarts_on_errors` to specify the number of times SkyPilot should try to restart the job.
resources:
job_recovery:
max_restarts_on_errors: 3 Retry 3 times before marking the job as failed
* Nvidia GPUs can now disable ECC (3676)
* New environment variable `SKYPILOT_NUM_NODES` to fetch the number of nodes in the cluster. (3656)
* SkyPilot [config can now be overridden](https://skypilot.readthedocs.io/en/latest/reference/yaml-spec.html#experimental-configurations) in the task definition with `experimental.config_override` (3689)
experimental:
config_override:
docker:
run_options: ...
kubernetes:
pod_config: ...
provision_timeout: ...
gcp:
managed_instance_group: ...
nvidia_gpus:
disable_ecc: ...
Enhancements
* SSH keys AddKeysToAgent for ssh config file and ssh cmd https://github.com/skypilot-org/skypilot/pull/3985
* SkyPilot runtime is now installed in a separate conda environment, reducing interference with user's environment. (3639)
* Similarly, the environment pre-configured in your docker image is no longer shadowed by SkyPilot's runtime environment (3874, 3867)
* `docker.run_options` now allows users to pass additional options when running docker containers. (3682)
Fixes
* Fix `sky cancel` not terminating all child processes (3919)
* Fix provisioning failures when multiple versions of SkyPilot are installed (3866)
* Shell autocomplete installation is now more robust (3892, 3893)
Kubernetes
New Features
* Observability improvements:
* `sky status --cloud kubernetes` shows all SkyPilot resources on the Kubernetes cluster. (4040, 4079)
* `sky show-gpus --cloud kubernetes` shows detailed GPU availability information on the cluster. (3816, 4085)
* SkyPilot now helps you set up your clusters for running SkyPilot jobs.
* If you already have a list of IPs and their SSH keys, `sky local up` can now [automatically set it up as a cluster](https://skypilot.readthedocs.io/en/latest/reservations/existing-machines.html) to be used for running jobs. (#3926)
* If you don't have a cluster yet, we provide a simple [one-click setup script](https://github.com/skypilot-org/skypilot/tree/master/examples/k8s_cloud_deploy) to deploy VMs with Kubernetes on cloud of your choice (#3929).
* SkyPilot job output is now piped to the container logs (3758)
* Use your existing logging tooling (`kubectl logs`, filebeat, etc.) to view SkyPilot job outputs.
* Support for Nvidia GPU operator labels (`nvidia.com/gpu.product`) for detecting GPU types. (3493)
* You no longer need to label GPUs if you have the Nvidia GPU operator installed.
* Spot instances are now supported on GKE clusters (3675)
* [Experimental] Multi-context support (3913, 3968, 3897, 3772, 4013)
Performance improvements:
* New command runner: 3x faster command submission for Kubernetes pods. (3157)
* `sky local up` for GPUs is now ~5x faster, provisioning in 2min 30s instead of 12min (3664)
* Our GPU images are now 3x smaller (1.5 GB), reducing the time to pull the image (3665)
* SSH jump pod is no longer required for `port-forward` mode (3657)
* SSH setup is now parallelized to speed up multi-node provisioning (4158)
Enhancements and fixes
* H100 Mega support on GKE (3891, 3627)
* Better handling for context names with special characters (4147)
* `--k8s` is now a valid alias for `--cloud kubernetes` (4151)
* Init containers are now supported on Kubernetes (3762)
* Auth: robust service account support and updated docs on minimal permissions (3632)
* Custom metadata annotations are now propagated to services, allowing configuration of internal load balancer services on cloud hosted Kubernetes clusters (3767)
* Provisioning errors are now surfaced clearly (3590, 3795, 3821)
* Cluster attributes (autodown, idle-minutes-to-autostop) are now added as annotations to the pod (3870)
* SkyServe controller is now automatically terminated when all replicas are terminated. (3984)
* Create namespace permission is no longer required in cluster launch flow (3714)
* If your cluster does not support `apparmor`, SkyPilot will now retry without requesting it. (4176)
Cloud: GCP
New Features
* New accelerators supported:
* H100 Mega (4099)
* TPU v5 (3814)
* [Dynamic Workload Scheduler (DWS)](https://cloud.google.com/blog/products/compute/introducing-dynamic-workload-scheduler) support (#3574, 3835)
* DWS helps get better availability on GCE through queuing and reservations.
* Faster `pd-extreme` disks with `disk_tier: ultra` (3860)
* New config `gcp.force_enable_external_ips` to force enable external IPs (3699)
* This is useful when communication within a VPC is desired and the VM needs to make calls to the public internet.
* TPU VMs can now run docker containers (4115)
Enhancements
* Provisioning is now 3x faster on GCP (4027)
* Faster networking support with gVNIC (4095)
* Upto ~2x faster in [pytorch distributed benchmarks](https://gist.github.com/romilbhardwaj/89f8399d8a5307df5d880cf1495ce957)
Cloud: AWS
New Features
* Capacity blocks and capacity reservations are now supported. (3852, 3853)
* You don’t have to wake up at 4:30am PDT to launch your job on a newly available capacity block: SkyPilot will wait for you until the start time of the capacity block.
* Faster `io2` disks with `disk_tier: ultra` (3860)
* Security groups: you can now specify security groups for your resources at a finer granularity. (3501)
* SkyPilot can now use encrypted EBS volumes (3765)
Enhancements
* Performance: provisioning now 3x faster on AWS (4091)
* Buckets created by SkyPilot are now tagged with labels specified in ~/.sky/config.yaml (3922)
* Label validation now handles `:` and other special characters. (3734)
Cloud: Azure
New Features
* You can now use any Azure community image with `--image-id` (4145)
* Azure Blob Storage is now supported (3032, 3796, 3807)
* Fractional A10 instance types are now supported (3877)
* You can now specify resource group for Azure instance provisioning (3764)
* Faster `Premium_LRS` disks with `disk_tier: high` (3921)
Enhancements
* Performance: provisioning is now 2x faster on Azure with our new provisioner and custom images (3697, 3704, 3696, 3700, 4139, 4167, 4205)
* Improved support for A10 GPUs (3707)
* Azure resource group is now waited to be deleted instead of erroring out (3712)
SkyServe
* Readiness probe timeout can now be set in the service spec (3472)
* You can now tear down a specific replica with `sky serve down --replica-id` (4032)
* SkyServe controller region is now chosen from the replica resources (4053)
Storage
* Azure Blob Storage is now supported. (3032, 3796, 3807)
* [`.skyignore` support](https://skypilot.readthedocs.io/en/latest/examples/syncing-code-artifacts.html#exclude-uploading-files) (4038)
* You can now add files to a `.skyignore` file to skip uploading them to cloud storage.
* GCSFuse is updated to 2.2.0, bringing better performance and reliability. (3619)
Other clouds
* Lambda Cloud support has been migrated to our new and more reliable provisioner (3865, 3889)
* Lambda Cloud now supports docker images (4115)
* CUDO now supports opening ports (3717)
* RunPod now supports opening ports (3748) and custom docker images (3728).
* FluidStack provisioning has been updated to their new API (3799)
* Paperspace now supports A4000 and P4000 GPUs (3991)
* OCI: bug fixes and improvements (4074, 4080)
Thanks to all contributors!
New contributors: winglian, Ultramann, jucor, BitPhinix, sethkimmel3, hyoxt121, BabyChouSr, wizenheimer, gurcangercek, shashank2000, ckgresla, bernardwin, kmushegi, Conless, JayThomason, colinjc, mtaran, Haijian06, KrishivPiduri, zpoint
Many thanks to all contributors who contributed to this release!
Contributors: Michaelvll, romilbhardwaj, cblmemo, landscapepainter, asaiacai, andylizf, yika, concretevitamin, colinjc, fozziethebeat, MaoZiming, JGSweets, Ultramann, Conless, jucor, wizenheimer, Haijian06, HysunHe, gurcangercek, bernardwin, JungleCatSW, BabyChouSr, hyoxt121, winglian, sethkimmel3, mjibril, shashank2000, ckgresla, zpoint, mtaran, KrishivPiduri, JayThomason, BitPhinix, kmushegi
**Full Changelog**: https://github.com/skypilot-org/skypilot/compare/v0.6.0...v0.7.0