Skypilot

This patch release brings more bug fixes, including fixes for cloud-specific networking and VPC configuration and managed spot.

What's Changed

This is a patch release with lots of bug fixes across the board, including many cloud-specific networking and VPC fixes.

Stay tuned for a detailed changelog coming up in v0.3!

What's Changed

This is a patch release with several bug fixes for TPU, Spot, Onprem and Storage.

Detailed announcements will be made in 0.3.0.

We are excited to release SkyPilot 0.2.0, which receives a host of new features, with many enhancements and fixes.

Highlights

* [**Managed Spot**](https://skypilot.readthedocs.io/en/latest/examples/spot-jobs.html) is made much more robust and easier to use.
* Try using `sky spot launch` on your existing yamls!
* We've seen users running 1000s of spot jobs in a recurring schedule.
* [**TPU Pods**](https://skypilot.readthedocs.io/en/latest/reference/tpu.html) are now supported.
* To use a TPU Pod, simply modify e.g., `accelerators: tpu-v2-8` to `accelerators: tpu-v2-32`.
* [**Benchmark**](https://skypilot.readthedocs.io/en/latest/reference/benchmark/index.html): use `sky bench` to easily measure the performance and cost of different cloud resources for your task.
* Provisioning is sped up by ~1 minute.
* Catalog is updated to [V3](https://github.com/skypilot-org/skypilot-catalog/tree/master/catalogs/v3) with 100s of resource changes and 1000s of price changes.
* `A100-80GB` is now available on 3 clouds. Check out `sky show-gpus -a` for GPU prices.
* No action needed as this will be automatically downloaded.

CLI & Task interface
New Features
* Add zone support in YAML 1014
* Add shell completion support for CLI by 1162
* Add `--no-setup` option to `sky launch` to allow for remounting of files without running setup commands again 1184
* Add `sky start --all` to start all clusters 1065
* Add glob support for `sky storage delete` 1117
* Add `--no-follow` option to `sky logs` and `sky spot logs` (print logs so far and exit)

Enhancements
* Show vCPUs in optimizer/benchmark messages 1076
* Make entrypoint optional: for quick VM launching, no more `sky launch <flags> ''`, simply do `sky launch <flags>` 1191
* Make `sky check` automatically enable necessary GCP APIs (1197, 1209); make it more robust for AWS checks (1194)

Managed spot
New Features
* `sky spot launch` now automatically translates file_mounts in a YAML to use cloud storage. 1081 1215
* This means the same YAML for on-demand resources launched by `sky launch` can now be launched by `sky spot launch`.
* Add `--retry-until-up` for `sky spot launch`; improve the responsiveness for `sky spot cancel` https://github.com/skypilot-org/skypilot/pull/1098
* Expose a `$SKYPILOT_RUN_ID` environment variable shared by all recoveries of the same spot job (useful for identifying it in Weights & Biases) 1196
* See the last Note block in [docs](https://skypilot.readthedocs.io/en/latest/examples/spot-jobs.html).

Enhancements
* Distinguish spot controller names for different users https://github.com/skypilot-org/skypilot/pull/1101
* **This may leak an old stopped controller** if you have used `spot launch` with <= 0.1.2.
* Add retry for spot cluster termination https://github.com/skypilot-org/skypilot/pull/1139
* Enable purge for spot controller https://github.com/skypilot-org/skypilot/pull/1107
* Show FAILED_CONTROLLER when controller exit abnormally https://github.com/skypilot-org/skypilot/pull/1143
* Make get_job_timestamp fetching more robust https://github.com/skypilot-org/skypilot/pull/1148
* Fail early when spot cluster name too long occurs on GCP https://github.com/skypilot-org/skypilot/pull/1183

Fixes
* Fix the retry logic for spot cluster launching https://github.com/skypilot-org/skypilot/pull/1150
* Fix non-persistent storage deletion for spot 974
* Fix spot recovery without cloud specified 1077
* Fix spot job duration https://github.com/skypilot-org/skypilot/pull/1104
* Fix `sky spot status -a` for resources and region information https://github.com/skypilot-org/skypilot/pull/1135

TPU support
* Support TPU Pod https://github.com/skypilot-org/skypilot/pull/1001

Provisioner
Enhancements
* Improving provision speed by ~1 minute (1092, 1103, 1108, 1111, 1126)
* Add host VM - GPU compatibility checks for GCP https://github.com/skypilot-org/skypilot/pull/989

Fixes
* Fix GCP VM leak issue https://github.com/skypilot-org/skypilot/pull/1102
* Fix GCP A100 launch error https://github.com/skypilot-org/skypilot/pull/1166
* Fix K80 gpunode by correcting GCP image version https://github.com/skypilot-org/skypilot/pull/1090

On-prem
Enhancements
* **Simplified on-prem deployment**
* `sky admin deploy` now automatically installs `skypilot`, `ray` (and `python3` and `pip3`) on the local cluster under admin user 1116
* Add cluster config schema check https://github.com/skypilot-org/skypilot/pull/1044
* Modify Sky Admin's Setup on Docs https://github.com/skypilot-org/skypilot/pull/1085
* Align Python Versions https://github.com/skypilot-org/skypilot/pull/1086

Fixes
* Fix Sky Status Logging https://github.com/skypilot-org/skypilot/pull/1041

Backend
Enhancements
* Catalog is updated to V3 with 100s of resource changes and 1000s of price changes 1204
* Canonicalize accelerator names in Resources https://github.com/skypilot-org/skypilot/pull/1075
* Reduce the frequency of job status update and remove parallel query https://github.com/skypilot-org/skypilot/pull/1096
* Increase thread limit and fix nofile limit https://github.com/skypilot-org/skypilot/pull/1128

Fixes
* [Storage] Fix public bucket source check in SkyPilot Storage https://github.com/skypilot-org/skypilot/pull/1087
* Fixes ray dashboard hanging problem (1088) https://github.com/skypilot-org/skypilot/pull/1109
* Fix placement group not scheduled issue (issue 1130) https://github.com/skypilot-org/skypilot/pull/1134

Misc. enhancements
* **New example**: Stable Diffusion https://github.com/skypilot-org/skypilot/pull/1149
* `pip install skypilot` now installs `skypilot[aws]` by default https://github.com/skypilot-org/skypilot/pull/1055
* Improve error messages for cloud import errors https://github.com/skypilot-org/skypilot/pull/1156
* Change `~/.ssh/config` permissions https://github.com/skypilot-org/skypilot/pull/1174
* Relative cluster yaml https://github.com/skypilot-org/skypilot/pull/1176
* UX: remove DURATION, move HOURLY_PRICE in status table (-a) https://github.com/skypilot-org/skypilot/pull/1129

Thanks to all Contributors!
New contributors
* sumanthgenz made their first contribution in https://github.com/skypilot-org/skypilot/pull/1065
* ewzeng made their first contribution in https://github.com/skypilot-org/skypilot/pull/1174

Many thanks to all contributors who contributed to this release!

Michaelvll, concretevitamin, infwinston, michaelzhiluo, WoosukKwon, romilbhardwaj, sumanthgenz, ewzeng, iojw, franklsf95

Highlights
This is our first release for SkyPilot -- a framework for easily running machine learning workloads on any cloud through a unified interface. No knowledge of cloud offerings is required or expected – you simply define the workload and its resource requirements, and SkyPilot will automatically execute it on AWS, Google Cloud Platform or Microsoft Azure.
Key features

- Run existing projects on the cloud with zero code changes
- Easily provision VMs across multiple cloud platforms (AWS, Azure or GCP)
- Easily manage multiple clusters to handle different projects
- Quick access to cloud instances for development
- Store datasets on the cloud and access them like you would on a local file system
- No cloud lock-in – seamlessly run your code across cloud providers

Thanks
Many thanks to all those who contributed to this release!
concretevitamin romilbhardwaj Michaelvll infwinston michaelzhiluo WoosukKwon suquark mraheja gmittal iojw lhqing franklsf95

**Full Changelog**: https://github.com/skypilot-org/skypilot/commits/v0.1.1