SkyPilot v0.12.0 brings major new capabilities: **Slurm integration** for running SkyPilot on existing Slurm clusters, **Job Groups** for heterogeneous parallel workloads like RL training, an **Agent Skill** that teaches AI coding agents to use SkyPilot, **Recipes** for sharing reusable YAML templates across teams, and significant **Pool enhancements** including autoscaling. This release also drops Python 3.7/3.8, with Python 3.9+ now required.
Get it now with:
bash
uv pip install "skypilot[all]>=0.12.0"
Or, upgrade your team SkyPilot API server:
bash
NAMESPACE=skypilot
RELEASE_NAME=skypilot
VERSION=0.12.0
helm repo update skypilot
helm upgrade -n $NAMESPACE $RELEASE_NAME skypilot/skypilot \
--set apiService.image=berkeleyskypilot/skypilot:$VERSION \
--version $VERSION --devel --reuse-values
**Deprecations & Breaking Changes**:
- **Dropped Python 3.7 and 3.8 support**—Python 3.9+ is now required ([8489](https://github.com/skypilot-org/skypilot/pull/8489)).
- **`sky.jobs.queue(version=1)` is deprecated** and will be removed in v0.13. Use `sky.jobs.queue(version=2)` instead. The new version returns richer [job metadata as dictionaries](https://docs.skypilot.co/en/latest/reference/api.html#sky-jobs-queue-v2) ([9118](https://github.com/skypilot-org/skypilot/pull/9118)).
Highlights
[New] [Slurm Support](https://docs.skypilot.co/en/latest/reference/slurm/slurm-getting-started.html)
SkyPilot now supports connecting your **Slurm clusters**, bringing its unified interface to one of the most widely used job schedulers in high-performance computing ([5491](https://github.com/skypilot-org/skypilot/pull/5491), [#8198](https://github.com/skypilot-org/skypilot/pull/8198), [#8219](https://github.com/skypilot-org/skypilot/pull/8219), [#8268](https://github.com/skypilot-org/skypilot/pull/8268), [#8291](https://github.com/skypilot-org/skypilot/pull/8291), [#8470](https://github.com/skypilot-org/skypilot/pull/8470), [#8604](https://github.com/skypilot-org/skypilot/pull/8604), [#8729](https://github.com/skypilot-org/skypilot/pull/8729), and 25+ additional PRs). Users can launch SkyPilot clusters and managed jobs on Slurm clusters with the same CLI and YAML they use for cloud and Kubernetes, enabling seamless workload portability across all AI infra.
<p align="center">
<img width="80%" alt="SkyPilot dashboard showing Slurm GPU availability" src="https://github.com/user-attachments/assets/2790a276-1fd7-4edb-8643-e04200236dae" />
</p>
Key capabilities include:
- **Multi-node clusters** with partition-level resource management
- **Container support** via pyxis/enroot for reproducible environments
- **SSH ProxyJump** for clusters behind bastion hosts
- **GPU availability viewing** with `sky show-gpus` for Slurm partitions
- **Custom sbatch directives** via `sbatch_options` in task YAML
- **Configurable workdir and tmpdir** for shared filesystem environments
- **Admin policy support** for Slurm partition routing
yaml
Launch on a Slurm cluster
resources:
accelerators: H100:8
infra: slurm
bash
View GPU availability across Slurm clusters
sky show-gpus --infra slurm
Launch a training job on Slurm
sky launch --infra slurm/my-cluster train.yaml
[New] [Agent Skill: AI Agents Meet SkyPilot](https://blog.skypilot.co/scaling-autoresearch/)
SkyPilot now ships an official **Agent Skill** that teaches AI coding agents—Claude Code, Codex, and others—how to use SkyPilot ([8823](https://github.com/skypilot-org/skypilot/pull/8823), [#9017](https://github.com/skypilot-org/skypilot/pull/9017), [#9037](https://github.com/skypilot-org/skypilot/pull/9037)). With the skill installed, your agent can launch clusters, run managed jobs, serve models, compare GPU pricing, and manage cloud resources—all through natural language.
Install the skill ([docs](https://docs.skypilot.co/en/latest/getting-started/skill.html#installation)) by telling your agent:
bash
Fetch and follow the install guide https://github.com/skypilot-org/skypilot/blob/HEAD/agent/INSTALL.md
In our [Scaling Autoresearch blog post](https://blog.skypilot.co/scaling-autoresearch/), we gave Claude Code the SkyPilot agent skill and access to a 16-GPU Kubernetes cluster. Over 8 hours, the agent autonomously submitted **~910 experiments** in parallel, achieving a **9x speedup** over sequential search—and even discovered hardware-specific optimizations on its own.
<p align="center">
<img width="60%" alt="Scaling Autoresearch with SkyPilot Agent Skill" src="https://blog.skypilot.co/scaling-autoresearch/assets/banner.png" />
</p>
Example interactions your agent can now handle:
| Capability | Example Prompt |
|---|---|
| Launch dev clusters | *"Launch a cluster with 4 A100 GPUs. Auto-stop after 30 min idle."* |
| Fine-tune models | *"Fine-tune Llama 3.1 8B on my dataset at s3://my-data. Use spot instances."* |
| Distributed training | *"Run PyTorch DDP training across 4 nodes with 8 H100s each."* |
| Serve models | *"Deploy Llama 3.1 70B with vLLM. Autoscale 1-3 replicas based on QPS."* |
| Compare pricing | *"What's the cheapest 8x H200 across AWS, GCP, Lambda, and CoreWeave?"* |
| Multi-cloud failover | *"Submit jobs that try our Slurm cluster first and fall back to AWS."* |
[New] [Job Groups: Heterogeneous Parallel Workloads](https://blog.skypilot.co/job-groups/)
SkyPilot Job Groups let you define **multiple tasks with different resource requirements** that run together as a single managed job ([8456](https://github.com/skypilot-org/skypilot/pull/8456), [#8664](https://github.com/skypilot-org/skypilot/pull/8664), [#8686](https://github.com/skypilot-org/skypilot/pull/8686), [#8688](https://github.com/skypilot-org/skypilot/pull/8688), [#8713](https://github.com/skypilot-org/skypilot/pull/8713), [#8940](https://github.com/skypilot-org/skypilot/pull/8940)). This is ideal for **reinforcement learning** workflows where training, inference, and data servers need different hardware—H100s for policy training, cheaper GPUs for rollout inference, and high-memory CPUs for replay buffers.
SkyPilot provisions all resources together, configures networking automatically with built-in **service discovery** (`{task_name}-{node_index}.{job_group_name}`), and manages the full lifecycle as one unit.
yaml
Multi-document YAML for a Job Group
---
name: rl-training
execution: parallel
primary_tasks: [ppo-trainer]
---
name: data-server
resources:
cpus: 4+
run: |
python data_server.py --port 8000
---
name: ppo-trainer
resources:
accelerators: H100:1
run: |
python ppo_trainer.py --data-server data-server-0.rl-training:8000
<p align="center">
<img width="60%" alt="Job Groups architecture" src="https://blog.skypilot.co/job-groups/assets/banner.png" />
</p>
[New] [Recipes: Shared YAML Registry](https://blog.skypilot.co/skypilot-recipes/)
**Recipes** allow teams to store and share SkyPilot YAMLs in a centralized, team-accessible registry ([8755](https://github.com/skypilot-org/skypilot/pull/8755), [#8825](https://github.com/skypilot-org/skypilot/pull/8825), [#8851](https://github.com/skypilot-org/skypilot/pull/8851), [#8873](https://github.com/skypilot-org/skypilot/pull/8873), [#8876](https://github.com/skypilot-org/skypilot/pull/8876), [#8882](https://github.com/skypilot-org/skypilot/pull/8882)). Launch standardized workloads directly from the CLI or dashboard without local YAML files, reducing DevOps overhead and ensuring consistent configurations across the team.
Recipes support clusters, managed jobs, pools, and SkyServe, with built-in validation that blocks local file dependencies to ensure portability.
bash
Launch a recipe directly
sky launch recipes:dev-cluster
Launch with custom overrides
sky launch recipes:gpu-cluster --cpus 16 --gpus H100:4 --env DATA_PATH=s3://my-data
<p align="center">
<img width="80%" alt="Recipes dashboard" src="https://blog.skypilot.co/skypilot-recipes/images/create-recipe.png" />
</p>
Pool Autoscaling & Enhancements
SkyPilot Pools receive major upgrades with **autoscaling**, **multiple jobs per worker**, **heterogeneous pools**, and **memory-aware scheduling** ([8483](https://github.com/skypilot-org/skypilot/pull/8483), [#8192](https://github.com/skypilot-org/skypilot/pull/8192), [#8315](https://github.com/skypilot-org/skypilot/pull/8315), [#8279](https://github.com/skypilot-org/skypilot/pull/8279), [#8509](https://github.com/skypilot-org/skypilot/pull/8509), [#7891](https://github.com/skypilot-org/skypilot/pull/7891)).
- **Autoscaling** ([8483](https://github.com/skypilot-org/skypilot/pull/8483)): Pools now automatically scale workers up and down (including to zero) based on queue length, maximizing GPU utilization while minimizing cost.
- **Multiple jobs per worker** ([8192](https://github.com/skypilot-org/skypilot/pull/8192)): Workers can now run multiple concurrent jobs, improving resource utilization for smaller workloads.
- **Heterogeneous pools** ([8315](https://github.com/skypilot-org/skypilot/pull/8315)): Pools can now contain workers with different resource configurations.
- **Memory-aware scheduling** ([8279](https://github.com/skypilot-org/skypilot/pull/8279)): The pool scheduler now considers memory requirements when assigning jobs to workers.
- **Fractional GPU improvements** ([8509](https://github.com/skypilot-org/skypilot/pull/8509), [#8480](https://github.com/skypilot-org/skypilot/pull/8480)): Fixed fractional GPU support across multiple workers and corrected dashboard display.
yaml
autoscaling-pool.yaml
pool:
min_workers: 0
max_workers: 10
resources:
accelerators: H100
setup: |
echo "Setup complete!"
Dashboard Performance & GPU Metrics
The SkyPilot dashboard receives significant performance improvements and new observability features ([8523](https://github.com/skypilot-org/skypilot/pull/8523), [#8718](https://github.com/skypilot-org/skypilot/pull/8718), [#8651](https://github.com/skypilot-org/skypilot/pull/8651), [#8534](https://github.com/skypilot-org/skypilot/pull/8534), [#8539](https://github.com/skypilot-org/skypilot/pull/8539)).
- **6x faster infra page** on first load with general speed improvements across all pages ([8523](https://github.com/skypilot-org/skypilot/pull/8523))
- **GPU metrics for managed jobs and job groups** ([8718](https://github.com/skypilot-org/skypilot/pull/8718)), including **GPU temperature panel** ([#8472](https://github.com/skypilot-org/skypilot/pull/8472)) and external Grafana links ([#8599](https://github.com/skypilot-org/skypilot/pull/8599))
- **Cluster burn rate metric** ([8683](https://github.com/skypilot-org/skypilot/pull/8683)) and **RPS-by-user metric** ([#8999](https://github.com/skypilot-org/skypilot/pull/8999)) in the API server Grafana dashboard
- **Progressive loading** for workspace and infra pages ([8575](https://github.com/skypilot-org/skypilot/pull/8575), [#8550](https://github.com/skypilot-org/skypilot/pull/8550))
Consolidation Mode by Default
Deploy-mode API servers now **auto-enable consolidation mode** ([9090](https://github.com/skypilot-org/skypilot/pull/9090)), which runs managed job controllers on the API server itself instead of launching separate controller VMs. This eliminates controller overhead costs and simplifies managed job operations for team deployments.
7x MOUNT_CACHED Speed Up
**Parallel uploads** are now the default for `MOUNT_CACHED` file mounts, delivering a **7x speedup** — flush time dropped from 151s to 21s for a ~14.6 GB test workload ([8455](https://github.com/skypilot-org/skypilot/pull/8455)). Advanced tuning options are available via `data.mount_cached` config ([#8810](https://github.com/skypilot-org/skypilot/pull/8810), [#8831](https://github.com/skypilot-org/skypilot/pull/8831), [#8994](https://github.com/skypilot-org/skypilot/pull/8994)).
<p align="center">
<img width="50%" alt="mount-cached-speedup" src="https://github.com/user-attachments/assets/03dab45c-3694-44ba-948e-166ea8a6450b" />
</p>
Automatic EFA on EKS
SkyPilot now automatically configures **Elastic Fabric Adapter (EFA)** on EKS, delivering **~78.8 GB/s inter-node bandwidth** (vs ~4.1 GB/s without EFA) — critical for distributed training performance ([8557](https://github.com/skypilot-org/skypilot/pull/8557), [#8771](https://github.com/skypilot-org/skypilot/pull/8771)):
yaml
resources:
network_tier: best
<p align="center">
<img width="50%" alt="efa-speedup" src="https://github.com/user-attachments/assets/e28b3d47-5378-41b6-a2eb-b8faf496f21a" />
</p>
Autostop Hooks
A new **autostop hook** mechanism allows running custom scripts before a cluster is automatically stopped — for example, to save checkpoints, sync W&B runs, or send notifications ([8412](https://github.com/skypilot-org/skypilot/pull/8412)):
yaml
resources:
autostop:
idle_minutes: 10
hook: |
wandb sync
curl -X POST $SLACK_WEBHOOK -d '{"text": "Cluster shutting down"}'
hook_timeout: 300
External Links (W&B Integration)
The SkyPilot dashboard now **automatically detects W&B links** and other external URLs generated by your AI workloads — no more digging through job logs ([8405](https://github.com/skypilot-org/skypilot/pull/8405)).
<p align="center">
<img width="60%" alt="External Links" src="https://github.com/user-attachments/assets/26c38623-60e1-4994-aab6-24d1f29249b6" />
</p>
Exit Code-Based Job Recovery
Users can now specify **exit codes that trigger automatic job recovery** in managed jobs, useful for transient failures with known error codes ([8324](https://github.com/skypilot-org/skypilot/pull/8324)):
yaml
resources:
job_recovery:
recover_on_exit_codes: [29]
Windows WSL Support
SkyPilot now **auto-detects WSL** and seamlessly configures VSCode Remote-SSH for Windows users ([8669](https://github.com/skypilot-org/skypilot/pull/8669)).
---