Runhouse

Quick release to allow passing Sky kwargs to ondemand_cluster (978).

**Full Changelog**: https://github.com/run-house/runhouse/compare/v0.0.30...v0.0.31

Highlights
[Alpha] On-demand Docker Clusters
This release adds support for using a base Docker image in conjunction with an [on-demand cluster.](https://www.run.house/docs/tutorials/api-clusters#on-demand-clusters) By specifying the `image_id` field in the format `docker:<registry>/<image>:<tag>` in the cluster factory, the corresponding Docker container will be downloaded when the cluster is launched. The Runhouse server is then started inside the Docker container, ensuring that anything that goes through Runhouse will be run inside the container environment.

For more information on usage, such as setting up environment variables for using private Docker registries, please refer to the [User Guide](https://www.run.house/docs/main/en/docker).

python
docker_cluster = rh.ondemand_cluster(
name="pytorch_cluster",
image_id="docker:nvcr.io/nvidia/pytorch:23.10-py3",
instance_type="CPU:2+",
provider="aws",
)
docker_cluster.up_if_not()


New Features
* Docker cluster support (803, 852, 830, 905)
* Add support for running in a marimo notebook (892)

Bug Fixes
* Handle string system for package `.to` (875)
* Properly save `config.yaml` for default conda env cluster (910)
* Minor fixes (874)

Highlights
This release improves autostop stability and robustness considerably, and introduces the ability to send an env or module to a specific node in a multinode cluster.

Improvements
* Simplify and improve Autostop by rohinb2 and dongreenberg in https://github.com/run-house/runhouse/pull/895, https://github.com/run-house/runhouse/pull/894
* Send env to a specific `node_idx`. by rohinb2 in https://github.com/run-house/runhouse/pull/835
* Update secrets login flow to be more opt-in by carolineechen in https://github.com/run-house/runhouse/pull/880
* Show information about active function calls in cluster.status() by rohinb2 in https://github.com/run-house/runhouse/pull/871 and https://github.com/run-house/runhouse/pull/896

Bugfixes
* [bug] Make `disable_den_auth` actually sync. by rohinb2 in https://github.com/run-house/runhouse/pull/865
* Move config.yaml creation to restart server() by BelSasha in https://github.com/run-house/runhouse/pull/868
* Bump SkyPilot Version to 0.6.0 and fix remote SkyPilot dependencies on Start by dongreenberg in https://github.com/run-house/runhouse/pull/855
* Consolidate periodic loops into one function updating Den and updating autostop. by rohinb2 in https://github.com/run-house/runhouse/pull/873
* Fix cluster factory bug with den_auth clusters not being saved. by rohinb2 in https://github.com/run-house/runhouse/pull/878
* Remove resource conversion check for secrets by carolineechen in https://github.com/run-house/runhouse/pull/881

Docs
* Clarify setup in docs and den quick start by mkandler in https://github.com/run-house/runhouse/pull/876
* Update status docs by BelSasha in https://github.com/run-house/runhouse/pull/889
* Llama 3 vLLM GCP example by mkandler in https://github.com/run-house/runhouse/pull/893
* Fix bug in starting example code block by mkandler in https://github.com/run-house/runhouse/pull/884
* Adds quotes to pip install in examples by mkandler in https://github.com/run-house/runhouse/pull/886
* Update secrets login in api tutorial by carolineechen in https://github.com/run-house/runhouse/pull/882

Testing
* Update multinode cluster fixtures. by rohinb2 in https://github.com/run-house/runhouse/pull/856
* minor changes to cluster status tests by BelSasha in https://github.com/run-house/runhouse/pull/891
* Group status tests together by dongreenberg in https://github.com/run-house/runhouse/pull/899
* Reorganzize default env tests and consolidate fixture into GCP fixture by dongreenberg in https://github.com/run-house/runhouse/pull/900
* Stop overwriting local dotenv in tests. by dongreenberg in https://github.com/run-house/runhouse/pull/901
* Consolidate static cluster fixtures into one by dongreenberg in https://github.com/run-house/runhouse/pull/902
* Change AutostopServlet into AutostopHelper, and test properly by dongreenberg in https://github.com/run-house/runhouse/pull/897
* cluster status scheduler tests by BelSasha in https://github.com/run-house/runhouse/pull/869

**Full Changelog**: https://github.com/run-house/runhouse/compare/v0.0.28...v0.0.29

server pid: 29395
• server port: 32300
• den auth: True
• server connection type: ssh
• backend config:
  • resource subtype: OnDemandCluster
  • use local telemetry: False
  • domain: None

Highlights
Custom cluster default env support and lots of new examples!

Cluster Default Env
Runhouse cluster now supports a `default_env` argument to provide more flexibility and isolation for your Runhouse needs. When you set up a cluster with the default env, Runhouse first installs the env on the cluster (any package installations and setup commands), then starts the Runhouse server inside that env, whether it be a bare metal or even conda env. Future Runhouse calls on/to the cluster, such as `cluster.run(cmd)`, `rh.function(local_fn).to(cluster)`, and so one, will default to run on this default env. Simply pass in any runhouse Env object, including it's package requirements, setup commands, working dir, etc, to the cluster factory.

python
my_default_env = rh.env(
name="my_default_env",
reqs=["pytest", "numpy"],
working_dir="./",
)
my_conda_env = rh.conda_env(name="conda_env1", env_vars={...}) conda env

cluster = rh.cluster(
name="my_cluster",
instance_type="CPU:2+",
default_env=my_default_env, or my_conda_env
)

cluster.run(["pip freeze | grep pytest"]) runs on default_env


Improvements
* Introduce support for custom cluster default env (678, 746, 760)
* Start our own Ray cluster instead of using SkyPilot's (742)
* Exception handling for Module (747)
* Disable timeout in AsyncClient (773)
* Only sync rh config to ondemand cluster (782)

Bug Fixes
* Set CPUs for ClusterServlet to 0 (772)
* previously, cluster servlet was taking up 1 cpu resource on the cluster. set this to zero instead
* Set den_auth default to None in cluster factory (784)
* non-None default argument causes the cluster to reconstruct from scratch (rather than reloaded from rns) if there's a non-None argument mismatch

Docs & Examples
See also [docs](https://www.run.house/docs) and [examples](https://www.run.house/examples) webpages.

New Examples
* Llama3 (741, 743, 744)
* Parallel embedding (759, 779, 783, 792)
* Hyperparameter optimization (770)
* Llama2 fine-tuning with LoRA (771)

New Tutorials
* Async tutorial in docs (768)

Fast-follow Ray bugfix

Bugfixes
* Start our own Ray cluster instead of using SkyPilot's (742)