Highlights
Changes to reqs, packages, and env installation
More explicit sycning behavior between local and remote development, and more flexible support for package installations on the cluster.
**`working_dir` env argument**
Previously, you could specify a "working directory" as part of your Runhouse env to sync over to a cluster. By default, this working directory is based on the folder that you are running your Runhouse Python code from (perhaps totally disconnected from the location of the coe you're sending over). This behavior changes in this release, and we now instead sync over the package in which the local class or function being sent to the cluster belongs. This eliminates unexpected sharp edges users encountered with working_dir setting, and we look to deprecate working_dir sometime soon (though this should not be disruptive). To specify folders to sync, one can directly pass local paths into the `reqs` of the Runhouse env.
env = rh.env(
name="env_with_local_package",
reqs=["pandas", "torch", "~/path/to/package"],
)
**Local path detection**
When sending over a function or module that imports local code to a remote cluster, we now automatically detect [1] the package on the filesystem corresponding to that code, add it to the env requirements, and sync it over to the cluster.
**Local package detection**
You can now pass the name of a local Python package, whether installed from source or editably, and we'll detect the correct directory to sync over. This is important for teams collaborating on shapred repos, who may not have their clones saved in the same place.
**Increased package installation support**
Pip install will now work properly for local folders, in addition to package strings. Including a requirement that is installed locally will also sync appropriately to the cluster.
* Pip install method will correctly work for local folders as well (998)
* Importing from a given file will now detect [1] a package on the filesystem to sync based on that file, add that to the envs reqs, and then sync it (914)
Updated Connection Flow
Improvements to detecting and checking the HTTP server connection. Improved and faster approach to fallbacks and retries when unable to reach the client. As part of the connection refactor, ondemand clusters are no longer automatically brought up when there is a call made to/on it. Rather, one needs to manually call `up()` or `up_if_not()` to bring up the cluster if it is not already running.
Deprecations
Remove support and clean up docs for features that are no longer maintained. See release notes below for complete list.
Release Notes
Improvements
**Reqs, Packages & Envs Revamp**
* Automatically add module pointers and reqs to env if they don't exist (914, 968)
* Change pointers to store absolute path (989)
* Insert at the beginning of sys.path for packages (990)
* Including a req that is installed locally will now sync appropriately (997)
* Support pip installing folders (998)
**Connection Flow**
* Fix tunnel hanging at termination (957, 959)
* Convert cluster client to a property (975)
* Add wrapper for running client methods (981)
* Move check server logic into cluster client call (982, 983)
* Handle Ray start in runhouse restart more reliably (1059)
**Other**
* Expand supported Ray versions (720)
* Handle SSH secret syncing in login (887)
* Add log support for clusters with port forwarding (928)
* Remove use of pexpect from password cluster login (940)
* Improve and wrap conda install command on the cluster (954)
* Add runhouse ssh support to local unsaved clusters (1012)
* Check IPs for ondemand clusters when loading from Den (1018)
* Add memory percent to cluster status (1030)
* Don’t log full config (1063)
Bug Fixes
* Propagate self.env if it was set for a module (916)
* Previously, if you pass reqs to a resource constructor as a list, it would get wiped when actually running `.to`
* Update conda env run commands to handle bash commands (920)
* Update config comparison check in cluster factory (955)
* Due to string v int mismatch in ips, sometimes cluster would reconstruct rather than load from Den
* Quote pip installs correctly (1032)
* Previously, package strings like `numpy==1.24.6` wouldnt install correctly due to parsing
* Pass through missing k8s cluster factory args (1067)
BC-Breaking
**Package & Envs**
* Locate working dir in breath first approach (908)
* Search for local of module based on it's file location, rather than runhouse or current working directory
* Remove default working dir from `\.` to `None` (915)
* Working directory is no longer synced over by default when sending to a cluster. Instead, it can be specified by passing in the corresponding Package in the env reqs
* Selectively sync runhouse for default restart behavior (1020)
* Only resync runhouse during restart server if it is a local editable package, or if it is explicitly set to True
**`cluster.run()`**
* Disallow passing node when on cluster (946)
* Remove run name support for running via ssh (936)
* Remove port_forward argument (934)
**Deprecations**
* Remove telemetry APIs and otel requirements (986)
* Remove KVstore, mapper, tables, and queue (994)
* Remove Run from docs (1011)
* Deprecate `system` arg from function and module factories
**Other**
* No longer automatically up an ondemand cluster when performing a call on it
* Previously, when making a call to or on an ondemand cluster that is not already running, Runhouse automatically brings up the cluster. This behavior changes in this release, and one will need to add run `cluster.up()` or `cluster.up_if_not()` to start the ondemand cluster.
Doc & Examples
* Add llama 3 fine tuning example (939, 942)
* Update module API example (950)
* Update architecture overview (1036)
* Add `up_if_not()` in examples and tutorials (999, 1009)
* Minor fixes in docs (1008, 1040, 1057)
[1] The detection algorithm for what to sync when you have an importable function or module is as follows:
* Start with the file that contains this importable function or module on the filesystem
* Go up directories till you find any of the following config files ".git", "setup.py", "setup.cfg", "pyproject.toml", "requirements.txt"