Highlights - Support conversion between Spark Dataframe and Ray Dataset - Support Ray 1.7.0 - Support Spark 3.2.0 - Various bug fixes and improvements
Thanks ConeyLiu zuston kira-lin mjschock edoakes carsonwang for their contributions to the release!
0.3.0
- Spark dynamic resource allocation support. This allows you to launch Spark external shuffle service on Ray and enable Spark dynamic resource allocation to maximize your resource utilization. - Spark-submit support. A command line utility bin/raydp-submit is provided for you to submit a scala/java/python Spark application to a Ray cluster. - MPI on Ray. This allows you to run MPI jobs on Ray. You can use this feature to construct pipelines like Spark + MPI on Ray. - Ray 1.3.0 support.
0.2.0
Several bug fixes. And also we have added some examples to show how RayDP works together with other libraries, such as PyTorch, Tensorflow, XGBoost, and Horovod.