Modin

This release updates the pandas version and fixes some minor bugs.

Bugfixes + Pandas Concordance (🐛 + 🐼)
----------------------------------------
* Fix `fill_value` parameter for `DataFrame.sub`, other binary ops (1109)

Code Quality + Testing 💯
-------------------------
* Make query compiler interface consistent for binary ops (1106)
* Add a bot message to display TeamCity test results (1124)

Dependencies 🔗
---------------
* Add randomly generated redis password for Ray by default (1107)
* Update pandas version to 1.0 and fix compatibility (1081) ⭐️

Contributors this release
-------------------------

The following users contributed code to Modin since the last release.

gshimansky (Maintainer)
devin-petersohn (Maintainer)

🎉🎉 Thank you! 🎉🎉

In this release we focused on improving testing infrastructure and fixing outstanding bugs. One particular longstanding bug with the Dask runtime, related to serialization, was fixed (1096). This revealed an oppourtinity for more optimization when it comes to serialization within the repository.

Bugfixes + Pandas Concordance (🐛 + 🐼)
----------------------------------------
* Fix loc when column mask is boolean and no rows are masked (1039)
* Fix opening a file from s3 when S is capitalized (1045)
* Fix pd.to_datetime when using a Series object. (1048)
* Fix issue with Dask engine where users were not able to use Da (1050)
* Fix `as_index=False` for `DataFrame.groupby` (1041)
* Fix ingesting parquet files that are coming from HDFS (1074)
* fixed bug where .to_frame(name) ignored name; added parity wit (1075)
* Fix column indexing when 2 or more columns have same name (1077)
* Fix Series.__getitem__ to accept a callable as key (1084)
* bugfix - sometimes df.loc[s] applies to columns instead of rows (1088)
* df.rename works with 'mapper' and 'axis' params (1057)
* Fix case where apply a Series across the columns threw error (1092)
* Fix Dask Serialization issue (1096)

New Functionality ✨
--------------------
* Add r operators to Series (1086)

Code Quality + Testing 💯
-------------------------
* Update Test infrastructure to use testmon when possible (1036)
* Re-enable testmon force selection (1040)
* Update master to track coverage again (1052)
* Update test for groupby and clean up groupby edge cases (1055)
* Change Error to warning for pandas version pin (1072)
* Upload coverage correctly from merged PRs (1076)

Documentation 📃
----------------
* Update Installation Documentation (1066)
* Documentation updates (1097)
* Add documentation for signed-off-by policy (1098)

Dependencies 🔗
---------------
* Unpin tables version for CI to support Ubuntu 19+ (1080)
* Pin msgpack < 1.0 (1090)

Contributors this release
-------------------------

The following users contributed code to Modin since the last release.

elonp (First time contributor) ⭐️
KevOBrien (First time contributor) ⭐️
devin-petersohn (Maintainer)

🎉🎉 Thank you! 🎉🎉

Modin 0.7 comes with the largest expansion of the API since the first release. Modin now supports over 83% of the pandas API, up from 71% last release. A number of long awaited features have been implemented to include: I/O support for Dask for parquet and other column stores and `groupby` with a list of column names.

Bugfixes + Pandas Concordance (🐛 + 🐼)
* Allow merging of named Series (879)
* Correctly merge `CategoricalDtype` dtypes (889)
* Fix issue where certain arguments were not defaulting to pandas (890)
* Send full path to workers on `read_csv` (899)
* Remove `__array_prepare__` from Series API (900)
* Add Series.str.title to API (901)
* Fix `df.squeeze` when `axis=0` on a 1x1 dataframe (902)
* Fix `skiprows` logic for `read_csv` (918)
* read_sql() will default to pandas when chuksize is given (920)
* Fix DeprecationWarning: invalid escape sequence \d (950)
* Fix `apply` when `args` is set (953)
* Fix inplace updates on partitions (962)
* Fix bug where certain encodings were throwing an error (980)
* Fix inplace operations without inplace keyword on emtpy dataframes (983)
* Support console with __repr__ like pandas (984)
* Fix `count` when `numeric_only=False` (1002)
* Fix bug in `loc` where slice on columns only threw Exception (1024)

New Functionality ✨
* support for duplicated() and drop_duplicates() (892)
* Create SeriesGroupBy wrapper to default to pandas and return to Modin (908)
* Bring I/O support to Dask for everything supported (955) ⭐️
* Add support for grouping by multiple columns when doing a reduction (987) ⭐️
* Implement `DataFrame.at_time` and `Series.at_time` (991)
* Add implementation for `between_time` for `Series` and `DataFram… (992)
* Implement `combine` for Series and DataFrame (995)
* Add implementation for `combine_first` for `DataFrame` and `Seri… (996)
* Add implementation for `droplevel` for `Series` and `DataFrame` (1000)
* Implement `assign` for `DataFrame` (998)
* Add implementation for `first` for `Series` and `DataFrame` (1006)
* Add implementation for `last` for `DataFrame` and `Series` (1007)
* Add implementation for `swapaxes` for DataFrame and Series (1010)
* Add implementation for `tz_convert` for `Series` and `DataFrame` (1013)
* Implement `tz_localize` for `Series` and `DataFrame` (1014)
* Add implementation for `tshift` (1016)
* Add implementation: `swaplevel` for `Series` and `DataFrame` (1018)
* Add implementation: `reorder_levels` for DataFrame and Series (1022)
* Add implementation: `take` for Series and DataFrame (1020)
* Add implentation: `truncate` for Series and DataFrame (1026)
* Fix bug where Parsing error was thrown when text spanned multip… (1027)


Code Quality + Testing 💯
* Update pytest and clean up tests a bit (903)
* CI updates (924, 925, 926, 927, 928, 929, 930, 931, 933, 936, 937, 938)
* Fix Windows Remove file test error (943)
* Add test script for simple execution of all unit tests (948)
* Fix pyarrow.parquet import in tests (952)
* Support parameters with run-tests.sh (959)
* Fix environment variables for CI and Master test suite (964)
* Make test_dataframe.py more granular with test suite (966)
* Cache pip depdendencies between builds to speed up process (967)
* Fix master CI workflow order and simplify workflow names (968)
* Update master build to allow coverage to be run (969)
* Optimize CI with minimal pip installs (971)
* use versioneer for versioning using VCS (1028)

Backend enhancements + Performance 🚀
* Improve performance of setting a column from an existing one (942)
* Increase `n_workers` for Dask from default to number of cores (965)

Documentation 📃
* Update README to have a more accurate API coverage section (974)
* Move API Coverage section in README to a more appropriate place (975)
* Update README to add advanced usage (988)

Dependencies 🔗
* Enforce only Python3+ on future releases (907)
* Change Coverage version to avoid sqlite3 errors (916)
* Remove top level import of `py` (935)
* Update Ray version to latest (941)
* Restructure import attempt to only try Ray if on a non-windows machine (945)
* Set `pure=False` for Dask `Client.submit` and `hash=False` for `… (957)


Contributors this release

The following users contributed code to Modin since the last release.

ecoughlan (First time contributor) ⭐️
aeroaks (First time contributor) ⭐️
eavidan (Returning contributor) 🌟
devin-petersohn (Maintainer)

🎉🎉 Thank you! 🎉🎉

The 0.6.3 release comes with several bugfixes and code quality improvements. Notably, the CI was moved to GitHub Actions for faster test completion time and better developer experience. There were also new additions to Modin functionality: `level` argument for `any` and `all` and new implementation for Dask readers with `read_csv` and `read_json`.

Bugfixes + Pandas Concordance (🐛 + 🐼)
* Fix level parameter for aggregations when names are `None` (838)
* Fix `map` and `apply` over a `Series` that contains list values (840)
* Fix metadata computation for `get_dummies` (845)
* fix for Python 3.7: re._pattern_type no longer exist (849)
* Fix integer overflow when doing a mask for Windows users (858)
* Fix issue where partition metadata would be off after insert + window (869)
* Fix issue for updating to Categorical Dtypes from `astype` (870)
* Fix psycopg2 connection functionality (736)

New Functionality ✨
* Add support for level argument in any, all (833)
* Add Implementation for Dask `read_csv` and `read_json`. (861)

Code Quality + Testing 💯
* Remove legacy _get_nan_block_id (844)
* ci: upgrade linters (flake8, black) (848)
* add GitHub Actions workflow for CI (850)
* Deprecate Travis CI in favor of GitHub Actions (856)
* Add CODECOV_SECRET to the environment variable to upload Codecov report (857)
* Remove debugging prints and add a test for prints (768)

Backend enhancements + Performance 🚀
* Refactor IO by breaking up into modular readers (754)

Documentation 📃
* Update actions name and create badge on README (854)

Dependencies 🔗
* Update to latest pandas version 0.25.2 (836)
* Upgrade to pandas 0.25.3 (860)

Contributors this release

The following users contributed code to Modin since the last release.

rliu4439 (First time contributor) ⭐️
smola (First time contributor) ⭐️
RehanSD (Returning contributor) 🌟
devin-petersohn (Maintainer)

🎉🎉 Thank you! 🎉🎉

This release contains a large number of new functionality for our internal Modin DataFrame abstraction, as well as many backend enhancements that dramatically improve performance.

We attempted to update to the latest version of Ray, but due to some API changes and changes in behavior, we had to revert that change until we can verify that this version of Ray will work with the large memory workloads of Modin.

Bugfixes + Pandas Concordance (🐛 + 🐼)
* Empty call queue after draining (808)

New Functionality ✨
* Add MapReduce to the functions and register (812)
* Initial addition of Fold Function in Pandas Query Compiler (827)

Backend enhancements + Performance 🚀
* Create `modin_frame.filter_full_axis` and update `query` and `dropna` to use (815)
* Add fast path for `concat` when the axes and partitioning is aligned (818)
* Remove dtype requirement from `head`, `tail`, etc. (819)
* Add mean to the Reductions list (825)
* Adding `quantile` to list of Reductions (826)
* Adding hooks to be able to custom distribute non-pandas objects (813)

Documentation 📃
* Fix build issue for documentation (811)

Dependencies 🔗
* Update Ray version to 0.7.5 (822)
* Revert "Update Ray version to 0.7.5 (822)" (831)

Contributors this release

The following users contributed code to Modin since the last release.

williamma12 (Maintainer)
devin-petersohn (Maintainer)

🎉🎉 Thank you! 🎉🎉

This release includes several bugfixes and improvements to the backend. It also fixes support for windows users and adds new install targets. See the README for more information.

Bugfixes + Pandas Concordance (🐛 + 🐼)
* Update components of groupby when level is being used (779)
* Support multiple `names` values for groupby on MultiLevelIndex (788)
* Update `Series.map` to support dictionary of operations and support (787)
* Fix `groupby` + `reduce` for partitions with only 1 row (795)
* Fix `DataFrame.apply` when the result of the `apply` is a `Series` (796)
* Update `any()` in groupby.py to accept arguments (802)

New Functionality ✨
* Support Windows with the proper requirements (780)

Backend enhancements + Performance 🚀
* Improve and correct metadata management internally (792)

Documentation 📃
* Update README to include more information (781)
* Fixup a broken docs link (799)

Dependencies 🔗
* Move pyarrow import after Ray (794)
* Fix import (804)
* Update io.py (784)

Contributors this release

The following users contributed code to Modin since the last release.

sbrugman (First time contributor) 🌟
xrmx (First time contributor) 🌟
rliu4439 (First time contributor) 🌟
williamma12 (Maintainer)
devin-petersohn (Maintainer)

🎉🎉 Thank you! 🎉🎉