Modin

This release contains a few key bugfixes and pandas version update.

Key Features and Updates
------------------------
* FIX-4376: Upgrade pandas to 1.4.2 (4377)
* FIX-4390: Add redis to Modin dependencies (4396)
* FIX-3527: Fix parquet partitioning issue causing negative row length partitions (4368)
* FIX-4330: Override the memory limit to start ray 1.11.0 on Macs. (4335)
* FIX-4394: Fix issue with multiindex metadata desync (4395)
* FIX-4373: fix usage of 'read_csv_glob' with 'usecols' parameter (4405)
* FIX-4425: Add parameters to groupby pct_change. (4429)

Contributors
------------
Garra1980, devin-petersohn, dchigarev, jeffreykennethli, mvashishtha, YarShev, anmyachev

This release contains significant upgrades to Developer API, as well as to Modin's documentation,
some refactor codebase and performance enhancements, and multiple bugfixes.

Key Features and Updates
------------------------

* Stability and Bugfixes
* FIX-https://github.com/modin-project/modin/issues/4058: Allow pickling empty dataframes and series (https://github.com/modin-project/modin/pull/4095)
* FIX-https://github.com/modin-project/modin/issues/4136: Fix exercise_3.ipynb example notebook (https://github.com/modin-project/modin/pull/4137)
* FIX-https://github.com/modin-project/modin/issues/4105: Fix names of pandas options to avoid `OptionError` (https://github.com/modin-project/modin/pull/4109)
* FIX-https://github.com/modin-project/modin/issues/3417: Fix read_csv with skiprows and header parameters (https://github.com/modin-project/modin/pull/3419)
* FIX-https://github.com/modin-project/modin/issues/4142: Fix OmniSci enabling (https://github.com/modin-project/modin/pull/4146)
* FIX-https://github.com/modin-project/modin/issues/4162: Use `skipif` instead of `skip` for compatibility with pytest 7.0 (https://github.com/modin-project/modin/pull/4163)
* FIX-https://github.com/modin-project/modin/issues/4158: Do not print OmniSci logs to stdout by default (https://github.com/modin-project/modin/pull/4159)
* FIX-https://github.com/modin-project/modin/issues/4177: Support read_feather from pathlike objects (https://github.com/modin-project/modin/issues/4177)
* FIX-https://github.com/modin-project/modin/issues/4234: Upgrade pandas to 1.4.1 (https://github.com/modin-project/modin/pull/4235)
* FIX-https://github.com/modin-project/modin/issues/3368: support unsigned integers in OmniSci backend (https://github.com/modin-project/modin/pull/4256)
* FIX-https://github.com/modin-project/modin/issues/4057: Allow reading an empty parquet file (https://github.com/modin-project/modin/pull/4075)
* FIX-https://github.com/modin-project/modin/issues/3884: Fix read_excel() dropping empty rows (https://github.com/modin-project/modin/pull/4161)
* FIX-https://github.com/modin-project/modin/issues/4257: Fix Categorical() for scalar categories (https://github.com/modin-project/modin/pull/4258)
* FIX-https://github.com/modin-project/modin/issues/4300: Fix Modin Categorical column dtype categories (https://github.com/modin-project/modin/pull/4276)
* FIX-https://github.com/modin-project/modin/issues/4208: Fix lazy metadata update for `PandasDataFrame.from_labels` (https://github.com/modin-project/modin/pull/4209)
* FIX-https://github.com/modin-project/modin/issues/3981, FIX-https://github.com/modin-project/modin/issues/3801, FIX-https://github.com/modin-project/modin/issues/4149: Stop broadcasting scalars to set items (https://github.com/modin-project/modin/pull/4160)
* FIX-https://github.com/modin-project/modin/issues/4185: Fix rolling across column partitions (https://github.com/modin-project/modin/pull/4262)
* FIX-https://github.com/modin-project/modin/issues/4303: Fix the syntax error in reading from postgres (https://github.com/modin-project/modin/pull/4304)
* FIX-https://github.com/modin-project/modin/issues/4308: Add proper error handling in df.set_index (https://github.com/modin-project/modin/pull/4309)
* FIX-https://github.com/modin-project/modin/issues/4056: Allow an empty parse_date list in `read_csv_glob` (https://github.com/modin-project/modin/pull/4074)
* FIX-https://github.com/modin-project/modin/issues/4312: Fix constructing categorical frame with duplicate column names (https://github.com/modin-project/modin/pull/4313).
* FIX-https://github.com/modin-project/modin/issues/4314: Allow passing a series of dtypes to astype (https://github.com/modin-project/modin/pull/4318)
* FIX-https://github.com/modin-project/modin/issues/4310: Handle lists of lists of ints in read_csv_glob (https://github.com/modin-project/modin/pull/4319)
* FIX-https://github.com/modin-project/modin/issues/4138, FIX-https://github.com/modin-project/modin/issues/4009: remove redundant sorting in the internal
* Performance enhancements
* FIX-https://github.com/modin-project/modin/issues/4138, FIX-https://github.com/modin-project/modin/issues/4009: remove redundant sorting in the internal '.mask()' flow (https://github.com/modin-project/modin/pull/4140)
* FIX-https://github.com/modin-project/modin/issues/4183: Stop shallow copies from creating global shared state. (https://github.com/modin-project/modin/pull/4184)
* Benchmarking enhancements
* FIX-https://github.com/modin-project/modin/issues/4221: add `wait` method for `PandasOnRayDataframeColumnPartition` class (https://github.com/modin-project/modin/pull/4231)
* Refactor Codebase
* REFACTOR-https://github.com/modin-project/modin/issues/3990: remove code duplication in `PandasDataframePartition` hierarchy (https://github.com/modin-project/modin/pull/3991)
* REFACTOR-https://github.com/modin-project/modin/issues/4229: remove unused `dask_client` global variable in `modin\pandas\__init__.py` (https://github.com/modin-project/modin/pull/4230)
* REFACTOR-https://github.com/modin-project/modin/issues/3997: remove code duplication for `broadcast_apply` method (https://github.com/modin-project/modin/pull/3996)
* REFACTOR-https://github.com/modin-project/modin/issues/3994: remove code duplication for `get_indices` function (https://github.com/modin-project/modin/pull/3995)
* REFACTOR-https://github.com/modin-project/modin/issues/4331: remove code duplication for `to_pandas`, `to_numpy` functions in `QueryCompiler` hierarchy (https://github.com/modin-project/modin/pull/4332)
* REFACTOR-https://github.com/modin-project/modin/issues/4213: Refactor `modin/examples/tutorial/` directory (https://github.com/modin-project/modin/pull/4214)
* REFACTOR-https://github.com/modin-project/modin/issues/4206: add assert check into `__init__` method of `PandasOnDaskDataframePartition` class (https://github.com/modin-project/modin/pull/4207)
* REFACTOR-https://github.com/modin-project/modin/issues/3900: add flake8-no-implicit-concat plugin and refactor flake8 error codes (https://github.com/modin-project/modin/pull/3901)
* REFACTOR-https://github.com/modin-project/modin/issues/4093: Refactor base to be smaller (https://github.com/modin-project/modin/pull/4220)
* REFACTOR-https://github.com/modin-project/modin/issues/4047: Rename `cluster` directory to `cloud` in examples (https://github.com/modin-project/modin/pull/4212)
* REFACTOR-https://github.com/modin-project/modin/issues/3853: interacting with Dask interface through `DaskWrapper` class (https://github.com/modin-project/modin/pull/3854)
* REFACTOR-https://github.com/modin-project/modin/issues/4322: Move is_reduce_fn outside of groupby_agg (https://github.com/modin-project/modin/pull/4323)
* Pandas API implementations and improvements
* FEAT-https://github.com/modin-project/modin/issues/3603: add experimental `read_custom_text` function that can read custom line-by-line text files (https://github.com/modin-project/modin/pull/3441)
* FEAT-https://github.com/modin-project/modin/issues/979: Enable reading from SQL server (https://github.com/modin-project/modin/pull/4279)
* Developer API enhancements
* FEAT-https://github.com/modin-project/modin/issues/4245: Define base interface for dataframe exchange protocol (https://github.com/modin-project/modin/pull/4246)
* FEAT-https://github.com/modin-project/modin/issues/4244: Implement dataframe exchange protocol for OmnisciOnNative execution (https://github.com/modin-project/modin/pull/4269)
* FEAT-https://github.com/modin-project/modin/issues/4144: Implement dataframe exchange protocol for pandas storage format (https://github.com/modin-project/modin/pull/4150)
* FEAT-https://github.com/modin-project/modin/issues/4342: Support `from_dataframe`` for pandas storage format (https://github.com/modin-project/modin/pull/4343)
* Update testing suite
* TEST-https://github.com/modin-project/modin/issues/3628: Report coverage data for `test-internals` CI job (https://github.com/modin-project/modin/pull/4198)
* TEST-https://github.com/modin-project/modin/issues/3938: Test tutorial notebooks in CI (https://github.com/modin-project/modin/pull/4145)
* TEST-https://github.com/modin-project/modin/issues/4153: Fix condition of running lint-commit and set of CI triggers (https://github.com/modin-project/modin/pull/4156)
* TEST-https://github.com/modin-project/modin/issues/4201: Add read_parquet, explode, tail, and various arithmetic functions to asv_bench (https://github.com/modin-project/modin/pull/4203)
* Documentation improvements
* DOCS-https://github.com/modin-project/modin/issues/4077: Add release notes template to docs folder (https://github.com/modin-project/modin/pull/4078)
* DOCS-https://github.com/modin-project/modin/issues/4082: Add pdf/epub/htmlzip formats for doc builds (https://github.com/modin-project/modin/pull/4083)
* DOCS-https://github.com/modin-project/modin/issues/4168: Fix rendering the examples on troubleshooting page (https://github.com/modin-project/modin/pull/4169)
* DOCS-https://github.com/modin-project/modin/issues/4151: Add info in troubleshooting page related to Dask engine usage (https://github.com/modin-project/modin/pull/4152)
* DOCS-https://github.com/modin-project/modin/issues/4172: Refresh Intel Distribution of Modin paragraph (https://github.com/modin-project/modin/pull/4175)
* DOCS-https://github.com/modin-project/modin/issues/4173: Mention strict channel priority in conda install section (https://github.com/modin-project/modin/pull/4178)
* DOCS-https://github.com/modin-project/modin/issues/4176: Update OmniSci usage section (https://github.com/modin-project/modin/pull/4192)
* DOCS-https://github.com/modin-project/modin/issues/4027: Add GIF images and chart to Modin README demonstrating speedups (https://github.com/modin-project/modin/pull/4232)
* DOCS-https://github.com/modin-project/modin/issues/3954: Add Dask example notebooks (https://github.com/modin-project/modin/pull/4139)
* DOCS-https://github.com/modin-project/modin/issues/4272: Add bar chart comparisons to quick start guide (https://github.com/modin-project/modin/pull/4277)
* DOCS-https://github.com/modin-project/modin/issues/3953: Add docs and notebook examples on running Modin with OmniSci (https://github.com/modin-project/modin/pull/4001)
* DOCS-https://github.com/modin-project/modin/issues/4280: Change links in jupyter notebooks (https://github.com/modin-project/modin/pull/4281)
* DOCS-https://github.com/modin-project/modin/issues/4290: Add changes for OmniSci notebooks (https://github.com/modin-project/modin/pull/4291)
* DOCS-https://github.com/modin-project/modin/issues/4241: Update warnings and docs regarding defaulting to pandas (https://github.com/modin-project/modin/pull/4242)
* DOCS-https://github.com/modin-project/modin/issues/3099: Fix `BasePandasDataSet` docstrings warnings (https://github.com/modin-project/modin/pull/4333)
* DOCS-https://github.com/modin-project/modin/issues/4339: Reformat I/O functions docstrings (https://github.com/modin-project/modin/pull/4341)
* DOCS-https://github.com/modin-project/modin/issues/4336: Reformat general utilities docstrings (https://github.com/modin-project/modin/pull/4338)
* Dependencies
* FIX-https://github.com/modin-project/modin/issues/4113, FIX-https://github.com/modin-project/modin/issues/4116, FIX-https://github.com/modin-project/modin/issues/4115: Apply new `black` formatting, fix pydocstyle check and readthedocs build (https://github.com/modin-project/modin/pull/4114)
* TEST-https://github.com/modin-project/modin/issues/3227: Use codecov github action instead of bash form in GA workflows (https://github.com/modin-project/modin/pull/3226)
* FIX-https://github.com/modin-project/modin/issues/4115: Unpin `pip` in readthedocs deps list (https://github.com/modin-project/modin/pull/4170)
* TEST-https://github.com/modin-project/modin/issues/4217: Pin `Dask<2022.2.0` as a temporary fix of CI (https://github.com/modin-project/modin/pull/4218)

Contributors
------------

prutskov, amyskov, paulovn, anmyachev, YarShev, RehanSD, devin-petersohn,
dchigarev, Garra1980, mvashishtha, naren-ponder, jeffreykennethli, dorisjlee, Rubtsowa

This release contains a few key bugfixes and pandas version update.

Key Features and Updates
------------------------
* Stability and Bugfixes
* Stop shallow dataframe copies from creating global shared state (4184)
* Make PandasOnRayDataframeColumnPartition conformant to partition interface (4231)
* Fix lazy metadata update for PandasDataFrame.from_labels (4209)
* Fix Categorical() for scalar categories (4258)
* Fix some cases when assigning a scalar to a subset of dataframe or series. (4160)
* Align read_excel() behaviour on empty rows with pandas 1.3+ (4161)
* Allow reading an empty parquet file. (4075)
* Pin Dask<2022.2.0 as a temporary fix. (4218)
* Add proper error handling in df.set_index. (4309)
* Documentation improvements
* Clarify OmniSci activation in its usage section. (4192)
* Upgrade pandas to 1.4.1 (4235)

Contributors
------------
mvashishtha anmyachev prutskov devin-petersohn naren-ponder YarShev Garra1980

This release contains documentation polishing and small user experience
improvements.

Key Features and Updates
------------------------
* Mention strict channel priority in conda install section (4178)
* Refresh Intel Distribution of Modin paragraph (4175)
* Add info in troubleshooting page related to Dask engine usage (4152)
* Do not print OmniSci logs to stdout by default (4159)
* Fix rendering the examples on troubleshooting page (4169)
* Use skipif instead of skip for compatibility with pytest 7.0 (4163)

Contributors
------------
RehanSD, YarShev, dchigarev, prutskov, Garra1980

This release contains a few key bugfixes and updates to the documentation.

Key Features and Updates
------------------------
* Stability and Bugfixes
* FIX-4058: Allow pickling empty dataframes and series (4095)
* FIX-4105: Fix names of pandas options to avoid `OptionError` (4109)
* FIX-4142: Fix OmniSci enabling (4146)
* Documentation improvements
* DOCS-4082: Add pdf/epub/htmlzip formats for doc builds (4083)
* DOCS-4079: Fix link to `PandasDataframe` in docs (4108)

Contributors
------------
prutskov, paulovn, YarShev, RehanSD, devin-petersohn,
mvashishtha

This release contains significant upgrades to Modin's documentation,
support for pandas 1.4, new algebra and partitioning layer APIs, and some bugfixes.

Key Features and Updates
------------------------
* Stability and bugfixes
* Support for subscripting Resampler (1a1edfd)
* Fix groupby with column name for `by` (a04d7b7)
* Workaround for groupby with `sort=False` with categorical keys (c67a7c5)
* Align default value of `REDIS_PASSWORD` with Ray's `DEFAULT_REDIS_PASSWORD` (f79cb85)
* Fix groupby dictionary aggregation when `by` and columns to aggregate overlap (d42c070)
* Fix `read_csv` when callables are provided for `skip_rows` parameter (7c84758)
* Ensure address is not passed to `ray.init` when running Ray in local mode (02a23d4)
* Ensure that `groupby.indices` returns positional indices (e9c06f2)
* Fix setting of categorical values (0e36e22)
* Ensure `df.__getitem__` respects step attribute of slice (7e85c5d)
* Ensure data argument is delievered to the Dataframe in experimental cloud mode (2f7da1f)
* Fix assigning to a Series with a single item (0d9d14e)
* Fix the default to pandas in pd.DataFrame.sparse.from_spmatrix (ab2855b)
* Fix `apply` result type inference (ac17ca1)
* Exclude "scripts" from setup package (6224aba)
* Fix assigning a Categorical to a column (cb4e727)
* Ensure `df.to_csv` propagates metadata (e.g. index) (154697b)
* Update `pyarrow` requirement in environment files (b55b08d)
* Performance enhancements
* Optimize `__getitem__` flow for .loc/.iloc (0947ee8)
* Delay instantiation of lazy `dtypes` on transpose (cd8db0c)
* Benchmarking enhancements
* Update benchmarks for groupby that are more representative (0582aa2)
* Refactor Codebase
* Update CODEOWNERS to reflect repository after refactor (cde6390)
* Remove duplicate import of `FactoryDispatcher` in Modin experimental pandas IO (2cfabaf)
* Update Modin to incorporate dataframe algebra (58bbcc3)
* Pandas API implementations and improvements
* Add support for `storage_options` argument to `read_csv_glob` (7c33afe)
* Add support for `dropna` argument for `groupby.indices` and `groupby.groups` (144a613)
* Ensure relabeling Modin Frame does not lose partition shape (3c740db)
* Update `Series.values` to default to `to_numpy()` (67228ef)
* Add support for `modin.pandas.show_versions` and `python -m modin --versions` (efe717f)
* Upgrade pandas support to 1.4 (39fbc57)
* OmniSci enhancements
* Update benchmarks for groupby that are more representative (9396f23)
* Update documentation on Native + OmniSci (edc1608)
* Add support for `getArrowTable()` (6882ec2)
* Fix segfault during `init` when only OmniSci is present (8c8a6a3)
* Optimize `append` with default arguments (67013f9)
* Fix OmniSci engine enabling for IO functions (9d1a334)
* XGBoost enhancements
* Developer API enhancements
* Add parameter for minimum partition size (1be66d1)
* Improve documentation for `read_csv_glob` and ensure warning raised if wildcard not in `filepath_or_buffer` (be10ba9)
* Expand virtual partitioning utility (8d1004f)
* Update testing suite
* Documentation improvements
* Improve documentation on pandas on Ray execution (b76dc57)
* Reformat documentation to match pandas documentation theme (cc96f5d)
* Improve documentation on pandas on Python execution (d590de0)
* Improve System view in architecture documentation (6d51921)
* Improve documentation on using pandas on Dask (003f338)
* Improve documentation on pandas on Dask execution (61bf043)
* Add documentation on using pandas on Python (195b668)
* Improve Modin Out of Core documentation (cf426c4)
* Improve documentation on OmniSci on native execution (689faee)
* Improve documentation on IO (ffa67c7)
* Add documentation on factories and parsers (6ca66db)
* Improve documentation for experimental pandas on Ray execution (20abddd)
* Improve documentation for `modin.core.dataframe.base` and `modin.core.dataframe.pandas` (cf1e541)
* Update troubleshooting documentation and add FAQs (cc95ae2)
* Improve README introduction and installation sections (a632d1f)
* Update copyright year (7da1dc8)
* Update a link to `pandas.read_json` (0315823)
* Improve documentation for Modin vs. Dask (34732cb)
* Fix links to the contributing page (81a06d6)
* Remove broken links from supported apis (c04502d)
* Change docs copyright statement to 'Modin Developers' (ed2a7a4)
* Rename Developer page to Development in docs (406af7c)
* Improve "Getting Started" section (4a62bba)
* Update Modin tutorials (76707bf)
* Add back quickstart notebook (4dd97ab)
* Fix links in README and update README and FAQs (5d84042)
* Update Modin module layout in architecture docs (7fcafa7)
* Update documentation with new algebra operators and `ModinDataframe` (4b70725)
* Add usage guide to documentation (4511566)
* Build docs with Python 3.8 (01c1876)
* Dependencies
* Update PyArrow to 6.0 and OmniSci to 5.10.1 (018515f)

Contributors
------------
anmyachev, prutskov, Rubtsowa, vnlitvinov, dchigarev, YarShev, amyskov,
mvashishtha, dorisjlee, devin-petersohn, jeffreykennethli, RehanSD,
novichkovg, Lozovskii-Aleksandr, naren-ponder, ahallermed, fexolm,
adityagp, susmitpy, ienkovich