This is a major update to the redivis-python client library, complete with _significant_ performance improvements, new mechanisms for working with larger tables, and new integrations with pyarrow, polars, and dask. It also includes updates that allow for the generation of output tables within a redivis notebook environment.
- [redivis-python library documentation](https://apidocs.redivis.com/client-libraries/redivis-python)
- [Python notebooks on Redivis](https://docs.redivis.com/reference/projects/notebook-nodes/python-notebooks)
New features 🚀
- New methods on Table, Query, and Upload classes:
- **to_dask_dataframe()**: read a redivis table as a [dask dataframe](https://docs.dask.org/en/stable/dataframe.html)
- **to_polars_lazyframe()**: read a redivis table as a [polars lazyframe](https://pola-rs.github.io/polars/py-polars/html/reference/lazyframe/index.html)
- **to_arrow_table()**: read a redivis table as a [pyarrow table](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html)
- **to_arrow_dataset()**: read a redivis table as a [pyarrow dataset](https://arrow.apache.org/docs/python/generated/pyarrow.dataset.Dataset.html)
- **to_arrow_batch_iterator()**: iterate over a table via [pyarrow RecordBatches](https://arrow.apache.org/docs/python/generated/pyarrow.RecordBatch.html)
- **to_geopandas_dataframe()**: read a redivis table with a geography variable as a [geopandas dataframe](https://geopandas.org/en/stable/docs/reference/api/geopandas.GeoDataFrame.html)
- **to_pandas_dataframe()**: read a redivis table as a [pandas dataframe](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html) for a table with a geography variable, using the new pyarrow dtypes by default (supercedes the old `to_dataframe()` method, which has been deprecated)
- Added `batch_preprocessor` argument to all above functions to efficiently pre-filter data as it is loaded
- Added `dtype_backend` argument to `to_[geo]pandas_dataframe()` methods. Specify either `pyarrow`, `numpy`, or `numpy_nullable`
- Add `date_as_object` boolean argument to_[geo]pandas_dataframe()` methods. Configure how dates should be stored in when using the legacy numpy dtypes.
- Add `redivis.current_notebook()`, returning an instance of the current notebook within a Redivis notebook environment (returns null in other environments)
- Add `Notebook.create_output_table(df)` method to generate an output table from a redivis notebook
Performance âš¡
- Multi-threaded data streaming, alongside performance improvements in the underlying transfer protocol, should lead to at least a 3x improvement in data transfer speeds
- Reduced memory overhead significantly, with new mechanisms to load larger-than-memory data
Deprecations
- The `to_dataframe` method has been deprecated, in favor of the more explicit `to_pandas_dataframe()` and `to_geopandas_dataframe()` methods. The new methods default to the "pyarrow" dtype_backend — to continue using numpy as the pandas backend, you must specify `dtype_backend="numpy"` for the new methods. However, pyarrow is significantly faster, and supports nullable data types. It is recommended if numpy isn't required, and will generally work with existing code.
Dependencies
Bumped the following dependencies:
- `pandas >= 2.0.0`
- `pyarrow >= 13.0.0`
- `pyarrow >= 0.14.0`
Added the following dependencies:
- `dask >= 2023.9.3`
- `polars >= 0.19`
- `shapely >= 2.0.1`