Scikit-learn-intelex

The release **Intel® Extension for Scikit-learn 2021.7.1** introduces the following changes:

📚 Support Materials
- [[Tabular Playground Series - Sep 2022] Tuning of ElasticNet hyperparameters](https://www.kaggle.com/code/alex97andreev/tuning-of-elasticnet-hyperparameters)
- [Accelerated Random Forest for Rent Prediction](https://www.kaggle.com/code/alex97andreev/accelerated-random-forest-for-rent-prediction)

🚨 What's New
- oneAPI interface for kNN regression
- Fix for wrong column names of pandas DataFrame in `sklearn.model_selection.train_test_split` patched function

The release **Intel® Extension for Scikit-learn 2021.6** introduces the following changes:

📚 Support Materials

Kaggle kernels:
- [Fast Feature Importance using scikit-learn-intelex](https://www.kaggle.com/lordozvlad/fast-feature-importance-using-scikit-learn-intelex)
- [[Tabular Playground Series - December 2021] Fast Feature Importance with sklearnex](https://www.kaggle.com/lordozvlad/tps-dec-fast-feature-importance-with-sklearnex?scriptVersionId=82254284)
- [[Tabular Playground Series - December 2021] SVC with sklearnex 20x speedup](https://www.kaggle.com/alexeykolobyanin/tps-dec-svc-with-sklearnex-20x-speedup)
- [[Tabular Playground Series - January 2022] Fast PyCaret with Scikit-learn-Intelex](https://www.kaggle.com/lordozvlad/tps-jan-fast-pycaret-with-scikit-learn-intelex)
- [[Tabular Playground Series - February 2022] KNN with sklearnex 13x speedup](https://www.kaggle.com/alexeykolobyanin/tps-feb-knn-with-sklearnex-13x-speedup)
- [Fast SVM for Sparse Data from NLP Problem](https://www.kaggle.com/alex97andreev/fast-svm-for-sparse-data-from-nlp-problem)
- [Introduction to scikit-learn-intelex](https://www.kaggle.com/lordozvlad/introduction-to-scikit-learn-intelex)
- [[Datasets] Fast Feature Importance using sklearnex](https://www.kaggle.com/lordozvlad/datasets-fast-feature-importance-using-sklearnex)
- [[Tabular Playground Series - March 2022] Fast workflow using scikit-learn-intelex](https://www.kaggle.com/lordozvlad/tps-mar-fast-workflow-using-scikit-learn-intelex)

🛠️ Library Engineering

- Reduced the size of oneDAL python run-time package by approximately 8%
- Added Python 3.10 support for daal4py and Intel(R) Extension for Scikit-learn packages

🚨 What's new

- Improved performance for the following Intel® Extension for Scikit-learn algorithms:

- t-SNE for “Burnes-Hut” algorithm

- Introduced new functionality for Intel® Extension for Scikit-learn:

- Manhattan, Minkowski, Chebyshev and Cosine distances for KNeighborsClassifier and NearestNeighbors with “brute” algorithm

- Fixed the following issues in Intel® Extension for Scikit-learn:

- An issue with the search of common data type in pandas DataFrame
- Patching overhead of finiteness checker for specific small data sizes
- Incorrect values in a tree visualization with `plot_tree` function in RandomForestClassifier
- Unexpected error for device strings in `{device}:{device_index}` format while using config context

The release **Intel® Extension for Scikit-learn 2021.5** introduces the following changes:

📚 Support Materials
- Kaggle kernels:
- [Tabular Playground Series - Sep 2021] [Ridge with sklearn-intelex 2x speedup](https://www.kaggle.com/alexeykolobyanin/tps-sep-ridge-with-sklearn-intelex-2x-speedup)
- [Tabular Playground Series - Oct 2021] [Fast AutoML with Intel Extension for Scikit-learn](https://www.kaggle.com/lordozvlad/fast-automl-with-intel-extension-for-scikit-learn/notebook)
- [Titanic – Machine Learning from Disaster] [AutoML with Intel Extension for Sklearn](https://www.kaggle.com/lordozvlad/titanic-automl-with-intel-extension-for-sklearn/notebook)
- [Tabular Playground Series - Nov 2021] [AutoML with Intel® Extension](https://www.kaggle.com/lordozvlad/tps-nov-automl-with-intel-extension)
- [Tabular Playground Series - Nov 2021] [Log Regression with sklearnex 17x speedup](https://www.kaggle.com/alexeykolobyanin/tps-nov-log-regression-with-sklearnex-17x-speedup)
- [Tabular Playground Series - Dec 2021] [SVC with sklearnex 20x speedup](https://www.kaggle.com/alexeykolobyanin/tps-dec-svc-with-sklearnex-20x-speedup)
- [Tabular Playground Series - Dec 2021] [Fast Feature Importance with sklearnex](https://www.kaggle.com/lordozvlad/tps-dec-fast-feature-importance-with-sklearnex)
- Added [demo samples](https://github.com/intel/scikit-learn-intelex/tree/rls/2021.5-rls/examples/notebooks) of the Intel® Extension for Scikit-learn usage with the performance comparison to original Scikit-learn for ElasticNet, K-means, Lasso Regression, Linear regression, and Ridge Regression
- Added [demo samples](https://github.com/intel/scikit-learn-intelex/tree/rls/2021.5-rls/examples/notebooks/NYCTaxi-E2E-Demo) of the Modin usage

🛠️ Library Engineering
- Reduced the size of oneDAL library by approximately ~15%, this is a required dependency of Intel® extension for scikit learn.

🚨 New Features
- Scikit-learn 1.0 support

🚀 ​Improved performance
- [GPU] `KNN` algorithm prediction
- [GPU] `SVC` and `SVR` algorithms training

🐛 Bug Fixes
- Stabilized the results of Linear Regression in oneDAL and Intel® Extension for Scikit-learn
- Fixed an issue with RPATH on MacOS

The release **Intel(R) Extension for Scikit-learn 2021.4** introduces the following changes:

📚 Support Materials
- Medium blogs:
- [Save Time and Money with Intel Extension for Scikit-learn](https://medium.com/intel-analytics-software/save-time-and-money-with-intel-extension-for-scikit-learn-33627425ae4)
- Anaconda blogs:
- [Scikit-learn Speed-up with Intel and Anaconda](https://www.anaconda.com/blog/scikit-learn-speed-up-with-intel-and-anaconda)
- Oracle blogs:
- [Accelerate your model build process with the Intel® Extension for Scikit-learn](https://blogs.oracle.com/ai-and-datascience/post/model-build-intel-extension-scikit-learn)
- Kaggle kernels:
- [Tabular Playground Series - Jun 2021] [Fast LogReg with scikit-learn-intelex](https://www.kaggle.com/owerbat/tps-jun-fast-stacking-with-scikit-learn-intelex)
- [Tabular Playground Series - Jun 2021] [AutoGluon with sklearnex](https://www.kaggle.com/alex97andreev/tps-jun-autogluon-with-sklearnex)
- [Tabular Playground Series - Jul 2021] [Fast RandomForest with sklearnex](https://www.kaggle.com/pahandrovich/tps-jul-2021-fast-randomforest-with-sklearnex)
- [Tabular Playground Series - Jul 2021] [RF with Intel Extension for Scikit-learn](https://www.kaggle.com/alexeykolobyanin/tps-jul-rf-with-intel-extension-for-scikit-learn)
- [Tabular Playground Series - Jul 2021] [Stacking with scikit-learn-intelex](https://www.kaggle.com/alexeykolobyanin/tps-jul-stacking-with-scikit-learn-intelex)
- [Tabular Playground Series - Aug 2021] [NuSVR with Intel Extension for Sklearn](https://www.kaggle.com/alexeykolobyanin/tps-aug-nusvr-with-intel-extension-for-sklearn)
- [Predict Future Sales] [Stacking with scikit-learn-intelex](https://www.kaggle.com/alexeykolobyanin/predict-sales-stacking-with-scikit-learn-intelex)
- [House Prices - Advanced Regression Techniques] [NuSVR sklearn-intelex 4x speedup](https://www.kaggle.com/alexeykolobyanin/house-prices-nusvr-sklearn-intelex-4x-speedup)
- Added [demo samples](https://github.com/intel/scikit-learn-intelex/tree/master/examples/notebooks) comparing the usage of Intel® Extension for Scikit-learn and the original Scikit-learn for KNN, Logistic Regression, SVM and Random Forest algorithms

🛠️ Library Engineering
- Introduced new functionality for [Intel® Extension for Scikit-learn*](https://github.com/intel/scikit-learn-intelex):
- Enabled patching for all Scikit-learn applications at once:
- You can enable global patching via command line:
- `python -m sklearnex.glob patch_sklearn`
- Or via code:
- from sklearnex import patch_sklearn
patch_sklearn(global_patch=True)
- Read more in [Intel® Extension for Scikit-learn documentation](https://intel.github.io/scikit-learn-intelex/index.html#usage).
- Added the support of Python 3.9 for both Intel® Extension for Scikit-learn and daal4py. The packages are available from PyPI and the Intel Channel on Anaconda Cloud.

🚨 New Features
- Enabled the global patching of all Scikit-learn applications
- Provided an integration with `dpctl` for heterogeneous computing (the support of `dpctl.tensor.usm_ndarray` for input and output)
- Extended API with `set_config` and `get_config` methods. Added the support of `target_offload` and `allow_fallback_to_host` options for device offloading scenarios
- Added the support of `predict_proba` in RandomForestClassifier estimator
- [CPU] Added the support of Sigmoid kernel in SVM algorithms
- [GPU] Added binary SVC support with Linear and RBF kernels

🚀 ​Improved performance
- [CPU] `SVR` algorithm training
- [CPU] `NuSVC` and `NuSVR` algorithms training
- [CPU] `RandomForestRegression` and `RandomForestClassifier` algorithms training and prediction
- [CPU] `KMeans` algorithm training

🐛 Bug Fixes
- Fixed an incorrectly raised exception during the patching of Random Forest algorithm when the number of trees was more than 7000.
- [CPU] Fixed an accuracy issue in `Random Forest` algorithm caused by the exclusion of constant features.
- [CPU] Fixed an issue in `NuSVC` Multiclass.
- [CPU] Fixed an issue with `KMeans` convergence inconsistency.
- [CPU] Fixed incorrect work of `train_test_split` with specific subset sizes.
- [GPU] Fixed incorrect bias calculation in `SVM`.

❗ Known Issues
- [GPU] For most algorithms, performance degradations were observed when the 2021.4 version of Intel® oneAPI DPC++ Compiler was used.
- [GPU] Examples are failing when run with Visual Studio Solutions on hardware that does not support double precision floating-point operations.

The release **Intel(R) Extension for Scikit-learn 2021.3** introduces the following changes:

📚 Support Materials
- Medium blogs:
- [Superior Machine Learning Performance on the Latest Intel Xeon Scalable Processors](https://medium.com/intel-analytics-software/superior-machine-learning-performance-on-the-latest-intel-xeon-scalable-processor-efdec279f5a3)
- [Leverage Intel Optimizations in Scikit-Learn (SVM Performance Training and Inference)](https://medium.com/intel-analytics-software/leverage-intel-optimizations-in-scikit-learn-f562cb9d5544)
- [Optimizing CatBoost Performance](https://medium.com/intel-analytics-software/optimizing-catboost-performance-4f73f0593071)
- [Performance Optimizations for End-to-End AI Pipelines](https://medium.com/intel-analytics-software/performance-optimizations-for-end-to-end-ai-pipelines-231e0966505a)
- Kaggle kernels:
- [Tabular Playground Series - Apr 2021] [RF with Intel Extension for Scikit-learn](https://www.kaggle.com/andreyrus/tps-apr-rf-with-intel-extension-for-scikit-learn)
- [Tabular Playground Series - Apr 2021] [SVM with Intel Extension for Scikit-learn](https://www.kaggle.com/napetrov/tps04-svm-with-intel-extension-for-scikit-learn)
- [Tabular Playground Series - Apr 2021] [SVM with Intel(R) Extension for Scikit-learn](https://www.kaggle.com/napetrov/tps04-svm-with-scikit-learn-intelex)
- [Tabular Playground Series - Jun 2021] [AutoGluon with Intel(R) Extension for Scikit-learn](https://www.kaggle.com/alex97andreev/tps-jun-autogluon-with-sklearnex)
- [Tabular Playground Series - Jun 2021] [Fast LogReg with Intel(R) Extension for Scikit-learn](https://www.kaggle.com/kppetrov/tps-jun-fast-logreg-with-scikit-learn-intelex)
- [Tabular Playground Series - Jun 2021] [Fast ML stack with Intel(R) Extension for Scikit-learn](https://www.kaggle.com/masdevas/fast-ml-stack-with-scikit-learn-intelex)
- [Tabular Playground Series - Jun 2021] [Fast Stacking with Intel(R) Extension for Scikit-learn](https://www.kaggle.com/owerbat/tps-jun-fast-stacking-with-scikit-learn-intelex)
- [Samples](https://github.com/intel/scikit-learn-intelex/tree/master/examples/notebooks) that illustrate the usage of Intel Extension for Scikit-learn

🛠️ Library Engineering
- Introduced optional dependencies on DPC++ runtime to Intel Extension for Scikit-learn and daal4py. To enable DPC++ backend, [install dpcpp_cpp_rt package](https://github.com/intel/scikit-learn-intelex#-installation). It reduces the default package size with all dependencies from 1.2GB to 400 MB.

🚨 New Features
- Introduced the support of scikit-learn 1.0 version in Intel(R) Extension for Scikit-learn. The 2021.3 release of Intel(R) Extension for Scikit-learn supports the latest scikit-learn releases: 0.22.X, 0.23.X, 0.24.X and 1.0.X.
- The support of `patch_sklearn` for several algorithms: *patch_sklearn(["SVC", "DBSCAN"])*
- [CPU] Acceleration of `SVR` estimator
- [CPU] Acceleration of `NuSVC` and `NuSVR` estimators
- [CPU] `Polynomial kernel` support in SVM algorithms

🚀 ​Improved performance
- [CPU] `SVM` algorithms training and prediction
- [CPU] `Linear`, `Ridge`, `ElasticNet`, and `Lasso` regressions prediction

🐛 Bug Fixes
- Fixed binary incompatibility for the versions of numpy earlier than 1.19.4
- Fixed an issue with a very large number of trees (> 7000) for `Random Forest` algorithm
- Fixed `patch_sklearn` to patch both fit and predict methods of `Logistic Regression` when the algorithm is given as a single parameter to `patch_sklearn`
- [CPU] Reduced the memory consumption of `SVM` prediction
- [GPU] Fixed an issue with kernel compilation on the platforms without hardware FP64 support

❗ Known Issues
- Intel(R) Extension for Scikit-learn package installed from PyPI repository can’t be found on Debian systems (including Google Collab). Mitigation: add “site-packages” folder into Python packages searching before importing the packages:
python
import sys
import os
import site
sys.path.append(os.path.join(os.path.dirname(site.getsitepackages()[0]), "site-packages"))

🚨 New Features
- Added support of patching scikit-learn version 1.0. scikit-learn version 0.21. * is no longer supported