Pingouin

--------------------

Major upgrade of the dependencies. This release requires **Python 3.7+, SciPy 1.7+, NumPy 1.19+ and Pandas 1.0+**. Pingouin uses the ``alternative`` argument that has been added to several statistical functions of Scipy 1.7+ (see below). However, SciPy 1.7+ requires Python 3.7+. We recommend all users to upgrade to the latest version of Pingouin.

Major enhancements
~~~~~~~~~~~~~~~~~~

**Directional testing**

The ``tail`` argument has been renamed to ``alternative`` in all Pingouin functions to be consistent with SciPy and R (`185 <https://github.com/raphaelvallat/pingouin/issues/185>`_). Furthermore, ``"alternative='one-sided'"`` has now been deprecated. Instead, ``alternative`` must be one of "two-sided" (default), "greater" or "less". Again, this is the same behavior as SciPy and R.

Added support for directional testing with ``"alternative='greater'"`` and ``"alternative='less'"`` in :py:func:`pingouin.corr` (`176 <https://github.com/raphaelvallat/pingouin/issues/176>`_). As a result, the p-value, confidence intervals and power of the correlation will change depending on the directionality of the test. Support for directional testing has also been added to :py:func:`pingouin.power_corr` and :py:func:`pingouin.compute_esci`.

Finally, the ``tail`` argument has been removed from :py:func:`pingouin.rm_corr`, :py:func:`pingouin.circ_corrcc` and :py:func:`pingouin.circ_corrcl` to be consistent with the original R / Matlab implementations.

**Partial correlation**

Major refactoring of :py:func:`pingouin.partial_corr`, which now uses the same method as the R `ppcor <https://cran.r-project.org/web/packages/ppcor/ppcor.pdf>`_ package, i.e. based on the inverse covariance matrix rather than the residuals of a linear regression. This new approach is faster and works better in some cases (such as Spearman partial correlation with binary variables, see `issue 147 <https://github.com/raphaelvallat/pingouin/issues/147>`_).
One caveat is that only the Pearson and Spearman correlation methods are now supported in partial/semi-partial correlation.

**Box M test**

Added the :py:func:`pingouin.box_m` function to calculate `Box's M test <https://en.wikipedia.org/wiki/Box%27s_M_test>`_ for equality of covariance matrices (`#175 <https://github.com/raphaelvallat/pingouin/pull/175>`_).

Minor enhancements
~~~~~~~~~~~~~~~~~~

* :py:func:`pingouin.wilcoxon` now supports a pre-computed array of differences, similar to :py:func:`scipy.stats.wilcoxon` (`issue 186 <https://github.com/raphaelvallat/pingouin/issues/186>`_).

* :py:func:`pingouin.mwu` and :py:func:`pingouin.wilcoxon` now support keywords arguments that are passed to the lower-level scipy functions.

* Added warning in :py:func:`pingouin.partial_corr` with ``method="skipped"``: the MCD algorithm does not give the same output in Python (scikit-learn) than in the original Matlab library (LIBRA), and this can lead to skipped correlations that are different in Pingouin than in the Matlab robust correlation toolbox (see `issue 164 <https://github.com/raphaelvallat/pingouin/issues/164>`_).

* :py:func:`pingouin.ancova` always uses statsmodels, regardless of the number of covariates. This fixes LinAlg errors in :py:func:`pingouin.ancova` and :py:func:`pingouin.rm_corr` (see `issue 184 <https://github.com/raphaelvallat/pingouin/issues/184>`_).

* Avoid RuntimeWarning when calculating CI and power of a perfect correlation in :py:func:`pingouin.corr` (see `issue 183 <https://github.com/raphaelvallat/pingouin/issues/183>`_).

* Use :py:func:`scipy.linalg.lstsq` instead of :py:func:`numpy.linalg.lstsq` whenever possible to better check for NaN and Inf in input (see `issue 184 <https://github.com/raphaelvallat/pingouin/issues/184>`_).

* flake8 requirements for max line length has been changed from 80 to 100 characters.

--------------------------------------------------------------------------------

------------------

**Bugfixes**

This release fixes a critical error in :py:func:`pingouin.partial_corr`: the number of covariates was not taken into account when calculating the degrees of freedom of the partial correlation, thus leading to incorrect results (except for the correlation coefficient which remained unaffected). For more details, please see `issue 171 <https://github.com/raphaelvallat/pingouin/issues/171>`_.

In addition to fixing the p-values and 95% confidence intervals, the statistical power and Bayes Factor have been removed from the output of :py:func:`pingouin.partial_corr`, at least temporary until we can make sure that these give exact results.

We have also fixed a minor bug in the robust skipped and shepherd correlation (see :py:func:`pingouin.corr`), for which the calculation of the confidence intervals and statistical power did not take into account the number of outliers. These are now calculated only on the cleaned data.

.. warning:: We therefore strongly recommend that all users UPDATE Pingouin (:code:`pip install -U pingouin`) and CHECK ANY RESULTS obtained with the :py:func:`pingouin.partial_corr` function.

**Enhancements**

a. Major refactoring of :py:func:`pingouin.plot_blandaltman`, which now has many additional parameters. It also uses a T distribution instead of a normal distribution to estimate the 95% confidence intervals of the mean difference and agreement limits. See `issue 167 <https://github.com/raphaelvallat/pingouin/issues/167>`_.
b. For clarity, the `z`, `r2` and `adj_r2` have been removed from the output of :py:func:`pingouin.corr` and :py:func:`pingouin.pairwise_corr`, as these can be readily calculated from the correlation coefficient.
c. Better testing against R for :py:func:`pingouin.partial_corr` and :py:func:`pingouin.corr`.

--------------------

**Bugfixes**

a. Fix invalid computation of the robust skipped correlation in :py:func:`pingouin.corr` (see `issue 164 <https://github.com/raphaelvallat/pingouin/issues/164>`_).
b. Passing a wrong ``tail`` argument to :py:func:`pingouin.corr` now *always* raises an error (see `PR 160 <https://github.com/raphaelvallat/pingouin/pull/160>`_).
In previous versions of pingouin, using any ``method`` other than ``"pearson"`` and a wrong ``tail`` argument such as ``"two-tailed"`` or ``"both"``
(instead of the correct ``"two-sided"``) may have resulted in silently returning a one-sided p-value.
c. Reverted changes made in :py:func:`pingouin.pairwise_corr` which led to Pingouin calculating the correlations between the DV columns and the covariates, thus artificially increasing the number of pairwise comparisons (see `issue 162 <https://github.com/raphaelvallat/pingouin/issues/162>`_).

-----------------------

**Bugfix**

This release fixes an error in the calculation of the p-values in the :py:func:`pingouin.pairwise_tukey` and :py:func:`pingouin.pairwise_gameshowell` functions (see `PR156 <https://github.com/raphaelvallat/pingouin/pull/156>`_). Old versions of Pingouin used an incorrect algorithm for the studentized range approximation, which resulted in (slightly) incorrect p-values. In most cases, the error did not seem to affect the significance of the p-values. The new version of Pingouin now uses `statsmodels internal implementation <https://github.com/statsmodels/statsmodels/blob/master/statsmodels/stats/libqsturng/qsturng_.py>`_ of the Gleason (1999) algorithm to estimate the p-values.

Please note that the Pingouin p-values may be slightly different than R (and JASP), because it uses a different algorithm. However, this does not seem to affect the significance levels of the p-values (i.e. a p-value below 0.05 in JASP is likely to be below 0.05 in Pingouin, and vice versa).

We therefore recommend that all users UPDATE Pingouin (:code:`pip install -U pingouin`) and CHECK ANY RESULTS obtained with the :py:func:`pingouin.pairwise_tukey` and :py:func:`pingouin.pairwise_gameshowell` functions.

---------------------

**Bugfix**

This release fixes a CRITICAL ERROR in the :py:func:`pingouin.pairwise_ttests` function (see `issue 151 <https://github.com/raphaelvallat/pingouin/issues/151>`_). The bug concerns one-way and two-way repeated measures pairwise T-tests. Until now, Pingouin implicitly assumed that the dataframe was sorted such that the ordering of the subject was the same across all repeated measurements (e.g. the third values in the repeated measurements always belonged to the same subject).
This led to incorrect results when the dataframe was not sorted in such a way.

We therefore strongly recommend that all users UPDATE Pingouin (:code:`pip install -U pingouin`) and CHECK ANY RESULTS obtained with the :py:func:`pingouin.pairwise_ttests` function. Note that the bug does not concern non-repeated measures pairwise T-test, since the ordering of the values does not matter in this case.

Furthermore, and to prevent a similar issue, we have now disabled ``marginal=False`` in two-way repeated measure design. As of this release, ``marginal=False`` will therefore only have an impact on the between-factor T-test(s) of a mixed design.

**Deprecation**

a. Removed the Glass delta effect size. Until now, Pingouin invalidly assumed that the control group was always the one with the lowest standard deviation. Since this cannot be verified, and to avoid any confusion, the Glass delta effect size has been completely removed from Pingouin.
See `issue 139 <https://github.com/raphaelvallat/pingouin/issues/139>`_.

**Enhancements**

a. :py:func:`pingouin.plot_paired` now supports an arbitrary number of within-levels as well as horizontal plotting. See `PR 133 <https://github.com/raphaelvallat/pingouin/pull/133>`_.
b. :py:func:`pingouin.linear_regression` now handles a rank deficient design matrix X by producing a warning and trying to calculate the sum of squared residuals without relying on :py:func:`np.linalg.lstsq`. See `issue 130 <https://github.com/raphaelvallat/pingouin/issues/130>`_.
c. :py:func:`pingouin.friedman` now has an option to choose between Chi square test or F test method.
d. Several minor improvements to the documentation and GitHub Actions. See `PR150 <https://github.com/raphaelvallat/pingouin/pull/150>`_.
e. Added support for ``kwargs`` in :py:func:`pingouin.corr` (see `issue 138 <https://github.com/raphaelvallat/pingouin/issues/138>`_).
f. Added ``confidence`` argument in :py:func:`pingouin.ttest` to allow for custom CI (see `issue 152 <https://github.com/raphaelvallat/pingouin/issues/152>`_).

-----------------------

**Bugfixes**

a. Fix a bug in in :py:func:`pingouin.ttest` in which the confidence intervals for one-sample T-test with y != 0 were invalid (e.g. ``pg.ttest(x=[4, 6, 7, 4], y=4)``). See `issue 119 <https://github.com/raphaelvallat/pingouin/issues/119>`_.

**New features**

a. Added a `pingouin.options` module which can be used to set default options. For example, one can set the default decimal rounding of the output dataframe, either for the entire dataframe, per column, per row, or per cell. See `PR120 <https://github.com/raphaelvallat/pingouin/pull/120>`_. For more details, please refer to `notebooks/06_others.ipynb <https://github.com/raphaelvallat/pingouin/blob/master/notebooks/06_Others.ipynb>`_.

.. code-block:: python

import pingouin as pg
pg.options['round'] = None Default: no rounding
pg.options['round'] = 4
pg.options['round.column.CI95%'] = 2
pg.options['round.row.T-test'] = 2
pg.options['round.cell.[T-test]x[CI95%]'] = 2


**Enhancements**

a. :py:func:`pingouin.linear_regression` now returns the processed X and y variables (Xw and yw for WLS) and the predicted values if ``as_dataframe=False``. See `issue 112 <https://github.com/raphaelvallat/pingouin/issues/112>`_.
b. The Common Language Effect Size (CLES) in :py:func:`pingouin.mwu` is now calculated using the formula given by Vargha and Delaney 2000, which works better when ties are present in data. This is consistent with the :py:func:`pingouin.wilcoxon` and :py:func:`pingouin.compute_effsize` functions. See `issue 114 <https://github.com/raphaelvallat/pingouin/issues/114>`_.
c. Better handling of kwargs arguments in :py:func:`pingouin.plot_paired` (see `PR 116 <https://github.com/raphaelvallat/pingouin/pull/116>`_).
d. Added ``boxplot_in_front`` argument to the :py:func:`pingouin.plot_paired`. When set to True, the boxplot is displayed in front of the lines with a slight transparency. This can make the overall plot more readable when plotting data from a large number of subjects. (see `PR 117 <https://github.com/raphaelvallat/pingouin/pull/117>`_).
e. Better handling of Categorical columns in several functions (e.g. ANOVA). See `issue 122 <https://github.com/raphaelvallat/pingouin/issues/122>`_.
f. :py:func:`multivariate_normality` now also returns the test statistic. This function also comes with better unit testing against the MVN R package.
g. :py:func:`pingouin.pairwise_corr` can now control for all covariates by excluding each specific set of column-combinations from the covariates to use for this combination, similar to :py:func:`pingouin.pcorr`. See `PR 124 <https://github.com/raphaelvallat/pingouin/pull/124>`_.
h. Bayes factor formatting is now handled via the options module. The default behaviour is unchanged (return as formatted string), but can easily be disabled by setting `pingouin.options["round.column.BF10"] = None`. See `PR 126 <https://github.com/raphaelvallat/pingouin/pull/126>`_.