Tea-tasting

What's Changed

**tea-tasting** is currently in beta. However, I consider it ready for important tasks and use it for analyzing switchback experiments at my work.

- Analysis of power for RatioOfMeans and Mean by e10v in https://github.com/e10v/tea-tasting/pull/68, https://github.com/e10v/tea-tasting/pull/70, https://github.com/e10v/tea-tasting/pull/71, https://github.com/e10v/tea-tasting/pull/72, https://github.com/e10v/tea-tasting/pull/75, and https://github.com/e10v/tea-tasting/pull/76
- Fix: make sure that k and n for binomtest are integer by e10v in https://github.com/e10v/tea-tasting/pull/73
- Create a guide on how to use tea-tasting with an arbitrary data backend by e10v in https://github.com/e10v/tea-tasting/pull/79
- Create a guide on custom metrics by e10v in https://github.com/e10v/tea-tasting/pull/81
- Update user guides, docstrings, and readme by e10v in https://github.com/e10v/tea-tasting/pull/77, https://github.com/e10v/tea-tasting/pull/78, https://github.com/e10v/tea-tasting/pull/80, and https://github.com/e10v/tea-tasting/pull/82
- Other minor changes by e10v in https://github.com/e10v/tea-tasting/pull/74 and https://github.com/e10v/tea-tasting/pull/83

**Full Changelog**: https://github.com/e10v/tea-tasting/compare/v0.0.5...v0.1.0

What's Changed

* Analysis of a statistic using bootstrap resampling by e10v in https://github.com/e10v/tea-tasting/pull/64
* Analysis of quantiles using bootstrap resampling by e10v in https://github.com/e10v/tea-tasting/pull/65
* Update docs by e10v in https://github.com/e10v/tea-tasting/pull/63 and https://github.com/e10v/tea-tasting/pull/67


**Full Changelog**: https://github.com/e10v/tea-tasting/compare/v0.0.4...v0.0.5

What's Changed

* Setup web docs with mkdocs by e10v in https://github.com/e10v/tea-tasting/pull/56
* Experiment result pretty formatting by e10v in https://github.com/e10v/tea-tasting/pull/57 and https://github.com/e10v/tea-tasting/pull/59
* Rename is_in to in_ by e10v in https://github.com/e10v/tea-tasting/pull/60
* Update docs and improve naming by e10v in https://github.com/e10v/tea-tasting/pull/61
* Support Ibis 9 by e10v in https://github.com/e10v/tea-tasting/pull/62


**Full Changelog**: https://github.com/e10v/tea-tasting/compare/v0.0.3...v0.0.4

What's Changed

* Sample ratio mismatch check by e10v in https://github.com/e10v/tea-tasting/pull/52
* Handle potential division by zero by e10v in https://github.com/e10v/tea-tasting/pull/53 and https://github.com/e10v/tea-tasting/pull/54
* Multiple minor improvements by e10v in https://github.com/e10v/tea-tasting/pull/55 and https://github.com/e10v/tea-tasting/pull/51


**Full Changelog**: https://github.com/e10v/tea-tasting/compare/v0.0.2...v0.0.3

**tea-tasting** is a Python package for statistical analysis of A/B tests that features:

- Student's t-test and Z-test out of the box.
- Extensible API: Define and use statistical tests of your choice.
- [Delta method](https://alexdeng.github.io/public/files/kdd2018-dm.pdf) for ratio metrics.
- Variance reduction with [CUPED](https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf)/[CUPAC](https://doordash.engineering/2020/06/08/improving-experimental-power-through-control-using-predictions-as-covariate-cupac/) (also in combination with delta method for ratio metrics).
- Confidence interval for both absolute and percent change.

**tea-tasting** calculates statistics within data backends such as BigQuery, ClickHouse, PostgreSQL, Snowflake, Spark, and other of 20+ backends supported by [Ibis](https://ibis-project.org/). This approach eliminates the need to import granular data into a Python environment, though Pandas DataFrames are also supported.

**tea-tasting** is still in alpha, but already includes all the features listed above. The following features are coming soon:

- Sample ratio mismatch check.
- More statistical tests:
- Asymptotic and exact tests for frequency data.
- Bootstrap.
- Quantile test (using Bootstrap).
- Mann–Whitney U test.
- Power analysis.
- A/A tests and simulations.
- Pretty output for experiment results (round etc.).
- Documentation on how to define metrics with custom statistical tests.
- Documentation with MkDocs and Material for MkDocs.
- More examples.

**tea-tasting** is a Python package for statistical analysis of A/B tests that features:

- Student's t-test, Z-test, and Bootstrap out of the box.
- Extensible API: Define and use statistical tests of your choice.
- [Delta method](https://alexdeng.github.io/public/files/kdd2018-dm.pdf) for ratio metrics.
- Variance reduction with [CUPED](https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf)/[CUPAC](https://doordash.engineering/2020/06/08/improving-experimental-power-through-control-using-predictions-as-covariate-cupac/) (also in combination with delta method for ratio metrics).
- [Fieller's confidence interval](https://en.wikipedia.org/wiki/Fieller%27s_theorem) for percent change.
- Sample ratio mismatch check.
- Power analysis.
- A/A tests.

Currently, **tea-tasting** is in the planning stage, and I'm starting with a README that outlines the proposed API — an approach known as Readme Driven Development (RDD).

Check out my [blog post](https://e10v.me/tea-tasting-rdd) where I explain the motivation for creating this package and the benefits of the RDD approach.