Tea-tasting

What's Changed

* Setup web docs with mkdocs by e10v in https://github.com/e10v/tea-tasting/pull/56
* Experiment result pretty formatting by e10v in https://github.com/e10v/tea-tasting/pull/57 and https://github.com/e10v/tea-tasting/pull/59
* Rename is_in to in_ by e10v in https://github.com/e10v/tea-tasting/pull/60
* Update docs and improve naming by e10v in https://github.com/e10v/tea-tasting/pull/61
* Support Ibis 9 by e10v in https://github.com/e10v/tea-tasting/pull/62


**Full Changelog**: https://github.com/e10v/tea-tasting/compare/v0.0.3...v0.0.4

What's Changed

* Sample ratio mismatch check by e10v in https://github.com/e10v/tea-tasting/pull/52
* Handle potential division by zero by e10v in https://github.com/e10v/tea-tasting/pull/53 and https://github.com/e10v/tea-tasting/pull/54
* Multiple minor improvements by e10v in https://github.com/e10v/tea-tasting/pull/55 and https://github.com/e10v/tea-tasting/pull/51


**Full Changelog**: https://github.com/e10v/tea-tasting/compare/v0.0.2...v0.0.3

**tea-tasting** is a Python package for statistical analysis of A/B tests that features:

- Student's t-test and Z-test out of the box.
- Extensible API: Define and use statistical tests of your choice.
- [Delta method](https://alexdeng.github.io/public/files/kdd2018-dm.pdf) for ratio metrics.
- Variance reduction with [CUPED](https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf)/[CUPAC](https://doordash.engineering/2020/06/08/improving-experimental-power-through-control-using-predictions-as-covariate-cupac/) (also in combination with delta method for ratio metrics).
- Confidence interval for both absolute and percent change.

**tea-tasting** calculates statistics within data backends such as BigQuery, ClickHouse, PostgreSQL, Snowflake, Spark, and other of 20+ backends supported by [Ibis](https://ibis-project.org/). This approach eliminates the need to import granular data into a Python environment, though Pandas DataFrames are also supported.

**tea-tasting** is still in alpha, but already includes all the features listed above. The following features are coming soon:

- Sample ratio mismatch check.
- More statistical tests:
- Asymptotic and exact tests for frequency data.
- Bootstrap.
- Quantile test (using Bootstrap).
- Mann–Whitney U test.
- Power analysis.
- A/A tests and simulations.
- Pretty output for experiment results (round etc.).
- Documentation on how to define metrics with custom statistical tests.
- Documentation with MkDocs and Material for MkDocs.
- More examples.

**tea-tasting** is a Python package for statistical analysis of A/B tests that features:

- Student's t-test, Z-test, and Bootstrap out of the box.
- Extensible API: Define and use statistical tests of your choice.
- [Delta method](https://alexdeng.github.io/public/files/kdd2018-dm.pdf) for ratio metrics.
- Variance reduction with [CUPED](https://exp-platform.com/Documents/2013-02-CUPED-ImprovingSensitivityOfControlledExperiments.pdf)/[CUPAC](https://doordash.engineering/2020/06/08/improving-experimental-power-through-control-using-predictions-as-covariate-cupac/) (also in combination with delta method for ratio metrics).
- [Fieller's confidence interval](https://en.wikipedia.org/wiki/Fieller%27s_theorem) for percent change.
- Sample ratio mismatch check.
- Power analysis.
- A/A tests.

Currently, **tea-tasting** is in the planning stage, and I'm starting with a README that outlines the proposed API — an approach known as Readme Driven Development (RDD).

Check out my [blog post](https://e10v.me/tea-tasting-rdd) where I explain the motivation for creating this package and the benefits of the RDD approach.