Langcheck

Breaking Changes

* To simplify installation, LangCheck v0.6.0 now splits the package into multiple languages.
* `pip install langcheck` will only install English metrics
* `pip install langcheck[all]` will install metrics for all languages
* `pip install langcheck[ja]` will only install Japanese and English metrics (use `[de]` for German and `[zh]` for Chinese)
* See [the "Installation" guide](https://langcheck.readthedocs.io/en/latest/installation.html) for more details.

New Features
* [Added the `pairwise_comparison()` metric](https://langcheck.readthedocs.io/en/latest/metrics.html#pairwise-text-quality-metrics) to automatically rank the output of two models!
* Enabled running OpenAI metrics asynchronously. To do this, set the `use_async` parameter for OpenAI metric functions, such as [`factual_consistency()`](https://langcheck.readthedocs.io/en/latest/langcheck.metrics.en.source_based_text_quality.html#langcheck.metrics.en.source_based_text_quality.factual_consistency)

Bug Fixes and Improvements
* Fixed the local Japanese, German, Chinese `factual_consistency()` metric to not throw an exception for long inputs (inputs longer than the model accepts are automatically truncated now)
* Updated the "Pytest" status badge in README.md
* Refactored implementation of local metrics

**Full Changelog**: https://github.com/citadel-ai/langcheck/compare/v0.5.0...v0.6.0

Breaking Changes

None

New Features
* Launched LangCheck metrics for Chinese (`langcheck.metrics.zh`) 🇨🇳 - thanks Vela-zz!
* New Model Manager architecture to unify the management of local evaluation models - thanks Vela-zz!
* Update the default OpenAI embedding model to `text-embedding-3-small` - thanks bioerrorlog!
* [New tutorial for LangCheckChat](https://langcheck.readthedocs.io/en/latest/tutorial_langcheckchat.html), a self-evaluating RAG application on LangCheck documentation

Bug Fixes and Improvements

* Improvements to support for German (README, factual consistency benchmarking, OpenAI prompts in German) 🇩🇪 - thanks ischender!
* Improvements to the OpenAI-based metrics for Japanese
* We now have pip install tests for different Python versions
* `nltk download` will now only run when needed (as opposed to always running on `import langcheck`)
* Add the tqdm progress bar to more metrics

**Full Changelog**: https://github.com/citadel-ai/langcheck/compare/v0.4.0...v0.5.0

Breaking Changes
None

New Features
* Launched LangCheck metrics for German (`langcheck.metrics.de`) 🇩🇪 - thanks ischender!
* Added `langcheck.metrics.context_relevance`, a metric to compute the relevance of a retrieved source text to the user's prompt
* Added `langcheck.metrics.answer_relevance`, a metric to compute the relevance of an output to the user's prompt
* Added `langcheck.augment.ja.synonym`, a Japanese text augmentation where some words are replaced with synonyms

Bug Fixes
None

**Full Changelog**: https://github.com/citadel-ai/langcheck/compare/v0.3.0...v0.4.0

Breaking Changes

- LangCheck v0.3.0 supports and requires `openai>=1`
- The required function parameters for OpenAI-based metrics have changed. [Please review the documentation to see the new API](https://langcheck.readthedocs.io/en/latest/metrics.html#computing-metrics-with-openai-models)!

New Features

- Launched `langcheck.augment` with customizable English text augmentations to test robustness of LLMs! [Check out the documentation to learn more](https://langcheck.readthedocs.io/en/latest/langcheck.augment.html).
- `langcheck.augment.rephrase()`
- `langcheck.augment.synonym()`
- `langcheck.augment.gender()`
- `langcheck.augment.keyboard_typo()`
- `langcheck.augment.ocr_typo()`
- `langcheck.augment.remove_punctuation()`
- `langcheck.augment.change_case()`
- Improved the quality of explanations and speed of the OpenAI-based `factual_consistency()` metric
- More deterministic evaluations for OpenAI-based metrics
- Added progress bar when computing metrics – thanks ThanhChinhBK!
- Added benchmarking for Japanese factual consistency metrics.

Bug Fixes

- Fixed "can't find Rust compiler" bug during installation for Python 3.10 by removing `detoxify` as a dependency and running their model directly (`detoxify` is unmaintained and pinned an old version of the `tokenizers` library)

**Full Changelog**: https://github.com/citadel-ai/langcheck/compare/v0.2.0...v0.3.0

Breaking Changes

- `MetricValue.metric_values` can now include `None` values when using OpenAI-based metrics. Metric values will be `None` if the OpenAI API rejects the request (e.g. due to their moderation policy) or returns an invalid response.

New Features

- Significantly improved all OpenAI-based metrics using chain-of-thought prompting, such as the `factual_consistency` metric
- All OpenAI-based metrics now output a text explanation for each score in `MetricValue.explanations`
- Added factual consistency benchmark datasets to evaluate different metric configurations
- Thresholds are now visualized when plotting metrics like `(toxicity_values > 0.9).scatter()` – thanks Vela-zz!
- Published new tutorial for email generation app
- Updated EN->JA translation model to `Helsinki-NLP/opus-mt-ja-en`
- (Beta) Added basic English text augmentations, full release with documentation coming soon

Bug Fixes

- Fixed "can't find Rust compiler" bug during installation for Python 3.11 due to older versions of the `tokenizers` library
- Temporarily pinned `openai<1.0.0` due to breaking changes in their API
- Fixed contributing guide for zsh – thanks shibuiwilliam!
- Cleaned up `noqa` showing up in documentation

First release 🎉 🎉