Langcheck

Breaking Changes

- LangCheck v0.3.0 supports and requires `openai>=1`
- The required function parameters for OpenAI-based metrics have changed. [Please review the documentation to see the new API](https://langcheck.readthedocs.io/en/latest/metrics.html#computing-metrics-with-openai-models)!

New Features

- Launched `langcheck.augment` with customizable English text augmentations to test robustness of LLMs! [Check out the documentation to learn more](https://langcheck.readthedocs.io/en/latest/langcheck.augment.html).
- `langcheck.augment.rephrase()`
- `langcheck.augment.synonym()`
- `langcheck.augment.gender()`
- `langcheck.augment.keyboard_typo()`
- `langcheck.augment.ocr_typo()`
- `langcheck.augment.remove_punctuation()`
- `langcheck.augment.change_case()`
- Improved the quality of explanations and speed of the OpenAI-based `factual_consistency()` metric
- More deterministic evaluations for OpenAI-based metrics
- Added progress bar when computing metrics – thanks ThanhChinhBK!
- Added benchmarking for Japanese factual consistency metrics.

Bug Fixes

- Fixed "can't find Rust compiler" bug during installation for Python 3.10 by removing `detoxify` as a dependency and running their model directly (`detoxify` is unmaintained and pinned an old version of the `tokenizers` library)

**Full Changelog**: https://github.com/citadel-ai/langcheck/compare/v0.2.0...v0.3.0

Breaking Changes

- `MetricValue.metric_values` can now include `None` values when using OpenAI-based metrics. Metric values will be `None` if the OpenAI API rejects the request (e.g. due to their moderation policy) or returns an invalid response.

New Features

- Significantly improved all OpenAI-based metrics using chain-of-thought prompting, such as the `factual_consistency` metric
- All OpenAI-based metrics now output a text explanation for each score in `MetricValue.explanations`
- Added factual consistency benchmark datasets to evaluate different metric configurations
- Thresholds are now visualized when plotting metrics like `(toxicity_values > 0.9).scatter()` – thanks Vela-zz!
- Published new tutorial for email generation app
- Updated EN->JA translation model to `Helsinki-NLP/opus-mt-ja-en`
- (Beta) Added basic English text augmentations, full release with documentation coming soon

Bug Fixes

- Fixed "can't find Rust compiler" bug during installation for Python 3.11 due to older versions of the `tokenizers` library
- Temporarily pinned `openai<1.0.0` due to breaking changes in their API
- Fixed contributing guide for zsh – thanks shibuiwilliam!
- Cleaned up `noqa` showing up in documentation

First release 🎉 🎉