Langfair

Latest version: v0.2.0

Safety actively analyzes 687767 Python packages for vulnerabilities to keep your Python projects secure.

Scan your dependencies

0.2.0

Highlights
- Upgrade version of LangChain to 0.3.7 to resolve dependency conflicts with later versions of LangChain community packages
- Refactor `ResponseGenerator`, `CounterfactualGenerator`, `AutoEval` to adjust for LangChain upgrade
- More intuitive printing in `AutoEval`
- Update unit tests
- Update documentation in notebooks for user-friendliness and to include MistralAI
- Improved exception handling
- Remove 'langchain: ' from print statements

What's Changed
* v0.1.2 Updates by dylanbouchard in https://github.com/cvs-health/langfair/pull/36
* upgrade langchain by dylanbouchard in https://github.com/cvs-health/langfair/pull/39
* Formatting changes made by vasisthasinghal in https://github.com/cvs-health/langfair/pull/41
* Resolve issue: upgrade version of langchain by dylanbouchard in https://github.com/cvs-health/langfair/pull/40
* Add pytest to dev dependencies by virenbajaj in https://github.com/cvs-health/langfair/pull/38
* Update exception handling and notebooks by dylanbouchard in https://github.com/cvs-health/langfair/pull/42
* Vb/handle suppressed exceptions by dylanbouchard in https://github.com/cvs-health/langfair/pull/44
* Release PR: v0.2.0 by dylanbouchard in https://github.com/cvs-health/langfair/pull/45

New Contributors
* vasisthasinghal made their first contribution in https://github.com/cvs-health/langfair/pull/41

**Full Changelog**: https://github.com/cvs-health/langfair/compare/v0.1.2...v0.2.0

0.1.2

Highlights
- Improved Readme for readability
- Improved notebook documentation for readability
- Removed `scipy`, `sklearn`, `openai` and `langchain-openai` dependencies
- Created new argument for `ResponseGenerator` and `CounterfactualGenerator` that allows users to specify which exceptions to suppress

What's Changed
* v0.1.1 -> Develop by dylanbouchard in https://github.com/cvs-health/langfair/pull/19
* Update readme and notebooks by dylanbouchard in https://github.com/cvs-health/langfair/pull/20
* Remove `scipy` dependency by dylanbouchard in https://github.com/cvs-health/langfair/pull/21
* Remove dependency on the scikit-learn confusion matrix by mohitcek in https://github.com/cvs-health/langfair/pull/23
* Add code of conduct by virenbajaj in https://github.com/cvs-health/langfair/pull/25
* Remove `openai`, `langchain-openai` dependencies by dylanbouchard in https://github.com/cvs-health/langfair/pull/22
* Add code of conduct: main -> develop by dylanbouchard in https://github.com/cvs-health/langfair/pull/26
* Move metrics section by virenbajaj in https://github.com/cvs-health/langfair/pull/27
* add user warning message by zeya30 in https://github.com/cvs-health/langfair/pull/28
* Fix links by dylanbouchard in https://github.com/cvs-health/langfair/pull/29
* DS/demo and pyproject updates by dskarbrevik in https://github.com/cvs-health/langfair/pull/30
* Update notebooks by dylanbouchard in https://github.com/cvs-health/langfair/pull/31
* remove openai import by dylanbouchard in https://github.com/cvs-health/langfair/pull/32
* Update notebook instructions by dylanbouchard in https://github.com/cvs-health/langfair/pull/33
* Release PR: v0.1.2 by dylanbouchard in https://github.com/cvs-health/langfair/pull/34

New Contributors
* mohitcek made their first contribution in https://github.com/cvs-health/langfair/pull/23
* virenbajaj made their first contribution in https://github.com/cvs-health/langfair/pull/25
* dskarbrevik made their first contribution in https://github.com/cvs-health/langfair/pull/30

**Full Changelog**: https://github.com/cvs-health/langfair/compare/v0.1.1...v0.1.2

0.1.1

What's Changed
* update docstring by dylanbouchard in https://github.com/cvs-health/langfair/pull/14
* Update readme by dylanbouchard in https://github.com/cvs-health/langfair/pull/16
* readme updates by dylanbouchard in https://github.com/cvs-health/langfair/pull/17
* Release PR - v0.1.1 by dylanbouchard in https://github.com/cvs-health/langfair/pull/18


**Full Changelog**: https://github.com/cvs-health/langfair/compare/v0.1.0...v0.1.1

0.1.0

LangFair is a Python library for conducting bias and fairness assessments of LLM use cases. This repository includes a framework for [choosing bias and fairness metrics](https://github.com/cvs-health/langfair/tree/develop#choosing-bias-and-fairness-metrics-for-an-llm-use-case), [demo notebooks](https://github.com/cvs-health/langfair/tree/develop/examples), and a LLM bias and fairness [technical playbook](https://arxiv.org/pdf/2407.10853) containing a thorough discussion of LLM bias and fairness risks, evaluation metrics, and best practices. Please refer to our [documentation site](https://cvs-health.github.io/langfair/) for more details on how to use LangFair.

Highlights
Bias and fairness metrics offered by LangFair fall into one of several categories. The full suite of metrics is displayed below.

Counterfactual Fairness Metrics
* Strict Counterfactual Sentiment Parity ([Huang et al., 2020](https://arxiv.org/pdf/1911.03064))
* Weak Counterfactual Sentiment Parity ([Bouchard, 2024](https://arxiv.org/pdf/2407.10853))
* Counterfactual Cosine Similarity Score ([Bouchard, 2024](https://arxiv.org/pdf/2407.10853))
* Counterfactual BLEU ([Bouchard, 2024](https://arxiv.org/pdf/2407.10853))
* Counterfactual ROUGE-L ([Bouchard, 2024](https://arxiv.org/pdf/2407.10853))

Stereotype Metrics
* Stereotypical Associations ([Liang et al., 2023](https://arxiv.org/pdf/2211.09110))
* Co-occurrence Bias Score ([Bordia & Bowman, 2019](https://aclanthology.org/N19-3002.pdf))
* Stereotype classifier metrics ([Zekun et al., 2023](https://arxiv.org/ftp/arxiv/papers/2311/2311.14126.pdf), [Bouchard, 2024](https://arxiv.org/pdf/2407.10853))

Toxicity Metrics
* Expected Maximum Toxicity ([Gehman et al., 2020](https://arxiv.org/pdf/2009.11462))
* Toxicity Probability ([Gehman et al., 2020](https://arxiv.org/pdf/2009.11462))
* Toxic Fraction ([Liang et al., 2023](https://arxiv.org/pdf/2211.09110))

Recommendation Fairness Metrics
* Jaccard Similarity ([Zhang et al., 2023](https://arxiv.org/pdf/2305.07609.pdf))
* Search Result Page Misinformation Score ([Zhang et al., 2023](https://arxiv.org/pdf/2305.07609.pdf))
* Pairwise Ranking Accuracy Gap ([Zhang et al., 2023](https://arxiv.org/pdf/2305.07609.pdf))

Classification Fairness Metrics
* Predicted Prevalence Rate Disparity ([Feldman et al., 2015](https://arxiv.org/abs/1412.3756); [Bellamy et al., 2018](https://arxiv.org/abs/1810.01943); [Saleiro et al., 2019](https://arxiv.org/abs/1811.05577))
* False Negative Rate Disparity ([Bellamy et al., 2018](https://arxiv.org/abs/1810.01943); [Saleiro et al., 2019](https://arxiv.org/abs/1811.05577))
* False Omission Rate Disparity ([Bellamy et al., 2018](https://arxiv.org/abs/1810.01943); [Saleiro et al., 2019](https://arxiv.org/abs/1811.05577))
* False Positive Rate Disparity ([Bellamy et al., 2018](https://arxiv.org/abs/1810.01943); [Saleiro et al., 2019](https://arxiv.org/abs/1811.05577))
* False Discovery Rate Disparity ([Bellamy et al., 2018](https://arxiv.org/abs/1810.01943); [Saleiro et al., 2019](https://arxiv.org/abs/1811.05577))

Links

Releases

Has known vulnerabilities

© 2024 Safety CLI Cybersecurity Inc. All Rights Reserved.