Promptfoo

What's Changed

Features

- `config get/set` commands, ability for users to set their email by [typpo](https://github.com/typpo) in [#971](https://github.com/promptfoo/promptfoo/pull/971)
- **webui**: Download as CSV by [typpo](https://github.com/typpo) in [#1000](https://github.com/promptfoo/promptfoo/pull/1000)
- Add support for Gemini default grader if credentials are present by [typpo](https://github.com/typpo) in [#998](https://github.com/promptfoo/promptfoo/pull/998)
- **redteam**: Allow arbitrary providers by [mldangelo](https://github.com/mldangelo) in [#1002](https://github.com/promptfoo/promptfoo/pull/1002)

Fixes

- Derived metrics by [typpo](https://github.com/typpo) in [#985](https://github.com/promptfoo/promptfoo/pull/985)
- Python provider can import modules with same name as built-ins by [typpo](https://github.com/typpo) in [#989](https://github.com/promptfoo/promptfoo/pull/989)
- Include error text in all cases by [typpo](https://github.com/typpo) in [#990](https://github.com/promptfoo/promptfoo/pull/990)
- Ensure tests inside scenarios are filtered by filter patterns by [mldangelo](https://github.com/mldangelo) in [#996](https://github.com/promptfoo/promptfoo/pull/996)
- Anthropic message API support for env vars by [typpo](https://github.com/typpo) in [#997](https://github.com/promptfoo/promptfoo/pull/997)

Documentation

- Add build documentation workflow and fix typos by [mldangelo](https://github.com/mldangelo) in [#993](https://github.com/promptfoo/promptfoo/pull/993)

Chores

- Block network calls in tests by [typpo](https://github.com/typpo) in [#972](https://github.com/promptfoo/promptfoo/pull/972)
- Export `AnthropicMessagesProvider` from providers by [greysteil](https://github.com/greysteil) in [#975](https://github.com/promptfoo/promptfoo/pull/975)
- Add Claude 3.5 sonnet pricing by [typpo](https://github.com/typpo) in [#976](https://github.com/promptfoo/promptfoo/pull/976)
- Pass `tool_choice` to Anthropic when set in config by [greysteil](https://github.com/greysteil) in [#977](https://github.com/promptfoo/promptfoo/pull/977)
- Fixed according to Ollama API specifications by [keishidev](https://github.com/keishidev) in [#981](https://github.com/promptfoo/promptfoo/pull/981)
- Add Dependabot config and update provider dependencies by [mldangelo](https://github.com/mldangelo) in [#984](https://github.com/promptfoo/promptfoo/pull/984)
- Don't commit `.env` to Git by [will-holley](https://github.com/will-holley) in [#991](https://github.com/promptfoo/promptfoo/pull/991)
- Update Docker base image to Node 20, improve self-hosting documentation, and add CI action for Docker build by [mldangelo](https://github.com/mldangelo) in [#995](https://github.com/promptfoo/promptfoo/pull/995)
- Allow variable cells to scroll instead of exploding the table height by [grrowl](https://github.com/grrowl) in [#973](https://github.com/promptfoo/promptfoo/pull/973)

New Contributors

- [keishidev](https://github.com/keishidev) made their first contribution in [#981](https://github.com/promptfoo/promptfoo/pull/981)
- [grrowl](https://github.com/grrowl) made their first contribution in [#973](https://github.com/promptfoo/promptfoo/pull/973)

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.65.2...0.66.0

What's Changed
* feat(redteam): make it easier to add non default plugins by typpo in https://github.com/promptfoo/promptfoo/pull/958
* feat: contains-sql assert by typpo in https://github.com/promptfoo/promptfoo/pull/964
* fix: handle absolute paths for js providers by typpo in https://github.com/promptfoo/promptfoo/pull/966
* fix(redteam): wire ui to plugins by typpo in https://github.com/promptfoo/promptfoo/pull/965
* fix(redteam): reduce overreliance, excessive-agency false positive rates by typpo in https://github.com/promptfoo/promptfoo/pull/963
* fix: label not showing problem when using eval with config option by suhwan-kang in https://github.com/promptfoo/promptfoo/pull/928
* fix: should return the whole message if the OpenAI return the content and the function call/tools at the same time. by qwlong in https://github.com/promptfoo/promptfoo/pull/968
* fix: label support for js prompts by typpo in https://github.com/promptfoo/promptfoo/pull/970
* docs: Add CLI delete command to docs by efung in https://github.com/promptfoo/promptfoo/pull/959
* docs: text to sql validation guide by typpo in https://github.com/promptfoo/promptfoo/pull/962

New Contributors
* suhwan-kang made their first contribution in https://github.com/promptfoo/promptfoo/pull/928
* qwlong made their first contribution in https://github.com/promptfoo/promptfoo/pull/968

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.65.1...0.65.2

Fixes related to web UI. See also [0.65.0 release notes](https://github.com/promptfoo/promptfoo/releases/tag/0.65.0)

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.65.0...0.65.1

What's Changed
* feat(webui): show pass/fail toggle by typpo in https://github.com/promptfoo/promptfoo/pull/938
* feat(webui): carousel for multiple failure reasons by typpo in https://github.com/promptfoo/promptfoo/pull/939
* feat(webui): clicking metric pills filters by nonzero only by typpo in https://github.com/promptfoo/promptfoo/pull/941
* feat: add hf to onboarding flow by typpo in https://github.com/promptfoo/promptfoo/pull/947
* feat: add support for `promptfoo export latest` by typpo in https://github.com/promptfoo/promptfoo/pull/948
* feat(redteam): political statements by typpo in https://github.com/promptfoo/promptfoo/pull/944
* feat(redteam): indicate performance with moderation filter by typpo in https://github.com/promptfoo/promptfoo/pull/933
* fix: serialize each item in `vars` when its type is a string (823) by YiFanChen99 in https://github.com/promptfoo/promptfoo/pull/943
* fix(redteam): more aggressive contract testing by typpo in https://github.com/promptfoo/promptfoo/pull/946
* chore(deps): update dependencies without breaking changes by mldangelo in https://github.com/promptfoo/promptfoo/pull/937
* chore(webui): split ResultsTable into separate files by typpo in https://github.com/promptfoo/promptfoo/pull/942


**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.64.0...0.65.0

What's Changed
* feat: export `promptfoo.cache` to node package by typpo in https://github.com/promptfoo/promptfoo/pull/923
* feat: add Voyage AI embeddings provider by typpo in https://github.com/promptfoo/promptfoo/pull/931
* feat(redteam): add unintended contracts test by mldangelo in https://github.com/promptfoo/promptfoo/pull/934
* feat(anthropic): support tool use by mldangelo in https://github.com/promptfoo/promptfoo/pull/932
* feat: Add more Portkey header provider options and create headers automatically by vrushankportkey in https://github.com/promptfoo/promptfoo/pull/909
* fix: handle openai chat-style messages better in `moderation` assert by typpo in https://github.com/promptfoo/promptfoo/pull/930
* fix(bedrock): support cohere embeddings by typpo in https://github.com/promptfoo/promptfoo/pull/924
* ci: add next.js build caching by typpo in https://github.com/promptfoo/promptfoo/pull/908
* chore(deps): bump braces from 3.0.2 to 3.0.3 by dependabot in https://github.com/promptfoo/promptfoo/pull/918
* chore(docs): update installation and GitHub Actions guides by mldangelo in https://github.com/promptfoo/promptfoo/pull/935
* chore(dependencies): bump LLM providers in package.json by mldangelo in https://github.com/promptfoo/promptfoo/pull/936

New Contributors
* vrushankportkey made their first contribution in https://github.com/promptfoo/promptfoo/pull/909

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.63.2...0.64.0

What's Changed
* fix(bedrock): default value for configs by typpo in https://github.com/promptfoo/promptfoo/pull/917