Promptfoo

What's Changed
* fix(vertex): correct handling of system instruction by typpo in https://github.com/promptfoo/promptfoo/pull/911
* fix(bedrock): support for llama, cohere command and command-r, mistral by typpo in https://github.com/promptfoo/promptfoo/pull/915

What's Changed
* feat(bedrock): Add support for mistral, llama, cohere by typpo in https://github.com/promptfoo/promptfoo/pull/885
* feat(ollama): add OLLAMA_API_KEY to support authentication by syndimann in https://github.com/promptfoo/promptfoo/pull/883
* feat(redteam): add test for competitor recommendations by typpo in https://github.com/promptfoo/promptfoo/pull/877
* feat(webui): Show the number of passes and failures by typpo in https://github.com/promptfoo/promptfoo/pull/888
* feat(webui): show manual grading record in test details view by typpo in https://github.com/promptfoo/promptfoo/pull/906
* feat(webui): use indexeddb instead of localstorage by typpo in https://github.com/promptfoo/promptfoo/pull/905
* feat: ability to set test case metric from csv by typpo in https://github.com/promptfoo/promptfoo/pull/889
* feat: interactive onboarding by typpo in https://github.com/promptfoo/promptfoo/pull/886
* feat: support `threshold` param from csv by typpo in https://github.com/promptfoo/promptfoo/pull/903
* feat: support array of values for `similar` assertion by typpo in https://github.com/promptfoo/promptfoo/pull/895
* fix(vertex): support var templating in system instruction by typpo in https://github.com/promptfoo/promptfoo/pull/902
* fix: Prompt variable reads unprocessed spaces on both sides by big-camel in https://github.com/promptfoo/promptfoo/pull/887
* fix: windows node 22 flake by typpo in https://github.com/promptfoo/promptfoo/pull/907
* [fix(webui): display latency when available](https://github.com/promptfoo/promptfoo/commit/bb335efbe9e8d6b23526c837402787a1cbba9969)
* [fix: ci passing despite failing build (](https://github.com/promptfoo/promptfoo/commit/ce6090be5d70fbe71c6da0a5ec1a73253a9d8a0e)https://github.com/promptfoo/promptfoo/pull/876[)](https://github.com/promptfoo/promptfoo/commit/ce6090be5d70fbe71c6da0a5ec1a73253a9d8a0e)
* [fix: incorrect migrations path in docker build](https://github.com/promptfoo/promptfoo/commit/6a1eef4e4b006b32de9ce6e5e2d7c0bd3b9fa95a) https://github.com/promptfoo/promptfoo/issues/861
* chore(ci): add `workflow_dispatch` trigger by mldangelo in https://github.com/promptfoo/promptfoo/pull/897
* chore(deps): update most dependencies to latest stable versions by mldangelo in https://github.com/promptfoo/promptfoo/pull/898
* chore: add more gemini models by typpo in https://github.com/promptfoo/promptfoo/pull/894
* chore: introduce eslint by mldangelo in https://github.com/promptfoo/promptfoo/pull/904
* chore: switch to SWC for faster Jest tests by mldangelo in https://github.com/promptfoo/promptfoo/pull/899
* chore: update to prettier 3 by mldangelo in https://github.com/promptfoo/promptfoo/pull/901
* [chore(openai): add tool_choice required type](https://github.com/promptfoo/promptfoo/commit/e97ce63221b0e06f7e03f46c466da36c5b713017)

New Contributors
* syndimann made their first contribution in https://github.com/promptfoo/promptfoo/pull/883

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.62.1...0.63.0

What's Changed
* fix: migrate database before writing results by typpo in https://github.com/promptfoo/promptfoo/pull/882
* feat(webui): Ability to suppress browser open on `promptfoo view` by typpo in https://github.com/promptfoo/promptfoo/pull/881


**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.62.0...0.62.1

Notes from 0.62.0 below

What's Changed
* chore: upgrade default graders to gpt-4o by typpo in https://github.com/promptfoo/promptfoo/pull/848
* ci: Introduce jest test coverage reports by mldangelo in https://github.com/promptfoo/promptfoo/pull/868
* ci: add support for node 22, remove support for node 16 by jacobdr in https://github.com/promptfoo/promptfoo/pull/836
* docs: Addresses minor typographical errors by mldangelo in https://github.com/promptfoo/promptfoo/pull/845
* docs: Help description of default `--output` by jamesbraza in https://github.com/promptfoo/promptfoo/pull/844
* feat(anthropic): add support for base url by typpo in https://github.com/promptfoo/promptfoo/pull/850
* feat(openai): Support function/tool callbacks by typpo in https://github.com/promptfoo/promptfoo/pull/830
* feat(vertex/gemini): add support for toolConfig and systemInstruction by typpo in https://github.com/promptfoo/promptfoo/pull/841
* feat(webui): Ability to filter to highlighted cells by typpo in https://github.com/promptfoo/promptfoo/pull/852
* feat(webui): ability to click to filter metric by typpo in https://github.com/promptfoo/promptfoo/pull/849
* feat(webui): add copy and highlight cell actions by typpo in https://github.com/promptfoo/promptfoo/pull/847
* feat: Add Red Team PII Tests by mldangelo in https://github.com/promptfoo/promptfoo/pull/862
* feat: Support custom gateway URLs in Portkey by roh26it in https://github.com/promptfoo/promptfoo/pull/840
* feat: add support for python embedding and classification providers by typpo in https://github.com/promptfoo/promptfoo/pull/864
* feat: add support for titan premier on bedrock by YiFanChen99 in https://github.com/promptfoo/promptfoo/pull/839
* feat: pass evalId in results by pelikhan in https://github.com/promptfoo/promptfoo/pull/758
* fix: Broken types by brendanmoore in https://github.com/promptfoo/promptfoo/pull/854
* fix: Fix broken progress callback in web ui by chrismaltais in https://github.com/promptfoo/promptfoo/pull/860
* fix: Fix formatting and add style check to CI by chrismaltais in https://github.com/promptfoo/promptfoo/pull/872
* fix: Fix type error eval page.tsx by drien in https://github.com/promptfoo/promptfoo/pull/867
* fix: Improve Error Handling for Python Assertions and Provider Exceptions by mldangelo in https://github.com/promptfoo/promptfoo/pull/863
* fix: Pass evaluateOptions from web ui yaml by chrismaltais in https://github.com/promptfoo/promptfoo/pull/859
* fix: Render multiple result images with markdown, if markdown contains multiple images by chrismaltais in https://github.com/promptfoo/promptfoo/pull/873
* fix: The values of defaultTest and evaluateOptions are not set when editing the eval yaml file. by big-camel in https://github.com/promptfoo/promptfoo/pull/834
* fix: crash on db migration when cache is disabled on first run by typpo in https://github.com/promptfoo/promptfoo/pull/842
* fix: csv and html outputs include both prompt and provider labels by typpo in https://github.com/promptfoo/promptfoo/pull/851
* fix: docker build and prepublish script by typpo in https://github.com/promptfoo/promptfoo/pull/846
* fix: show labels for custom provider by typpo in https://github.com/promptfoo/promptfoo/pull/875

New Contributors
* YiFanChen99 made their first contribution in https://github.com/promptfoo/promptfoo/pull/839
* roh26it made their first contribution in https://github.com/promptfoo/promptfoo/pull/840
* pelikhan made their first contribution in https://github.com/promptfoo/promptfoo/pull/758
* brendanmoore made their first contribution in https://github.com/promptfoo/promptfoo/pull/854
* drien made their first contribution in https://github.com/promptfoo/promptfoo/pull/867

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.61.0...0.62.0

What's Changed
* feat(anthropic): add support for base url by typpo in https://github.com/promptfoo/promptfoo/pull/850
* feat(openai): Support function/tool callbacks by typpo in https://github.com/promptfoo/promptfoo/pull/830
* feat(vertex/gemini): add support for toolConfig and systemInstruction by typpo in https://github.com/promptfoo/promptfoo/pull/841
* feat(webui): Ability to filter to highlighted cells by typpo in https://github.com/promptfoo/promptfoo/pull/852
* feat(webui): ability to click to filter metric by typpo in https://github.com/promptfoo/promptfoo/pull/849
* feat(webui): add copy and highlight cell actions by typpo in https://github.com/promptfoo/promptfoo/pull/847
* feat: Add Red Team PII Tests by mldangelo in https://github.com/promptfoo/promptfoo/pull/862
* feat: Support custom gateway URLs in Portkey by roh26it in https://github.com/promptfoo/promptfoo/pull/840
* feat: add support for python embedding and classification providers by typpo in https://github.com/promptfoo/promptfoo/pull/864
* feat: add support for titan premier on bedrock by YiFanChen99 in https://github.com/promptfoo/promptfoo/pull/839
* feat: pass evalId in results by pelikhan in https://github.com/promptfoo/promptfoo/pull/758
* feat: upgrade default graders to gpt-4o by typpo in https://github.com/promptfoo/promptfoo/pull/848
* fix: Broken types by brendanmoore in https://github.com/promptfoo/promptfoo/pull/854
* fix: Fix broken progress callback in web ui by chrismaltais in https://github.com/promptfoo/promptfoo/pull/860
* fix: Fix formatting and add style check to CI by chrismaltais in https://github.com/promptfoo/promptfoo/pull/872
* fix: Fix type error eval page.tsx by drien in https://github.com/promptfoo/promptfoo/pull/867
* fix: Improve Error Handling for Python Assertions and Provider Exceptions by mldangelo in https://github.com/promptfoo/promptfoo/pull/863
* fix: Pass evaluateOptions from web ui yaml by chrismaltais in https://github.com/promptfoo/promptfoo/pull/859
* fix: Render multiple result images with markdown, if markdown contains multiple images by chrismaltais in https://github.com/promptfoo/promptfoo/pull/873
* fix: The values of defaultTest and evaluateOptions are not set when editing the eval yaml file. by big-camel in https://github.com/promptfoo/promptfoo/pull/834
* fix: crash on db migration when cache is disabled on first run by typpo in https://github.com/promptfoo/promptfoo/pull/842
* fix: csv and html outputs include both prompt and provider labels by typpo in https://github.com/promptfoo/promptfoo/pull/851
* fix: docker build and prepublish script by typpo in https://github.com/promptfoo/promptfoo/pull/846
* fix: show labels for custom provider by typpo in https://github.com/promptfoo/promptfoo/pull/875
* ci: Introduce jest test coverage reports by mldangelo in https://github.com/promptfoo/promptfoo/pull/868
* ci: add support for node 22, remove support for node 16 by jacobdr in https://github.com/promptfoo/promptfoo/pull/836
* docs: Addresses minor typographical errors by mldangelo in https://github.com/promptfoo/promptfoo/pull/845
* docs: Help description of default `--output` by jamesbraza in https://github.com/promptfoo/promptfoo/pull/844

New Contributors
* YiFanChen99 made their first contribution in https://github.com/promptfoo/promptfoo/pull/839
* roh26it made their first contribution in https://github.com/promptfoo/promptfoo/pull/840
* pelikhan made their first contribution in https://github.com/promptfoo/promptfoo/pull/758
* brendanmoore made their first contribution in https://github.com/promptfoo/promptfoo/pull/854
* drien made their first contribution in https://github.com/promptfoo/promptfoo/pull/867

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.61.0...0.62.0

What's Changed
* feat: `moderation` assert type by typpo in https://github.com/promptfoo/promptfoo/pull/821
* feat: general purpose http/https provider by typpo in https://github.com/promptfoo/promptfoo/pull/822
* feat: add portkey provider by typpo in https://github.com/promptfoo/promptfoo/pull/819
* feat: Add Cloudflare AI Provider by jacobdr in https://github.com/promptfoo/promptfoo/pull/817
* fix: Remove duplicate logging line by efung in https://github.com/promptfoo/promptfoo/pull/825
* fix: The ‘defaultTest’ option has no effect during evaluation. by big-camel in https://github.com/promptfoo/promptfoo/pull/829
* fix: Improve Error Handling in Python Script Execution by mldangelo in https://github.com/promptfoo/promptfoo/pull/833
* docs: How to red team LLMs by typpo in https://github.com/promptfoo/promptfoo/pull/828
* chore(mistral): add codestral by typpo in https://github.com/promptfoo/promptfoo/pull/831

New Contributors
* big-camel made their first contribution in https://github.com/promptfoo/promptfoo/pull/829
* jacobdr made their first contribution in https://github.com/promptfoo/promptfoo/pull/817
* mldangelo made their first contribution in https://github.com/promptfoo/promptfoo/pull/833

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.60.0...0.61.0

What's Changed
* feat: redteam testset generation by typpo in https://github.com/promptfoo/promptfoo/pull/804
* feat: support for deep equality check in equals assertion by mikanfactory in https://github.com/promptfoo/promptfoo/pull/805
* feat(webui): Add image viewer by typpo in https://github.com/promptfoo/promptfoo/pull/816
* feat: Allow functions in renderVarsInObject by tomquist in https://github.com/promptfoo/promptfoo/pull/813
* feat: ability to reference previous llm outputs via storeOutputAs by typpo in https://github.com/promptfoo/promptfoo/pull/808
* feat: support for prompt objects by typpo in https://github.com/promptfoo/promptfoo/pull/818
* fix: huggingface api key handling by typpo in https://github.com/promptfoo/promptfoo/pull/809
* docs: Restore ProviderResponse class name by alexclaydon in https://github.com/promptfoo/promptfoo/pull/806
* docs: Fix typo in local build command by chrismaltais in https://github.com/promptfoo/promptfoo/pull/811

New Contributors
* mikanfactory made their first contribution in https://github.com/promptfoo/promptfoo/pull/805
* alexclaydon made their first contribution in https://github.com/promptfoo/promptfoo/pull/806
* chrismaltais made their first contribution in https://github.com/promptfoo/promptfoo/pull/811
* tomquist made their first contribution in https://github.com/promptfoo/promptfoo/pull/813

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.59.1...0.60.0