Promptfoo

What's Changed
* feat: add `delete eval latest` and `delete eval all` by typpo in https://github.com/promptfoo/promptfoo/pull/1383
* feat: bfla and bofa plugins by typpo in https://github.com/promptfoo/promptfoo/pull/1406
* feat: Support loading tools from multiple files by x1a0 in https://github.com/promptfoo/promptfoo/pull/1384
* feat: `promptfoo eval --description` override by typpo in https://github.com/promptfoo/promptfoo/pull/1399
* feat: add `default` strategy and remove `--add-strategies` by typpo in https://github.com/promptfoo/promptfoo/pull/1401
* feat: assume unrecognized openai models are chat models by typpo in https://github.com/promptfoo/promptfoo/pull/1404
* feat: excessive agency grader looks at tools by typpo in https://github.com/promptfoo/promptfoo/pull/1403
* fix(types): allow boolean values in VarsSchema by mldangelo in https://github.com/promptfoo/promptfoo/pull/1386
* fix: dont check SSL certs by typpo in https://github.com/promptfoo/promptfoo/pull/1396
* fix: reduce rbac and moderation false positives by typpo in https://github.com/promptfoo/promptfoo/pull/1400
* fix: `redteam` property was not read in config by typpo in https://github.com/promptfoo/promptfoo/pull/1407
* fix: Do not ignored derived metrics by tomquist in https://github.com/promptfoo/promptfoo/pull/1381
* fix: add indexes for sqlite by typpo in https://github.com/promptfoo/promptfoo/pull/1382
* chore(deps-dev): bump aws-sdk/client-bedrock-runtime from 3.623.0 to 3.624.0 by dependabot in https://github.com/promptfoo/promptfoo/pull/1379
* chore(deps): bump openai from 4.54.0 to 4.55.0 by dependabot in https://github.com/promptfoo/promptfoo/pull/1387
* chore(deps): bump openai from 4.55.0 to 4.55.1 by dependabot in https://github.com/promptfoo/promptfoo/pull/1392
* chore(deps): bump anthropic-ai/sdk from 0.25.0 to 0.25.1 by dependabot in https://github.com/promptfoo/promptfoo/pull/1397
* chore(deps): bump openai from 4.55.1 to 4.55.3 by dependabot in https://github.com/promptfoo/promptfoo/pull/1398

New Contributors
* x1a0 made their first contribution in https://github.com/promptfoo/promptfoo/pull/1384

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.75.2...0.76.0

What's Changed
* fix: Update "Edit Comment" dialog background for the dark mode by dimusic in https://github.com/promptfoo/promptfoo/pull/1374
* fix: undefined var in hallucination template by typpo in https://github.com/promptfoo/promptfoo/pull/1375
* fix: restore harmCategory var by typpo in https://github.com/promptfoo/promptfoo/pull/1380

New Contributors
* dimusic made their first contribution in https://github.com/promptfoo/promptfoo/pull/1374

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.75.1...0.75.2

What's Changed

* chore(deps): update dependencies by mldangelo in https://github.com/promptfoo/promptfoo/pull/1373
* fix: temporarily disable nunjucks strict mode by typpo

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.75.0...0.75.1

What's Changed

Features
* feat(webui): Download report as PDF by typpo in 1348
* feat(redteam): Add custom policy plugin by mldangelo in 1346
* feat: Implement defaultTest metadata in tests and scenarios by mldangelo in 1361
* feat(config): Add writePromptfooConfig function and orderKeys utility by mldangelo in 1360
* feat(redteam): Add purpose and entities to defaultTest metadata by mldangelo in 1359
* feat(webui): Show metadata in details dialog by typpo in 1362
* feat(redteam): Add some simple requested strategies by typpo in 1364
* feat!: Add `default` plugin collection and remove --add-plugins by typpo in 1369

Bug Fixes
* fix: Moderation assert and iterative provider handle output objects by typpo in 1353
* fix: Improve PII grader by typpo in 1354
* fix: Improve RBAC grading by typpo in 1347
* fix: Make graders set assertion value by typpo in 1355
* fix: Allow falsy provider response outputs by typpo in 1356
* fix: Improve entity extraction and enable for PII by typpo in 1358
* fix: Do not dereference external tool files by typpo in 1357
* fix: Google sheets output by typpo in 1367

Documentation
* docs: How to red team RAG applications by typpo in 1368

Refactoring
* refactor(redteam): Consolidate graders and plugins by mldangelo in 1370

Chores
* chore(deps): Bump openai from 4.53.2 to 4.54.0 by dependabot in 1349
* chore(redteam): Collect user consent for harmful generation by typpo in 1365
* chore(deps-dev): Bump aws-sdk/client-bedrock-runtime from 3.622.0 to 3.623.0 by dependabot in 1372

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.74.0...0.75.0

What's Changed

Features
- **feat**: Split types vs validators for prompts, providers, and redteam [1325](https://github.com/promptfoo/promptfoo/pull/1325) by [typpo](https://github.com/typpo)
- **feat**: Load provider `tools` and `functions` from external file [1342](https://github.com/promptfoo/promptfoo/pull/1342) by [typpo](https://github.com/typpo)

Fixes
- **fix**: Show gray icon when there are no tests in report [1335](https://github.com/promptfoo/promptfoo/pull/1335) by [typpo](https://github.com/typpo)
- **fix**: numTests calculation for previous evals [1336](https://github.com/promptfoo/promptfoo/pull/1336) by [onyck](https://github.com/onyck)
- **fix**: Only show the number of tests actually run in the eval [1338](https://github.com/promptfoo/promptfoo/pull/1338) by [typpo](https://github.com/typpo)
- **fix**: better-sqlite3 in arm64 docker image [1344](https://github.com/promptfoo/promptfoo/pull/1344) by [cmrfrd](https://github.com/cmrfrd)
- **fix**: Correct positive example in DEFAULT_GRADING_PROMPT [1337](https://github.com/promptfoo/promptfoo/pull/1337) by [tbuckley](https://github.com/tbuckley)

Chores
- **chore**: Integrate red team evaluation into promptfoo init [1334](https://github.com/promptfoo/promptfoo/pull/1334) by [mldangelo](https://github.com/mldangelo)
- **chore(deps)**: Bump azure/identity from 4.4.0 to 4.4.1 [1340](https://github.com/promptfoo/promptfoo/pull/1340) by [dependabot](https://github.com/dependabot)
- **chore(deps)**: Bump the github-actions group with 3 updates [1339](https://github.com/promptfoo/promptfoo/pull/1339) by [dependabot](https://github.com/dependabot)
- **chore**: Enforce consistent type imports [1341](https://github.com/promptfoo/promptfoo/pull/1341) by [mldangelo](https://github.com/mldangelo)

Refactoring
- **refactor(redteam)**: Update plugin architecture and improve error handling [1343](https://github.com/promptfoo/promptfoo/pull/1343) by [mldangelo](https://github.com/mldangelo)

Documentation
- **docs**: Expand installation instructions in README and docs [1345](https://github.com/promptfoo/promptfoo/pull/1345) by [mldangelo](https://github.com/mldangelo)


New Contributors
- [tbuckley](https://github.com/tbuckley) made their first contribution in [#1337](https://github.com/promptfoo/promptfoo/pull/1337)

**Full Changelog**: [0.73.9...0.74.0](https://github.com/promptfoo/promptfoo/compare/0.73.9...0.74.0)

What's Changed
- **Features:**
- feat: handle rate limits in generic fetch path by typpo in https://github.com/promptfoo/promptfoo/pull/1324

- **Fixes:**
- fix: show default vars in table by typpo in https://github.com/promptfoo/promptfoo/pull/1306
- fix: chain provider and test transform by fvdnabee in https://github.com/promptfoo/promptfoo/pull/1316

- **Refactors:**
- refactor(redteam): extract entity and purpose logic, update imitation plugin by mldangelo in https://github.com/promptfoo/promptfoo/pull/1301

- **Chores:**
- chore(deps): bump openai from 4.53.1 to 4.53.2 by dependabot in https://github.com/promptfoo/promptfoo/pull/1314
- chore: set page titles by typpo in https://github.com/promptfoo/promptfoo/pull/1315
- chore: add devcontainer setup by cmrfrd in https://github.com/promptfoo/promptfoo/pull/1317
- chore(webui): persist column selection in evals view by mldangelo in https://github.com/promptfoo/promptfoo/pull/1302
- chore(redteam): allow multiple provider selection by mldangelo in https://github.com/promptfoo/promptfoo/pull/1319
- chore(deps): bump anthropic-ai/sdk from 0.24.3 to 0.25.0 by dependabot in https://github.com/promptfoo/promptfoo/pull/1326
- chore(deps-dev): bump aws-sdk/client-bedrock-runtime from 3.620.0 to 3.620.1 by dependabot in https://github.com/promptfoo/promptfoo/pull/1327
- chore(deps): update dev dependencies and minor package versions by mldangelo in https://github.com/promptfoo/promptfoo/pull/1331

- **CI/CD:**
- ci: add assets generation job and update json schema by mldangelo in https://github.com/promptfoo/promptfoo/pull/1321

- **Documentation:**
- docs: add CITATION.cff file by mldangelo in https://github.com/promptfoo/promptfoo/pull/1322
- docs: update examples and docs to use gpt-4o and gpt-4o-mini models by mldangelo in https://github.com/promptfoo/promptfoo/pull/1323

New Contributors
- cmrfrd made their first contribution in https://github.com/promptfoo/promptfoo/pull/1317

**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.73.8...0.73.9