Promptfoo

What's Changed
* Support for custom functions as prompts by typpo in https://github.com/promptfoo/promptfoo/pull/147
* Refactor parts of util into more descriptive files by typpo in https://github.com/promptfoo/promptfoo/pull/148
* Misc fixes and improvements


**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.23.1...0.24.0

What's Changed
* Improvements to custom grading by typpo in https://github.com/promptfoo/promptfoo/pull/140
* Support for Google Vertex and PaLM chat APIs by typpo in https://github.com/promptfoo/promptfoo/pull/131
* Add support for including files in defaultTest by typpo in https://github.com/promptfoo/promptfoo/pull/137
* Add support for disabling cache in evaluate() options by typpo in https://github.com/promptfoo/promptfoo/pull/135
* Add support for loading vars directly from file by typpo in https://github.com/promptfoo/promptfoo/pull/139
* Include `provider` in `EvaluateResult`
* Other misc improvements and fixes


**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.22.0...0.23.1

What's Changed
* Improvements to custom grading by typpo in https://github.com/promptfoo/promptfoo/pull/140
* Support for Google Vertex and PaLM chat APIs by typpo in https://github.com/promptfoo/promptfoo/pull/131
* Add support for including files in defaultTest by typpo in https://github.com/promptfoo/promptfoo/pull/137
* Add support for disabling cache in evaluate() options by typpo in https://github.com/promptfoo/promptfoo/pull/135
* Add support for loading vars directly from file by typpo in https://github.com/promptfoo/promptfoo/pull/139
* Include `provider` in `EvaluateResult`
* Other misc improvements and fixes


**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.22.0...0.23.0

What's Changed
* Add OpenAI factuality and closed-QA graders by typpo in https://github.com/promptfoo/promptfoo/pull/126. These new graders implement OpenAI's eval methodology.
* Auto-escape vars when prompt is a JSON object by typpo in https://github.com/promptfoo/promptfoo/pull/127.
* Improvements to custom providers - Pass context including `vars` to callApi and make `TestCase` generic for ease of typing
* Add `prompt` to Javascript, Python, and Webhook assertion context
* Fix llama.cpp usage of provider config overrides
* Fix ollama provider parsing for llama versions like llama:13b, llama:70b etc.
* Trim var strings in CLI table (prevents slowness during CLI table output)


**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.21.4...0.22.0

What's Changed
* Add support for test case threshold value by typpo in https://github.com/promptfoo/promptfoo/pull/125
* Add support for pass/fail threshold for javascript and python numeric return values


**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.21.3...0.21.4

What's Changed
* Increase request backoff and add optional delay between API calls by typpo in https://github.com/promptfoo/promptfoo/pull/122


**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.21.2...0.21.3