Breaking update to python custom assertions
Python assertions now expect a `get_assert` function which returns a native value, rather than parsing stdout (594). This means instead of:
py
print(json.dumps((result))
You should just return the assertion result:
py
return result
Here's a full example of a `custom_assert.py`:
py
def get_assert(output, context) -> Union[bool, float, Dict[str, Any]]
print('Prompt:', context['prompt'])
print('Vars', context['vars']['topic']
Determine the result...
result = test_output(output)
Here's an example GradingResult dict
result = {
'pass': True,
'score': 0.6,
'reason': 'Looks good to me',
}
return result
See [documentation](https://promptfoo.dev/docs/configuration/expected-outputs/python/#external-py)
What's Changed
* chore: improve json parsing errors by typpo in https://github.com/promptfoo/promptfoo/pull/620
* feat: ability to override path to python binary by typpo in https://github.com/promptfoo/promptfoo/pull/619
* feat(webui): store settings in localstorage by typpo in https://github.com/promptfoo/promptfoo/pull/617
* feat(azureopenai): apiKeyEnvar support by typpo in https://github.com/promptfoo/promptfoo/pull/628
* Add documentation for openai vision by CamdenClark in https://github.com/promptfoo/promptfoo/pull/637
* Support claude vision and images by CamdenClark in https://github.com/promptfoo/promptfoo/pull/639
* fix(webui): ability to save defaultTest and evaluateOptions in yaml editor by typpo in https://github.com/promptfoo/promptfoo/pull/629
* fix: assertion files use relative path by typpo in https://github.com/promptfoo/promptfoo/pull/624
* feat: add provider reference to prompt function by guilhermetk in https://github.com/promptfoo/promptfoo/pull/633
* feat(webui): "progress" page that shows provider/prompt pairs by typpo in https://github.com/promptfoo/promptfoo/pull/631
* feat: ability to import vars using glob by typpo in https://github.com/promptfoo/promptfoo/pull/641
* feat!: return values directly in python assertions by typpo in https://github.com/promptfoo/promptfoo/pull/638
New Contributors
* CamdenClark made their first contribution in https://github.com/promptfoo/promptfoo/pull/637
* guilhermetk made their first contribution in https://github.com/promptfoo/promptfoo/pull/633
**Full Changelog**: https://github.com/promptfoo/promptfoo/compare/0.50.1...0.51.0