Modelgauge

What's Changed
* Add the wildguard private annotator, with some refactoring. by rogthefrog in https://github.com/mlcommons/modelgauge/pull/554
* HuggingFace Inference SUT by bkorycki in https://github.com/mlcommons/modelgauge/pull/561
* Safetests use first batch of v1.0 prompts by bkorycki in https://github.com/mlcommons/modelgauge/pull/563

**Full Changelog**: https://github.com/mlcommons/modelgauge/compare/v0.6.2...v0.6.3

What's Changed
* Officially add new annotators by bkorycki in https://github.com/mlcommons/modelgauge/pull/550



**Full Changelog**: https://github.com/mlcommons/modelgauge/compare/v0.6.1...v0.6.2

What's Changed
* Fix bug where bad raw annotations are cached forever
* Remove safetest base class
* Minor improvements for pipeline debugging
* Adding 'system' role to openai_client _ROLE_MAP by shachihk-intel
* Better together API errors
* Keep track of items that can't be processed
* Updated dependencies and add notebook linter
* Remove deprecated Together models, and update tests to match

New Contributors
* rogthefrog made their first contribution in https://github.com/mlcommons/modelgauge/pull/512
* shachihk-intel made their first contribution in https://github.com/mlcommons/modelgauge/pull/534

**Full Changelog**: https://github.com/mlcommons/modelgauge/compare/v0.6.0...v0.6.1

What's Changed
* Together and HuggingFace SUTs can now return log probs in their responses when requested.
* New CLI option `--plugin-dir` loads local plugins at runtime.
* Increase reliability of downloading test data.
* Prepare modelgauge infra files for safety evaluator testing (new "System" chat role, minor `llama_guard_annotator` refactor).
* Documentation updates, including initial API reference.
* Introduce `Pipeline` and related classes to serve as the base for a composable set of objects that handle common bulk processing tasks like running prompts, getting annotations, and any other slow I/O-bound workloads.
* SafeTests use files from dev deployment of modellab.
* New `run-csv-items` command quickly runs batches of prompts and/or responses in a CSV file through some SUTs and/or annotators.
* Add new v1.0 SafeTest class and place-holder test `safe-dfm-1.0`. Version 0.5 tests (e.g. `safe-cae`) are not affected.
* Move Together plugin files + SafeTest into core modelgauge library.

New Contributors
* tsunamit made their first contribution in https://github.com/mlcommons/modelgauge/pull/449
* HuaizhengZhang made their first contribution in https://github.com/mlcommons/modelgauge/pull/489

**Full Changelog**: https://github.com/mlcommons/modelgauge/compare/v0.5.1...v0.6.0

What's Changed
* Updated docs
* SafeTest compatible with python 3.11+
* Add new [Llama Guard 2](https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-guard-2) to `LlamaGuardAnnotator`
* Can configure `LlamaGuardAnnotator` with optional `llama_guard_version` parameter. Defaults to Llama Guard 2
* Minor changes to prompt/category formatting for Llama Guard 1. This may affect results.
* SafeTest can also be configured to use Llama Guard 1 or 2 as it's annotator. Defaults to version 2.


**Full Changelog**: https://github.com/mlcommons/modelgauge/compare/v0.5.0...v0.5.1

What's Changed

* Renamed to ModelGauge and started pushing to PyPI!
* A whole bunch of cleanups and preparation for the more public release.
* Caching now supports dicts.
* Unit tests to ensure you can install from PyPI and run in a notebook.
* Expand range of supported python versions to 3.10 and up.
* Remove benign hazard from SafeTest.
* Start setting up ReadTheDocs.


**Full Changelog**: https://github.com/mlcommons/modelgauge/compare/v0.3.3...v0.5.0