Garak

What's Changed
Tuning & fixes
* defensive coding around capture results in xss content detector by jmartin-tech https://github.com/NVIDIA/garak/pull/1120

**Full Changelog**: https://github.com/NVIDIA/garak/compare/v0.10.3...v0.10.3.1

What's Changed
New plugins
* Added watsonx.ai generator by iamnotcj in https://github.com/NVIDIA/garak/pull/1058
* Additional XSS Exfil Probes by erickgalinkin in https://github.com/NVIDIA/garak/pull/1060

New features
* Generators: add option to skip output btw given delimiters; add hook for postprocessing by leondz in https://github.com/NVIDIA/garak/pull/1097
* Feature: align hallucinated package named with outputs by leondz in https://github.com/NVIDIA/garak/pull/1076
* support `module.classname` config specification for plugins by jmartin-tech in https://github.com/NVIDIA/garak/pull/1108

Documentation
* pedantic spelling by dltemple in https://github.com/NVIDIA/garak/pull/1085
* update contributing checklist numbers to be ascending by shane-rosse in https://github.com/NVIDIA/garak/pull/1115

Tuning & fixes
* update: reorder ditw prompts by leondz in https://github.com/NVIDIA/garak/pull/1098
* update openai model list by leondz in https://github.com/NVIDIA/garak/pull/1100
* update: Give clearer message when header encoding fails by leondz in https://github.com/NVIDIA/garak/pull/1088

What's Changed
* uninclude donotanswer from default probes by leondz in https://github.com/NVIDIA/garak/pull/1111
* stop forcing generation count for mini phrasing probes by leondz in https://github.com/NVIDIA/garak/pull/1109

New Contributors
* iamnotcj made their first contribution in https://github.com/NVIDIA/garak/pull/1058
* dltemple made their first contribution in https://github.com/NVIDIA/garak/pull/1085
* shane-rosse made their first contribution in https://github.com/NVIDIA/garak/pull/1115

**Full Changelog**: https://github.com/NVIDIA/garak/compare/v0.10.2...v.0.10.3

What's Changed
New plugins
* Detector: Shields for testing LLM Application Firewalls by Eric-Hacker in https://github.com/NVIDIA/garak/pull/1059

New features
* Warn if api_key in Config by erickgalinkin in https://github.com/NVIDIA/garak/pull/1049
* Feature: configuration based rest proxy support by jmartin-tech in https://github.com/NVIDIA/garak/pull/1073
* Feature: configuration based rest ssl suppression by jmartin-tech in https://github.com/NVIDIA/garak/pull/1074
* Detector: add `startswith` matching for `StringDetector` base class by leondz in https://github.com/NVIDIA/garak/pull/1075

Documentation
* docs: update index.rst by eltociear in https://github.com/NVIDIA/garak/pull/1053
* docs: update readme by leondz in https://github.com/NVIDIA/garak/pull/1050
* adjust label ref syntax by jmartin-tech in https://github.com/NVIDIA/garak/pull/1057

Tuning & fixes
* `StringDetector.case_sensitive` now operates as expected by leondz in https://github.com/NVIDIA/garak/pull/1072
* guard against `None` in `leakreplay` attempt history management by leondz in https://github.com/NVIDIA/garak/pull/1081
* init zscore with enough scope to cover its uses by leondz in https://github.com/NVIDIA/garak/pull/1086
* Update for latest ollama support by jmartin-tech in https://github.com/NVIDIA/garak/pull/1092

New Contributors
* eltociear made their first contribution in https://github.com/NVIDIA/garak/pull/1053
* Eric-Hacker made their first contribution in https://github.com/NVIDIA/garak/pull/1059

**Full Changelog**: https://github.com/NVIDIA/garak/compare/v0.10.1...v0.10.2

What's Changed
New plugins
* probe: ANSI terminal takeover by leondz in https://github.com/NVIDIA/garak/pull/1025
* probe: Past Tense Vulnerability by Shine-afk in https://github.com/NVIDIA/garak/pull/924
* generator: promote OpenAICompatible as first class generator by jmartin-tech in https://github.com/NVIDIA/garak/pull/1021

New features
* feature: amend garak user-agent by leondz in https://github.com/NVIDIA/garak/pull/960
* feature: reinforce `trust_remote_code=False` by leondz in https://github.com/NVIDIA/garak/pull/979
* generators: add option to specify HTTP codes to skip generation on, for `RestGenerator` by leondz in https://github.com/NVIDIA/garak/pull/999
* feature: add experimental features flag to be only accessible in core config by leondz in https://github.com/NVIDIA/garak/pull/1003
* feature: enable configuration updates by jmartin-tech in https://github.com/NVIDIA/garak/pull/1026
* feature: use hf chat support by jmartin-tech in https://github.com/NVIDIA/garak/pull/1047
* feature: per-probe tags now adjustable based on payload selection by leondz in https://github.com/NVIDIA/garak/pull/1031

Documentation
* doc: Update README.md to notify of migration by leondz in https://github.com/NVIDIA/garak/pull/1002
* docs: add arxiv lozenge by leondz in https://github.com/NVIDIA/garak/pull/1030

Tuning & fixes
* remove no longer needed skip-duplicate-actions by jmartin-tech in https://github.com/NVIDIA/garak/pull/976
* Sanity test for pip install from repository by jmartin-tech in https://github.com/NVIDIA/garak/pull/1000
* add CODEOWNERS by jmartin-tech in https://github.com/NVIDIA/garak/pull/1001
* Update reference to reflect migration of org by jmartin-tech in https://github.com/NVIDIA/garak/pull/1004
* use allowed action name by jmartin-tech in https://github.com/NVIDIA/garak/pull/1005
* Fixing a few typos by zoenolan in https://github.com/NVIDIA/garak/pull/1006
* Fix huggingface inference endpoint name by jmartin-tech in https://github.com/NVIDIA/garak/pull/1011
* detect if tokenizer is not loaded and adjust by jmartin-tech in https://github.com/NVIDIA/garak/pull/1012
* fix: correct argument in TAP by harshraj172 in https://github.com/NVIDIA/garak/pull/1022
* refactor attempt to utilized property annotations by jmartin-tech in https://github.com/NVIDIA/garak/pull/1027
* Restrict transformers version until MPS issue is addressed by jmartin-tech in https://github.com/NVIDIA/garak/pull/1039
* pass device to transfomers pipeline by jmartin-tech in https://github.com/NVIDIA/garak/pull/1042
* Fix tap.PAIR run error - run_tap param evaluator_model by rafaelsandroni in https://github.com/NVIDIA/garak/pull/1045

New Contributors
* zoenolan made their first contribution in https://github.com/NVIDIA/garak/pull/1006
* harshraj172 made their first contribution in https://github.com/NVIDIA/garak/pull/1022
* Shine-afk made their first contribution in https://github.com/NVIDIA/garak/pull/924
* rafaelsandroni made their first contribution in https://github.com/NVIDIA/garak/pull/1045

**Full Changelog**: https://github.com/NVIDIA/garak/compare/v0.10.0...v0.10.1

What's Changed
New plugins
* detector: llm as a judge by jmartin-tech in https://github.com/leondz/garak/pull/956
* generator: update `openai` for o1 models by leondz in https://github.com/leondz/garak/pull/922
* generator: Ollama by martinebl in https://github.com/leondz/garak/pull/876
* generator: support of azure openai by eric-therond in https://github.com/leondz/garak/pull/817
* generator: vision nims by leondz in https://github.com/leondz/garak/pull/959
* probe: add whois injection to latentinjection by leondz in https://github.com/leondz/garak/pull/947

New features
* payload/probe separation by leondz in https://github.com/leondz/garak/pull/870
* data file override support with precedence by jmartin-tech in https://github.com/leondz/garak/pull/916
* cli: `--parallel_attempts` hint by leondz in https://github.com/leondz/garak/pull/932
* reporting: tidy config details, add payload audit info by leondz in https://github.com/leondz/garak/pull/936
* payloads: web-related, incl. `probes.encoding` migration by leondz in https://github.com/leondz/garak/pull/933

Documentation
* Document configuration of garak using YAML by leondz in https://github.com/leondz/garak/pull/911
* RST formatting updates to restgenerator page by leondz in https://github.com/leondz/garak/pull/912
* doc: give generatiors.rest list the space it wants and deserves by leondz in https://github.com/leondz/garak/pull/914
* latent injection doc fix by leondz in https://github.com/leondz/garak/pull/921
* Contributing Probes documentation by erickgalinkin in https://github.com/leondz/garak/pull/919
* docs: add example of using `Configurable` to specify a generator name for `load_plugin` by leondz in https://github.com/leondz/garak/pull/929
* docs: NVIDIA generators by leondz in https://github.com/leondz/garak/pull/917
* update bag reference by emmanuel-ferdman in https://github.com/leondz/garak/pull/953
* docs: update faq by leondz in https://github.com/leondz/garak/pull/957
* docs: describe top-level objectives in reference guide by leondz in https://github.com/leondz/garak/pull/958
* docs: centralise faq by leondz in https://github.com/leondz/garak/pull/965
* docs: make project scope clear and note that contributors should check it by leondz in https://github.com/leondz/garak/pull/964
* doc: update to use rst toctree href syntax by leondz in https://github.com/leondz/garak/pull/967
* docs: clarify "contributing" document by leondz in https://github.com/leondz/garak/pull/968
* docs: require core config top- and second-level params to be documented by leondz in https://github.com/leondz/garak/pull/966

Tuning & fixes
* plugin classes should not access `DEFAULT_PARAMS` by jmartin-tech in https://github.com/leondz/garak/pull/906
* add test status lozenges for lin/win/osx by leondz in https://github.com/leondz/garak/pull/903
* utilized the cache to filter `base` classes by jmartin-tech in https://github.com/leondz/garak/pull/905
* paraphrase fast consistent model device by jmartin-tech in https://github.com/leondz/garak/pull/898
* Force cache build in automation by jmartin-tech in https://github.com/leondz/garak/pull/907
* tweak: defer import of slow Llava* classes by leondz in https://github.com/leondz/garak/pull/908
* add issue and PR templates by jmartin-tech in https://github.com/leondz/garak/pull/909
* tweak: new latentinjection probe, add configurability to mini version by leondz in https://github.com/leondz/garak/pull/910
* bug: set match type correctly in `riskywords` detectors by leondz in https://github.com/leondz/garak/pull/918
* bugfix: enable latent injection mini latent jailbreak probe by default by leondz in https://github.com/leondz/garak/pull/923
* tweak: update migitation strings by leondz in https://github.com/leondz/garak/pull/925
* bugfix: use distinct report entry type for payload init by leondz in https://github.com/leondz/garak/pull/930
* add code coverage options by jmartin-tech in https://github.com/leondz/garak/pull/934
* reporting: update link to bag by leondz in https://github.com/leondz/garak/pull/935
* probe: missing newline in `latentinjection.LatentInjectionFactSnippetEiffel` by leondz in https://github.com/leondz/garak/pull/938
* update: extend calibration with more probes by leondz in https://github.com/leondz/garak/pull/939
* limit project actions to primary repo by jmartin-tech in https://github.com/leondz/garak/pull/940
* Add error handling for empty `node_results` to `base.py`. by erickgalinkin in https://github.com/leondz/garak/pull/942
* update: rename `replay` plugins to `divergence` by leondz in https://github.com/leondz/garak/pull/945
* update hint to reference --parallel_attempts by jmartin-tech in https://github.com/leondz/garak/pull/951
* Configurable TAP probe, refactor judge resources to shared red team by jmartin-tech in https://github.com/leondz/garak/pull/949
* architecture: factor HFCompatible out by leondz in https://github.com/leondz/garak/pull/954
* generators: fix rasa issues 961 & 962 by rgstephens in https://github.com/leondz/garak/pull/963
* fix: add garak/data to pip package by jmartin-tech in https://github.com/leondz/garak/pull/969

New Contributors
* martinebl made their first contribution in https://github.com/leondz/garak/pull/876
* emmanuel-ferdman made their first contribution in https://github.com/leondz/garak/pull/953
* eric-therond made their first contribution in https://github.com/leondz/garak/pull/817

**Full Changelog**: https://github.com/leondz/garak/compare/v0.9.0.16...v0.10.0

What's Changed

New plugins

* probe: topic pushing by leondz in https://github.com/leondz/garak/pull/764
* probe: ruby package hallucination by arjun-krishna1 in https://github.com/leondz/garak/pull/851
* probe: Latent prompt injection by leondz in https://github.com/leondz/garak/pull/877
* probe: npm package hallucination by arjun-krishna1 in https://github.com/leondz/garak/pull/861
* probe: rust crate hallucination by arjun-krishna1 in https://github.com/leondz/garak/pull/873
* generator: Groq API by mmilenkovic-groq in https://github.com/leondz/garak/pull/896

New features

* extract `generations` param to be set on probes instead of generators by jmartin-tech in https://github.com/leondz/garak/pull/837
* secure garak HF assets by leondz in https://github.com/leondz/garak/pull/854
* Rename `knownbadsignatures` to `av_spam_scanning` by leondz in https://github.com/leondz/garak/pull/850
* enable tokenizer customization in HFDetector by jmartin-tech in https://github.com/leondz/garak/pull/855
* refactor calibration / z-score code, so z-scores can be shown in CLI at run time by leondz in https://github.com/leondz/garak/pull/847

Tuning & fixes

* add automation action for plugin_cache.json by jmartin-tech in https://github.com/leondz/garak/pull/819
* add automation action for plugin_cache.json by jmartin-tech in https://github.com/leondz/garak/pull/819
* Docs/readmefaqdc32 by leondz in https://github.com/leondz/garak/pull/828
* fix initialization bug for kwargs in function by soumilinandi in https://github.com/leondz/garak/pull/827
* suppress LiteLLM logging during import by jmartin-tech in https://github.com/leondz/garak/pull/834
* relax `--probe_tags` as supported when other probe options are passed by jmartin-tech in https://github.com/leondz/garak/pull/836
* rename harness test module by leondz in https://github.com/leondz/garak/pull/848
* Surface underlying exception msg at plugin load by leondz in https://github.com/leondz/garak/pull/846
* relax litellm provider constraint by arjun-krishna1 in https://github.com/leondz/garak/pull/820
* plugin metadata tests by leondz in https://github.com/leondz/garak/pull/849
* Add explanation of how we treat the word "vulnerability" by leondz in https://github.com/leondz/garak/pull/852
* add faq items by leondz in https://github.com/leondz/garak/pull/857
* test buff transform() methods, to get them to load and run by leondz in https://github.com/leondz/garak/pull/856
* expand doc string that fails test requirements for verbosity by jmartin-tech in https://github.com/leondz/garak/pull/859
* refresh tutorial on contributing a generator by leondz in https://github.com/leondz/garak/pull/858
* avoid lambda in `_config` module level code by jmartin-tech in https://github.com/leondz/garak/pull/860
* add logging dir, new generator questions to FAQ by leondz in https://github.com/leondz/garak/pull/866
* revise default generations value to 5 by leondz in https://github.com/leondz/garak/pull/868
* validate detector doc_uri by leondz in https://github.com/leondz/garak/pull/865
* fix output error when parallel_requests > 1 and supports_multiple_generations is False by arjun-krishna1 in https://github.com/leondz/garak/pull/864
* improved messaging & error handling around NIM names by leondz in https://github.com/leondz/garak/pull/867
* look at the right var for NIM name checking by leondz in https://github.com/leondz/garak/pull/871
* filter detector template in tests by jmartin-tech in https://github.com/leondz/garak/pull/874
* have CLI UI render 100% w/o using scientific notation by leondz in https://github.com/leondz/garak/pull/882
* Generator docs are now tested for as intended by leondz in https://github.com/leondz/garak/pull/897
* reporting tweaks by leondz in https://github.com/leondz/garak/pull/901

New Contributors
* soumilinandi made their first contribution in https://github.com/leondz/garak/pull/827
* arjun-krishna1 made their first contribution in https://github.com/leondz/garak/pull/820
* mmilenkovic-groq made their first contribution in https://github.com/leondz/garak/pull/896
* greshake made their first contribution in https://github.com/leondz/garak/pull/877

**Full Changelog**: https://github.com/leondz/garak/compare/v0.9.0.15...v0.9.0.16