Garak

What's Changed

New plugins
* probe: assess 🤗 repos for potentially malicious files (`fileformats`) by leondz in https://github.com/leondz/garak/pull/767

New features
* use XDG paths for configuration data and caching by jmartin-tech in https://github.com/leondz/garak/pull/799
* Report comparative info for detector scores by leondz in https://github.com/leondz/garak/pull/814
* enable HFDetector model configuration with hf_args by jmartin-tech in https://github.com/leondz/garak/pull/810
* Flags for varying NIM seed and temperature every call by leondz in https://github.com/leondz/garak/pull/808

Tuning & fixes
* don't skip listing probes in 100%-pass-rate modules by leondz in https://github.com/leondz/garak/pull/781
* make default regex `raw` to remove python 3.12 warnings by jmartin-tech in https://github.com/leondz/garak/pull/809
* suppress HF `warning`s and lower during detector model instantiation by leondz in https://github.com/leondz/garak/pull/807
* validate cache consistency on first access by jmartin-tech in https://github.com/leondz/garak/pull/815
* clarify `continuation` probes by leondz in https://github.com/leondz/garak/pull/813
* print logfile path when invoked from CLI by leondz in https://github.com/leondz/garak/pull/811
* update slurprompts filename in encoding pkg by leondz in https://github.com/leondz/garak/pull/818

**Full Changelog**: https://github.com/leondz/garak/compare/v0.9.0.14...v0.9.0.15

What's Changed

New plugins
* BEAST Attack Implementation by erickgalinkin in https://github.com/leondz/garak/pull/728
* add probe for "grandma" attack by leondz in https://github.com/leondz/garak/pull/740
* generator: lorem ipsum by leondz in https://github.com/leondz/garak/pull/776

New features
* Feature: configurable plugins by jmartin-tech in https://github.com/leondz/garak/pull/711
* Add JSON Path support for RestGenerator by leondz in https://github.com/leondz/garak/pull/651
* Make banner line test dynamic by jmartin-tech in https://github.com/leondz/garak/pull/670
* Tutorial for contributing a generator by leondz in https://github.com/leondz/garak/pull/688
* add 'complete' message to report at end of run by leondz in https://github.com/leondz/garak/pull/735
* support chat modality in attempts by leondz in https://github.com/leondz/garak/pull/644
* add grandma substance synth probe, update docs and var names by leondz in https://github.com/leondz/garak/pull/763
* update NVCF to support versioning, custom payload keys by leondz in https://github.com/leondz/garak/pull/772
* WINDOWS & OSX SUPPORT: test by default on linux, osx, win by leondz in https://github.com/leondz/garak/pull/793
* enable nested setting of plugin configs by leondz in https://github.com/leondz/garak/pull/773
* always log nvcf payload by leondz in https://github.com/leondz/garak/pull/778
* update ecoji for windows support by jmartin-tech in https://github.com/leondz/garak/pull/787
* remove extraneous default params for nims that expect conservative pa… by leondz in https://github.com/leondz/garak/pull/749
* require probes to match input modality of the generator by jmartin-tech in https://github.com/leondz/garak/pull/738
* Improve cloze data for `leakreplay` probe by leondz in https://github.com/leondz/garak/pull/708
* add CONTRIBUTING.md, update contributing guide in docs by leondz in https://github.com/leondz/garak/pull/706
* add DEFAULT_PARAMS to base types that are configurables by leondz in https://github.com/leondz/garak/pull/771
* atkgen - add configurability, allow config of which adversarial generator to use by leondz in https://github.com/leondz/garak/pull/777
* enhancement: enable lazy setting of nested dicts by leondz in https://github.com/leondz/garak/pull/775
* convert mutation_generator to use `load_plugin` by jmartin-tech in https://github.com/leondz/garak/pull/795
* Add support for gpt-4o, gpt-4o-mini by erickgalinkin in https://github.com/leondz/garak/pull/798
* bump transformers ver to min reqd for llama 3.1 by leondz in https://github.com/leondz/garak/pull/800
* add mitigation strings, add test for string-based mitigation skipping by leondz in https://github.com/leondz/garak/pull/762
* speedup report generation by using class attribs instead of instantiating by leondz in https://github.com/leondz/garak/pull/707
* cached plugin enum by jmartin-tech in https://github.com/leondz/garak/pull/768

Tuning & fixes
* add some actionable opinions about linting by leondz in https://github.com/leondz/garak/pull/668
* handle target model returning None in atkgen by leondz in https://github.com/leondz/garak/pull/678
* bump & fix black version by leondz in https://github.com/leondz/garak/pull/686
* Bump/black 24.4.2 -- and apply formatting by leondz in https://github.com/leondz/garak/pull/687
* convert module-level constant names to uppercase by leondz in https://github.com/leondz/garak/pull/685
* Guard cli run with invalid or incomplete config by jmartin-tech in https://github.com/leondz/garak/pull/677
* Catch generic exception from transformers in huggingface by jmartin-tech in https://github.com/leondz/garak/pull/690
* stablize openai parallel by jmartin-tech in https://github.com/leondz/garak/pull/689
* rest generator response must be a list by jmartin-tech in https://github.com/leondz/garak/pull/693
* Set generator `_call_model()` and `_generate()` type hints; amend functions by leondz in https://github.com/leondz/garak/pull/694
* typo in racial slur prompt by leondz in https://github.com/leondz/garak/pull/697
* Generator streamlining, docs by leondz in https://github.com/leondz/garak/pull/682
* Split NVCF generator into completion chat by leondz in https://github.com/leondz/garak/pull/696
* Update the quality taxonomy by leondz in https://github.com/leondz/garak/pull/703
* generator & detector structure validation by leondz in https://github.com/leondz/garak/pull/702
* doc & inheritance fixes to DanInTheWildMini by leondz in https://github.com/leondz/garak/pull/712
* Remove support for openai api v0 by leondz in https://github.com/leondz/garak/pull/718
* restore `req_template_json_object` lost in refactor by jmartin-tech in https://github.com/leondz/garak/pull/729
* utilize GarakException in cli by jmartin-tech in https://github.com/leondz/garak/pull/730
* ensure report is printed in `utf-8` for Windows by jmartin-tech in https://github.com/leondz/garak/pull/732
* Linting - 20240612 by leondz in https://github.com/leondz/garak/pull/733
* add garak preprint paper by leondz in https://github.com/leondz/garak/pull/739
* only set generations if still default by jmartin-tech in https://github.com/leondz/garak/pull/741
* return value from `batch_decode` is already a list by jmartin-tech in https://github.com/leondz/garak/pull/743
* Handle openai 400 errors more gracefully by leondz in https://github.com/leondz/garak/pull/744
* Refactor `huggingface` config support by jmartin-tech in https://github.com/leondz/garak/pull/742
* rename class level probe doc_uri by jmartin-tech in https://github.com/leondz/garak/pull/759
* retry openaicompatible requests if invalid content received by leondz in https://github.com/leondz/garak/pull/761
* add colour theme config by leondz in https://github.com/leondz/garak/pull/769
* reduce zalgo-ness by leondz in https://github.com/leondz/garak/pull/770
* move REST generator exception into garak.exceptions by leondz in https://github.com/leondz/garak/pull/779
* defensive coding: allow python generators more places by leondz in https://github.com/leondz/garak/pull/782
* stop encoding.payloads accumulating each load by leondz in https://github.com/leondz/garak/pull/780
* correct sorting for set json in plugin_cache.json by jmartin-tech in https://github.com/leondz/garak/pull/794
* corrections to config format for TAP internal generators by jmartin-tech in https://github.com/leondz/garak/pull/796
* add missing comma, extra test case in probes.test.Test by leondz in https://github.com/leondz/garak/pull/797
* Bugfix/leakreplay output setting by leondz in https://github.com/leondz/garak/pull/790


**Full Changelog**: https://github.com/leondz/garak/compare/v0.9.0.13...v0.9.0.14

What's Changed

garak's 1st birthday! 🎂

Headlines in this birthday release:
* Multimodal support! LLaVa + FigStep - HUGE thanks to DavidLee528
* LiteLLM generator support - thanks to Tien-Cheng!
* DoNotAnswer probe for prompts an LLM should decline - thanks to AhsanAyub!
* LangChain Serve generator support - thanks to GustavFredrikson!
* Support for NIM generators, NVIDIA Inference Microservices
* OpenAI abstraction with parallelisation support
* Windows compatibility enhanced, with optional testing in the workflow
* Hugging Face speedup

plugins
* LiteLLM generator by Tien-Cheng in https://github.com/leondz/garak/pull/572
* Support for using 'langchain serve' endpoints by GustavFredrikson in https://github.com/leondz/garak/pull/588
* Enable parallel OpenAI calls by jmartin-tech in https://github.com/leondz/garak/pull/645
* Multi-modal Jailbreaking Attack on LLaVA by DavidLee528 in https://github.com/leondz/garak/pull/587
* bump openai module version to match paths in latest litellm by leondz in https://github.com/leondz/garak/pull/664
* generator: NIM by leondz in https://github.com/leondz/garak/pull/637
* Probe: Do Not Answer by AhsanAyub in https://github.com/leondz/garak/pull/608

architecture
* change supported pythons to 3.10-3.12 by leondz in https://github.com/leondz/garak/pull/503
* add more detailed TAP docs by leondz in https://github.com/leondz/garak/pull/504
* add multiple-result aggregator by leondz in https://github.com/leondz/garak/pull/505
* add post buff hook by erickgalinkin in https://github.com/leondz/garak/pull/506
* Add fleshed-out docs to all probes by leondz in https://github.com/leondz/garak/pull/507
* add bibtex under citation info in readme by leondz in https://github.com/leondz/garak/pull/511
* define broad test, all probes, 1 gen per by leondz in https://github.com/leondz/garak/pull/514
* Feature/taxonomy payloads by leondz in https://github.com/leondz/garak/pull/519
* include paraphrasing in broad conf by leondz in https://github.com/leondz/garak/pull/521
* choose whether buffing will also include the original prompt by leondz in https://github.com/leondz/garak/pull/523
* add config var for capping max buffed prompts to add per buff by leondz in https://github.com/leondz/garak/pull/526
* document Probe.probe(); skip a buff hook if no buffs by leondz in https://github.com/leondz/garak/pull/527
* add type hints to base.Probe; fix base probe rst by leondz in https://github.com/leondz/garak/pull/528
* Bump datasets package by shubhobm in https://github.com/leondz/garak/pull/536
* Add ConversationalPipeline for huggingface models by erickgalinkin in https://github.com/leondz/garak/pull/539
* add generator for supporting openai module v0.x by leondz in https://github.com/leondz/garak/pull/553
* Update README.md by erickgalinkin in https://github.com/leondz/garak/pull/558
* Minor typo in FAQ by jmartin-tech in https://github.com/leondz/garak/pull/562
* Add additional error message when __doc__ is None type by DavidLee528 in https://github.com/leondz/garak/pull/566
* shared constant & string literal by jmartin-tech in https://github.com/leondz/garak/pull/571
* Spelling corrections for multiple locations by jmartin-tech in https://github.com/leondz/garak/pull/564
* Reduce Huggingface GPU utilization by erickgalinkin in https://github.com/leondz/garak/pull/567
* skip `verbose` flag in secondary parser by jmartin-tech in https://github.com/leondz/garak/pull/576
* Added project twitter link and corrected a grammatic error by codebrain001 in https://github.com/leondz/garak/pull/578
* Convert GGML to expect GGUF format by jmartin-tech in https://github.com/leondz/garak/pull/581
* Update workflows: CLA asst bump, PR & manual testing by leondz in https://github.com/leondz/garak/pull/591
* add test de-duping using skip-duplicate-actions by leondz in https://github.com/leondz/garak/pull/597
* Remove `!` entries from files not intended as executables by jmartin-tech in https://github.com/leondz/garak/pull/612
* Further align shebangs with code that has executable entry points by leondz in https://github.com/leondz/garak/pull/613
* interactive mode intro by leondz in https://github.com/leondz/garak/pull/614
* add tests for `ggml` generator by jmartin-tech in https://github.com/leondz/garak/pull/618
* add var for generator context_len and populate this for some generators by leondz in https://github.com/leondz/garak/pull/616
* allow generators.Base.generate() to take an optional param specifying generation count by leondz in https://github.com/leondz/garak/pull/600
* Enable windows tests as github action by jmartin-tech in https://github.com/leondz/garak/pull/626
* add on-demand macos testing by leondz in https://github.com/leondz/garak/pull/631
* macOS test install from correct path by jmartin-tech in https://github.com/leondz/garak/pull/633
* consolidate test file cleanup by jmartin-tech in https://github.com/leondz/garak/pull/634
* bump discord link by leondz in https://github.com/leondz/garak/pull/648
* meta the arguments a bit for GET vs other request types by jmartin-tech in https://github.com/leondz/garak/pull/640

stability
* fail gracefully if nvcf rejects input; compact zalgo prompts by leondz in https://github.com/leondz/garak/pull/509
* log & skip past NVCF 4xx errors by leondz in https://github.com/leondz/garak/pull/533
* fix empty autodan prompts & poor detector behaviour by leondz in https://github.com/leondz/garak/pull/534
* Fix AutoDAN issues by erickgalinkin in https://github.com/leondz/garak/pull/537
* fix bad nonetype handling in atkgen probe by leondz in https://github.com/leondz/garak/pull/538
* Division by zero error fixed in HTML report generation by CoderMayhem in https://github.com/leondz/garak/pull/545
* cap cohere lib version by leondz in https://github.com/leondz/garak/pull/569
* rm deprecated model from example by leondz in https://github.com/leondz/garak/pull/575
* Attack fixes by erickgalinkin in https://github.com/leondz/garak/pull/555
* More regex as string literal by jmartin-tech in https://github.com/leondz/garak/pull/586
* Bugfix/action dedupe by leondz in https://github.com/leondz/garak/pull/598
* wrap cli exec to gracefully catch keyboard exit signal by jmartin-tech in https://github.com/leondz/garak/pull/603
* Enforce warning output for `garak` classes. by jmartin-tech in https://github.com/leondz/garak/pull/605
* bump hf transformers v to avoid transformers30076 by leondz in https://github.com/leondz/garak/pull/636
* update avidtools to remove typing reference by jmartin-tech in https://github.com/leondz/garak/pull/639
* torch v bump by leondz in https://github.com/leondz/garak/pull/649
* Pause FigStepTiny by leondz in https://github.com/leondz/garak/pull/652
* Bugfix/visual jailbreak pause by leondz in https://github.com/leondz/garak/pull/653
* limit push test to main by jmartin-tech in https://github.com/leondz/garak/pull/661
* Update MANIFEST.in so all resources are installed by JKL98ISR in https://github.com/leondz/garak/pull/660
* handle extant but closed `hitlogfile` file by leondz in https://github.com/leondz/garak/pull/665


New Contributors
* CoderMayhem made their first contribution in https://github.com/leondz/garak/pull/545
* jmartin-tech made their first contribution in https://github.com/leondz/garak/pull/562
* codebrain001 made their first contribution in https://github.com/leondz/garak/pull/578
* Tien-Cheng made their first contribution in https://github.com/leondz/garak/pull/572
* GustavFredrikson made their first contribution in https://github.com/leondz/garak/pull/588
* JKL98ISR made their first contribution in https://github.com/leondz/garak/pull/660
* AhsanAyub made their first contribution in https://github.com/leondz/garak/pull/608

**Full Changelog**: https://github.com/leondz/garak/compare/v0.9.0.12...v0.9.0.13

What's Changed

plugins
* New encoding probes by zmackie in https://github.com/leondz/garak/pull/459
* OpenAI upgrade by erickgalinkin in https://github.com/leondz/garak/pull/477
* Low Resource Languages Buff by erickgalinkin in https://github.com/leondz/garak/pull/478
* Add Rasa generator by rgstephens in https://github.com/leondz/garak/pull/453
* Tree of Attacks by erickgalinkin in https://github.com/leondz/garak/pull/446

functionality improvements
* support multiple buffs by leondz in https://github.com/leondz/garak/pull/497
* wrap exception printing in repr by leondz in https://github.com/leondz/garak/pull/425
* add generators.function docs & examples by leondz in https://github.com/leondz/garak/pull/437
* update doc indices, add test to check them by leondz in https://github.com/leondz/garak/pull/450
* fix & unify REST generator timeout param names; set default request timeout to 20s by leondz in https://github.com/leondz/garak/pull/451
* add test to keep requirements in sync by leondz in https://github.com/leondz/garak/pull/465
* docs for buffs by leondz in https://github.com/leondz/garak/pull/466
* autosearch in the configs/ subdir for configs (no yaml extension should be given) by leondz in https://github.com/leondz/garak/pull/467
* Update function.py by erickgalinkin in https://github.com/leondz/garak/pull/500
* add warning when using a lite/default profile by leondz in https://github.com/leondz/garak/pull/476
* rename default output dir to garak_runs/; by leondz in https://github.com/leondz/garak/pull/488
* update openai model list by leondz in https://github.com/leondz/garak/pull/494
* make test_openai generation tests skip if no OAI API key set by leondz in https://github.com/leondz/garak/pull/491

fixes
* html report now uses correct basedir by leondz in https://github.com/leondz/garak/pull/439
* typos & clarifications in rest generator by leondz in https://github.com/leondz/garak/pull/436
* update manifest by leondz in https://github.com/leondz/garak/pull/454
* Avoid divide by zero error by erickgalinkin in https://github.com/leondz/garak/pull/458
* Fix/test pytest-8.0.0 order by leondz in https://github.com/leondz/garak/pull/472
* Check & enable Python 3.12 support by leondz in https://github.com/leondz/garak/pull/475
* move pathlib uses to _config.transient.basedir by leondz in https://github.com/leondz/garak/pull/499
* catch & handle HF hub exceptions loading dataset for package hallucination by leondz in https://github.com/leondz/garak/pull/470


New Contributors
* zmackie made their first contribution in https://github.com/leondz/garak/pull/459
* rgstephens made their first contribution in https://github.com/leondz/garak/pull/453

**Full Changelog**: https://github.com/leondz/garak/compare/v0.9.0.11...v0.9.0.12

What's Changed
* wrap exception printing in repr by leondz in https://github.com/leondz/garak/pull/425
* typos & clarifications in rest generator by leondz in https://github.com/leondz/garak/pull/430, https://github.com/leondz/garak/pull/436
* add generators.function docs & examples by leondz in https://github.com/leondz/garak/pull/437
* html report now uses correct basedir by leondz in https://github.com/leondz/garak/pull/439

**Full Changelog**: https://github.com/leondz/garak/compare/v0.9.0.11...v0.9.0.11.post1

What's Changed
* Probe for repetition-based nudging into replay/spurious generation by leondz in https://github.com/leondz/garak/pull/404
* Probe for invisible text prompt injections by leondz in https://github.com/leondz/garak/pull/397
* Probe for the 'DAN in the wild' paper's library of jailbreak prompts by leondz in https://github.com/leondz/garak/pull/405
* Probe for NYT & The Guardian content in training data by leondz in https://github.com/leondz/garak/pull/402
* Add NVIDIA cloud functions generator by leondz in https://github.com/leondz/garak/pull/398
* Add toxicity generation deep test config by leondz in https://github.com/leondz/garak/pull/413
* Generator enhancements and minor improvements by shubhobm in https://github.com/leondz/garak/pull/391
* Update HF inf api generator to match their current expectations by leondz in https://github.com/leondz/garak/pull/400
* Invoke garak on the command line, with `garak` by leondz in https://github.com/leondz/garak/pull/410
* Mitigate continuation probe oversensitivity by leondz in https://github.com/leondz/garak/pull/394
* Handle nvcf container timeouts by leondz in https://github.com/leondz/garak/pull/399
* Fixing Exception Cause By Type Error When Scanning LLMs Via Replicate by DavidLee528 in https://github.com/leondz/garak/pull/401
* Make sure triggers attempt.note is saved in hitlog by leondz in https://github.com/leondz/garak/pull/403
* Repeat replay now optionally overrides generator max len by leondz in https://github.com/leondz/garak/pull/408
* Replay.Repeat now preserves attempt when restoring generator max_tokens by leondz in https://github.com/leondz/garak/pull/409
* Gracefully handle NVCF request timeouts & other failures by leondz in https://github.com/leondz/garak/pull/411
* Fix deprecated encoding by leondz in https://github.com/leondz/garak/pull/412
* Better coverage in mitigation bypass detector

**Full Changelog**: https://github.com/leondz/garak/compare/v0.9.0.10...v0.9.0.11