Guidance

What's Changed
* Replace Python-based EarleyParser with lexeme-based rust implementation by hudson-ai & mmoskal in https://github.com/guidance-ai/guidance/pull/951

This pre-release version of guidance uses a new Rust-based Earley lexer + parser that offers significant speedups and bugfixes to guidance grammar parsing. See PR https://github.com/guidance-ai/guidance/pull/951 for full details of the optimizations. Expect some changes from current Guidance (v0.1.16) behavior during pre-release.

**Full Changelog**: https://github.com/guidance-ai/guidance/compare/0.1.16...0.2.0rc1

What's Changed
* Allow generating arbitrary (schemaless) JSON by hudson-ai in https://github.com/guidance-ai/guidance/pull/892
* [Bug] Fix Llama3ChatTemplate by jjkjkj in https://github.com/guidance-ai/guidance/pull/884
* Use python's `re` module instead of `pyformlang` to build regex ASTs by hudson-ai in https://github.com/guidance-ai/guidance/pull/854
* [Feature] Add Support for Phi-3-small by ambisinister in https://github.com/guidance-ai/guidance/pull/935
* [Feature] Constant elements in JSON schemas by wjn0 in https://github.com/guidance-ai/guidance/pull/886
* [Bug] Fix for LlamaCpp tokeniser prepending spaces by riedgar-ms in https://github.com/guidance-ai/guidance/pull/903
* New Benchmarking system by nopdive in https://github.com/guidance-ai/guidance/pull/843
* Mistral chat template integrated by Harsha-Nori in https://github.com/guidance-ai/guidance/pull/919
* [Bug] Passing through kwargs to `TransformersTokenizer` by riedgar-ms in https://github.com/guidance-ai/guidance/pull/922
* [Feature] Add extra support to JSON string schema by riedgar-ms in https://github.com/guidance-ai/guidance/pull/927
* Update protobuf definitions for remote list_append fix by nking-1 in https://github.com/guidance-ai/guidance/pull/937
* Downgrade protobuf required version using protoc 25.3 by nking-1 in https://github.com/guidance-ai/guidance/pull/946
* [Bug] Fix regex for urls in image function by kklemon in https://github.com/guidance-ai/guidance/pull/928
* Dedent fixes for Python 3.12 by Harsha-Nori in https://github.com/guidance-ai/guidance/pull/959
* Refector `TransformersTokenizer` and change fallback behavior for `byte_tokens` by hudson-ai in https://github.com/guidance-ai/guidance/pull/973
* Fix Transformer models with bos_token null by Mihaiii in https://github.com/guidance-ai/guidance/pull/990

New Contributors
* nopdive made their first contribution in https://github.com/guidance-ai/guidance/pull/843
* jjkjkj made their first contribution in https://github.com/guidance-ai/guidance/pull/884
* wjn0 made their first contribution in https://github.com/guidance-ai/guidance/pull/886
* kklemon made their first contribution in https://github.com/guidance-ai/guidance/pull/928
* ambisinister made their first contribution in https://github.com/guidance-ai/guidance/pull/935
* Mihaiii made their first contribution in https://github.com/guidance-ai/guidance/pull/990

**Full Changelog**: https://github.com/guidance-ai/guidance/compare/0.1.15...0.1.16

What's Changed
* Support for Anthropic Claude 3 and Phi-3 models by Harsha-Nori
* [Feature] Metrics for Grammarless models by riedgar-ms in https://github.com/guidance-ai/guidance/pull/817
* Remove Instruct/Chat versions of models & introduce a new ChatTemplate API by Harsha-Nori in https://github.com/guidance-ai/guidance/pull/820
* Fix set_attribute by slundberg in https://github.com/guidance-ai/guidance/pull/835
* Fix for unstable tokenization schemes by Harsha-Nori in https://github.com/guidance-ai/guidance/pull/830
* Support null captures (Fix 826) by slundberg in https://github.com/guidance-ai/guidance/pull/834
* Allow capture of raw functions by slundberg in https://github.com/guidance-ai/guidance/pull/837
* [Build] Significant notebook testing updates by riedgar-ms in https://github.com/guidance-ai/guidance/pull/825
* [Docs] ReadTheDocs Now Works! by riedgar-ms in https://github.com/guidance-ai/guidance/pull/829
* add initial support for azure guidance server API by paulbkoch in https://github.com/guidance-ai/guidance/pull/827
* [Build] Adjust llama-cpp-python versions by riedgar-ms in https://github.com/guidance-ai/guidance/pull/832
* [Test] Fix for LlamaCpp tests by paulbkoch in https://github.com/guidance-ai/guidance/pull/839
* Remove default max token limit by slundberg in https://github.com/guidance-ai/guidance/pull/836
* [Build] General Maintenance by riedgar-ms in https://github.com/guidance-ai/guidance/pull/842
* Updated readme by nking-1 in https://github.com/guidance-ai/guidance/pull/810


**Full Changelog**: https://github.com/guidance-ai/guidance/compare/0.1.14...0.1.15

This release brings a series of new features, bug fixes, and changes, including:

- Improved compatibility for loading many `transformers` models, including newly tested support for `Phi-3`
- New capabilities in the `guidance.json` function for generating json, including early support for pydantic schemas.
- Support for AzureAI models in new `guidance.models.AzureAIStudio*` classes
- Significant dependency reductions on the default guidance installation
- Small grammar optimizations
- Many improvements to the infrastructure, testing, and release pipelines :)

What's Changed
* replace catches of the ImportError exception with the more specific ModuleNotFoundError exception by paulbkoch in https://github.com/guidance-ai/guidance/pull/733
* fix for issue 660 where the call to llama_batch_free is failing during process shutdown by paulbkoch in https://github.com/guidance-ai/guidance/pull/734
* Fix 713 for mistral models in transformers by slundberg in https://github.com/guidance-ai/guidance/pull/740
* Update JSON API and example notebook by riedgar-ms in https://github.com/guidance-ai/guidance/pull/699
* remove redundant options from select() by mmoskal in https://github.com/guidance-ai/guidance/pull/712
* Add support for `prefixItems`, `minItems`, and `maxItems` for `JSON` array generation by hudson-ai in https://github.com/guidance-ai/guidance/pull/732
* [Docs] Expand documentation by riedgar-ms in https://github.com/guidance-ai/guidance/pull/750
* Generation with `pydantic` schemas by hudson-ai in https://github.com/guidance-ai/guidance/pull/724
* First stab at minimal mypy implementation by ryanpeach in https://github.com/guidance-ai/guidance/pull/738
* remove openai hard dependency by slundberg in https://github.com/guidance-ai/guidance/pull/763
* from guidance import guidance by ryanpeach in https://github.com/guidance-ai/guidance/pull/759
* Run black on codebase by Harsha-Nori in https://github.com/guidance-ai/guidance/pull/765
* add rust bindings by paulbkoch in https://github.com/guidance-ai/guidance/pull/736
* fix Transformers tokenizer issues when models have extended vocabulary and/or special tokens by Harsha-Nori in https://github.com/guidance-ai/guidance/pull/766
* [Install] Rework optional imports by riedgar-ms in https://github.com/guidance-ai/guidance/pull/767
* Update _llama_cpp.py based on new contract of llama_get_logits by czoccolo in https://github.com/guidance-ai/guidance/pull/769
* [Bug] Missing support for AzureAI by riedgar-ms in https://github.com/guidance-ai/guidance/pull/770
* [Doc] Add developer notes by riedgar-ms in https://github.com/guidance-ai/guidance/pull/774
* [Feature] Improved JSON Schema support by riedgar-ms in https://github.com/guidance-ai/guidance/pull/787
* [Bug] Fix Regex for identifying OpenAI models by alexander-brady in https://github.com/guidance-ai/guidance/pull/778
* Fix models like phi-3 which have a mismatched tokenizer definition and model output tensor size by Harsha-Nori in https://github.com/guidance-ai/guidance/pull/792
* [Feature] Add `temperature` to JSON generator by riedgar-ms in https://github.com/guidance-ai/guidance/pull/784
* [Feature] Support for Azure AI Studio by riedgar-ms in https://github.com/guidance-ai/guidance/pull/779
* Add guard for running build from setup.py, redirecting users to pip by Harsha-Nori in https://github.com/guidance-ai/guidance/pull/805
* *Many* testing and CI improvements by riedgar-ms

New Contributors
* mmoskal made their first contribution in https://github.com/guidance-ai/guidance/pull/712
* ryanpeach made their first contribution in https://github.com/guidance-ai/guidance/pull/738
* czoccolo made their first contribution in https://github.com/guidance-ai/guidance/pull/769
* alexander-brady made their first contribution in https://github.com/guidance-ai/guidance/pull/778

**Full Changelog**: https://github.com/guidance-ai/guidance/compare/0.1.13...0.1.14

What's Changed
* Fixed PyPi publishing
* Workflow for GPU Runner by riedgar-ms in https://github.com/guidance-ai/guidance/pull/694
* Add Llama to model test matrix by riedgar-ms in https://github.com/guidance-ai/guidance/pull/703


**Full Changelog**: https://github.com/guidance-ai/guidance/compare/0.1.12...0.1.13

Lots of new features and bug fixes :)

What's Changed
* Adding Caching to OpenAI chat by adamgordonbell in https://github.com/guidance-ai/guidance/pull/603
* Minor Server Tweaks by riedgar-ms in https://github.com/guidance-ai/guidance/pull/612
* Dependency trimming by riedgar-ms in https://github.com/guidance-ai/guidance/pull/613
* Add Python 3.12 by riedgar-ms in https://github.com/guidance-ai/guidance/pull/623
* Grammar Builder for JSON Schema by riedgar-ms in https://github.com/guidance-ai/guidance/pull/619
* remove unused code by paulbkoch in https://github.com/guidance-ai/guidance/pull/629
* fix: missing tokenizer choice in AzureOpenAI subclasses by imarquart in https://github.com/guidance-ai/guidance/pull/641
* Pop first argument from guidance wrapper by hudson-ai in https://github.com/guidance-ai/guidance/pull/634
* Fix behavior of `_json_schema_to_grammar` for schemas with many properties by hudson-ai in https://github.com/guidance-ai/guidance/pull/645
* Exclude tests packages from distribution by shawnz in https://github.com/guidance-ai/guidance/pull/646
* Support multiple additions to ModelStream by shawnz in https://github.com/guidance-ai/guidance/pull/639
* Enable AzureOpenAI model testing by riedgar-ms in https://github.com/guidance-ai/guidance/pull/514
* CI Build Schedule by riedgar-ms in https://github.com/guidance-ai/guidance/pull/662
* Add extensible tests to transformers by riedgar-ms in https://github.com/guidance-ai/guidance/pull/663
* Update _llama_cpp.py to use new bindings by Nking92 in https://github.com/guidance-ai/guidance/pull/665
* Propagate errors in ModelStream by shawnz in https://github.com/guidance-ai/guidance/pull/640
* Fix test_transformers.py by riedgar-ms in https://github.com/guidance-ai/guidance/pull/673
* Together.ai Support by minosvasilias in https://github.com/guidance-ai/guidance/pull/654
* Move get_logits and EngineCallResponse out of the Engine.__call__ function so that the remaining parts can be lowered to Rust or C++ in the future and for simplification of LLM servers that operate in batches by paulbkoch in https://github.com/guidance-ai/guidance/pull/647
* Make Nulls Terminal by hudson-ai in https://github.com/guidance-ai/guidance/pull/679
* [Test] Expand to other OSes by riedgar-ms in https://github.com/guidance-ai/guidance/pull/680
* JSON Schema Rewrite by riedgar-ms in https://github.com/guidance-ai/guidance/pull/651
* [Test] MacOS Issue with Server tests by riedgar-ms in https://github.com/guidance-ai/guidance/pull/684
* Support mutual recursion in `GrammarFunctions` by hudson-ai in https://github.com/guidance-ai/guidance/pull/683
* More JSON schema features by hudson-ai in https://github.com/guidance-ai/guidance/pull/685
* Improve negative test cases for JSON generation by riedgar-ms in https://github.com/guidance-ai/guidance/pull/693
* Separate out server tests by riedgar-ms in https://github.com/guidance-ai/guidance/pull/688
* Add fixture for an LLM by riedgar-ms in https://github.com/guidance-ai/guidance/pull/674
* Removed side effect on the stop argument from _gen by VincentToups in https://github.com/guidance-ai/guidance/pull/677
* Fix for test_gen by riedgar-ms in https://github.com/guidance-ai/guidance/pull/695
* Fix 681 and 682 by slundberg in https://github.com/guidance-ai/guidance/pull/704

New Contributors
* adamgordonbell made their first contribution in https://github.com/guidance-ai/guidance/pull/603
* imarquart made their first contribution in https://github.com/guidance-ai/guidance/pull/641
* hudson-ai made their first contribution in https://github.com/guidance-ai/guidance/pull/634
* shawnz made their first contribution in https://github.com/guidance-ai/guidance/pull/646
* Nking92 made their first contribution in https://github.com/guidance-ai/guidance/pull/665
* minosvasilias made their first contribution in https://github.com/guidance-ai/guidance/pull/654
* VincentToups made their first contribution in https://github.com/guidance-ai/guidance/pull/677

**Full Changelog**: https://github.com/guidance-ai/guidance/compare/0.1.11...0.1.12