Importnat Changes
write abstract for update talk about unitxt covering the following topics:
- **Criteria based LLM as Judges** - Improved class of llm as judges with customizable judging criteria [(read more)](https://www.unitxt.ai/en/latest/docs/llm_as_judge.html)
- **Unitxt assistant** - A textual assistant expert in unitxt to help developers [(read more)](https://www.unitxt.ai/en/latest/docs/llm_as_judge.html)
- **New benchmarks: Tables, Vision** - Benchmarks for table understanding and image understanding compiled by the community and collaborators [(read more)](https://www.unitxt.ai/en/latest/docs/llm_as_judge.html)
- **Support for all major inference providers** - Inference for evaluation or llm as judges can be channel to any inference provider such as: azure, aws and watsonx [(read more)](https://www.unitxt.ai/en/latest/docs/llm_as_judge.html)
Detailed Changes
* Fix typing notation for python 3.8 by elronbandel in https://github.com/IBM/unitxt/pull/1453
* Instance_metric and apply_metric keep only one instance at a time in mem, at the expense of repeated passes over input stream (2 times for instance_metric, metrics for apply_metric) by dafnapension in https://github.com/IBM/unitxt/pull/1448
* simplify class parameter listing on web page by dafnapension in https://github.com/IBM/unitxt/pull/1454
* Bring code coverage tests back to life by elronbandel in https://github.com/IBM/unitxt/pull/1455
* Fix coverage tests by elronbandel in https://github.com/IBM/unitxt/pull/1456
* make demos_pool a local var rather than a separate stream by dafnapension in https://github.com/IBM/unitxt/pull/1436
* Adding upper case and last non empty line processor by antonpibm in https://github.com/IBM/unitxt/pull/1458
* performance by bluebench by dafnapension in https://github.com/IBM/unitxt/pull/1457
* Add UNITXT_MOCK_INFERENCE_MODE environment variable to performance workflow by elronbandel in https://github.com/IBM/unitxt/pull/1461
* remove redundant lines from performance.yml by dafnapension in https://github.com/IBM/unitxt/pull/1462
* Benjams/add bioasq miniwiki datasets by BenjSz in https://github.com/IBM/unitxt/pull/1460
* Add SocialIQA dataset by elronbandel in https://github.com/IBM/unitxt/pull/1468
* Add parallelization to RITS inference by arielge in https://github.com/IBM/unitxt/pull/1441
* Fix the type handeling for tasks to support string types by elronbandel in https://github.com/IBM/unitxt/pull/1470
* Update version to 1.16.1 by elronbandel in https://github.com/IBM/unitxt/pull/1472
* extend choices arrangement functionality with ReorderableMultipleChoi… by eliyahabba in https://github.com/IBM/unitxt/pull/1464
* Add GPQA dataset by elronbandel in https://github.com/IBM/unitxt/pull/1474
* Add simple QA dataset by elronbandel in https://github.com/IBM/unitxt/pull/1475
* Add LongBench V2 dataset by elronbandel in https://github.com/IBM/unitxt/pull/1476
* Adding typed recipe test by antonpibm in https://github.com/IBM/unitxt/pull/1473
* Add place_correct_choice_position to set the correct choice index and… by eliyahabba in https://github.com/IBM/unitxt/pull/1481
* Add MapReduceMetric a new base class to integrate all metrics into by elronbandel in https://github.com/IBM/unitxt/pull/1459
* Add multi document support and FRAMES benchmark by elronbandel in https://github.com/IBM/unitxt/pull/1477
* Update version to 1.16.2 by elronbandel in https://github.com/IBM/unitxt/pull/1483
* Add Azure support and expand OpenAI model options in inference engine by elronbandel in https://github.com/IBM/unitxt/pull/1485
* Benjams/fix bioasq card by BenjSz in https://github.com/IBM/unitxt/pull/1486
* add separator to csv loader by BenjSz in https://github.com/IBM/unitxt/pull/1488
* Fix bug in metrics loading in tasks by elronbandel in https://github.com/IBM/unitxt/pull/1487
* Update version to 1.16.3 by elronbandel in https://github.com/IBM/unitxt/pull/1489
* Fix bootstrap condition to handle cases with insufficient instances by elronbandel in https://github.com/IBM/unitxt/pull/1490
* Update version to 1.16.4 by elronbandel in https://github.com/IBM/unitxt/pull/1491
* Simplify artifact link [Non Backward Compatible!] by elronbandel in https://github.com/IBM/unitxt/pull/1494
* Added NER example by yoavkatz in https://github.com/IBM/unitxt/pull/1492
* Add example for evaluating tables as images using Unitxt APIs by elronbandel in https://github.com/IBM/unitxt/pull/1495
* Mm updates by alfassy in https://github.com/IBM/unitxt/pull/1465
* Fix wrong saving of artifact initial dict by elronbandel in https://github.com/IBM/unitxt/pull/1499
* Accelerate and improve RAG Metrics by elronbandel in https://github.com/IBM/unitxt/pull/1497
* Make clinc preparation faster by elronbandel in https://github.com/IBM/unitxt/pull/1501
* Fix templates lists in vision cards by elronbandel in https://github.com/IBM/unitxt/pull/1500
* Add vision benchmark example by elronbandel in https://github.com/IBM/unitxt/pull/1502
* Update vis bench by elronbandel in https://github.com/IBM/unitxt/pull/1505
* Add Balance operator by elronbandel in https://github.com/IBM/unitxt/pull/1507
* Fix for demos_pool with images. by elronbandel in https://github.com/IBM/unitxt/pull/1509
* Remove new balance operator and use existing implementation by elronbandel in https://github.com/IBM/unitxt/pull/1510
* Fixes and adjustment in rag metrics and related inference engines by lilacheden in https://github.com/IBM/unitxt/pull/1466
* Tables bench by ShirApp in https://github.com/IBM/unitxt/pull/1506
* Keep metadata over main unitxt stages by eladven in https://github.com/IBM/unitxt/pull/1512
* Fix: Improved handling of `place_correct_choice_position` for flexibl… by eliyahabba in https://github.com/IBM/unitxt/pull/1511
* Fixes in LLMJudge by lilacheden in https://github.com/IBM/unitxt/pull/1498
* Verify metrics prediction_type without loading metric by elronbandel in https://github.com/IBM/unitxt/pull/1519
* Add Unitxt Assistant beta by elronbandel in https://github.com/IBM/unitxt/pull/1513
* Ensure fusion do not call streams before use by elronbandel in https://github.com/IBM/unitxt/pull/1518
* Minor llm as judge fix/changes by martinscooper in https://github.com/IBM/unitxt/pull/1467
* Fix: Selected option for supporting negative indexes in place_correct… by eliyahabba in https://github.com/IBM/unitxt/pull/1522
* Refactor rag metrics and judges by lilacheden in https://github.com/IBM/unitxt/pull/1515
* Add Llama 3.1 on Vertex AI to CrossProviderInferenceEngine by yifanmai in https://github.com/IBM/unitxt/pull/1525
* fix external_rag example by lilacheden in https://github.com/IBM/unitxt/pull/1526
* Add search to assistant for much faster response by elronbandel in https://github.com/IBM/unitxt/pull/1524
* fixed division by 0 in compare performance results by dafnapension in https://github.com/IBM/unitxt/pull/1523
* Add two criteria based direct llm judges by lilacheden in https://github.com/IBM/unitxt/pull/1527
* Update version to 1.17.0 by elronbandel in https://github.com/IBM/unitxt/pull/1535
New Contributors
* eliyahabba made their first contribution in https://github.com/IBM/unitxt/pull/1464
**Full Changelog**: https://github.com/IBM/unitxt/compare/1.16.0...1.17.0