Litellm

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.34.19...v1.34.20

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 78 | 82.77907037711888 | 1.5764725188758335 | 0.0 | 472 | 0 | 72.8318520000073 | 463.46175899998343 |
| /health/liveliness | Passed ✅ | 62 | 64.35756584088988 | 15.617765886151266 | 0.0 | 4676 | 0 | 59.69151200002898 | 1073.4929960000272 |
| /health/readiness | Passed ✅ | 62 | 64.7144725812361 | 15.13012820022781 | 0.0033399841501606646 | 4530 | 1 | 59.79837799998222 | 1565.471099999968 |
| Aggregated | Passed ✅ | 62 | 65.42304813835499 | 32.32436660525491 | 0.0033399841501606646 | 9678 | 1 | 59.69151200002898 | 1565.471099999968 |

What's Changed
* docs: Update references to Ollama repository url by DaxServer in https://github.com/BerriAI/litellm/pull/2778
* fix(docs): Correct Docker pull command in deploy.md by DaxServer in https://github.com/BerriAI/litellm/pull/2779
* (fix) improve async perf by 100ms by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2774
* support cohere_chat in get_api_key by phact in https://github.com/BerriAI/litellm/pull/2782
* fix(router.py): fix check for context window fallbacks by krrishdholakia in https://github.com/BerriAI/litellm/pull/2783
* [Feat] Proxy - high traffic redis caching - when using `url` by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2785
* fix(proxy_server.py): don't require scope for team-based jwt access by krrishdholakia in https://github.com/BerriAI/litellm/pull/2787
* [Feat] Allow using model = * on proxy config.yaml by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2788

New Contributors
* DaxServer made their first contribution in https://github.com/BerriAI/litellm/pull/2778
* phact made their first contribution in https://github.com/BerriAI/litellm/pull/2782

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.34.18...v1.34.19

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 82 | 89.06715224896365 | 1.6099163983391747 | 0.0 | 482 | 0 | 75.70292299999437 | 1423.435052000002 |
| /health/liveliness | Passed ✅ | 66 | 68.38107856032434 | 15.254124878039441 | 0.0033400755152265035 | 4567 | 1 | 63.28286600000865 | 1515.330511000002 |
| /health/readiness | Passed ✅ | 66 | 69.006023757576 | 15.54137137234892 | 0.0 | 4653 | 0 | 63.62064799998279 | 1312.5853199999824 |
| Aggregated | Passed ✅ | 66 | 69.70849120933853 | 32.405412648727534 | 0.0033400755152265035 | 9702 | 1 | 63.28286600000865 | 1515.330511000002 |

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.34.17...v1.34.18










Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 95 | 98.99946042212315 | 1.4798141591944631 | 0.0 | 443 | 0 | 87.83134200001541 | 516.7399519999947 |
| /health/liveliness | Passed ✅ | 78 | 80.24621814689517 | 15.599846779770074 | 0.0 | 4670 | 0 | 74.0505100000064 | 1447.6485150000258 |
| /health/readiness | Passed ✅ | 78 | 81.60481963803868 | 15.052014901422687 | 0.006680876565212023 | 4506 | 2 | 74.04748900000868 | 1498.5097569999652 |
| Aggregated | Passed ✅ | 78 | 81.74632674935023 | 32.131675840387224 | 0.006680876565212023 | 9619 | 2 | 74.04748900000868 | 1498.5097569999652 |

What's Changed
* fix(factory.py): parse list in xml tool calling response (anthropic) by krrishdholakia in https://github.com/BerriAI/litellm/pull/2748
* [UI] Edit Teams by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2771
* [UI] view team name in usage by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2772
* (ui) dont let failed prediction block spend view by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2770
* fix(tpm_rpm_limiter.py): enable redis caching for tpm/rpm checks on keys/user/teams by krrishdholakia in https://github.com/BerriAI/litellm/pull/2775


**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.34.16...v1.34.17

What's Changed
* fix: still verify kid on single pub key fetched by jacobnguyenn in https://github.com/BerriAI/litellm/pull/2762
* fix(auth_checks.py): make global spend checks more accurate by krrishdholakia in https://github.com/BerriAI/litellm/pull/2757
* [Feat] Update integration with Lunary by vincelwt in https://github.com/BerriAI/litellm/pull/2244

New Contributors
* jacobnguyenn made their first contribution in https://github.com/BerriAI/litellm/pull/2762

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.34.14...v1.34.16

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Failed ❌ | 1800.0 | 1804.1266242200957 | 1.3962453502626808 | 1.3962453502626808 | 418 | 418 | 379.11375700002736 | 3296.3548129999936 |
| /health/liveliness | Passed ✅ | 78 | 79.54824791932579 | 14.864334470499832 | 0.0 | 4450 | 0 | 73.51694799999109 | 1067.832205000002 |
| /health/readiness | Passed ✅ | 78 | 80.41093312383722 | 15.07811366288455 | 0.0 | 4514 | 0 | 73.90703699996948 | 1273.2291260000181 |
| Aggregated | Passed ✅ | 78 | 156.79914562843686 | 31.338693483647063 | 1.3962453502626808 | 9382 | 418 | 73.51694799999109 | 3296.3548129999936 |

Excited to launch major improvement to LiteLLM UI - Create Teams, Keys and Track LLM Spend

📈 New `/metrics` endpoint for viewing metrics on Prometheus: https://docs.litellm.ai/docs/proxy/grafana_metrics

Admin UI Upgrades:

⚡️ Test Keys on UI (make an LLM Request)

🚀 Delete Teams

⭐️ Add `all_models` to Teams, Keys


What's Changed
* [Fix + Docs] grafana / prometheus metrics by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2752
* [UI] /team/delete by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2759
* build(deps): bump express from 4.18.2 to 4.19.2 in /docs/my-website by dependabot in https://github.com/BerriAI/litellm/pull/2689
* build(deps): bump webpack-dev-middleware from 5.3.3 to 5.3.4 in /docs/my-website by dependabot in https://github.com/BerriAI/litellm/pull/2628
* [UI] team, key add `all_models` by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2760


**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.34.13...v1.34.14

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Failed ❌ | 1800.0 | 1760.7637035391895 | 1.4060758597299565 | 1.4060758597299565 | 421 | 421 | 342.6113250000071 | 3130.634011999973 |
| /health/liveliness | Passed ✅ | 37 | 39.48335569165602 | 14.969197157505143 | 0.0 | 4482 | 0 | 35.50325700001622 | 1117.3830359999783 |
| /health/readiness | Passed ✅ | 37 | 40.18390747290733 | 15.162908321078392 | 0.0033398476478146235 | 4540 | 1 | 35.52504999998973 | 1222.5566220000132 |
| Aggregated | Passed ✅ | 37 | 116.56050612379507 | 31.538181338313493 | 1.409415707377771 | 9443 | 422 | 35.50325700001622 | 3130.634011999973 |