Litellm

What's Changed
* Add command-r-plus by themrzmaster in https://github.com/BerriAI/litellm/pull/2847
* Fix: Remove non-existent gemini-1.5-pro-vision model. by Manouchehri in https://github.com/BerriAI/litellm/pull/2841
* [WIP] feat(vertex_ai_anthropic.py): Add support for claude 3 on vertex ai by krrishdholakia in https://github.com/BerriAI/litellm/pull/2665
* feat: add gemini-experimental to vertex ai by cheahjs in https://github.com/BerriAI/litellm/pull/2859

New Contributors
* themrzmaster made their first contribution in https://github.com/BerriAI/litellm/pull/2847
* cheahjs made their first contribution in https://github.com/BerriAI/litellm/pull/2859

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.34.27...v1.34.28.dev12

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 81 | 85.18298957883415 | 1.546449649654907 | 0.0 | 463 | 0 | 75.98635799999442 | 262.51808699998946 |
| /health/liveliness | Passed ✅ | 66 | 68.25648587302278 | 15.203971501574808 | 0.0 | 4552 | 0 | 63.103840999986005 | 1338.210661000005 |
| /health/readiness | Passed ✅ | 66 | 68.6630459758205 | 15.471176624625333 | 0.0 | 4632 | 0 | 63.46115799999552 | 1323.8507300000038 |
| Aggregated | Passed ✅ | 66 | 69.26406933025822 | 32.221597775855045 | 0.0 | 9647 | 0 | 63.103840999986005 | 1338.210661000005 |

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.34.26...v1.34.27

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 93 | 109.87938594267533 | 1.5730803323619171 | 0.003339873317116597 | 471 | 1 | 86.13480499997195 | 1296.3781700000254 |
| /health/liveliness | Passed ✅ | 78 | 80.21681031098781 | 15.046129293610269 | 0.0 | 4505 | 0 | 73.91940100001193 | 1017.9840530000206 |
| /health/readiness | Passed ✅ | 78 | 82.1807569235042 | 15.630607124105673 | 0.010019619951349791 | 4680 | 3 | 68.7019019999866 | 3399.4434679999586 |
| Aggregated | Passed ✅ | 78 | 82.61556168517045 | 32.24981675007786 | 0.013359493268466388 | 9656 | 4 | 68.7019019999866 | 3399.4434679999586 |

Track Cost for Custom Tags
Start here: Live now: https://docs.litellm.ai/docs/proxy/enterprise#tracking-spend-for-custom-tags


👉 /cache/delete live now - delete specific cache keys https://docs.litellm.ai/docs/proxy/caching#deleting-cache-keys---cachedelete

⭐️ Cache Hit Responses include cache-key used in response headers

What's Changed
* fix(proxy_server.py): persist models added via `/model/new` to db by krrishdholakia in https://github.com/BerriAI/litellm/pull/2827
* Fix link to providers in README by rjmacarthy in https://github.com/BerriAI/litellm/pull/2836
* [FEAT] Proxy - Delete Cache Keys + return cache key in responses by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2840
* docs - `delete` cache keys by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2846
* [UI] Show Usage Per Tag by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2850
* feat(ui): add models via ui by krrishdholakia in https://github.com/BerriAI/litellm/pull/2851


New Contributors
* rjmacarthy made their first contribution in https://github.com/BerriAI/litellm/pull/2836

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.34.25...v1.34.26

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 81 | 90.22493932662181 | 1.4930019313847342 | 0.0 | 447 | 0 | 76.62882199997512 | 1052.6339859999894 |
| /health/liveliness | Passed ✅ | 66 | 69.8399791989995 | 15.35754559397541 | 0.0 | 4598 | 0 | 63.2494559999941 | 1154.2658040000333 |
| /health/readiness | Passed ✅ | 66 | 68.53321177929936 | 15.36088564303891 | 0.0 | 4599 | 0 | 63.40399799995566 | 973.5464270000307 |
| Aggregated | Passed ✅ | 66 | 70.16165628463285 | 32.211433168399054 | 0.0 | 9644 | 0 | 63.2494559999941 | 1154.2658040000333 |

What's Changed
* [FEAT] UI - simplify creating Keys (clean up form and require a model to be selected on UI) by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2821
* FIXED - ui edit keys bug (Edit did not work for a pre-existing key) by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2822
* UI - Fix deleting a key - (this was flaky and would randomly fail) by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2823
* [UI] Fix Team Edit Bug by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2825
* (ui) fix bug when editing keys - selecting models by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2828


**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.34.23-stable...v1.34.25















Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 81 | 91.6600482698744 | 1.5965906599047226 | 0.0 | 478 | 0 | 76.00831600001356 | 1157.996044000015 |
| /health/liveliness | Passed ✅ | 65 | 67.88568940271182 | 15.518326790622051 | 0.01002044347220537 | 4646 | 3 | 63.564229999997224 | 1351.7788230000178 |
| /health/readiness | Passed ✅ | 65 | 68.612596960124 | 15.077427277845015 | 0.0 | 4514 | 0 | 63.44934800000601 | 1280.5795630000034 |
| Aggregated | Passed ✅ | 65 | 69.40523746793949 | 32.192344728371786 | 0.01002044347220537 | 9638 | 3 | 63.44934800000601 | 1351.7788230000178 |

v1.34.23-stable
**Full Changelog**: https://github.com/BerriAI/litellm/compare/v.1.32.34-stable...v1.34.23-stable

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 83 | 98.39840399774896 | 1.4831160494372213 | 0.0 | 444 | 0 | 76.62480600004073 | 1318.882416000065 |
| /health/liveliness | Passed ✅ | 67 | 68.70456653985288 | 15.505911489836896 | 0.0 | 4642 | 0 | 63.61812900001951 | 865.8068830000047 |
| /health/readiness | Passed ✅ | 67 | 68.8459329603138 | 15.318851807925896 | 0.0 | 4586 | 0 | 63.58104399998865 | 1370.0413990000015 |
| Aggregated | Passed ✅ | 67 | 70.13471234584335 | 32.307879347200014 | 0.0 | 9672 | 0 | 63.58104399998865 | 1370.0413990000015 |

What's Changed
* Delete node_modules directory by w3bdesign in https://github.com/BerriAI/litellm/pull/2791
* [UI] Edit Keys by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2795
* [UI] Quality Improvements - show correct team based usage, add litellm favicon by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2797
* fix(openai.py): return logprobs for text completion calls by krrishdholakia in https://github.com/BerriAI/litellm/pull/2799
* [UI] use all_models alias by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2801
* UI - use all-team-models as an alias by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2807
* (fix) Include gemini models in `model_list` by foragerr in https://github.com/BerriAI/litellm/pull/2806
* [feat] use `all-proxy-models` and `all-team-models` with Admin UI by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2808
* [UI] QA Fix Edit Key flow - return `token_id` in /key/generate respose by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2809

New Contributors
* w3bdesign made their first contribution in https://github.com/BerriAI/litellm/pull/2791
* foragerr made their first contribution in https://github.com/BerriAI/litellm/pull/2806

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.34.21...v1.34.22







Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 83 | 94.11182962381028 | 1.4027733713580515 | 0.0 | 420 | 0 | 76.28952499999286 | 1063.0291819999798 |
| /health/liveliness | Passed ✅ | 66 | 69.01898528408572 | 15.236790762227216 | 0.0 | 4562 | 0 | 63.48201800000197 | 1289.3061539999735 |
| /health/readiness | Passed ✅ | 67 | 68.88402493579531 | 15.554084739082013 | 0.0 | 4657 | 0 | 63.672586999985015 | 1306.7071749999855 |
| Aggregated | Passed ✅ | 67 | 70.04715047556802 | 32.19364887266728 | 0.0 | 9639 | 0 | 63.48201800000197 | 1306.7071749999855 |

What's Changed
* Add sync iterator by turmind in https://github.com/BerriAI/litellm/pull/2794
* fix(proxy_server.py): allow user to set in-memory + redis ttl by krrishdholakia in https://github.com/BerriAI/litellm/pull/2789
* Fix max_tokens type in main.py by phact in https://github.com/BerriAI/litellm/pull/2790

New Contributors
* turmind made their first contribution in https://github.com/BerriAI/litellm/pull/2794

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.34.20...v1.34.21

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 82 | 99.44767344857672 | 1.5262768715973858 | 0.0 | 457 | 0 | 76.33821500002114 | 1827.514999000016 |
| /health/liveliness | Passed ✅ | 66 | 68.59761310860081 | 15.376321043401234 | 0.0 | 4604 | 0 | 63.56413200001043 | 1420.0152670000534 |
| /health/readiness | Passed ✅ | 66 | 68.43111482729716 | 15.122498193857686 | 0.0 | 4528 | 0 | 63.35147300001154 | 1183.1644380000057 |
| Aggregated | Passed ✅ | 66 | 69.9892674372716 | 32.0250961088563 | 0.0 | 9589 | 0 | 63.35147300001154 | 1827.514999000016 |