Litellm

**Full Changelog**: https://github.com/BerriAI/litellm/compare/1.35.5.dev2...v1.35.5

Call 100+ LLMS, run /health checks on Admin UI

👉 Edit + Test langfuse and SlackHQ configurations on the LiteLLM UI

🛠️ UI - fix - adding azure OpenAI on admin ui

⚡️ [Fix] Load proxy models when proxy starts up

✅ [LiteLLM UI] Show Error message for 10-20s (h/t Graham Neubig for this request)

😇 QA - Added Tests for /health endpoints on Proxy
<img width="623" alt="Screenshot 2024-04-13 at 10 12 53 PM" src="https://github.com/BerriAI/litellm/assets/29436595/536719af-cd08-4a7c-bc97-6127f23f8666">

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 92 | 97.28560691555401 | 1.5031977156223721 | 0.0 | 450 | 0 | 86.17738300006295 | 926.6810239999472 |
| /health/liveliness | Passed ✅ | 78 | 80.18245176222777 | 15.299212305667698 | 0.003340439368049716 | 4580 | 1 | 74.2219969999951 | 1033.391943999959 |
| /health/readiness | Passed ✅ | 78 | 80.9458593949494 | 15.342638017452344 | 0.0 | 4593 | 0 | 74.08499699999993 | 1307.7433869999595 |
| Aggregated | Passed ✅ | 78 | 81.34661585617866 | 32.14504803874242 | 0.003340439368049716 | 9623 | 1 | 74.08499699999993 | 1307.7433869999595 |

What's Changed
* UI - fix - adding azure OpenAI on admin ui by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3004
* [test] - Test /health endpoints by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3003
* [Fix] Load proxy models when proxy starts up by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3005
* UI - edit Langfuse vars on ui by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3006
* [UI] Show Errors for 20s by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3008
* UI - test langfuse callbacks on Admin UI by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3007
* [UI] QA fixes adding langfuse callback by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3010
* [UI] Add / Test Slack Alerting by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3011
* Admin UI -run /health check through admin ui by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3014
* Litellm prometheus service metrics by krrishdholakia in https://github.com/BerriAI/litellm/pull/3017


**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.35.4...1.35.5.dev2

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 80 | 85.68544724824264 | 1.4263860259563899 | 0.0 | 427 | 0 | 75.07200899999589 | 949.8076319999882 |
| /health/liveliness | Passed ✅ | 65 | 68.0963399051023 | 15.38292189585287 | 0.0 | 4605 | 0 | 63.464997000096446 | 851.2580190000563 |
| /health/readiness | Passed ✅ | 65 | 68.89657557841379 | 15.506519748219116 | 0.0 | 4642 | 0 | 63.50266200001897 | 1345.5118699999957 |
| Aggregated | Passed ✅ | 65 | 69.25669165526033 | 32.315827670028376 | 0.0 | 9674 | 0 | 63.464997000096446 | 1345.5118699999957 |

What's Changed
* fix(router.py): initial commit for semaphores on router by krrishdholakia in https://github.com/BerriAI/litellm/pull/2996


**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.35.3...v1.35.4

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 55 | 61.81466973473701 | 1.5865369739586979 | 0.0 | 475 | 0 | 50.56273999997529 | 1291.203577999994 |
| /health/liveliness | Passed ✅ | 40 | 43.40270568389331 | 15.511321488556195 | 0.0 | 4644 | 0 | 38.63372099999651 | 1381.9246239999927 |
| /health/readiness | Passed ✅ | 41 | 42.992702731896564 | 15.497961177196542 | 0.003340077839913048 | 4640 | 1 | 38.45442200000093 | 1314.9427230000015 |
| Aggregated | Passed ✅ | 41 | 44.103932164771024 | 32.595819639711436 | 0.003340077839913048 | 9759 | 1 | 38.45442200000093 | 1381.9246239999927 |

What's Changed
* fix - use copy.deepcopy on /model/info endpoints by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2997


**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.35.2.dev4...v1.35.3

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 81 | 86.15952575669651 | 1.4964555491804343 | 0.0 | 448 | 0 | 75.1446659999715 | 912.2716670000273 |
| /health/liveliness | Passed ✅ | 66 | 69.61895297303029 | 15.729484779220236 | 0.0 | 4709 | 0 | 63.55963599997949 | 1172.4433609999778 |
| /health/readiness | Passed ✅ | 66 | 69.02371470551194 | 15.392114220141611 | 0.0 | 4608 | 0 | 63.66152200001807 | 1251.2778020000042 |
| Aggregated | Passed ✅ | 66 | 70.0969169945723 | 32.61805454854228 | 0.0 | 9765 | 0 | 63.55963599997949 | 1251.2778020000042 |

What's Changed
* Add JSON mode to Gemini (Vertex AI) by Manouchehri in https://github.com/BerriAI/litellm/pull/2964
* fix - delete key from inMemory Cache after /key/update by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2965
* fix(handle_jwt.py): User cost tracking via JWT Auth by krrishdholakia in https://github.com/BerriAI/litellm/pull/2970
* fix(proxy_server.py): support tracking org spend by krrishdholakia in https://github.com/BerriAI/litellm/pull/2978


**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.35.1.dev2...v1.35.2

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 41 | 44.1075405043854 | 1.523085528016257 | 0.0 | 456 | 0 | 35.037465000016255 | 749.7959620000074 |
| /health/liveliness | Passed ✅ | 25 | 27.538079022664157 | 15.474682568638856 | 0.0 | 4633 | 0 | 23.358772999984012 | 1158.9303129999848 |
| /health/readiness | Passed ✅ | 25 | 27.88288945533811 | 15.70514945774658 | 0.0 | 4702 | 0 | 23.31466300000784 | 1296.1237730000335 |
| Aggregated | Passed ✅ | 25 | 28.475365621591475 | 32.702917554401694 | 0.0 | 9791 | 0 | 23.31466300000784 | 1296.1237730000335 |

What's Changed
* fix(proxy_server.py): support tracking org spend by krrishdholakia in https://github.com/BerriAI/litellm/pull/2978
* Fix .dockerignore by bcvanmeurs in https://github.com/BerriAI/litellm/pull/2982
* fix - stop printing api_key in debug mode by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2977
* truncate long slack msg by Dev-Khant in https://github.com/BerriAI/litellm/pull/2984
* [Feat] Support + Test caching for TextCompletion by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2991
* ui - fix setting slack alerting by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2985

New Contributors
* bcvanmeurs made their first contribution in https://github.com/BerriAI/litellm/pull/2982
* Dev-Khant made their first contribution in https://github.com/BerriAI/litellm/pull/2984

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.35.2.dev3...v1.35.2.dev4

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 92 | 99.49399542663635 | 1.4799813646970934 | 0.0 | 443 | 0 | 85.82718999997496 | 1355.355900999939 |
| /health/liveliness | Passed ✅ | 78 | 80.01183602425647 | 15.28757274233386 | 0.003340815721663868 | 4576 | 1 | 73.83316000002083 | 846.8029710000167 |
| /health/readiness | Passed ✅ | 78 | 80.38568712813309 | 15.46129515986038 | 0.0 | 4628 | 0 | 73.86776399999917 | 1050.429261000005 |
| Aggregated | Passed ✅ | 78 | 81.08582581631553 | 32.22884926689133 | 0.003340815721663868 | 9647 | 1 | 73.83316000002083 | 1355.355900999939 |