Litellm

What's Changed
* [Fix] Add better observability for debugging lowest latency routing by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3283
* [Feat] lowest-latency routing - penalize timeout errors from api endpoints by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3281


**Full Changelog**: https://github.com/BerriAI/litellm/compare/1.35.24.dev6...v1.35.25

Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 95 | 104.03587220041199 | 1.616816849539448 | 0.0 | 484 | 0 | 88.34031499998218 | 1242.1794910000017 |
| /health/liveliness | Passed ✅ | 79 | 83.35600463999175 | 15.319673702454356 | 0.0 | 4586 | 0 | 77.10968600002843 | 1444.1510330000256 |
| /health/readiness | Passed ✅ | 79 | 82.3850293131141 | 15.309652110411756 | 0.0 | 4583 | 0 | 77.28314799999225 | 1293.70391599997 |
| Aggregated | Passed ✅ | 79 | 83.93189565585871 | 32.24614266240556 | 0.0 | 9653 | 0 | 77.10968600002843 | 1444.1510330000256 |

🚨 UI Issue - INVESTIGATING. Use v`1.35.23` for last stable UI release.

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.35.24.dev1...v1.35.24

Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 57 | 62.57754433107982 | 1.482967464223559 | 0.0 | 444 | 0 | 50.958461999982774 | 817.4045920000026 |
| /health/liveliness | Passed ✅ | 40 | 43.222332060980996 | 15.664678845064172 | 0.003340016811314322 | 4690 | 1 | 38.411224000014954 | 999.1422629999818 |
| /health/readiness | Passed ✅ | 41 | 43.62012660392403 | 15.490997970875826 | 0.003340016811314322 | 4638 | 1 | 38.66341999997758 | 1356.7161670000019 |
| Aggregated | Passed ✅ | 41 | 44.29055610294732 | 32.638644280163554 | 0.006680033622628644 | 9772 | 2 | 38.411224000014954 | 1356.7161670000019 |

What's Changed
* Support for Greenscale AI logging by greenscale-nandesh in https://github.com/BerriAI/litellm/pull/3098
* [UI] V0 - Edit Model tpm, rpm, api_base by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3277
* [Feat] Show model, api base in APITimeoutError exceptions by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3279
* [UI-Polish] filter by models on dropdown by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3280
* fix(proxy_server.py): fix `/config/update` by krrishdholakia in https://github.com/BerriAI/litellm/pull/3282

New Contributors
* greenscale-nandesh made their first contribution in https://github.com/BerriAI/litellm/pull/3098

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.35.24...1.35.24.dev6

Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 94 | 104.27768922222346 | 1.4429443368713684 | 0.0 | 432 | 0 | 86.3644869999689 | 763.5629149999659 |
| /health/liveliness | Passed ✅ | 78 | 80.55345714471746 | 15.394746408889207 | 0.0 | 4609 | 0 | 73.8546030000009 | 696.1782949999815 |
| /health/readiness | Passed ✅ | 78 | 82.0756289784257 | 15.481590281015723 | 0.0033401489279429823 | 4635 | 1 | 74.05194300000062 | 1385.1709909999954 |
| Aggregated | Passed ✅ | 78 | 82.34181335665636 | 32.319281026776295 | 0.0033401489279429823 | 9676 | 1 | 73.8546030000009 | 1385.1709909999954 |

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.35.23...v1.35.24.dev1

Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 96 | 104.08889620803765 | 1.4128712196268784 | 0.0 | 423 | 0 | 90.27053599999135 | 1337.9505959999847 |
| /health/liveliness | Passed ✅ | 79 | 81.64779973608695 | 15.605045716540841 | 0.0 | 4672 | 0 | 77.07790199998499 | 1228.4024810000176 |
| /health/readiness | Passed ✅ | 79 | 83.50619211328004 | 14.890259804010931 | 0.00668024217317673 | 4458 | 2 | 77.08443000001353 | 1429.602591000048 |
| Aggregated | Passed ✅ | 79 | 83.50871222694458 | 31.90817674017865 | 0.00668024217317673 | 9553 | 2 | 77.07790199998499 | 1429.602591000048 |

What's Changed
* build(ui): user_dashboard.tsx by krrishdholakia in https://github.com/BerriAI/litellm/pull/3260


**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.35.21...v1.35.23

Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 95 | 104.34841019318016 | 1.469716999607524 | 0.0 | 440 | 0 | 86.94174600003635 | 1376.3201009999761 |
| /health/liveliness | Passed ✅ | 78 | 80.76396643555898 | 15.161466957314891 | 0.0066805318163978365 | 4539 | 2 | 73.81250000003092 | 1444.6542679999652 |
| /health/readiness | Passed ✅ | 78 | 80.15375961581503 | 15.50217407995118 | 0.0 | 4641 | 0 | 74.04167700002517 | 931.0259220000034 |
| Aggregated | Passed ✅ | 78 | 81.54828924251524 | 32.133358036873595 | 0.0066805318163978365 | 9620 | 2 | 73.81250000003092 | 1444.6542679999652 |

What's Changed
* ui - cleanup showing router settings by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3249
* (utils.py) - Add seed for Groq by Manouchehri in https://github.com/BerriAI/litellm/pull/3252
* Add missing tool_calls and name to messages by elisalimli in https://github.com/BerriAI/litellm/pull/3124
* [Fix] Refactor slack alerting, use one httpx client by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3255
* fix(utils.py): fix 'no-cache': true when caching is turned on by krrishdholakia in https://github.com/BerriAI/litellm/pull/3250
* UI - filter metrics by model group by ishaan-jaff in https://github.com/BerriAI/litellm/pull/3259


**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.35.20.dev2...v1.35.21

Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat



Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat



Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 58 | 64.9741058238994 | 1.5931582433810807 | 0.0 | 477 | 0 | 51.917337999952906 | 869.7110660000362 |
| /health/liveliness | Passed ✅ | 41 | 44.58716365170725 | 15.554167587894534 | 0.006679908777279165 | 4657 | 2 | 38.56498999999758 | 1391.5798419999987 |
| /health/readiness | Passed ✅ | 41 | 43.37650948992081 | 15.243551829751054 | 0.0 | 4564 | 0 | 38.630514000033145 | 1020.0773849999791 |
| Aggregated | Passed ✅ | 41 | 45.02015455929064 | 32.39087766102667 | 0.006679908777279165 | 9698 | 2 | 38.56498999999758 | 1391.5798419999987 |