Litellm

What's Changed
* fix: Lunary integration by 7HR4IZ3 in https://github.com/BerriAI/litellm/pull/4379
* [Fix] - Admin UI login bug by ishaan-jaff in https://github.com/BerriAI/litellm/pull/4382
* Support custom model info for router logic (e.g. `max_input_tokens`) by krrishdholakia in https://github.com/BerriAI/litellm/pull/4388
* fix(vertex_httpx.py): cover gemini content violation (on prompt) by krrishdholakia in https://github.com/BerriAI/litellm/pull/4392
* [Feat-Enterprise] - Allow setting custom public routes by ishaan-jaff in https://github.com/BerriAI/litellm/pull/4389
* docs control available public routes by ishaan-jaff in https://github.com/BerriAI/litellm/pull/4394
* Log rejected router requests to langfuse by krrishdholakia in https://github.com/BerriAI/litellm/pull/4390
* Fix /spend/calculate use model_group_alias when set by ishaan-jaff in https://github.com/BerriAI/litellm/pull/4395

New Contributors
* 7HR4IZ3 made their first contribution in https://github.com/BerriAI/litellm/pull/4379

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.40.25...v1.40.26



Docker Run LiteLLM Proxy


docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.26



Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 130.0 | 156.53979613674755 | 6.329682565511454 | 0.0 | 1894 | 0 | 109.70386399998233 | 2175.4312479999953 |
| Aggregated | Passed ✅ | 130.0 | 156.53979613674755 | 6.329682565511454 | 0.0 | 1894 | 0 | 109.70386399998233 | 2175.4312479999953 |

What's Changed
* feat(dynamic_rate_limiter.py): Dynamic tpm quota (multiple projects) by krrishdholakia in https://github.com/BerriAI/litellm/pull/4349
* fix(router.py): Content Policy Fallbacks for Azure 'content_filter' responses by krrishdholakia in https://github.com/BerriAI/litellm/pull/4365
* Disable message redaction in logs via request header by msabramo in https://github.com/BerriAI/litellm/pull/4352


**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.40.24...v1.40.25



Docker Run LiteLLM Proxy


docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.25



Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 120.0 | 137.1328113917474 | 6.561515229902106 | 0.0 | 1963 | 0 | 98.4713450000072 | 1831.7410280000104 |
| Aggregated | Passed ✅ | 120.0 | 137.1328113917474 | 6.561515229902106 | 0.0 | 1963 | 0 | 98.4713450000072 | 1831.7410280000104 |

What's Changed
* refactor(litellm_logging.py): refactors how slack_alerting generates langfuse trace url by krrishdholakia in https://github.com/BerriAI/litellm/pull/4344
* [Security Fix - Proxy Server ADMIN UI] - Store credentials in cookies + use strong JWT signing secret by ishaan-jaff in https://github.com/BerriAI/litellm/pull/4357
* [Test] Test routes on LiteLLM Proxy always includes OpenAI Routes by ishaan-jaff in https://github.com/BerriAI/litellm/pull/4356
* fix - Can't access /v1/audio/speech with some user key by ishaan-jaff in https://github.com/BerriAI/litellm/pull/4360


**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.40.22...v1.40.24



Docker Run LiteLLM Proxy


docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.24



Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 130.0 | 154.96801309401323 | 6.361794835434345 | 0.0 | 1904 | 0 | 115.54615199997897 | 1161.0779019999882 |
| Aggregated | Passed ✅ | 130.0 | 154.96801309401323 | 6.361794835434345 | 0.0 | 1904 | 0 | 115.54615199997897 | 1161.0779019999882 |

What's Changed
* fix: use per-token costs for claude via vertex_ai by spdustin in https://github.com/BerriAI/litellm/pull/4337
* [Feat] Admin UI - Show Cache hit stats by ishaan-jaff in https://github.com/BerriAI/litellm/pull/4340
* fix - liteLLM proxy /moderations endpoint returns 500 error when model is not specified by ishaan-jaff in https://github.com/BerriAI/litellm/pull/4342
* [Fix + Test] - Spend tags not getting stored on 1.40.9 by ishaan-jaff in https://github.com/BerriAI/litellm/pull/4345
* Print content window fallbacks on startup to help verify configuration by lolsborn in https://github.com/BerriAI/litellm/pull/4350

New Contributors
* lolsborn made their first contribution in https://github.com/BerriAI/litellm/pull/4350

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.40.21...v1.40.22



Docker Run LiteLLM Proxy


docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.22



Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 120.0 | 147.06905652027004 | 6.431081863451109 | 0.0 | 1924 | 0 | 100.04098199999589 | 1834.3141159999732 |
| Aggregated | Passed ✅ | 120.0 | 147.06905652027004 | 6.431081863451109 | 0.0 | 1924 | 0 | 100.04098199999589 | 1834.3141159999732 |

What's Changed
* feat: friendli ai support by pocca2048 in https://github.com/BerriAI/litellm/pull/4121

New Contributors
* pocca2048 made their first contribution in https://github.com/BerriAI/litellm/pull/4121

**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.40.20...v1.40.21



Docker Run LiteLLM Proxy


docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.21



Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 140.0 | 163.37791068962042 | 6.407997421786419 | 0.0 | 1917 | 0 | 114.2956310000045 | 1749.6762069999932 |
| Aggregated | Passed ✅ | 140.0 | 163.37791068962042 | 6.407997421786419 | 0.0 | 1917 | 0 | 114.2956310000045 | 1749.6762069999932 |

What's Changed
* docs - add algolia search 🫡 by ishaan-jaff in https://github.com/BerriAI/litellm/pull/4320
* [Feat] allow using custom router strategy by ishaan-jaff in https://github.com/BerriAI/litellm/pull/4318
* fix(utils.py): allow dropping specific openai params by krrishdholakia in https://github.com/BerriAI/litellm/pull/4313
* fix(user_api_key_auth.py): ensure user has access to fallback models by krrishdholakia in https://github.com/BerriAI/litellm/pull/4321
* Update proxy_cli.py by vanpelt in https://github.com/BerriAI/litellm/pull/4325
* fix(key_management_endpoints.py): use common _duration_in_seconds function by krrishdholakia in https://github.com/BerriAI/litellm/pull/4323
* feat(router.py): allow user to call specific deployment via id by krrishdholakia in https://github.com/BerriAI/litellm/pull/4290
* test(test_python_38.py): add coverage for non-gen settings config.yaml flow by krrishdholakia in https://github.com/BerriAI/litellm/pull/4328
* [Fix] user field and user_api_key_* is sometimes omitted randomly by ishaan-jaff in https://github.com/BerriAI/litellm/pull/4322


**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.40.19...v1.40.20



Docker Run LiteLLM Proxy


docker run \
-e STORE_MODEL_IN_DB=True \
-p 4000:4000 \
ghcr.io/berriai/litellm:main-v1.40.20



Don't want to maintain your internal proxy? get in touch 🎉
Hosted Proxy Alpha: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

Load Test LiteLLM Proxy Results

| Name | Status | Median Response Time (ms) | Average Response Time (ms) | Requests/s | Failures/s | Request Count | Failure Count | Min Response Time (ms) | Max Response Time (ms) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| /chat/completions | Passed ✅ | 79 | 105.64488981589672 | 6.515118790194818 | 0.0 | 1950 | 0 | 67.60507300003837 | 3342.9461570000285 |
| Aggregated | Passed ✅ | 79 | 105.64488981589672 | 6.515118790194818 | 0.0 | 1950 | 0 | 67.60507300003837 | 3342.9461570000285 |