1.Incognito Requests - Don't log anything - docs: https://docs.litellm.ai/docs/proxy/enterprise#incognito-requests---dont-log-anything
When `no-log=True`, the request will **not be logged on any callbacks** and there will be **no server logs on litellm**
python
import openai
client = openai.OpenAI(
api_key="anything", proxy api-key
base_url="http://0.0.0.0:8000" # litellm proxy
)
response = client.chat.completions.create(
model="gpt-3.5-turbo",
messages = [
{
"role": "user",
"content": "this is a test request, write a short poem"
}
],
extra_body={
"no-log": True
}
)
print(response)
2. Allow user to pass messages.name for claude-3, perplexity
Note: Before this pr - the two providers would raise errors with the `name` param
LiteLLM SDK
python
import litellm
response = litellm.completion(
model="claude-3-opus-20240229",
messages = [
{"role": "user", "content": "Hi gm!", "name": "ishaan"},
]
)
LiteLLM Proxy Server
python
import openai
client = openai.OpenAI(
api_key="anything",
base_url="http://0.0.0.0:8000"
)
response = client.chat.completions.create(
model="claude-3-opus-20240229"",
messages = [
{"role": "user", "content": "Hi gm!", "name": "ishaan"},
])
print(response)
3. If user is using `run_gunicorn` use cpu_count to select optimal `num_workers`
4. AzureOpenAI - Pass api_version to litellm proxy per request
Usage - sending a request to litellm proxy
python
from openai import AzureOpenAI
client = AzureOpenAI(
api_key="dummy",
I want to use a specific api_version, other than default 2023-07-01-preview
api_version="2023-05-15",
OpenAI Proxy Endpoint
azure_endpoint="https://openai-proxy.domain.com"
)
response = client.chat.completions.create(
model="gpt-35-turbo-16k-qt",
messages=[
{"role": "user", "content": "Some content"}
],
)
What's Changed
* [Feat] Support messages.name for claude-3, perplexity ai API by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2399
* docs: fix yaml typo in proxy/configs.md by GuillermoBlasco in https://github.com/BerriAI/litellm/pull/2402
* [Feat] LiteLLM - use cpu_count for default num_workers, run locust load test by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2406
* [FEAT] AzureOpenAI - Pass `api_version` to litellm per request by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2403
* Add quickstart deploy with k8s by GuillermoBlasco in https://github.com/BerriAI/litellm/pull/2409
* Update Docs for Kubernetes by H0llyW00dzZ in https://github.com/BerriAI/litellm/pull/2411
* [FEAT-liteLLM Proxy] Incognito Requests - Don't log anything by ishaan-jaff in https://github.com/BerriAI/litellm/pull/2408
* Fix Docs Formatting in Website by H0llyW00dzZ in https://github.com/BerriAI/litellm/pull/2413
New Contributors
* GuillermoBlasco made their first contribution in https://github.com/BerriAI/litellm/pull/2402
* H0llyW00dzZ made their first contribution in https://github.com/BerriAI/litellm/pull/2411
**Full Changelog**: https://github.com/BerriAI/litellm/compare/v1.30.3...v1.30.4