Friendli-client

Now it is available to use API calls to Friendli Dedicated Endpoints.

python
from friendli import Friendli

client = Friendli(use_dedicated_endpoint=True)
chat = client.chat.completions.create(
model="{endpoint_id}",
messages=[
{
"role": "user",
"content": "Give three tips for staying healthy.",
}
]
)


If you want to send a request to a specific adapter of the Multi-LoRA endpoint, provide "{endpoint_id}:{adapter_route}" to `model` argument:

python
from friendli import Friendli

client = Friendli(use_dedicated_endpoint=True)
chat = client.chat.completions.create(
model="{endpoint_id}:{adapter_route}",
messages=[
{
"role": "user",
"content": "Give three tips for staying healthy.",
}
]
)

- Deprecate model conversion and quantization. Alternatively, please use [`friendli-model-optmizer`](https://github.com/friendliai/friendli-model-optimizer) to quantize your models.
- Increase default HTTP timeout.

- Support for Tool Calling API: Added new API to support tool calling.
- Phi3 INT8 Support: Implemented support for Phi3 INT8.
- Snowflake Arctic FP8 Quantizer: Introduced new quantizer for Snowflake Arctic FP8.
- Added support for INT8 quantization for Llama and refactored quantizer to use only safetensors.

Updating Patch Version

This patch version Introduces explicit resource management to prevent unexpected resource leaks.
By default, the library closes underlying HTTP and gRPC connections when the client is garbage-collected. However, you can now manually close the `Friendli` or `AsyncFriendli` client using the `.close()` method or utilize a context manager to ensure proper closure when exiting a `with` block.

Usage examples

python
import asyncio
from friendli import AsyncFriendli

client = AsyncFriendli(base_url="0.0.0.0:8000", use_grpc=True)

async def run():
async with client:
stream = await client.completions.create(
prompt="Explain what gRPC is. Also give me a Python code snippet of gRPC client.",
stream=True,
top_k=1,
)

async for chunk in stream:
print(chunk.text, end="", flush=True)

asyncio.run(run())

- gRPC client support for completions API.

- Minor: add a default value for the "index" and "text" fields of the completion stream's chunk.