What's Changed
Add `StreamedResponse` and `AsyncStreamedResponse` to enable parsing responses that contain both text _and_ tool calls. See PR https://github.com/jackmpcollins/magentic/pull/383 or the new docs (copied below) https://magentic.dev/streaming/#StreamedResponse for more details.
⚡ StreamedResponse
Some LLMs have the ability to generate text output and make tool calls in the same response. This allows them to perform chain-of-thought reasoning or provide additional context to the user. In magentic, the `StreamedResponse` (or `AsyncStreamedResponse`) class can be used to request this type of output. This object is an iterable of `StreamedStr` (or `AsyncStreamedStr`) and `FunctionCall` instances.
!!! warning "Consuming StreamedStr"
The StreamedStr object must be iterated over before the next item in the `StreamedResponse` is processed, otherwise the string output will be lost. This is because the `StreamedResponse` and `StreamedStr` share the same underlying generator, so advancing the `StreamedResponse` iterator skips over the `StreamedStr` items. The `StreamedStr` object has internal caching so after iterating over it once the chunks will remain available.
In the example below, we request that the LLM generates a greeting and then calls a function to get the weather for two cities. The `StreamedResponse` object is then iterated over to print the output, and the `StreamedStr` and `FunctionCall` items are processed separately.
python
from magentic import prompt, FunctionCall, StreamedResponse, StreamedStr
def get_weather(city: str) -> str:
return f"The weather in {city} is 20°C."
prompt(
"Say hello, then get the weather for: {cities}",
functions=[get_weather],
)
def describe_weather(cities: list[str]) -> StreamedResponse: ...
response = describe_weather(["Cape Town", "San Francisco"])
for item in response:
if isinstance(item, StreamedStr):
for chunk in item:
print the chunks as they are received
print(chunk, sep="", end="")
print()
if isinstance(item, FunctionCall):
print the function call, then call it and print the result
print(item)
print(item())
Hello! I'll get the weather for Cape Town and San Francisco for you.
FunctionCall(<function get_weather at 0x1109825c0>, 'Cape Town')
The weather in Cape Town is 20°C.
FunctionCall(<function get_weather at 0x1109825c0>, 'San Francisco')
The weather in San Francisco is 20°C.
PRs
* Test Ollama via `OpenaiChatModel` by jackmpcollins in https://github.com/jackmpcollins/magentic/pull/281
* Rename test to test_openai_chat_model_acomplete_ollama by jackmpcollins in https://github.com/jackmpcollins/magentic/pull/381
* Add `(Async)StreamedResponse` for multi-part responses by jackmpcollins in https://github.com/jackmpcollins/magentic/pull/383
**Full Changelog**: https://github.com/jackmpcollins/magentic/compare/v0.33.0...v0.34.0