Livekit-agents

✨ [NEW] OpenAI Realtime API support

We're partnering with OpenAI on a new `MultimodalAgent` API in the Agents framework. This class completely wraps OpenAI’s Realtime API, abstract away the raw wire protocol, and provide an ultra-low latency WebRTC transport between GPT-4o and your users’ devices. This same stack powers Advanced Voice in the ChatGPT app.

- Try the Realtime API in our [playground](https://playground.livekit.io/) [[code](https://github.com/livekit-examples/realtime-playground)]
- Check out our [guide](https://docs.livekit.io/agents/openai) to building your first app with this new API

Patch Changes

- bump livekit to v0.17.2 - [815](https://github.com/livekit/agents/pull/815) ([theomonnom](https://github.com/theomonnom))

- silero: support any sample rate - [805](https://github.com/livekit/agents/pull/805) ([theomonnom](https://github.com/theomonnom))


livekit-plugins-openai0.8.5
Patch Changes

- Fix function for OpenAI Assistants - [784](https://github.com/livekit/agents/pull/784) ([keepingitneil](https://github.com/keepingitneil))

Minor Changes

- rename voice_assistant.state to lk.agent.state - [772](https://github.com/livekit/agents/pull/772) ([bcherry](https://github.com/bcherry))

Patch Changes

- bump rtc - [782](https://github.com/livekit/agents/pull/782) ([nbsp](https://github.com/nbsp))

- improve graceful shutdown - [756](https://github.com/livekit/agents/pull/756) ([theomonnom](https://github.com/theomonnom))

- avoid returning tiny frames from TTS - [747](https://github.com/livekit/agents/pull/747) ([theomonnom](https://github.com/theomonnom))

- windows: default to threaded executor & fix dev mode - [755](https://github.com/livekit/agents/pull/755) ([theomonnom](https://github.com/theomonnom))

- 11labs: send phoneme in one entire xml chunk - [766](https://github.com/livekit/agents/pull/766) ([theomonnom](https://github.com/theomonnom))

- fix: process not starting if num_idle_processes is zero - [763](https://github.com/livekit/agents/pull/763) ([theomonnom](https://github.com/theomonnom))

- voiceassistant: avoid tiny frames on playout - [750](https://github.com/livekit/agents/pull/750) ([theomonnom](https://github.com/theomonnom))

- voiceassistant: expose turn_completion_delay - [752](https://github.com/livekit/agents/pull/752) ([theomonnom](https://github.com/theomonnom))

- limit concurrent process init to 1 - [751](https://github.com/livekit/agents/pull/751) ([theomonnom](https://github.com/theomonnom))

- Add typing-extensions as a dependency - [778](https://github.com/livekit/agents/pull/778) ([keepingitneil](https://github.com/keepingitneil))

- Allow setting LLM temperature with VoiceAssistant - [741](https://github.com/livekit/agents/pull/741) ([davidzhao](https://github.com/davidzhao))

- better dev defaults - [762](https://github.com/livekit/agents/pull/762) ([theomonnom](https://github.com/theomonnom))

- voiceassistant: allow to cancel llm generation inside before_llm_cb - [753](https://github.com/livekit/agents/pull/753) ([theomonnom](https://github.com/theomonnom))

- use os.exit to exit forcefully - [770](https://github.com/livekit/agents/pull/770) ([theomonnom](https://github.com/theomonnom))


livekit-plugins-openai0.8.3
Patch Changes

- Introduce function calling to OpenAI Assistants - [710](https://github.com/livekit/agents/pull/710) ([keepingitneil](https://github.com/keepingitneil))

- Add Cerebras to OpenAI Plugin - [731](https://github.com/livekit/agents/pull/731) ([henrytwo](https://github.com/henrytwo))


livekit-plugins-deepgram0.6.7
Patch Changes

- Only send actual audio to Deepgram using a basic audio RMS filter - [738](https://github.com/livekit/agents/pull/738) ([keepingitneil](https://github.com/keepingitneil))

- defaults to nova-2-general model - [726](https://github.com/livekit/agents/pull/726) ([davidzhao](https://github.com/davidzhao))


livekit-plugins-cartesia0.4.2
Patch Changes

- Add support for cartesia voice control - [740](https://github.com/livekit/agents/pull/740) ([bcherry](https://github.com/bcherry))


livekit-plugins-anthropic0.2.1
Patch Changes

- Fixes to Anthropic Function Calling - [708](https://github.com/livekit/agents/pull/708) ([keepingitneil](https://github.com/keepingitneil))

Patch Changes

- tts_forwarder: don't raise inside mark_{audio,text}\_segment_end when nothing was pushed - [730](https://github.com/livekit/agents/pull/730) ([theomonnom](https://github.com/theomonnom))


livekit-plugins-openai0.8.2
Patch Changes

- Add deepseek LLMs at OpenAI plugin - [714](https://github.com/livekit/agents/pull/714) ([lenage](https://github.com/lenage))

- skip processing of choice.delta when it is None - [705](https://github.com/livekit/agents/pull/705) ([theomonnom](https://github.com/theomonnom))


livekit-plugins-google0.7.0
Minor Changes

- Enable use of Google STT with Application Default Credentials. - [721](https://github.com/livekit/agents/pull/721) ([rsinnet](https://github.com/rsinnet))

Patch Changes

- google-tts: ignore wav header - [703](https://github.com/livekit/agents/pull/703) ([theomonnom](https://github.com/theomonnom))


livekit-plugins-elevenlabs0.7.4
Patch Changes

- elevenlabs: expose enable_ssml_parsing - [723](https://github.com/livekit/agents/pull/723) ([theomonnom](https://github.com/theomonnom))


livekit-plugins-anthropic0.2.0
Minor Changes

- bump anthropic for release - [724](https://github.com/livekit/agents/pull/724) ([theomonnom](https://github.com/theomonnom))

Patch Changes

- improve gracefully_cancel logic - [720](https://github.com/livekit/agents/pull/720) ([theomonnom](https://github.com/theomonnom))

- Make ctx.room.name available prior to connection - [716](https://github.com/livekit/agents/pull/716) ([davidzhao](https://github.com/davidzhao))

- ipc: add threaded job runner - [684](https://github.com/livekit/agents/pull/684) ([theomonnom](https://github.com/theomonnom))

- voiceassistant: add VoiceAssistantState - [654](https://github.com/livekit/agents/pull/654) ([lukasIO](https://github.com/lukasIO))

- add JobContext.wait_for_participant - [712](https://github.com/livekit/agents/pull/712) ([theomonnom](https://github.com/theomonnom))

- fix non pickleable log - [691](https://github.com/livekit/agents/pull/691) ([theomonnom](https://github.com/theomonnom))

- voiceassistant: skip speech initialization if interrupted - [715](https://github.com/livekit/agents/pull/715) ([theomonnom](https://github.com/theomonnom))

- bump required livekit version to 0.15.2 - [722](https://github.com/livekit/agents/pull/722) ([theomonnom](https://github.com/theomonnom))

- voiceassistant: add will_synthesize_assistant_speech - [706](https://github.com/livekit/agents/pull/706) ([theomonnom](https://github.com/theomonnom))

- voiceassistant: fix mark_audio_segment_end with no audio data - [719](https://github.com/livekit/agents/pull/719) ([theomonnom](https://github.com/theomonnom))


livekit-plugins-google0.6.3
Patch Changes

- Fix Google STT exception when no valid speech is recognized - [680](https://github.com/livekit/agents/pull/680) ([davidzhao](https://github.com/davidzhao))


livekit-plugins-deepgram0.6.6
Patch Changes

- deepgram: switch the default model to phonecall - [676](https://github.com/livekit/agents/pull/676) ([theomonnom](https://github.com/theomonnom))

Patch Changes

- Pass JobContext to participant entrypoint function - [694](https://github.com/livekit/agents/pull/694) ([davidzhao](https://github.com/davidzhao))

- voiceassistant: keep punctuations when sending agent transcription - [648](https://github.com/livekit/agents/pull/648) ([theomonnom](https://github.com/theomonnom))


livekit-plugins-anthropic0.1.1
livekit-plugins-anthropic