TL;DR:

  • LiveKit Agents is a Python framework for real-time voice AI, handling WebRTC transport, STT/LLM/TTS chaining, and interruption detection out of the box
  • You write the agent logic; LiveKit handles the low-level audio plumbing, turn detection, and connection management
  • Supported integrations include Deepgram, Whisper, OpenAI, Anthropic, ElevenLabs, Cartesia, and Silero VAD

If you’ve tried to build a voice AI agent from scratch, you know the work that isn’t the LLM call. Turn detection. Barge-in handling. Audio buffering. WebRTC negotiation. Latency management across three separate API calls. It’s a non-trivial infrastructure problem before you’ve written a single line of agent logic.

LiveKit Agents is built to absorb that complexity. It gives you a Python framework where you define what your agent does, and the framework handles how audio flows in and out in real time.

What LiveKit Is

LiveKit is an open-source platform for real-time audio and video communication, built on WebRTC. The core infrastructure (LiveKit Server) handles room management, track routing, and signalling. LiveKit Agents runs on top of this as a Python framework for agents that participate in rooms as audio participants.

That WebRTC foundation matters. It means your voice agent is accessible directly from a browser without a native app, uses end-to-end encryption, and handles adaptive bitrate and packet loss recovery automatically.

The Pipeline Model

A voice AI agent built with LiveKit Agents follows a three-stage pipeline:

  1. STT (Speech to Text): audio from the user is transcribed in real time. Supported: Deepgram Nova, Whisper (via OpenAI or local), Azure Speech, Google Cloud Speech.

  2. LLM: the transcript is sent to a language model. Supported: OpenAI (GPT-4o, GPT-4o-mini), Anthropic (Claude), Groq, local models via Ollama.

  3. TTS (Text to Speech): the LLM response is synthesised and streamed back as audio. Supported: ElevenLabs, Cartesia, OpenAI TTS, Azure TTS, Google TTS.

What makes this work in real time is that TTS starts streaming back to the user as soon as the first tokens arrive from the LLM, rather than waiting for the full response. LiveKit Agents calls this “speech streaming” and it significantly reduces perceived latency.

A Minimal Agent

Install the framework and your chosen providers:

pip install livekit-agents livekit-plugins-openai livekit-plugins-deepgram livekit-plugins-elevenlabs

A basic agent looks like this:

from livekit.agents import AutoSubscribe, JobContext, WorkerOptions, cli, llm
from livekit.agents.voice_assistant import VoiceAssistant
from livekit.plugins import deepgram, elevenlabs, openai, silero

async def entrypoint(ctx: JobContext):
    initial_ctx = llm.ChatContext().append(
        role="system",
        text="You are a helpful voice assistant. Keep responses concise for voice."
    )

    await ctx.connect(auto_subscribe=AutoSubscribe.AUDIO_ONLY)

    assistant = VoiceAssistant(
        vad=silero.VAD.load(),
        stt=deepgram.STT(),
        llm=openai.LLM(model="gpt-4o-mini"),
        tts=elevenlabs.TTS(),
        chat_ctx=initial_ctx,
    )

    assistant.start(ctx.room)
    await assistant.say("Hello, how can I help you?", allow_interruptions=True)

if __name__ == "__main__":
    cli.run_app(WorkerOptions(entrypoint_fnc=entrypoint))

That is the entire voice agent. LiveKit handles audio input from whoever joins the room, routes it through Deepgram for transcription, sends the transcript to GPT-4o-mini, streams the response through ElevenLabs, and plays it back.

Interruption Handling

One of the harder problems in voice AI is barge-in: what happens when the user starts speaking while the agent is talking. The naive approach is to ignore it or cut off mid-word awkwardly.

LiveKit Agents handles this via Voice Activity Detection (VAD), typically using Silero VAD. When VAD detects speech while the agent is talking, the current TTS stream is cancelled, the audio buffer is cleared, and the new utterance starts processing. This produces natural-feeling interruptions rather than robotic turn-taking.

You can control interruption sensitivity:

assistant = VoiceAssistant(
    vad=silero.VAD.load(),
    allow_interruptions=True,
    interrupt_speech_duration=0.5,  # seconds of speech before interrupting
    interrupt_min_words=0,
)

Connecting from a Browser

LiveKit’s client SDKs are available for JavaScript, iOS, and Android. A browser-based connection looks like:

import { Room } from "livekit-client";

const room = new Room();
await room.connect(livekitServerUrl, userToken);
await room.localParticipant.setMicrophoneEnabled(true);

The token is a JWT you generate server-side with your LiveKit API credentials. The user’s browser connects to your LiveKit Server, the agent joins the same room, and audio flows bidirectionally over WebRTC.

When to Use LiveKit Agents

LiveKit Agents is the right choice when you’re building a real-time voice interface: customer service bots, voice-controlled dashboards, phone call agents (via SIP integration), or interview assistants.

For turn-based voice (press to talk, wait for response), simpler approaches using the OpenAI Audio API or ElevenLabs’ conversational AI product are lower overhead. LiveKit Agents earns its complexity when you need continuous, low-latency, interruption-aware voice that works at scale.

The framework has a hosted cloud option (LiveKit Cloud) and a self-hosted path using the open-source LiveKit Server. For production, LiveKit Cloud handles the media infrastructure; you just run your agent workers.

References