Building voice AI that feels live: a hands-on guide to Higgs Realtime

Real-time voice AI is easy to demo. Building it well is much harder.

A good voice agent can’t simply wait for someone to finish speaking, turn the audio into text, generate an answer, and read it back. Real conversations don’t behave like a sequence of clean requests and responses. People pause. They interrupt. They change direction halfway through a sentence. They expect the system to remember what came before, use tools when necessary, and respond without making the conversation feel like a series of transactions.

That changes how developers need to think about building voice applications—so today we are releasing the Higgs Realtime API Tutorial, a build-it-yourself guide for developers who want to understand how real-time voice AI works and build voice applications with sub-second latency at a fraction of the usual cost. Rather than starting with a finished demo and hiding the complexity underneath it, the tutorial builds a working browser voice assistant one layer at a time.

Traditional AI applications have a familiar rhythm: request, inference, response. Real-time voice runs on a different model of the world.

The application is no longer simply calling a model. It is participating in a live conversation—while maintaining conversational state and keeping the experience responsive.

Higgs Realtime was designed for this environment: an audio-native speech-to-speech system that listens, reasons, calls tools, and responds in real time. It handles interruption and conversational turn-taking, maintains multi-turn state, supports more than 100 languages including code-switching, and can invoke tools as part of the conversation. And with sub-second speech-in to speech-out latency, the interaction stays close to the rhythm people expect from conversation.

But a capable real-time model is only one part of the system. Developers still need to build the application around it. That’s what this tutorial is about.

The tutorial assumes you know JavaScript or TypeScript and basic React. It does not assume you know audio programming or real-time protocols. Starting from zero, you progressively build a browser-based voice assistant:

By the end, you haven’t just copied a voice demo. You’ve built the pieces that make one work.

One of the biggest differences between chat and voice is interruption. In a text interface, the boundary between user and assistant is explicit: someone sends a message, then the other side responds. Conversation doesn’t have that boundary. A person may say:

The tutorial lets you experience this directly. You’ll stream microphone audio into Higgs Realtime, use server-side turn detection, receive generated audio incrementally, and handle interruptions as part of the normal application flow. This isn’t an enhancement added on top of voice AI. It is part of what makes voice interaction feel like conversation.

Natural conversation is useful. Natural conversation connected to real systems is much more useful. A customer-support agent needs to retrieve an account. A travel agent needs to search availability. A sales agent may need to look up a product, calculate a quote, or schedule an appointment.

That’s why tool calling is built into the tutorial rather than treated as an advanced topic. You’ll define a tool, expose it to Higgs Realtime, detect when the model calls it, execute it in your application, return the result, and let the model continue the spoken conversation. The result is the architectural pattern voice agents are built on:

Real-time applications can be surprisingly difficult to debug. Was the problem caused by the microphone? The audio encoding? The WebSocket connection? Turn detection? The model? Or the sequence of events coming back from the API? The tutorial builds two things that make the answer findable:

When we introduced Higgs Realtime, our goal was to make capable real-time voice AI practical to deploy: audio-native interaction, tool calling, interruption handling, multilingual conversation, sub-second responsiveness, and economics that work at scale. This tutorial addresses the other side of that goal—making real-time voice AI practical to build.

The repository intentionally keeps the surrounding application simple. There is no production database, no authentication system, no complex backend; even the search tool uses a small local dataset. That simplicity is deliberate: it lets you see the mechanics of real-time voice interaction without hiding them behind an application framework. Once those mechanics make sense, you can replace the tutorial components with your own tools, business logic, data, and workflows—customer support, sales, travel, scheduling, training, personal assistants, enterprise workflows. The conversational foundation is largely the same.

The entire tutorial is open source and available right now. The best way to understand real-time voice AI isn’t just to hear a demo.

We can’t wait to see what you build.