AI lets us build virtual worlds, full of exciting characters, ready to interact with each other and with human players. It also offers the opportunity to build responsive call centers that help customers with their queries, service requests or during their purchase. This requires five components: first, the ability to understand audio, second, the ability to reason about the request, third, the ability to communicate the answer in a natural manner, fourth, tools to orchestrate the entire process and lastly, the ability to align and tune models such that they behave in the intended manner.

Boson AI's Roleplay Technology allows for interactive storytelling and character simulation, blending advanced language models, voice generation and speech recognition (audio understanding). Let's review the tools needed to make models behave in a manner that delights and informs users.

Finetuning and Alignment

Pretrained foundation models for speech, voice or text are only half the story. After all, they won't know what is needed to create engaging content, to follow instructions or to create a meaningful dialogue. This is where finetuning and alignment comes in. We use a wide range of technologies to make the models work:

We designed algorithms for targeted content crawling to ensure that our models are competent when it comes to human interactions.
Not all conversations are created equal. In some cases we want the models to converse with users in a specific, positive manner. This is where semi-synthetic dialogue generation comes in handy. It ensures that we have plenty of positive examples to reinforce positive behavior.
Human feedback is the ultimate yardstick to measure whether a model works. This is where human annotators come in handy.
A good sales representative will adapt in real time to the needs and preferences of the customer. In the same manner, the model needs to learn what a user wants, ideally while a user is talking to the agent. We use LLMs to accomplish this goal.

Judger and AI Feedback

Human feedback is expensive, yet vital. This is supplemented with additional feedback, obtained by using effective prompting of a suitably configured general purpose model. To make the best out of human feedback and automatic annotation, we learn a high quality judger model, allowing us to assess large amounts of data efficiently.

Judgers can be used for fine-tuning large language and audio models
Prompt engineering is vital when designing judgers that are able to assess the quality of the components. It is often much easier (and more reliable) to distinguish between good and bad outputs, when compared to actually generating such outputs.
Reinforcement learning from human feedback is a mechanism to capturing annotations and to turn this into an objective function (that can be optimized). We use it (and related tools) in our model building.

Integration

Lastly, all these components need to be combined (audio understanding, audio generation and LLM) to produce an engaging and realistic experience. This is where Higgs Audio Understanding and Higgs Audio Generation models come in handy.

Higgs Audio Understanding can be used to recognize emotions and to annotate the transcripts (or to let the model decide on an appropriate course of action).
Emotional audio generation is just as vital. After all, it is very difficult to annotate text in emotionally competent and nuanced ways. Having a model that is naturally competent in terms of nuance, interruption, and emphasis and accent is highly relevant.
When designing dialogs that only capture one party at a time, a lot of the interplay between speakers is lost. The conversation becomes one of two interleaved monologues. Instead, we are able to generate natural sounding conversation.
While automation and competence in annotation is good, we still want flexibility to influence the outcomes of our model more directly. This is possible by providing our model with emotional and stylistic instructions for communication.

For a demo that integrates all these components check out our Magic Broom Store. Make sure to enter the backstage view that illustrates how our model is capable of understanding and logging the interaction flow. Please contact us for more details and to discuss how we can help you deploy rich interactive agents to serve your customers.

Roleplay and Agents

Related Technologies

Audio Understanding

Voice Generation