Foundation models have become the backbone of modern generative AI. In particular, Largue Language Models (LLMs) have opened the door to many surprising abilities, from reasoning to understanding to translation and to the dedicated creation of content. Pretraining such models is an arduous task, requiring large amounts of compute, data and time. Over recent years, the proliferation of freely available high quality models such as Llama or Qwen have enabled an entire ecosystem of AI applications that can be built at an acceptable cost.

This has shifted the emphasis onto post-training, i.e. the science and art of taking pretrained models and adjusting them further to make them fit for a specific purpose. Boson AI has extensive expertise in training and aligning models for roleplay, instruction following and workflows. To succeed, one needs to master a number of skills, ranging from the collection of sufficient amounts of fine-tuning and preference-tuning data, to algorithms serving rapidly updating models, to strategies for reinforcement learning.

Datasets

Just like with pretraining, the selection of suitable datasets is key. For instance, to do well in roleplay, it matters to obtain data that addresses the corresponding aspects, e.g. from public game chat, storytelling and interaction data. In this sense, we need to obtain data to design a suitable curriculum for the learning algorithm. We have significant expertise in identifying, crawling and processing such data.

Task Specific Finetuning

Having a model reproduce data is nice but not necessarily useful in its own right. This is particularly true when it comes to expensive (and often rare) task specific data. For instance, if we were to train a model to do well in addition, we might not want to weigh all parts of 'The answer is 42' equally. In particular, the only part that really matters is the last token (42), rather than the rest. We would thus design tasks that emphasize the last piece of the answer.

While somewhat contrived, it is clear that problem-specific objectives are useful in getting the most value from expensive and rare data. This is particularly true, when such data arises from a business process and thus cannot be increased, even if it were desirable to do so.

Serving for Training

An often overlooked aspect of fine-tuning and aligning models is that we need to continuously serve the to-be-trained model. This is necessary as we need to draw samples (i.e. observe) the new model to determine which action to take next. Such a task can signifcantly increase the memory, computation and coordination footprint for training, particularly in the case of very large models: request rates need to be matched, parameters to be updated, and we need to maintain a replay buffer to make use of data, once it's been generated.

Our team can help you implement effective algorithms for modeling and finetuning LLMs. This can be significantly more cost effective than training models from scratch. Contact us today to discuss your specific needs and how our expertise can benefit your projects.

Language

Related Technologies

Training

Data