This paper was accepted at the Workshop on Navigating and Addressing Data Problems for Foundation Models at ICLR 2026.
Language models achieve impressive performance on a variety of knowledge, language, and reasoning tasks due to the scale and diversity of pretraining data available. The standard training recipe is a two-stage paradigm: pretraining first on the full corpus of data followed by specialization on a subset of high quality, specialized data from the full corpus. In the multi-domain setting, this involves continued pretraining of multiple models on each specialized domain, referred… Read More
Optimal Splitting of Language Models from Mixtures to Specialized Domains
This paper was accepted at the Workshop on Navigating and Addressing Data Problems for Foundation Models at ICLR 2026.
Language models achieve impressive performance on a variety of knowledge, language, and reasoning tasks due to the scale and diversity of pretraining data available. The standard training recipe is a two-stage paradigm: pretraining first on the full corpus of data followed by specialization on a subset of high quality, specialized data from the full corpus. In the multi-domain setting, this involves continued pretraining of multiple models on each specialized domain, referred…
Related Posts
How to Build a General-Purpose AI Agent in 131 Lines of Python
The following article originally appeared on Hugo Bowne-Anderson’s newsletter, Vanishing Gradients, and is being republished…
BERT Models and Its Variants
This article is divided into two parts; they are: • Architecture and Training of BERT…
ExpertLens: Activation Steering Features Are Highly Interpretable
This paper was accepted at the Workshop on Unifying Representations in Neural Models (UniReps) at…