Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A popular method is Classifier-based Quality Filtering (CQF), which trains a binary classifier to distinguish between pretraining data and a small, high-quality set. It assigns each pretraining document a quality score defined as the classifier’s score and retains only the top-scoring ones. We provide an in-depth analysis of CQF. We show that while CQF improves downstream task performance, it does not necessarily enhance language modeling on the high-quality… Read More
The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining
Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A popular method is Classifier-based Quality Filtering (CQF), which trains a binary classifier to distinguish between pretraining data and a small, high-quality set. It assigns each pretraining document a quality score defined as the classifier’s score and retains only the top-scoring ones. We provide an in-depth analysis of CQF. We show that while CQF improves downstream task performance, it does not necessarily enhance language modeling on the high-quality…
Related Posts
The Google guide for holiday help
Learn more about using Google products like Gemini, Search, Shopping, Pixel and more over the…
We’re publishing an AI playbook to help others with sustainability reporting.
We’re sharing a practical playbook to help organizations streamline and enhance sustainability reporting with AI.Corporate…
CPEP: Contrastive Pose-EMG Pre-training Enhances Gesture Generalization on EMG Signals
This paper was accepted at the Foundation Models for the Brain and Body Workshop at…