One of the first pre-processing steps for constructing web-scale LLM pretraining datasets involves extracting text from HTML. Despite the immense diversity of web content, existing open-source datasets predominantly apply a single fixed extractor to all webpages. In this work, we investigate whether this practice leads to suboptimal coverage and utilization of Internet data. We first show that while different extractors may lead to similar model performance on standard language understanding tasks, the pages surviving a fixed filtering pipeline can differ substantially. This suggests a simple… Read More
Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining
One of the first pre-processing steps for constructing web-scale LLM pretraining datasets involves extracting text from HTML. Despite the immense diversity of web content, existing open-source datasets predominantly apply a single fixed extractor to all webpages. In this work, we investigate whether this practice leads to suboptimal coverage and utilization of Internet data. We first show that while different extractors may lead to similar model performance on standard language understanding tasks, the pages surviving a fixed filtering pipeline can differ substantially. This suggests a simple…
Related Posts
Gradient Canvas: Celebrating over a decade of artistic collaborations with AI
Gradient Canvas is a new art exhibition celebrating a decade of creative collaborations between artists…
The Complete Guide to Model Context Protocol
Language models can generate text and reason impressively, yet they remain isolated by default.
Use Circle to Search and Google Lens to spot scam messages.
You can now use Circle to Search and Google Lens to detect scammy messages you…