While scaling laws for Large Language Models (LLMs) traditionally focus on proxy metrics like pretraining loss, predicting downstream task performance has been considered unreliable. This paper challenges that view by proposing a direct framework to model the scaling of benchmark performance from the training budget. We find that for a fixed token-to-parameter ratio, a simple power law can accurately describe the scaling behavior of log accuracy on multiple popular downstream tasks. Our results show that the direct approach extrapolates better than the previously proposed two-stage procedure… Read More
Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
While scaling laws for Large Language Models (LLMs) traditionally focus on proxy metrics like pretraining loss, predicting downstream task performance has been considered unreliable. This paper challenges that view by proposing a direct framework to model the scaling of benchmark performance from the training budget. We find that for a fixed token-to-parameter ratio, a simple power law can accurately describe the scaling behavior of log accuracy on multiple popular downstream tasks. Our results show that the direct approach extrapolates better than the previously proposed two-stage procedure…
Related Posts
PREAMBLE: Private and Efficient Aggregation via Block Sparse Vectors
We revisit the problem of secure aggregation of high-dimensional vectors in a two-server system such…
Google’s year in review: 8 areas with research breakthroughs in 2025
This year saw new AI models, transformative products and new breakthroughs in science and robotics.
Neurons receive precisely tailored teaching signals as we learn
New work suggests the brain can deliver neuron-specific feedback during learning — resembling the error…