Evolution Timeline
Paper added: Emergent Abilities Mirage
Are Emergent Abilities of Large Language Models a Mirage?
We show that emergent abilities may be artifacts of evaluation metrics. Continuous metrics reveal smooth improvements rather than sudden phase transitions.
Controversy: Does scaling continue to work?
Debate over whether recent models show diminishing returns from scale or if we need new architectures
Scaling Laws and the Limits of Transformer Language Models
We examine whether scaling laws continue to hold for modern large language models. Our analysis reveals potential saturation effects in certain capabilities while others continue to improve predictably with scale.
Paper added: Chinchilla Scaling Laws
Training Compute-Optimal Large Language Models
We investigate optimal model sizes and training dataset sizes. Current models are significantly undertrained, suggesting scaling should balance model size and training tokens.
Programme founded
Programme created to investigate limits and breakdown of neural scaling laws
Emergent Abilities of Large Language Models
We document dozens of examples where large language models display abilities not present in smaller models. These emergent abilities appear unpredictably as model scale increases, raising fundamental questions about scaling.