The Power of Power Law: Asymmetry Enables Compositional Reasoning

Zixuan Wang, Xingyu Dang, Jason D. Lee, Kaifeng Lyu

Abstract: Natural language data follows power-law frequency patterns, where many skills and facts appear rarely. This work studies why such skewed data can help compositional reasoning: across state-tracking and multi-step arithmetic tasks, power-law training distributions outperform uniform ones. We introduce a minimalist skill-composition model and show that power-law sampling can reduce the data needed for learning by creating a useful asymmetry in the loss landscape, helping models first master frequent compositions and then transfer that progress to rare long-tail skills.