Why Do Asymmetric Power Laws Help Reasoning?

Links: paper, PDF, and Eric Michaud's quanta essay.

How to make LLM learn to reason efficiently?

Suppose you are asked to train an LLM to solve multi-step arithmetic (e.g. $a+b\times c -d\div e=?$) with as few training steps as possible. How should you design your training distribution?

Let’s say you are only allowed to change the training distribution of the numbers, which is the most basic “knowledge” of the arithmetic. One option is the standard uniform distribution, sampling every number with roughly equal probability. Another option is an asymmetric power-law distribution: rank all the numbers randomly, assigning probabilities according to power law, and sample accordingly. Which one will you choose?

One may choose uniform distribution as it seems to be a fix to the asymmetry of the power law, making up for the long-tail effect. But, when the numbers in the problems are sampled from a power law distribution, the model learns much faster than the uniform one!

We then checked whether the pattern survives beyond the first arithmetic example:

• State tracking, where the model composes a sequence of group actions.
• Multi-hop QA, where the model follows chains of synthetic natural language facts.
• Synthetic GSM-style math, where answers depend on a graph of arithmetic operations (written in natural language).

The same surprise keeps showing up: power law consistently wins.

So why does power law help reasoning so much? The post will explain the interesting phenomenon in three steps:

1. Rethink the intuition. Uniform distribution is the right instinct for one-hop memorization, but composition changes the bottleneck.
2. Build a toy theory. A minimalist composition model shows how uniform distribution can create a symmetric, nearly flat initial landscape, while power law creates a useful descent direction.
3. Check the mechanism. Transformer experiments show the same stage-wise picture: escape first, learn head skills next, and use them as stepping stones for scarce tail skills.

Why power law v.s. uniform? Why does uniform feel right?

First, why do we consider power law distribution as the competitor of uniform? The motivation is straightforward: power laws are one of the most natural shapes in language. At the word level, Zipf’s law says that a few words appear constantly while most words are rare. More generally, the “items” may not be words at all: they may be latent skills or knowledge pieces whose occurrence frequencies follow a power-law distribution, $p_i\propto i^{-\alpha}$. This viewpoint can also explain why loss may decrease smoothly as a power law: many discrete skill-learning events get averaged together as the model reaches farther into the tail (Michaud et al.).

But this picture of power law also exposes a problem: the long tail effect. Under a power-law distribution, rare skills are observed only when the dataset becomes very large, while the most frequent skills may be sampled far beyond what is necessary for learning them.

If the goal is to learn atomic knowledge or individual skills faster, the obvious data-curation move is to flatten the distribution: up-weight low-frequency skills, down-weight high-frequency ones, and move closer to a uniform distribution over skills. Given enough knowledge about the data and enough budget for curation, this sounds like the ideal long-tail fix. In this case, shouldn’t a more uniform distribution help?

Sanity check: one-hop memorization

We start with a task that mainly requires memorizing atomic knowledge: one-hop question answering (QA). Each example contains a single fact of the form “entity – relation –> answer.” The question asks for that answer directly. There is no intermediate entity to carry, no second relation to apply, and no hidden chain to execute. This is memorization in the cleanest sense.

In this setting, if a relation is rare under a power-law distribution, the model simply sees fewer direct examples of that relation. Since the test set asks one-hop questions across all relations, the bottleneck is coverage of long-tail relation skills. Therefore, uniform distribution should help on this task.

The experiment behaves exactly this way. We randomly rank relations, train one model with uniformly sampled relations and another with power-law sampled relations, and evaluate exact match on one-hop questions. Uniform distribution wins this race.

Overall, if the task were only to store isolated facts, “use a power-law distribution” would be a strange recommendation. High-frequency skills are already frequent; scarce long-tail skills need data. Shifting towards a uniform distribution gives every skill a fairer chance.

What if the task is multi-hop?

However, natural language tasks are not just about single-hop memorization. Reasoning tasks, for example, often require combining multiple thinking steps or pieces of atomic knowledge to solve a problem. What happens when the task becomes more reasoning-heavy?

Remark. This framing is close to work on knowledge manipulation, implicit multi-hop reasoning, state tracking, parallelism, and chain of thought.

Now we change only the task to a more reasoning-like, multi-hop QA task. Instead of asking for one relation, the questions ask for a chain of relations. The model must apply the relations one by one and compose all of them. For example:

When memorizing atomic facts, each relation can be learned almost independently. In contrast, in a two-hop QA problem, the model has to retrieve the first fact, use its answer as the input to the second fact, and only then produce the final answer.

Intuitively, if a chain uses $k$ skills with frequencies roughly $p_1,\ldots,p_k$, the full combination is much rarer than any one skill alone. From a pure coverage view, power-law distribution should look especially bad here: it undersamples scarce long-tail skills, and rare chains involve rare pieces. The naive prediction makes the uniform distribution even more tempting. Surprisingly, the experiment goes the other way instead: power-law distribution exhibits a clear gain in training speed. We evaluate on a uniform test distribution, so the gain is not coming from matching the test distribution.

This raises the main question: what changes when a model has to compose skills rather than recall them one at a time? Why does a power-law distribution help language models learn reasoning?

A minimalist model of skill composition

To understand why only a switch of training distribution helps in implicit compositional reasoning tasks, we look for a controlled setting of skill composition. Transformer experiments mix many effects: representation learning, attention, finite samples, and hidden multi-step operations. To isolate the role of the data distribution, consider a toy task called $k$-multiplicative composition.

There are $d$ atomic skills. A skill can be read as a relation in multi-hop QA or a basic operation in arithmetic. In the toy task, skill $i$ has a hidden sign $w_i^\star\in{-1,+1}$. A training example samples $k$ skill indices $I_1,\ldots,I_k$ from a distribution $p$ and asks the model to predict the product of their hidden signs:

The model has one learnable parameter $w_i$ for each skill and predicts the same kind of product:

So the task is simple but still compositional: the model must learn the hidden value of each skill and use several learned skills together. The only thing we change is the sampling distribution $p$: uniform over skills, or Zipf / power law with $p_j\propto j^{-\alpha}$.

To analyze gradient descent, we consider the matched learner $f_w(X)=\prod_{t=1}^k w_{I_t}$ and optimize the population square loss $\mathcal L(w)=\frac12\mathbb E_X[(f_w(X)-f_{w^\star}(X))^2]$.

The two quantities that control the dynamics are the weighted inner product and weighted norm

where $A(t)$ is the similarity between the current model and the ground truth under the training distribution, while $B(t)$ is the corresponding weighted norm. The key object is the population gradient. A short calculation gives

Equivalently, the expected gradient descent update for coordinate $j$ is

This equation is the mechanism in miniature. The factor $p_j$ is the local sampling frequency of skill $j$. The factor $A(t)^{k-1}$ is the global composition signal: it is large only when the current model already has some weighted similarity with the hidden target. Near initialization, the key question is whether the signal term $kp_jA(t)^{k-1}w_j^\star$ is large enough for gradient descent to escape the flat initial region.

Uniform distribution induces hardness

The intuition behind the failure of uniform distribution is, surprisingly, symmetry. Under uniform inputs, the possible hidden targets are too balanced, and correlation-based gradient queries reveal very little about which hidden vector is correct. The specific issue can be seen in the gradient dynamics. If $w_i(0)\sim\mathcal N(0,r^2)$, then

Under uniform distribution, every skill has probability $1/d$, so $\sum_i p_i^2=1/d$. The initial similarity is an average over all $d$ random coordinates, and its typical size is about $\lvert A(w_0)\rvert\approx r/\sqrt d$. Composition raises this small quantity to the power $k-1$. The useful initial signal therefore behaves like $(r/\sqrt d)^{k-1}$, exponentially slowing down the training.

The results thus indicate that the more complex the task is (with an increasing number $k$), the slower the training is. A larger number of skills (a larger $d$) will make the issue even worse.

Punchline. Uniform distribution fixes long-tail exposure, but for composition it can also create a symmetric hard instance: the initial alignment is tiny, the useful gradient is tiny, and gradient descent sees an almost flat landscape.

Remark. The rigorous theoretical result uses another tool called correlational statistical query (CSQ) lower bound. Under a uniform input distribution, learning the compositional task requires either very accurate gradient queries or a large amount of data/compute.

Power-law distribution re-enables efficient training

What about the more natural power-law distribution? Can it enable the learning of the simple model? The positive theorem says yes: online minibatch gradient descent can learn the same $k$-multiplicative composition task efficiently under a Zipf distribution.

Recall the population gradient above. Under a power-law distribution, the frequent skills occur with constant probability. Equivalently, $\sum_i p_i^2$ does not shrink like $1/d$. Therefore the initial weighted similarity is not washed out by averaging over all $d$ skills: $\lvert A(0)\rvert\approx \Theta(r)$. At the same time, for small constant initialization scale, $B(0)\approx \Theta(r^2)$. Thus the signal term dominates the first update:

Or, in vector form near initialization,

This is the beneficial asymmetry. For high-frequency skills, $p_j$ is large enough that the initial gradient is no longer tiny. The loss landscape near initialization now has a clearer descent direction toward the lower-loss region. This is Stage I of training: power law helps the model escape the flatness. After this, the optimization becomes much easier and goes through two more stages.

Stage II: high-frequency skills help the tail. Though the initial gradient signal is large enough to escape the flat region, the hidden scalars behind the skills are not learned simultaneously. Recall the coordinate update:

The head skills have large sampling probabilities. For a constant-rank skill $i=O(1)$ under a Zipf law with $\alpha>1$, we have $p_i=\Theta(1)$. These skills therefore grow first from the initialization scale $r$ to a large constant. Once enough head skills are aligned, the weighted similarity

increases from the initialization scale $O(r)$ to $O(1)$. This matters because every coordinate update contains the same global factor $A(t)^{k-1}$. For a tail skill $j=\Omega(d)$, the local factor $p_j$ is still small, but the signal term

is now much larger than it was near initialization. In this sense, the head skills act as a stepping stone: power law first learns high-frequency skills, and those learned skills increase the gradient signal for scarce long-tail skills.

Stage III: the long-tail drawback returns. Once all hidden scalars have non-trivial accuracy, training enters the convergence phase. At this point, the original long-tail intuition finally comes back. Skills with large rank $j=\Omega(d)$ have small sampling probability, roughly $p_j=O(d^{-\alpha})$, so they are updated much less often. Even though the model has already found a useful compositional direction, final convergence on the tail is slowed by the low probability of sampling tail skills.

This is why the result is not “power law makes the tail easy.” The tail is still hard at the end. The advantage is that the first two stages let the model start composing before the usual long-tail bottleneck dominates.

Punchline. Power law helps not by showing the tail more often, but by breaking the symmetry first. The head creates a visible descent direction; only after that can head skills become stepping stones for the tail.

The transformer check: state tracking

To test the generality of the theory prediction, the next step is to check whether the same mechanism appears in transformers. As a standard testbed, we consider the $S_5$ state tracking task, a synthetic composition task related to Allen-Zhu’s DePO/canon-layer setup and Merrill and Sabharwal’s transformer limits work (parallelism, chain of thought).

In this task, the input is a sequence of group elements $g_1,g_2,\ldots,g_k\in S_5$, and the target is their composition $g_1\circ g_2\circ\cdots\circ g_k$ without chain-of-thought. The skills are the permutations themselves. The model cannot solve the task by recognizing one update rule in isolation; it has to compose the sequence.

This task is known to be hard under uniform training distribution without intermediate supervision, even though there exists an efficient transformer construction for the computation. Simply switching the skill distribution to power law enables the same model to learn the task efficiently, without curriculum or intermediate thinking traces.

Stage I: power law enables escaping from the flat region. To visualize the landscape, take the training trajectories of the $S_5$ state tracking task under both uniform and power-law distributions. Compute the top two PCA directions from consecutive checkpoint differences and plot the loss landscape together with the trajectory. The initial region under uniform distribution is much flatter; the power-law distribution creates a clearer descent direction to the lower-loss region.

Stage II and III: head-to-tail learning, then long-tail convergence. Figure 8 checks the remaining stages above. The permutations are separated by rank into bins. Once the head bin starts to learn, the expected gradient norm on samples requiring a tail permutation becomes much larger when the other input permutations come from the learned head bin. This is the empirical counterpart of increasing $A(t)$ in Stage II. Later, the tail bins still converge more slowly, matching the Stage III long-tail bottleneck.

So the state-tracking experiment mirrors the theory: power-law distribution first improves the initial landscape, then creates an implicit curriculum through high-frequency skills, and finally faces the ordinary long-tail drawback.

Back to reasoning tasks

Finally, we can ask whether this understanding generalizes beyond the minimalist model and state tracking. Consider two synthetic natural-language reasoning tasks: multi-hop question answering and synthetic grade-school math. Only the training distribution is changed; the test sets are sampled uniformly over skills.

The first setting is multi-hop QA we mentioned earlier. The data is based on synthetic facts over relations between individuals, which can be viewed as a dependency graph:

and questions formed by chaining relations:

Each relation is treated as an atomic skill, and each hop in the question corresponds to one relation. The model has to answer the multi-hop query directly, without explicit chain-of-thought supervision. We vary the number of people (entity $\lvert E\rvert$) and the number of different kinds of relations ($\lvert R\rvert$) to control the hardness of the task.

The second setting is synthetic GSM-style arithmetic, following the spirit of controlled math-reasoning generators in Physics of Language Models: Part 2.1 and GSM-Infinite. These problems are generated from layered dependency graphs and natural-language templates. The answer requires composing several arithmetic operations, so each problem can be seen as a composition of basic operations on the dependency graph.

Across these tasks, the pattern is consistent with the theory. Power-law training learns faster than uniform training even though evaluation is uniform. Multi-hop QA also shows a similar stage-wise mechanism and a steeper loss landscape, which generalizes the mechanistic findings from the $S_5$ setting. Synthetic GSM-style arithmetic shows that the advantage is not limited to relation chaining.

What this suggests in practice

How do we apply the analysis to practice? A natural next step is to tune the shape of the real-world skill distribution, and try to understand if the distribution itself can help the learning of reasoning. One conjecture is that the power law already present in natural language helps LLMs learn essential reasoning circuits. A coarser-grained and more conceptual framework, such as Skill-Mix, Instruct-SkillMix, and related work on learning skill composition from examples, can be a starting point for asking this question at the level of real language skills rather than toy coordinates.

This also suggests a different way to think about curricula and synthetic data in agentic tasks. This post only studies the simplest chain-like composition. Agentic tasks often have a richer compositional graph, involving tool calls, branching decisions, memory updates, verification steps, and recovery from failed actions. A useful future direction is to ask whether changing the distribution over these latent skills and subgraphs can make agent training easier: keep enough high-frequency scaffolding skills to create a useful optimization path, while deliberately sampling rare but important tail skills once the model has enough compositional structure to benefit from them.

Conclusion

The main lesson is not that power-law data is always better, or that uniform data is always worse. It is more specific: when a task requires implicit composition, the training distribution changes not only which skills are seen, but also what gradient descent sees at the beginning of training. Uniform sampling can be better for exposing rare facts, while power-law sampling can make the first useful compositional direction easier to find.

References

• Zixuan Wang, Xingyu Dang, Jason D. Lee, and Kaifeng Lyu. The Power of Power Law: Asymmetry Enables Compositional Reasoning. arXiv, 2026.
• Eric J. Michaud. On neural scaling and the quanta hypothesis. 2026.
• Eric J. Michaud, Ziming Liu, Uzay Girit, and Max Tegmark. The Quantization Model of Neural Scaling. NeurIPS, 2023.
• Zeyuan Allen-Zhu. Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers. SSRN, 2025.
• Tian Ye, Zicheng Xu, Yuanzhi Li, and Zeyuan Allen-Zhu. Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process. arXiv, 2024.
• Zeyuan Allen-Zhu and Yuanzhi Li. Physics of Language Models: Part 3.2, Knowledge Manipulation. arXiv, 2023.
• Yang Zhou, Hongyi Liu, Zhuoming Chen, Yuandong Tian, and Beidi Chen. GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?. arXiv, 2025.
• Dingli Yu, Simran Kaur, Arushi Gupta, Jonah Brown-Cohen, Anirudh Goyal, and Sanjeev Arora. Skill-Mix: a Flexible and Expandable Family of Evaluations for AI models. arXiv, 2023.
• Simran Kaur, Simon Park, Anirudh Goyal, and Sanjeev Arora. Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning. arXiv, 2024.
• Haoyu Zhao, Simran Kaur, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. Can Models Learn Skill Composition from Examples?. arXiv, 2024.
• William Merrill and Ashish Sabharwal. The Parallelism Tradeoff: Limitations of Log-Precision Transformers. TACL, 2023.
• William Merrill and Ashish Sabharwal. The Expressive Power of Transformers with Chain of Thought. arXiv, 2023.
• Nouha Dziri et al. Faith and Fate: Limits of Transformers on Compositionality. arXiv, 2023.
• Yuekun Yao, Yupei Du, Dawei Zhu, Michael Hahn, and Alexander Koller. Language Models Can Learn Implicit Multi-Hop Reasoning, but Only if They Have Lots of Training Data. arXiv, 2025.
• Sanjeev Arora and Anirudh Goyal. A Theory for Emergence of Complex Skills in Language Models. arXiv, 2023.
• Michael Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM, 1998.
• Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. Curriculum Learning. ICML, 2009.
• Aaron Clauset, Cosma Rohilla Shalizi, and M. E. J. Newman. Power-law distributions in empirical data. SIAM Review, 2009.