When a Zero-Parameter Cache Overtakes a Transformer

A recent experiment highlights the performance crossover between a 1.43 million-parameter transformer and a simple zero-parameter document cache. By testing both systems on documents of varying lengths, researchers found that while the transformer dominates at short context windows (around 60 tokens), its performance plateaus. Conversely, the cache mechanism scales effectively with document length, outperforming the neural network as the text grows to 250 and 1000 tokens. The study demonstrates that the transformer's contribution becomes negligible at longer lengths, with the cache providing a 43% relative margin advantage at 1000 tokens. The findings suggest that document length is a critical resource that favors simple statistical counting over fixed-window neural models. The author emphasizes that comparing systems without identifying the specific resource axes they depend on can lead to misleading conclusions, as performance rankings can flip entirely depending on the operating context.