Just Train More: Measuring the Exchange Rate

A recent technical analysis explores the relationship between training data volume and transformer model performance compared to simple count-based caches. The study investigates the common argument that increasing training data will eventually allow small transformer models to outperform zero-parameter cache mechanisms in document completion tasks. By evaluating checkpoints at 500K, 2M, and 8M tokens, the author calculates an 'exchange rate' between training data and context window access. The findings suggest that while increasing training data does improve performance, the crossover point where a transformer surpasses a simple cache moves slowly. Specifically, the research indicates that buying document access via a larger context window is significantly more cost-effective than scaling training data. The author concludes that at fixed short contexts, transformers struggle to compete with simple, zero-parameter mechanisms, suggesting that model architecture and context window size remain critical, often confounded variables in current AI research.