Neural network by recipe: calculating weights instead of storing matrices

This Habr article explores alternative methods for storing neural network parameters. Instead of the traditional approach of storing massive weight matrices, the authors propose using 'recipes'—algorithmic ways to generate weights on the fly. The piece examines various approaches, ranging from using small auxiliary generator networks to mathematical formulas and fractal structures. The primary goal of such research is to radically reduce the memory footprint required for model operation, which is critical for running modern LLMs on resource-constrained devices. The author analyzes the effectiveness of these methods, weighing the gains in compactness against the computational costs of weight generation. Ultimately, the article raises the question of how close modern language models are to becoming runnable on any consumer device, much like classic games were in the past.