Can a small LLM be taught to learn without increasing weights? Part 2: Why retraining hit a ceiling

The author continues a series of experiments with small language models aimed at developing the LANN-4 architecture. The core concept involves training compact neural networks capable of 'lifelong learning' without increasing parameter counts, utilizing external memory and periodic fine-tuning. During the research, the author encountered a significant issue: retraining the same model on new data is only effective during the first cycle, after which progress plateaus. A cumulative effect from multiple training iterations was not achieved. The author shares both successes and failures, noting that these models serve as tools for testing hypotheses rather than final products. Currently, the researcher is seeking community input to identify potential flaws in the architecture before committing to resource-intensive computational tests. The article invites readers to discuss how to overcome the learning ceiling in small LLMs.