BroadChain News, April 24, 14:00. Large Language Models (LLMs) enter a "frozen" state after training is complete, and once deployed, they can only rely on external patches such as context windows and Retrieval-Augmented Generation (RAG) to function. Two partners at a16z point out that this is akin to the protagonist in the film "Memento": capable of retrieving information but unable to truly learn new knowledge. They systematically outline the cutting-edge research direction of "continual learning," analyzing the field from three dimensions: context, modules, and weight updates.
In-context learning (ICL) is effective, but it only applies to problems where answers or fragments already exist in the world. For tasks requiring genuine discovery (e.g., new mathematical proofs), adversarial scenarios (e.g., security red teaming), or tacit knowledge that is difficult to articulate, models need to directly write new experiences into their parameters after deployment. In-context learning is temporary; true learning requires compression.
Continual learning is not a new concept (traceable back to 1989), but a16z believes it is one of the most important directions in AI today. The explosive growth in model capabilities over the past two to three years has widened the gap between what models "know" and what they "can know." If models can learn their own memory architecture rather than relying on external tools, it may unlock entirely new dimensions of scaling.