BroadChain News, April 25, 13:02 - In the film *Memento*, the protagonist suffers from brain damage that prevents him from forming new memories, relying only on tattoos and Polaroid photos to piece together reality. Large language models (LLMs) face a similar dilemma: after training, vast amounts of knowledge are frozen in parameters, unable to update themselves based on new experiences. To compensate for this flaw, developers have built a "scaffold" for them—chat history serves as short-term notes, retrieval systems act as external notebooks, and system prompts function like tattoos. But the model itself never truly internalizes this new information.
An increasing number of researchers believe that in-context learning (ICL) has fundamental limitations. It can only solve problems where answers already exist somewhere in the world, but for tasks requiring genuine discovery (e.g., new mathematical proofs), adversarial scenarios (e.g., security attacks and defenses), or tacit knowledge that is difficult to articulate, models must be able to directly integrate new knowledge and experience into their parameters after deployment. In-context learning is temporary; true learning requires compression.
This research area is known as "continual learning." Although the concept is not new (traceable back to a 1989 paper), a16z crypto believes it is one of the most important directions in AI research today. The explosive growth in model capabilities over the past two to three years has made the gap between what a model "knows" and what it "can know" increasingly apparent. This article aims to share insights from top researchers in this field, clarify different paths of continual learning, and promote the implementation of this topic in the entrepreneurial ecosystem.
Before arguing for parameter learning (i.e., updating model weights), it must be acknowledged that in-context learning is effective, and there are strong reasons to believe it will continue to dominate. The essence of a Transformer is a conditional token predictor based on sequences. Given the right sequence, astonishingly rich behaviors can be achieved without touching the weights. Cursor's article on scaling autonomous coding agents is a case in point: the model weights are fixed, and what truly drives the system is the careful orchestration of context. OpenClaw is another example, elevating the "shell design" of agents into an independent discipline.
When prompt engineering first emerged, many researchers questioned whether "mere prompts" could serve as a legitimate interface. But this is a native product of the Transformer architecture, requiring no retraining and automatically enhancing with model upgrades. The stronger the model, the stronger the prompts. However, the goal of continual learning is to enable models to learn their own memory architecture, rather than relying on external custom tools. If achieved, it could unlock entirely new dimensions of scaling.