Recently, there have been significant discussions about scaling laws, particularly with the introduction of OpenAI’s new scaling law for o1 called test-time compute. The world’s loudest AI critic Gary Marcus, however, disagrees. He argues that scaling LLMs across different parameters is not the ultimate path to AGI.
“I don’t think LLMs are a good way to get there (AGI). They might be part of the answer, but I don’t think they are the whole answer,” Marcus said in an exclusive interview with AIM, stressing that LLMs are not “useless”. He also expressed optimism about AGI, describing it as a machine capable of approaching new problems with the flexibility and resourcefulness of a smart human being. “I think we’ll see it someday,” he further said.
Sharing his two cents on OpenAI’s test-time compute, Marcus suggested that he doesn’t believe it will be a viable solution in the long term. According to him, while scaling is necessary, the so-called “scaling laws” aren’t universal truths. “They’re not like gravity. They’re generalisations that work for a while. Like Moore’s law, which worked for a while but didn’t last forever,” he said.
Marcus further explained that scaling in this field has mostly involved adding more data to solve problems. “That worked to some degree for a while, but it never really solved hallucinations and stupid errors, and now we’re running out of fresh data to keep doing that,” he added.
While suggesting that OpenAI’s new method of scaling is worth exploring, Marcus pointed out that it only works in domains with a lot of synthetic data.
Gary’s Answer to AGI
Marcus, who has long been critical of the over-reliance on scaling deep learning models, reiterated his belief that the field must change course. “We spent the last four years pursuing mostly one hypothesis, which was scaling, and I think that hypothesis was a mistake,” he said.
According to Marcus, the future of AI should focus on neuro-symbolic systems, which combine the learning capabilities of deep learning with the structured, rule-based reasoning of symbolic AI.
In his 2020 paper ‘The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence’, Marcus outlined a four-step strategy for the AI community to follow: prioritising neuro-symbolic AI, building large-scale knowledge databases, improving reasoning techniques, and developing cognitive models. Despite the difficulty of these tasks, he argued that they are essential for progress. “I think that’s what we need to do now,” he said.
While Marcus critiqued companies like OpenAI for their focus on scaling models like GPT-4o, he applauded DeepMind, especially for its work on AlphaFold.
“DeepMind has done the most interesting work in the field,” he said, adding that AlphaFold, although not as general-purpose as GPT-4o, demonstrates the successful application of neuro-symbolic AI in the domain of protein folding.
Marcus also acknowledged that DeepMind is likely on a better path towards AGI compared to its competitors. He, however, indicated that no company has yet found the definitive route to AGI. “Of the major companies working on this, DeepMind is most likely to be on the correct path,” he said.
Marcus and Yann LeCun’s Debate
Recently, Marcus and Meta’s chief AI scientist Yann LeCun engaged in a conversation on Threads, debating who first predicted that LLMs wouldn’t lead to AGI. “Auto-regressive LLMs are hitting a performance ceiling. I’ve been predicting this before most people had even heard of LLMs,” claimed LeCun.
He added that he has always been heavily criticised for saying that LLMs were useful, but were an off-ramp on the road to human-level AI. “I’ve said that reaching human-level AI will require new architectures and new paradigms,” he further said.
Marcus told AIM that, back in 2019, he had predicted that language models wouldn’t lead to AGI. “When I originally said that, LeCun said, ‘You’re fighting a rear-guard action,’ and at the time, he was much more bullish on large language models,” he recalled.
“He’s trying to reinvent himself and pretend that he was always opposed to them. He was hostile to me when I first criticised them and was highly critical of my 2022 paper ‘Deep Learning is Hitting a Wall’,” Marcus said, adding, “He’s using the famous rhetorical technique of a straw man and misrepresenting my idea in order to make me sound like a fool.”
Marcus disclosed that his 2022 paper was focused on large language models. “It was about the limits of GPT-3 that were unlikely to be solved by scaling. I was always arguing that large language models were the best example of deep learning we had, but they were bound to encounter problems. Now those problems have surfaced, and he’s (Yann LeCun) trying to rewrite history and pretend he was the first person to see them,” he added.
Betting Big on Cognitive Models
Marcus stressed that the future of AI requires advances in cognitive models, a concept he has advocated for years. These models can reason and understand the world in a way that LLMs cannot.
As he described in his 2020 essay, these cognitive models would allow AI systems to represent individual objects and entities in a better way, differentiating between unique instances rather than treating everything as part of a generic category.
“I wrote about cognitive models in 2020, and he (Yann LeCun) has also been writing about them. He calls them world models, but it’s just different names for the same thing,” Marcus said.
Marcus believes that representing individuals separately from their kind is one of the things known to be necessary for artificial intelligence, while agreeing that the exact method to achieve this is unclear. He compared the current state of AI development to Leonardo da Vinci’s designs for a helicopter, which he described as an idea ahead of its time but not yet possible to implement due to a lack of supporting technology.
In his latest book ‘Taming Silicon Valley’, Marcus discusses how current AI systems, despite their impressive capabilities, are still far from achieving human-level intelligence. They are often brittle and prone to errors, especially when faced with unexpected situations.
Compute is Not the Answer
Marcus believes that the right answer to AI won’t require so much computing. Using his own children as an example, he said, “My children don’t have nearly as much compute as we have on this planet, and yet they are smarter than all the best machines that we have on the planet.”
Looking to the future, Marcus advised young AI researchers to avoid following the trend of working exclusively on LLMs. “The brightest students should try to find their own path. Large language models are not going to last forever. Someone who invents something really different could transform the world,” he concluded.
The post Why You Should Take Gary Marcus Seriously appeared first on Analytics India Magazine.
