The human brain has approximately 86 billion neurons. And it’s quite a complex task to mimic the human brain by building neural networks such as RNNs, CNNs, or Transformers since scaling up the network to that level is not feasible. This also comes along with the problem of collecting huge amounts of labelled training data.
Researchers at MIT’s Computer Science and AI Laboratory (CSAIL) have come up with a new technique to solve these problems; it’s called liquid neural networks or LNNs. These are time-continuous RNNs that process data sequentially, keep the memory of past inputs, and adjust its behaviours based on new inputs.
Daniela Rus, the director of CSAIL, recently demonstrated the use of these LNNs in running a self-driving autonomous vehicle. She demonstrated that the researchers were able to build a fairly working self-driving car trained within the city using 100,000 artificial neurons. The problem, she pointed out, was that the cameras were picking up too much noise in its attention map, which is not required.
Leveraging just 19 artificial neurons through LNNs, the attention map got clearer and became more focused on the path it had to take. This was done by converting these neurons into decision trees, for letting the machine decide which path to take by itself, which is kind of a temporal neural network.
Dropping intelligence?
Reducing the number of neurons actually sounds like reducing the intelligence of these models. But Rus also demonstrated how this technique can be leveraged in various tasks such as embodied robotics as she claims that LNNs are causal — they understand cause and effect — something which has been touted as missing within Transformer-based neural networks.
While the human brain has billions of neurons, LNNs can achieve similar functionality with a significantly smaller number of artificial neurons. It is similar to Yann LeCun’s idea of achieving dog and cat-like intelligence before human-like intelligence. This compactness also offers advantages in terms of computational efficiency and scalability.
Researchers of LNNs highlight how by applying liquid neural networks to robotics, there is great potential for enhancing the reliability of autonomous navigation systems. This advancement would have numerous applications, such as improving search and rescue operations, facilitating wildlife monitoring efforts, and enhancing delivery services, among other possibilities.
As urban areas become increasingly crowded, the concept of smart mobility becomes crucial, and according to Ramin Hasani, the pioneer of LNNs, these offer a significant advantage due to their compact size while also bringing down the cost of training large models.
How does this work?
According to Hasani, the inspiration to build these systems came from a 1mm long worm called nematode C Elegans. The neural system of these worms consists of just 302 neurons. Hasani said that even with just less number of neurons, these worms are able to complete tasks that can have unexpectedly complex dynamics.
Hasani explains how typical machine-learning systems have neural networks that learn solely during the training phase, after which their parameters remain unchanged. However, LNNs possess the remarkable ability to continue learning even after their initial training. They achieve this by employing “liquid” algorithms that dynamically adapt to new information, much like the way neurons and synapses interact in living organisms’ brains.
For example, in an experiment performed on a drone, using these liquid networks, the drone was able to track the object better than other deep neural networks. This is because the neurons are less, it is able to stay focused on the task, and not depend on the context around.
A possible limitation with LNNs in self-driving vehicles is that since it is not accounting for all the surroundings and just focused on where it wants to go, it is difficult to avoid collisions with objects in the way. The research is being done on empty roads without any obstacles. This area, according to Hasani, is still being explored and improved.
LNNs for LLMs?
LNNs possibly open a whole new arena for research in AI applications. When it comes to LLMs, the field is built on Transformers and thus on the number of parameters. With liquid networks, the parameters change over time, based on the results of a nested set of differential equations, which essentially means it understands new tasks by itself, and thus does not require vast amounts of training.
LLMs, such as GPT-3 and GPT-4, have already demonstrated impressive language generation capabilities. However, they are limited by their static nature, where the parameters are fixed after the initial training. LNNs, on the other hand, possess the ability to continue learning and adapting to new information, similar to how neurons and synapses interact in living organisms’ brains.
By integrating LNNs into LLMs, these models can continuously update their parameters and adapt to evolving language patterns and contexts. This dynamic nature allows LLMs to improve their performance over time and respond more effectively to changing user needs and preferences. On the other hand, since models built on these architectures won’t have large amounts of information, it is highly unlikely to expect intelligent behaviour that we see hints of within current LLMs. More exploration of the network would definitely open up more avenues for LNNs.
Despite the challenges, researchers are striving to optimise LNNs by exploring ways to minimise the number of neurons required for specific tasks. Additionally, the limited literature on LNN implementation, application, and benefits makes understanding their potential and limitations challenging, unlike more widely recognised neural network architectures like CNNs, RNNs, or transformer models.
The post Can LNNs Replace Transformers? appeared first on Analytics India Magazine.
Generative AI is Driving Investment in the Cloud
Cloud Providers Seek to Keep Up with AI Demand
“Unlike general-purpose cloud offerings that run multiple workloads in parallel, HPE GreenLake for LLMs runs on an AI-native architecture uniquely designed to run a single large-scale AI training and simulation workload, and at full computing capacity,” the company said in a release. “The offering will support AI and HPC jobs on hundreds or thousands of CPUs or GPUs at once. This capability is significantly more effective, reliable, and efficient to train AI and create more accurate models, allowing enterprises to speed up their journey from POC to production to solve problems faster.”
