Meta focuses on AI models for mobile devices – Computerworld

Facebook parent company Meta is working on developing a new mobile-compatible small language model (SLM) that aims to run apps on devices while reducing power consumption during model inference tasks, according to a paper published by researchers at the company.

To set the context, large language models (LLMs) have many more parameters. For example, Mistral-22B has 22 billion parameters, while GPT-4 has 1.76 trillion parameters. In contrast, smaller language models have relatively fewer parameters, such as Microsoft’s Phi-3 family of SLMs, which have various versions starting at 3.8 billion parameters.

The parameter helps the LLM decide between different answers it can give to queries – the greater the number of parameters, the greater the need for a larger computing infrastructure.

However, the Meta researchers believe it is possible to develop efficient SLM models with fewer than a billion parameters, which would enable the adoption of generative AI across a variety of use cases involving mobile devices, which have relatively less computing infrastructure than a server or rack.

As we read in the paper, the researchers conducted experiments with models of different architectures, with 125 million and 350 million parameters, and found that smaller models, in which depth is more important than width, improved model performance.

“Contrary to the common belief that data and the number of parameters play a key role in determining model quality, our study highlights the importance of model architecture for subbillion-scale LLM models,” the researchers wrote.

“By leveraging deep and thin architectures, combined with shared embedding and attention mechanisms for group queries, we build a robust core network, denoted MobileLLM, that achieves a remarkable 2.7%/4.3% accuracy improvement over previous state-of-the-art 125M/350M models,” they added.

The 125 and 350 million models, dubbed MobileLLM, were as effective as large language models like Llama 2 in handling chat and several API call tasks, the researchers said, highlighting the capabilities of small models for common on-device use cases. Although MobileLLM is not available in any of Meta’s products for public use, the researchers made the code and data for the experiment available with the paper.