Feature: Qualcomm makes the case for …

Durga Malladi, Vice President and General Manager, Technology Planning and Edge Solutions, Qualcomm Technologies (shown in the photo), made a compelling case for putting AI on devices rather than in the cloud during a June analyst and media workshop, arguing that the current focus on the technology is not another false signal.

As a chip company, it’s no surprise that Qualcomm’s case for the need to implement AI on devices is compelling: Malladi explained that the company is on an “ongoing mission” to move computing from the cloud “to the edge, directly to devices.”

He noted that the huge leaps in computing capabilities of today’s devices, as well as advances in connectivity technologies, make such a shift entirely possible, although he conceded that the same high-speed networks that enable AI on devices could very well support a decent cloud service, especially as the number of parameters in models in use starts to reach into the billions.

Malladi noted that questions about the scalability of AI on devices remain prevalent, despite the “computing power” available.

Running AI on a device can be difficult, but a Qualcomm executive argues that the pros outweigh the cons of using the cloud, citing the costs and increasing complexity of the tasks the technology is expected to handle.

Malladi explained that the cost of inference “grows exponentially if you run everything exclusively in the cloud,” noting that this could prove problematic in the future.

This is not a chatbot anymore

Durga Malladi – Vice President and General Manager, Technology Planning and Edge Solutions

Qualcomm Technologies

He explained this by referring to research published by Reuters Agency in 2023 to the level of generative AI processing needed to run a portion of Google Search queries through the cloud, which showed the cost was “staggering.” That would offset any benefits gained from lowering the price of the hardware involved, Malladi said.

“The second thing is that the types of applications are becoming very rich now. It’s not a chatbot anymore.”

Services are embracing more multimodality, Malladi said, pointing to images, voice and other add-ons that he explained make it “more challenging” and harder to scale. Add to that the sheer number of actual users, and the numbers involved in “generating tokens or processing images” become even more daunting.

Malladi highlighted environmental concerns about the growing demand for cloud computing, citing projections that the amount of energy needed for AI could account for 3.5% of total global energy consumption by 2030.

A New Dawn
Malladi called the current hype around AI the third spring of a technology that, he explained, has been around since at least the middle of the last century.

He noted the development of the Turing test in the 1950s, which Encyclopedia Britannica is an assessment of a computer’s ability to reason in the way humans would, one of the first steps in what he called the first “spring” of artificial intelligence.

This spring was characterized by a number of original ideas, including the development of ELIZA, described by the New Jersey Institute of Technology as a natural language processing program, written in the mid-1960s by Professor Joseph Weizenbaum of the Massachusetts Institute of Technology.

Interestingly, ELIZA was originally called a chatterbot, now the term is slightly shortened to chatbot.

That initial spring quickly turned into winter, Malladi said, as research later conducted in the 1960s concluded that the amount of knowledge these chatterbots could acquire fell far short of the “lofty goals” expected of them.

The second spring of AI had to wait until the early 1980s, when expert systems, deep convolutional networks, and parallel distributed computing capabilities paved the way. Malladi explained that factors such as human expertise and the beginning of widespread PC adoption led to the collapse of this round of interest in the technology in the early 1990s, the second winter.

Despite this second failure, Malladi noted progress in handwriting and numeric recognition, pointing to the ability of ATMs to recognize numerals on deposited checks.

Ironically, it was events that took place later in the 1990s that convinced Malladi that the current trend in AI development would not die out again.

He pointed to the birth of the consumer internet, which brought access to vast amounts of data that had been a barrier in the previous two decades. The second factor was the dramatic increase in available computing power. Malladi noted that desktop and laptop computers have gained more processing capabilities, changing the foundations of artificial intelligence.

“So we’re in the third spring of AI, and we think there’s no turning back,” Malladi said, explaining that the processing power of devices and the amount of data available from public and enterprise sources mean “there’s a ton of automation that can be done right now” for consumer and productivity use cases.

Security
Malladi returned to this issue by examining the types of data we currently deal with in the context of on-device AI.

The executive noted the growing demand for more personalized responses from AI-powered consumer services, but also for higher levels of security. Using medical records as an example, Malladi explained that an AI voice assistant needs to offer personalized information rather than rely on public domain details, arguing that this poses risks when cloud computing is involved.

“Do you want to be able to access that data and then send it to the cloud so it can do some inference and come back? Why would I want to do that when I can run it directly on the device?”

Another potential use case was demonstrated at the Qualcomm Snapdragon Summit in 2023, when a person was looking for information about what they were looking at by pointing their phone at it. Malladi explained that context is required to generate the response, including deriving the user’s position from various sensors, a task that involves “a lot of information” that is “very local and contextual.”

Malladi argued that these examples of the need to protect data privacy are why “it is best to keep data private on devices.”

He explained that in corporate scenarios, there may be a need to access data from off-premises, noting that access to corporate servers or cloud services may vary depending on where an employee is located.

“But regardless of connectivity, you want to have a unified AI experience,” he explained, noting that if you can run the technology directly on the device, “you actually have the ability to get the answer without any impact on how the connectivity works.”

Common goals
As with many recent high-level discussions on AI, Malladi stressed the importance of partnerships and ethics.

He emphasized that Qualcomm does not create genAI models, which means developing standard approaches to evaluating them is increasingly important as developers tend to apply their own rules for what is fair and safe.

Qualcomm is contributing to these standards, and Malladi references his work on the AI ​​Security Working Group at ML Commons, an engineering consortium focused on the technology.

This is a really good initiative that is considered, at least in the US, as a starting point.

Durga Malladi – Vice President and General Manager, Technology Planning and Edge Solutions

Qualcomm Technologies

The company’s partnerships play a role in advancing ethics and principles: Malladi said that alongside device OEMs, Qualcomm is working with governments and regulators, in part to clarify what AI is “and what it isn’t,” and is also working with developers, which includes offering access to testing through a center centered around the company’s various compatible chips.

“Our job is not to explain to them the intricacies of our NPU and our CPU, but to make it easier for them to access the ‘chips’ without having to know all the details.”

Malladi argued that storing data locally, rather than using the cloud, could also play a key role in AI ethics, although he acknowledged that security remains an important factor even when information is stored on a device. “This has nothing to do with AI per se, but I think in the context of AI it becomes even more important.”

The executive noted growing regulatory concerns about deepfakes, explaining that much of the issue revolves around what actually constitutes a fake. He questioned whether making a few simple edits to a photo could be considered a fake, adding that Qualcomm considers that to be an original element, an extension another, and entirely synthetic images a third.

He added that Qualcomm is working with secure content transparency tool provider Truepic to verify metadata covering all three elements and provide a “certificate of authenticity” that provides some degree of transparency.

Malladi noted that in addition to the fact that many flagship smartphones now include AI directly, the pace of development of language models also serves Qualcomm’s mission as companies do more with fewer parameters to choose from.

As an example, he cited Meta Platform’s Llama3, which offers 8 billion and 70 billion parameter options, while its predecessor offered 7 billion, 13 billion, and 70 billion, respectively.

“In short, what we call smaller models are vastly superior to yesterday’s larger models,” which in turn enables richer use in popular devices.

While Malladi’s presentation obviously focused on Qualcomm’s core competencies and its push into devices, his views carry weight because of his background as a technologist who studied neural networks, among other things, at university.

His talk reflects a growing consensus on the fundamental challenges of implementing AI, as well as a growing understanding of the need for collaboration, education, and, of course, data.