The next generation of AI devices has arrived. Who will be the next market leader after Nvidia?

“The next thing we know and the gate for horses to enter this race opened a few days ago,” he said. “It’s an AI computer and an AI smartphone.”

Pollak referred to the Copilot+ computers announced last week by Microsoft, which the company describes as a “new era” in computing. These are AI-centric laptops that, at least for now, run exclusively on Qualcomm chips.

They will also use chipsets from Intel and AMD later this year, but all first-generation Copilot+ PCs when they go on sale in June will be based on Qualcomm processors.

Meanwhile, artificial intelligence smartphones, which began appearing late last year when Google released the Pixel 8 and accelerated earlier this year with Samsung’s release of the Galaxy S 24, are expected to reach their peak next month, when Apple is expected to announce changes to A.I. on their iPhones. Devices of this type either use Qualcomm chips or, in the case of these three particular companies, chips that are in the same family as Qualcomm chips.

But to understand what all this has to do with Nvidia, you first need to understand the difference between two big consumers of processor power in the AI ​​era: training and inference.

Training an AI model, which in the case of the generative AI models built by companies like Google, OpenAI, Microsoft and Meta, involves vacuuming up all the data in the world and looking for statistical connections between things, requires a lot of computing power and has seen customers line up in queue to fill their data centers with powerful Nvidia systems.

However, inference, which involves taking a model and making it do something useful like writing an email, doesn’t require that much processing power. Also, inference has typically been run centrally in huge data centers powered by Nvidia (or similar) chips, but this is starting to change.

What AI phones from Google, Samsung and (soon) Apple have in common – as well as Microsoft’s Copilot+ computers – is that they all do AI inference locally, on low-power chips inside the device, rather than on high-power chips inside the device. cloud .

Neural processing units

Training essentially stays in the cloud, but inference spreads to edge devices.

For example, to qualify for the Copilot+ PC label, laptops must be equipped with an inference chip, called a neural processing unit, capable of performing 40 trillion operations per second, or 40 TOPS.

The Qualcomm Snapdragon X Elite chipset in the first generation of Copilot+ computers will actually be capable of something more – 45 TOPS.

That’s not much processing power compared to the 800 TOPS provided by Nvidia’s laptop GPUs, but Microsoft is betting it’ll be enough to reason about artificial intelligence, even if it’s not enough to train it.

Indeed, to help inference work more efficiently on consumer devices like PCs and AI phones, Microsoft, Google and others are training new, lightweight versions of their models that run fast on low-power NPUs but still have sufficient accuracy to satisfy consumers.

Microsoft’s Copilot+ computers will have 40 different models of different sizes, and similarly Google has multiple sizes of its Gemini model, some of which will be small enough to be inferred “on-device” and some so large that they will still need to run in cloud data centers.

From an AI stock investing perspective, Loftus’ Pollak says it’s still an open question how much value the move to NPU for inference will divert from Nvidia and give to companies like Qualcomm.

However, it opens up the possibility of creating an entirely new generation of applications that will use local AI inference to produce results that would be either impossible or impractical to achieve using the cloud.

Even though local inference on small models has the disadvantage of not being as accurate as cloud inference on large models, it has the distinct advantage of being fast, cheap, and, above all, private.

When asked which of these applications is worth investing in, Pollak did not want to answer. This is just the beginning, and we don’t yet know how app developers will use the new AI-powered desktops and phones.

Just like with the early days of the Internet and smartphones, these will most likely be applications that no one has even thought of yet.