Why No Big Tech Company Has Been Able to Dethrone Nvidia as the King of AI

Nvidia (NVDA) is known for building AI chips, but its most important design is the business barrier that keeps customers and competitors at bay. That barrier is made of software as much as silicon.

Over the past two decades, Nvidia has created what’s known in tech as a “walled garden,” not dissimilar to the one Apple (AAPL) has created. While Apple’s ecosystem of software and services is aimed at consumers, Nvidia has long focused on developers who build AI systems and other software with its chips.

Most Read in The Wall Street Journal

Nvidia’s walled-off system explains why, despite competition from other chipmakers and even tech giants like Google and Amazon, Nvidia is unlikely to lose significant share of the AI ​​market in the next few years.

It also explains why, in the long term, the fight for territory where Nvidia currently dominates will likely center on the company’s coding prowess, not just the design of its circuitry — and why its rivals are racing to develop software that can get around Nvidia’s guardrail.

The key to understanding Nvidia’s closed garden is a software platform called CUDA. When it launched in 2007, the platform was a solution to a problem no one else had: how to run non-graphical software, like encryption algorithms and cryptocurrency mining, on Nvidia’s specialized chips, which were designed for labor-intensive applications like 3D graphics and video games.

CUDA enabled all sorts of other computations on these chips, known as graphics processing units, or GPUs. Among the applications that CUDA allowed Nvidia’s chips to run was AI software, whose explosive growth in recent years has made Nvidia one of the most valuable companies in the world.

Moreover, and this is key, CUDA was just the beginning. Year after year, Nvidia responded to the needs of software developers by releasing specialized code libraries that enabled a huge number of tasks to be performed on its GPUs at speeds that were impossible with conventional general-purpose processors such as those made by Intel and AMD.

The importance of Nvidia’s software platforms explains why Nvidia has for years employed more software engineers than hardware engineers. Nvidia CEO Jensen Huang recently called his company’s focus on combining hardware and software “full-stack computing,” meaning Nvidia makes everything from chips to AI software.

The story continues

Every time a rival announces AI chips to compete with Nvidia’s, it’s competing with systems that Nvidia customers have been using for more than 15 years to write mountains of code. That software can be difficult to port to a competitor’s system.

At its June shareholder meeting, Nvidia announced that CUDA now includes more than 300 code libraries and 600 AI models, and supports 3,700 GPU-accelerated applications used by more than five million developers at about 40,000 companies.

The sheer size of the AI ​​computing market has prompted a number of companies to band together to take on Nvidia. Atif Malik, a semiconductor and networking equipment analyst at Citi Research, predicts that the AI-related chip market will reach $400 billion annually by 2027. (Nvidia’s revenue for the fiscal year ending in January was about $61 billion.)

Much of that collaboration is focused on developing open-source alternatives to CUDA, says Bill Pearson, Intel’s vice president of AI for cloud customers. Intel engineers are contributing to two such projects, one involving Arm, Google, Samsung and Qualcomm. OpenAI, the company behind ChatGPT, is working on its own open-source project.

Investors are flocking to startups working to develop alternatives to CUDA. That investment is fueled in part by the possibility that engineers from many of the world’s tech giants could come together to let companies use any chips they want—and stop paying what some in the industry are calling the “CUDA tax.”

One startup that could benefit from this open source software, Groq, recently announced a $640 million investment at a $2.8 billion valuation to produce chips that will compete with Nvidia.

Tech giants are also investing in their own alternatives to Nvidia chips. Google (GOOG) and Amazon (AMZN) are making their own AI training and deployment chips, and Microsoft (MSFT) announced in 2023 that it would follow suit.

One of the most successful competitors to Nvidia’s dominance of AI chips is AMD (AMD). It’s still a fraction of Nvidia’s size in the market—AMD is forecasting $4.5 billion in revenue in 2024 for its Instinct AI chip line—but it’s investing heavily in hiring software engineers, says Andrew Dieckman, AMD’s executive vice president.

“We’ve expanded our software resources tremendously,” he says. AMD announced last month that it would acquire Silo AI for $665 million, adding 300 AI engineers.

Microsoft and Meta Platforms (META), two major Nvidia customers, are buying AI chips from AMD, reflecting a desire to spur competition for one of the most expensive products in the tech giants’ budgets.

Regardless, Malik of Citi Research expects Nvidia to maintain its share of the AI-related chipset market at around 90% for the next two to three years.

To understand the pros and cons of the alternatives, it’s worth understanding what’s needed to build a ChatGPT-style AI without using Nvidia hardware or software.

Babak Pahlavan, CEO of startup NinjaTech AI, says he would use Nvidia hardware and software to run his company—if he could afford it. But shortages of Nvidia’s powerful H100 chips keep prices high and availability difficult.

Pahlavan and his co-founders eventually turned to Amazon, which makes its own AI training chips, the process by which such systems “learn” from massive data sets. After months of effort, the team finally managed to train their AI on Amazon’s chips, known as Trainium. It wasn’t easy.

“There were a lot of challenges and bugs,” says Pahlavan, whose team at NinjaTech AI met four times a week for months with Amazon’s software team. Eventually, the two companies worked out the kinks, and NinjaTech’s AI “agents” that perform tasks for users went live in May. The company says its service has more than 1 million monthly active users, all of whom are powered by models trained and running on Amazon chips.

“There were some early mistakes on both sides,” says Amazon Web Services Director Gadi Hutt, whose team worked with NinjaTech AI. But now, he says, “we’re off to the races.”

Customers using Amazon’s custom AI chips include Anthropic, Airbnb, Pinterest and Snap. Amazon offers its cloud computing customers access to Nvidia’s chips, but they are more expensive to use than Amazon’s own AI chips. Still, it would take time for customers to make the switch, Hutt says.

NinjaTech AI’s experience illustrates one of the main reasons startups like it have to endure the difficulties and extra time it takes to develop AI outside Nvidia’s gated garden: the cost.

To support more than a million monthly users, NinjaTech’s cloud bill at Amazon is about $250,000 per month, Pahlavan says. If he ran the same AI on Nvidia chips, it would be between $750,000 and $1.2 million, he adds.

Nvidia is fully aware of all these competitive pressures and that its chips are expensive to buy and run. Huang, its CEO, has promised that the company’s next generation of AI-focused chips will lower the cost of training AI on the company’s hardware.

For the foreseeable future, Nvidia’s fate is a matter of inertia—the same kind of inertia that has kept companies and customers confined to walled gardens throughout history. Apple included.

For more analysis, reviews, advice and headlines from WSJ Technology, click here.

Write to Christopher Mims at [email protected]

Most Read in The Wall Street Journal