The Only Real Moat in AI Isn’t Hardware—It’s Nvidia’s CUDA

Let me apologize upfront for leaning on an overused phrase—once a niche finance term that has recently wormed its way into mainstream tech vocabulary—but I cannot write about the modern AI industry without talking about “moats.” Popularized decades ago by Warren Buffett to describe a company’s lasting, defensible competitive advantage, the term blew up in Silicon Valley startup circles after an allegedly leaked Google memo titled “We Have No Moat, and Neither Does OpenAI” warned that open-source AI would overrun big tech’s dominant market position.

A few years on, that predicted collapse of big tech’s “castle” has not come to pass. Outside of a short-lived panic when open-source lab DeepSeek first launched, open-source AI models have not managed to massively outperform closed, proprietary alternatives from leading frontier labs. Even so, none of those top labs—OpenAI, Anthropic, Google—can actually claim to hold a real, unassailable moat. The one company that does is Nvidia.

Nvidia CEO Jensen Huang has called CUDA the firm’s most precious treasure, and that claim holds up. Contrary to what you might assume from a leading chipmaker, Nvidia’s moat is not a cutting-edge piece of silicon hardware—it is CUDA. The name might sound like a banned chemical compound you’d see on an FDA recall list, but it is actually the only true, unbeatable competitive advantage in the entire AI sector today. Technically, CUDA stands for Compute Unified Device Architecture, but like common acronyms such as laser or scuba, no one ever uses the full expansion; everyone just refers to it as “KOO-duh.”

So what does this incredibly valuable asset actually do? If I had to sum it up in one word: parallel processing. Let’s break that down with a simple example. Imagine you task a computer with calculating every entry in a 9×9 multiplication table. A single-core CPU will run all 81 calculations one after another, slowly and methodically. But a GPU with nine cores can split the work evenly: each core handles one full column of the table, cutting total run time by a factor of nine. Modern GPUs can be even more efficient: if programmed to recognize that 7×9 equals 9×7, they can cut out duplicate work, shrinking 81 operations to just 45 and almost halving the total workload. When a single large AI training run can cost over $100 million, every tiny efficiency improvement adds up to massive gains.

Nvidia’s GPUs were originally built to render graphics for video games. Back in the early 2000s, a Stanford PhD student named Ian Buck—who first fell in love with GPUs as a hardcore gamer—realized the GPU architecture could be repurposed for general high-performance computing. He built an early programming framework called Brook, was hired by Nvidia, and led the development of CUDA alongside John Nickolls. If you ever wonder how we arrived at a world of AI-driven white-collar displacement and autonomous weapons, you can trace that path back to a gamer who wanted demons in Doom to jiggle at 60 frames per second.

CUDA is not just a programming language—it is an entire development platform. That vague term fits because, much like The New York Times is a newspaper that also operates a surprisingly successful gaming business, CUDA has grown over decades into a sprawling, interconnected bundle of software libraries built specifically for AI development. Every library is optimized to shave nanoseconds off individual mathematical operations; when you add those tiny gains together, you get the blazing fast GPU performance the entire AI industry relies on.

A modern graphics card is not just a hunk of circuit board stuffed with chips, memory, and cooling fans. It is a complex system of cache hierarchies and specialized processing units called tensor cores and streaming multiprocessors. To use a simple metaphor: what chipmakers sell is like a professional restaurant kitchen, and more processing cores are equivalent to more grilling stations. But even a kitchen with 30 grilling stations will not cook orders any faster without a skilled head chef to assign tasks efficiently—and that is exactly what CUDA does for GPU cores. Extending that metaphor further, the hand-tuned CUDA libraries optimized for specific matrix operations are like specialized kitchen tools: a cherry pitter, a shrimp deveiner. Those might feel like overkill for a home cook, but when you have 10,000 shrimp to process, they cut your work time dramatically.

That brings us back to DeepSeek. Its engineers went below CUDA’s layer of abstraction to code directly in PTX, a low-level assembly language for Nvidia GPUs. Sticking with the kitchen metaphor: if peeling garlic is your task, an unoptimized GPU would just tell you “peel the skin with your fingernails.” CUDA gives you a clearer, faster instruction: “smash the clove with the flat of your knife first.” PTX lets you dictate every tiny step of the process: “lift the blade 2.35 inches above the cutting board, align it parallel to the center of the clove, and strike downward with 36.2 newtons of force.” That level of control is why CUDA is so valuable to Nvidia, and why no competitor has been able to replicate it.

Tweaking GPU performance to squeeze out every last bit of speed is an incredibly difficult problem. You cannot just hire a new grad, hand them a Claude AI subscription, and expect them to build working, optimized GPU kernels. Writing low-level optimized code for GPUs is grueling, painstaking work—only a handful of engineers in the world are good at it, and most of them work for Nvidia.

Full disclosure: in previous Machine Readable columns, I was already familiar with the programming languages I analyzed. That was not the case here. To keep my usual standard of firsthand experience, I spent a full day working with CUDA, and it ruined my entire afternoon. A simple matrix multiplication that takes three lines of code in PyTorch (the most popular machine learning framework) took me more than 50 lines of clunky CUDA code to get working. Squeezing out maximum performance, it turns out, is noble work, but incredibly tedious. After dipping my toe into Nvidia’s moat, I can confirm: it is deep, cold, and very hard to cross.

CUDA’s dominance is not just built on the quality of its ecosystem—it is built on powerful network lock-in. Because all major modern machine learning frameworks are built to run on CUDA, which only works natively on Nvidia chips, competing chips from AMD underperform even when they have more cores and more memory on paper. Comparing GPUs by spec sheet is like comparing race cars by how many cylinders they have: actual performance can only be measured on the road.

Another disclosure: I originally planned to run my own benchmarks of top Nvidia and AMD chips, but buying an H100 and an MI300X would have gotten me blacklisted by Condé Nast’s finance team for excessive expenses. Instead, you can take the word of independent researchers, who have consistently found that even with better on-paper specs, AMD cannot match Nvidia’s real-world performance. One big reason for Nvidia’s software edge is unusual for a chipmaker: the company hires more software engineers than hardware engineers. If I were running AMD, that is the first change I would make (though no one asked me for my opinion).

Every year, new challengers emerge trying to drain Nvidia’s moat, only to get stuck and fail. OpenCL, an open standard backed by a consortium including Apple, AMD and Qualcomm, was meant to be the Android to CUDA’s iOS, but it never gained any meaningful traction among AI developers. AMD’s own answer to CUDA, ROCm, has a name that is almost unpronounceable—do you call it “rock cum”? (Before AMD hires more engineers, they should hire a new branding team.) It is also plagued by so many bugs and compatibility issues that its Reddit community reads more like a support group for frustrated developers.

We cannot forget Intel, either: it is easy to write off Intel as a struggling chipmaker, but its recent history shows it is also a struggling software company. Its last-ditch effort to compete, oneAPI, has barely made a dent, and there is no sign it will overtake CUDA any time before 2026, if ever. The most credible challenger right now is probably Modular, led by legendary programming language designer Chris Lattner, who created Apple’s Swift and LLVM. But the open secret in the industry is that just like theoretical physicists usually cannot change a tire to save their lives, most AI researchers cannot write even a single line of working C++.

Long before AI researchers became famous influencers on social media, these skilled kernel engineers were quietly grinding away on CUDA, getting little public recognition for their work. Even the best AI coding assistants still struggle to write working, optimized kernel code.

In the end, Nvidia is actually more similar to Apple than it is to AMD or Intel. It is a great hardware company because it is first a great software company. Apple’s moat against Android was never just the iPhone—it was the entire ecosystem: iOS, the App Store, the network of developers building for it. Sure, you can fold a Samsung Galaxy in half, but do you actually want to switch your entire payment system to Samsung Pay?

For the foreseeable future, the AI industry will just have to live with Nvidia’s notoriously high chip prices. This is the first installment of a three-part Machine Readable series looking at the programming languages and platforms that power modern AI.