The Only Real Moat in AI Isn’t Hardware—It’s Nvidia’s CUDA

Let me apologize for opening with an overused phrase: a piece of classic finance jargon that has recently wormed its way into mainstream tech vocabulary. But there’s no avoiding it— I have to talk about “moats.”

Popularized decades ago by Warren Buffett to describe a company’s durable, unbeatable competitive advantage, the term firmly embedded itself in Silicon Valley pitch decks after a supposedly leaked internal Google memo, titled “We Have No Moat, and Neither Does OpenAI,” warned that open-source AI would overrun Big Tech’s metaphorical market castle.

A few years on, those castle walls remain standing. Outside of a short-lived panic when Chinese AI lab DeepSeek launched its first open-source models, open-source AI has yet to drastically outperform closed, proprietary models from leading research groups. Even so, none of the world’s top frontier AI labs—OpenAI, Anthropic, Google—can claim to have a meaningful competitive moat at all.

The one company that does hold a true, unassailable moat is Nvidia. CEO Jensen Huang has called it the firm’s most precious “treasure.” Contrary to what you might assume from a leading chipmaker, this moat is not a piece of cutting-edge hardware. It is a tool called CUDA. What sounds like a restricted pharmaceutical compound banned by the FDA may actually be the only unbeatable competitive advantage in the entire AI industry.

Technically, CUDA stands for Compute Unified Device Architecture, but just like “laser” or “scuba,” no one bothers to use the full acronym anymore—everyone just calls it “KOO-duh.” So what does this all-important treasure actually do? If forced to sum it up in one word: parallelization.

Let’s break that down with a simple example. Suppose we ask a computer to fill out a 9×9 multiplication table. On a single-core processor, all 81 calculations run one after another, one at a time. But a GPU with nine separate cores can split the work evenly: each core handles an entire column—one runs all 1x multiples, another all 2x multiples, and so on—delivering a roughly ninefold speed gain. Modern GPUs are even smarter than that: if programmed to recognize that 7×9 = 9×7, they can skip redundant work, cutting 81 operations down to 45 and nearly halving the total workload. When a single large AI training run can cost more than $100 million, every tiny efficiency gain adds up to massive difference.

Nvidia’s GPUs were originally built to render graphics for video games. Back in the early 2000s, a Stanford PhD student named Ian Buck—who first fell in love with GPUs as an avid gamer—realized their parallel architecture could be repurposed for general high-performance computing. He built an early prototype framework called Brook, was hired by Nvidia, and teamed up with John Nickolls to lead the development of CUDA. So if AI eventually ushers in an era of permanent white-collar underemployment and deadly autonomous weapons, you can trace it all back to a gamer who wanted a Doom demon’s anatomy to jiggle smoothly at 60 frames per second.

CUDA is not a programming language in itself—it is a “development platform.” I use that vague catch-all term because, just like The New York Times is a newspaper that has also become a major gaming company, CUDA has grown over 20 years into a sprawling, layered bundle of specialized software libraries built specifically for AI workloads. Every library shaves nanoseconds off individual mathematical operations, and when you add all those gains together, it lets Nvidia GPUs run AI work far faster than any competing hardware, to use the industry meme, “go brrr.”

A modern graphics card is not just a circuit board crammed with chips, memory, and cooling fans. It is an intricate system of cache hierarchies and specialized processing units called “tensor cores” and “streaming multiprocessors.” In that sense, what chip companies sell is like a professional commercial kitchen, and more processing cores are equivalent to more grilling stations. But even a kitchen with 30 grilling stations won’t fill orders any faster without a skilled head chef to deftly assign work—and that is exactly what CUDA does for GPU cores.

To extend that metaphor: hand-tuned CUDA libraries optimized for specific matrix operations are the equivalent of specialty kitchen tools built for one job only—a cherry pitter, a shrimp deveiner. Those are nice extras for home cooks, but they are non-negotiable when you have 10,000 shrimp to process. Which brings us back to DeepSeek. Its engineers went even lower than CUDA’s existing layer of abstraction to code directly in PTX, a low-level assembly language native to Nvidia GPUs. To use the garlic peeling example: an unoptimized GPU would be told “peel the skin with your fingernails.” CUDA can instruct: “smash the clove with the flat of your knife to loosen the skin.” PTX lets you dictate every tiny sub-step: “lift the blade 2.35 inches above the cutting board, align it parallel to the clove’s equator, and strike downward with 36.2 newtons of force.”

That example makes it clear why CUDA is so valuable to Nvidia, and so hard for any competitor to replicate. Tuning GPU performance is an extraordinarily gnarly, labor-intensive problem. You can’t just hire a new grad out of college, hand them a Claude Max subscription, and expect them to crank out optimized GPU kernel code. Writing code at this level of granularity is grueling, slogging work—even for most experienced developers, outside of elite teams like DeepSeek’s.

Full disclosure: In past installments of my Machine Readable column, I was already familiar with the languages I analyzed. That was not the case here. To uphold my standard of first-hand experience, I decided to spend a full day working with CUDA. It ruined my afternoon.

A simple matrix multiplication that takes me just three lines of code in PyTorch—the world’s most popular machine learning framework—took me more than 50 lines of code to get working in CUDA. It turns out squeezing every last drop of performance out of a GPU is an admirable goal, but an incredibly tedious one. After dipping my own toe into Nvidia’s moat, I can confirm: it is indeed deep and forbidding.

CUDA’s dominance is built not just on the quality of its developer ecosystem, but on powerful network lock-in. Because every major modern machine learning framework is built on top of CUDA, which runs natively only on Nvidia chips, competing chips from rivals like AMD underperform even when they have more cores and memory on paper. Comparing GPUs by published specs is like comparing race cars by cylinder count: real performance can only be measured on the track.

Another disclosure: I originally planned to run my own benchmarks comparing top chips from Nvidia and AMD, but there was no way I could expense a $10,000+ Nvidia H100 and AMD MI300X without landing on Condé Nast’s permanent blacklist. Instead, you’ll have to take the word of independent researchers, who have consistently found that even with better on-paper specs, AMD’s best chips are outmatched by Nvidia’s top models.

A big part of Nvidia’s software edge comes down to one unusual choice for a chipmaker: it hires more software engineers than hardware engineers. If I were running AMD, that would be my first order of business. Not that anyone asked.

Every year, new hopefuls emerge trying to drain Nvidia’s moat, only to drown in it themselves. OpenCL, an open standard backed by a consortium including Apple, AMD, and Qualcomm, was supposed to be an Android-like alternative to CUDA’s closed “iOS” model. It barely gained any developer traction.

Meanwhile, AMD’s own answer to CUDA, ROCm, has an even worse name than CUDA—for the record, is it supposed to be pronounced “rock cum”? (Forget hiring more programmers first—hire a new marketing team.) It has also been so plagued by bugs and compatibility issues that its subreddit reads like a support group for frustrated developers.

Let’s not forget Intel. While it’s easy to write Intel off as a struggling chipmaker, its recent history proves it is also a struggling software company. In a last-ditch push for relevance, it launched oneAPI, but even looking ahead to 2026, it is all but guaranteed that CUDA will still reign supreme. The only plausible challenger on the horizon is Modular, led by legendary language designer Chris Lattner—creator of Apple’s Swift and the LLVM compiler infrastructure.

But the open industry secret is that, just like most theoretical physicists can’t change a tire to save their lives, most AI researchers can’t even write a single line of functional C++ code. There are very few skilled GPU kernel engineers in the entire world, and a large majority of them already work for Nvidia. Long before AI researchers started chasing industry clout, these engineers were diligently grinding away on CUDA with no public recognition. Even the most advanced AI coding assistants still hobble through writing working kernel code.

In the end, Nvidia is far more similar to Apple than it is to AMD or Intel. It is a great hardware company because it is first a great software company. Apple’s moat against Android was never just the iPhone—it was the entire ecosystem: iOS, the App Store, and the network of developers building for it. Sure, you can fold a Samsung Galaxy in half, but do you actually want to switch to using Samsung Pay for everything? For the foreseeable future, the AI industry will just have to live with Nvidia’s notoriously sky-high chip prices.

This is the first installment of a three-part Machine Readable series exploring the programming languages that power modern AI.