With intelligent SaaS—anything using large language models or agents—the biggest moat isn't the model or the interface. It's cost efficiency. Every API call costs money. And the more your users rely on the product, the more those costs pile up.
At small scale, it's easy to ignore. But when usage grows, it becomes a real problem. Some companies end up spending half their revenue on inference. Others try to raise prices, which rarely works.
A lot of the work goes into the usual things: caching, prompt trimming, routing simple tasks to cheaper models, or switching to smaller local models when possible. That helps. But it's only part of it.
What matters just as much is how the prompt gets built—how the context is prepared before the model sees anything. If the input is wrong, the output is useless. Shit in, shit out.
The Context Problem
That's where retrieval comes in. Not generic RAG setups, but retrieval that's tuned for the actual problem your product solves. Whether it's HR, legal, support, or something else, the AI needs the right context, in the right format, every time. It's not just about stuffing documents into a vector DB and calling it a day. It's about structuring the inputs so the model can actually do its job without guessing.
Over time, this becomes the real moat. Not just "our AI does X," but "our system gives the AI exactly what it needs to do X well—and does it cheaply."
The Efficiency Advantage
Most products won't be replaced because someone has a better prompt. They'll be replaced because someone else runs the whole thing 5x more efficiently and gets better output with smaller models. That's where this is heading.
The companies that figure out how to deliver the same quality at a fraction of the cost will own their markets. It's not just about the technology—it's about the entire system working together: smart routing, efficient retrieval, and context that's engineered for the task.
"The real moat isn't the AI. It's making the AI work efficiently at scale."
This is the unglamorous work that actually matters. While everyone's talking about the latest model capabilities, the winners are quietly building systems that make those capabilities affordable and reliable for real users.