Four years ago, VAST Data began building for cloud providers that wanted to run tens of thousands of GPUs in a single build-out, on an architecture its engineers insisted had to be all-flash at a time when much of the storage industry considered flash too niche for computing at that scale.
The company had come to the problem through high-performance computing, where the hardest mathematical workloads expose every weakness in a system as it grows, and it designed for cost and performance in equal measure. When ChatGPT pushed AI infrastructure spending into the mainstream, those early design choices suddenly mattered to a far larger set of buyers.
Storage, long the last line item addressed in an AI build-out, has since become one of its most contested. Jason Hammons, VP Systems Engineering at VAST Data, spoke to GEC Newswire about why the category has filled up so quickly, how the company handles the fragmented data that decides whether AI pilots ever scale, and what customers preparing for 2027 are asking for. The performance and efficiency figures he cites are VAST Data’s own; no independent benchmark or named comparison provider was shared during the interview.
Storage has suddenly become one of the most crowded parts of the AI infrastructure market. Why is everyone rushing into it now?
It is getting crowded because it has classically been underserved. Storage is usually the last part that gets addressed in a build-out, because everyone has been thinking about GPUs, networking and everything around them, and those things are important. But if data becomes the bottleneck for GPU utilisation, you become less effective, you deliver a poorer customer experience, you struggle to scale and you end up with availability problems. That is when people start thinking about it a lot more, and when a lot more money goes into it. What is happening right now is that people are discovering a lot of platforms out there are not really fit for purpose for AI at scale, and you don’t find that out until you get there.
VAST Data describes itself as an AI operating system spanning storage, analytics and more. Did you build those layers one after another, or all at once?
We approached it in a lot of different ways, because AI factories are built differently in different sectors. We started with cloud providers four years ago, building additional services and capabilities so they could scale up massively and cater for tens of thousands of GPUs in a single build-out. There are a lot of hard problems to solve there. You can build plenty of things at small scale that appear to work. Once you reach a certain level of scale, whether it is storage, a vector database or an event broker, things start breaking, and suddenly you discover a dependency nobody expected.
Having gone through that with some of our bigger customers three or four years ago, we have already done the hard work on scalability, and the product is designed for it. We are now at the point where reliability is the number one thing people ask us about, because most storage systems go down all the time. VAST stays up, and that has been extremely important for some very big customers. The other piece is building on flash with the best possible economics.
Much of the market once dismissed flash as too niche for this kind of work. What gave VAST the conviction to bet on it anyway?
We were quite lucky, because our CEO, our engineering team and our architects all stood very firmly on one point. When you solve difficult problems with a lot of compute at scale, performance matters enormously to customers, and cost matters just as much. GPUs were not as prominent then as they are now. We built the product to solve some of the toughest mathematical problems in large-scale computation, and a lot of that started in HPC, with large x86 and GPU environments. Going through that teaches you what needs to be hardened, and that most customers make their decisions on a cost-performance equation.
So we made a decision early on that it had to be all-flash, built so that it was structurally highly scalable and also very efficient. We focused on data reduction algorithms that did not sacrifice performance. We call the architecture disaggregated, shared-everything, and the way it functions turned out to be even more beneficial for AI than for HPC.
What changed once ChatGPT arrived?
Once we crossed that chasm, cloud providers started appearing and we were with them from the first hours. They realised we had the best cost-performance, and that the performance properties we had created were the best in the industry for practical AI use. In the old world of HPC, a lot of synthetic benchmarking and theoretical maths went into judging the speed of a storage system. Some of it applied to scientific workloads. Nobody was really building for AI back then, though, and the industry did not understand the need for the massive read performance we engineered into the product. Our read performance under mass concurrency across a large namespace outperforms the competition, and we use about half the flash the next closest provider needs to deliver it.
Enterprise data has been accumulating in fragmented form for more than a decade, and the rush to deploy AI has exposed it. How does VAST deal with fragmentation and data quality, which ultimately decide whether pilots scale?
We have focused on building very scalable and stable data structures with this architecture. That lets us roll out native services that use our native metadata structures efficiently. The alternative is layering on a gateway here, a gateway there and another gateway on top, until the whole thing is bolted onto the architecture and no longer part of it. Bolt-ons solve short-term problems, and they don’t scale. If you want to scale efficiently, you need extreme efficiency and access to those native metadata structures across thousands of drives at once, because GPUs drive mass concurrency and that concurrency only keeps growing. You have to stay behind the architecture and resist bolting things on to fix a short-term problem.
We take our time as a result. We spent about two years building the world’s most scalable vector database, and it is in the market today handling trillions of vectors and embeddings in a SQL data warehouse, where many competitors are standing up hundreds of clusters to deal with the same thing. Then there is security and sovereignty: who can see what, and how you audit what an LLM did with a piece of data and how it joined that data with another table. We can track that, because it is a native service built into the system, using our native permissioning schema, security standards and audit logging, all exposed when they are needed.
Organisations in the UAE are working towards national AI adoption timelines that run into 2028, which makes 2027 a decisive year. Based on your customer conversations, what will they need most?
The number one thing is helping them get access to flash, because AI doesn’t work without flash. We proved this four years ago, and some of the market didn’t believe us. Now everybody realises that if you want it to work, you have to have it. The challenge is getting it into customers’ hands at the lowest possible cost. It sounds very basic, but it is a really important problem we are solving right now for our cloud providers, our AI factories and our enterprise customers, who need half the flash with us that they would need with someone else.
After that, it is about making native tooling simple to use inside the product. With our vector database, for example, you can instantiate it in three clicks. If you tried to get that through your IT department, it would take a week or two of approvals for compute, networking and security sign-off, the typical way an enterprise deploys anything. All of that is built into the product, it is simple to turn on, and it retains all the security and permission properties we discussed. You get through security, everyone is happy, and off you go with that extreme efficiency. Beyond that, it comes down to choosing the right use cases and advising customers on how to use our tooling.
