Wiring and powering GPUs differently can swing AI latency by orders of magnitude, says CoreWeave
That has architectural consequences for teams building inference-heavy services with bursty demand. If document pipelines, agent workflows or retrieval systems spike unpredictably, the cloud decision shifts from raw accelerator access to end-to-end tail-latency control. Practitioners should probe how a provider handles multi-GPU communication, noisy-neighbor isolation, storage locality and cross-cloud connectivity, especially when web tiers, data stores and security controls remain on AWS, Azure or Google Cloud. A faster inference tier can still lose its advantage if requests traverse slow integration paths or data egress bottlenecks.
The interoperability point also deserves scrutiny. “Open” networking and standard APIs can reduce lock-in, but portability in AI stacks is rarely automatic once observability, evaluation, orchestration and model-serving pipelines are wired into a vendor’s platform. Teams should distinguish between CUDA-level compatibility and operational portability.
Before committing, enterprises should treat AI infrastructure selection like a benchmarked systems exercise, not a capacity purchase. Measure p95/p99 latency under burst, startup time for new capacity, queueing behavior, data transfer penalties and failure-domain impact during regional or rack-level disruption. Those factors will matter more to production inference than headline GPU counts.
The neocloud market is moving past its origins as a stopgap for scarce graphics processing units. AI-native startups now choose their infrastructure on latency, burst capacity and openness, not just chip availability.
That shift is playing out at CoreWeave Inc., which is expanding beyond GPU compute into networking, storage and software as inference demand grows. At the same time, LlamaIndex Inc. has evolved from an open-source framework for retrieval-augmented generation into a model builder that rents its compute rather than owning it, according to Jerry Liu (pictured, right), co-founder and chief executive officer of LlamaIndex.
“We’re effectively a specialized AI lab right now that’s purely focused on building models for document parsing and extraction. We post-train open-weight models, we gather our own datasets and we make it really, really good at analyzing and reading documents to basically extract that data,” Liu said. “We care a lot about making sure that we can actually tailor everything we’re doing at the Pareto frontier of performance, cost, and latency for our customers.”
Liu and Lukas Biewald (left), senior vice president of AI initiatives at CoreWeave, spoke with theCUBE’s John Furrier and Dave Vellante at Fully Connected, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed long-running agents, governance and why AI-native startups are turning to specialized clouds for inference-heavy workloads. (* Disclosure below.)
Why bursty AI workloads favor the neocloud model
LlamaIndex’s compute footprint barely existed a year ago. Today its workload runs about 75% inference and 25% training, and it processes millions of document pages per day for finance, legal and insurance customers whose paperwork arrives in bursts, according to Liu. The company owns no GPU cluster, so guaranteed capacity matters more than hardware ownership.
“We serve a lot of different customers at extremely persistent and also spiky workloads,” ,” Liu said. “We really, really need to make sure that we have the right capacity to serve our customers without getting throttled.”
CoreWeave is betting that capacity alone is not the differentiator. Biewald joined the company through its acquisition of Weights & Biases, the AI observability startup he co-founded, and CoreWeave used the event to launch CoreWeave Forge, a development layer that runs training, inference, evaluation and agent development in one connected environment. Coming from software, he initially questioned how much chip configuration could really matter, Biewald noted.
“I’ll tell you, the answer is ‘massive difference,’” Biewald said. “I’m talking orders of magnitude difference depending on how you do the networking for the chips [and] how you do the power distribution.”
Openness also separates CoreWeave from the hyperscalers, according to Biewald. Where providers such as Amazon Web Services Inc. lean on proprietary application programming interfaces that make workloads hard to move, CoreWeave follows the standard networking protocols recommended by Nvidia Corp., which brings broader open-source support. Analysts have observed that CoreWeave is broadening its portfolio much as AWS did in its early days, even as the company bristles at the neocloud label.
“CoreWeave knows that everyone is coming from a different cloud,” Biewald said. “Everyone’s going to host their web service on AWS or GCP, not on CoreWeave. CoreWeave is okay with that, so CoreWeave plays much more nicely with the other clouds.”
That ecosystem points to a larger change in who gets to build intelligence, Liu noted. Post-training a small open-weight model remains a skill limited to a narrow group of specialists today. Abundant neocloud capacity, combined with fast-improving coding agents, could open that work to far more people.
“Everyone is starting to get really good at defining observability and evals and the right metrics to focus on,” Liu said. “I think there’s going to be a world where we’re basically just going to automate this entire loop and make it accessible to everybody.”
Here’s the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of Fully Connected:
(* Disclosure: TheCUBE is a paid media partner for the Fully Connected 2026 event. Neither CoreWeave, the sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)
Wiring and powering GPUs differently can swing AI latency by orders of magnitude, says CoreWeave
Enjoyed this article? Sign up for our newsletter to receive regular insights and stay connected.