AI inference infrastructure is becoming a system-level challenge as organizations move generative and agentic applications into production. Graphics processing unit performance remains essential, but storage latency, network bandwidth, data movement and power consumption increasingly determine the cost and speed of producing tokens.

The requirements also vary by workload. Interactive chat prioritizes latency, batch inference emphasizes throughput and agentic systems create expanding contexts. At the same time, retrieval-augmented generation and multi-tenant AI factories intensify demands across the stack, according to Ka Wai Leung (pictured, center), AI solutions product management at IBM Corp.

“You need to understand what type of workload,” he said. “Based upon the workload, you understand the characteristics of the workload, and you build your system behind it. That’s how you scale.”

Leung; William Li (left), general manager of solution management at Super Micro Computer Inc.; and Anders Graham (right), senior director of SSD marketing and business development at Kioxia Holdings Corp., spoke with theCUBE’s Rob Strechay for the Supermicro Open Storage Summit interview series, during an exclusive broadcast on theCUBE, SiliconANGLE Media’s livestreaming studio. They discussed how workload requirements, data architectures and coordinated infrastructure design shape inference performance at scale. (* Disclosure below.)

AI inference infrastructure starts with the workload

Training workloads commonly depend on high sequential throughput to feed models and complete checkpointing operations without leaving GPUs idle. AI inference infrastructure brings greater emphasis to low-latency random reads and writes, particularly when applications continually retrieve proprietary or recently updated information to supply context, according to Graham.

“I think we’re now in the year of inference … low latency is critical,” he said. “We also see retrieval-augmented generation … where you basically are going to provide additional context to the models on the fly using new data or proprietary data, whatever is necessary to retrain the model.”

Production inference also depends on the freshness and reliability of the context supplied to models. That requirement becomes harder when structured, unstructured and multimodal information spans mainframes and other enterprise environments, according to Leung.

“I think the first challenge is … enterprise accounts, where they have data just scattered everywhere. How do you as an organization or an architect provide access to all these types of data without ingesting these massive amounts into your AI factories? Data gravity — data sovereignty — is a big consideration.”

Full-stack design sustains inference efficiency

Power efficiency is another critical consideration in AI inference infrastructure as data centers face limits on available energy and physical capacity. Comparing its newer BiCS8-based CM9 drives with the preceding CM7 generation, Kioxia measured significant improvements in random-read and random-write input/output operations per second per unit of power, according to Graham.

“If we take a look at the random workload … you get similar metrics where the random read [is] 76% improvement and random write over 100% improvement,” he said. “Moving to next-generation technologies and taking advantage of the new NAND technologies is a great way to have better power efficiency.”

Supermicro’s reference design combines an Nvidia Corp. HGX B300 compute environment with the IBM Storage Scale Erasure Code Edition and Kioxia drives. The architecture uses a high-performance storage tier to support latency-sensitive workloads and can add a capacity tier for colder data, Li noted.

“Storage plays the most important role at the heart of the data center total solutions; we call it Data Center Building Block Solutions,” he said. “You don’t need to go to different vendors to look for server rack integration, liquid cooling deployment, storage deployment; you can … go to Supermicro and work with all the very active partners, including IBM [and] Kioxia, to provide you with other software solutions, as well.”

Nvidia, IBM and Supermicro tested IBM Storage Scale as a shared KV cache. The approach allows previously computed context to be reused without keeping all cached data in limited GPU memory or system RAM, and the test produced subsecond time-to-first-token responses across the tested prompt lengths. Nvidia’s team then asked that heavy traffic be added to the Spectrum-X network to test how its throughput advantage over an uncached baseline would hold under real-world contention, according to Leung.

“The number went down a little bit, but I think from 4.8 requests per second to about 3.6, so instead of 22 times, it’s like 18 times more efficient,” he said. “That’s because we bombarded the network with a lot of network noise to try to mimic real-world conditions.”

Stay tuned for the complete video interview, part of SiliconANGLE’s and theCUBE’s coverage of the Supermicro Open Storage Summit interview series.

(* Disclosure: TheCUBE is a paid media partner for the Supermicro Open Storage Summit interview series. Neither Supermicro, the sponsor of theCUBE’s event coverage, nor other sponsors have editorial control over content on theCUBE or SiliconANGLE.)

Photo: SiliconANGLE