Red Hat has announced the launch of its AI Inference Server, a development aimed at enhancing generative AI capabilities across hybrid cloud environments. The server is built on the vLLM community project and integrates Neural Magic technologies to improve speed, efficiency, and cost-effectiveness. This initiative aligns with Red Hat’s vision of enabling any generative AI model to run on any AI accelerator in any cloud setting.
Joe Fernandes, Vice President and General Manager of Red Hat’s AI Business Unit, emphasized the importance of inference in AI operations: “Inference is where the real promise of gen AI is delivered…but it must be delivered in an effective and cost-efficient way.” The new server aims to meet these demands by providing a common inference layer that supports various models across different environments.
The vLLM project, which originated from the University of California, Berkeley in 2023, forms the backbone of this offering. It supports high-throughput generative AI inference and multi-GPU model acceleration. Red Hat’s adoption of vLLM as part of its solution underscores its role as a standard for future AI inference innovations.
Key industry figures have expressed support for Red Hat’s latest venture. Ramine Roane from AMD highlighted their collaboration with Red Hat to provide efficient generative AI solutions using AMD Instinct GPUs. Jeremy Foster from Cisco noted that the server offers speed, consistency, and flexibility necessary for modern AI workloads. Intel’s Bill Pearson remarked on their excitement about integrating Intel Gaudi accelerators with the server to enhance performance and efficiency.
NVIDIA’s John Fanelli also commented on the potential benefits: “With open, full-stack NVIDIA accelerated computing and Red Hat AI Inference Server…developers can run efficient reasoning at scale across hybrid clouds.”
Red Hat aims to simplify deploying generative AI through this innovation while supporting third-party platforms for greater flexibility.
