NVIDIA HGX B300 Benchmarking for AI Infrastructure: Evaluating Performance, Power, and AI Workload

The NVIDIA HGX B300 is an eight-GPU AI server platform based on the company’s Blackwell Ultra architecture. It is designed with a stronger emphasis on AI inference performance and introduces a new set of performance, power, thermal, and infrastructure considerations for organizations planning large-scale AI deployments.

TechInsights conducted its own benchmarking of an ASRock 8U16X-GNR2-B300 rackmount server, which is fitted with eight NVIDIA B300 accelerators on an HGX B300 NVL8 baseboard. The study examines the server's performance when running workloads that are representative of actual production conditions, such as AI inference, training, computing, and interconnect tests; it also looks at power consumption and thermal performance at the node level.

To understand how a complete server performs under continuous workloads, infrastructure teams also need to know about throughput, latency, power demand, cooling requirements, and multi-GPU scaling. We carried out tests using a single physical HGX B300 server node; the AC power is measured at the wall for both of the power-supply inputs, the actual temperatures are recorded by means of external thermocouples, and data on GPU power consumption, thermal performance, clock speed, and utilization are collected during the workloads. This enables evaluation of the B300 against metrics relevant to real-world deployment conditions, such as performance per watt and energy-to-solution.

Figure 1 – NVIDIA HGX B300 server benchmarking covering AI inference, power, thermal, and NVLink performance (Source: TechInsights)

Benchmarking AI Inference and Training

A sweep covering DeepSeek-R1, GLM-5, and Qwen3.5 was carried out using Unofficial InferenceX for multiple models and precision levels, the precision levels being FP4, FP8, and BF16; measurements were taken for token throughput, time to first token, time per output token, inter-token latency, and the distributions of end-to-end latency.

TechInsights has also conducted MLPerf Inference tests on DeepSeek-R1, GPT-OSS-120B, and Llama 2 70B. The training workloads included Llama 2 70B LoRA fine-tuning and Llama 3.1 8B pre-training. It is important to note that inference, fine-tuning, and pre-training place different demands on the system, so a single benchmark is not sufficient to capture all of them.

Measuring Server Compute and Interconnect Performance

The report also looks at the compute and communication layers, which support multi-GPU workloads. The GPU microbenchmarks assess the sustained FP16 training, FP16 inference-decode, and FP64 scientific throughput per device. Interconnect testing measures the NVLink bus bandwidth for the various NVIDIA Collective Communications Library (NCCL) collective patterns. These tests help explain how the eight GPUs exchange data and how interconnect behavior might affect workload scaling across the node.

Why is server benchmarking important for AI infrastructure?

Datacenter architects may use the performance, power, and thermal figures at the node level to aid cluster planning and determine capacity. AI infrastructure engineers can compare the throughput and latency of various inference and training workloads. Meanwhile, server original equipment manufacturers (OEMs) and original design manufacturers (ODMs) can evaluate the platform and baseboard implementations, while suppliers of capital equipment can use the findings to better understand the power and thermal requirements of next-generation AI servers.

Marketing and strategy teams can also gain visibility into independent data on per-watt performance and energy-to-solution. This information can be used to conduct a competitive evaluation and establish product positioning without relying solely on vendor specifications.

What the Full Analysis Includes

The full report includes the system configuration, test methodology, workload-level results, latency distributions, token throughput measurements, power and thermal telemetry, GPU compute results, and NVLink interconnect data.

TechInsights is also conducting related analysis of the B300 processor, interconnect and packaging architecture, and HBM3E memory modules used in the server. Together, these studies extend the analysis beyond server-level benchmarking to the components and integration technologies inside the platform.