Hardware Innovations in Multi-GPU Configurations

Recent technical discussions on community forums, including Reddit, have highlighted emerging methods for scaling GPU density using PCIe switches. This hardware configuration allows multiple graphics processing units (GPUs) to be connected to a single PCIe x16 slot, effectively bypassing the physical limitations of standard motherboard layouts.

One documented setup involves the integration of four Radeon AI PRO R9700 cards and two RTX 3090 units through a PCIe switch. According to reports from users on Reddit, this configuration enables the system to run multiple cards at high speeds from a single upstream port. However, technical observers have noted potential limitations, specifically regarding Peer-to-Peer (P2P) capabilities on certain hardware and bottlenecks occurring at the PCIe-to-CPU interface during hybrid inference tasks. Some users have suggested that while switches facilitate high internal traffic speeds, they may require specific BIOS configurations to manage address space limitations.

Performance Benchmarks for Large-Scale Inference

In a separate technical report shared on Reddit, a user detailed the performance of a custom-built multi-user inference server designed for business applications involving sensitive data. The system, which utilizes an AMD EPYC 7402P CPU and an Asrock ROMED8-2T motherboard, incorporates four Radeon AI PRO R9700 GPUs. The total build cost was reported to be approximately $16,000 CAD.

The benchmarks, conducted using the Radiance software (v1.17.0) on Ubuntu 26.04.1, focused on the Qwen 3.8-Flash-Next model. The results, while unverified by independent third-party testing, indicate significant throughput for long-context concurrency:

  • Prefill Performance: The system achieved a total prefill rate of approximately 6,800 tokens per second (tok/s), regardless of the number of users.
  • Inference Speed: For a single user with a 128K context, the writing speed was recorded at 189 tok/s. Under a heavy load of eight concurrent users with 128K contexts, the writing speed per user was measured between 70 and 83 tok/s.
  • Latency: For follow-up queries where 94% to 99% of the data is retrieved from the cache, the first-token latency remained between 0.2 and 2.5 seconds, even with eight concurrent users.

Thermal and Power Considerations

The hardware setup required significant power management, utilizing a Seasonic Prime TX-1600W power supply. The user reported that all four Radeon cards were power-limited to 210W. During peak workloads, specifically during the prefill stage of 8 x 128K tokens, the middle two cards reached hotspot temperatures of 91°C and memory temperatures of 92°C.

The build was housed in a Phanteks Enthoo Pro 2 Server case, which the builder noted provided sufficient space for the massive components and necessary airflow, despite the high thermal output of the GPUs. The user indicated that the primary objective of the build was to provide a high-speed, secure local inference environment to mitigate the risks associated with using external AI services for sensitive customer data.