Localized AI Infrastructure Deployment for Data Extraction

A business owner operating USA Arms has reported successful deployment of a high-VRAM multi-GPU server cluster designed to facilitate local large language model (LLM) inference. The move to localized hardware follows reported restrictions from hosted AI providers regarding the processing of specific industry-related data.

According to a detailed report shared on Reddit, the infrastructure consists of three identical server units, each equipped with four ASRock Radeon AI PRO R9700 Creator GPUs, providing a total of 384 GB of VRAM across the cluster. The hardware configuration utilizes AMD EPYC 7452 processors and ASRock Rack ROMED8-2T/BCM motherboards. The total cost per machine is approximately $10,733.91, including tax and shipping.

Performance Benchmarks and Model Efficiency

The deployment utilizes the vLLM-Radiance engine (vLLM 0.30.0, ROCm 7.14) to run Qwen/Qwen3.8-27B-FP8. In testing conducted by the operator, the system demonstrated significant throughput capabilities for structured data extraction.

Key performance metrics for a single unit include:

  • Single User (MT-Bench): 53.1 tokens per second (tok/s) with a Time to First Token (TTFT) of 125 ms.
  • 16 Users (MT-Bench): 484 tok/s aggregate throughput.
  • 32 Users: 603 tok/s aggregate throughput.

The operator noted that the system’s performance is particularly efficient for JSON extraction, where draft acceptance rates reached 83%. When testing long-context windows, the system achieved a prefill rate of 7,044 tok/s for 4K context, which decreased to 2,719 tok/s as context reached 256K.

Comparative Analysis of Precision Formats

The deployment included a comparison between FP8 and MXFP4 precision formats to determine the optimal balance between accuracy and throughput for business operations.

The results indicated that while MXFP4 provided higher throughput—processing 4,819 validated jobs per hour compared to 4,477 for FP8—the FP8 format maintained a higher accuracy rate (98.0/100 vs 95.3/100). The operator concluded that FP8 was the preferred configuration to mitigate the risk of “false facts” entering the business’s data system.

The operator also compared the cost-efficiency of this custom build against a fleet of Apple Mac Studios. The report suggests that scaling the Mac Studio approach to match the throughput of the current GPU cluster would have required approximately 41 additional units, representing an estimated capital expenditure of $280,000.

Hardware and Operational Integration

The custom builds integrate SilverStone HELA 2050R Platinum power supplies and SilverStone RM44 4U chassis. Beyond AI inference, the operator reported that the servers successfully handle CI/CD (Continuous Integration/Continuous Deployment) workloads, such as Rust builds, with only a 3-8% reduction in build throughput.