Comparative Performance Analysis of Small Mixture-of-Experts (MoE) Models
A recent technical evaluation conducted via the Atomic Agent v0.6.10 framework has provided a comparative performance analysis of three newly released small Mixture-of-Experts (MoE) models: Flint 40B-A2.5B, Ornith 1.5 35B-A3B, and Nemotron 3.5 Lightning 30B-A3B. The testing, reported by a developer on Reddit, utilized a complex coding task involving Three.js to generate a 20-second Halloween-themed animation, requiring the production of approximately 1,000 to 1,700 lines of code across multiple files.
Benchmarking Methodology and Setup
The testing environment utilized a single RTX PRO 6000 Blackwell 96 GB GPU per model, running Q8_0 GGUF quantizations via llama.cpp. The models operated as “workers” in a multi-agent setup, where an Anthropic Opus 5.5 model acted as the central planner. The task was executed in a single turn without human intervention, with three workers running in parallel for each model.
Efficiency and Execution Metrics
According to the data released by the tester, raw decoding speed did not directly correlate with overall task completion efficiency. While Nemotron 3.5 Lightning demonstrated the highest raw performance metrics—including a decode speed of 295 t/s and a prefill rate of 7,670 t/s—it proved to be the least efficient in an agentic workflow. Nemotron required 369 requests to the model and generated 397K local tokens, resulting in a total Opus spend of $0.95. The model also required six planning waves to complete the task, with a reported edit success rate of approximately 17%.
In contrast, Flint 40B-A2.5B emerged as the most efficient model across several key metrics. It achieved the shortest wall time at 7:36 and required only 164 requests. Flint generated 74K local tokens and required only three planning waves, resulting in the lowest Opus expenditure at $0.41. The model required only six fixes to reach a functional state, primarily consisting of minor syntax errors such as typos and undeclared variables.
Ornith 1.5 35B-A3B occupied the middle ground in terms of wall time (9:04) and Opus spend ($0.49), though the tester noted an instance where an Ornith worker reportedly overwrote a 367-line file with an ellipsis (...), a failure mitigated by the presence of the other two parallel workers.
Key Findings in Agentic Workflows
The comparison highlights a significant discrepancy between theoretical token throughput and practical application in agentic coding. The data suggests that the “worker” model significantly influences the cost of the “planner” model; the Opus spend increased from $0.41 to $0.95 simply by switching the underlying worker model.
The tester noted that the primary cost driver in these workflows is the volume of data the planner must read rather than the generation of new code, with input-to-output ratios ranging from 48x to 76x. While Nemotron offered superior raw speed, its tendency to repeat failed edits—such as repeatedly adding stray braces to a file—resulted in a significantly longer total execution time compared to Flint.