Technical Discrepancies in Local LLM Inference

A recent report from a user on Reddit has highlighted performance variations between llama.cpp and Ollama when running the “Huihui-Qwen3.6-35B-A3B-abliterated” model. According to the user’s benchmarks on an Intel i7-13th gen system with an Nvidia RTX 4070 (12GB VRAM), llama.cpp achieved a generation speed of 55 tokens per second (tps), while Ollama reached 61 tps using the same prompt.

Technical discussions following the report suggest several potential causes for this discrepancy. Community contributors noted that llama.cpp requires precise configuration of context length and thread allocation to match Ollama’s default settings. Specifically, users pointed out that llama.cpp may default to a different context window than Ollama’s 4k default, and that the -t flag in llama.cpp refers to CPU offloading rather than GPU threads. Experts in the thread suggested that performance could be further optimized through speculative decoding or by ensuring the model is fully saturated within the available VRAM.

Performance Gains via PCI-E Peer-to-Peer Enabling

In a separate technical demonstration, a user reported significant performance improvements when enabling PCI-E Peer-to-Peer (P2P) communication for consumer-grade Nvidia GPUs. Testing was conducted on a multi-GPU setup utilizing four Nvidia RTX 5060 Ti (16GB) cards.

The benchmarks, conducted using the llama-benchy tool on a Qwen/Qwen3.6-27B-FP8 model, showed that enabling P2P increased prefill (PP) speeds by approximately 25%. Without P2P, the prefill speed for a 2048 token prompt was recorded at 1648.96 t/s; with P2P enabled, this rose to 2305.20 t/s.

The user noted that achieving these results requires enabling Resizable BAR in the BIOS and using patched Linux kernel modules to bypass Nvidia’s driver-level restrictions on P2P for consumer cards. While the results show a measurable increase in throughput, the process requires specific hardware support and non-standard software configurations.

Model Pruning and Size Reduction in Kimi K3

New developments in model optimization have emerged regarding the Kimi K3 model. According to reports on HuggingFace and Reddit, a modified version of the Kimi K3 (Unsloth) IQ2-XXS model has been released, which significantly reduces the model size from 711GB to 478GB.

The reduction was achieved by pruning the “multi-lingual fat” of the model, effectively removing non-English language components to retain only English-language intelligence. While the developer claims the model maintains its high intelligence levels, some community members have expressed skepticism, noting that language capabilities in Large Language Models (LLMs) are often entangled and that removing specific “experts” in a Mixture-of-Experts (MoE) architecture may cause unforeseen degradation in reasoning or coding tasks.

Preliminary testing by a user using the SWE-Lancer benchmark indicated that while the 478GB version was able to solve specific tasks where other 2-bit versions failed, the inference speed was slow, averaging 2.5 hours per task due to the complexities of streaming the model from an SSD.