Advancements in Low-Memory LLM Training
New research shared on GitHub and discussed via Reddit suggests a potential shift in the accessibility of Large Language Model (LLM) training, specifically regarding the hardware requirements for consumer-grade GPUs. An AI researcher has released code for a new optimizer designed to reduce the VRAM required during training.
According to the researcher’s documentation, the optimizer aims to reduce memory consumption from the approximately 16 bytes per weight required by the standard AdamW optimizer to as little as 1 byte or less per weight. This is achieved by focusing on ternary networks, a method that aims to match the quality of full-precision models at scale.
The researcher claims that this reduction allows for the training of 10-20 billion parameter models on consumer hardware. Specifically, the researcher reported fitting the training of a 27B parameter model on a single 24GB NVIDIA RTX 3090 and a 23B model on a 16GB GPU. The resulting models reportedly cost approximately 1.6 bits per weight, which is significantly smaller than standard 16-bit or Q8 quantization.
In unverified tests involving the conversion of existing models, the researcher reported that a quantized and trained Qwen 3.8 27B model achieved a score of 0.895 on the GSM8K benchmark, comparable to full-precision versions, and 0.84 on HumanEval, compared to 0.93 for full precision. The model reportedly ran at approximately 2^3 tokens per second on a 12GB laptop GPU. However, the researcher noted that these models currently exhibit limitations, such as looping behavior and difficulty retaining specific facts. The code for this training method is currently available for non-commercial use.
Financial Volatility and Infrastructure Risks in AI
As the demand for AI compute continues to drive massive capital expenditures, analysts are raising questions regarding the long-term sustainability of the current infrastructure model.
CoreWeave Inc., a major provider of GPU-accelerated cloud services, has recently reported revenue growth and lifted its guidance, which initially eased concerns regarding a potential AI slowdown. However, some market observers remain cautious due to the high debt levels associated with such rapid expansion.
Critics point out that a significant portion of the debt used to fund these operations is secured by infrastructure, specifically GPUs, which are rapidly depreciating assets. There are concerns that the business models of major providers like CoreWeave Inc. rely heavily on the ability of large-scale customers, such as OpenAI and Anthropic, to meet massive multi-billion dollar commitments. If the projected productivity gains from AI do not materialize as expected, the ability of these companies to service their debts could become a point of systemic risk.
Forensic Technology Procurement in India
In a separate development regarding digital privacy and law enforcement, procurement records from the Delhi Police Crime Branch and the Delhi Government’s Forensic Science Laboratory have revealed significant investments in mobile data extraction tools.
According to reports from Internet Freedom Foundation (IFF), the Delhi Police recently concluded a ₹4.59 crore contract for two units of MSAB XRY Pro, a Swedish-made forensic tool. The procurement was conducted via a single-seller bid, with the justification that it would not be in the public interest to use an advertised tender.
The technical specifications for these tools include the ability to extract data from over 100 cloud services using stored login tokens. This includes access to platforms such as Google, iCloud, WhatsApp, and various social media services. Furthermore, the specifications require capabilities for “brute-force” attacks on encrypted backups and the ability to bypass screen locks on specific mobile devices. The tools also include advanced AI-driven features, such as facial recognition and the ability to automatically categorize images into categories including “extremism,” “drugs,” and “weapons.”
