KoboldCpp has fixed a bug that stopped its Mac build from using GPU acceleration. The fix is included in version 1.123.1, with both the maintainer and the user who reported the problem confirming it was…
Local AI models are getting more efficient, and two releases this week show how reducing memory requirements can help people run more capable models on their existing hardware. PrismML’s new Bonsai 2 models use ternary…
llama.cpp 0.7.0 is out with a GPU expert cache for mixture-of-experts models that do not fit entirely in graphics memory. Released on October 15, the update includes a way to keep selected experts close to…
Running local AI on a GPU with limited VRAM already involves compromises. You might need a smaller quantization, a shorter context window, or even move some model layers into system RAM. Speculative decoding can help…
One of the more intriguing ways to speed up local AI is speculative decoding. People with plenty of VRAM report excellent results, sometimes generating tokens several times faster than usual. Things get trickier on 8GB.…
Google DeepMind’s new EmbeddingGemma 2 is a small 740 million parameter model made for running directly on everyday devices like phones and laptops. Instead of focusing only on text, it can work with text, code,…
PewDiePie has revealed Ajax, a local AI model he's building as a personal assistant for his Odysseus workspace. It's designed to run on your own hardware, work with your everyday tools, and refuse fewer requests.…
There’s a new AI trend where people are merging games, and the results look incredible. Legal and technical issues aside, this article explains how AI is being used to merge games and whether local AI…
Why can a 16 GB model run on an 8 GB GPU? Quantization. This article shows how it works with a small Python script, then explains the GGUF file format and what the name Q4_K_M…