A 27B LLM model that fits in just 4GB of VRAM sounds impossible, but here we are
www.makeuseof.com/27b-llm-model-fits-4gb-vram/
A 27B model squeezed down to 1 bit per weight
Same Qwen underneath, with a trick that takes 54GB down to 3.9GB
- How much brain survives the squeeze
Math and coding hold up, instruction following takes the hit - Fast on a GPU, grim on a CPU
Mac numbers, an RTX 5070 Ti, and a Ryzen that struggles - Who should actually download this
8GB and 12GB GPUs and Apple silicon, mostly