Will it run? Check if a local LLM fits your GPU

Pick your graphics card or Mac, a model, a quant and a context length. The checker adds up the memory llama.cpp needs and shows the sum line by line. If we have run the model ourselves, our measured speeds appear under the result.

Updated 22 models85 cards and Macs

How the checker works

A model running in llama.cpp needs memory for three things, and the checker adds them up:

  • Weights: the size of the GGUF file for the quant you picked, as listed on Hugging Face. With every layer on the GPU, all of it sits in GPU memory.
  • KV cache: the keys and values of every token in the context. Each layer that keeps a cache stores 2 × KV heads × head size values per token, at 2 bytes each in f16. Some models cache fewer layers. Gemma keeps most of its layers to a short sliding window, and Qwen3.5 and LFM3 mix in layers that hold a small fixed state in place of a cache.
  • Overhead: the working buffers llama.cpp allocates. The biggest holds the output scores for a 512-token batch, vocabulary size × 512 × 4 bytes, so a model with a 262,144-token vocabulary needs 0.5 GiB for it alone. We add 256 MiB for the rest.

We checked the formula against the 20 runs in our benchmark table, measured on an 8 GB RX 5700 XT with flash attention on and up to 4,736 tokens of context. On every run it came out above the most VRAM the model used, by 160 to 730 MiB. The gap is widest for models with a big vocabulary and separate input and output embeddings: on a graphics card, llama.cpp keeps the input embedding table in system RAM, and the formula counts it anyway to stay on the safe side.

The verdict compares the total with the memory available:

  • Fits: the total is at most 90% of it.
  • Tight fit: it fits with less than 10% to spare. Close other programs that use the GPU, such as a browser or a game, or shorten the context.
  • Does not fit: llama.cpp would have to put some layers in system RAM, which is much slower. The checker suggests the largest quant and the longest context that do fit.

Macs

On a Mac the GPU shares memory with everything else, and macOS caps how much it can take. On our 8 GB M1 MacBook Air, llama.cpp reports 5,461 MiB for the GPU, two thirds of the total. The checker uses two thirds for Macs with up to 36 GB and three quarters above that. We have not measured a Mac with more than 8 GB, so the three-quarters figure is the default that owners of larger Macs report. Run llama-cli --list-devices to see your own number. sudo sysctl iogpu.wired_limit_mb=<MB> raises the cap until the next restart, and leaves less memory for everything else.

What it leaves out

  • Speed. The checker only works out memory. The measured runs under the result come from our own card, and yours will differ.
  • Image input. Vision models load a separate projector file (mmproj) next to the model. For Qwen3.5 9B that file is 0.92 GB in F16.
  • Other programs. Your desktop and browser also take memory on the card, which matters most when the verdict is a tight fit.
  • Long contexts and flash attention off. Our runs stopped at 4,736 tokens, with flash attention on. llama.cpp turns it on by default when the card supports it. With it off, llama.cpp allocates a buffer for the attention scores that grows with the context, and the checker leaves it out.
  • Non-default settings. A bigger batch (-ub) raises the overhead. The checker assumes llama.cpp's default of 512.

Models in the checker

File sizes and attention layouts come from the Hugging Face Hub, fetched on : the size of each quant's GGUF file, and the layer and head counts from the model's config.json. Card memory comes from NVIDIA, AMD and Intel's published specifications.

ModelParameters (B)Trained contextQuants
LFM3 1.7B1.73262,144Q4_K_M, Q5_K_M, Q6_K, Q8_0
Llama 3.2 3B Instruct3.21131,072Q4_K_M, Q5_K_M, Q6_K, Q8_0
Phi-4 mini instruct3.84131,072Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Gemma 3 4B it3.88131,072Q2_K, Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Qwen3.5 4B4.21262,144Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Mistral 7B Instruct v0.37.2532,768Q2_K, Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
DeepSeek-R1 Distill Qwen 7B7.62131,072Q2_K, Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Llama 3.1 8B Instruct8.03131,072Q2_K, Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Qwen3 8B8.1940,960Q2_K, Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Qwen3.5 9B8.95262,144Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Ternary Bonsai 2 9B8.95262,144Q2_0
Gemma 4 12B it11.91262,144Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Qwen3 14B14.7740,960Q2_K, Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Phi-4 14B14.6616,384Q2_K, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
gpt-oss 20B (mixture of experts)20.91131,072Q2_K, Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Mistral Small 3.2 24B23.57131,072Q2_K, Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Gemma 4 26B-A4B it (mixture of experts)25.23262,144Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Qwen3.8 27B27.32262,144Q4_K_M, Q5_K_M, Q6_K, Q8_0
Ternary Bonsai 2 27B26.9262,144Q2_0
Qwen3 30B-A3B Instruct 2507 (mixture of experts)30.53262,144Q2_K, Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Qwen3.6 35B-A3B (mixture of experts)34.66262,144Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0
Llama 3.3 70B Instruct70.55131,072Q2_K, Q3_K_S, Q3_K_M, Q4_K_M, Q5_K_M, Q6_K, Q8_0