Local LLM benchmarks on an 8 GB GPU
Tokens per second and peak VRAM for 10 open models, measured with llama-bench on an 8 GB Radeon RX 5700 XT. Each row is one dated run, and the CSV adds the exact command behind it.
What fits in 8 GB
In the 18 November 2026 run, every model up to 9B parameters at Q4_K_M fit on the card with all its layers on the GPU. The largest of them, Qwen3.5 9B, peaked at 5,610 MiB with 4,096 tokens already in the context. We ran Qwen3 14B only at Q3_K_S, to show where 8 GB runs out: it peaked at 7,118 MiB of the 8,176 MiB the card reports and generated 28 to 31 tokens per second. LFM3 1.7B was the fastest at 178 tokens per second. For a model or card we have not run, the Will it run? checker works out the memory it needs.
Generation speed mostly follows file size. With an empty context, Gemma 3 4B and Phi-4 mini, both 2.5 GB files, generate 95 and 104 tokens per second; the 7B and 8B models, 4.4 to 4.9 GB, generate 63 to 68. Ternary Bonsai 2 9B breaks the pattern: its file is 2.6 GB, about the size of Gemma's, yet it generates 60 tokens per second, close to the 7B models.
Results
| LFM3 1.7B | Q4_K_M | 1.10 | 0 | 2,443 ±3.2 | 177.7 ±4.4 | 210 | 1,335 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| LFM3 1.7B | Q4_K_M | 1.10 | 4,096 | 2,060 ±35.5 | 171.8 ±3.1 | 249 | 1,397 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Phi-4 mini instruct | Q4_K_M | 2.49 | 0 | 1,139 ±0.4 | 104.1 ±0.3 | 449 | 2,863 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Gemma 3 4B it | Q4_K_M | 2.49 | 0 | 1,193 ±0.9 | 95.4 ±0.8 | 429 | 2,986 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Phi-4 mini instruct | Q4_K_M | 2.49 | 4,096 | 759 ±6.9 | 90.9 ±0.2 | 675 | 3,396 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Gemma 3 4B it | Q4_K_M | 2.49 | 4,096 | 1,076 ±3.8 | 88.8 ±0.4 | 476 | 3,203 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Qwen3.5 4B | Q4_K_M | 2.74 | 0 | 1,025 ±0.7 | 85.2 ±0.6 | 500 | 3,192 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Qwen3.5 4B | Q4_K_M | 2.74 | 4,096 | 913 ±3.9 | 82.4 ±0.2 | 561 | 3,340 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| DeepSeek-R1 Distill Qwen 7B | Q4_K_M | 4.68 | 0 | 595 ±0.5 | 68.1 ±0.2 | 861 | 4,536 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Mistral 7B Instruct v0.3 | Q4_K_M | 4.37 | 0 | 567 ±0.3 | 67.8 ±0.2 | 903 | 4,294 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Llama 3.1 8B Instruct | Q4_K_M | 4.92 | 0 | 567 ±0.3 | 63.5 ±0.3 | 903 | 4,766 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| DeepSeek-R1 Distill Qwen 7B | Q4_K_M | 4.68 | 4,096 | 473 ±2.4 | 63.0 ±0.2 | 1,083 | 4,772 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Mistral 7B Instruct v0.3 | Q4_K_M | 4.37 | 4,096 | 426 ±1.9 | 60.8 ±0.2 | 1,202 | 4,826 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Ternary Bonsai 2 9B | Q2_0 | 2.56 | 0 | 609 ±0.3 | 59.7 ±0.1 | 841 | 2,835 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Ternary Bonsai 2 9B | Q2_0 | 2.56 | 4,096 | 564 ±1.8 | 58.1 ±0.5 | 908 | 2,979 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Llama 3.1 8B Instruct | Q4_K_M | 4.92 | 4,096 | 426 ±2.0 | 57.2 ±0.1 | 1,203 | 5,298 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Qwen3.5 9B | Q4_K_M | 5.68 | 0 | 555 ±0.4 | 53.4 ±0.2 | 922 | 5,462 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Qwen3.5 9B | Q4_K_M | 5.68 | 4,096 | 519 ±2.2 | 51.9 ±0.3 | 987 | 5,610 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Qwen3 14B | Q3_K_S | 6.66 | 0 | 263 ±0.3 | 31.0 ±0.1 | 1,947 | 6,458 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) | |
| Qwen3 14B | Q3_K_S | 6.66 | 4,096 | 210 ±0.5 | 28.5 ±0.0 | 2,440 | 7,118 of 8,176 | RX 5700 XT 8 GB (VM gpu-debian) Vulkan, llama.cpp b12445 (ef3848f9d) |
How we tested
The card is an AMD Radeon RX 5700 XT with 8 GB of GDDR6, passed through to a Debian virtual machine on Proxmox. Vulkan reports 8,176 MiB of it. It runs our own undervolt: the core tops out at 2,000 MHz at 1,100 mV, with a 195 W power cap. A card at stock settings may score differently.
The software is llama.cpp build b12445 (commit ef3848f9d) with the Vulkan backend, on the RADV driver from Mesa 26.1.6. Every row names its backend and build, because llama.cpp speeds change from one release to the next. Each model ran twice, once with an empty context and once with 4,096 tokens already in it:
llama-bench -m <model>.gguf -ngl 99 -fa 1 -p 512 -n 128 -d 0 -r 3 -o json
llama-bench -m <model>.gguf -ngl 99 -fa 1 -p 512 -n 128 -d 4096 -r 3 -o json-ngl 99 puts every layer on the GPU and -fa 1 turns on flash attention. Each test repeats three times. The columns mean:
- Prompt t/s: how fast the model reads a 512-token prompt.
- Generation t/s: how fast it writes 128 tokens.
- Context: tokens already in the context when the test starts. At 4,096 the model works behind a long chat history, which is slower.
- 512-token prompt: 512 divided by the prompt speed, so the time to read a 512-token prompt. It leaves out loading the model, and at 4,096 tokens of context it covers only the 512 new tokens. It is not the time to first token you see in a chat app.
- Peak VRAM: the highest reading of the card's used memory (
mem_info_vram_usedfrom the amdgpu driver), sampled every 200 ms during the run. It counts the whole card, including the 11 MiB in use at idle.
The GGUF files came from Hugging Face, and before each run we checked the file's sha256 against the hash the Hub publishes. Most models are Q4_K_M. Qwen3 14B is Q3_K_S, to show where 8 GB runs out. Ternary Bonsai 2 9B ran as Q2_0 on upstream llama.cpp; PrismML's own PQ2_0 files need their llama.cpp fork and are not in the table. We skip any file over 7 GiB, since it would leave no room for the context on an 8 GB card.
llama-bench feeds the model synthetic tokens instead of real text. Speed does not depend on what the words say, but the table tells you nothing about answer quality. While a model runs, the driver also reports 150 to 850 MiB of GTT (system RAM the GPU can address), which llama.cpp uses for buffers on the host side; every layer stayed in VRAM, and no run touched swap.
For the software side of running models at home, see Self-hosting LLMs in 2026: a guide to privacy, hardware and cost and how to build a privacy-first AI server.
Download the data
The full data is one CSV file with one row per run: the model file, its size and sha256, every driver reading, and the exact command. Download bench.csv. You can reuse it under CC BY 4.0: credit Kextcache and link to this page.
Changelog
- : first run. 10 models on the RX 5700 XT with llama.cpp b12445, each with an empty context and with 4,096 tokens in it, 20 rows.