How the Speed Ratings Work
Most ratings on AIHWPrices.com are calculated from a device's memory bandwidth and usable memory, then checked against real benchmark results. Phone ratings are hand-entered estimates (see Phones below). This page explains the method, what it gets right, and where it is only an estimate.
How Speed Ratings Are Estimated
- What's rated: A single device running one reference model per tier at 4-bit quantization (Q4_K_M / GGUF) with a baseline context of 2048 tokens.
- Reference models: 3B (Small Dense), 8B (Consumer Dense), 32B (Mid Dense), 30B-A3B (Mid MoE), 70B (Large Dense), gpt-oss-120b (Large MoE) and 671B (Enterprise).
- Rating levels: Very Fast is 50+ tokens/second, Fast is 25-49, Medium is 10-24 and Slow is under 10.
- How speeds are estimated: Token generation speed is calculated from the device's memory bandwidth and the model's size: time per token = data read ÷ (bandwidth × efficiency) + a small fixed overhead. The efficiency and overhead are fitted separately for NVIDIA/AMD GPUs, Apple Silicon and unified-memory mini PCs and laptops, using measured llama.cpp results (CUDA, Vulkan, ROCm, Metal).
- How accurate that is: Against 43 published benchmark results the formula is typically within about 5% and at worst about 20%. The calculation is automatic and is checked against these benchmarks every time the site is built.
- What is less certain: Most benchmarks used are 7B models. A few larger results (a 120B MoE on three unified-memory systems and a 70B dense model on two RTX 3090s) check the larger tiers. GPUs that spill into system RAM, and the 671B tier, have no benchmark behind them and are extrapolated from the same formula.
- The "~" mark: A rating marked ~ is within about 10% of the cutoff between two tiers and could land in the neighbouring one.
- GPUs when the model doesn't fit in VRAM: Desktop GPUs are rated assuming a typical PC with at least 64 GB of dual-channel DDR5, with part of the model running from system RAM (for MoE models, mainly the experts). Laptops with a discrete GPU use the machine's own RAM size, so a model that spills past that RAM is shown as N/A. These ratings vary widely with RAM speed and the rest of the system, and are the least certain.
- Macs and unified-memory mini PCs: There is no spill to extra RAM, so the model must fit in the device's usable memory.
- N/A means the model does not fit on the device even under these assumptions.
- Clusters: The cluster icon marks devices their maker supports linking together. Cluster speeds are not rated.
- Real-world variation: Actual performance depends on thermals, drivers and the exact quantization.
GPU VRAM Overhead
- For GPUs, the usable memory metric assumes a Windows 11 baseline, which takes 1.5GB - 2.5GB of VRAM for desktop graphics.
- Your active chat history eats another 2GB - 4GB (KV Cache), so always pick models that leave a ~3GB safety buffer below your card's maximum limit.
- Linux users can expect roughly ~2GB of extra headroom.
Unified Memory Systems
- For Apple Silicon Macs, the usable memory metric is the default macOS limit for GPU memory, about 75% of the installed unified memory.
- For Ryzen AI Max+ mini PCs, it is the amount the GPU can use by default (96 GB of a 128 GB system).
- These limits can usually be raised manually, so treat them as safe defaults rather than hard ceilings.
Laptops
- Laptop GPUs (RTX 5090 Laptop, RTX PRO) are power-limited, so real speed varies with the model, cooling and power mode. Their ratings use the same memory-bandwidth formula as desktop cards, but there are no laptop benchmarks behind them, so treat them as the least certain ratings on this site.
- Laptop prices are a typical or starting price for the configuration listed. Other configurations cost more or less.
- NVIDIA has not published the memory bandwidth of its RTX Spark (N1X) chip. The figure used for those laptops (273 GB/s) is an estimate based on the DGX Spark, and their usable memory is the amount the GPU used in reviewer testing.
Phones
- Hand-entered, not calculated: Phone ratings do not use the formula above. They use the same idea (speed is roughly real memory speed divided by the GB read per token), but with the real speed seen in community reports instead of the chip's paper maximum.
- Where the numbers come from: Community llama.cpp and PocketPal benchmark reports. The real memory speeds used are about 40 GB/s for Snapdragon 8 Elite and Elite Gen 5, about 60 GB/s for the iPhone 17 Pro (A19 Pro), about 90 GB/s for the iPhone 18 Pro (A20 Pro) and about 15 GB/s for Google Tensor.
- No benchmarks yet for some chips: The A20 Pro figure is the A19 Pro result scaled by Apple's +50% memory bandwidth claim, and the Tensor G6 figure comes from older Pixel reports. Both will be updated when real results appear.
- Sustained, not burst: Ratings assume speeds after a minute or more of use. Phones heat up and throttle, so short bursts run 20-30% faster.
- The bandwidth column: For phones it is the chip's maximum, calculated from memory type, speed and bus width where the maker does not publish it. In practice phones reach roughly 20-80% of that peak depending on the chip.
- Usable memory: 75% of RAM for Android phones. For iPhones it is about 6.5 GB of the 12 GB, a limit reported by developers and users rather than published by Apple.
- Less certain: Phone ratings carry no "~" marks, and community tests often use faster 4-bit formats than the Q4_K_M standard used elsewhere on this site.
Power Ratings
- The power column is not one measurement.
- For GPUs it is the card's board power rating. For most mini PCs it is the chip's rated power, and for the DGX Spark it is the whole system's peak draw.
- For Apple Macs it is Apple's published maximum power draw measured at the wall (includes power supply losses), shown only where Apple lists that exact chip configuration. Apple does not publish a TDP.
The Benchmarks Behind the Formula
These 43 measured results (token generation, tokens/second) calibrate the formula and are re-checked on every site build. Most are from public llama.cpp benchmark threads; each row links to its source.
| Device | Platform | Model | Backend | Measured (tok/s) | Source |
|---|---|---|---|---|---|
| Apple M5 Ultra 80c | Apple Silicon | Llama 2 7B Q4_0 | Metal | 179.1 | llama.cpp #4167 · build c1d0e7a |
| Apple M5 Max 40c | Apple Silicon | Llama 2 7B Q4_0 | Metal | 119.92 | llama.cpp #4167 · build c1d0e7a |
| Apple M3 Ultra 80c | Apple Silicon | Llama 2 7B Q4_0 | Metal | 92.14 | llama.cpp #4167 · build 8e672ef; thread lists 800 GB/s, Apple spec is 819 |
| Apple M4 Max 40c | Apple Silicon | Llama 2 7B Q4_0 | Metal | 83.06 | llama.cpp #4167 · build 8e672ef |
| Apple M5 Ultra 80c | Apple Silicon | Llama 2 7B F16 | Metal | 73.6 | llama.cpp #4167 · build c1d0e7a |
| Apple M5 Pro 20c | Apple Silicon | Llama 2 7B Q4_0 | Metal | 66.33 | llama.cpp #4167 · build c1d0e7a |
| Apple M4 Pro 20c | Apple Silicon | Llama 2 7B Q4_0 | Metal | 50.74 | llama.cpp #4167 · build 8e672ef |
| Apple M3 Ultra 80c | Apple Silicon | Llama 2 7B F16 | Metal | 39.78 | llama.cpp #4167 · build 8e672ef |
| Apple M5 Max 40c | Apple Silicon | Llama 2 7B F16 | Metal | 37.11 | llama.cpp #4167 · build c1d0e7a |
| Apple M5 | Apple Silicon | Llama 2 7B Q4_0 | Metal | 31.88 | llama.cpp #4167 · build c1d0e7a |
| Apple M4 Max 40c | Apple Silicon | Llama 2 7B F16 | Metal | 31.64 | llama.cpp #4167 · build 8e672ef |
| Apple M4 | Apple Silicon | Llama 2 7B Q4_0 | Metal | 24.11 | llama.cpp #4167 · build 8e672ef (2023, slower than current) |
| Apple M5 Pro 20c | Apple Silicon | Llama 2 7B F16 | Metal | 21.55 | llama.cpp #4167 · build c1d0e7a |
| Apple M4 Pro 20c | Apple Silicon | Llama 2 7B F16 | Metal | 17.18 | llama.cpp #4167 · build 8e672ef |
| Apple M4 | Apple Silicon | Llama 2 7B F16 | Metal | 7.43 | llama.cpp #4167 · build 8e672ef |
| RTX 5090 | GPU | Llama 2 7B Q4_0 | CUDA | 290.02 | llama.cpp #15013 |
| RTX PRO 6000 Blackwell | GPU | Llama 2 7B Q4_0 | CUDA | 274.2 | llama.cpp #15013 |
| RX 7900 XTX | GPU | Llama 2 7B Q4_0 | Vulkan | 191.28 | llama.cpp #10879 |
| RTX 4090 | GPU | Llama 2 7B Q4_0 | CUDA | 186.21 | llama.cpp #15013 |
| RTX 5080 | GPU | Llama 2 7B Q4_0 | CUDA | 181.99 | llama.cpp #15013 |
| RTX 5070 Ti | GPU | Llama 2 7B Q4_0 | CUDA | 176.85 | llama.cpp #15013 |
| RTX 3090 | GPU | Llama 2 7B Q4_0 | CUDA | 158.16 | llama.cpp #15013 |
| RTX 3080 10GB | GPU | Llama 2 7B Q4_0 | CUDA | 139.65 | llama.cpp #15013 |
| RX 9070 XT | GPU | Llama 2 7B Q4_0 | Vulkan | 137.11 | llama.cpp #10879 |
| Radeon AI PRO R9700 | GPU | Llama 2 7B Q4_0 | Vulkan | 133.27 | llama.cpp #10879 |
| Razer Blade 16 (RTX 5090 Laptop 24GB) | GPU | Llama 3.1 8B (MLPerf Client, llama.cpp CUDA) | CUDA | 128.8 | storagereview.com · laptop power limit applies |
| RTX 5070 | GPU | Llama 2 7B Q4_0 | CUDA | 127.54 | llama.cpp #15013 |
| Arc Pro B70 | GPU | Llama 2 7B Q4_0 | Vulkan | 112.02 | llama.cpp #10879 · Intel Vulkan results vary a lot by driver |
| RTX 5060 Ti 16GB | GPU | Llama 2 7B Q4_0 | CUDA | 90.94 | llama.cpp #15013 |
| RTX 3060 12GB | GPU | Llama 2 7B Q4_0 | CUDA | 75.57 | llama.cpp #15013 |
| Arc Pro B60 | GPU | Llama 2 7B Q4_0 | Vulkan | 68.55 | llama.cpp #10879 · Sibling of B70; Intel Vulkan varies a lot by driver |
| RTX 4060 Ti 8GB | GPU | Llama 2 7B Q4_0 | CUDA | 63.86 | llama.cpp #15013 |
| 2x RTX 3090 (layer split) | GPU | Llama 70B Q4_0 (36.20 GiB) | CUDA | 21.17 | llama.cpp #4167 · tg128, Dec 2023. Layer split, so each token still reads through one card's bandwidth at a time; 1x 3090 bandwidth used. |
| HP OmniBook Ultra 16 (RTX Spark N1X) | Unified memory | Qwen3.6-35B-A3B 4-bit (llama.cpp, plain decode) | CUDA | 67.64 | theregister.com |
| NVIDIA DGX Spark | Unified memory | Qwen3.6-35B-A3B 4-bit (llama.cpp, plain decode) | CUDA | 63.77 | theregister.com |
| NVIDIA DGX Spark | Unified memory | gpt-oss 120B MXFP4 MoE (59.02 GiB, ~5B active) | CUDA | 60.57 | llama.cpp #16578 · tg32, build 6922 + kernel 6.17.1, Nov 2025 (same device, later software) |
| DGX Spark | Unified memory | Llama 2 7B Q4_0 | CUDA | 57.21 | llama.cpp #15013 |
| HP OmniBook Ultra 16 (RTX Spark N1X 128GB) | Unified memory | Llama 3.1 8B (MLPerf Client, llama.cpp CUDA) | CUDA | 56.5 | storagereview.com · quant not stated in review; Windows on Arm |
| Ryzen AI Max+ 395 (Strix Halo) | Unified memory | Llama 2 7B Q4_0 | Vulkan | 53.59 | llama.cpp #10879 · Single report |
| HP Z2 Mini G1a (Ryzen AI Max+ 395) | Unified memory | Qwen3.6-35B-A3B 4-bit (llama.cpp, plain decode) | Vulkan/ROCm (not stated) | 50.37 | theregister.com |
| GMKtec EVO-X2 (Strix Halo) | Unified memory | gpt-oss 120B MXFP4 MoE (59.02 GiB, ~5B active) | ROCm | 47.49 | llama.cpp #16578 · tg32, build ~6816, Oct 2025 |
| NVIDIA DGX Spark | Unified memory | gpt-oss 120B MXFP4 MoE (59.02 GiB, ~5B active) | CUDA | 45.87 | llama.cpp #16578 · tg32, build 6816, Oct 2025 |
| NVIDIA Jetson Thor | Unified memory | gpt-oss 120B MXFP4 MoE (59.02 GiB, ~5B active) | CUDA | 42 | llama.cpp #16578 · tg32, build 6771, Oct 2025 |