Choosing a model
| Name | Memory | Download | What it is |
|---|---|---|---|
| tiny | 2 GB | 0.5 GB | Qwen2.5 0.5B Instruct, 4-bit |
| small | 4 GB | 1.1 GB | Qwen2.5 1.5B Instruct, 4-bit (the default for most laptops) |
| medium | 8 GB | 3.8 GB | Qwen2.5 7B Instruct, 3-bit-class (Q3_K_M), tuned for 8 GB computers |
| standard | 12 GB | 4.7 GB | Qwen2.5 7B Instruct, 4-bit (Q4_K_M) |
OwnLLM picks the best installed model that fits your computer and the memory that is free right now, so it does not slow your machine down. Pin one with ownllm ask --model medium ... (the same model and question give the same answer).
Why medium is Q3_K_M (the 8 GB decision)
A 7B model is just small enough for an 8 GB computer if its file is small. We compared the three realistic 7B files with the same engine settings on the same machine:
| 7B file | Size | Memory in use | Perplexity ↓ | 40-question check |
|---|---|---|---|---|
| Q3_K_M | 3.81 GB | 3.83 GB | 7.36 | 36 / 40 |
| Q4_0 | 4.43 GB | 4.41 GB | 7.00 | 37 / 40 |
| Q4_K_M | 4.68 GB | 4.58 GB | 6.86 | 37 / 40 |
The raw output of this run, and the script that produced it, are in docs/benchmarks/ and scripts/bench_models.py.
*Perplexity* was measured on 60 passages of a public-domain novel (lower is better). The 40-question check is arithmetic, facts, instruction following and short reasoning, answered with no randomness. One question is within noise, so treat it as "no obvious difference", not a ranking. On the standard measure Q3_K_M is about 5% worse than Q4_0 and 7% worse than Q4_K_M.
We chose Q3_K_M for 8 GB because Q4_0 needs about 0.6 GB more, and on an 8 GB computer the operating system and your other programs already use 3 GB or more. Staying out of swap matters more than a 5% difference in perplexity. If you have the room, standard is better.
What makes it run on 8 GB
- A compressed (8-bit) attention cache, flash attention, small working batches, one conversation at a time.
- A 4,096-token context on computers with 8 GB or less (larger on bigger machines).
- The model file is memory-mapped, so the system loads only what it needs.
- Memory-aware model choice, and
ownllm doctorshows how much memory is free.
Honest limits. These numbers were measured on a Mac with plenty of memory, using the engine's own memory report, not on a physical 8 GB laptop. Speed on a thin 8 GB laptop will be much lower than the numbers you may see elsewhere; expect a few words per second. If it feels slow, ownllm get small.
Same answer every time ("deterministic mode")
By default answers use no randomness (temperature 0, a fixed seed) and the server's prompt cache is off, because we measured that a cache can change the arithmetic between requests and so the wording. With these settings the same model, same question, same settings and the same computer gave the same answer on every run in our tests. This makes results *repeatable*, which helps checking and testing. It does not make an answer correct. For more variety, set OWNLLM_CREATIVE=1 or use ownllm ask --creative.