Local LLM benchmark results
Setup and models
M1 Max: 21 GPU cores, 8 performance CPU cores, 2 efficiency CPU cores. The chip has a theoretical memory bandwidth of 400 GB/s.
LM Studio is used for everything. OMLX on occasion just to see if did better on the MTP variant of Qwen.
Which models do I test?
- Qwen 3.6 is the gold standard for coding, which produces good results but is also slow because it’s a dense model.
- Gemma 4 26B A4B QAT is interesting because it’s one of the first out there using Quantization-Aware Training.
- Ornith is allegedly a “benchmaxed” refinement of Qwen, which has some positive reviews on Reddit.
Bounded read-only benchmark
Each model is run with this same prompt:
pi --provider lmstudio --model $MODELNAME \
--thinking low --tools read,grep,find,ls \
--name "benchmark-$MODELNAME-bounded" \
"Inspect this repository for one real bug or fragile area. \
Make at most 8 tool calls total. Never call the same tool \
with the same arguments twice. Never read the same file twice. \
After finding one credible issue, stop using tools immediately \
and give the final answer. If no issue is confirmed after 8 \
tool calls, stop and report the strongest candidate. Your final \
answer must be under 500 words and contain exactly these\
headings: Location, Evidence, Failure scenario, Root cause, \
Minimal patch. Do not edit files. Do not discuss your search \
process. Do not output internal reasoning."
And here are the results:
| Model | Runtime | TTFT | TPS | Output | Decode time | Outcome |
|---|---|---|---|---|---|---|
| Ornith 35B | LM Studio | 4.46s | 43.9 | 3,273 | 74.6s | Completed, verbose |
| Qwen 3.6 27B non-MTP | LM Studio | 37.85s | 11.0 | 1,012 | 92.3s | Completed, most restrained |
| Gemma 4 26B A4B QAT, tuned | LM Studio | 4.56s | 41.7 | 9,919 | 237.9s | Completed, very verbose |
| Qwen 3.6 27B oQ8 MTP | oMLX | 25.13s | 11.1 | 2,811 | 252.1s | Completed |
| Qwen 3.6 27B MTP Q8_0 | LM Studio | 15.64s | 11.9 | 3,554 | 297.6s | Completed, inefficient |
| Qwen 3.6 27B MLX 8-bit | oMLX | 23.81s | 10.2 | 4,245 | 416.1s | Completed, poor overall |
| Gemma, original settings | LM Studio | 15.09s | 35.3 | 16,070 | 453.6s | Failed to terminate |
| Ornith, original prompt | LM Studio | 18.51s | 40.1 | 19,566 | 487.5s | Hit output limit |
Current conclusion: Surprisingly, Ornith 1.0 35B through LM Studio is the best fit so far. Gemma is correct but inefficient. Ordinary Qwen is restrained in token use, but the objective benchmark exposed a concurrency bug, so it does not technicall qualify as a correct solution.
Objective coding benchmark
The objective benchmark is a deliberately broken Python repository designed to test whether a coding agent can produce a functionally correct patch.
The repository contains a function called get_or_load(key, loader).
The method should return a cached value when one exists.
On a cache miss, the method calls loader(), caches the result, and returns it.
It is deliberately broken under concurrency.
Each model gets the same prompt:
You must preserve the existing API while meeting these requirements:
Concurrent requests for the same missing key invoke the loader exactly once.
All callers waiting for that load receive the same value.
If the loader raises, every waiting caller receives that exception.
A loader failure is not cached, so a later request can retry.
None is a valid value and must be cached.
Different keys can load concurrently.
No external dependencies are added.
The patch should be reasonably small.
Existing tests must be run.
Existing tests must not be weakened just to obtain a pass.
Aside from the included test suite, there is an extra test suite that the model does not have access to. This checks for quality of implementation. For example, it checks if two loaders can execute at the same time for different keys, and whether the key was loaded multiple times even though there was no expiry. It also checks if TTL expiry causes correct behaviour.
Here are the results
| Model | Result | TTFT | TPS | Output | Decode time | Input |
|---|---|---|---|---|---|---|
| Ornith 1.0 35B 8-bit | 5/5 PASS | 21.02s | 47.3 | 7,921 | 167.3s | 78k |
| Gemma 4 26B A4B QAT 4-bit | 5/5 PASS | 16.79s | 31.5 | 30,883 | 981.1s | 363k |
| Qwen 3.6 27B 8-bit | 4/5 FAIL | 46.02s | 7.2 | 3,872 | 540.0s | 59k |
Findings
- Ornith 5/5 hidden checks. Fastest correct result and current winner.
- Gemma: 5/5 hidden checks, but generated 30,883 tokens and repeatedly reconsidered a correct approach.
- Qwen: 4/5 hidden checks. The patch coalesced successful same-key loads but failed to distribute one loader exception to all concurrent waiters. The failing loader ran six times instead of once.
Conclusions
Tokens/second is important. But also is the time to first token (TTFT). Keeping both of these down is essential for iterative agentic coding.
Qwen remains the best philosopher and still excels with output brevity. It did not fully pass the test and i suspect more bespoke tests will be needed to get a better feel about how it interprets vague prompts. More interestingly, MTP only made a difference of 1 tok/sec on this machine. I have seen MTP make a bigger difference in the past but was not able to reproduce that here.
While Gemma QAT looked good at face value, it really struggled to stay focused and kept introducing doubt. Temperature reduction did not improve this. It still may be reasonable as a general purpose agent but I will not be using it for any coding. There is a bigger version (31b qat) but I could not get it to run properly with pi or opencode.