local vs cloud / apple silicon
How fast is local, really? 7 models, 147 tests, one honest benchmark.
A reproducible harness that runs local Ollama models against the Claude API on Apple Silicon, across 21 coding prompts in 7 categories. It measures tokens per second, time to first token, quality (scored by Claude Sonnet), and dollar cost, then writes the reports so the numbers are not just vibes.
Models7
Tests147
Prompts21 / 7 cats
HardwareM2 Max 32GB
Total cost$3.95
// results
Throughput, sorted by tokens per second
Bar length is tokens per second. The gap between cloud and local is the whole story: local is real and free, but on this machine it trails the API on raw speed.
Cloud (Claude API)
Local (Ollama)
Results from one benchmark run on an M2 Max (32GB unified memory). Quality is scored by Claude Sonnet 4.6 out of 10; 62 of the 147 scores are Claude judging Claude, which is flagged in every report rather than hidden and may inflate cloud quality. Cost is the dollar total for the cloud calls in the run; local models cost nothing to run. TTFT is time to first token.
// what it measures
Three axes, one run
01
Speed
Tokens per second for sustained generation, plus time to first token for how quickly a model starts answering.
169.7 tok/s fastest · 0.5s best TTFT
02
Quality
Every output is scored by Claude Sonnet 4.6 on correctness (40%), completeness (35%), and clarity (25%).
8.65 / 10 top quality
03
Cost
Dollar cost of the cloud calls and the judging pass. Local models running through Ollama are free.
$3.95 total · $2.69 run + $1.26 judge
// how it works
Prompt bank to reports, one pipeline
A Python package (llm_bench) runs the same 21 prompts against every model, hands the outputs to a Claude judge, and generates Word, PowerPoint, web, and dashboard reports.
input
Prompt bank
21 YAML prompts, 7 categories
→
runner
Executor
runs all models, captures metrics
→
adapters
Ollama + Anthropic
4 local, 3 cloud
→
judge
Claude Sonnet
weighted quality scoring
→
output
Reports
docx, pptx, web, React dashboard
// prompt categories
Seven categories, three prompts each
Each model runs all 21 prompts, so 7 models produce 147 tests total.
| Category | File | What it tests |
|---|---|---|
| Code Generation | code_generation.yaml | Write new code from specifications |
| Debugging & Reasoning | debugging_reasoning.yaml | Find and fix bugs, trace logic |
| Refactoring | refactoring.yaml | Improve existing code structure |
| Explanation & Teaching | explanation_teaching.yaml | Explain concepts clearly |
| Short Quick Tasks | short_quick.yaml | Fast utility operations |
| Long Complex Research | long_complex.yaml | Deep architecture and research tasks |
| Tool Calling / Agentic | tool_calling.yaml | Agentic tool use patterns |
// install and run
Clone it, run your own numbers
Set ANTHROPIC_API_KEY for the cloud models and start `ollama serve` for the local ones. Then run the benchmark.
# Clone
git clone https://github.com/parsamivehchi/tps.sh.git
cd tps.sh
# Install dependencies
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
# Configure API key (for cloud models)
echo "ANTHROPIC_API_KEY=sk-ant-..." > .env
# Start Ollama (for local models)
ollama serve
# Run the benchmark
python -m llm_bench run// requirements
Python 3.12+
Ollama local models
Anthropic API key cloud models
Apple Silicon M2 Max 32GB used
Node / Bun optional, dashboard
See the full interactive results
Charts, per-category breakdowns, and the hardware guide live at the interactive site.
// license
PolyForm Noncommercial 1.0.0
Free for personal and non-commercial use. See the full license for the exact terms. Licensor: Parsa Mivehchi.