Running Claude Code Fully Local Is Not Worth It Yet

TL;DR: It works, and I would not do it. Model Runner compiles every tool's JSON Schema into a GBNF grammar, llama.cpp rejects it, and the session 400s on the first request; an eighty-line proxy fixes that. What the proxy cannot fix is the arithmetic. A 35,482-token handshake eats half a 64k window before you type, one file question took 4 minutes 45 seconds, and a machine that does this properly costs about the same per month as the hosted subscription you were trying to cancel. Full local is not the win. Splitting the work is, and that is the third article.

No translation layer needed, and why that is not the whole story

Every "run Claude Code on a local model" guide from a year ago had a translation layer in the middle. That was not optional. Claude Code speaks the Anthropic messages protocol, almost every local inference server speaks the OpenAI chat-completions protocol, and something has to reshape the JSON between them.

Docker Model Runner removed the need. Its API surface carries three dialects at once:

Route	Dialect
/engines/v1/chat/completions	OpenAI
/anthropic/v1/messages	Anthropic
/api/chat	Ollama

That middle row means no protocol translation. Claude Code sets its base URL, posts the request it was always going to post, and the runner answers in the shape it expects.

It does not, as it turns out, mean nothing has to sit in between. Something does, for a completely different reason, and that is most of this article.

I ran everything below on an M4 Pro with 24 GB of unified memory, Docker 29.6.2, and Claude Code 2.1.246. Version numbers matter here more than usual, because docker model grew several of these subcommands recently and the older ones are missing on Docker Desktop builds from early 2026.

Step 1: check the runner before you touch anything

docker model status

On a working install you get the backend as well as the state:

Docker Model Runner is running
BACKEND    STATUS         DETAILS
llama.cpp  Running        llama.cpp b9879-metal
diffusers  Not Installed

llama.cpp b9879-metal is the part worth reading. On Apple silicon the Metal build is what puts the model on the GPU; if you see a plain CPU build, everything below still works and will be roughly an order of magnitude slower.

If the command reports the runner is not running, install or start it:

docker model install-runner

Step 2: open the TCP port

By default the runner answers containers at model-runner.docker.internal and nothing else. Claude Code is a process on your host, not a container, so it needs a real port.

docker desktop enable model-runner --tcp 12434

The Docker Desktop UI has the same switch under Settings, AI. Confirm it took:

curl -s http://localhost:12434/

Docker Model Runner

The service is running.

If you get a connection refused here, stop and fix it. Everything after this point produces confusing errors when the port is closed, because Claude Code reports a failed API call rather than a failed connection.

Step 3: pick a model, then pull it

Docker Hub's ai/ namespace is a normal registry namespace holding models as OCI artifacts, so docker model pull works exactly like docker pull. Search from the CLI rather than the website, because the CLI prints the download size:

docker model search coder

Here is what the coding-relevant part of the catalogue looked like on 28 August 2026:

Model	Size	Notes
ai/gpt-oss	12.1 GB	20B, MXFP4, small default context
ai/devstral-small-2	15.2 GB	24B dense, tuned for agentic edits
ai/glm-4.7-flash	18.3 GB	30B MoE, ~3B active
ai/qwen3-coder	18.6 GB	30B MoE, long native context
ai/deepcoder-preview	9.0 GB	14B, reasoning-flavoured
ai/qwen3-coder-next	48.4 GB	80B MoE, needs a workstation

That size column is close to the resident memory the weights will occupy. On a 16 GB machine, ai/gpt-oss is the realistic ceiling. On 24 GB the 18 GB models load but leave very little for the KV cache once you widen the context, which is the next section's problem.

docker model pull ai/devstral-small-2

Pulls are plain registry traffic, so they inherit whatever your network gives you. Mine ran at about 1.5 MB/s through a corporate proxy, which turns an 18 GB model into an overnight job. Worth knowing before you start it during a standup.

Step 4: prove the endpoint works before Claude Code is involved

Debugging two unfamiliar things at once is how people conclude "it doesn't work". Test the bridge on its own:

curl -s http://localhost:12434/anthropic/v1/messages \
  -H "Content-Type: application/json" \
  -d '{
    "model": "ai/devstral-small-2",
    "max_tokens": 64,
    "messages": [{"role": "user", "content": "Reply with the word ready."}]
  }'

A healthy response is Anthropic-shaped. Mine came back like this, trimmed:

{
  "type": "message",
  "role": "assistant",
  "content": [{ "type": "thinking", "thinking": "..." }],
  "model": ".../Qwen3-8B-Q4_K_M.gguf",
  "stop_reason": "max_tokens",
  "usage": { "input_tokens": 14, "output_tokens": 64 }
}

Two things worth noticing. The model field echoes the actual GGUF file, which is how you find out what quantisation you were really given. And /v1/messages works as well as /anthropic/v1/messages; the official guide uses the short form, the API reference documents the long one, and both returned 200 on my box.

If you get a 404 here, check the model name against docker model ls first. The runner returns 404 for an unknown model on a route that does exist, which reads exactly like a wrong URL and is not one. The runner log confirms it:

docker model logs | grep -i "anthropic api request" -A1

Step 5: the context trap

This is the step the official guide mentions in passing and the step that actually decides whether your session is usable.

Claude Code does not send your prompt. It sends its system prompt, the JSON schema for every tool it can call, your project's CLAUDE.md, and then your prompt. On an empty directory that handshake is already several thousand tokens, and it grows with every MCP server you have connected.

There is a second half to this that the guide does not mention at all. Claude Code has to guess your model's window, and for a model name it does not recognise it guesses 200k:

"ai/qwen3" is not a model this version of Claude Code recognizes, so
auto-compact will keep this session within 200k tokens (the context
window it assumes).

Every Docker Hub model is unrecognised, so this fires every time. If the model actually has 40k, Claude Code will happily fill 200k before it thinks about compacting, and the runner truncates underneath. Tell it the truth:

export CLAUDE_CODE_MAX_CONTEXT_TOKENS=40960

Then make sure the model can hold that. Note that docker model inspect is not much help here, at least on the build I tested. It returns the digest and "format": "gguf" and nothing about the window, and docker model ls printed empty columns for parameters, quantisation and context:

MODEL NAME  PARAMETERS  QUANTIZATION  ARCHITECTURE  MODEL ID      CONTEXT  SIZE
qwen3                                               977d1540140c

So do not go hunting for the number. Just repackage anything you are unsure about. Model Runner rewrites the artifact locally without re-downloading the weights:

docker model package --from ai/gpt-oss --context-size 32000 gpt-oss:32k

Then use the new tag. Widening context is not free: the KV cache scales with it, so a 32k window on a 12 GB model can add several gigabytes of resident memory. On a 24 GB laptop that is the difference between a session that runs and a session that swaps.

Models that already ship a long default, ai/qwen3-coder and ai/glm-4.7-flash among them, skip this step entirely. That alone is a reason to prefer them.

Step 6: launch Claude Code

From your project directory:

ANTHROPIC_BASE_URL=http://localhost:12434 claude --model ai/devstral-small-2

On Windows PowerShell:

$env:ANTHROPIC_BASE_URL = "http://localhost:12434"
claude --model ai/devstral-small-2

Inside the session, run /status. It prints the API base URL in use, which is the only trustworthy confirmation that you are talking to your own machine and not quietly burning hosted tokens because a typo in the variable name made the export a no-op.

There is also a shortcut that sets both variables for you:

docker model launch claude

Ask it what it would do first:

docker model launch claude --config

Configuration for claude (host app):
  Environment:
    ANTHROPIC_BASE_URL=http://localhost/exp/vDD4.40/anthropic
    ANTHROPIC_API_KEY=sk-docker-model-runner

Note the base URL it picks is a Docker Desktop internal socket path, not port 12434. It works, but it is version-stamped (vDD4.40) and will change under you. For anything you write down and reuse, the explicit port is the stable choice.

Watching what Claude Code actually sends

The runner keeps a request log, which is genuinely useful for understanding why a small model behaves badly:

docker model requests --follow --model ai/devstral-small-2 | jq .

Leave that running in a second terminal while you use Claude Code in the first. Do this once and the context problem stops being abstract. You can see the size of the system prompt, and you can see the tool-call JSON the model is being asked to produce.

The failure that actually stopped me: "failed to parse grammar"

My first real session did not run out of context. It died instantly:

API Error: 400 Failed to initialize samplers: failed to parse grammar

That error is worth explaining, because nothing about it points at the cause and searching it gets you nowhere useful.

To make a local model emit valid tool calls, Model Runner does not just ask nicely. It converts the JSON Schema of every tool in your session into a GBNF grammar and constrains llama.cpp's sampler to it. You can watch it happen:

docker model logs | grep '::='

You will see one grammar rule per field of every tool you have. And llama.cpp refuses to compile the grammar if any of them is too big. A JSON Schema string with maxLength: 524288 becomes char{0,524288}, and the parser gives up on the whole thing.

I bisected it against the endpoint directly, one tool, one string field, nothing else moving:

maxLength	HTTP
1,000	200
1,500	200
1,900	200
2,000	400
4,096	400
524,288	400

The cliff sits just under 2,000. Anything above it takes down the entire session, not just that one tool, because the grammar is compiled as a single unit.

Find your own offenders:

docker model logs | grep -oE 'tool-[A-Za-z-]+-schema-[A-Za-z-]+ ::= .*char\{0,[0-9]{4,}\}'

In my session two tools were over the line, and one of them was enough on its own. The unpleasant part is that --disallowedTools does not save you. I tried:

claude --model ai/qwen3 \
  --strict-mcp-config --mcp-config '{"mcpServers":{}}' \
  --disallowedTools "Workflow,Artifact" \
  -p "Reply with exactly: READY"

Same 400. Disallowing a tool stops Claude Code from calling it; the schema still goes out in the request, so the grammar is still built from it.

Nor does dropping to a single tool, or a clean config directory with no MCP servers at all. So I bisected the real request instead. The proxy below dumps it; from there it is a loop over subsets.

Two independent causes, and one limit:

1. pattern is the main one. Model Runner hands each schema's regex to llama.cpp's converter, which cannot express several ordinary constructs. Claude Code's Artifact and RemoteTrigger tools each fail on their own, and deleting pattern alone turns both from 400 into 200.

2. Size bounds are the second. Measured on a single one-field tool: maxLength 1,900 compiles, 2,000 does not.

3. Total grammar complexity is a hard ceiling. With every constraint stripped, the first 50 of my 81 tools compiled and 60 did not. Interestingly this is not a tool count: 200 synthetic two-field tools compiled fine. It is the combined complexity, so a session with many MCP servers connected fails no matter what you strip.

Causes one and two can be fixed on the wire. A small proxy that walks the request and deletes those keywords is about eighty lines:

function stripBounds(node: unknown): void {
  if (Array.isArray(node)) {
    node.forEach(stripBounds);
    return;
  }
  if (node === null || typeof node !== 'object') {
    return;
  }
  const obj = node as Record<string, unknown>;
  if (typeof obj.pattern === 'string') {
    delete obj.pattern;
  }
  for (const key of ['maxLength', 'maxItems'] as const) {
    if (typeof obj[key] === 'number' && (obj[key] as number) >= 1500) {
      delete obj[key];
    }
  }
  Object.values(obj).forEach(stripBounds);
}

Sit that in front of port 12434, rewrite tools on the way through, pipe the response back untouched so SSE streaming still works, and point ANTHROPIC_BASE_URL at the proxy instead. Nothing is lost: those keywords are input validation, and Claude Code validates tool inputs on its own side anyway.

With the proxy in place and MCP servers off, it runs:

ANTHROPIC_BASE_URL=http://localhost:12435 claude --model ai/gpt-oss \
  --strict-mcp-config --mcp-config '{"mcpServers":{}}' \
  -p "Read sample.ts and tell me the exported value, nothing else."

42

That is a full agentic loop: system prompt, tool schemas, a Read call, and an answer, entirely on the laptop.

Two more walls between there and a usable session

The grammar was the interesting failure. These two are the ones that will actually eat your evening.

The handshake is 35,482 tokens. Not "several thousand", as I guessed before measuring. Claude Code told me itself:

API Error: 400 request (35482 tokens) exceeds the available context size (4096 tokens)

docker model package --context-size did not move it, incidentally. The runtime config is the lever that works:

docker model configure --context-size 65536 --keep-alive 15m ai/gpt-oss

And remember to raise CLAUDE_CODE_MAX_CONTEXT_TOKENS to match, or Claude Code refuses the prompt from its own side before the runner ever sees it.

Two resident models will not fit. ai/qwen3-coder is the better coding model and it does not work on 24 GB, because 18.6 GB of weights plus a context wide enough for a 35k handshake overflows the GPU:

error: Insufficient Memory (kIOGPUCommandBufferCallbackErrorOutOfMemory)

Worse, --keep-alive means an earlier model is still resident when you start the next one, so a configuration that worked ten minutes ago fails now for no visible reason. docker model ps shows the truth, and docker model unload --all before each session removes the whole class of problem.

Context width has a cost curve too, not just a cliff. At 98k the runner allocates KV across four slots, the model spills, and generation dropped to about 1 token per second. At 64k the same model returned a complete tool-using answer. Wider is not better; wide enough is better.

Half the window is gone before you type. This is the part I underestimated. A 64k context minus a 35,482-token handshake leaves roughly 30k for everything else, and /clear does not give any of it back. Clearing wipes the transcript, not the system prompt and the tool schemas, so a cleared session restarts at the same 35k floor.

Thirty thousand tokens sounds like room. It isn't, once a tool is involved. I ran one command that listed a batch of Jira tasks, the model spent two minutes twenty reasoning about the output, and the session hit the ceiling on that single turn. Reasoning tokens count against the window like any other. So does a git log, a directory listing, a file over a few hundred lines.

The practical rule I settled on: local sessions are for one file and one edit. Anything that fans out, reads widely, or pipes a command's output back into the model belongs on the hosted side. Widening the window doesn't rescue it either, because 98k is where generation collapsed to a token a second.

So what does the working setup cost

On the M4 Pro, ai/gpt-oss at 64k answered that one-file question in about 4 minutes 45 seconds, of which prompt processing ran at roughly 190 tokens/second. ai/qwen3 at 8B did the same task correctly in 6 minutes 51 seconds, because it is a thinking model and spends most of the budget reasoning about a trivial question.

Both are correct. Neither is fast. Calibrate accordingly.

What a dedicated box actually costs

The laptop is the wrong test. Nobody sensible runs a shared coding model on the machine they are typing on, so price the thing people actually propose: a dedicated Linux box in the corner.

Two builds, both real configurations rather than aspirational ones. Prices are what I found on the German used and retail market in August 2026, in euros, and they will drift.

Item	Single-GPU	Dual-GPU
GPU	1x RTX 3090 24 GB, used	2x RTX 3090 24 GB, used
	700	1,400
Board, CPU, 64 GB DDR5	550	650
PSU	850 W, 120	1,200 W, 190
Case, NVMe, fans	180	230
Build total	1,550	2,470

Now run it. A 3090 pulls around 350 W flat out, and --keep-alive means the model sits resident between your questions rather than reloading, so there is an idle floor too. Four hours a day of real generation on the dual build, an 80 W idle draw the rest of the time, at 0.35 EUR/kWh:

Line	Single-GPU	Dual-GPU
Hardware, spread over 24 months	65	103
Electricity per month	28	45
Monthly, EUR	93	148

Set that against the thing it replaces. A Claude Max subscription at the 5x tier is 100 USD a month, roughly 92 EUR at the rate that week. The 20x tier is double that.

So the dual-GPU box, the one that comfortably holds a 30B model with a window wide enough for the handshake, costs more per month than the subscription, for two years, before anyone has fixed a driver. The single-GPU build lands at a dead heat with the 5x tier while running a weaker model in a narrower window.

And that comparison flatters the local side, because it prices only the electricity and the parts. It does not price the evening I spent bisecting a GBNF grammar, the docker model configure versus docker model package detour, or the fact that every Docker update can move one of those subcommands under you. Nor does it price the wait: 4 minutes 45 seconds for a question about one file, against a few seconds hosted. Ten of those a day is 45 minutes.

The honest version of "free local model" is that the tokens are free and nothing else is.

What you should not expect

Be honest about the ceiling. A 15 GB model on a laptop is not a substitute for a hosted frontier model, and pretending otherwise wastes an afternoon.

Long agentic runs are where the gap shows. A hosted model will chain twenty tool calls across a refactor without losing the thread; a small local model drifts, forgets a file it already read, or re-runs the same search. Note that malformed tool JSON, the classic complaint about local models, is not the failure mode here. Grammar-constrained sampling makes it structurally impossible for the model to emit invalid JSON. It can still pick the wrong tool or the wrong argument, and now it does so in perfectly well-formed syntax.

Where local genuinely wins is narrower and real. Code that cannot leave the building. Working on a train. Grinding through a repetitive mechanical edit across forty files, where you do not care whether the model is clever and you do care that it is free. Our practical rundown of AI coding tools covers where each tier of model earns its place, and the Claude Code workflow for React assumes the hosted model for a reason.

Which of these Docker Hub models is worth the disk space for TypeScript and React specifically is a separate question, with a much less obvious answer than the parameter counts suggest. I worked through it in the local model comparison.

The verdict, and the setup I actually kept

I set out to replace the hosted model and I could not, on any reading of the numbers.

Nothing here failed. The grammar problem has a fix, the context problem has a fix, the model returns correct answers to real questions about real files. The setup works. It just loses on every axis that matters at once: it is slower by two orders of magnitude on wall-clock, weaker on multi-file reasoning, and once you buy hardware that holds a decent model it is not even cheaper. Three losses out of three is not a trade-off, it is a verdict.

What survived is narrower and I still use it. Code that cannot leave the building. A train with no wifi. And the interesting one, which is that not every request Claude Code sends is hard. Reading a file, summarising a tool result, checking a status: a 12 GB model handles those, and they are a real share of a session.

That is not "switch to local". It is "stop sending the easy half somewhere expensive", and it needs a router rather than an environment variable. Everything you installed here is the prerequisite for it, so the hybrid setup, step by step picks up exactly where this article stops.

Rolling it back

Nothing here is sticky unless you made it sticky.

unset ANTHROPIC_BASE_URL     # back to the hosted model
docker model unload --all    # free the RAM, keep the weights
docker model rm ai/devstral-small-2   # free the disk

If you exported the variable in ~/.zshrc, remove it there too. An orphaned ANTHROPIC_BASE_URL pointing at a runner you have since stopped produces a connection error on every claude invocation, and it is not obvious where it came from three weeks later.