Requel 2.0 shipped AI requirements review: per-type review definitions for goals, stories, use cases, scenarios, steps, actors and glossary terms, each tuned against a fixture project until a frontier model found every flaw I had planted in it. I tuned them through Requel’s dev-only cli provider, which shells out to the claude command, and that left one question for 2.1: how much of that holds up on a model you can run on your own laptop, with no API key and nothing leaving the machine? This post is how we answered it for issue #365: which models we picked and why, the exact commands I ran, the scripts that scored the results, and what came out. As with the rest of Requel, I did this pairing with Claude. It wrote the run instructions and the tooling, I ran everything on my Mac, and it read the results back over the desktop app’s file bridge.

Why a local model at all

Requel’s AI review works with OpenAI, Anthropic and Gemini, and with any server that speaks the OpenAI chat-completions API, which includes Ollama. Two kinds of people want the last one: people who can’t send requirements outside the building, and people like me who don’t have an API budget. The bundled Ollama default, qwen2.5:3b, proves the pipeline works and not much else; its findings are too rough to be worth reading. The question was whether a bigger local model, one that still fits on a developer’s laptop, is good enough to recommend.

Picking the candidates

The machine is a MacBook Pro with an M3 Pro and 36 GB of unified memory. That sets the budget more than anything else:

  • The GPU doesn’t get all 36 GB. When Ollama starts it logs what Metal will give it, and on this Mac that is 28.1 GiB. MySQL, Requel’s JVM, an IDE and a browser live in what’s left.
  • The model has to share that with its context. A review prompt is the definition’s instructions plus a context pack about the entity (the entity itself, its siblings, its use cases, the glossary), and Requel allows packs up to 16,000 tokens. With room for the reply, a 32K context is the target, and the context’s cache comes out of the same 28 GB.
  • It has to return JSON. Over Ollama’s OpenAI-compatible endpoint, Spring AI embeds the output schema in the prompt rather than enforcing it, so the model has to follow the format on its own. A model that drifts into prose fails the review.

That rules out the 70B class and makes the 14B class feel like a waste of the hardware. The sweet spot is a 4-bit model around 17–19 GB on disk. The release plan had pencilled in a 14B, with a 27B if it fit. When we got to the ticket, we picked two current models in that range that are built very differently:

Model Ollama tag Size Shape
Qwen 3.6, 27B qwen3.6:27b ~17 GB dense: every parameter works on every token
Gemma 4, 26B gemma4:26b ~18 GB mixture-of-experts: about 4B parameters active per token

A dense model and a mixture-of-experts model of about the same size is a useful pair. You’d expect the dense one to reason better and the sparse one to be faster, and running both on the same fixture tells you whether either expectation holds for this job.

The setup

Two terminals. The first runs Ollama and stays open:

command -v ollama >/dev/null || brew install ollama
osascript -e 'quit app "Ollama"' 2>/dev/null; brew services stop ollama 2>/dev/null; pkill -f "ollama serve" 2>/dev/null; sleep 2
OLLAMA_CONTEXT_LENGTH=32768 ollama serve

The second line matters more than it looks. The Ollama desktop app and a Homebrew service both hold port 11434, and either one would serve requests with its own settings instead of the one I was starting. The third line sets the context explicitly. Older Ollama releases default to 4,096 tokens and cut a longer prompt without an error, so the model quietly reviews half a prompt. This release (0.40.14) picks a default from GPU memory, and on this Mac it logged:

msg="vram-based default context" total_vram="28.1 GiB" default_num_ctx=32768

So the default happened to be right here, but it won’t be on a smaller machine, and Requel’s setup guide now says to set it every time.

The second terminal does everything else: waits for Ollama, checks MySQL is up, pulls both models (about 35 GB together), asks the model for JSON directly to check it can, builds Requel, starts a capture proxy and Requel, and runs a one-type trial:

cd ~/gh-acc/rreganjr/Requel
until curl -s http://localhost:11434/api/version >/dev/null; do echo "waiting for ollama..."; sleep 2; done
nc -z 127.0.0.1 3306 || echo "!!! MySQL is not running on 3306 - start it before Requel"
ollama pull qwen3.6:27b && ollama pull gemma4:26b
MODEL=qwen3.6:27b
time curl -s http://localhost:11434/v1/chat/completions -H 'Content-Type: application/json' -d "{\"model\":\"$MODEL\",\"max_tokens\":2000,\"messages\":[{\"role\":\"system\",\"content\":\"You review one requirement. Reply with JSON only, no prose and no code fence, matching {\\\"findings\\\": [{\\\"type\\\": string, \\\"text\\\": string}]}.\"},{\"role\":\"user\",\"content\":\"Requirement: The system shall be fast and easy to use.\"}]}" | python3 -c 'import sys,json; r=json.load(sys.stdin); c=r["choices"][0]; m=c["message"]; t=(m.get("content") or "").strip(); print("finish:", c["finish_reason"], "| reasoning chars:", len(m.get("reasoning") or ""), "| usage:", r.get("usage")); print(json.dumps(json.loads(t.removeprefix("```json").removesuffix("```").strip()), indent=2))'
ollama ps
mvn -pl modules/requel-app -am package -DskipTests -q
mkdir -p target/ai-eval
python3 scripts/ai-eval/capture-proxy.py --out target/ai-eval/capture-qwen3.6 > target/ai-eval/proxy-qwen3.6.log 2>&1 &
REQUEL_AI_MODEL=qwen3.6:27b REQUEL_AI_BASE_URL=http://localhost:11435 java -jar modules/requel-app/target/requel-app-2.1.0-dev.jar --spring.profiles.active=dev,ai-ollama --server.port=8080 '--spring.datasource.url=jdbc:mysql://127.0.0.1:3306/requel?createDatabaseIfNotExist=true&useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC' --spring.datasource.username=root --spring.datasource.password=password > target/ai-eval/requel-qwen3.6.log 2>&1 &
until curl -s http://localhost:8080/actuator/health | grep -q UP; do echo "waiting for requel..."; sleep 5; done
python3 scripts/ai-eval/score.py --type Step --runs 1 --timeout 900 --delete-after --label "ollama/qwen3.6:27b trial"
cat target/ai-eval/proxy-qwen3.6.log

The curl check answered the first question in about 85 seconds: finish: stop, a clean findings list calling “fast” and “easy to use” unmeasurable, and 5,493 characters of reasoning before it. ollama ps confirmed the model was loaded the way I wanted:

NAME           ID              SIZE     PROCESSOR    CONTEXT    RUNNER    UNTIL
qwen3.6:27b    fb512b7f3368    19 GB    100% GPU     32768      mlx       4 minutes from now

100% GPU is the line to check. If the model and its context don’t fit, Ollama splits it between CPU and GPU and everything slows down; the fix is a smaller context, such as 24576.

Requel itself starts with the ai-ollama profile, which points the OpenAI client at Ollama, and REQUEL_AI_MODEL picks the model. REQUEL_AI_BASE_URL points it at port 11435 instead of 11434, which is where the next piece comes in.

The scripts

The capture proxy

Requel records every AI run (who asked, which definition, tokens, status), but not the request body or the reply. With a cloud model that’s the privacy-preserving default. With a local model it leaves you guessing when something goes wrong, so Claude wrote a small dev tool, scripts/ai-eval/capture-proxy.py, that sits between Requel and Ollama. It’s Python standard library only: it listens on 11435, forwards to 11434, saves each exchange as NNNN-request.json and NNNN-response.json, and prints one line per call:

0007 qwen3.6:27b 200 41.2s prompt=7311 completion=912 finish=stop reasoning=0 json=ok

Each field answers a question that otherwise needs a debugger:

  • json says whether the reply’s content parses as JSON (ok, BAD, EMPTY).
  • finish=length means the reply hit the output cap and was cut off.
  • prompt near 32768 means the prompt filled the context.
  • reasoning is how many characters of thinking the model returned beside its answer.

It earned its keep in the first five minutes. The first trial failed every review, and the proxy showed why: six calls, every one a 400 in 0.0 seconds with no tokens at all. Ollama was saying “EOF” because it never saw a request body. Spring AI’s HTTP client sends bodies with Transfer-Encoding: chunked, and the first version of the proxy only read a Content-Length body, so it forwarded an empty one. One _read_body that understands chunked encoding later, the same trial ran clean. Without the proxy, all I’d have had was “failed” three times.

The scoring script

scripts/ai-eval/score.py is the harness the review definitions were tuned with, and the point of #365 was to run it unchanged against a local model. It:

  1. Imports a fixture project, a small tool library, under a fresh name.
  2. Asks Requel to review each entity, through the same POST /api/ai/reviews the UI uses, and polls until the run finishes.
  3. Scores each run’s findings against expectations.json.
  4. Deletes the project and writes report.md and raw.json.

The fixture has 39 entities across the seven types. Most carry one planted flaw, each a classic requirements problem:

  • a goal that’s unmeasurable;
  • a goal that names a solution instead of an outcome;
  • a goal that’s really a constraint;
  • a story that’s really a scenario;
  • a use case with no failure path;
  • a step that does two things;
  • an actor defined by who it isn’t;
  • a glossary term defined in terms of itself.

Each expectation names the finding types that count as a hit plus keyword patterns. Anything else the model raises counts as spurious, except findings on an “also valid” list of real flaws the fixture doesn’t target. Twelve entities are silent: written to be clean, so the right answer is to raise nothing. One is a trap, a story with made-up figures that must not be confirmed as correct. Each review also runs the terminology policy, which checks the text against the project glossary, as a second call.

Every run is re-scorable from its raw JSON, so the expectations can be argued about after the fact without re-running a model. That’s also how I compared against the frontier result: the 2.0 run, re-scored with today’s expectations.

Thinking costs more than time

The first working trial (the three Step entities, one review each) found both planted flaws and kept quiet on the clean step. It was also slow: 50 to 262 seconds per call. The proxy showed why: Qwen 3.6 “thinks” by default, writing thousands of characters of reasoning before every answer. Requel throws the reasoning away; it only reads the reply.

Ollama maps the OpenAI reasoning_effort field to its own thinking switch, and Spring AI exposes it as a property, so it can be turned off from Requel’s command line with --spring.ai.openai.chat.options.reasoning-effort=none. Before trusting it, a direct check against Ollama with and without the field:

thinking on (default): 73.3s reasoning=4804 completion=1294 json=ok
reasoning_effort=none: 3.0s reasoning=0 completion=53 json=ok

Same valid JSON, 24 times faster. With thinking off, the same three-step trial found the same two flaws, still left the clean step alone, and every call finished in 25 to 99 seconds. But it added two findings that weren’t there before: a guess that one step was really two, and a policy finding that “member” should be the glossary’s “Member”, a difference of one capital letter.

That’s when I asked whether intelligence doesn’t require thinking. The answer the full runs gave is more specific: thinking didn’t help the model find more, but it made it more careful about what it reported.

The overnight run

Three full runs, unattended, one after the other:

  1. Qwen with thinking off, three reviews per entity.
  2. Gemma with thinking off, three reviews per entity.
  3. Qwen with thinking on, one review per entity (enough to compare, at a fraction of the night).
cd ~/gh-acc/rreganjr/Requel
run_eval() {
  model="$1"; name="$2"; runs="$3"; flag="$4"
  pkill -f requel-app-2.1.0-dev.jar; pkill -f capture-proxy.py; sleep 5
  ollama stop qwen3.6:27b 2>/dev/null; ollama stop gemma4:26b 2>/dev/null
  python3 scripts/ai-eval/capture-proxy.py --out "target/ai-eval/capture-$name" > "target/ai-eval/proxy-$name.log" 2>&1 &
  REQUEL_AI_MODEL="$model" REQUEL_AI_BASE_URL=http://localhost:11435 java -jar modules/requel-app/target/requel-app-2.1.0-dev.jar --spring.profiles.active=dev,ai-ollama --server.port=8080 $flag '--spring.datasource.url=jdbc:mysql://127.0.0.1:3306/requel?createDatabaseIfNotExist=true&useSSL=false&allowPublicKeyRetrieval=true&serverTimezone=UTC' --spring.datasource.username=root --spring.datasource.password=password > "target/ai-eval/requel-$name.log" 2>&1 &
  sleep 10; until curl -s http://localhost:8080/actuator/health | grep -q UP; do sleep 5; done
  echo "=== $(date) starting $name"
  python3 scripts/ai-eval/score.py --runs "$runs" --timeout 900 --delete-after --label "ollama/$model $name #365"
  echo "=== $(date) finished $name"
}
run_eval qwen3.6:27b qwen3.6-nothink-full 3 --spring.ai.openai.chat.options.reasoning-effort=none
run_eval gemma4:26b gemma4-nothink-full 3 --spring.ai.openai.chat.options.reasoning-effort=none
run_eval qwen3.6:27b qwen3.6-think-full 1 ""
pkill -f requel-app-2.1.0-dev.jar; pkill -f capture-proxy.py
ls -d target/ai-eval/2026*/ | tail -3

Each run gets a fresh Requel, its own capture directory and log, and the previous model is unloaded first so the two never compete for memory. It started at 11:46 pm and finished at 5:28 am.

The results

Like for like, on the 35 entities the frontier run also scored:

  Frontier (cli, 2.0) Gemma, thinking off Qwen, thinking off Qwen, thinking on
Planted flaws found 100% 80% 84% 79%
Spurious findings per review 0.2 0.5 0.7 0.1
Clean entities left alone 96% 67% 67% 100%
Failed replies 0 0 3 3
Policy: spurious per review 0.0 0.0 0.4 0.0
Median time per review ~16 s 14 s 68 s 239 s
Full run — 34 min 2 h 25 min 2 h 43 min (1 run)

(The review time includes the policy call, which waits behind the review because Ollama serves one request at a time on this machine.)

A few things stood out.

The mixture-of-experts model was the fast one, by a lot. Gemma 4 took 14 seconds per review, the same as the frontier model through the CLI, and finished three passes over the fixture in 34 minutes. Qwen took five times as long. With only about 4B parameters active per token, Gemma does far less work per token, and for this job it gave up very little for it.

Both local models are good at the plain flaws. A person named where a role belongs, two actions in one step, a circular definition, a story with no benefit, a use case with no failure path: both models found every one, in every run.

Both miss the same four, every time. All of them are goal flaws where the evidence is something the goal doesn’t say:

  • a goal that duplicates a sibling goal at a different level (“members find any tool within a minute” versus “members search the catalogue by name”);
  • a rule that says only who can’t borrow and never who can;
  • “every tool already carries exactly one barcode”, an assumption stated as fact that nothing protects;
  • “all alerts are acknowledged within a day”, which is trivially true if the monitor raises none.

The local models raised something else instead, usually “this names a solution, not an outcome”. That’s not wrong, but it’s not the point. Thinking didn’t recover any of the four, which is why we didn’t change the definitions. The instructions already explain each of these flaws. Making them blunter would risk the 100% the frontier model gets without helping a model that can’t make the second inference.

Gemma’s extra findings are mostly arguable. On the clean “renew a loan” use case and scenario, it asks what happens if someone reserves the tool between loading the page and clicking Renew. That’s a fair question the fixture just doesn’t count. Its policy pass matched the frontier model’s exactly.

Qwen has two habits. On some entities with nothing to report, it returns an empty {} instead of a summary and an empty findings list. Requel rejects that as a schema failure: 7 of 234 calls. Its terminology policy also flags “member” and “Members” as non-canonical forms of the glossary term “Member” over and over: 50 findings in 114 policy runs.

Thinking makes Qwen precise and slow. With thinking on, spurious findings fell from 0.7 to 0.1 per review, and the policy pass went clean. But reviews took four minutes, and three of 39 never answered at all. The thinking counts against Requel’s 4,000-token output cap, and on those three the model spent all of it reasoning, ending at finish=length with nothing to parse. Raising the cap would fix the failures, but not the four minutes.

What changed in Requel

No review definition changed. The setup guide’s local-model section does:

  • Use gemma4:26b with thinking off. It’s as fast as the frontier model, it never broke the JSON, and the policy pass is clean. Expect it to find about four in five of the flaws the frontier model finds.
  • Set OLLAMA_CONTEXT_LENGTH explicitly, and check ollama ps for 100% GPU.
  • Turn thinking off with --spring.ai.openai.chat.options.reasoning-effort=none.
  • Use the capture proxy when you need to see what’s going back and forth.

The full write-up, with per-type and per-entity tables and the raw findings for all three runs, is in the repo: doc/work/2.1/365-local-model-eval.md. The scripts are in scripts/ai-eval.

What I’d take away

A local model on a laptop is now a reasonable reviewer for the mechanical half of requirements quality, the part a careful colleague would catch on a first read. It isn’t a substitute for the second read, the one that compares a goal with its neighbours or asks what an “every” quantifies over. That’s the part I most wanted an assistant for, and it’s still where the frontier models earn their cost.

The other takeaway is about method. None of this would have been worth much without a fixture with known answers and a script that scores against it. “It seems pretty good” is what I’d have concluded from clicking around with Gemma, and I’d have missed both the four-flaw gap and the fact that thinking buys precision rather than insight. A night of unattended runs and a re-scorable JSON file per run turned a hunch into a table.