← Short version

semif vs Jev

A local 4B “System 1” judge running on a Mac, measured against TypeSafe’s real Jev on the same questions: latency, cost, and accuracy on real past decisions from my own dev campaigns.

Date 2026-09-23 Machine Apple M2 Max · 64 GB This Mac SemIf-OpenJev · Qwen3.5-4B · MLX Real Jev typesafe/jev-1.13-20260917 via OpenRouter /api/v1/systemone

Ranking

Both rank well where it counts: “same defect?” AUROC 0.88 local / 0.95 Jev (n=18). Jev ranks higher in 4 of 5 comparisons.

Thresholds

The local model is badly overconfident: AUROC 0.905 but accuracy at 0.5 only 0.478. Jev at 0.5: 0.891.

Severity

Neither can rank severity. At 95% precision both catch 1 in 6 real P0s.

Speed & cost

Warm calls 0.2–0.5 s on both. Local has a 9.2 s cold start; Jev is ~$0.000012 per yes/no call.

Accuracy on real orchestration decisions

Five decision families pulled from past agent-loop logs from my own software projects, labelled with what the log itself recorded, then asked of both backends with the same question text. AUROC measures ranking: 0.5 is a coin flip, 1.0 is perfect.

Family	n	AUROC local / Jev	Acc@0.5 local / Jev	Recall @95% prec.	Mean ms	Verdict
F1 dedup“Do these two findings describe the same defect?”	18	0.877 / 0.951	0.667 / 0.833	0.667 / 0.778	401 / 378	Shadow only
F1 dedup, cloud-eligible rowsrows allowed to leave the machine	11	0.750 / 0.833	n/a	n/a	n/a	Shadow only
F2 stuckround N vs N+1: STUCK or PROGRESS	5	1.000 / 0.833	0.600 / 0.800	1.000 / 0.667	613 / 332	Demo, N too small
F3 falsified claim returningno positive examples survived	12	n/a	1.000 / 1.000	n/a	538 / 336	Not usable
F4 severitydefect → P0–P3 (P0 AUROC shown)	41	0.620 / 0.739	top-1 0.390 / 0.268	0.167 / 0.167	296 / 325	Not usable
F5 verdict typeCONFIRMED vs UNRESOLVED	46	0.905 / 0.926	0.478 / 0.891	n/a	338 / 366	Shadow only

For anyone wiring this into an agent loop: the local 4B model sorts reasonably but its probabilities are not calibrated. On F5 it scored 0.905 AUROC but called CONFIRMED on items that weren’t, so a fixed 0.5 cutoff got 48% accuracy. Use it to order work and to add scrutiny, not to skip work, until a per-decision threshold is measured on held-out labels. Backends disagreed on 17% (F1), 41% (F5) and 56% (F4) of items.

Latency and cost

Measurement	This Mac	Real Jev	How measured
Single yes/no, warm	190–258 ms	289–534 ms	CLI `ms` field, several runs
First call, cold	9.2 s	n/a	model load + shader compile; ~9 GB unified memory
Through the web UI’s API, one question	293 ms	490 ms	wall clock; “both” runs concurrently ≈ the slower one
Batch: 20 items × 3 questions, both backends	15.8 s total		one `/api/ask` batch call
Eval, mean per call (5 families)	296–613 ms	325–378 ms	CLI in batch; local includes process startup and some contention
Cost	$0	$1.2e-5 / call	`usage.cost` on a 287-input-token yes/no; $0.042 per M input, output free
Cost of the whole eval (128 Jev calls)	$0	≈ $0.0015	estimated from the per-call rate

Same inputs, side by side

Input	This Mac	Real Jev
yes/no: “Is this person asking me to do something?” (invoice by Friday)	0.9998	0.99
same question, an FYI notice	0.0086	0.03
pick: bug / feature / praise (“crashes when I open settings”)	bug 0.9995	bug 1.00
rate: outage urgency, low → critical	critical 0.74 · high 0.24	critical 1.00
rank 4 messages: “Is this a bug report?”	0.99 / 0.97 / 0.05 / 0.003	0.66 / 0.65 / 0.08 / 0.04

Same ordering on every rank tested. Jev returns two-decimal probabilities and saturates to 0 or 1 on pick and rate; the local model’s raw probabilities are more extreme on yes/no.

A real use: Google Ads search terms

Three questions per search term in one request: is the searcher seeking care, what’s their intent, and does the term name a place outside the service area. The 20 terms are examples written for the test, not account data. A term is a negative-keyword candidate when seeking-care is under 30% or the intent is job, learn or unrelated.

Search term	Seeking care	Intent (local / Jev)	Outside area	Negative?

Result: intent agreed on 18 of 20. Both flagged the same 9 negatives, and all 9 look right on a read-through (a human should still check before adding negatives). The two disagreements were the local model calling “child therapist hialeah” and “key largo family counseling” a specific other provider; Jev said find-care, the better read for generic terms. The local outside-area scores were also noisy (“…salary florida” 75%). Jev cost for the batch: $0.00046, about 2¢ per 1,000 terms.

Drafting requests with an LLM

Drafter	Time	Cost	Notes
gpt-oss:20b via Ollama (fully local)	13.8 s cold	$0	7.6 s of it model load; 340 tokens; valid request once score levels were a list; local semif then answered it
Claude Sonnet 5 via OpenRouter	6–15 s	$0.011–0.017	strict JSON from the prompt, server-side schema check, one repair retry (n=3)
Claude Haiku 4.5 via OpenRouter	6–15 s*	$0.0035	n=1; in its one try it packed three log lines into one state, so the prompt now forbids that. *range across all four cloud drafts, not timed per model

Things that bit

Jev rejects an empty criteria on yes/no questions. {"type":"noul","criteria":{}} returns HTTP 400 (the schema wants criteria.true/criteria.false or no criteria at all). The local SemIf accepts both. Fixed in the semif client by dropping empty noul criteria before remote calls.
Score criteria are an ordered list, 2–10 levels, lowest first, not a name→description map.
Thresholds fit in-sample are overfit. Jev’s 0.44 cutoff on F1 sits exactly on one example’s score, and the local 0.20 cutoff sits on a tie. Neither ships as a rule.

Caveats

Small N everywhere (5–46 per family). These are signals, not certifications.
Labels come from what past logs recorded, a few inferred. Some label noise is certain.
Rows from sensitive sources were excluded from cloud calls and from these tables; F1’s cloud-eligible subset (n=11) is the fair Jev-vs-local comparison.
One machine, one day; the local latency had other work competing for the model at times.

Reproduce

semif yes "Is this person asking me to do something?" -s "Hey, can you send me the updated invoice by Friday?"
semif-jev yes "Is this person asking me to do something?" -s "Hey, can you send me the updated invoice by Friday?"
printf '%s\n' '{"id":"a","state":"The app crashes on launch"}' '{"id":"b","state":"Please add a dark mode"}' \
  | semif rank "Is this a bug report?" -k 2