semif vs Jev
Free 4B model on my M2 Max vs TypeSafe’s real Jev. Same questions. 2026-09-23.
| semif (local) | Jev (cloud) | |
|---|---|---|
| Speed per call | ~0.25 s | ~0.4 s |
| Cold start | 9 s | none |
| Cost | free | $0.00001 |
| “Same bug?” ranking (AUROC, n=18) | 0.88 | 0.95 |
| Accuracy at 0.5 cutoff (n=46) | 48% | 89% |
| Severity: real P0s caught | 1 in 6 | 1 in 6 |
| Ad search terms, same answer | 18 of 20 | |
Takeaway: local ranks fine but is overconfident. Jev is better calibrated. Neither can judge severity.
Small samples, labels from my own past agent logs. Full numbers →