Decision models in software · 2026-10-05

Where can a local model compete with Jev?

Unofficial, independent testing. Use as is, without warranty or vendor endorsement.

LIVE API RUN COMPLETE · 17,576/17,576 live requests completed. Local alternatives may still be running. Only complete task groups appear below.
Updated 2026-10-05T00:49:57.233828+00:00. Fresh strong-task comparison: complete for the seven priority alternatives.

Live referenceJev 1.13.0Exact version, current API calls
New public test cases800Plus 400 development examples
Estimated API spend$0.4585Reported input tokens × dated public price

Open the shareable graphics: accuracy, response time, technology, confidence and business value

The practical use cases

Software workflowValue to measureDeployment decision
Engineering document intakeSuggest drawing, inspection, purchase or maintenance-record folders in a document management system.Start with a reviewer accepting or correcting suggestions. Measure seconds saved per document, corrections and search failures.
Manufacturing and oil-and-gas maintenance helpdeskSuggest maintenance, procurement, document control or IT queues; refer mixed requests for review.Evaluate on real, labelled tickets with technicians and document controllers. Track misrouting and resolution delay. This test does not cover equipment control or work-permit approval.
Helpdesk message screeningPrioritize suspected spam or scam messages for review and reduce irrelevant workload.Measure false positives on legitimate messages. Public spam/phishing labels do not establish safe automatic deletion or security protection.
Content and knowledge-base organizationAttach broad topic labels to incoming articles; use confidence to offer a shortlist.AG News is a transfer benchmark, not evidence that the same model understands a company's technical taxonomy.

A fair comparison

Every model receives the same frozen input, question and complete option list. We chose whole task groups where local alternatives had shown promise, then drew new examples without consulting model outputs. These fresh results are shown separately from vendor replication, historical Jev results and the harder general suites. No individual cases were selected because a local model answered correctly.

Accuracy includes failed requests. Tables show raw counts, uncertainty and paired differences against the live API. Differences are descriptive and are not adjusted for multiple model comparisons. A narrow lead is not sufficient evidence of a general winner. Calibration uses the returned probability vectors; typed, valid output can still be wrong.

Confidence thresholds use development cases only. A threshold qualifies when its development accuracy has a Wilson 95% lower bound of at least 95%, with at least 30 accepted cases. The chosen threshold is then fixed for the test set. Zero accepted coverage means the experiment does not justify automatic acceptance. Tiny synthetic business sets are intended for reviewing behaviour, not certifying a threshold.

The exact banking pilot has all 72 long option descriptions. Laya's native question budget truncates these descriptions on every banking case, so its score measures an unsuitable default configuration for that request shape. It must not be read as a general banking-language score. Native truncation counts are shown alongside the relevant model.

The new business set contains 32 bilingual scenario families: 8 for development and 24 for testing. Labels and translations are assistant-authored and await domain-expert review. Public datasets may have been seen during training. Laya reports spam and phishing in its training mix; perfect scores here do not establish general security performance. Von reports emotion data in training and fitting its confidence map on public JevBench. Other model cards disclose broader public multi-task training; Jev's row-level training exposure is unknown. This is a comparison of released models, not a guarantee of unseen training data. A planned full Intern reproduction overlaps the AG News corpus; overlapping results are not additional independent evidence.

News categories · fresh public test

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API178/20089.0%95% CI 83.9%–92.6%89.3%0Reference0.0%
Laya183/20091.5%95% CI 86.8%–94.6%91.6%0+2.5 pp 95% paired interval -2.5 to +8.00.0%
Laya multilingual183/20091.5%95% CI 86.8%–94.6%91.7%0+2.5 pp 95% paired interval -1.5 to +7.00.0%
Decider 0.8B174/20087.0%95% CI 81.6%–91.0%87.2%0-2.0 pp 95% paired interval -6.5 to +2.50.0%
Decider 2B176/20088.0%95% CI 82.8%–91.8%88.2%0-1.0 pp 95% paired interval -5.0 to +3.50.0%
Decider 4B180/20090.0%95% CI 85.1%–93.4%90.3%0+1.0 pp 95% paired interval -2.5 to +5.00.0%
Von165/20082.5%95% CI 76.6%–87.1%82.3%0-6.5 pp 95% paired interval -12.5 to -0.50.0%
GLiNER Decide 340M140/20070.0%95% CI 63.3%–75.9%67.8%0-19.0 pp 95% paired interval -25.0 to -13.00.0%
CLM 8B INT893/20046.5%95% CI 39.7%–53.4%40.3%0-42.5 pp 95% paired interval -50.5 to -34.50.0%
Cygnet 12B NF4 (local variant)169/20084.5%95% CI 78.8%–88.9%84.8%0-4.5 pp 95% paired interval -8.5 to -1.00.0%
Decision 4B v1.2174/20087.0%95% CI 81.6%–91.0%87.4%0-2.0 pp 95% paired interval -4.5 to +0.00.0%
Decision 2.0 Eos 0.8B165/20082.5%95% CI 76.6%–87.1%82.6%0-6.5 pp 95% paired interval -11.0 to -2.00.0%
Decision 2.0 Kai 0.6B167/20083.5%95% CI 77.7%–88.0%83.6%0-5.5 pp 95% paired interval -9.5 to -1.50.0%
Decision 2.0 Nox 4B174/20087.0%95% CI 81.6%–91.0%87.3%0-2.0 pp 95% paired interval -5.0 to +0.50.0%
Decision 2.0 Sol 2B167/20083.5%95% CI 77.7%–88.0%83.6%0-5.5 pp 95% paired interval -9.5 to -2.00.0%
GLiNER Decide 1B142/20071.0%95% CI 64.4%–76.8%70.7%0-18.0 pp 95% paired interval -23.5 to -12.50.0%
GLiNER Decide multilingual149/20074.5%95% CI 68.0%–80.0%74.5%0-14.5 pp 95% paired interval -20.0 to -9.50.0%
Imajev 2B168/20084.0%95% CI 78.3%–88.4%84.9%44 native abstentions included-5.0 pp 95% paired interval -9.0 to -1.00.0%
Imajev 4B168/20084.0%95% CI 78.3%–88.4%87.1%1313 native abstentions included-5.0 pp 95% paired interval -8.5 to -2.00.0%
Intern-Decision 4B176/20088.0%95% CI 82.8%–91.8%88.3%0-1.0 pp 95% paired interval -4.0 to +2.00.0%
Jev-Omni 12B NF4 (local variant)170/20085.0%95% CI 79.4%–89.3%85.3%0-4.0 pp 95% paired interval -8.0 to -0.50.0%
JevK5 2B170/20085.0%95% CI 79.4%–89.3%85.3%0-4.0 pp 95% paired interval -8.0 to +0.00.0%
JevK5 4B173/20086.5%95% CI 81.1%–90.6%86.8%0-2.5 pp 95% paired interval -6.0 to +0.50.0%
Julia 144M179/20089.5%95% CI 84.5%–93.0%89.5%0+0.5 pp 95% paired interval -4.5 to +6.00.0%
Kev 0.8B177/20088.5%95% CI 83.3%–92.2%88.7%0-0.5 pp 95% paired interval -5.0 to +4.50.0%
Kev 4B179/20089.5%95% CI 84.5%–93.0%89.8%0+0.5 pp 95% paired interval -3.5 to +4.50.0%
Nev 2B Q8177/20088.5%95% CI 83.3%–92.2%88.8%0-0.5 pp 95% paired interval -3.5 to +3.00.0%
Nimble 9B NF4176/20088.0%95% CI 82.8%–91.8%88.3%0-1.0 pp 95% paired interval -4.0 to +2.00.0%
Plumb 4B171/20085.5%95% CI 80.0%–89.7%85.9%0-3.5 pp 95% paired interval -7.0 to -0.50.0%
Tev1 4B175/20087.5%95% CI 82.2%–91.4%87.9%0-1.5 pp 95% paired interval -5.0 to +2.00.0%
Wald 4B v1.2170/20085.0%95% CI 79.4%–89.3%85.2%0-4.0 pp 95% paired interval -7.0 to -1.00.0%
Winnow 12B Q8 (Windows CUDA)170/20085.0%95% CI 79.4%–89.3%85.4%0-4.0 pp 95% paired interval -7.5 to -1.00.0%

Emotion categories · fresh public test

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API120/20060.0%95% CI 53.1%–66.5%50.1%0Reference0.0%
Laya121/20060.5%95% CI 53.6%–67.0%44.7%0+0.5 pp 95% paired interval -6.0 to +7.00.0%
Laya multilingual103/20051.5%95% CI 44.6%–58.3%38.7%0-8.5 pp 95% paired interval -15.5 to -1.50.0%
Decider 0.8B150/20075.0%95% CI 68.6%–80.5%56.4%0+15.0 pp 95% paired interval +9.0 to +21.50.0%
Decider 2B166/20083.0%95% CI 77.2%–87.6%70.0%0+23.0 pp 95% paired interval +16.5 to +30.50.0%
Decider 4B153/20076.5%95% CI 70.2%–81.8%66.7%0+16.5 pp 95% paired interval +11.0 to +22.50.0%
Von177/20088.5%95% CI 83.3%–92.2%71.0%0+28.5 pp 95% paired interval +22.0 to +36.00.0%
GLiNER Decide 340M106/20053.0%95% CI 46.1%–59.8%40.6%0-7.0 pp 95% paired interval -13.5 to +0.00.0%
CLM 8B INT851/20025.5%95% CI 20.0%–32.0%10.0%0-34.5 pp 95% paired interval -43.0 to -25.00.0%
Cygnet 12B NF4 (local variant)49/20024.5%95% CI 19.1%–30.9%8.0%0-35.5 pp 95% paired interval -44.0 to -26.50.0%
Decision 4B v1.2116/20058.0%95% CI 51.1%–64.6%44.9%0-2.0 pp 95% paired interval -7.0 to +3.50.0%
Decision 2.0 Eos 0.8B102/20051.0%95% CI 44.1%–57.8%42.3%0-9.0 pp 95% paired interval -16.0 to -1.50.0%
Decision 2.0 Kai 0.6B114/20057.0%95% CI 50.1%–63.7%45.5%0-3.0 pp 95% paired interval -9.5 to +3.50.0%
Decision 2.0 Nox 4B118/20059.0%95% CI 52.1%–65.6%46.1%0-1.0 pp 95% paired interval -6.0 to +4.50.0%
Decision 2.0 Sol 2B58/20029.0%95% CI 23.2%–35.6%9.5%0-31.0 pp 95% paired interval -39.5 to -22.00.0%
GLiNER Decide 1B91/20045.5%95% CI 38.7%–52.4%47.5%0-14.5 pp 95% paired interval -22.5 to -6.50.0%
GLiNER Decide multilingual78/20039.0%95% CI 32.5%–45.9%31.9%0-21.0 pp 95% paired interval -28.5 to -13.50.0%
Imajev 2B121/20060.5%95% CI 53.6%–67.0%45.3%99 native abstentions included+0.5 pp 95% paired interval -4.5 to +5.50.0%
Imajev 4B80/20040.0%95% CI 33.5%–46.9%38.3%6060 native abstentions included-20.0 pp 95% paired interval -26.5 to -13.50.0%
Intern-Decision 4B96/20048.0%95% CI 41.2%–54.9%36.5%0-12.0 pp 95% paired interval -18.5 to -5.50.0%
Jev-Omni 12B NF4 (local variant)106/20053.0%95% CI 46.1%–59.8%41.6%0-7.0 pp 95% paired interval -12.5 to -1.50.0%
JevK5 2B124/20062.0%95% CI 55.1%–68.4%45.6%0+2.0 pp 95% paired interval -3.0 to +7.00.0%
JevK5 4B121/20060.5%95% CI 53.6%–67.0%51.0%0+0.5 pp 95% paired interval -3.5 to +5.00.0%
Julia 144M182/20091.0%95% CI 86.2%–94.2%78.9%0+31.0 pp 95% paired interval +24.5 to +38.50.0%
Kev 0.8B107/20053.5%95% CI 46.6%–60.3%41.2%0-6.5 pp 95% paired interval -13.0 to +0.00.0%
Kev 4B114/20057.0%95% CI 50.1%–63.7%46.7%0-3.0 pp 95% paired interval -8.5 to +2.50.0%
Nev 2B Q8123/20061.5%95% CI 54.6%–68.0%43.0%0+1.5 pp 95% paired interval -4.0 to +7.00.0%
Nimble 9B NF4118/20059.0%95% CI 52.1%–65.6%48.5%0-1.0 pp 95% paired interval -5.0 to +3.50.0%
Plumb 4B116/20058.0%95% CI 51.1%–64.6%48.9%0-2.0 pp 95% paired interval -7.0 to +3.00.0%
Tev1 4B118/20059.0%95% CI 52.1%–65.6%49.0%0-1.0 pp 95% paired interval -5.5 to +4.00.0%
Wald 4B v1.2121/20060.5%95% CI 53.6%–67.0%48.9%0+0.5 pp 95% paired interval -4.0 to +5.00.0%
Winnow 12B Q8 (Windows CUDA)120/20060.0%95% CI 53.1%–66.5%49.7%0+0.0 pp 95% paired interval -4.5 to +4.50.0%

Spam screening · fresh public test

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API196/20098.0%95% CI 95.0%–99.2%98.0%0Reference95.5%189/191 accepted test cases correct
Laya39 inputs truncated by native defaults200/200100.0%95% CI 98.1%–100.0%100.0%0+2.0 pp 95% paired interval +0.5 to +4.099.5%199/199 accepted test cases correct
Laya multilingual5 inputs truncated by native defaults200/200100.0%95% CI 98.1%–100.0%100.0%0+2.0 pp 95% paired interval +0.5 to +4.00.0%
Decider 0.8B196/20098.0%95% CI 95.0%–99.2%98.0%0+0.0 pp 95% paired interval -1.5 to +1.593.0%186/186 accepted test cases correct
Decider 2B196/20098.0%95% CI 95.0%–99.2%98.0%0+0.0 pp 95% paired interval -1.5 to +1.583.0%166/166 accepted test cases correct
Decider 4B195/20097.5%95% CI 94.3%–98.9%97.5%0-0.5 pp 95% paired interval -2.5 to +1.093.5%186/187 accepted test cases correct
Von125/20062.5%95% CI 55.6%–68.9%56.0%0-35.5 pp 95% paired interval -42.5 to -29.00.0%
GLiNER Decide 340M143/20071.5%95% CI 64.9%–77.3%67.4%0-26.5 pp 95% paired interval -33.0 to -20.50.0%
CLM 8B INT8132/20066.0%95% CI 59.2%–72.2%65.9%0-32.0 pp 95% paired interval -39.0 to -25.00.0%
Cygnet 12B NF4 (local variant)187/20093.5%95% CI 89.2%–96.2%93.4%0-4.5 pp 95% paired interval -8.0 to -1.50.0%
Decision 4B v1.2189/20094.5%95% CI 90.4%–96.9%94.5%0-3.5 pp 95% paired interval -6.5 to -1.00.0%
Decision 2.0 Eos 0.8B139/20069.5%95% CI 62.8%–75.5%69.2%0-28.5 pp 95% paired interval -35.0 to -22.50.0%
Decision 2.0 Kai 0.6B109/20054.5%95% CI 47.6%–61.3%35.3%0-43.5 pp 95% paired interval -50.5 to -36.50.0%
Decision 2.0 Nox 4B186/20093.0%95% CI 88.6%–95.8%92.9%0-5.0 pp 95% paired interval -8.5 to -2.00.0%
Decision 2.0 Sol 2B149/20074.5%95% CI 68.0%–80.0%74.4%0-23.5 pp 95% paired interval -29.5 to -18.00.0%
GLiNER Decide 1B173/20086.5%95% CI 81.1%–90.6%86.3%0-11.5 pp 95% paired interval -16.5 to -7.00.0%
GLiNER Decide multilingual125/20062.5%95% CI 55.6%–68.9%58.4%0-35.5 pp 95% paired interval -43.0 to -29.00.0%
Imajev 2B136/20068.0%95% CI 61.2%–74.1%64.9%1515 native abstentions included-30.0 pp 95% paired interval -36.5 to -24.00.0%
Imajev 4B169/20084.5%95% CI 78.8%–88.9%84.8%66 native abstentions included-13.5 pp 95% paired interval -18.0 to -9.00.0%
Intern-Decision 4B188/20094.0%95% CI 89.8%–96.5%93.9%0-4.0 pp 95% paired interval -7.0 to -1.50.0%
Jev-Omni 12B NF4 (local variant)193/20096.5%95% CI 93.0%–98.3%96.5%0-1.5 pp 95% paired interval -4.0 to +0.50.0%
JevK5 2B141/20070.5%95% CI 63.8%–76.4%70.5%0-27.5 pp 95% paired interval -34.0 to -21.50.0%
JevK5 4B166/20083.0%95% CI 77.2%–87.6%81.8%0-15.0 pp 95% paired interval -20.5 to -10.00.0%
Julia 144M93/20046.5%95% CI 39.7%–53.4%46.5%0-51.5 pp 95% paired interval -59.0 to -44.50.0%
Kev 0.8B139/20069.5%95% CI 62.8%–75.5%67.4%0-28.5 pp 95% paired interval -35.0 to -22.50.0%
Kev 4B186/20093.0%95% CI 88.6%–95.8%92.8%0-5.0 pp 95% paired interval -8.5 to -2.00.0%
Nev 2B Q8166/20083.0%95% CI 77.2%–87.6%81.9%0-15.0 pp 95% paired interval -20.0 to -10.50.0%
Nimble 9B NF4165/20082.5%95% CI 76.6%–87.1%81.2%0-15.5 pp 95% paired interval -21.0 to -11.00.0%
Plumb 4B184/20092.0%95% CI 87.4%–95.0%91.8%0-6.0 pp 95% paired interval -9.5 to -3.00.0%
Tev1 4B190/20095.0%95% CI 91.0%–97.3%94.9%0-3.0 pp 95% paired interval -5.5 to -1.00.0%
Wald 4B v1.2184/20092.0%95% CI 87.4%–95.0%91.8%0-6.0 pp 95% paired interval -9.5 to -3.00.0%
Winnow 12B Q8 (Windows CUDA)187/20093.5%95% CI 89.2%–96.2%93.4%0-4.5 pp 95% paired interval -7.5 to -2.070.0%139/140 accepted test cases correct

Phishing / scam screening · fresh public test

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API179/20089.5%95% CI 84.5%–93.0%88.2%0Reference0.0%
Laya56 inputs truncated by native defaults200/200100.0%95% CI 98.1%–100.0%100.0%0+10.5 pp 95% paired interval +6.5 to +15.099.0%198/198 accepted test cases correct
Laya multilingual4 inputs truncated by native defaults200/200100.0%95% CI 98.1%–100.0%100.0%0+10.5 pp 95% paired interval +6.5 to +15.0100.0%200/200 accepted test cases correct
Decider 0.8B184/20092.0%95% CI 87.4%–95.0%91.4%0+2.5 pp 95% paired interval -1.5 to +6.587.0%169/174 accepted test cases correct
Decider 2B189/20094.5%95% CI 90.4%–96.9%94.2%0+5.0 pp 95% paired interval +0.0 to +10.086.5%170/173 accepted test cases correct
Decider 4B194/20097.0%95% CI 93.6%–98.6%96.8%0+7.5 pp 95% paired interval +3.0 to +12.580.5%160/161 accepted test cases correct
Von152/20076.0%95% CI 69.6%–81.4%75.0%0-13.5 pp 95% paired interval -20.5 to -6.50.0%
GLiNER Decide 340M127/20063.5%95% CI 56.6%–69.9%48.9%0-26.0 pp 95% paired interval -32.0 to -20.00.0%
CLM 8B INT880/20040.0%95% CI 33.5%–46.9%31.6%0-49.5 pp 95% paired interval -58.5 to -40.50.0%
Cygnet 12B NF4 (local variant)178/20089.0%95% CI 83.9%–92.6%88.3%0-0.5 pp 95% paired interval -5.0 to +4.00.0%
Decision 4B v1.2185/20092.5%95% CI 88.0%–95.4%92.1%0+3.0 pp 95% paired interval -2.0 to +8.00.0%
Decision 2.0 Eos 0.8B152/20076.0%95% CI 69.6%–81.4%75.6%0-13.5 pp 95% paired interval -20.5 to -7.00.0%
Decision 2.0 Kai 0.6B137/20068.5%95% CI 61.8%–74.5%64.2%0-21.0 pp 95% paired interval -27.5 to -15.00.0%
Decision 2.0 Nox 4B185/20092.5%95% CI 88.0%–95.4%92.1%0+3.0 pp 95% paired interval -1.5 to +7.589.5%172/179 accepted test cases correct
Decision 2.0 Sol 2B172/20086.0%95% CI 80.5%–90.1%85.1%0-3.5 pp 95% paired interval -9.0 to +2.00.0%
GLiNER Decide 1B122/20061.0%95% CI 54.1%–67.5%48.0%0-28.5 pp 95% paired interval -35.0 to -22.00.0%
GLiNER Decide multilingual129/20064.5%95% CI 57.7%–70.8%55.2%0-25.0 pp 95% paired interval -31.5 to -18.50.0%
Imajev 2B144/20072.0%95% CI 65.4%–77.8%74.4%2222 native abstentions included-17.5 pp 95% paired interval -23.5 to -12.00.0%
Imajev 4B164/20082.0%95% CI 76.1%–86.7%80.9%55 native abstentions included-7.5 pp 95% paired interval -11.5 to -3.50.0%
Intern-Decision 4B183/20091.5%95% CI 86.8%–94.6%90.6%0+2.0 pp 95% paired interval -1.0 to +5.50.0%
Jev-Omni 12B NF4 (local variant)176/20088.0%95% CI 82.8%–91.8%87.0%0-1.5 pp 95% paired interval -5.0 to +2.50.0%
JevK5 2B147/20073.5%95% CI 67.0%–79.1%73.4%0-16.0 pp 95% paired interval -23.5 to -8.00.0%
JevK5 4B180/20090.0%95% CI 85.1%–93.4%88.9%0+0.5 pp 95% paired interval -3.0 to +4.00.0%
Julia 144M76/20038.0%95% CI 31.6%–44.9%31.4%0-51.5 pp 95% paired interval -60.5 to -42.00.0%
Kev 0.8B132/20066.0%95% CI 59.2%–72.2%56.4%0-23.5 pp 95% paired interval -30.0 to -17.00.0%
Kev 4B182/20091.0%95% CI 86.2%–94.2%90.0%0+1.5 pp 95% paired interval -1.5 to +4.50.0%
Nev 2B Q8136/20068.0%95% CI 61.2%–74.1%54.8%0-21.5 pp 95% paired interval -27.5 to -16.00.0%
Nimble 9B NF4170/20085.0%95% CI 79.4%–89.3%83.0%0-4.5 pp 95% paired interval -8.5 to -0.50.0%
Plumb 4B183/20091.5%95% CI 86.8%–94.6%90.6%0+2.0 pp 95% paired interval -0.5 to +5.00.0%
Tev1 4B175/20087.5%95% CI 82.2%–91.4%85.7%0-2.0 pp 95% paired interval -5.0 to +1.00.0%
Wald 4B v1.2181/20090.5%95% CI 85.6%–93.8%89.6%0+1.0 pp 95% paired interval -2.5 to +5.00.0%
Winnow 12B Q8 (Windows CUDA)182/20091.0%95% CI 86.2%–94.2%90.2%0+1.5 pp 95% paired interval -1.5 to +5.00.0%
fresh_business · routing · en · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API11/1291.7%95% CI 64.6%–98.5%93.1%0Reference0.0%
Laya8/1266.7%95% CI 39.1%–86.2%67.4%0-25.0 pp 95% paired interval -58.3 to +8.30.0%
Laya multilingual6/1250.0%95% CI 25.4%–74.6%50.0%0-41.7 pp 95% paired interval -66.7 to -16.70.0%
Decider 0.8B6/1250.0%95% CI 25.4%–74.6%52.7%0-41.7 pp 95% paired interval -66.7 to -16.70.0%
Decider 2B9/1275.0%95% CI 46.8%–91.1%77.3%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
Decider 4B10/1283.3%95% CI 55.2%–95.3%86.7%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Von9/1275.0%95% CI 46.8%–91.1%67.9%0-16.7 pp 95% paired interval -50.0 to +16.70.0%
GLiNER Decide 340M8/1266.7%95% CI 39.1%–86.2%70.7%0-25.0 pp 95% paired interval -50.0 to +0.00.0%
CLM 8B INT82/1216.7%95% CI 4.7%–44.8%5.7%0-75.0 pp 95% paired interval -91.7 to -50.00.0%
Cygnet 12B NF4 (local variant)10/1283.3%95% CI 55.2%–95.3%86.7%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Decision 4B v1.211/1291.7%95% CI 64.6%–98.5%93.1%0+0.0 pp 95% paired interval -25.0 to +25.00.0%
Decision 2.0 Eos 0.8B6/1250.0%95% CI 25.4%–74.6%48.3%0-41.7 pp 95% paired interval -66.7 to -16.70.0%
Decision 2.0 Kai 0.6B7/1258.3%95% CI 32.0%–80.7%57.7%0-33.3 pp 95% paired interval -58.3 to -8.30.0%
Decision 2.0 Nox 4B10/1283.3%95% CI 55.2%–95.3%85.3%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Decision 2.0 Sol 2B7/1258.3%95% CI 32.0%–80.7%55.6%0-33.3 pp 95% paired interval -58.3 to -8.30.0%
GLiNER Decide 1B7/1258.3%95% CI 32.0%–80.7%58.7%0-33.3 pp 95% paired interval -58.3 to -8.30.0%
GLiNER Decide multilingual5/1241.7%95% CI 19.3%–68.0%44.0%0-50.0 pp 95% paired interval -75.0 to -25.00.0%
Imajev 2B9/1275.0%95% CI 46.8%–91.1%77.3%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
Imajev 4B10/1283.3%95% CI 55.2%–95.3%85.3%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Intern-Decision 4B10/1283.3%95% CI 55.2%–95.3%85.3%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Jev-Omni 12B NF4 (local variant)11/1291.7%95% CI 64.6%–98.5%93.1%0+0.0 pp 95% paired interval -25.0 to +25.00.0%
JevK5 2B9/1275.0%95% CI 46.8%–91.1%77.3%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
JevK5 4B10/1283.3%95% CI 55.2%–95.3%85.3%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Julia 144M3/1225.0%95% CI 8.9%–53.2%13.0%0-66.7 pp 95% paired interval -91.7 to -41.70.0%
Kev 0.8B8/1266.7%95% CI 39.1%–86.2%68.1%0-25.0 pp 95% paired interval -50.0 to +0.00.0%
Kev 4B9/1275.0%95% CI 46.8%–91.1%77.3%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
Nev 2B Q89/1275.0%95% CI 46.8%–91.1%77.3%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
Nimble 9B NF410/1283.3%95% CI 55.2%–95.3%86.7%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Plumb 4B9/1275.0%95% CI 46.8%–91.1%76.8%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
Tev1 4B8/1266.7%95% CI 39.1%–86.2%69.3%0-25.0 pp 95% paired interval -50.0 to +0.00.0%
Wald 4B v1.210/1283.3%95% CI 55.2%–95.3%85.3%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Winnow 12B Q8 (Windows CUDA)11/1291.7%95% CI 64.6%–98.5%93.1%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Fixed keyword baseline4/1233.3%Label-only rules; no confidence estimate.
fresh_business · routing · nb · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API10/1283.3%95% CI 55.2%–95.3%85.3%0Reference0.0%
Laya12 inputs truncated by native defaults3/1225.0%95% CI 8.9%–53.2%8.0%0-58.3 pp 95% paired interval -91.7 to -16.70.0%
Laya multilingual6/1250.0%95% CI 25.4%–74.6%34.7%0-33.3 pp 95% paired interval -75.0 to +8.30.0%
Decider 0.8B5/1241.7%95% CI 19.3%–68.0%36.2%0-41.7 pp 95% paired interval -66.7 to -16.70.0%
Decider 2B6/1250.0%95% CI 25.4%–74.6%47.7%0-33.3 pp 95% paired interval -58.3 to -8.30.0%
Decider 4B8/1266.7%95% CI 39.1%–86.2%67.3%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
Von2/1216.7%95% CI 4.7%–44.8%10.7%0-66.7 pp 95% paired interval -100.0 to -25.00.0%
GLiNER Decide 340M4/1233.3%95% CI 13.8%–60.9%31.3%0-50.0 pp 95% paired interval -75.0 to -16.70.0%
CLM 8B INT82/1216.7%95% CI 4.7%–44.8%17.0%0-66.7 pp 95% paired interval -91.7 to -41.70.0%
Cygnet 12B NF4 (local variant)10/1283.3%95% CI 55.2%–95.3%85.3%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Decision 4B v1.29/1275.0%95% CI 46.8%–91.1%77.4%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Decision 2.0 Eos 0.8B5/1241.7%95% CI 19.3%–68.0%37.3%0-41.7 pp 95% paired interval -66.7 to -16.70.0%
Decision 2.0 Kai 0.6B2/1216.7%95% CI 4.7%–44.8%6.7%0-66.7 pp 95% paired interval -100.0 to -25.00.0%
Decision 2.0 Nox 4B10/1283.3%95% CI 55.2%–95.3%85.3%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Decision 2.0 Sol 2B7/1258.3%95% CI 32.0%–80.7%55.0%0-25.0 pp 95% paired interval -50.0 to +0.00.0%
GLiNER Decide 1B2/1216.7%95% CI 4.7%–44.8%17.3%0-66.7 pp 95% paired interval -100.0 to -25.00.0%
GLiNER Decide multilingual3/1225.0%95% CI 8.9%–53.2%24.0%0-58.3 pp 95% paired interval -83.3 to -33.30.0%
Imajev 2B8/1266.7%95% CI 39.1%–86.2%69.4%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
Imajev 4B9/1275.0%95% CI 46.8%–91.1%76.0%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Intern-Decision 4B9/1275.0%95% CI 46.8%–91.1%76.0%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Jev-Omni 12B NF4 (local variant)12/12100.0%95% CI 75.8%–100.0%100.0%0+16.7 pp 95% paired interval +0.0 to +41.70.0%
JevK5 2B5/1241.7%95% CI 19.3%–68.0%24.0%0-41.7 pp 95% paired interval -83.3 to +0.00.0%
JevK5 4B9/1275.0%95% CI 46.8%–91.1%78.1%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Julia 144M4/1233.3%95% CI 13.8%–60.9%23.3%0-50.0 pp 95% paired interval -75.0 to -25.00.0%
Kev 0.8B4/1233.3%95% CI 13.8%–60.9%20.9%0-50.0 pp 95% paired interval -83.3 to -8.30.0%
Kev 4B8/1266.7%95% CI 39.1%–86.2%70.0%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
Nev 2B Q86/1250.0%95% CI 25.4%–74.6%53.3%0-33.3 pp 95% paired interval -58.3 to -8.30.0%
Nimble 9B NF410/1283.3%95% CI 55.2%–95.3%85.3%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Plumb 4B9/1275.0%95% CI 46.8%–91.1%76.0%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Tev1 4B10/1283.3%95% CI 55.2%–95.3%85.3%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Wald 4B v1.29/1275.0%95% CI 46.8%–91.1%77.3%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Winnow 12B Q8 (Windows CUDA)11/1291.7%95% CI 64.6%–98.5%93.1%0+8.3 pp 95% paired interval +0.0 to +25.00.0%
Fixed keyword baseline6/1250.0%Label-only rules; no confidence estimate.
fresh_business · documents · en · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API12/12100.0%95% CI 75.8%–100.0%100.0%0Reference0.0%
Laya8/1266.7%95% CI 39.1%–86.2%66.7%0-33.3 pp 95% paired interval -58.3 to -8.30.0%
Laya multilingual8/1266.7%95% CI 39.1%–86.2%66.0%0-33.3 pp 95% paired interval -58.3 to -8.30.0%
Decider 0.8B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Decider 2B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Decider 4B11/1291.7%95% CI 64.6%–98.5%91.4%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Von9/1275.0%95% CI 46.8%–91.1%72.7%0-25.0 pp 95% paired interval -50.0 to +0.00.0%
GLiNER Decide 340M10/1283.3%95% CI 55.2%–95.3%82.9%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
CLM 8B INT86/1250.0%95% CI 25.4%–74.6%35.1%0-50.0 pp 95% paired interval -75.0 to -25.00.0%
Cygnet 12B NF4 (local variant)12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Decision 4B v1.212/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Decision 2.0 Eos 0.8B10/1283.3%95% CI 55.2%–95.3%83.1%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
Decision 2.0 Kai 0.6B11/1291.7%95% CI 64.6%–98.5%91.4%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Decision 2.0 Nox 4B11/1291.7%95% CI 64.6%–98.5%91.4%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Decision 2.0 Sol 2B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
GLiNER Decide 1B10/1283.3%95% CI 55.2%–95.3%82.9%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
GLiNER Decide multilingual3/1225.0%95% CI 8.9%–53.2%22.9%0-75.0 pp 95% paired interval -100.0 to -50.00.0%
Imajev 2B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Imajev 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Intern-Decision 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Jev-Omni 12B NF4 (local variant)11/1291.7%95% CI 64.6%–98.5%91.4%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
JevK5 2B11/1291.7%95% CI 64.6%–98.5%91.4%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
JevK5 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Julia 144M6/1250.0%95% CI 25.4%–74.6%48.6%0-50.0 pp 95% paired interval -75.0 to -16.70.0%
Kev 0.8B11/1291.7%95% CI 64.6%–98.5%91.4%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Kev 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Nev 2B Q812/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Nimble 9B NF412/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Plumb 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Tev1 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Wald 4B v1.212/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Winnow 12B Q8 (Windows CUDA)12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Fixed keyword baseline2/1216.7%Label-only rules; no confidence estimate.
fresh_business · documents · nb · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API12/12100.0%95% CI 75.8%–100.0%100.0%0Reference0.0%
Laya4/1233.3%95% CI 13.8%–60.9%26.1%0-66.7 pp 95% paired interval -91.7 to -41.70.0%
Laya multilingual10/1283.3%95% CI 55.2%–95.3%83.1%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
Decider 0.8B11/1291.7%95% CI 64.6%–98.5%91.4%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Decider 2B11/1291.7%95% CI 64.6%–98.5%91.4%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Decider 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Von6/1250.0%95% CI 25.4%–74.6%45.0%0-50.0 pp 95% paired interval -75.0 to -25.00.0%
GLiNER Decide 340M11/1291.7%95% CI 64.6%–98.5%91.4%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
CLM 8B INT84/1233.3%95% CI 13.8%–60.9%23.2%0-66.7 pp 95% paired interval -91.7 to -41.70.0%
Cygnet 12B NF4 (local variant)12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Decision 4B v1.212/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Decision 2.0 Eos 0.8B10/1283.3%95% CI 55.2%–95.3%80.4%0-16.7 pp 95% paired interval -41.7 to +0.00.0%
Decision 2.0 Kai 0.6B11/1291.7%95% CI 64.6%–98.5%91.4%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Decision 2.0 Nox 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Decision 2.0 Sol 2B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
GLiNER Decide 1B7/1258.3%95% CI 32.0%–80.7%58.3%0-41.7 pp 95% paired interval -66.7 to -16.70.0%
GLiNER Decide multilingual6/1250.0%95% CI 25.4%–74.6%41.0%0-50.0 pp 95% paired interval -75.0 to -25.00.0%
Imajev 2B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Imajev 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Intern-Decision 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Jev-Omni 12B NF4 (local variant)12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
JevK5 2B8/1266.7%95% CI 39.1%–86.2%65.4%0-33.3 pp 95% paired interval -58.3 to -8.30.0%
JevK5 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Julia 144M5/1241.7%95% CI 19.3%–68.0%42.5%0-58.3 pp 95% paired interval -83.3 to -25.00.0%
Kev 0.8B11/1291.7%95% CI 64.6%–98.5%91.4%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Kev 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Nev 2B Q812/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Nimble 9B NF411/1291.7%95% CI 64.6%–98.5%91.4%0-8.3 pp 95% paired interval -25.0 to +0.00.0%
Plumb 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Tev1 4B12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Wald 4B v1.212/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Winnow 12B Q8 (Windows CUDA)12/12100.0%95% CI 75.8%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.0.0%
Fixed keyword baseline3/1225.0%Label-only rules; no confidence estimate.

Ticket routing · English

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API40/40100.0%95% CI 91.2%–100.0%100.0%0Reference—
Laya29/4072.5%95% CI 57.2%–83.9%72.5%0-27.5 pp 95% paired interval -42.5 to -15.0—
Laya multilingual20/4050.0%95% CI 35.2%–64.8%46.6%0-50.0 pp 95% paired interval -65.0 to -35.0—
Decider 0.8B35/4087.5%95% CI 73.9%–94.5%86.5%0-12.5 pp 95% paired interval -25.0 to -2.5—
Decider 2B35/4087.5%95% CI 73.9%–94.5%87.1%0-12.5 pp 95% paired interval -22.6 to -2.5—
Decider 4B38/4095.0%95% CI 83.5%–98.6%95.0%0-5.0 pp 95% paired interval -12.5 to +0.0—
Von34/4085.0%95% CI 70.9%–92.9%83.4%0-15.0 pp 95% paired interval -25.0 to -5.0—
GLiNER Decide 340M34/4085.0%95% CI 70.9%–92.9%84.0%0-15.0 pp 95% paired interval -27.5 to -5.0—
CLM 8B INT88/4020.0%95% CI 10.5%–34.8%6.7%0-80.0 pp 95% paired interval -90.0 to -67.5—
Cygnet 12B NF4 (local variant)40/40100.0%95% CI 91.2%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decision 4B v1.240/40100.0%95% CI 91.2%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decision 2.0 Eos 0.8B34/4085.0%95% CI 70.9%–92.9%84.9%0-15.0 pp 95% paired interval -27.5 to -5.0—
Decision 2.0 Kai 0.6B35/4087.5%95% CI 73.9%–94.5%87.2%0-12.5 pp 95% paired interval -25.0 to -2.5—
Decision 2.0 Nox 4B38/4095.0%95% CI 83.5%–98.6%95.0%0-5.0 pp 95% paired interval -12.5 to +0.0—
Decision 2.0 Sol 2B35/4087.5%95% CI 73.9%–94.5%88.0%0-12.5 pp 95% paired interval -22.5 to -2.5—
GLiNER Decide 1B32/4080.0%95% CI 65.2%–89.5%76.6%0-20.0 pp 95% paired interval -32.5 to -7.5—
GLiNER Decide multilingual27/4067.5%95% CI 52.0%–79.9%61.8%0-32.5 pp 95% paired interval -47.5 to -17.5—
Imajev 2B36/4090.0%95% CI 76.9%–96.0%88.6%0-10.0 pp 95% paired interval -20.0 to -2.5—
Imajev 4B38/4095.0%95% CI 83.5%–98.6%95.0%0-5.0 pp 95% paired interval -12.5 to +0.0—
Intern-Decision 4B39/4097.5%95% CI 87.1%–99.6%97.5%0-2.5 pp 95% paired interval -7.5 to +0.0—
Jev-Omni 12B NF4 (local variant)40/40100.0%95% CI 91.2%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
JevK5 2B37/4092.5%95% CI 80.1%–97.4%92.0%0-7.5 pp 95% paired interval -17.5 to +0.0—
JevK5 4B37/4092.5%95% CI 80.1%–97.4%92.4%0-7.5 pp 95% paired interval -15.0 to +0.0—
Julia 144M19/4047.5%95% CI 32.9%–62.5%41.3%0-52.5 pp 95% paired interval -67.5 to -37.5—
Kev 0.8B33/4082.5%95% CI 68.1%–91.3%83.6%0-17.5 pp 95% paired interval -30.0 to -7.5—
Kev 4B35/4087.5%95% CI 73.9%–94.5%87.6%0-12.5 pp 95% paired interval -22.5 to -2.5—
Nev 2B Q834/4085.0%95% CI 70.9%–92.9%84.0%0-15.0 pp 95% paired interval -27.5 to -5.0—
Nimble 9B NF439/4097.5%95% CI 87.1%–99.6%97.5%0-2.5 pp 95% paired interval -7.5 to +0.0—
Plumb 4B38/4095.0%95% CI 83.5%–98.6%95.0%0-5.0 pp 95% paired interval -12.5 to +0.0—
Tev1 4B39/4097.5%95% CI 87.1%–99.6%97.5%0-2.5 pp 95% paired interval -7.5 to +0.0—
Wald 4B v1.238/4095.0%95% CI 83.5%–98.6%94.9%0-5.0 pp 95% paired interval -12.5 to +0.0—
Winnow 12B Q8 (Windows CUDA)40/40100.0%95% CI 91.2%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Fixed keyword baseline18/4045.0%Label-only rules; no confidence estimate.

Ticket routing · Norwegian

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API39/4097.5%95% CI 87.1%–99.6%97.5%0Reference—
Laya20/4050.0%95% CI 35.2%–64.8%45.4%0-47.5 pp 95% paired interval -62.5 to -30.0—
Laya multilingual18/4045.0%95% CI 30.7%–60.2%39.5%0-52.5 pp 95% paired interval -67.5 to -37.5—
Decider 0.8B30/4075.0%95% CI 59.8%–85.8%73.5%0-22.5 pp 95% paired interval -35.0 to -10.0—
Decider 2B30/4075.0%95% CI 59.8%–85.8%74.7%0-22.5 pp 95% paired interval -37.5 to -10.0—
Decider 4B33/4082.5%95% CI 68.1%–91.3%82.1%0-15.0 pp 95% paired interval -27.5 to -5.0—
Von15/4037.5%95% CI 24.2%–53.0%25.7%0-60.0 pp 95% paired interval -75.0 to -42.5—
GLiNER Decide 340M26/4065.0%95% CI 49.5%–77.9%64.7%0-32.5 pp 95% paired interval -47.5 to -20.0—
CLM 8B INT813/4032.5%95% CI 20.1%–48.0%22.1%0-65.0 pp 95% paired interval -80.0 to -50.0—
Cygnet 12B NF4 (local variant)38/4095.0%95% CI 83.5%–98.6%95.0%0-2.5 pp 95% paired interval -12.5 to +5.0—
Decision 4B v1.238/4095.0%95% CI 83.5%–98.6%95.0%0-2.5 pp 95% paired interval -7.5 to +0.0—
Decision 2.0 Eos 0.8B26/4065.0%95% CI 49.5%–77.9%66.0%0-32.5 pp 95% paired interval -47.5 to -20.0—
Decision 2.0 Kai 0.6B31/4077.5%95% CI 62.5%–87.7%77.6%0-20.0 pp 95% paired interval -35.0 to -7.5—
Decision 2.0 Nox 4B35/4087.5%95% CI 73.9%–94.5%86.3%0-10.0 pp 95% paired interval -20.0 to -2.5—
Decision 2.0 Sol 2B30/4075.0%95% CI 59.8%–85.8%75.9%0-22.5 pp 95% paired interval -37.5 to -7.5—
GLiNER Decide 1B23/4057.5%95% CI 42.2%–71.5%58.0%0-40.0 pp 95% paired interval -55.0 to -25.0—
GLiNER Decide multilingual27/4067.5%95% CI 52.0%–79.9%60.4%0-30.0 pp 95% paired interval -42.6 to -17.4—
Imajev 2B30/4075.0%95% CI 59.8%–85.8%73.4%0-22.5 pp 95% paired interval -35.0 to -10.0—
Imajev 4B37/4092.5%95% CI 80.1%–97.4%92.3%0-5.0 pp 95% paired interval -12.5 to +0.0—
Intern-Decision 4B39/4097.5%95% CI 87.1%–99.6%97.5%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Jev-Omni 12B NF4 (local variant)39/4097.5%95% CI 87.1%–99.6%97.5%0+0.0 pp 95% paired interval -7.5 to +7.5—
JevK5 2B32/4080.0%95% CI 65.2%–89.5%78.1%0-17.5 pp 95% paired interval -30.0 to -7.4—
JevK5 4B36/4090.0%95% CI 76.9%–96.0%88.9%0-7.5 pp 95% paired interval -17.5 to +0.0—
Julia 144M11/4027.5%95% CI 16.1%–42.8%24.3%0-70.0 pp 95% paired interval -85.0 to -52.5—
Kev 0.8B29/4072.5%95% CI 57.2%–83.9%74.0%0-25.0 pp 95% paired interval -40.0 to -10.0—
Kev 4B35/4087.5%95% CI 73.9%–94.5%87.2%0-10.0 pp 95% paired interval -20.0 to -2.5—
Nev 2B Q834/4085.0%95% CI 70.9%–92.9%82.8%0-12.5 pp 95% paired interval -25.0 to -2.5—
Nimble 9B NF439/4097.5%95% CI 87.1%–99.6%97.5%0+0.0 pp 95% paired interval -7.5 to +7.5—
Plumb 4B37/4092.5%95% CI 80.1%–97.4%92.0%0-5.0 pp 95% paired interval -12.5 to +0.0—
Tev1 4B37/4092.5%95% CI 80.1%–97.4%92.3%0-5.0 pp 95% paired interval -12.5 to +0.0—
Wald 4B v1.236/4090.0%95% CI 76.9%–96.0%88.8%0-7.5 pp 95% paired interval -17.5 to +0.0—
Winnow 12B Q8 (Windows CUDA)40/40100.0%95% CI 91.2%–100.0%100.0%0+2.5 pp 95% paired interval +0.0 to +7.5—
Fixed keyword baseline21/4052.5%Label-only rules; no confidence estimate.

Engineering documents · English

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API28/28100.0%95% CI 87.9%–100.0%100.0%0Reference—
Laya25/2889.3%95% CI 72.8%–96.3%89.1%0-10.7 pp 95% paired interval -21.4 to +0.0—
Laya multilingual23/2882.1%95% CI 64.4%–92.1%81.8%0-17.9 pp 95% paired interval -32.1 to -3.6—
Decider 0.8B26/2892.9%95% CI 77.4%–98.0%93.0%0-7.1 pp 95% paired interval -17.9 to +0.0—
Decider 2B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decider 4B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Von26/2892.9%95% CI 77.4%–98.0%92.5%0-7.1 pp 95% paired interval -17.9 to +0.0—
GLiNER Decide 340M26/2892.9%95% CI 77.4%–98.0%92.8%0-7.1 pp 95% paired interval -17.9 to +0.0—
CLM 8B INT815/2853.6%95% CI 35.8%–70.5%46.0%0-46.4 pp 95% paired interval -64.3 to -28.6—
Cygnet 12B NF4 (local variant)27/2896.4%95% CI 82.3%–99.4%96.4%0-3.6 pp 95% paired interval -10.7 to +0.0—
Decision 4B v1.227/2896.4%95% CI 82.3%–99.4%96.4%0-3.6 pp 95% paired interval -10.7 to +0.0—
Decision 2.0 Eos 0.8B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decision 2.0 Kai 0.6B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decision 2.0 Nox 4B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decision 2.0 Sol 2B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
GLiNER Decide 1B26/2892.9%95% CI 77.4%–98.0%92.5%0-7.1 pp 95% paired interval -17.9 to +0.0—
GLiNER Decide multilingual22/2878.6%95% CI 60.5%–89.8%79.8%0-21.4 pp 95% paired interval -35.7 to -7.1—
Imajev 2B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Imajev 4B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Intern-Decision 4B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Jev-Omni 12B NF4 (local variant)28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
JevK5 2B27/2896.4%95% CI 82.3%–99.4%96.4%0-3.6 pp 95% paired interval -10.7 to +0.0—
JevK5 4B27/2896.4%95% CI 82.3%–99.4%96.4%0-3.6 pp 95% paired interval -10.7 to +0.0—
Julia 144M18/2864.3%95% CI 45.8%–79.3%61.3%0-35.7 pp 95% paired interval -53.6 to -21.4—
Kev 0.8B27/2896.4%95% CI 82.3%–99.4%96.4%0-3.6 pp 95% paired interval -10.7 to +0.0—
Kev 4B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Nev 2B Q828/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Nimble 9B NF428/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Plumb 4B27/2896.4%95% CI 82.3%–99.4%96.4%0-3.6 pp 95% paired interval -10.7 to +0.0—
Tev1 4B27/2896.4%95% CI 82.3%–99.4%96.4%0-3.6 pp 95% paired interval -10.7 to +0.0—
Wald 4B v1.228/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Winnow 12B Q8 (Windows CUDA)28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Fixed keyword baseline11/2839.3%Label-only rules; no confidence estimate.

Engineering documents · Norwegian

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API28/28100.0%95% CI 87.9%–100.0%100.0%0Reference—
Laya21/2875.0%95% CI 56.6%–87.3%75.5%0-25.0 pp 95% paired interval -42.9 to -10.7—
Laya multilingual22/2878.6%95% CI 60.5%–89.8%77.9%0-21.4 pp 95% paired interval -35.7 to -7.1—
Decider 0.8B27/2896.4%95% CI 82.3%–99.4%96.4%0-3.6 pp 95% paired interval -10.7 to +0.0—
Decider 2B27/2896.4%95% CI 82.3%–99.4%96.4%0-3.6 pp 95% paired interval -10.7 to +0.0—
Decider 4B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Von12/2842.9%95% CI 26.5%–60.9%29.6%0-57.1 pp 95% paired interval -75.0 to -39.3—
GLiNER Decide 340M22/2878.6%95% CI 60.5%–89.8%78.4%0-21.4 pp 95% paired interval -35.7 to -7.1—
CLM 8B INT88/2828.6%95% CI 15.3%–47.1%16.5%0-71.4 pp 95% paired interval -85.7 to -53.6—
Cygnet 12B NF4 (local variant)28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decision 4B v1.224/2885.7%95% CI 68.5%–94.3%85.7%0-14.3 pp 95% paired interval -28.6 to -3.6—
Decision 2.0 Eos 0.8B25/2889.3%95% CI 72.8%–96.3%89.0%0-10.7 pp 95% paired interval -21.5 to +0.0—
Decision 2.0 Kai 0.6B26/2892.9%95% CI 77.4%–98.0%92.8%0-7.1 pp 95% paired interval -17.9 to +0.0—
Decision 2.0 Nox 4B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decision 2.0 Sol 2B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
GLiNER Decide 1B21/2875.0%95% CI 56.6%–87.3%76.4%0-25.0 pp 95% paired interval -42.9 to -10.7—
GLiNER Decide multilingual21/2875.0%95% CI 56.6%–87.3%75.6%0-25.0 pp 95% paired interval -42.9 to -10.7—
Imajev 2B27/2896.4%95% CI 82.3%–99.4%96.4%0-3.6 pp 95% paired interval -10.7 to +0.0—
Imajev 4B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Intern-Decision 4B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Jev-Omni 12B NF4 (local variant)26/2892.9%95% CI 77.4%–98.0%93.0%0-7.1 pp 95% paired interval -17.9 to +0.0—
JevK5 2B27/2896.4%95% CI 82.3%–99.4%96.4%0-3.6 pp 95% paired interval -10.7 to +0.0—
JevK5 4B27/2896.4%95% CI 82.3%–99.4%96.4%0-3.6 pp 95% paired interval -10.7 to +0.0—
Julia 144M14/2850.0%95% CI 32.6%–67.4%42.4%0-50.0 pp 95% paired interval -67.9 to -32.1—
Kev 0.8B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Kev 4B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Nev 2B Q827/2896.4%95% CI 82.3%–99.4%96.4%0-3.6 pp 95% paired interval -10.7 to +0.0—
Nimble 9B NF428/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Plumb 4B26/2892.9%95% CI 77.4%–98.0%92.7%0-7.1 pp 95% paired interval -17.9 to +0.0—
Tev1 4B28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Wald 4B v1.228/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Winnow 12B Q8 (Windows CUDA)28/28100.0%95% CI 87.9%–100.0%100.0%0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Fixed keyword baseline15/2853.6%Label-only rules; no confidence estimate.
laya_claim · news.replication · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API346/40086.5%95% CI 82.8%–89.5%85.4%0Reference—
Laya381/40095.2%95% CI 92.7%–96.9%94.7%0+8.8 pp 95% paired interval +5.5 to +12.5—
Laya multilingual375/40093.8%95% CI 90.9%–95.7%93.1%0+7.2 pp 95% paired interval +3.8 to +11.0—
Decider 0.8B362/40090.5%95% CI 87.2%–93.0%89.7%0+4.0 pp 95% paired interval +1.2 to +7.0—
Decider 2B356/40089.0%95% CI 85.6%–91.7%88.0%0+2.5 pp 95% paired interval +0.0 to +5.0—
Decider 4B365/40091.2%95% CI 88.1%–93.6%90.5%0+4.8 pp 95% paired interval +2.0 to +7.5—
Von346/40086.5%95% CI 82.8%–89.5%84.8%0+0.0 pp 95% paired interval -3.5 to +3.8—
GLiNER Decide 340M284/40071.0%95% CI 66.4%–75.2%66.8%0-15.5 pp 95% paired interval -19.5 to -11.5—
CLM 8B INT8155/40038.8%95% CI 34.1%–43.6%36.2%0-47.8 pp 95% paired interval -53.5 to -41.8—
Cygnet 12B NF4 (local variant)338/40084.5%95% CI 80.6%–87.7%83.1%0-2.0 pp 95% paired interval -4.0 to +0.0—
Decision 4B v1.2339/40084.8%95% CI 80.9%–87.9%83.5%0-1.8 pp 95% paired interval -4.0 to +0.5—
Decision 2.0 Eos 0.8B316/40079.0%95% CI 74.7%–82.7%77.4%0-7.5 pp 95% paired interval -10.8 to -4.2—
Decision 2.0 Kai 0.6B335/40083.8%95% CI 79.8%–87.0%82.2%0-2.8 pp 95% paired interval -5.5 to -0.2—
Decision 2.0 Nox 4B342/40085.5%95% CI 81.7%–88.6%84.2%0-1.0 pp 95% paired interval -3.2 to +1.2—
Decision 2.0 Sol 2B328/40082.0%95% CI 77.9%–85.5%80.3%0-4.5 pp 95% paired interval -7.2 to -2.0—
GLiNER Decide 1B285/40071.2%95% CI 66.6%–75.5%67.8%0-15.2 pp 95% paired interval -19.2 to -11.5—
GLiNER Decide multilingual295/40073.8%95% CI 69.2%–77.8%71.5%0-12.8 pp 95% paired interval -16.5 to -9.2—
Imajev 2B326/40081.5%95% CI 77.4%–85.0%80.7%77 native abstentions included-5.0 pp 95% paired interval -7.8 to -2.5—
Imajev 4B320/40080.0%95% CI 75.8%–83.6%80.5%2121 native abstentions included-6.5 pp 95% paired interval -9.0 to -4.0—
Intern-Decision 4B355/40088.8%95% CI 85.3%–91.5%87.7%0+2.2 pp 95% paired interval -0.2 to +4.8—
Jev-Omni 12B NF4 (local variant)331/40082.8%95% CI 78.7%–86.1%81.4%0-3.8 pp 95% paired interval -6.0 to -1.5—
JevK5 2B338/40084.5%95% CI 80.6%–87.7%83.2%0-2.0 pp 95% paired interval -4.5 to +0.5—
JevK5 4B333/40083.2%95% CI 79.3%–86.6%81.6%0-3.2 pp 95% paired interval -5.8 to -1.0—
Julia 144M344/40086.0%95% CI 82.3%–89.1%85.5%0-0.5 pp 95% paired interval -4.0 to +2.8—
Kev 0.8B358/40089.5%95% CI 86.1%–92.1%88.6%0+3.0 pp 95% paired interval -0.2 to +6.2—
Kev 4B370/40092.5%95% CI 89.5%–94.7%91.8%0+6.0 pp 95% paired interval +3.2 to +9.0—
Nev 2B Q8355/40088.8%95% CI 85.3%–91.5%87.9%0+2.2 pp 95% paired interval +0.0 to +4.5—
Nimble 9B NF4354/40088.5%95% CI 85.0%–91.3%87.7%0+2.0 pp 95% paired interval -0.5 to +4.2—
Plumb 4B336/40084.0%95% CI 80.1%–87.3%82.6%0-2.5 pp 95% paired interval -5.0 to -0.2—
Tev1 4B352/40088.0%95% CI 84.4%–90.8%87.0%0+1.5 pp 95% paired interval -1.0 to +4.0—
Wald 4B v1.2338/40084.5%95% CI 80.6%–87.7%83.0%0-2.0 pp 95% paired interval -4.5 to +0.2—
Winnow 12B Q8 (Windows CUDA)337/40084.2%95% CI 80.4%–87.5%82.9%0-2.2 pp 95% paired interval -4.2 to -0.5—
laya_claim · emotion.replication · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API237/40059.2%95% CI 54.4%–64.0%49.3%0Reference—
Laya239/40059.8%95% CI 54.9%–64.4%47.8%0+0.5 pp 95% paired interval -4.0 to +4.8—
Laya multilingual215/40053.8%95% CI 48.9%–58.6%43.4%0-5.5 pp 95% paired interval -10.2 to -0.8—
Decider 0.8B321/40080.2%95% CI 76.1%–83.9%70.4%0+21.0 pp 95% paired interval +16.5 to +25.8—
Decider 2B340/40085.0%95% CI 81.2%–88.2%78.5%0+25.8 pp 95% paired interval +20.8 to +30.8—
Decider 4B326/40081.5%95% CI 77.4%–85.0%75.1%0+22.2 pp 95% paired interval +17.5 to +27.0—
Von355/40088.8%95% CI 85.3%–91.5%76.3%0+29.5 pp 95% paired interval +25.0 to +34.2—
GLiNER Decide 340M222/40055.5%95% CI 50.6%–60.3%51.0%0-3.8 pp 95% paired interval -8.5 to +1.2—
CLM 8B INT899/40024.8%95% CI 20.8%–29.2%9.1%0-34.5 pp 95% paired interval -40.0 to -28.8—
Cygnet 12B NF4 (local variant)107/40026.8%95% CI 22.6%–31.3%9.2%0-32.5 pp 95% paired interval -38.2 to -26.5—
Decision 4B v1.2217/40054.2%95% CI 49.4%–59.1%46.5%0-5.0 pp 95% paired interval -8.5 to -1.2—
Decision 2.0 Eos 0.8B211/40052.8%95% CI 47.9%–57.6%46.5%0-6.5 pp 95% paired interval -11.5 to -1.5—
Decision 2.0 Kai 0.6B220/40055.0%95% CI 50.1%–59.8%45.8%0-4.2 pp 95% paired interval -8.8 to +0.5—
Decision 2.0 Nox 4B232/40058.0%95% CI 53.1%–62.7%50.7%0-1.2 pp 95% paired interval -5.0 to +2.8—
Decision 2.0 Sol 2B126/40031.5%95% CI 27.1%–36.2%11.4%0-27.8 pp 95% paired interval -33.2 to -22.0—
GLiNER Decide 1B196/40049.0%95% CI 44.1%–53.9%45.4%0-10.2 pp 95% paired interval -15.5 to -4.8—
GLiNER Decide multilingual171/40042.8%95% CI 38.0%–47.6%35.7%0-16.5 pp 95% paired interval -22.2 to -10.5—
Imajev 2B239/40059.8%95% CI 54.9%–64.4%50.8%3434 native abstentions included+0.5 pp 95% paired interval -3.2 to +4.2—
Imajev 4B169/40042.2%95% CI 37.5%–47.1%43.5%130130 native abstentions included-17.0 pp 95% paired interval -21.3 to -12.5—
Intern-Decision 4B215/40053.8%95% CI 48.9%–58.6%46.6%0-5.5 pp 95% paired interval -9.8 to -1.2—
Jev-Omni 12B NF4 (local variant)215/40053.8%95% CI 48.9%–58.6%44.2%0-5.5 pp 95% paired interval -9.0 to -2.0—
JevK5 2B234/40058.5%95% CI 53.6%–63.2%48.9%0-0.8 pp 95% paired interval -4.8 to +3.2—
JevK5 4B240/40060.0%95% CI 55.1%–64.7%52.3%0+0.8 pp 95% paired interval -2.5 to +4.2—
Julia 144M376/40094.0%95% CI 91.2%–95.9%89.1%0+34.8 pp 95% paired interval +30.0 to +39.5—
Kev 0.8B220/40055.0%95% CI 50.1%–59.8%44.7%0-4.2 pp 95% paired interval -8.5 to +0.2—
Kev 4B224/40056.0%95% CI 51.1%–60.8%48.7%0-3.2 pp 95% paired interval -6.8 to +0.2—
Nev 2B Q8237/40059.2%95% CI 54.4%–64.0%48.4%0+0.0 pp 95% paired interval -3.8 to +3.8—
Nimble 9B NF4234/40058.5%95% CI 53.6%–63.2%47.6%0-0.8 pp 95% paired interval -3.5 to +2.0—
Plumb 4B218/40054.5%95% CI 49.6%–59.3%44.2%0-4.8 pp 95% paired interval -8.0 to -1.2—
Tev1 4B231/40057.8%95% CI 52.9%–62.5%48.6%0-1.5 pp 95% paired interval -5.0 to +2.2—
Wald 4B v1.2239/40059.8%95% CI 54.9%–64.4%50.6%0+0.5 pp 95% paired interval -3.2 to +4.0—
Winnow 12B Q8 (Windows CUDA)246/40061.5%95% CI 56.6%–66.1%52.6%0+2.2 pp 95% paired interval -1.0 to +5.5—
laya_claim · spam.replication · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API393/40098.2%95% CI 96.4%–99.1%98.2%0Reference—
Laya91 inputs truncated by native defaults399/40099.8%95% CI 98.6%–100.0%99.7%0+1.5 pp 95% paired interval +0.5 to +2.8—
Laya multilingual9 inputs truncated by native defaults399/40099.8%95% CI 98.6%–100.0%99.7%0+1.5 pp 95% paired interval +0.2 to +3.0—
Decider 0.8B388/40097.0%95% CI 94.8%–98.3%97.0%0-1.2 pp 95% paired interval -2.8 to +0.2—
Decider 2B391/40097.8%95% CI 95.8%–98.8%97.7%0-0.5 pp 95% paired interval -2.0 to +0.8—
Decider 4B395/40098.8%95% CI 97.1%–99.5%98.7%0+0.5 pp 95% paired interval -0.8 to +1.8—
Von235/40058.8%95% CI 53.9%–63.5%55.3%0-39.5 pp 95% paired interval -44.8 to -34.5—
GLiNER Decide 340M275/40068.8%95% CI 64.0%–73.1%67.3%0-29.5 pp 95% paired interval -34.2 to -25.0—
CLM 8B INT8274/40068.5%95% CI 63.8%–72.9%68.5%0-29.8 pp 95% paired interval -34.5 to -25.0—
Cygnet 12B NF4 (local variant)380/40095.0%95% CI 92.4%–96.7%95.0%0-3.2 pp 95% paired interval -5.2 to -1.5—
Decision 4B v1.2381/40095.2%95% CI 92.7%–96.9%95.2%0-3.0 pp 95% paired interval -4.5 to -1.5—
Decision 2.0 Eos 0.8B300/40075.0%95% CI 70.5%–79.0%74.1%0-23.2 pp 95% paired interval -27.5 to -19.0—
Decision 2.0 Kai 0.6B192/40048.0%95% CI 43.1%–52.9%34.0%0-50.2 pp 95% paired interval -55.2 to -45.2—
Decision 2.0 Nox 4B374/40093.5%95% CI 90.6%–95.5%93.5%0-4.8 pp 95% paired interval -7.0 to -2.8—
Decision 2.0 Sol 2B320/40080.0%95% CI 75.8%–83.6%79.9%0-18.2 pp 95% paired interval -22.0 to -14.5—
GLiNER Decide 1B339/40084.8%95% CI 80.9%–87.9%84.6%0-13.5 pp 95% paired interval -17.2 to -9.8—
GLiNER Decide multilingual211/40052.8%95% CI 47.9%–57.6%50.4%0-45.5 pp 95% paired interval -50.8 to -40.5—
Imajev 2B277/40069.2%95% CI 64.6%–73.6%69.9%2323 native abstentions included-29.0 pp 95% paired interval -33.8 to -24.2—
Imajev 4B333/40083.2%95% CI 79.3%–86.6%84.6%1414 native abstentions included-15.0 pp 95% paired interval -18.5 to -11.5—
Intern-Decision 4B378/40094.5%95% CI 91.8%–96.3%94.5%0-3.8 pp 95% paired interval -5.8 to -1.8—
Jev-Omni 12B NF4 (local variant)384/40096.0%95% CI 93.6%–97.5%96.0%0-2.2 pp 95% paired interval -4.0 to -0.8—
JevK5 2B315/40078.8%95% CI 74.5%–82.5%78.6%0-19.5 pp 95% paired interval -23.2 to -15.8—
JevK5 4B337/40084.2%95% CI 80.4%–87.5%84.1%0-14.0 pp 95% paired interval -17.5 to -10.8—
Julia 144M180/40045.0%95% CI 40.2%–49.9%44.5%0-53.2 pp 95% paired interval -58.2 to -48.2—
Kev 0.8B277/40069.2%95% CI 64.6%–73.6%68.6%0-29.0 pp 95% paired interval -33.5 to -24.8—
Kev 4B365/40091.2%95% CI 88.1%–93.6%91.2%0-7.0 pp 95% paired interval -9.5 to -4.5—
Nev 2B Q8337/40084.2%95% CI 80.4%–87.5%84.2%0-14.0 pp 95% paired interval -17.5 to -10.5—
Nimble 9B NF4337/40084.2%95% CI 80.4%–87.5%84.1%0-14.0 pp 95% paired interval -17.8 to -10.5—
Plumb 4B369/40092.2%95% CI 89.2%–94.5%92.2%0-6.0 pp 95% paired interval -8.5 to -3.8—
Tev1 4B380/40095.0%95% CI 92.4%–96.7%95.0%0-3.2 pp 95% paired interval -5.0 to -1.5—
Wald 4B v1.2363/40090.8%95% CI 87.5%–93.2%90.7%0-7.5 pp 95% paired interval -10.2 to -5.0—
Winnow 12B Q8 (Windows CUDA)388/40097.0%95% CI 94.8%–98.3%97.0%0-1.2 pp 95% paired interval -2.2 to -0.2—
laya_claim · phishing.replication · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API360/40090.0%95% CI 86.7%–92.6%88.8%0Reference—
Laya101 inputs truncated by native defaults391/40097.8%95% CI 95.8%–98.8%97.6%0+7.8 pp 95% paired interval +4.5 to +11.0—
Laya multilingual7 inputs truncated by native defaults397/40099.2%95% CI 97.8%–99.7%99.2%0+9.2 pp 95% paired interval +6.0 to +12.5—
Decider 0.8B370/40092.5%95% CI 89.5%–94.7%91.8%0+2.5 pp 95% paired interval -0.2 to +5.5—
Decider 2B376/40094.0%95% CI 91.2%–95.9%93.6%0+4.0 pp 95% paired interval +1.2 to +7.0—
Decider 4B384/40096.0%95% CI 93.6%–97.5%95.8%0+6.0 pp 95% paired interval +2.7 to +9.5—
Von278/40069.5%95% CI 64.8%–73.8%67.5%0-20.5 pp 95% paired interval -25.5 to -15.2—
GLiNER Decide 340M258/40064.5%95% CI 59.7%–69.0%50.6%0-25.5 pp 95% paired interval -30.0 to -21.0—
CLM 8B INT8169/40042.2%95% CI 37.5%–47.1%34.6%0-47.8 pp 95% paired interval -54.5 to -41.2—
Cygnet 12B NF4 (local variant)362/40090.5%95% CI 87.2%–93.0%89.8%0+0.5 pp 95% paired interval -2.2 to +3.5—
Decision 4B v1.2366/40091.5%95% CI 88.4%–93.9%91.0%0+1.5 pp 95% paired interval -1.5 to +4.5—
Decision 2.0 Eos 0.8B306/40076.5%95% CI 72.1%–80.4%76.0%0-13.5 pp 95% paired interval -18.0 to -8.8—
Decision 2.0 Kai 0.6B286/40071.5%95% CI 66.9%–75.7%67.0%0-18.5 pp 95% paired interval -23.2 to -14.0—
Decision 2.0 Nox 4B368/40092.0%95% CI 88.9%–94.3%91.4%0+2.0 pp 95% paired interval -0.8 to +5.0—
Decision 2.0 Sol 2B341/40085.2%95% CI 81.4%–88.4%84.7%0-4.8 pp 95% paired interval -9.0 to -0.5—
GLiNER Decide 1B233/40058.2%95% CI 53.4%–63.0%43.0%0-31.8 pp 95% paired interval -36.2 to -27.0—
GLiNER Decide multilingual256/40064.0%95% CI 59.2%–68.6%55.1%0-26.0 pp 95% paired interval -31.0 to -21.0—
Imajev 2B296/40074.0%95% CI 69.5%–78.1%76.4%4444 native abstentions included-16.0 pp 95% paired interval -20.2 to -12.2—
Imajev 4B335/40083.8%95% CI 79.8%–87.0%83.8%1919 native abstentions included-6.2 pp 95% paired interval -9.2 to -3.5—
Intern-Decision 4B368/40092.0%95% CI 88.9%–94.3%91.2%0+2.0 pp 95% paired interval +0.2 to +3.8—
Jev-Omni 12B NF4 (local variant)348/40087.0%95% CI 83.3%–89.9%86.0%0-3.0 pp 95% paired interval -5.8 to -0.2—
JevK5 2B296/40074.0%95% CI 69.5%–78.1%74.0%0-16.0 pp 95% paired interval -21.2 to -10.5—
JevK5 4B354/40088.5%95% CI 85.0%–91.3%86.9%0-1.5 pp 95% paired interval -3.8 to +0.8—
Julia 144M150/40037.5%95% CI 32.9%–42.3%30.1%0-52.5 pp 95% paired interval -58.8 to -46.2—
Kev 0.8B265/40066.2%95% CI 61.5%–70.7%56.3%0-23.8 pp 95% paired interval -28.8 to -19.0—
Kev 4B361/40090.2%95% CI 86.9%–92.8%89.3%0+0.2 pp 95% paired interval -2.5 to +3.0—
Nev 2B Q8287/40071.8%95% CI 67.1%–75.9%61.9%0-18.2 pp 95% paired interval -22.2 to -14.5—
Nimble 9B NF4337/40084.2%95% CI 80.4%–87.5%81.7%0-5.8 pp 95% paired interval -8.5 to -3.2—
Plumb 4B364/40091.0%95% CI 87.8%–93.4%90.0%0+1.0 pp 95% paired interval -1.2 to +3.2—
Tev1 4B339/40084.8%95% CI 80.9%–87.9%82.2%0-5.2 pp 95% paired interval -7.8 to -2.8—
Wald 4B v1.2358/40089.5%95% CI 86.1%–92.1%88.3%0-0.5 pp 95% paired interval -2.5 to +1.8—
Winnow 12B Q8 (Windows CUDA)353/40088.2%95% CI 84.7%–91.0%87.0%0-1.8 pp 95% paired interval -4.0 to +0.2—
jev_verified · agnews · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API91/10091.0%95% CI 83.8%–95.2%90.5%0Reference—
Laya97/10097.0%95% CI 91.5%–99.0%97.0%0+6.0 pp 95% paired interval +0.0 to +12.0—
Laya multilingual91/10091.0%95% CI 83.8%–95.2%90.9%0+0.0 pp 95% paired interval -8.0 to +8.0—
Decider 0.8B81/10081.0%95% CI 72.2%–87.5%77.0%0-10.0 pp 95% paired interval -16.0 to -4.0—
Decider 2B93/10093.0%95% CI 86.3%–96.6%92.7%0+2.0 pp 95% paired interval -3.0 to +7.0—
Decider 4B96/10096.0%95% CI 90.2%–98.4%95.9%0+5.0 pp 95% paired interval -1.0 to +11.0—
Von89/10089.0%95% CI 81.4%–93.7%89.0%0-2.0 pp 95% paired interval -10.0 to +5.0—
GLiNER Decide 340M87/10087.0%95% CI 79.0%–92.2%86.0%0-4.0 pp 95% paired interval -10.0 to +2.0—
CLM 8B INT847/10047.0%95% CI 37.5%–56.7%42.1%0-44.0 pp 95% paired interval -54.0 to -34.0—
Cygnet 12B NF4 (local variant)88/10088.0%95% CI 80.2%–93.0%87.3%0-3.0 pp 95% paired interval -7.0 to +1.0—
Decision 4B v1.291/10091.0%95% CI 83.8%–95.2%90.4%0+0.0 pp 95% paired interval -3.0 to +3.0—
Decision 2.0 Eos 0.8B73/10073.0%95% CI 63.6%–80.7%68.0%0-18.0 pp 95% paired interval -26.0 to -11.0—
Decision 2.0 Kai 0.6B83/10083.0%95% CI 74.5%–89.1%81.6%0-8.0 pp 95% paired interval -14.0 to -3.0—
Decision 2.0 Nox 4B85/10085.0%95% CI 76.7%–90.7%83.0%0-6.0 pp 95% paired interval -11.0 to -2.0—
Decision 2.0 Sol 2B80/10080.0%95% CI 71.1%–86.7%76.8%0-11.0 pp 95% paired interval -18.0 to -6.0—
GLiNER Decide 1B74/10074.0%95% CI 64.6%–81.6%72.4%0-17.0 pp 95% paired interval -26.0 to -9.0—
GLiNER Decide multilingual78/10078.0%95% CI 68.9%–85.0%76.9%0-13.0 pp 95% paired interval -22.0 to -5.0—
Imajev 2B83/10083.0%95% CI 74.5%–89.1%79.9%0-8.0 pp 95% paired interval -14.0 to -2.0—
Imajev 4B86/10086.0%95% CI 77.9%–91.5%84.1%0-5.0 pp 95% paired interval -10.0 to +0.0—
Intern-Decision 4B92/10092.0%95% CI 85.0%–95.9%91.6%0+1.0 pp 95% paired interval -3.0 to +6.0—
Jev-Omni 12B NF4 (local variant)85/10085.0%95% CI 76.7%–90.7%82.9%0-6.0 pp 95% paired interval -11.0 to -2.0—
JevK5 2B85/10085.0%95% CI 76.7%–90.7%83.1%0-6.0 pp 95% paired interval -12.0 to -1.0—
JevK5 4B86/10086.0%95% CI 77.9%–91.5%84.3%0-5.0 pp 95% paired interval -10.0 to -1.0—
Julia 144M94/10094.0%95% CI 87.5%–97.2%94.0%0+3.0 pp 95% paired interval -4.0 to +10.0—
Kev 0.8B92/10092.0%95% CI 85.0%–95.9%91.9%0+1.0 pp 95% paired interval -6.0 to +8.0—
Kev 4B95/10095.0%95% CI 88.8%–97.8%95.0%0+4.0 pp 95% paired interval -3.0 to +11.0—
Nev 2B Q888/10088.0%95% CI 80.2%–93.0%87.0%0-3.0 pp 95% paired interval -7.0 to +0.0—
Nimble 9B NF494/10094.0%95% CI 87.5%–97.2%93.9%0+3.0 pp 95% paired interval -3.0 to +9.0—
Plumb 4B89/10089.0%95% CI 81.4%–93.7%88.0%0-2.0 pp 95% paired interval -5.0 to +0.0—
Tev1 4B96/10096.0%95% CI 90.2%–98.4%96.0%0+5.0 pp 95% paired interval -1.0 to +11.0—
Wald 4B v1.283/10083.0%95% CI 74.5%–89.1%80.1%0-8.0 pp 95% paired interval -14.0 to -3.0—
Winnow 12B Q8 (Windows CUDA)88/10088.0%95% CI 80.2%–93.0%86.8%0-3.0 pp 95% paired interval -7.0 to +0.0—
jev_verified · emotiondair · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API48/10048.0%95% CI 38.5%–57.7%48.0%0Reference—
Laya40/10040.0%95% CI 30.9%–49.8%37.4%0-8.0 pp 95% paired interval -17.0 to +1.0—
Laya multilingual22/10022.0%95% CI 15.0%–31.1%16.1%0-26.0 pp 95% paired interval -37.0 to -15.0—
Decider 0.8B61/10061.0%95% CI 51.2%–70.0%59.9%0+13.0 pp 95% paired interval +2.0 to +24.0—
Decider 2B76/10076.0%95% CI 66.8%–83.3%75.8%0+28.0 pp 95% paired interval +18.0 to +38.0—
Decider 4B71/10071.0%95% CI 61.5%–79.0%70.8%0+23.0 pp 95% paired interval +13.0 to +34.0—
Von71/10071.0%95% CI 61.5%–79.0%68.0%0+23.0 pp 95% paired interval +11.0 to +36.0—
GLiNER Decide 340M56/10056.0%95% CI 46.2%–65.3%55.8%0+8.0 pp 95% paired interval -2.0 to +18.0—
CLM 8B INT822/10022.0%95% CI 15.0%–31.1%12.7%0-26.0 pp 95% paired interval -36.0 to -16.0—
Cygnet 12B NF4 (local variant)43/10043.0%95% CI 33.7%–52.8%41.3%0-5.0 pp 95% paired interval -13.0 to +3.0—
Decision 4B v1.251/10051.0%95% CI 41.3%–60.6%51.4%0+3.0 pp 95% paired interval -5.0 to +11.0—
Decision 2.0 Eos 0.8B49/10049.0%95% CI 39.4%–58.7%48.7%0+1.0 pp 95% paired interval -10.0 to +12.0—
Decision 2.0 Kai 0.6B53/10053.0%95% CI 43.3%–62.5%53.4%0+5.0 pp 95% paired interval -3.0 to +13.0—
Decision 2.0 Nox 4B51/10051.0%95% CI 41.3%–60.6%51.3%0+3.0 pp 95% paired interval -6.0 to +12.0—
Decision 2.0 Sol 2B52/10052.0%95% CI 42.3%–61.5%50.0%0+4.0 pp 95% paired interval -5.0 to +14.0—
GLiNER Decide 1B48/10048.0%95% CI 38.5%–57.7%45.4%0+0.0 pp 95% paired interval -12.0 to +12.0—
GLiNER Decide multilingual49/10049.0%95% CI 39.4%–58.7%45.8%0+1.0 pp 95% paired interval -10.0 to +12.0—
Imajev 2B45/10045.0%95% CI 35.6%–54.8%47.1%1414 native abstentions included-3.0 pp 95% paired interval -10.0 to +4.0—
Imajev 4B36/10036.0%95% CI 27.3%–45.8%41.2%3939 native abstentions included-12.0 pp 95% paired interval -21.0 to -3.0—
Intern-Decision 4B47/10047.0%95% CI 37.5%–56.7%45.4%0-1.0 pp 95% paired interval -9.0 to +7.0—
Jev-Omni 12B NF4 (local variant)46/10046.0%95% CI 36.6%–55.7%45.7%0-2.0 pp 95% paired interval -10.0 to +6.0—
JevK5 2B46/10046.0%95% CI 36.6%–55.7%46.2%0-2.0 pp 95% paired interval -10.0 to +7.0—
JevK5 4B49/10049.0%95% CI 39.4%–58.7%47.0%0+1.0 pp 95% paired interval -7.0 to +10.0—
Julia 144M86/10086.0%95% CI 77.9%–91.5%85.6%0+38.0 pp 95% paired interval +28.0 to +49.0—
Kev 0.8B45/10045.0%95% CI 35.6%–54.8%44.7%0-3.0 pp 95% paired interval -14.0 to +8.0—
Kev 4B45/10045.0%95% CI 35.6%–54.8%46.2%0-3.0 pp 95% paired interval -11.0 to +6.0—
Nev 2B Q848/10048.0%95% CI 38.5%–57.7%49.8%0+0.0 pp 95% paired interval -9.0 to +9.0—
Nimble 9B NF445/10045.0%95% CI 35.6%–54.8%44.3%0-3.0 pp 95% paired interval -11.0 to +5.0—
Plumb 4B47/10047.0%95% CI 37.5%–56.7%47.8%0-1.0 pp 95% paired interval -9.0 to +7.0—
Tev1 4B48/10048.0%95% CI 38.5%–57.7%46.9%0+0.0 pp 95% paired interval -8.0 to +8.0—
Wald 4B v1.243/10043.0%95% CI 33.7%–52.8%43.8%0-5.0 pp 95% paired interval -13.0 to +3.0—
Winnow 12B Q8 (Windows CUDA)52/10052.0%95% CI 42.3%–61.5%51.0%0+4.0 pp 95% paired interval -2.0 to +11.0—
jev_verified · banking77 · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API89/10089.0%95% CI 81.4%–93.7%88.3%0Reference—
Laya100 inputs truncated by native defaults2/1002.0%95% CI 0.6%–7.0%0.1%0-87.0 pp 95% paired interval -94.0 to -79.0—
Laya multilingual100 inputs truncated by native defaults2/1002.0%95% CI 0.6%–7.0%0.1%0-87.0 pp 95% paired interval -93.0 to -80.0—
Decider 0.8B78/10078.0%95% CI 68.9%–85.0%75.7%0-11.0 pp 95% paired interval -20.0 to -2.0—
Decider 2B78/10078.0%95% CI 68.9%–85.0%73.5%0-11.0 pp 95% paired interval -19.0 to -4.0—
Decider 4B89/10089.0%95% CI 81.4%–93.7%86.8%0+0.0 pp 95% paired interval -6.0 to +6.0—
Von78/10078.0%95% CI 68.9%–85.0%75.7%0-11.0 pp 95% paired interval -18.0 to -4.0—
GLiNER Decide 340M75/10075.0%95% CI 65.7%–82.5%72.3%0-14.0 pp 95% paired interval -22.0 to -6.0—
CLM 8B INT88/1008.0%95% CI 4.1%–15.0%4.7%0-81.0 pp 95% paired interval -89.0 to -72.0—
Cygnet 12B NF4 (local variant)0/1000.0%95% CI 0.0%–3.7%0.0%100-89.0 pp 95% paired interval -95.0 to -82.0—
Decision 4B v1.20/1000.0%95% CI 0.0%–3.7%0.0%100-89.0 pp 95% paired interval -95.0 to -82.0—
Decision 2.0 Eos 0.8B90/10090.0%95% CI 82.6%–94.5%89.4%0+1.0 pp 95% paired interval -6.0 to +9.0—
Decision 2.0 Kai 0.6B81/10081.0%95% CI 72.2%–87.5%81.1%0-8.0 pp 95% paired interval -15.0 to -2.0—
Decision 2.0 Nox 4B86/10086.0%95% CI 77.9%–91.5%85.7%0-3.0 pp 95% paired interval -8.0 to +2.0—
Decision 2.0 Sol 2B89/10089.0%95% CI 81.4%–93.7%88.4%0+0.0 pp 95% paired interval -5.0 to +6.0—
GLiNER Decide 1B44/10044.0%95% CI 34.7%–53.8%38.2%0-45.0 pp 95% paired interval -55.0 to -34.0—
GLiNER Decide multilingual61/10061.0%95% CI 51.2%–70.0%58.8%0-28.0 pp 95% paired interval -37.0 to -20.0—
Imajev 2B77/10077.0%95% CI 67.8%–84.2%72.6%0-12.0 pp 95% paired interval -20.0 to -5.0—
Imajev 4B88/10088.0%95% CI 80.2%–93.0%85.0%0-1.0 pp 95% paired interval -6.0 to +4.0—
Intern-Decision 4B0/1000.0%95% CI 0.0%–3.7%0.0%100-89.0 pp 95% paired interval -95.0 to -82.0—
Jev-Omni 12B NF4 (local variant)52/10052.0%95% CI 42.3%–61.5%35.6%0-37.0 pp 95% paired interval -47.0 to -27.0—
JevK5 2B61/10061.0%95% CI 51.2%–70.0%57.4%0-28.0 pp 95% paired interval -37.0 to -19.0—
JevK5 4B76/10076.0%95% CI 66.8%–83.3%74.2%0-13.0 pp 95% paired interval -21.0 to -6.0—
Julia 144M0/1000.0%95% CI 0.0%–3.7%0.0%100-89.0 pp 95% paired interval -95.0 to -82.0—
Kev 0.8B87/10087.0%95% CI 79.0%–92.2%85.9%0-2.0 pp 95% paired interval -8.0 to +4.0—
Kev 4B88/10088.0%95% CI 80.2%–93.0%86.6%0-1.0 pp 95% paired interval -7.0 to +5.0—
Nev 2B Q80/1000.0%95% CI 0.0%–3.7%0.0%100-89.0 pp 95% paired interval -95.0 to -82.0—
Nimble 9B NF484/10084.0%95% CI 75.6%–89.9%83.4%0-5.0 pp 95% paired interval -12.0 to +1.0—
Plumb 4B77/10077.0%95% CI 67.8%–84.2%73.3%0-12.0 pp 95% paired interval -19.0 to -5.0—
Tev1 4B0/1000.0%95% CI 0.0%–3.7%0.0%100-89.0 pp 95% paired interval -95.0 to -82.0—
Wald 4B v1.284/10084.0%95% CI 75.6%–89.9%81.1%0-5.0 pp 95% paired interval -12.0 to +1.0—
Winnow 12B Q8 (Windows CUDA)0/1000.0%95% CI 0.0%–3.7%0.0%100-89.0 pp 95% paired interval -95.0 to -82.0—
jevbench_public · easy · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API48/48100.0%95% CI 92.6%–100.0%—0Reference—
Laya46/4895.8%95% CI 86.0%–98.8%—0-4.2 pp 95% paired interval -10.4 to +0.0—
Laya multilingual43/4889.6%95% CI 77.8%–95.5%—0-10.4 pp 95% paired interval -18.8 to -2.1—
Decider 0.8B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decider 2B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decider 4B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Von48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
GLiNER Decide 340M47/4897.9%95% CI 89.1%–99.6%—0-2.1 pp 95% paired interval -6.2 to +0.0—
CLM 8B INT833/4868.8%95% CI 54.7%–80.1%—0-31.2 pp 95% paired interval -43.8 to -18.8—
Cygnet 12B NF4 (local variant)48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decision 4B v1.248/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decision 2.0 Eos 0.8B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decision 2.0 Kai 0.6B47/4897.9%95% CI 89.1%–99.6%—0-2.1 pp 95% paired interval -6.2 to +0.0—
Decision 2.0 Nox 4B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Decision 2.0 Sol 2B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
GLiNER Decide 1B47/4897.9%95% CI 89.1%–99.6%—0-2.1 pp 95% paired interval -6.2 to +0.0—
GLiNER Decide multilingual47/4897.9%95% CI 89.1%–99.6%—0-2.1 pp 95% paired interval -6.2 to +0.0—
Imajev 2B47/4897.9%95% CI 89.1%–99.6%—11 native abstentions included-2.1 pp 95% paired interval -6.2 to +0.0—
Imajev 4B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Intern-Decision 4B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Jev-Omni 12B NF4 (local variant)48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
JevK5 2B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
JevK5 4B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Julia 144M35/4872.9%95% CI 59.0%–83.4%—0-27.1 pp 95% paired interval -39.6 to -14.6—
Kev 0.8B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Kev 4B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Nev 2B Q848/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Nimble 9B NF448/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Plumb 4B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Tev1 4B48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Wald 4B v1.248/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
Winnow 12B Q8 (Windows CUDA)48/48100.0%95% CI 92.6%–100.0%—0+0.0 pp No variation in observed paired differences; bootstrap cannot establish equivalence.—
jevbench_public · original · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API70/7297.2%95% source-cluster CI 93.1%–100.0%—0Reference—
Laya52/7272.2%95% source-cluster CI 59.7%–83.3%—0-25.0 pp 95% paired interval -37.5 to -12.5—
Laya multilingual29/7240.3%95% source-cluster CI 27.8%–52.8%—0-56.9 pp 95% paired interval -69.4 to -43.1—
Decider 0.8B61/7284.7%95% source-cluster CI 75.0%–93.1%—0-12.5 pp 95% paired interval -22.2 to -4.1—
Decider 2B64/7288.9%95% source-cluster CI 79.2%–97.2%—0-8.3 pp 95% paired interval -18.1 to +0.0—
Decider 4B71/7298.6%95% source-cluster CI 95.8%–100.0%—0+1.4 pp 95% paired interval -2.8 to +5.6—
Von46/7263.9%95% source-cluster CI 50.0%–77.8%—0-33.3 pp 95% paired interval -47.2 to -18.1—
GLiNER Decide 340M48/7266.7%95% source-cluster CI 54.2%–77.8%—0-30.6 pp 95% paired interval -44.4 to -18.1—
CLM 8B INT825/7234.7%95% source-cluster CI 20.8%–48.6%—0-62.5 pp 95% paired interval -75.0 to -48.6—
Cygnet 12B NF4 (local variant)65/7290.3%95% source-cluster CI 80.6%–97.2%—0-6.9 pp 95% paired interval -16.7 to +0.0—
Decision 4B v1.269/7295.8%95% source-cluster CI 88.9%–100.0%—0-1.4 pp 95% paired interval -6.9 to +2.8—
Decision 2.0 Eos 0.8B64/7288.9%95% source-cluster CI 80.6%–95.8%—0-8.3 pp 95% paired interval -16.7 to -1.4—
Decision 2.0 Kai 0.6B61/7284.7%95% source-cluster CI 76.4%–93.1%—0-12.5 pp 95% paired interval -20.8 to -5.6—
Decision 2.0 Nox 4B68/7294.4%95% source-cluster CI 86.1%–100.0%—0-2.8 pp 95% paired interval -12.5 to +4.2—
Decision 2.0 Sol 2B67/7293.1%95% source-cluster CI 84.7%–98.6%—0-4.2 pp 95% paired interval -11.1 to +1.4—
GLiNER Decide 1B35/7248.6%95% source-cluster CI 34.7%–63.9%—0-48.6 pp 95% paired interval -62.5 to -33.3—
GLiNER Decide multilingual42/7258.3%95% source-cluster CI 44.4%–73.6%—0-38.9 pp 95% paired interval -54.2 to -23.6—
Imajev 2B65/7290.3%95% source-cluster CI 81.9%–97.2%—11 native abstentions included-6.9 pp 95% paired interval -15.3 to +0.0—
Imajev 4B70/7297.2%95% source-cluster CI 93.1%–100.0%—11 native abstentions included+0.0 pp 95% paired interval -4.2 to +4.2—
Intern-Decision 4B71/7298.6%95% source-cluster CI 95.8%–100.0%—0+1.4 pp 95% paired interval -2.8 to +6.9—
Jev-Omni 12B NF4 (local variant)69/7295.8%95% source-cluster CI 90.3%–100.0%—0-1.4 pp 95% paired interval -6.9 to +2.8—
JevK5 2B57/7279.2%95% source-cluster CI 68.1%–90.3%—0-18.1 pp 95% paired interval -29.2 to -8.3—
JevK5 4B68/7294.4%95% source-cluster CI 87.5%–100.0%—0-2.8 pp 95% paired interval -11.1 to +4.2—
Julia 144M34/7247.2%95% source-cluster CI 31.9%–61.1%—0-50.0 pp 95% paired interval -66.7 to -36.1—
Kev 0.8B58/7280.6%95% source-cluster CI 69.4%–90.3%—0-16.7 pp 95% paired interval -27.8 to -5.6—
Kev 4B67/7293.1%95% source-cluster CI 83.3%–100.0%—0-4.2 pp 95% paired interval -13.9 to +4.2—
Nev 2B Q863/7287.5%95% source-cluster CI 80.6%–94.4%—0-9.7 pp 95% paired interval -18.1 to -2.8—
Nimble 9B NF468/7294.4%95% source-cluster CI 86.1%–100.0%—0-2.8 pp 95% paired interval -12.5 to +4.2—
Plumb 4B70/7297.2%95% source-cluster CI 93.1%–100.0%—0+0.0 pp 95% paired interval -5.6 to +5.6—
Tev1 4B69/7295.8%95% source-cluster CI 88.9%–100.0%—0-1.4 pp 95% paired interval -9.7 to +5.6—
Wald 4B v1.272/72100.0%95% source-cluster CI 100.0%–100.0%—0+2.8 pp 95% paired interval +0.0 to +6.9—
Winnow 12B Q8 (Windows CUDA)69/7295.8%95% source-cluster CI 88.9%–100.0%—0-1.4 pp 95% paired interval -8.3 to +4.2—
jevbench_public · hard · inspect complete results
Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API81/11173.0%95% CI 64.0%–80.4%—0Reference—
Laya57 inputs truncated by native defaults37/11133.3%95% CI 25.3%–42.5%—0-39.6 pp 95% paired interval -51.4 to -27.0—
Laya multilingual44 inputs truncated by native defaults37/11133.3%95% CI 25.3%–42.5%—0-39.6 pp 95% paired interval -51.4 to -27.9—
Decider 0.8B43/11138.7%95% CI 30.2%–48.0%—0-34.2 pp 95% paired interval -45.0 to -23.4—
Decider 2B63/11156.8%95% CI 47.5%–65.6%—0-16.2 pp 95% paired interval -26.1 to -6.3—
Decider 4B72/11164.9%95% CI 55.6%–73.1%—0-8.1 pp 95% paired interval -16.2 to +0.0—
Von50/11145.0%95% CI 36.1%–54.3%—0-27.9 pp 95% paired interval -38.7 to -16.2—
GLiNER Decide 340M36/11132.4%95% CI 24.4%–41.6%—0-40.5 pp 95% paired interval -50.5 to -29.7—
CLM 8B INT838/11134.2%95% CI 26.1%–43.5%—0-38.7 pp 95% paired interval -51.4 to -25.2—
Cygnet 12B NF4 (local variant)82/11173.9%95% CI 65.0%–81.1%—0+0.9 pp 95% paired interval -8.1 to +9.9—
Decision 4B v1.286/11177.5%95% CI 68.9%–84.3%—0+4.5 pp 95% paired interval -3.6 to +12.6—
Decision 2.0 Eos 0.8B51/11145.9%95% CI 37.0%–55.2%—0-27.0 pp 95% paired interval -37.8 to -16.2—
Decision 2.0 Kai 0.6B44/11139.6%95% CI 31.0%–48.9%—0-33.3 pp 95% paired interval -44.1 to -22.5—
Decision 2.0 Nox 4B69/11162.2%95% CI 52.9%–70.6%—0-10.8 pp 95% paired interval -18.0 to -3.6—
Decision 2.0 Sol 2B51/11145.9%95% CI 37.0%–55.2%—0-27.0 pp 95% paired interval -36.9 to -17.1—
GLiNER Decide 1B39/11135.1%95% CI 26.9%–44.4%—0-37.8 pp 95% paired interval -47.7 to -27.9—
GLiNER Decide multilingual42/11137.8%95% CI 29.4%–47.1%—0-35.1 pp 95% paired interval -45.9 to -23.4—
Imajev 2B62/11155.9%95% CI 46.6%–64.7%—22 native abstentions included-17.1 pp 95% paired interval -27.0 to -7.2—
Imajev 4B80/11172.1%95% CI 63.1%–79.6%—0-0.9 pp 95% paired interval -7.2 to +5.4—
Intern-Decision 4B82/11173.9%95% CI 65.0%–81.1%—0+0.9 pp 95% paired interval -8.1 to +9.9—
Jev-Omni 12B NF4 (local variant)81/11173.0%95% CI 64.0%–80.4%—0+0.0 pp 95% paired interval -8.1 to +8.1—
JevK5 2B68/11161.3%95% CI 52.0%–69.8%—0-11.7 pp 95% paired interval -21.6 to -1.8—
JevK5 4B86/11177.5%95% CI 68.9%–84.3%—0+4.5 pp 95% paired interval -2.7 to +11.7—
Julia 144M40/11136.0%95% CI 27.7%–45.3%—0-36.9 pp 95% paired interval -48.6 to -25.2—
Kev 0.8B41/11136.9%95% CI 28.5%–46.2%—0-36.0 pp 95% paired interval -46.8 to -25.2—
Kev 4B60/11154.1%95% CI 44.8%–63.0%—0-18.9 pp 95% paired interval -27.9 to -10.8—
Nev 2B Q875/11167.6%95% CI 58.4%–75.6%—0-5.4 pp 95% paired interval -14.4 to +4.5—
Nimble 9B NF465/11158.6%95% CI 49.3%–67.3%—0-14.4 pp 95% paired interval -22.5 to -6.3—
Plumb 4B89/11180.2%95% CI 71.8%–86.5%—0+7.2 pp 95% paired interval +0.9 to +14.4—
Tev1 4B57/11151.4%95% CI 42.2%–60.4%—0-21.6 pp 95% paired interval -30.6 to -13.5—
Wald 4B v1.277/11169.4%95% CI 60.3%–77.2%—10-3.6 pp 95% paired interval -12.6 to +5.4—
Winnow 12B Q8 (Windows CUDA)82/11173.9%95% CI 65.0%–81.1%—0+0.9 pp 95% paired interval -7.2 to +9.0—
Public financial-document decisions · inspect complete results

Accuracy above counts a request correct only when every question is correct; per-question accuracy is also shown. Kev was trained on these task distributions: CFPB complaint templates, generated skill families and public developer-data proxies. These tests do not establish independent industrial-domain performance. The common developer benchmark retains all 1073 questions; the separate claim audit follows the publisher amendment to 1071.

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API465/57481.0%95% CI 77.6%–84.0%820/936 individual questions correct (87.6%)—0Reference—
Laya275 inputs truncated by native defaults222/57438.7%95% CI 34.8%–42.7%513/936 individual questions correct (54.8%)—0-42.3 pp 95% paired interval -46.9 to -38.0—
Laya multilingual159 inputs truncated by native defaults175/57430.5%95% CI 26.9%–34.4%436/936 individual questions correct (46.6%)—0-50.5 pp 95% paired interval -55.2 to -46.2—
Decider 0.8B320/57455.7%95% CI 51.7%–59.8%652/936 individual questions correct (69.7%)—0-25.3 pp 95% paired interval -29.4 to -21.3—
Decider 2B373/57465.0%95% CI 61.0%–68.8%723/936 individual questions correct (77.2%)—0-16.0 pp 95% paired interval -19.9 to -12.2—
Decider 4B415/57472.3%95% CI 68.5%–75.8%761/936 individual questions correct (81.3%)—0-8.7 pp 95% paired interval -12.4 to -5.2—
Von229/57439.9%95% CI 36.0%–44.0%525/936 individual questions correct (56.1%)—0-41.1 pp 95% paired interval -45.8 to -36.8—
GLiNER Decide 340M237/57441.3%95% CI 37.3%–45.4%528/936 individual questions correct (56.4%)—0-39.7 pp 95% paired interval -44.1 to -35.0—
CLM 8B INT857/5749.9%95% CI 7.7%–12.6%233/936 individual questions correct (24.9%)—0-71.1 pp 95% paired interval -74.9 to -67.2—
Cygnet 12B NF4 (local variant)449/57478.2%95% CI 74.7%–81.4%800/936 individual questions correct (85.5%)—0-2.8 pp 95% paired interval -6.1 to +0.3—
Decision 4B v1.2413/57472.0%95% CI 68.1%–75.5%762/936 individual questions correct (81.4%)—0-9.1 pp 95% paired interval -12.7 to -5.4—
Decision 2.0 Eos 0.8B291/57450.7%95% CI 46.6%–54.8%609/936 individual questions correct (65.1%)—0-30.3 pp 95% paired interval -34.7 to -25.8—
Decision 2.0 Kai 0.6B270/57447.0%95% CI 43.0%–51.1%581/936 individual questions correct (62.1%)—0-34.0 pp 95% paired interval -38.2 to -29.6—
Decision 2.0 Nox 4B403/57470.2%95% CI 66.3%–73.8%750/936 individual questions correct (80.1%)—0-10.8 pp 95% paired interval -14.1 to -7.5—
Decision 2.0 Sol 2B300/57452.3%95% CI 48.2%–56.3%622/936 individual questions correct (66.5%)—0-28.7 pp 95% paired interval -32.9 to -24.4—
GLiNER Decide 1B242/57442.2%95% CI 38.2%–46.2%541/936 individual questions correct (57.8%)—0-38.9 pp 95% paired interval -43.2 to -34.1—
GLiNER Decide multilingual242/57442.2%95% CI 38.2%–46.2%532/936 individual questions correct (56.8%)—0-38.9 pp 95% paired interval -43.2 to -34.1—
Imajev 2B345/57460.1%95% CI 56.0%–64.0%682/936 individual questions correct (72.9%)—4544 native abstentions included-20.9 pp 95% paired interval -25.1 to -16.7—
Imajev 4B353/57461.5%95% CI 57.5%–65.4%678/936 individual questions correct (72.4%)—119118 native abstentions included-19.5 pp 95% paired interval -23.7 to -15.3—
Intern-Decision 4B426/57474.2%95% CI 70.5%–77.6%778/936 individual questions correct (83.1%)—0-6.8 pp 95% paired interval -10.3 to -3.5—
Jev-Omni 12B NF4 (local variant)417/57472.6%95% CI 68.9%–76.1%773/936 individual questions correct (82.6%)—0-8.4 pp 95% paired interval -11.8 to -5.1—
JevK5 2B350/57461.0%95% CI 56.9%–64.9%695/936 individual questions correct (74.3%)—0-20.0 pp 95% paired interval -24.0 to -16.0—
JevK5 4B417/57472.6%95% CI 68.9%–76.1%770/936 individual questions correct (82.3%)—1-8.4 pp 95% paired interval -11.8 to -5.1—
Julia 144M195/57434.0%95% CI 30.2%–37.9%459/936 individual questions correct (49.0%)—0-47.0 pp 95% paired interval -51.6 to -42.5—
Kev 0.8B442/57477.0%95% CI 73.4%–80.3%796/936 individual questions correct (85.0%)—0-4.0 pp 95% paired interval -7.8 to -0.3—
Kev 4B488/57485.0%95% CI 81.9%–87.7%848/936 individual questions correct (90.6%)—0+4.0 pp 95% paired interval +0.9 to +7.1—
Nev 2B Q8350/57461.0%95% CI 56.9%–64.9%690/936 individual questions correct (73.7%)—0-20.0 pp 95% paired interval -24.2 to -16.0—
Nimble 9B NF4398/57469.3%95% CI 65.4%–73.0%748/936 individual questions correct (79.9%)—1-11.7 pp 95% paired interval -15.9 to -8.0—
Plumb 4B428/57474.6%95% CI 70.8%–78.0%779/936 individual questions correct (83.2%)—1-6.4 pp 95% paired interval -9.9 to -3.1—
Tev1 4B432/57475.3%95% CI 71.6%–78.6%780/936 individual questions correct (83.3%)—0-5.7 pp 95% paired interval -9.1 to -2.3—
Wald 4B v1.2403/57470.2%95% CI 66.3%–73.8%749/936 individual questions correct (80.0%)—8-10.8 pp 95% paired interval -14.3 to -7.5—
Winnow 12B Q8 (Windows CUDA)461/57480.3%95% CI 76.9%–83.4%820/936 individual questions correct (87.6%)—0-0.7 pp 95% paired interval -3.7 to +1.9—
Templated decision skills · inspect complete results

Accuracy above counts a request correct only when every question is correct; per-question accuracy is also shown. Kev was trained on these task distributions: CFPB complaint templates, generated skill families and public developer-data proxies. These tests do not establish independent industrial-domain performance. The common developer benchmark retains all 1073 questions; the separate claim audit follows the publisher amendment to 1071.

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API490/70070.0%95% source-cluster CI 66.6%–73.3%849/1088 individual questions correct (78.0%)—0Reference—
Laya100 inputs truncated by native defaults113/70016.1%95% source-cluster CI 13.5%–18.9%310/1088 individual questions correct (28.5%)—0-53.9 pp 95% paired interval -58.2 to -49.6—
Laya multilingual100 inputs truncated by native defaults116/70016.6%95% source-cluster CI 13.9%–19.4%320/1088 individual questions correct (29.4%)—0-53.4 pp 95% paired interval -57.7 to -49.1—
Decider 0.8B178/70025.4%95% source-cluster CI 22.3%–28.7%412/1088 individual questions correct (37.9%)—0-44.6 pp 95% paired interval -48.8 to -40.1—
Decider 2B269/70038.4%95% source-cluster CI 35.0%–42.2%560/1088 individual questions correct (51.5%)—0-31.6 pp 95% paired interval -35.7 to -27.5—
Decider 4B363/70051.9%95% source-cluster CI 48.3%–55.5%689/1088 individual questions correct (63.3%)—0-18.1 pp 95% paired interval -22.4 to -14.0—
Von141/70020.1%95% source-cluster CI 17.2%–23.0%368/1088 individual questions correct (33.8%)—0-49.9 pp 95% paired interval -54.3 to -45.6—
GLiNER Decide 340M162/70023.1%95% source-cluster CI 20.1%–26.2%324/1088 individual questions correct (29.8%)—0-46.9 pp 95% paired interval -51.3 to -42.5—
CLM 8B INT8134/70019.1%95% source-cluster CI 16.0%–22.1%354/1088 individual questions correct (32.5%)—0-50.9 pp 95% paired interval -55.5 to -46.5—
Cygnet 12B NF4 (local variant)288/70041.1%95% source-cluster CI 37.6%–44.9%610/1088 individual questions correct (56.1%)—0-28.9 pp 95% paired interval -33.5 to -24.3—
Decision 4B v1.2364/70052.0%95% source-cluster CI 48.2%–55.6%672/1088 individual questions correct (61.8%)—0-18.0 pp 95% paired interval -22.2 to -14.0—
Decision 2.0 Eos 0.8B161/70023.0%95% source-cluster CI 20.0%–26.2%362/1088 individual questions correct (33.3%)—0-47.0 pp 95% paired interval -51.3 to -42.8—
Decision 2.0 Kai 0.6B173/70024.7%95% source-cluster CI 21.5%–28.0%383/1088 individual questions correct (35.2%)—0-45.3 pp 95% paired interval -49.6 to -40.9—
Decision 2.0 Nox 4B352/70050.3%95% source-cluster CI 46.8%–53.8%672/1088 individual questions correct (61.8%)—0-19.7 pp 95% paired interval -23.6 to -15.7—
Decision 2.0 Sol 2B161/70023.0%95% source-cluster CI 19.9%–26.1%396/1088 individual questions correct (36.4%)—0-47.0 pp 95% paired interval -51.1 to -42.8—
GLiNER Decide 1B135/70019.3%95% source-cluster CI 16.4%–22.3%317/1088 individual questions correct (29.1%)—0-50.7 pp 95% paired interval -55.2 to -46.2—
GLiNER Decide multilingual116/70016.6%95% source-cluster CI 13.8%–19.4%302/1088 individual questions correct (27.8%)—0-53.4 pp 95% paired interval -57.7 to -49.1—
Imajev 2B222/70031.7%95% source-cluster CI 28.4%–35.1%470/1088 individual questions correct (43.2%)—31-38.3 pp 95% paired interval -42.6 to -34.0—
Imajev 4B409/70058.4%95% source-cluster CI 54.9%–61.9%719/1088 individual questions correct (66.1%)—321 native abstentions included-11.6 pp 95% paired interval -15.5 to -7.4—
Intern-Decision 4B293/70041.9%95% source-cluster CI 38.4%–45.5%591/1088 individual questions correct (54.3%)—11-28.1 pp 95% paired interval -32.0 to -24.1—
Jev-Omni 12B NF4 (local variant)366/70052.3%95% source-cluster CI 48.6%–56.1%693/1088 individual questions correct (63.7%)—0-17.7 pp 95% paired interval -21.8 to -13.8—
JevK5 2B221/70031.6%95% source-cluster CI 28.2%–35.0%491/1088 individual questions correct (45.1%)—0-38.4 pp 95% paired interval -42.7 to -33.9—
JevK5 4B333/70047.6%95% source-cluster CI 43.8%–51.2%621/1088 individual questions correct (57.1%)—42-22.4 pp 95% paired interval -26.5 to -18.3—
Julia 144M159/70022.7%95% source-cluster CI 19.6%–25.9%361/1088 individual questions correct (33.2%)—0-47.3 pp 95% paired interval -51.8 to -42.7—
Kev 0.8B378/70054.0%95% source-cluster CI 50.2%–57.6%722/1088 individual questions correct (66.4%)—0-16.0 pp 95% paired interval -20.9 to -11.5—
Kev 4B507/70072.4%95% source-cluster CI 69.1%–75.7%876/1088 individual questions correct (80.5%)—0+2.4 pp 95% paired interval -1.6 to +6.3—
Nev 2B Q8219/70031.3%95% source-cluster CI 28.0%–34.7%491/1088 individual questions correct (45.1%)—0-38.7 pp 95% paired interval -43.0 to -34.3—
Nimble 9B NF4316/70045.1%95% source-cluster CI 41.8%–48.8%593/1088 individual questions correct (54.5%)—51-24.9 pp 95% paired interval -28.9 to -20.5—
Plumb 4B337/70048.1%95% source-cluster CI 44.3%–51.9%618/1088 individual questions correct (56.8%)—41-21.9 pp 95% paired interval -26.2 to -17.8—
Tev1 4B316/70045.1%95% source-cluster CI 41.5%–48.7%616/1088 individual questions correct (56.6%)—0-24.9 pp 95% paired interval -29.1 to -20.6—
Wald 4B v1.2363/70051.9%95% source-cluster CI 48.2%–55.5%649/1088 individual questions correct (59.7%)—68-18.1 pp 95% paired interval -22.1 to -14.1—
Winnow 12B Q8 (Windows CUDA)428/70061.1%95% source-cluster CI 57.4%–64.7%769/1088 individual questions correct (70.7%)—0-8.9 pp 95% paired interval -12.8 to -5.0—
Public developer-tooling decisions · inspect complete results

Accuracy above counts a request correct only when every question is correct; per-question accuracy is also shown. Kev was trained on these task distributions: CFPB complaint templates, generated skill families and public developer-data proxies. These tests do not establish independent industrial-domain performance. The common developer benchmark retains all 1073 questions; the separate claim audit follows the publisher amendment to 1071.

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API629/90069.9%95% source-cluster CI 66.2%–74.1%791/1073 individual questions correct (73.7%)—0Reference—
Laya168 inputs truncated by native defaults439/90048.8%95% source-cluster CI 45.6%–51.7%568/1073 individual questions correct (52.9%)—0-21.1 pp 95% paired interval -26.4 to -16.4—
Laya multilingual47 inputs truncated by native defaults411/90045.7%95% source-cluster CI 41.1%–49.5%522/1073 individual questions correct (48.6%)—0-24.2 pp 95% paired interval -31.1 to -18.2—
Decider 0.8B431/90047.9%95% source-cluster CI 45.1%–50.6%561/1073 individual questions correct (52.3%)—0-22.0 pp 95% paired interval -26.9 to -17.7—
Decider 2B434/90048.2%95% source-cluster CI 45.0%–51.2%568/1073 individual questions correct (52.9%)—0-21.7 pp 95% paired interval -26.8 to -17.2—
Decider 4B531/90059.0%95% source-cluster CI 55.8%–63.0%689/1073 individual questions correct (64.2%)—0-10.9 pp 95% paired interval -14.6 to -7.4—
Von389/90043.2%95% source-cluster CI 38.2%–47.7%519/1073 individual questions correct (48.4%)—0-26.7 pp 95% paired interval -33.7 to -20.1—
GLiNER Decide 340M443/90049.2%95% source-cluster CI 46.5%–51.9%588/1073 individual questions correct (54.8%)—0-20.7 pp 95% paired interval -25.3 to -16.5—
CLM 8B INT8391/90043.4%95% source-cluster CI 36.7%–49.9%481/1073 individual questions correct (44.8%)—0-26.4 pp 95% paired interval -35.4 to -17.5—
Cygnet 12B NF4 (local variant)602/90066.9%95% source-cluster CI 62.8%–71.9%765/1073 individual questions correct (71.3%)—0-3.0 pp 95% paired interval -5.6 to -0.3—
Decision 4B v1.2578/90064.2%95% source-cluster CI 60.4%–68.6%733/1073 individual questions correct (68.3%)—0-5.7 pp 95% paired interval -9.6 to -2.2—
Decision 2.0 Eos 0.8B438/90048.7%95% source-cluster CI 45.5%–51.6%568/1073 individual questions correct (52.9%)—0-21.2 pp 95% paired interval -26.7 to -16.2—
Decision 2.0 Kai 0.6B420/90046.7%95% source-cluster CI 43.9%–49.3%548/1073 individual questions correct (51.1%)—0-23.2 pp 95% paired interval -28.3 to -18.9—
Decision 2.0 Nox 4B588/90065.3%95% source-cluster CI 61.4%–69.7%752/1073 individual questions correct (70.1%)—0-4.6 pp 95% paired interval -7.6 to -1.8—
Decision 2.0 Sol 2B512/90056.9%95% source-cluster CI 53.4%–60.4%656/1073 individual questions correct (61.1%)—0-13.0 pp 95% paired interval -17.8 to -8.8—
GLiNER Decide 1B472/90052.4%95% source-cluster CI 49.2%–55.5%593/1073 individual questions correct (55.3%)—0-17.4 pp 95% paired interval -22.2 to -13.1—
GLiNER Decide multilingual393/90043.7%95% source-cluster CI 39.7%–47.3%504/1073 individual questions correct (47.0%)—0-26.2 pp 95% paired interval -32.0 to -21.2—
Imajev 2B498/90055.3%95% source-cluster CI 52.3%–58.3%653/1073 individual questions correct (60.9%)—55 native abstentions included-14.6 pp 95% paired interval -19.5 to -10.1—
Imajev 4B553/90061.4%95% source-cluster CI 57.6%–66.0%715/1073 individual questions correct (66.6%)—3737 native abstentions included-8.4 pp 95% paired interval -12.8 to -4.6—
Intern-Decision 4B557/90061.9%95% source-cluster CI 58.4%–65.6%721/1073 individual questions correct (67.2%)—0-8.0 pp 95% paired interval -11.2 to -5.1—
Jev-Omni 12B NF4 (local variant)573/90063.7%95% source-cluster CI 59.5%–67.4%739/1073 individual questions correct (68.9%)—0-6.2 pp 95% paired interval -11.5 to -1.0—
JevK5 2B412/90045.8%95% source-cluster CI 42.6%–48.6%549/1073 individual questions correct (51.2%)—0-24.1 pp 95% paired interval -29.2 to -19.4—
JevK5 4B530/90058.9%95% source-cluster CI 55.7%–62.8%683/1073 individual questions correct (63.7%)—0-11.0 pp 95% paired interval -14.4 to -7.9—
Julia 144M326/90036.2%95% source-cluster CI 30.9%–40.6%436/1073 individual questions correct (40.6%)—0-33.7 pp 95% paired interval -41.5 to -26.7—
Kev 0.8B517/90057.4%95% source-cluster CI 50.9%–63.4%684/1073 individual questions correct (63.7%)—0-12.4 pp 95% paired interval -21.0 to -4.4—
Kev 4B640/90071.1%95% source-cluster CI 67.6%–74.0%810/1073 individual questions correct (75.5%)—0+1.2 pp 95% paired interval -3.8 to +5.7—
Nev 2B Q8461/90051.2%95% source-cluster CI 48.4%–54.2%605/1073 individual questions correct (56.4%)—0-18.7 pp 95% paired interval -23.0 to -14.9—
Nimble 9B NF4584/90064.9%95% source-cluster CI 60.9%–69.8%745/1073 individual questions correct (69.4%)—0-5.0 pp 95% paired interval -8.7 to -0.9—
Plumb 4B579/90064.3%95% source-cluster CI 59.7%–68.6%742/1073 individual questions correct (69.2%)—0-5.6 pp 95% paired interval -10.9 to -0.3—
Tev1 4B584/90064.9%95% source-cluster CI 60.8%–69.6%750/1073 individual questions correct (69.9%)—0-5.0 pp 95% paired interval -7.9 to -2.4—
Wald 4B v1.2595/90066.1%95% source-cluster CI 62.0%–71.2%747/1073 individual questions correct (69.6%)—1-3.8 pp 95% paired interval -7.4 to +0.0—
Winnow 12B Q8 (Windows CUDA)610/90067.8%95% source-cluster CI 63.8%–72.8%770/1073 individual questions correct (71.8%)—0-2.1 pp 95% paired interval -6.2 to +1.4—

18 intent scenarios · English

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API2550/294886.5%95% CI 85.2%–87.7%86.1%0Reference—
Laya1962/294866.6%95% CI 64.8%–68.2%69.0%0-19.9 pp 95% paired interval -21.6 to -18.4—
Laya multilingual1815/294861.6%95% CI 59.8%–63.3%61.3%0-24.9 pp 95% paired interval -26.7 to -23.2—
Decider 0.8B2423/294882.2%95% CI 80.8%–83.5%81.7%0-4.3 pp 95% paired interval -5.5 to -3.1—
Decider 2B2433/294882.5%95% CI 81.1%–83.9%82.8%0-4.0 pp 95% paired interval -5.1 to -2.9—
Decider 4B2603/294888.3%95% CI 87.1%–89.4%88.0%0+1.8 pp 95% paired interval +0.9 to +2.6—
Von1665/294856.5%95% CI 54.7%–58.3%61.5%0-30.0 pp 95% paired interval -31.9 to -28.1—
GLiNER Decide 340M2067/294870.1%95% CI 68.4%–71.7%70.7%0-16.4 pp 95% paired interval -17.9 to -14.8—
CLM 8B INT81229/294841.7%95% CI 39.9%–43.5%37.3%0-44.8 pp 95% paired interval -46.7 to -42.7—
Cygnet 12B NF4 (local variant)2272/294877.1%95% CI 75.5%–78.6%76.9%0-9.4 pp 95% paired interval -10.9 to -8.0—
Decision 4B v1.22425/294882.3%95% CI 80.8%–83.6%82.3%0-4.2 pp 95% paired interval -5.3 to -3.2—
Decision 2.0 Eos 0.8B1989/294867.5%95% CI 65.8%–69.1%67.4%0-19.0 pp 95% paired interval -20.7 to -17.5—
Decision 2.0 Kai 0.6B2149/294872.9%95% CI 71.3%–74.5%73.6%0-13.6 pp 95% paired interval -15.0 to -12.1—
Decision 2.0 Nox 4B2512/294885.2%95% CI 83.9%–86.4%84.8%0-1.3 pp 95% paired interval -2.2 to -0.3—
Decision 2.0 Sol 2B2165/294873.4%95% CI 71.8%–75.0%72.5%0-13.1 pp 95% paired interval -14.5 to -11.7—
GLiNER Decide 1B1729/294858.6%95% CI 56.9%–60.4%62.2%0-27.8 pp 95% paired interval -29.7 to -25.9—
GLiNER Decide multilingual1625/294855.1%95% CI 53.3%–56.9%56.6%0-31.4 pp 95% paired interval -33.3 to -29.4—
Imajev 2B2439/294882.7%95% CI 81.3%–84.1%84.4%6262 native abstentions included-3.8 pp 95% paired interval -5.0 to -2.6—
Imajev 4B2524/294885.6%95% CI 84.3%–86.8%87.3%154154 native abstentions included-0.9 pp 95% paired interval -1.9 to +0.2—
Intern-Decision 4B2448/294883.0%95% CI 81.6%–84.4%83.1%0-3.5 pp 95% paired interval -4.5 to -2.4—
Jev-Omni 12B NF4 (local variant)2365/294880.2%95% CI 78.7%–81.6%77.8%0-6.3 pp 95% paired interval -7.5 to -5.0—
JevK5 2B2089/294870.9%95% CI 69.2%–72.5%73.3%0-15.6 pp 95% paired interval -17.2 to -14.1—
JevK5 4B2375/294880.6%95% CI 79.1%–82.0%80.5%0-5.9 pp 95% paired interval -7.1 to -4.7—
Julia 144M1256/294842.6%95% CI 40.8%–44.4%44.6%0-43.9 pp 95% paired interval -45.8 to -42.1—
Kev 0.8B2114/294871.7%95% CI 70.1%–73.3%73.6%0-14.8 pp 95% paired interval -16.2 to -13.2—
Kev 4B2327/294878.9%95% CI 77.4%–80.4%79.6%0-7.6 pp 95% paired interval -8.8 to -6.3—
Nev 2B Q82449/294883.1%95% CI 81.7%–84.4%82.7%0-3.4 pp 95% paired interval -4.5 to -2.4—
Nimble 9B NF42411/294881.8%95% CI 80.3%–83.1%82.6%0-4.7 pp 95% paired interval -5.9 to -3.6—
Plumb 4B2438/294882.7%95% CI 81.3%–84.0%81.6%0-3.8 pp 95% paired interval -4.9 to -2.7—
Tev1 4B2421/294882.1%95% CI 80.7%–83.5%82.1%0-4.4 pp 95% paired interval -5.5 to -3.3—
Wald 4B v1.22507/294885.0%95% CI 83.7%–86.3%85.3%0-1.5 pp 95% paired interval -2.4 to -0.4—
Winnow 12B Q8 (Windows CUDA)2454/294883.2%95% CI 81.9%–84.5%83.8%0-3.3 pp 95% paired interval -4.3 to -2.2—

18 intent scenarios · Norwegian Bokmål

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API2412/294881.8%95% CI 80.4%–83.2%81.6%0Reference—
Laya621/294821.1%95% CI 19.6%–22.6%25.6%0-60.8 pp 95% paired interval -62.7 to -58.9—
Laya multilingual1511/294851.3%95% CI 49.5%–53.1%52.5%0-30.6 pp 95% paired interval -32.4 to -28.7—
Decider 0.8B2021/294868.6%95% CI 66.9%–70.2%67.5%0-13.3 pp 95% paired interval -14.8 to -11.7—
Decider 2B2099/294871.2%95% CI 69.5%–72.8%73.2%0-10.6 pp 95% paired interval -12.1 to -9.1—
Decider 4B2418/294882.0%95% CI 80.6%–83.4%81.6%0+0.2 pp 95% paired interval -0.9 to +1.3—
Von394/294813.4%95% CI 12.2%–14.6%14.5%0-68.5 pp 95% paired interval -70.2 to -66.6—
GLiNER Decide 340M1557/294852.8%95% CI 51.0%–54.6%51.7%0-29.0 pp 95% paired interval -30.8 to -27.1—
CLM 8B INT8894/294830.3%95% CI 28.7%–32.0%25.1%0-51.5 pp 95% paired interval -53.5 to -49.4—
Cygnet 12B NF4 (local variant)2013/294868.3%95% CI 66.6%–69.9%67.7%0-13.5 pp 95% paired interval -15.2 to -11.9—
Decision 4B v1.22150/294872.9%95% CI 71.3%–74.5%73.6%0-8.9 pp 95% paired interval -10.3 to -7.5—
Decision 2.0 Eos 0.8B1551/294852.6%95% CI 50.8%–54.4%53.5%0-29.2 pp 95% paired interval -31.1 to -27.6—
Decision 2.0 Kai 0.6B1614/294854.7%95% CI 52.9%–56.5%56.7%0-27.1 pp 95% paired interval -28.9 to -25.3—
Decision 2.0 Nox 4B2367/294880.3%95% CI 78.8%–81.7%79.1%0-1.5 pp 95% paired interval -2.6 to -0.4—
Decision 2.0 Sol 2B1898/294864.4%95% CI 62.6%–66.1%64.6%0-17.4 pp 95% paired interval -19.1 to -15.7—
GLiNER Decide 1B729/294824.7%95% CI 23.2%–26.3%27.3%0-57.1 pp 95% paired interval -59.0 to -55.3—
GLiNER Decide multilingual1455/294849.4%95% CI 47.6%–51.2%51.7%0-32.5 pp 95% paired interval -34.5 to -30.4—
Imajev 2B2156/294873.1%95% CI 71.5%–74.7%74.6%3737 native abstentions included-8.7 pp 95% paired interval -10.1 to -7.3—
Imajev 4B2424/294882.2%95% CI 80.8%–83.6%83.0%135135 native abstentions included+0.4 pp 95% paired interval -0.9 to +1.7—
Intern-Decision 4B2126/294872.1%95% CI 70.5%–73.7%73.0%0-9.7 pp 95% paired interval -11.0 to -8.3—
Jev-Omni 12B NF4 (local variant)2055/294869.7%95% CI 68.0%–71.3%67.9%0-12.1 pp 95% paired interval -13.5 to -10.6—
JevK5 2B1682/294857.1%95% CI 55.3%–58.8%61.5%0-24.8 pp 95% paired interval -26.5 to -23.0—
JevK5 4B2133/294872.4%95% CI 70.7%–73.9%72.0%0-9.5 pp 95% paired interval -10.8 to -8.1—
Julia 144M1174/294839.8%95% CI 38.1%–41.6%39.2%0-42.0 pp 95% paired interval -44.0 to -40.0—
Kev 0.8B1719/294858.3%95% CI 56.5%–60.1%62.5%0-23.5 pp 95% paired interval -25.3 to -21.7—
Kev 4B2123/294872.0%95% CI 70.4%–73.6%72.8%0-9.8 pp 95% paired interval -11.1 to -8.4—
Nev 2B Q82183/294874.1%95% CI 72.4%–75.6%74.2%0-7.8 pp 95% paired interval -9.2 to -6.3—
Nimble 9B NF42181/294874.0%95% CI 72.4%–75.5%75.5%0-7.8 pp 95% paired interval -9.2 to -6.5—
Plumb 4B2174/294873.7%95% CI 72.1%–75.3%72.9%0-8.1 pp 95% paired interval -9.4 to -6.9—
Tev1 4B2143/294872.7%95% CI 71.1%–74.3%73.7%0-9.1 pp 95% paired interval -10.4 to -7.8—
Wald 4B v1.22379/294880.7%95% CI 79.2%–82.1%80.6%0-1.1 pp 95% paired interval -2.3 to +0.1—
Winnow 12B Q8 (Windows CUDA)2299/294878.0%95% CI 76.5%–79.4%79.5%0-3.8 pp 95% paired interval -5.0 to -2.6—

Norwegian review sentiment

Completed taskAccuracyMacro F1FailedDifference vs live JevAccepted without review
Jev 1.13.0 · live API843/117371.9%95% source-cluster CI 68.6%–75.0%70.0%0Reference—
Laya626/117353.4%95% source-cluster CI 50.5%–56.3%47.0%0-18.5 pp 95% paired interval -23.1 to -13.6—
Laya multilingual696/117359.3%95% source-cluster CI 56.2%–62.2%53.4%0-12.5 pp 95% paired interval -16.3 to -8.7—
Decider 0.8B793/117367.6%95% source-cluster CI 64.3%–70.8%62.5%0-4.3 pp 95% paired interval -9.0 to +0.2—
Decider 2B818/117369.7%95% source-cluster CI 67.0%–72.6%65.7%0-2.1 pp 95% paired interval -5.6 to +1.3—
Decider 4B850/117372.5%95% source-cluster CI 69.2%–75.6%68.8%0+0.6 pp 95% paired interval -2.2 to +3.3—
Von582/117349.6%95% source-cluster CI 45.2%–53.7%29.6%0-22.3 pp 95% paired interval -28.2 to -16.5—
GLiNER Decide 340M703/117359.9%95% source-cluster CI 56.4%–63.2%54.4%0-11.9 pp 95% paired interval -16.0 to -8.1—
CLM 8B INT8522/117344.5%95% source-cluster CI 39.4%–48.8%21.8%0-27.4 pp 95% paired interval -33.7 to -21.3—
Cygnet 12B NF4 (local variant)758/117364.6%95% source-cluster CI 60.5%–68.5%63.1%0-7.2 pp 95% paired interval -10.1 to -4.3—
Decision 4B v1.2787/117367.1%95% source-cluster CI 63.4%–70.6%64.9%0-4.8 pp 95% paired interval -7.2 to -2.3—
Decision 2.0 Eos 0.8B626/117353.4%95% source-cluster CI 50.2%–56.2%50.9%0-18.5 pp 95% paired interval -22.0 to -15.2—
Decision 2.0 Kai 0.6B677/117357.7%95% source-cluster CI 55.2%–60.4%54.3%0-14.2 pp 95% paired interval -17.6 to -10.6—
Decision 2.0 Nox 4B844/117372.0%95% source-cluster CI 68.7%–75.5%68.8%0+0.1 pp 95% paired interval -2.9 to +3.2—
Decision 2.0 Sol 2B671/117357.2%95% source-cluster CI 54.7%–59.8%56.2%0-14.7 pp 95% paired interval -18.1 to -11.2—
GLiNER Decide 1B484/117341.3%95% source-cluster CI 37.7%–44.8%37.6%0-30.6 pp 95% paired interval -35.6 to -25.8—
GLiNER Decide multilingual780/117366.5%95% source-cluster CI 62.4%–70.2%56.9%0-5.4 pp 95% paired interval -9.8 to -1.4—
Imajev 2B770/117365.6%95% source-cluster CI 62.2%–68.9%59.8%2424 native abstentions included-6.2 pp 95% paired interval -10.8 to -2.0—
Imajev 4B819/117369.8%95% source-cluster CI 66.7%–72.9%65.2%2323 native abstentions included-2.0 pp 95% paired interval -5.6 to +1.4—
Intern-Decision 4B802/117368.4%95% source-cluster CI 65.0%–71.7%65.6%0-3.5 pp 95% paired interval -5.8 to -1.2—
Jev-Omni 12B NF4 (local variant)743/117363.3%95% source-cluster CI 59.8%–67.0%62.3%0-8.5 pp 95% paired interval -11.3 to -5.6—
JevK5 2B612/117352.2%95% source-cluster CI 49.1%–55.1%50.8%0-19.7 pp 95% paired interval -23.1 to -16.5—
JevK5 4B813/117369.3%95% source-cluster CI 66.1%–72.5%66.2%0-2.6 pp 95% paired interval -5.1 to -0.2—
Julia 144M541/117346.1%95% source-cluster CI 43.3%–49.0%41.3%0-25.7 pp 95% paired interval -30.3 to -21.2—
Kev 0.8B757/117364.5%95% source-cluster CI 61.3%–67.6%58.0%0-7.3 pp 95% paired interval -11.6 to -3.4—
Kev 4B867/117373.9%95% source-cluster CI 70.6%–77.2%68.9%0+2.0 pp 95% paired interval -1.2 to +5.2—
Nev 2B Q8797/117367.9%95% source-cluster CI 65.1%–70.7%64.7%0-3.9 pp 95% paired interval -6.7 to -0.9—
Nimble 9B NF4636/117354.2%95% source-cluster CI 49.4%–58.6%53.1%0-17.6 pp 95% paired interval -22.0 to -14.0—
Plumb 4B806/117368.7%95% source-cluster CI 65.4%–71.8%66.2%0-3.2 pp 95% paired interval -5.6 to -0.9—
Tev1 4B857/117373.1%95% source-cluster CI 69.9%–76.2%70.3%0+1.2 pp 95% paired interval -1.7 to +3.9—
Wald 4B v1.2854/117372.8%95% source-cluster CI 69.5%–76.2%67.4%0+0.9 pp 95% paired interval -2.8 to +4.5—
Winnow 12B Q8 (Windows CUDA)863/117373.6%95% source-cluster CI 70.8%–76.4%71.5%0+1.7 pp 95% paired interval -0.6 to +4.0—

Repeatability and option order

Two exact repeats each preserved 111/112 and 111/112 sampled Jev decisions. Reversing the available choice order preserved 87/88. These probes use cases selected by a fixed ID hash, independent of their results; they are diagnostic repeats, not additional accuracy evidence.

Changed case IDs and probability changes · Request, model and spend reconciliation

Inspect a business example

These are saved benchmark outputs. Selecting a case makes no API call. Review the expected label and model suggestions to understand failure modes.

ModelSuggested labelConfidenceMatches expected label
Inspect full saved probabilities


Translate a pilot into business value
These are editable assumptions, not measured savings. Use net time saved after review and corrections. Infrastructure, integration, support and API costs belong in annual operating cost.


Use one consistent currency for the hourly rate, operating cost and result. This model does not monetize avoided downtime or safety outcomes.


Audit and limitations
Machine-readable results and calibration metrics · Frozen protocol · Fresh-case provenance · Token and budget ledger summary · Full local comparison · Published claim audit · Historical Jev verification


Local models run on an RTX 5070 Ti 16 GB in Python virtual environments; precision and adapter details are retained per model. API timings include network transport and concurrent requests, and are not directly comparable with local GPU throughput or H100 marketing numbers. The current run cannot establish advertised speed or cost multiples against unrelated LLM workflows. Published input pricing was $42 per billion tokens; output tokens are free according to the API schema. Estimated spend is not an account statement.