Unofficial local software evaluation · 2026-10-05

Decision models under test

31 of 31 models have recorded results for all shared fixtures.
Updated 2026-10-05T00:49:25.539223+00:00. Partial groups are excluded from accuracy tables. Native rejections and runtime failures remain in the denominator. Use as is, without warranty or vendor endorsement.

The question is where a small local model makes software more useful: classifying documents, suggesting ticket queues, or screening engineering records. Public benchmark replication and independent task transfer answer different questions. There is no pooled winner across unrelated tasks.

Live and verified Jev comparisons

Current live Jev comparison and business findings covers the shared fixtures and new disjoint test cases. Historical Jev comparison on matching cases uses 231 verified public per-case outcomes and a separate exact 300-case pilot. Claim-by-claim audit separates reproduced, near-reproduced and unverifiable statements. Historical scores, current API calls and sealed leaderboard claims remain separate.

Execution status

RTX 5070 Ti 16 GB; Windows; Python venvs; 78% Torch allocation cap. Failed requests stay in accuracy denominators. An unfinished or failed adapter is not a measured zero.

Model	Practical cases	Public claim cases	Typed cases · 5 decisions each	Exact Jev pilot	Fresh cases · dev + test
CLM 8B INT8	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Cygnet 12B NF4 (local variant)	12,341 / 12,341200 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 300100 failed requests	1,264 / 1,2640 failed requests
Decider 0.8B	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Decider 2B	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Decider 4B	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Decision 4B v1.2	12,341 / 12,341200 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 300100 failed requests	1,264 / 1,2640 failed requests
Decision 2.0 Eos 0.8B	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Decision 2.0 Kai 0.6B	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Decision 2.0 Nox 4B	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Decision 2.0 Sol 2B	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
GLiNER Decide 1B	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
GLiNER Decide multilingual	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
GLiNER Decide 340M	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Imajev 2B	12,341 / 12,3410 failed requests	3,271 / 3,27132 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Imajev 4B	12,341 / 12,3410 failed requests	3,271 / 3,27132 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Intern-Decision 4B	12,341 / 12,341100 failed requests	3,271 / 3,27111 failed requests	400 / 4000 failed requests	300 / 300100 failed requests	1,264 / 1,2640 failed requests
Jev-Omni 12B NF4 (local variant)	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
JevK5 2B	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
JevK5 4B	12,341 / 12,3410 failed requests	3,271 / 3,27143 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Julia 144M	12,341 / 12,341200 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 300100 failed requests	1,264 / 1,2640 failed requests
Kev 0.8B	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Kev 4B	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Laya multilingual	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Laya	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Nev 2B Q8	12,341 / 12,341200 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 300100 failed requests	1,264 / 1,2640 failed requests
Nimble 9B NF4	12,341 / 12,3410 failed requests	3,271 / 3,27152 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Plumb 4B	12,341 / 12,3410 failed requests	3,271 / 3,27142 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Tev1 4B	12,341 / 12,341200 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 300100 failed requests	1,264 / 1,2640 failed requests
Von	12,341 / 12,3410 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Wald 4B v1.2	12,341 / 12,3410 failed requests	3,271 / 3,27187 failed requests	400 / 4000 failed requests	300 / 3000 failed requests	1,264 / 1,2640 failed requests
Winnow 12B Q8 (Windows CUDA)	12,341 / 12,341100 failed requests	3,271 / 3,2710 failed requests	400 / 4000 failed requests	300 / 300100 failed requests	1,264 / 1,2640 failed requests

Julia: first native claim check

Our run: 72.55% (1,451/2,000 decisions). Published: 73.15%. Difference: -0.60 percentage points.

Unmodified publisher reproduce_typed.py; verified model and Parquet hashes; CPU fp32 native Torch backend, batch one, 1024-token context. Near reproduction within the predeclared 2 percentage point band; not exact numerical equality. This verifies a public benchmark result, not deployment performance.

Published metrics · Local result

English intent scenarios

Completed suite	Accuracy	95% interval	Request coverage
Decider 4B	88.3%	87.1%–89.4%	100.0%
Imajev 4B	85.6%	84.3%–86.8%	94.8%
Decision 2.0 Nox 4B	85.2%	83.9%–86.4%	100.0%
Wald 4B v1.2	85.0%	83.7%–86.3%	100.0%
Winnow 12B Q8 (Windows CUDA)	83.2%	81.9%–84.5%	100.0%
Nev 2B Q8	83.1%	81.7%–84.4%	100.0%
Intern-Decision 4B	83.0%	81.6%–84.4%	100.0%
Imajev 2B	82.7%	81.3%–84.1%	97.9%
Plumb 4B	82.7%	81.3%–84.0%	100.0%
Decider 2B	82.5%	81.1%–83.9%	100.0%
Decision 4B v1.2	82.3%	80.8%–83.6%	100.0%
Decider 0.8B	82.2%	80.8%–83.5%	100.0%
Tev1 4B	82.1%	80.7%–83.5%	100.0%
Nimble 9B NF4	81.8%	80.3%–83.1%	100.0%
JevK5 4B	80.6%	79.1%–82.0%	100.0%
Jev-Omni 12B NF4 (local variant)	80.2%	78.7%–81.6%	100.0%
Kev 4B	78.9%	77.4%–80.4%	100.0%
Cygnet 12B NF4 (local variant)	77.1%	75.5%–78.6%	100.0%
Decision 2.0 Sol 2B	73.4%	71.8%–75.0%	100.0%
Decision 2.0 Kai 0.6B	72.9%	71.3%–74.5%	100.0%
Kev 0.8B	71.7%	70.1%–73.3%	100.0%
JevK5 2B	70.9%	69.2%–72.5%	100.0%
GLiNER Decide 340M	70.1%	68.4%–71.7%	100.0%
Decision 2.0 Eos 0.8B	67.5%	65.8%–69.1%	100.0%
Laya	66.6%	64.8%–68.2%	100.0%
Laya multilingual	61.6%	59.8%–63.3%	100.0%
GLiNER Decide 1B	58.6%	56.9%–60.4%	100.0%
Von	56.5%	54.7%–58.3%	100.0%
GLiNER Decide multilingual	55.1%	53.3%–56.9%	100.0%
Julia 144M	42.6%	40.8%–44.4%	100.0%
CLM 8B INT8	41.7%	39.9%–43.5%	100.0%

Norwegian intent scenarios

Completed suite	Accuracy	95% interval	Request coverage
Imajev 4B	82.2%	80.8%–83.6%	95.4%
Decider 4B	82.0%	80.6%–83.4%	100.0%
Wald 4B v1.2	80.7%	79.2%–82.1%	100.0%
Decision 2.0 Nox 4B	80.3%	78.8%–81.7%	100.0%
Winnow 12B Q8 (Windows CUDA)	78.0%	76.5%–79.4%	100.0%
Nev 2B Q8	74.1%	72.4%–75.6%	100.0%
Nimble 9B NF4	74.0%	72.4%–75.5%	100.0%
Plumb 4B	73.7%	72.1%–75.3%	100.0%
Imajev 2B	73.1%	71.5%–74.7%	98.7%
Decision 4B v1.2	72.9%	71.3%–74.5%	100.0%
Tev1 4B	72.7%	71.1%–74.3%	100.0%
JevK5 4B	72.4%	70.7%–73.9%	100.0%
Intern-Decision 4B	72.1%	70.5%–73.7%	100.0%
Kev 4B	72.0%	70.4%–73.6%	100.0%
Decider 2B	71.2%	69.5%–72.8%	100.0%
Jev-Omni 12B NF4 (local variant)	69.7%	68.0%–71.3%	100.0%
Decider 0.8B	68.6%	66.9%–70.2%	100.0%
Cygnet 12B NF4 (local variant)	68.3%	66.6%–69.9%	100.0%
Decision 2.0 Sol 2B	64.4%	62.6%–66.1%	100.0%
Kev 0.8B	58.3%	56.5%–60.1%	100.0%
JevK5 2B	57.1%	55.3%–58.8%	100.0%
Decision 2.0 Kai 0.6B	54.7%	52.9%–56.5%	100.0%
GLiNER Decide 340M	52.8%	51.0%–54.6%	100.0%
Decision 2.0 Eos 0.8B	52.6%	50.8%–54.4%	100.0%
Laya multilingual	51.3%	49.5%–53.1%	100.0%
GLiNER Decide multilingual	49.4%	47.6%–51.2%	100.0%
Julia 144M	39.8%	38.1%–41.6%	100.0%
CLM 8B INT8	30.3%	28.7%–32.0%	100.0%
GLiNER Decide 1B	24.7%	23.2%–26.3%	100.0%
Laya	21.1%	19.6%–22.6%	100.0%
Von	13.4%	12.2%–14.6%	100.0%

Industry routing · English

Completed suite	Accuracy	95% interval	Request coverage
Winnow 12B Q8 (Windows CUDA)	100.0%	91.2%–100.0%	100.0%
Jev-Omni 12B NF4 (local variant)	100.0%	91.2%–100.0%	100.0%
Decision 4B v1.2	100.0%	91.2%–100.0%	100.0%
Cygnet 12B NF4 (local variant)	100.0%	91.2%–100.0%	100.0%
Tev1 4B	97.5%	87.1%–99.6%	100.0%
Nimble 9B NF4	97.5%	87.1%–99.6%	100.0%
Intern-Decision 4B	97.5%	87.1%–99.6%	100.0%
Wald 4B v1.2	95.0%	83.5%–98.6%	100.0%
Plumb 4B	95.0%	83.5%–98.6%	100.0%
Imajev 4B	95.0%	83.5%–98.6%	100.0%
Decision 2.0 Nox 4B	95.0%	83.5%–98.6%	100.0%
Decider 4B	95.0%	83.5%–98.6%	100.0%
JevK5 4B	92.5%	80.1%–97.4%	100.0%
JevK5 2B	92.5%	80.1%–97.4%	100.0%
Imajev 2B	90.0%	76.9%–96.0%	100.0%
Kev 4B	87.5%	73.9%–94.5%	100.0%
Decision 2.0 Sol 2B	87.5%	73.9%–94.5%	100.0%
Decision 2.0 Kai 0.6B	87.5%	73.9%–94.5%	100.0%
Decider 2B	87.5%	73.9%–94.5%	100.0%
Decider 0.8B	87.5%	73.9%–94.5%	100.0%
Von	85.0%	70.9%–92.9%	100.0%
Nev 2B Q8	85.0%	70.9%–92.9%	100.0%
GLiNER Decide 340M	85.0%	70.9%–92.9%	100.0%
Decision 2.0 Eos 0.8B	85.0%	70.9%–92.9%	100.0%
Kev 0.8B	82.5%	68.1%–91.3%	100.0%
GLiNER Decide 1B	80.0%	65.2%–89.5%	100.0%
Laya	72.5%	57.2%–83.9%	100.0%
GLiNER Decide multilingual	67.5%	52.0%–79.9%	100.0%
Laya multilingual	50.0%	35.2%–64.8%	100.0%
Julia 144M	47.5%	32.9%–62.5%	100.0%
CLM 8B INT8	20.0%	10.5%–34.8%	100.0%

Industry routing · Norwegian

Completed suite	Accuracy	95% interval	Request coverage
Winnow 12B Q8 (Windows CUDA)	100.0%	91.2%–100.0%	100.0%
Nimble 9B NF4	97.5%	87.1%–99.6%	100.0%
Jev-Omni 12B NF4 (local variant)	97.5%	87.1%–99.6%	100.0%
Intern-Decision 4B	97.5%	87.1%–99.6%	100.0%
Decision 4B v1.2	95.0%	83.5%–98.6%	100.0%
Cygnet 12B NF4 (local variant)	95.0%	83.5%–98.6%	100.0%
Tev1 4B	92.5%	80.1%–97.4%	100.0%
Plumb 4B	92.5%	80.1%–97.4%	100.0%
Imajev 4B	92.5%	80.1%–97.4%	100.0%
Wald 4B v1.2	90.0%	76.9%–96.0%	100.0%
JevK5 4B	90.0%	76.9%–96.0%	100.0%
Kev 4B	87.5%	73.9%–94.5%	100.0%
Decision 2.0 Nox 4B	87.5%	73.9%–94.5%	100.0%
Nev 2B Q8	85.0%	70.9%–92.9%	100.0%
Decider 4B	82.5%	68.1%–91.3%	100.0%
JevK5 2B	80.0%	65.2%–89.5%	100.0%
Decision 2.0 Kai 0.6B	77.5%	62.5%–87.7%	100.0%
Imajev 2B	75.0%	59.8%–85.8%	100.0%
Decision 2.0 Sol 2B	75.0%	59.8%–85.8%	100.0%
Decider 2B	75.0%	59.8%–85.8%	100.0%
Decider 0.8B	75.0%	59.8%–85.8%	100.0%
Kev 0.8B	72.5%	57.2%–83.9%	100.0%
GLiNER Decide multilingual	67.5%	52.0%–79.9%	100.0%
GLiNER Decide 340M	65.0%	49.5%–77.9%	100.0%
Decision 2.0 Eos 0.8B	65.0%	49.5%–77.9%	100.0%
GLiNER Decide 1B	57.5%	42.2%–71.5%	100.0%
Laya	50.0%	35.2%–64.8%	100.0%
Laya multilingual	45.0%	30.7%–60.2%	100.0%
Von	37.5%	24.2%–53.0%	100.0%
CLM 8B INT8	32.5%	20.1%–48.0%	100.0%
Julia 144M	27.5%	16.1%–42.8%	100.0%

Document classification · English

Completed suite	Accuracy	95% interval	Request coverage
Winnow 12B Q8 (Windows CUDA)	100.0%	87.9%–100.0%	100.0%
Wald 4B v1.2	100.0%	87.9%–100.0%	100.0%
Nimble 9B NF4	100.0%	87.9%–100.0%	100.0%
Nev 2B Q8	100.0%	87.9%–100.0%	100.0%
Kev 4B	100.0%	87.9%–100.0%	100.0%
Jev-Omni 12B NF4 (local variant)	100.0%	87.9%–100.0%	100.0%
Intern-Decision 4B	100.0%	87.9%–100.0%	100.0%
Imajev 4B	100.0%	87.9%–100.0%	100.0%
Imajev 2B	100.0%	87.9%–100.0%	100.0%
Decision 2.0 Sol 2B	100.0%	87.9%–100.0%	100.0%
Decision 2.0 Nox 4B	100.0%	87.9%–100.0%	100.0%
Decision 2.0 Kai 0.6B	100.0%	87.9%–100.0%	100.0%
Decision 2.0 Eos 0.8B	100.0%	87.9%–100.0%	100.0%
Decider 4B	100.0%	87.9%–100.0%	100.0%
Decider 2B	100.0%	87.9%–100.0%	100.0%
Tev1 4B	96.4%	82.3%–99.4%	100.0%
Plumb 4B	96.4%	82.3%–99.4%	100.0%
Kev 0.8B	96.4%	82.3%–99.4%	100.0%
JevK5 4B	96.4%	82.3%–99.4%	100.0%
JevK5 2B	96.4%	82.3%–99.4%	100.0%
Decision 4B v1.2	96.4%	82.3%–99.4%	100.0%
Cygnet 12B NF4 (local variant)	96.4%	82.3%–99.4%	100.0%
Von	92.9%	77.4%–98.0%	100.0%
GLiNER Decide 340M	92.9%	77.4%–98.0%	100.0%
GLiNER Decide 1B	92.9%	77.4%–98.0%	100.0%
Decider 0.8B	92.9%	77.4%–98.0%	100.0%
Laya	89.3%	72.8%–96.3%	100.0%
Laya multilingual	82.1%	64.4%–92.1%	100.0%
GLiNER Decide multilingual	78.6%	60.5%–89.8%	100.0%
Julia 144M	64.3%	45.8%–79.3%	100.0%
CLM 8B INT8	53.6%	35.8%–70.5%	100.0%

Document classification · Norwegian

Completed suite	Accuracy	95% interval	Request coverage
Winnow 12B Q8 (Windows CUDA)	100.0%	87.9%–100.0%	100.0%
Wald 4B v1.2	100.0%	87.9%–100.0%	100.0%
Tev1 4B	100.0%	87.9%–100.0%	100.0%
Nimble 9B NF4	100.0%	87.9%–100.0%	100.0%
Kev 4B	100.0%	87.9%–100.0%	100.0%
Kev 0.8B	100.0%	87.9%–100.0%	100.0%
Intern-Decision 4B	100.0%	87.9%–100.0%	100.0%
Imajev 4B	100.0%	87.9%–100.0%	100.0%
Decision 2.0 Sol 2B	100.0%	87.9%–100.0%	100.0%
Decision 2.0 Nox 4B	100.0%	87.9%–100.0%	100.0%
Decider 4B	100.0%	87.9%–100.0%	100.0%
Cygnet 12B NF4 (local variant)	100.0%	87.9%–100.0%	100.0%
Nev 2B Q8	96.4%	82.3%–99.4%	100.0%
JevK5 4B	96.4%	82.3%–99.4%	100.0%
JevK5 2B	96.4%	82.3%–99.4%	100.0%
Imajev 2B	96.4%	82.3%–99.4%	100.0%
Decider 2B	96.4%	82.3%–99.4%	100.0%
Decider 0.8B	96.4%	82.3%–99.4%	100.0%
Plumb 4B	92.9%	77.4%–98.0%	100.0%
Jev-Omni 12B NF4 (local variant)	92.9%	77.4%–98.0%	100.0%
Decision 2.0 Kai 0.6B	92.9%	77.4%–98.0%	100.0%
Decision 2.0 Eos 0.8B	89.3%	72.8%–96.3%	100.0%
Decision 4B v1.2	85.7%	68.5%–94.3%	100.0%
Laya multilingual	78.6%	60.5%–89.8%	100.0%
GLiNER Decide 340M	78.6%	60.5%–89.8%	100.0%
Laya	75.0%	56.6%–87.3%	100.0%
GLiNER Decide multilingual	75.0%	56.6%–87.3%	100.0%
GLiNER Decide 1B	75.0%	56.6%–87.3%	100.0%
Julia 144M	50.0%	32.6%–67.4%	100.0%
Von	42.9%	26.5%–60.9%	100.0%
CLM 8B INT8	28.6%	15.3%–47.1%	100.0%

How to interpret this

MASSIVE measures 18 intent scenarios in English and Norwegian Bokmål, using 2,948 deduplicated test examples per language. The industry set contains assistant-authored scenarios: 40 routing and 28 document cases per language. These small synthetic samples need domain-expert review before a production decision. Reversing option order is a separate diagnostic, not additional independent evidence.

Accuracy includes rejected requests. Coverage reports the proportion receiving valid outputs. Confidence intervals are Wilson intervals for individual cases; they do not establish performance on new organizations or languages. Public publisher fixtures can have training or calibration exposure. Typed-decision reference labels use teacher agreement. Claim scoring uses the publisher's rules where available, including JevBench's distribution checks and tie-breaking.

Latency and peak memory need a separate controlled timing run. Current batch timings include different batching implementations; CLM can reuse an embedding cache, and Nev uses an external Vulkan process. These figures must not be compared with advertised H100 server latency.

Business value to validate

Helpdesk: suggest a queue and retain the evidence for an agent to accept or correct. Oil and gas / engineering: suggest a document type or responsible discipline for incoming records. Manufacturing: organize maintenance descriptions and route exception reports. The benchmark does not validate safety decisions or equipment control.

Measure review time saved, correction rate, routing delays, and local deployment cost in a pilot. Net annual value = cases × adoption rate × net seconds saved ÷ 3,600 × loaded hourly cost − annual operating cost. Seconds saved must include correction and review time; benchmark accuracy alone cannot supply this number.