Evidence revision completion-2026-10-04 · Updated 2026-10-05
Presentation revision overview-2026-10-05 · 5 October 2026

Completing the same benchmark for every model

Each local model is evaluated against the same five frozen fixtures: 17,576 records. This pass has added 76,658 previously missing records; 0 remain. Completion retries recovered 847 recorded memory failures. 235 remain after at least three retained retry attempts each. Incorrect answers and native rejections are retained.

Jev 1.13.0 now has 17,576 records with zero API failures after two pinned-version retries. Original attempts are preserved. Existing successful local predictions and fixture hashes are verified against the pre-run baseline.

Task leaders across all local models · Compare every model by accuracy, latency and VRAM

Controlled timing coverage

340 of 348 timing groups are complete across 29 models. Missing groups are listed in the limitations and aggregate evidence. The graphs also withhold complete timing groups with fewer than 90% answered calls.

Coverage and failure accounting

Recorded means that a case has a validated saved result; it does not mean that the model answered correctly or answered at all. Native rejections include option limits and native context/schema restrictions.

Business interpretation

Compare models by the actual workflow: suggested ticket categories, engineering document labels, screening or developer-tool selection. These tests support choosing candidates for a reviewed pilot. They do not measure staff-time savings, production safety or financial return.

• All 17,576 fixture records include development examples; task accuracy excludes development using the original split rules.
• Recorded coverage, answered coverage, native rejection and infrastructure failure are separate.
• Native Windows adapters and pinned precision are retained; no Linux/vLLM or publisher hardware parity claim.
• Public training overlap is unknown. Industrial cases are small authored diagnostics, not domain-expert validation.
• Recovery-worker time is excluded from controlled latency plots. API latency includes network transport.
• JevK5 4B: 2 of 2 explicitly selected integration-recovery cases recorded through the publisher native serial question API. Existing successful predictions and the original parity tolerance are unchanged. The exact input hashes, original crashes and recovery configuration are retained locally.
• Plumb 4B: 1 of 1 explicitly selected integration-recovery cases recorded through the publisher native serial question API. Existing successful predictions and the original parity tolerance are unchanged. The exact input hashes, original crashes and recovery configuration are retained locally.
• JevK5 4B / kev_claim/devtools-v1: Developer-tool timing unavailable: three fresh workers failed native/batched probability parity during fixed warm-up (maximum difference 0.047281). Native-serial diagnostics changed 1 of 12 saved decisions, so alternative-path latency is withheld from comparison with the original accuracy score. No timing value is estimated; existing accuracy records are preserved.
• Winnow 12B Q8 retains 7 historical timing gaps outside the approved restoration queue. Its full accuracy results are available; no latency or VRAM values are estimated for the missing groups.

Aggregate completion evidence · Final verification · Accuracy, latency and VRAM graphs · Full matched API comparison