| decider-4bJevBench public easy | 100.00% published
100.00% measured | matches published rounding | Same verified public cases and native scorer; bf16 Windows/RTX runtime, graphs disabled. The 2pp band is an engineering check, not a significance test.
Primary source |
|---|
| decider-4bJevBench public original | 98.60% published
98.61% measured | matches published rounding | Same verified public cases and native scorer; bf16 Windows/RTX runtime, graphs disabled. The 2pp band is an engineering check, not a significance test.
Primary source |
|---|
| decider-4bJevBench public hard | 64.90% published
64.86% measured | matches published rounding | Same verified public cases and native scorer; bf16 Windows/RTX runtime, graphs disabled. The 2pp band is an engineering check, not a significance test.
Primary source |
|---|
| decider-2bJevBench public easy | 100.00% published
100.00% measured | matches published rounding | Same verified public cases and native scorer; bf16 Windows/RTX runtime, graphs disabled. The 2pp band is an engineering check, not a significance test.
Primary source |
|---|
| decider-2bJevBench public original | 88.90% published
88.89% measured | matches published rounding | Same verified public cases and native scorer; bf16 Windows/RTX runtime, graphs disabled. The 2pp band is an engineering check, not a significance test.
Primary source |
|---|
| decider-2bJevBench public hard | 57.70% published
56.76% measured | near reproduction within 2pp | Same verified public cases and native scorer; bf16 Windows/RTX runtime, graphs disabled. The 2pp band is an engineering check, not a significance test.
Primary source |
|---|
| layaPublisher news.replication | 95.00% published
95.25% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| layaPublisher emotion.replication | 59.50% published
59.75% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| layaPublisher spam.replication | 99.25% published
99.75% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| layaPublisher phishing.replication | 98.00% published
97.75% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| layaPublisher jailbreaking | 70.75% published
71.25% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| layaPublisher toxicity | 53.00% published
53.00% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| layaPublisher intent | 78.33% published
78.00% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| layaPublisher scenario | 60.33% published
59.67% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| laya-multilingualPublisher news.replication | 93.00% published
93.75% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| laya-multilingualPublisher emotion.replication | 53.00% published
53.75% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| laya-multilingualPublisher spam.replication | 99.25% published
99.75% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| laya-multilingualPublisher phishing.replication | 99.25% published
99.25% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| laya-multilingualPublisher jailbreaking | 75.50% published
80.50% measured | outside 2pp repeatability band | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| laya-multilingualPublisher toxicity | 52.50% published
53.50% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| laya-multilingualPublisher intent | 65.67% published
65.33% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| laya-multilingualPublisher scenario | 56.00% published
56.00% measured | near reproduction within 2pp | Reconstructed publisher questions and sampling. Publisher did not pin dataset revisions; our source revisions are fixed. SDK and device differ. Task training exposure is explicitly reported.
Primary source |
|---|
| gliner-decideFast Decisions advertised 5100-example held-out average | — | not independently verifiable with public fixture | The accessible fixture has 1700 development examples. Its results cannot verify or refute a claim on 5100 different held-out examples.
Primary source |
|---|
| gliner-decide-1bFast Decisions advertised 5100-example held-out average | — | not independently verifiable with public fixture | The accessible fixture has 1700 development examples. Its results cannot verify or refute a claim on 5100 different held-out examples.
Primary source |
|---|
| gliner-decide-multiFast Decisions advertised 5100-example held-out average | — | not independently verifiable with public fixture | The accessible fixture has 1700 development examples. Its results cannot verify or refute a claim on 5100 different held-out examples.
Primary source |
|---|
| kev-08bdocuments-v1 locked test / question accuracy | 85.10% published
85.04% measured | near reproduction within 2pp | Same pinned public test and question metric; bf16 local serving instead of publisher fp32 claim evaluation. Devtools uses the publisher amendment: remove both rows sharing the duplicated CodeReviewer source ID (1071 questions). The common cross-model benchmark retains all 1073 questions. Hard-v1 holds out templates of trained skills; devtools labels are public proxies; documents use the same CFPB source and question templates as training.
Primary source |
|---|
| kev-08bhard-v1 locked test / question accuracy | 66.50% published
66.36% measured | near reproduction within 2pp | Same pinned public test and question metric; bf16 local serving instead of publisher fp32 claim evaluation. Devtools uses the publisher amendment: remove both rows sharing the duplicated CodeReviewer source ID (1071 questions). The common cross-model benchmark retains all 1073 questions. Hard-v1 holds out templates of trained skills; devtools labels are public proxies; documents use the same CFPB source and question templates as training.
Primary source |
|---|
| kev-08bdevtools-v1 locked test / question accuracy | 63.70% published
63.77% measured | near reproduction within 2pp | Same pinned public test and question metric; bf16 local serving instead of publisher fp32 claim evaluation. Devtools uses the publisher amendment: remove both rows sharing the duplicated CodeReviewer source ID (1071 questions). The common cross-model benchmark retains all 1073 questions. Hard-v1 holds out templates of trained skills; devtools labels are public proxies; documents use the same CFPB source and question templates as training.
Primary source |
|---|
| kev-4bhard-v1 locked test / question accuracy | 80.30% published
80.51% measured | near reproduction within 2pp | Same pinned public test and question metric; bf16 local serving instead of publisher fp32 claim evaluation. Devtools uses the publisher amendment: remove both rows sharing the duplicated CodeReviewer source ID (1071 questions). The common cross-model benchmark retains all 1073 questions. Hard-v1 holds out templates of trained skills; devtools labels are public proxies; documents use the same CFPB source and question templates as training.
Primary source |
|---|
| kev-4bdevtools-v1 locked test / question accuracy | 75.60% published
75.54% measured | near reproduction within 2pp | Same pinned public test and question metric; bf16 local serving instead of publisher fp32 claim evaluation. Devtools uses the publisher amendment: remove both rows sharing the duplicated CodeReviewer source ID (1071 questions). The common cross-model benchmark retains all 1073 questions. Hard-v1 holds out templates of trained skills; devtools labels are public proxies; documents use the same CFPB source and question templates as training.
Primary source |
|---|
| kev-4bPrevious round-8 documents-v1 locked test 90.4% | — | different checkpoint version | The tested checkpoint is the later round-10 skills release. Its model card also documents older results. A round-10 score cannot verify or refute the older round-8 90.4% claim.
Primary source |
|---|
| vonJabr v2 macro accuracy 72.0% on 869 cases | — | fixture count mismatch | Our pinned complete TOML contains 866 cases. Report it as a transfer result, not an exact replication. Von reports fitting its confidence map on the public JevBench cases and emotion in training.
Primary source |
|---|
| juliaNative typed-decisions 2000 heads | 73.15% published
72.55% measured | Near reproduction within the predeclared 2 percentage point band; not exact numerical equality | Unmodified publisher reproduce_typed.py; verified model and Parquet hashes; CPU fp32 native Torch backend, batch one, 1024-token context
Primary source |
|---|
| clm-int8CLM DeepSWE best-of-N 31/38 (81.6%) | — | exact downstream reproduction from published embeddings | Earlier local run used the authors' evaluator and public precomputed embeddings. This confirms the downstream evaluation, not full encoder/runtime equivalence. INT8 local classification is a separate test.
|
|---|