Accuracy before automation
Choose a task to compare both checkpoints with a simple trained classifier. Different tasks stay separate; there is no misleading overall accuracy score.
Correct decisions on held-out examples
Human-labelled public test data; the trained baseline has access to the official training split. Laya is zero-shot.What the comparison says
Quality and local performance
| Checkpoint | Accuracy / 95% interval | Macro F1 | Warm p50 / p95 | Batch throughput |
|---|
Latency is measured separately with one request at a time. Throughput uses batches. Downloads, cold load, application and network overhead are excluded.
Turn quality into a business decision
Potential applications include suggesting a support queue, tagging feedback, and prioritizing review. Local inference gives you control over where message content is processed.
Editable cost scenario · NOK / month
Illustrative net monthly benefit
A practical first release
- Suggest a category inside the existing workflow and let staff correct it.
- Measure handling time, overrides and downstream mistakes on a private holdout.
- Validate separate quality targets for Bokmål, Nynorsk and high-cost categories.
- Automate only the cases that meet those targets; keep the rest in human review.
Budget for monitoring, maintenance and fallback handling. Keep irreversible actions behind application permissions and appropriate review.
Confidence has to be earned
A confident answer can still be wrong. Calibration and threshold selection use different validation groups; the final policy is measured on untouched test examples.
| Model | Raw ECE → fitted ECE | Errors at ≥90% raw confidence | Selected coverage | Selected accuracy / 95% interval |
|---|
ECE measures the gap between confidence and correctness (lower is better). The policy needs ≥30 validation decisions and a 95% lower accuracy bound ≥95%. A policy qualifying on validation may still fail on test.
Norwegian is more than one test
Bokmål: matched English–Norwegian request routing provides a direct language comparison.
Norwegian reviews: human-labelled negative, neutral and positive sentences. Variety is not tagged per row in this dataset.
Nynorsk: authored support and negation diagnostics only. This is insufficient to certify Nynorsk production accuracy.
Sensitivity to option order
The first 100 examples in deterministic label-blind order are rerun with reversed choices. This reveals fragility without replacing the original score.Authored diagnostic results: support, negation, scoring and long inputs
| Model | Diagnostic | Cases | Accuracy | Macro F1 | Truncated |
|---|
Per-category precision, recall, F1 and confusion matrices
Evidence you can inspect and reproduce
Evaluation protocol
- Pinned model and dataset revisions; local Python virtual environment; no hosted inference or LLM judge.
- Official train/validation/test splits. Exact normalized text duplicates removed with test priority. NoReC review groups checked for split leakage.
- MASSIVE English and Bokmål use the same example IDs and all 18 service categories. This is not a benchmark of all 60 intents.
- Temperature fitted on one half of validation source groups; policy selected on the other half. Test outcomes do not tune prompts, temperatures or thresholds.
- 95% accuracy intervals use 2,000 source-group bootstrap resamples. Multiple NoReC sentences from one review remain grouped.
- All public test errors remain in the results. Failed runs are explicitly marked incomplete.
Local environment
Evidence files
Business findings (Markdown)
Full metrics & run manifest (JSON)
Every prediction (JSONL) (local-only evidence)
Error review (CSV) (local-only evidence)
Measured truncation
The SDK's native token limits are used in the primary evaluation. Long-document diagnostics additionally test a 2,048-token multilingual setting. The SDK rounds probabilities to four decimals; calibration renormalizes them and clips log-loss zeros at 1e-8.
Sources
Laya model card · Multilingual model card · MASSIVE · NoReC sentence