Benchmarks · sources and method
Benchmark sources and method
One benchmark cannot describe every task. This comparison gives each source the same space and includes missing results. It shows the TypeSafe reference model, not Decision Models models.
| Reference model | Decision Index0.2.1Chance-corrected index / 100Independent community · run by apolinario (@multimodalart) on Hugging Face | Workflow Evalsdatasets 2026-09-28 · code 0ac3b8aModel-reference agreementTypeSafe · reference vendor | Jev Rerank Bench2026-09-25Dataset-macro nDCG@10 / 1Independent author | JevBench1.5.1Composite score / 100Same founder as Decision Models | ImageJevBench0.1.4Composite score / 100Same founder as Decision Models |
|---|---|---|---|---|---|
| TypeSafe referenceJev 1.13.0 | 57.91 | 67.8 | 0.670 | 72.13 | Not measured¹ |
¹ Not measured means no comparable published measurement was verified for this model; it is not a zero. The table shows published results for the TypeSafe reference model, not scores of Decision Models models and not a measurement of the Decision Models service.
Disclosure: JevBench and ImageJevBench are run by the founder of Decision Models. Workflow Evals is published by TypeSafe AI, the vendor of the reference model. It measures model-reference agreement, not human-labelled accuracy. No overall score is published while comparable coverage is incomplete. Method and sources →
Equal weight, visible gaps
All five benchmarks have equal-width columns and the same treatment. Metrics retain their own units: an index, reference agreement, nDCG and a composite score measure different things. Submetrics and model cards that repeat the same results are not counted as extra votes.
There is no overall score because a common set of comparable measured results is missing. Any future overall must use the unweighted arithmetic mean over a declared common benchmark set: mean(100 × score / scale). Missing values are never silently omitted or replaced with zero. Chance correction already applied by a source is retained.
What this page does not show
The table lists published results for the TypeSafe reference model only. It shows no scores for the Decision Models service or for the models behind s1-fast, s1-pro and s1-vision.
Sources and limitations
Decision Index · 0.2.1
38 index benchmarks across five areas, chance-corrected and coverage-adjusted. The value is the TypeSafe reference result on the board; it is not a run of the Decision Models service.
Primary source → · Original data → · Published: 2026-09-28. Checked: 2026-09-30.
Workflow Evals · datasets 2026-09-28 · code 0ac3b8a
Four public vendor workflow datasets, 705 cases; unweighted mean of the headline consensus metric reported in each manifest for Jev 1.13.0: invoice and customer service use exact action sets (accuracy), security incidents use exact action sets (agreement), and agent traces use primary-action disposition. Reference mode: all questions, reasoning off. Reference labels blend large-model answers, with some fallback contributors; agreement is not human-labelled accuracy.
Primary source → · WorkflowEvals code → · Public datasets → · Published: 2026-09-28. Checked: 2026-09-30.
Jev Rerank Bench · 2026-09-25
Eight English retrieval datasets; 1,617 scored questions. The value uses the yes/no-per-pair method. Public dataset training overlap cannot be excluded. nDCG is ranking utility, not classification accuracy. The reference also has a stronger multi-level, batched configuration, so the shown value is not the reference model’s ceiling.
Primary source → · Published: 2026-09-25. Checked: 2026-09-30.
JevBench · 1.5.1
Official released edition shown above. Its composite weights intelligence, calibration, speed and cost equally. This comparison uses only published aggregate results; no sealed item text was used to build it. The TypeSafe reference uses observed hosted API latency and list price. It is not a measurement of the Decision Models service.
Primary source → · Original data → · Published: 2026-09-29. Checked: 2026-09-30.
ImageJevBench · 0.1.4
Public edition shown above. TypeSafe Jev 1.13.0 has no comparable image row; other image-model results are not substituted. The held next release is not used.
Primary source → · Published: 2026-09-29. Checked: 2026-09-30.
Back to our models’ benchmark results → · Our own measurements and raw numbers →