How we measure Survey Coding quality

Last measured: 16 November 2026 · model s1-fast-2 (Decision Models API, EU tier) · Beta

Survey Coding asks the model one yes/no question per answer and code ("Does the answer mention or express this theme?") and gets back a probability. You decide what happens with uncertain answers: they are flagged for your review.

What we measured

We used a held-out test set the model and our settings never saw during tuning:

Data set Answers Codes Micro-F1 of model suggestions (after calibration on 40 answers) Answers flagged for review Micro-F1 after you review the flagged answers
Synthetic: NPS reasons (SaaS), EN 350 12 0.95 22 % 0.98
Synthetic: NPS reasons (online shop), DE 350 12 0.98 7 % 0.99
Synthetic: employee survey, EN 350 12 0.95 26 % 0.99
Synthetic: employee survey, DE 350 12 0.97 13 % 0.99
MAMS restaurant reviews (human-labelled), EN 400 8 0.67 86 % 0.97

Micro-F1 combines precision (suggested codes that are right) and recall (right codes that were suggested) over all code decisions; 1.00 would mean every decision matches the reference labels.

Calibration on 40 answers is the step the app offers for projects with 150+ answers: you review 40 random answers, the app fits its thresholds to your decisions, then codes the rest. In the measurement, the reference labels played the part of the reviewer; we report the mean over 20 random samples of 40.

How the data sets were made — please read before you compare

What this means for you

Your data can differ. Use the free tier (up to 500 answers) and the calibration step to see how the model does on your own codebook before you rely on it. All model suggestions are AI-generated; you review and own the final codes.

Measured quality