Metric · Confidence
Users are sure of their decision
What it tells you
How certain people are about the decision they made with the AI. Set against whether each decision was right, it shows whether their certainty follows their accuracy or stays just as high when they are wrong.
How, in 2 minutes
One item after each decision, then compare the average certainty on the decisions people got right with the average on the ones they got wrong.
Benchmark
No benchmark yet. Run the kit and send us yours.
Plan your test explains how to choose metrics and how many participants you need.
Questionnaire items
- How sure are you that this decision is right? (1 to 5)
Answer on a 1 to 7 scale from strongly disagree to strongly agree unless the item states its own scale. The ULTRA team wrote or adapted these items for the kits and has not checked their reliability, so report them as ULTRA kit items and name any questionnaire they were adapted from.
Log events
decision_made {task, item, correct}decision_confidence {task, item, rating}Pairs with
Research side · the property behind this metric
Property
References
- Donoso-Guzmán et al. 2025, Table 9: confidence.
- Van den Bossche and Abdul Khalil 2026, IUI Companion: the alignment between expressed confidence and actual performance as an indicator of calibrated reliance.