Metric · Trust
Users rely on it the right amount
What it tells you
Whether people follow the AI when it is right and check it when it is wrong. Too much trust and errors slip through; too little and the efficiency gain evaporates.
How, in 10 minutes
Three questionnaire items after the task, plus two numbers from your logs: how often a suggestion was accepted, and how often a wrong suggestion was accepted.
Benchmark
No benchmark yet. Run the kit and send us yours.
Plan your test explains how to choose metrics and how many participants you need.
Questionnaire items
- I can rely on this system to do what it should.
- I would follow this system's suggestion without checking, for routine cases.
- When the system is unsure, I notice it.
Answer on a 1 to 7 scale from strongly disagree to strongly agree unless the item states its own scale. The ULTRA team wrote or adapted these items for the kits and has not checked their reliability, so report them as ULTRA kit items and name any questionnaire they were adapted from.
Log events
suggestion_shown {pattern, field, confidence}suggestion_accepted {pattern, field, was_correct}suggestion_overridden {pattern, field}Pairs with
Research side · the property behind this metric
Property
References
- Tintarev and Masthoff, definition of trust as perceived confidence in a system's competence.
- Donoso-Guzmán et al. 2025, Table 9 and B.11.