Metric · Reliance
View
How often users follow the AI
What it tells you
How often people accept what the AI proposes, and, the part that matters, how often they accept it when it is wrong. Without the second number, an acceptance rate says nothing about whether the AI was right.
How, from your logs
From the logs: acceptance rate of AI proposals, and acceptance rate of proposals you know to be wrong, using seeded errors or a labelled sample.
Benchmark
- VAIA FormFlow pilot: non-top suggestion chosen: median 1 (pre-filled) vs 2 (on-demand) (n=6, course providers)
Plan your test explains how to choose metrics and how many participants you need.
Log events
proposal_shown {pattern, item, was_correct}proposal_accepted {pattern, item, was_correct}proposal_rejected {pattern, item, was_correct}Pairs with
Research side · the property behind this metric
Property
Reliance (Interaction). Whether the user is willing to provide control to the machine for the given task. Donoso-Guzmán et al. 2025, Table 9
References
- Donoso-Guzmán et al. 2025, Table 9 and section 4.12.7: reliance as the willingness to hand decision control to the system, asked directly; measures of how much a decision changes after seeing the AI are coded there as explanation power. The acceptance rates on this card are ULTRA's behavioural measure.
- Zhang, Liao and Bellamy 2020, trust calibration in AI-assisted decision making.
Editors
Practice: Thomas Van den Bossche · v0.1