About ULTRA · How we test
Plan your test
Five steps to decide what to measure, from Donoso-Guzmán et al. (2025), section 6.4, and what this site gives you at each step.
Select relevant properties
Start from what people do with the AI: decision support (a person decides, the AI advises), model improvement (someone analyses the model and improves it, during development or after deployment) or capability assessment (someone judges whether the model is good enough for a job). Pick the properties that matter there, not the ones that are easy to measure.
On this site. The compass does this step for you and names the rule behind each metric. Find my patterns
Check which measurements exist
Each property has questionnaire items or log events that others have used. If none fit your product, plan a qualitative study instead of inventing a scale.
On this site. Every metric card lists its items and events. The nine metric cards
Separate what needs users
Log-based measures run without a session. Questionnaire items need people in front of a screen. Knowing which is which decides your recruitment.
On this site. Each card says how many minutes it takes, or "from your logs". See the cards
Prioritise by the time users have
A twenty-item survey does not fit a five-minute session. Choose the few items that answer your question and let the rest go.
On this site. Two or three metric cards fit ten minutes per participant: three questionnaire items plus two counts from your logs is the usual kit. Example kit
Revise the plan
If only a few properties can be measured with closed questions, switch to a mixed-methods design: think-aloud, logs and interviews together.
On this site. The receipts on the cases show what that looks like in practice. A receipt
How many participants do you need?
The honest answer from the 2025 systematic review, and how it shapes the labels here.
Average in the review
34.5
participants per study on average, across the 82 studies in the review.
Too small for a hypothesis test
16 or fewer
With that few, run a thorough qualitative study instead and report per user group.
On this site
17 or more
A controlled comparison above that floor shows as Tested; below it, as Tested (pilot). The floor is a minimum, not a guarantee: the number you need depends on the effect and the design. How a pattern earns its label
Keep the property names as they are, so your result can be compared with the studies on the research side.
Your metric kit
Nine cards. Pick the two or three that fit ten minutes per participant; each shows its items, its log events and the time it takes.
Efficiency
Faster, fewer errorsWhether the AI feature actually saves time and reduces mistakes, compared with doing the task without it. Efficiency is the time; performance is the error rate. Report both.from your logsReliance
How often users follow the AIHow often people accept what the AI proposes, and, the part that matters, how often they accept it when it is wrong. Without the second number, an acceptance rate says nothing about whether the AI was right.from your logsConfidence
Users are sure of their decisionHow certain people are about the decision they made with the AI. Set against whether each decision was right, it shows whether their certainty follows their accuracy or stays just as high when they are wrong.2 minIntention to use
They want to keep using itWhether people would choose the feature again tomorrow.3 minCognitive load
Mental effortHow hard people had to think to use the AI feature. In the KU Leuven review, the size and structure of an explanation were found to influence it.5 minInformation correctness
Catching made-up factsWhether people notice when generated text contains a wrong or invented fact. The KU Leuven framework suggests considering this property for any system that uses generative AI.6 minControllability
Users feel in controlWhether people feel they can steer, correct and undo what the AI does. The KU Leuven framework names four forms of that control: reversing actions, correcting the system, filtering or zooming the explanation, and asking questions.8 minUnderstanding model behaviour
Users get how it decidesWhether people can predict and explain what the AI will do with an input. The KU Leuven framework defines it as the user's ability to correctly interpret the model behaviour.10 minTrust
Users rely on it the right amountWhether people follow the AI when it is right and check it when it is wrong. Too much trust and errors slip through; too little and the efficiency gain evaporates.10 minTest with us
Rather test it in the UX Lab?
Bring your product and your users. We build the two versions, run the session and write the receipt with you.