About ULTRA · Research side
Vocabulary bridge
The research side uses the property names of the KU Leuven evaluation framework verbatim, so studies stay comparable. The practice side uses plain words. This table is the contract between the two.
Components of the framework used in the table: Situational characteristics · Subjective system aspects · User experience · Interaction · Objective system aspects. A linked plain name has a ten-minute metric card.
| Property (research) | Component | Plain name (practice) | How the ULTRA kit measures it |
|---|---|---|---|
| Case difficulty | Situational characteristics | How hard the task was | No ten-minute kit yet |
| Cognitive load | Subjective system aspects | Mental effort | Two items adapted from the NASA Task Load Index (NASA-TLX), mental demand and frustration, on a 1 to 7 scale after the task, plus time on task from the logs. |
| Confidence | User experience | Users are sure of their decision | One item after each decision, then compare the average certainty on the decisions people got right with the average on the ones they got wrong. |
| Controllability | User experience | Users feel in control | Three questionnaire items after the task, plus two log counts: how often people override or edit an AI proposal, and how often they undo. |
| Efficiency | Interaction | Faster | Time on task and error count per task, with and without the AI feature, on the same tasks. Add re-check time: how long people spend verifying AI output. |
| Information correctness | Subjective system aspects | Catching made-up facts | Seed the test material with two known errors, such as a wrong date and an invented name, and count how many participants catch each one. Add one item on perceived accuracy. |
| Intention to use | User experience | They want to keep using it | Three items after the task. For a product already in use, add the share of users who turn the feature on in week four. |
| AI model certainty | Objective system aspects | How sure the model really is | No ten-minute kit yet |
| Performance | Interaction | Fewer errors | Time on task and error count per task, with and without the AI feature, on the same tasks. Add re-check time: how long people spend verifying AI output. |
| Reliance | Interaction | How often users follow the AI | From the logs: acceptance rate of AI proposals, and acceptance rate of proposals you know to be wrong, using seeded errors or a labelled sample. |
| Satisfaction | User experience | They like it | Three items after the task. For a product already in use, add the share of users who turn the feature on in week four. |
| Trust | User experience | Users rely on it the right amount | Three questionnaire items after the task, plus two numbers from your logs: how often a suggestion was accepted, and how often a wrong suggestion was accepted. |
| Understanding explanation | User experience | Users got the explanation | No ten-minute kit yet |
| Understanding model behaviour | User experience | Users get how it decides | Two short comprehension questions per task, such as 'Why was this e-mail marked urgent?' scored against the actual reason, plus one confidence rating on the answer. |
IUI 2026 is Van den Bossche and Abdul Khalil (2026), listed under Research on the About page. Its four design goals map onto the framework properties as follows, to be confirmed by KU Leuven Augment:
- The right amount of trust: Trust, Reliance and AI model certainty.
- The user stays in charge: Controllability.
- The system shows how it works: Understanding model behaviour and Understanding explanation.
- The system adapts to the person: Alignment with situational context, and Satisfaction across user groups.