How to design usability studies for AI agent features with non-deterministic outputs

Traditional usability testing relies on a quiet assumption: every participant sees the same interface and gets the same results. If five people struggle with the same screen, you've found a problem.

AI agent features break that assumption. When a participant asks an agent to summarize a document, draft a reply, or complete a multi-step task, the output can differ every time—even with an identical prompt. One participant gets a sharp, accurate result. The next gets something vague. A third gets a confident answer that's wrong.

This makes it harder to know what you're actually observing. Is the participant confused because the design is unclear, or because the agent did something unexpected? Did the task succeed because the experience works, or because that participant got lucky with the output?

This guide covers how to plan, run, and analyze usability studies for AI agent features when outputs are non-deterministic. It's written for UX researchers, product managers, and designers who need findings they can trust.

Why standard usability testing falls short for AI agents

In a typical usability test, the interface is the constant and the participant is the variable. You hold the design steady and watch how different people move through it.

With AI agents, you now have two variables at once: participant behavior and system behavior. That creates a few specific problems.

Task success becomes ambiguous. A participant might follow the "right" path and still fail because the agent produced a poor result. Another might do something unexpected and succeed because the agent recovered well.

Findings are harder to compare. If one participant saw a three-step plan and another saw a seven-step plan, their reactions aren't directly comparable.

Problems can hide. If most sessions happen to produce good outputs, you may never see how the design holds up when the agent gets it wrong—which is often where the most important usability issues live.

Prompting skill varies. With open-ended input, participants who phrase requests well get better results. You can end up measuring participant skill rather than design quality.

None of this means usability testing doesn't work for AI features. It means you need to design studies that account for output variance instead of ignoring it.

Shift what you're testing

The first step is reframing your research questions. With deterministic features, you often ask, "Can users complete this task?" With agent features, the more useful questions tend to be:

• Can users tell whether the output is good?
• Do they know what the agent did and why?
• Can they correct, redirect, or undo the agent's work easily?
• Does their trust match the agent's actual reliability?
• What do they do when the result isn't what they expected?

These questions focus on the interaction around the output, not just the output itself. That's where design has the most influence. You can't fully control what the model produces, but you can control how results are presented, explained, verified, and corrected.

Write your research questions with this in mind before you plan anything else. They'll shape how much variance you want to control, which tasks you choose, and what you measure.

Decide how much output variance to control

One of the most important study design decisions is how much you let the real model drive the session. There's a spectrum of options.

Live model outputs

Participants interact with the actual agent, and outputs vary naturally.

Use this when: You want to understand real-world performance, discover failure modes you haven't anticipated, or validate a feature close to launch.

Trade-off: You get realism but lose comparability. You'll need a larger sample and careful logging to make sense of results.

Curated or seeded outputs

You constrain the agent so outputs are more consistent—for example, by fixing inputs, using test data sets, lowering randomness settings where available, or pre-generating a bank of outputs and serving them based on the task.

Use this when: You want to compare design variants or test a specific interaction pattern without output quality muddying the results.

Trade-off: Participants may notice the agent feels less responsive to their input, especially if they go off-script.

Wizard of Oz

A researcher or teammate behind the scenes writes or selects the agent's responses in real time.

Use this when: The feature isn't built yet, or you want to deliberately stage specific situations—a refusal, an ambiguous result, a confident error, a partial completion.

Trade-off: Responses are slower and may feel less natural. It works best for testing concepts and interaction flows rather than measuring performance.

Planned output conditions

A useful middle ground is to define a small set of output conditions in advance and assign participants to them deliberately. For example:

• Good output: Accurate, complete, matches the request
• Partially correct output: Mostly right with one meaningful error
• Poor output: Off-target or clearly wrong
• Uncertain output: The agent asks a clarifying question or signals low confidence

By making sure each condition appears across multiple participants, you can compare how people respond to each situation. This is often the most practical way to test error handling and trust, which are hard to observe if you wait for failures to happen naturally.

Many teams combine approaches across a research program: controlled or Wizard of Oz studies early to shape the interaction design, then live-model studies later to check how the real system performs.

Design tasks around goals, not paths

Because agents can take different routes to the same result, scripted step-by-step tasks don't fit well. Write tasks around outcomes participants care about.

Instead of: "Click the agent button and ask it to summarize the Q3 report."

Try: "You need to brief your manager on the Q3 report in five minutes. Use whatever you'd like in the product to get ready."

Goal-based tasks let you see whether participants choose to use the agent at all, how they frame their request, and how they judge whether the result is good enough.

A few other task design tips:

Use realistic content. Agents behave very differently on real data versus toy examples. If possible, let participants use their own documents or scenarios, or create test materials that closely mirror real ones.

Include tasks where verification matters. Build in at least one task where the participant needs to check the agent's work—for example, where a factual error would have consequences. This shows whether people actually verify outputs or accept them as-is.

Include multi-turn tasks. Many agent problems only appear after the first response, when participants try to refine, correct, or extend what the agent did.

Standardize the starting point. Even if outputs vary, make sure every participant begins with the same context, data, and instructions. This reduces noise you can control.

Plan your sample around output variance

Because output quality is an extra variable, you'll likely need more sessions than a standard usability test. Five participants might surface the main issues in a deterministic flow, but with an agent, those five may all happen to get good outputs.

Some practical guidance:

• For exploratory qualitative studies, plan for roughly 8–12 participants per key user segment.
• If you're using planned output conditions, make sure each condition is seen by at least three or four participants.
• Recruit for a range of experience with AI tools. People who use AI tools daily often prompt, verify, and trust differently than people who rarely do.
• Consider supplementing moderated sessions with unmoderated tests or product analytics to see patterns across a larger number of interactions.

Run sessions that capture both sides of the interaction

Log every output

This is the single most important practice. For every session, save exactly what the agent produced: the participant's input, the agent's response, any intermediate steps or tool calls it showed, and timestamps. Without this, you can't tell whether a participant's reaction was to the design or to a specific output.

If your team can export agent logs or traces, link them to the session recording. If not, screen recordings plus a note-taker capturing outputs verbatim will work.

Ask about expectations before the output appears

Before the agent responds, ask participants what they expect it to do. "What do you think will happen when you send that?" or "What would a good answer look like here?"

This gives you a baseline. When the output arrives, you can compare it against what the participant wanted, which helps explain reactions like surprise, disappointment, or over-trust.

Use think-aloud, but probe evaluation

Standard think-aloud works, but add prompts that focus on how participants judge the output:

• "How would you know if this is correct?"
• "Would you use this as-is? What would you change?"
• "How confident are you in this result?"

Pay attention to the gap between stated confidence and actual output quality. A participant who says "this looks right" about a flawed output is showing you a trust calibration problem.

Test recovery deliberately

Don't wait for errors to happen. If you're using live outputs and the agent performs well, introduce a follow-up task that makes correction necessary—for example, "Your manager says this summary is missing the budget numbers. Fix it."

Watch how participants try to recover. Do they re-prompt? Edit manually? Start over? Abandon the agent? Recovery flows are often where agent features succeed or fail in real use.

Debrief on trust and control

At the end of each session, ask broader questions about the experience:

• When did you feel in control? When didn't you?
• Was there anything the agent did that you didn't understand?
• Would you let it do this task without checking its work?

Choose metrics that account for variance

Traditional metrics like task completion and time on task still matter, but they need context. Consider adding:

Output quality rating. Have a researcher or subject-matter expert rate each output independently using a simple rubric (for example, 1–4 on accuracy and completeness). This lets you analyze participant behavior in light of what they actually saw.

Outcome success. Did the participant end up with a usable result, regardless of how they got there?

Correction effort. How many turns, edits, or retries did it take to reach a usable result?

Verification behavior. Did the participant check the output? How?

Trust calibration. Did the participant's confidence match the output's actual quality? Over-trust of poor outputs and under-trust of good outputs are both problems.

Intervention rate. For agents that take actions, how often did participants stop, override, or undo what the agent did?

Report these metrics broken down by output quality, not just as overall averages. "80% task success" means very different things if most participants got good outputs versus a mix.

Analyze findings by output condition

Analysis is where non-deterministic studies need the most care.

Tag output quality alongside behavior

When you code your session data, treat output quality as its own tag or attribute. Then tag participant behaviors, quotes, and pain points as usual. This lets you filter findings by condition—for example, "all moments of confusion when the output was partially correct."

A research repository like Dovetail is useful here because you can keep session recordings, transcripts, and output logs together, tag them consistently, and filter highlights across sessions to see how reactions differ depending on what the agent produced.

Separate model issues from design issues

As you review findings, sort them into rough groups:

• Design issues: Participants struggled even when the output was good. Examples include unclear presentation, hidden controls, or confusing explanations.
• Error-handling issues: Participants struggled when the output was poor, and the design didn't help them notice or recover.
• Trust issues: Participants accepted bad outputs or rejected good ones.
• Model quality issues: The agent consistently produced poor results for a certain kind of request.

This separation makes your findings more actionable. Design teams can act on the first three directly. Model quality issues should go to the people responsible for prompts, evaluation, or model selection—with session evidence attached.

Look for patterns across variance, not just within it

Some findings hold no matter what the agent produces. If every participant scrolled past the agent's explanation without reading it, that's a reliable finding regardless of output quality. Prioritize issues that appear across conditions—they're the most robust.

Report findings honestly

When you share results, be explicit about variance. Stakeholders may assume every participant had the same experience unless you tell them otherwise.

Good reporting practices include:

• Stating how outputs were generated (live, curated, Wizard of Oz) and why
• Showing the distribution of output quality across sessions
• Pairing key quotes or clips with the output the participant was reacting to
• Flagging which findings depend on output quality and which don't
• Being clear about what you couldn't observe—for example, if no participant encountered a particular failure mode

This kind of transparency builds credibility and helps teams make better decisions about what to fix first.

Common mistakes to avoid

Treating each session as equivalent. Without logging outputs, you can't interpret reactions accurately.

Only testing the happy path. If all your sessions produce good outputs, you've learned little about how the feature performs in real use.

Over-scripting tasks. Rigid instructions hide how people would naturally use, or avoid, the agent.

Measuring prompting skill instead of design. Recruit a range of participants and watch for cases where results depend more on how someone phrased a request than on the interface.

Reporting averages without context. Overall success rates can hide the fact that the experience works well only when the agent performs well.

Final thoughts

Non-deterministic outputs make usability testing harder, but not impossible. The key is to stop treating output variance as noise and start treating it as part of what you're studying. Decide how much variance to control, log every output, design tasks around real goals, and analyze findings in light of what each participant actually saw.

Done well, these studies tell you more than whether the agent works. They show you whether people can understand it, judge it, correct it, and trust it the right amount—which is what determines whether an AI feature is actually useful.

You usually need more participants than a traditional usability test, because output quality is now an extra variable alongside user behavior. For exploratory, qualitative studies, 8–12 participants per key segment is a reasonable starting point. Aim for enough sessions to see several examples of good, mediocre, and poor agent outputs. If you're comparing design variants or measuring task success rates, plan for larger samples or supplement moderated sessions with unmoderated tests and log analysis so you can separate design effects from output variance.

It depends on what you're trying to learn. Scripted or Wizard of Oz outputs give you control, which is useful when you want to test how people react to specific situations like a confident wrong answer, a partial result, or a refusal. Live model outputs show you how the real system performs and surface failure modes you didn't anticipate. Many teams use both: controlled outputs early to test interaction patterns and recovery flows, then live outputs later to validate the experience under realistic conditions.

Record the actual agent output for every session and rate its quality independently of how the participant responded. If participants struggle even when the output was good, the problem is likely in the interface—how results are presented, explained, or acted on. If participants only struggle when the output was poor, look at how the design handles errors, uncertainty, and correction. Patterns where good outputs are distrusted or bad outputs are accepted without question usually point to trust calibration issues that both design and model teams need to address.