How to Judge AI Follow-Up Questions in Customer Research

A longer interview transcript gives you more material to read. Whether it gives you a better reason to change your product, pricing or checkout is a separate question. I can imagine feeling reassured by pages of detailed answers, then struggling to point to the passage that actually changed my mind.

I think AI follow-up questions should be judged by the new, decision-relevant evidence they uncover, with repetition and leading questions counted against them.

The September 14 methodological commentary on Verasight’s AI-moderation experiment describes a randomized study involving 3,160 participants, with an AI-moderated interview condition compared with a written survey using the same four questions. In the commentator’s account, the AI condition produced roughly five times as many words and twice as much usable evidence per participant, with about 80% of the lift attributed to adaptive follow-up probing. Those are the commentary’s reported figures, not findings I have independently checked against the original paper. They suggest a useful possibility: asking a relevant follow-up may recover detail that a written answer leaves out. They also leave me wanting to inspect how “usable evidence” was defined and how the contribution of probing was isolated.

What a follow-up needs to earn

I agree with the case for adaptive probing. People often give an answer that makes sense to them because they already know the story behind it. The researcher doesn’t.

Take a hypothetical checkout study. Someone says delivery was inconvenient. That could mean the price, the arrival date, the need to be home or a previous delivery problem. A follow-up that asks them to describe what happened on their last attempted order could separate those explanations.

That is the kind of gain I would want to see in the transcript. A concrete event can tell you which part of the experience deserves another look. A longer explanation of how much the person values convenience may leave you exactly where you started.

ORB’s September 2026 article describes testing semi-structured AI interviews that ask programmed questions, respond to answers and probe for detail. That gives this debate a practical setting: teams are testing how much freedom to give an interviewer between the questions they planned. The supplied account doesn’t establish a deployment count or an outcome, so I wouldn’t use it as evidence that this approach has already succeeded at scale.

My disagreement is with treating answer volume as a verdict. Word count can show that the experience elicited more speech or text. It cannot, by itself, tell you whether any uncertainty was resolved.

What the comparison can tell you

The written-survey comparison is useful for a founder deciding whether follow-ups might improve an existing study. But the same starting questions do not mean participants had the same opportunity to explain themselves. One condition could ask again, and adapt. That is part of the experience being tested.

If the practical question is whether that whole experience produces more useful material than a written survey, the comparison fits. If the question is whether AI interviews outperform skilled human interviewers, this comparison cannot answer it. There was no skilled-human-interviewer condition in the comparison described in the brief.

I would also separate a finding about the full interview experience from a claim about what caused its gains. Before leaning on the reported attribution to adaptive probing, I’d want the original paper’s methods. Was probing varied independently? How was evidence coded? Could reviewers see which condition produced an answer?

Until those questions are answered, I would keep the numerical effects out of a buying decision. The more useful move is to inspect whether the follow-ups recover evidence your current study misses.

Our recommended review rubric

This is our recommendation for reviewing adaptive interviews, not a validated scoring system from the Verasight study. Read the initial answer, the follow-up and the resulting answer together. A polished final summary can hide how a claim entered the conversation.

New decision-relevant evidence

Mark a passage here when it adds something you did not have before and helps answer the study’s question. Look for a specific example or a sequence of events. A constraint can qualify too, as can a contradiction that makes the first answer less straightforward.

In the hypothetical checkout study, learning that the participant needed the order before leaving home on Friday gives “inconvenient delivery” a more useful meaning. It points toward investigating delivery timing and how clearly it is shown. It does not yet prove that changing it would recover the sale.

For each passage, write down what uncertainty it reduces. If you can’t name one, reconsider whether it belongs in this category.

Repetition

Use this label when the participant restates an existing point without adding a relevant detail. More adjectives, stronger emphasis and a second explanation of the same preference may all fall here.

Repetition can help you understand how someone expresses a concern. I still wouldn’t count each restatement as another piece of evidence for a product decision. Otherwise, the most persistent interviewer can appear to have found the most support.

Unsupported interpretation

Use this when the interviewer or analysis adds a meaning the participant has not established. Suppose someone describes an unclear delivery date, and the summary turns that into distrust of the brand. Distrust might be worth investigating. It remains an interpretation until there is evidence for it.

Keep that interpretation visibly separate from what the person actually said. I would rather carry an unresolved question into the next study than quietly turn my own explanation into a customer finding.

Leading probes

Flag follow-ups that introduce the answer they seem to want. In our hypothetical example, asking whether expensive shipping caused the abandonment supplies both a cause and a judgment. Asking what happened after the participant saw the delivery options leaves more room for their account.

A leading probe can produce a detailed answer. The detail doesn’t repair the question. Flag the resulting passage rather than treating it as clean evidence. Our guide to How to Write Survey Questions That Don't Lead People applies to follow-ups too.

These labels need not be mutually exclusive. An answer can contain a new detail and still have been elicited by a leading question.

A small test before wider use

Start with one decision your team is actually facing. For the checkout example, you might ask what prevented recent would-be buyers from completing an order. Recruit people who experienced that event, rather than people willing to speculate about it.

Then pilot the interview and review the exchanges using the rubric. Keep the original answers beside the follow-ups so you can see what was added. Where practical, have someone who didn’t design the questions review the passages without seeing the tool’s summary first. Ask them to explain any evidence label you disagree on.

If you compare versions, keep the starting questions and participant eligibility consistent. Record participant effort as well as evidence gained. A longer session needs to earn the extra attention it asks for, and a small pilot should guide revisions rather than support a sweeping superiority claim.

Before expanding the study, choose a follow-up you think worked well. Put its new evidence beside the decision it could inform. If that connection is hard to write down, revise the probe before asking more people to answer it.