Survey Exclusion Rules: Fast Answers Aren’t Always Bad Answers

A participant finishes your survey much faster than you expected. You can see the completion time. You still have to decide what it tells you. In its September 18 case study, Prolific describes a three-week study of AI sycophancy involving 1,364 participants. The researcher used the platform’s auto-reject feature for sessions described as completed “inhumanly quickly”, alongside her own preregistered attention checks. The account says these checks excluded a very small share of participants in each study. The supplied excerpt gives no exact exclusion rate or estimate of how accurately the checks identified unusable responses.

I think this supports setting exclusion rules before launch, with speed treated as a reason to investigate rather than proof that a response is bad.

I agree with combining platform checks and study-specific rules. A platform can flag patterns across sessions; a researcher can define what a participant needs to understand or complete for a particular study. But I wouldn’t borrow an unexplained speed threshold from this example. There isn’t enough information here to know whether it would make sense for your survey.

What a completion time can tell you

Imagine you ask a repeat customer which product they last ordered and why they bought it again. They may know immediately. Someone considering your brand for the first time might need longer to read the same product names.

A short completion time could reflect familiarity, a shorter route through the survey, or skipped work. You need more than the clock to distinguish those possibilities.

The reverse deserves a look too. A long session might mean someone considered every question carefully. It might also mean they left the survey open while making lunch. Elapsed time measures elapsed time. Any judgment about attention needs further support.

I’m wary of my own instincts here. If a fast respondent gives an answer I dislike, it would be easy to inspect their timing more closely than the timing of someone who agrees with me. Writing the rule first makes that selective scrutiny harder.

The case study’s use of preregistered attention checks is useful in that respect. The checks were specified in advance. For a founder running a small customer survey, I’d recommend at least a dated exclusion plan saved before responses arrive. That’s an editorial recommendation, not a method validated by this case study, and an internal note isn’t equivalent to formal preregistration.

A survey exclusion-plan template

Keep the plan short enough that the person reviewing responses will actually use it. I’d create an entry for each signal you intend to check, using these five fields.

Signal

Write down exactly what will trigger a flag. Avoid labels such as “suspicious respondent”, which leave the reviewer to invent the rule later.

For speed, specify the timing measure, the survey route it applies to, and the review boundary you’ve chosen. Before launch, pilot the survey with people resembling your intended audience and inspect how the timing is recorded. Use that to inform a boundary you can explain. Don’t copy a number from an unrelated study.

If you cannot justify an automatic exclusion threshold, keep the signal as a review flag.

Rationale

Explain how the signal could make a response unusable for your research question.

For example, a study comparing packaging depends on participants seeing the packaging. A completion time that appears too short to view the material raises a specific concern. A quick answer to a familiar purchase-history question raises a different, weaker concern.

This field should connect the rule to the task. “Faster than I expected” needs more work.

Review step

State what the reviewer will inspect and what decisions they can make. For a speed flag, that might include the participant’s survey route, whether the required material loaded, any relevant attention check, and whether their answers address the questions asked.

Define whether you’ll retain the response, exclude only an affected answer, or exclude the whole submission. A missing explanation does not automatically make a completed product-choice question unusable.

Apply the same review to answers you like and answers you don’t. Where practical, assess quality flags before looking at which option the participant preferred.

Participant-payment rules

Record the current rules of the recruitment platform or agreement you’re using. Include the permitted grounds for rejection, any required review or contact steps, and how uncertain cases will be handled.

Excluding a response from your analysis and refusing payment are separate decisions. A submission may be unsuitable for a particular analysis while the participant remains entitled to payment. Don’t assume a speed flag permits rejection, and don’t introduce payment conditions after someone has completed the work.

For research with your own customers, write down any incentive commitments too. The absence of a panel platform doesn’t remove those commitments.

Reporting

Decide how you’ll report the number of submissions received, the number flagged, and the number ultimately excluded, with reasons. Keep overlapping flags visible so one response isn’t accidentally counted as several exclusions.

Record any rule changes made after launch and why you made them. A genuine technical fault may require a new decision. Label that decision honestly rather than presenting it as something you planned from the start.

Human responses can still be unusable

We covered the identity question in What Prolific’s “100% Human” Research Guarantee Covers. This question starts after that one. Even when a participant is human, you still need to assess whether their response can support your analysis.

A real customer can misunderstand a question. An attentive participant can encounter a broken image. Someone can give a brief, useful answer without explaining every part of their thinking.

I’d avoid calling these cases fraud unless there is evidence of deception. “Excluded because the required image didn’t load” tells a reader what happened. “Low-quality participant” hides the actual decision and puts the blame on the person.

That also gives you a chance to fix the study. If a check catches people because its wording is confusing, excluding them won’t repair the question.

Borderline responses and your decision

Preserve the raw responses as collected, within your consent and retention commitments. Keep a separate exclusion log and build your analysis from a working copy. You should be able to trace every removal without reconstructing it from memory.

Then check whether the borderline cases change your conclusion. Run the planned analysis with those responses excluded, and compare it with a version that retains them. Keep clearly ineligible submissions separate from uncertain quality judgments.

Suppose you’re choosing which product benefit to lead with on a landing page. If the preferred benefit changes when you restore the speed-flagged responses, I wouldn’t present the cleaned result as a settled customer preference. I’d report that sensitivity and gather more evidence before committing to the change.

If the conclusion holds, you can say it held under that check. That still doesn’t prove every retained response was sound. It tells you something narrower and useful about the decision in front of you.

These are my recommendations for reviewing a small study. The Prolific example doesn’t establish that this process, or any particular threshold, will catch every bad response.

Before you launch your next survey, open a note beside the questionnaire. Write the first signal you plan to flag, why it could affect your answer, and what you’ll inspect before removing a response. If the only rationale is that someone might finish faster than you would, leave room for them to know the answer.