How to run effective research critique sessions: peer-reviewing study designs before fieldwork

Most research teams discover methodological problems too late. A discussion guide turns out to be leading. The sample skews toward power users. The research questions are too broad to produce useful findings. These issues surface during analysis—or worse, during a readout when a stakeholder asks a question the study was never designed to answer.

Research critique sessions are a straightforward way to catch these problems before they become expensive. The concept is simple: before fieldwork begins, the lead researcher presents their study design to a small group of peers who review it for blind spots, assumptions, and methodological weaknesses.

This article covers how to structure these sessions, what to include, who should participate, and how to build a critique culture that actually works.

Why peer review matters before fieldwork

In academic research, peer review happens before publication. In applied UX and product research, there is rarely a formal equivalent. Study designs are often reviewed only by the researcher who created them, or briefly discussed in a 1:1 with a manager. This means the plan is shaped by one person's perspective.

That is not a problem of competence. It is a problem of perspective. Every researcher carries assumptions—about the problem space, the users, the method, the right questions to ask. A second or third set of eyes can surface things the lead researcher has internalized without realizing it.

Common issues that peer critique catches include:

• Leading or loaded questions in discussion guides that nudge participants toward a particular answer
• Unclear or overly broad research questions that will produce findings too vague to act on
• Sampling criteria that unintentionally exclude important user segments
• Method-problem mismatch—choosing a method that cannot actually answer the research question (e.g., running a usability test when the real question is about adoption motivation)
• Missing success criteria that make it unclear what a "good" finding would look like
• Analytical assumptions baked into the plan without being stated

Catching even one of these before fieldwork starts can save weeks of effort and prevent a study from producing ambiguous or misleading results.

What to include in the study design document

A critique session is only as useful as the material being reviewed. The researcher should prepare a written study design document and share it with reviewers at least two business days before the session. Asking people to review a plan live, with no preparation, leads to surface-level feedback.

The document does not need to be long. One to three pages is usually sufficient. It should cover:

Research objectives and questions

State what the study is trying to learn and why it matters now. List the specific research questions the study is designed to answer. These should be precise enough that reviewers can evaluate whether the proposed method and protocol will actually address them.

A vague objective like "understand the onboarding experience" gives reviewers little to work with. A specific one like "identify where and why new users abandon the setup flow within their first session" is much easier to critique.

Method and rationale

Describe the chosen method—interviews, usability testing, diary study, survey, contextual inquiry, etc.—and briefly explain why it is the right fit for the research questions. This is where method-problem mismatches become visible.

Participant criteria and recruitment plan

Define who will be included in the study and how they will be recruited. Specify screening criteria, target sample size, and any demographic or behavioral characteristics that matter. Note who is deliberately excluded and why.

Discussion guide or protocol

Include the full set of questions, tasks, or prompts that will be used during sessions. This is usually the most productive section for critique. Reviewers can identify leading phrasing, missing topics, question ordering effects, and tasks that are too easy or too difficult to reveal useful behavior.

Analysis approach

Describe how the data will be analyzed. Will sessions be coded thematically? Will usability issues be rated by severity? Will survey responses be segmented by user type? Stating this upfront helps reviewers assess whether the study design will actually support the intended analysis.

Timeline and constraints

Note any constraints that shaped the study design—budget limits, tight timelines, limited access to a particular user group. This context helps reviewers distinguish between genuine blind spots and deliberate trade-offs the researcher has already considered.

How to structure the session

A well-run critique session typically takes 45 to 60 minutes. Longer sessions lose focus. Shorter ones do not leave enough time to work through the material.

Before the session

• Share the study design document at least two days in advance
• Ask reviewers to read it and come with written notes
• Designate a facilitator (this can be the lead researcher, but a neutral facilitator often works better)

Opening: context and ground rules (5 minutes)

The lead researcher gives a brief verbal summary of the study—its purpose, its stakes, and any context not captured in the document. The facilitator sets ground rules:

• Feedback should focus on the study design, not the researcher
• Critiques should be specific and accompanied by a suggestion or question, not just an objection
• The goal is to strengthen the plan, not redesign it from scratch

Structured walkthrough (25–35 minutes)

Rather than opening the floor for general comments, walk through the document section by section. This prevents the conversation from clustering around the first issue someone raises while neglecting the rest of the plan.

For each section, the facilitator asks: "What concerns, questions, or suggestions does anyone have about this part of the plan?"

The most productive critique sessions focus on a few key questions:

• Will this method answer the stated research questions? If there is a gap between what the study asks and what the method can reveal, that is the most important issue to surface.
• Are the questions or tasks phrased neutrally? Review each discussion guide question for leading language, double-barreled phrasing, or assumptions embedded in the wording.
• Who is missing from the sample? Consider whether the participant criteria inadvertently exclude a group whose perspective matters.
• What could go wrong during sessions? Anticipate practical problems—tasks that depend on account data participants may not have, questions that require recall over long time periods, or session lengths that are too ambitious.
• Will the findings be actionable? Consider whether the team receiving the results will be able to make decisions based on what this study produces.

Closing: prioritize and decide (10 minutes)

At the end of the session, the facilitator helps the group prioritize. Not every suggestion needs to be adopted. The lead researcher should leave with a clear list of changes they plan to make and, if applicable, a brief explanation of feedback they considered but chose not to act on.

Document the decisions. A shared record of what was raised and what was changed (or not) creates accountability and makes it easier to reflect on the study later.

Who should participate

The composition of the review group matters. Three to five reviewers is the right size. Beyond that, the conversation becomes unwieldy.

At least one experienced researcher. Someone with strong methodological knowledge who can evaluate whether the method, sample, and protocol are sound.

Someone with domain expertise. A person who knows the problem space well—whether that is a product manager, a designer, or another researcher who has studied the same area. They can identify missing questions or participant segments that a methodological reviewer might not catch.

An occasional outsider. Including someone who is not close to the project can be surprisingly useful. They will ask basic questions that expose assumptions the core team has stopped noticing.

People who should generally not participate: the project's direct stakeholders. Their presence can shift the conversation from methodological quality to strategic preferences, which is a different discussion.

Common pitfalls and how to avoid them

The session turns into a redesign meeting

This happens when reviewers suggest entirely different research approaches instead of critiquing the one on the table. The facilitator should redirect: "Given the method and scope that's been chosen, how can we make this study stronger?" Fundamental method disagreements should be flagged but resolved separately.

Feedback is too vague to act on

Comments like "the discussion guide feels off" or "I'm not sure about the sample" are not actionable. The facilitator should ask follow-up questions: "Which specific questions feel off, and what would you change?" Encourage reviewers to be concrete.

The researcher becomes defensive

Having your work critiqued is uncomfortable. It helps to frame the session explicitly as a quality practice, not a performance evaluation. Some teams rotate the critique role so that everyone's work gets reviewed, normalizing the process. Over time, researchers often come to value these sessions because they produce better studies and reduce the anxiety of discovering problems late.

Only senior researchers get critiqued

In some teams, critique sessions are treated as a development exercise for junior researchers. This undermines the practice. Senior researchers benefit equally—often more, since their plans may receive less scrutiny through informal channels. Make critique a standard step for all studies above a certain scope, regardless of who leads them.

Building a critique culture on your team

Running one critique session is easy. Making it a durable practice requires intentional effort.

Make it a process step, not a favor. If critique sessions depend on goodwill and open calendars, they will not survive the first busy quarter. Build them into your team's research workflow as a defined step between study planning and fieldwork.

Keep the time commitment reasonable. Reviewers should spend no more than 30 minutes reading the document and 60 minutes in the session. If the burden is higher than that, people will stop participating.

Track the impact. After a study concludes, briefly note whether the critique session surfaced issues that would have affected the findings. Over time, this creates a compelling case for continuing the practice.

Use a shared template for study design documents. A consistent format makes it faster for reviewers to read and evaluate plans. It also helps researchers remember to include all the necessary components. Tools like Dovetail can centralize these documents alongside your research repository, making it easy for reviewers to reference past studies and see how similar methodological questions were handled before.

Rotate facilitation. Having the same person facilitate every session creates a single point of failure and limits development opportunities. Rotating the role also gives everyone practice in giving structured feedback.

When to skip a critique session

Not every study needs a full peer review. Quick evaluative tests with established protocols, short surveys using validated instruments, or lightweight guerrilla research may not warrant the overhead. A useful threshold: if the study will take more than a week of effort, influence a significant product decision, or use a method the researcher has limited experience with, it is worth the 90 minutes of total reviewer time.

For smaller studies, a lightweight alternative works well: share the discussion guide or protocol in a Slack channel or shared doc and ask for asynchronous comments within 24 hours. This captures many of the same benefits with less coordination cost.

Making the findings from critique sessions discoverable

One underappreciated benefit of research critique sessions is the institutional knowledge they generate. The methodological discussions, trade-off decisions, and alternative approaches considered during critique are valuable beyond the immediate study. When these are documented and stored in a research repository—platforms like Dovetail are designed for exactly this kind of knowledge management—future researchers can learn from past decisions instead of repeating the same debates.

Over time, a team that runs consistent critique sessions develops shared standards for research quality. Questions that once required 15 minutes of discussion ("Should we include non-users in this sample?") become settled norms that reviewers can reference quickly. This is how a research practice matures—not through top-down mandates, but through repeated, structured conversation about what good research looks like.

A research critique session is a structured meeting where peers review a proposed study design before fieldwork begins. The goal is to identify methodological blind spots—things like biased interview questions, unclear research objectives, sampling gaps, or analytical assumptions—that the lead researcher may not see on their own. These sessions are not about judging the researcher's competence. They are about improving the quality of the research itself by bringing multiple perspectives to bear on the plan before data collection starts.

Three to five reviewers is generally the right range. Fewer than three limits the diversity of perspectives, while more than five tends to dilute focus and make discussions harder to manage. The group should include at least one person with strong methodological expertise and, ideally, someone with domain knowledge of the topic being studied. Including a product manager or designer can also help ensure the study design will produce findings that are actionable for the team.

The session should happen after the researcher has a drafted study plan—including research questions, method selection, participant criteria, and a discussion guide or protocol—but before any recruitment or fieldwork begins. This timing is critical. Running a critique too early, before the plan is concrete enough to evaluate, wastes everyone's time. Running it too late, after participants are scheduled, makes it difficult to act on feedback without delaying the project. A good rule of thumb is to schedule the critique at least one week before the planned start of fieldwork.