How to code & analyze open-ended questions

Want to grow your business more efficiently and enhance your bottom line? Understanding your customer insights and feedback is the key. However, analyzing survey responses can be overwhelming and time-consuming. 

By coding the responses to open-ended questions, you can streamline the process and save days of manual labor. The insights you gain will help you drive improvements at every stage of the sales and marketing funnel, boosting your revenue and customer success. (Note: in this context, to “code” = giving a piece of data a label).

When you ask customers open-ended questions, such as "What do you think about our pricing structure?", it usually takes more time and effort to sort through their answers than reviewing closed-ended responses to questions like "Does the subscription provide good value?"

That’s because open-ended questions allow customers to provide their thoughts and opinions in their own words, which are bound to be more varied and complex than just a “yes” or “no.”

Survey coding is a way to review your entire set of open-ended responses, allowing you to identify common themes and sort them into appropriate categories or groups using tags.

Ideally, you’ll see some patterns emerge, allowing you to accurately analyze common sentiments (neutral, positive, or negative) and gradually build a better, even more customer-centric business.

A coding frame is a predefined set of categories, themes, or topics to categorize open-ended survey responses. The coding frame guides sorting responses into relevant categories, making it easier to analyze and draw insights from the data. The coding frame can be created manually or through automated means. This frame could be flat or hierarchical, representing the structure of your research's themes.

Flat coding frame

Using a flat coding frame for manual coding has some advantages, as it can be a simpler and faster approach. However, as the number of codes increases, organizing and navigating the themes and concepts becomes more challenging.

Furthermore, it can be challenging to determine which themes are more important, which could slow down the decision-making process. Therefore, it's essential to carefully consider the pros and cons of using a flat coding frame and determine if it is the most effective method for your specific needs.

Hierarchical coding frame

Hierarchical frames are a way to organize code based on their relationship. For instance, you can organize codes according to your customers' sentiments on a particular topic. Suppose you conduct a survey to ask customers about their experience with your customer service. You may choose to use codes that only capture their feedback related to customer service.

However, upon analyzing the survey responses, you may also find many comments about your company's products, and you must code all the responses from scratch. To avoid this, you can use a flexible coding frame that covers different topics and insights. Such a coding frame will allow you to reuse the results later and obtain a comprehensive understanding of your customer's feedback.

Here is a sample hierarchical frame:

In this example, codes are organized hierarchically by topic, sentiment, and attribute/theme:

1. The top code pertains to customer service. 

2. The middle-level code determines whether the sentiment is positive or negative. 

3. The bottom level provides more information about the topic by focusing on a specific attribute or associated theme.

12 common coding categories

Here are some popular, predefined codes that might apply to your product or service (but feel free to create your own):

1. UXD - UX design

2. FTR - Feature requests

3. PRC - Pricing

4. TCHSPT - Technical support

5. PFM - Performance

6. SEC - Security

7. UPD - Updates and upgrades

8. USG - User segmentation

9. ANL - Analytics

10. CSTM - Customer satisfaction

11. MTKT - Marketing

12. RSR - User research

It's essential to ensure comprehensive coverage of your categories. Take time to plan your coding categories thoughtfully, and don't add extra codes for the sake of it. However, if you realize the need to add a new category once the coding is underway, reevaluate your previously reviewed responses, ensuring your method is consistent.

Organizations commonly use a hybrid approach to survey coding, leveraging AI for initial sorting and coding (automation), followed by a layer of human quality control (manual). Understanding the advantages and drawbacks of both approaches will help you adopt an effective survey analysis strategy.

Manual coding:

Manual coding puts a human in control of every step of the process. Someone will read through all the open-ended survey responses received and categorize each response as they go. They'll assign a tag once they decide what category the response falls under. 

Pros of manually coding surveys

• Proximity to data—Having close contact with the data at every step.

• Nuanced, deeper insights—Perceiving in ways that AI-powered software or machine learning (ML) tools can't.

• Enhanced survey value—The potential to extract deeper insights and better understand your customers. 

Cons of manually coding surveys

• Time-consuming—Manual coding tends to be time-intensive, verging on painful.

• Lacks scalability—This can be unrealistic with a small or rapidly growing team.

• Error-prone—Likelihood of introducing human error or subjectivity stemming from biases.

Automated coding:

This type of coding leverages the power of machine learning (ML) algorithms and AI software to streamline the process. While a manual review is usually still necessary during and after, automated coding is an increasingly reliable option, mainly if you conduct many surveys or collect large amounts of data. 

Pros of automatically coding surveys

• Large volume efficiency—If you want to run frequent surveys or work with large datasets, automated coding is a time-saving option compared to manual coding.

• Resource optimization—Efficiently processing survey responses saves time and money, allowing the option to redirect attention to more business-critical responsibilities and decisions.

• Scalability and sustainability—If you want to grow and sustain surveys as a vital feedback loop, automated response coding will streamline the process and make data substantially easier to analyze.

Cons of automatically coding surveys

• Potential for errors—While automated coding dramatically reduces human error, it can still make mistakes, especially if nuanced details, context, and cultural factors need to be detected.

• Distance from data—There will be a degree of separation between analyst and data, which some managers and project leads will (understandably) consider a new risk to manage.

• Requires a trial run—Adopting new tools or processes that reduce human oversight always calls for a trial run to identify and ward off potential issues before full implementation.

Manual and automatic coding are equally valuable methods for capturing and displaying survey responses. The choice of which you should use depends mainly on the size of your team and the number of surveys you plan to conduct.

Manual coding is generally best for managing small samples of survey data, while automated coding is ideal for large survey samples.

Manual coding is usually the gold standard for academic or medical research. 

Customer service, support, marketing, and product research teams use automatic coding extensively.

Again, you don't necessarily have to choose one or the other—finding a hybrid approach can be a practical option.

Consider how many employees you have on staff and how much data you will be working with, and create a plan accordingly. Then, consider trialing a tool for coding and analyzing open-ended questions.

Whether you opt for manual or automated coding, you should follow some best practices. These guidelines help you streamline organizational processes and review survey responses with accuracy, consistency, and reliability.

Create a coding framework:

Before you start coding responses, develop a coding framework or manual. This document should outline the key themes, concepts, or categories you are looking for in the responses. Clearly define each code and provide examples to guide the coding process. A well-defined framework helps maintain consistency across different coders and ensures a standardized approach to analysis.

Train coders and establish inter-rater reliability:

If multiple coders will be involved, providing training sessions to ensure a shared understanding of the coding framework and process is crucial. Training helps reduce ambiguity and ensures that coders interpret and apply codes consistently. After the initial training, conduct coding exercises on sample responses to assess inter-coder reliability. This step helps identify discrepancies in coding interpretations among coders, allowing for adjustments and further clarification.

Use software:

Tools like Dovetail provide features for organizing, coding, and analyzing qualitative data. Using software can facilitate collaboration among coders, allowing for easy tracking of coded segments and supporting the exploration of patterns and relationships within the data.

Regularly review and refine codes:

Periodically check the coded data to ensure the ongoing relevance and accuracy of the coding framework. As you gain a deeper understanding of the data, you may discover new overlooked themes or subcategories. Be open to refining and expanding the coding framework as needed. Regular reviews also help maintain consistency over time and ensure that any changes in coding reflect an improved understanding of the data rather than inconsistencies in interpretation.

Remember that coding qualitative survey responses is an iterative process that requires attention to detail, ongoing communication among coders, and a commitment to refining the framework.

Studying customer feedback is about more than just gathering responses. It's about turning those responses into valuable insights to help you make necessary changes and share learnings with stakeholders. With Dovetail, you can seamlessly import data, highlight critical themes, and collaborate with your team. By working together and analyzing data, you can uncover patterns and make informed decisions that drive growth and success.

To present findings from open-ended survey questions to key stakeholders, focus on clarity and relevance. Begin with a brief intro outlining the purpose of the survey and the significance of open-ended responses. Summarize themes, and consider excerpting a few illustrative quotes to capture sentiments. Highlight any notable patterns or variations and emphasize the most relevant insights. Utilize visuals like word clouds, bar charts, or grids for added impact. Acknowledge potential limitations transparently. Conclude with a summary of actionable recommendations and their likely impact.

Open-ended questions inspire candid sharing of thoughts and opinions (qualitative feedback). They typically ask What, How, Who, or Why.

A few examples include:

"How would you describe your experience with our company?"

"How would you compare our customer service to other companies you've dealt with?"

"What would improve that?"

“What is the most helpful product/service feature? Why?”

Coding responses involves organizing and tagging your data to identify themes and relationships. During this stage of the survey process, you'll assign labels to specific words or phrases that represent recurrent themes. It is a crucial step in understanding customer insights since you'll be able to start to recognize points and overarching themes in the responses received.