Comparison: Manual or automatic semantic analysis?

January 21, 2021

When conducting surveys or wanting to analyze customer feedback in-depth, it is very difficult to know how to go about it.

Indeed, when dealing with open responses or verbatim review sites, the task is much more difficult than taking an average with closed responses.

Yet this is the richest source of information about your customers. It would be a shame to do without it, wouldn't it?

This is why we are going to review in this article the two main methods used to perform a semantic analysis of your customer verbatim: manual analysis and automatic analysis.

This is why we are going to review in this article the two main methods used to perform a semantic analysis of your customer verbatim: manual analysis and automatic analysis.

1 / Manual analysis of client's verbatim

It is traditionally carried out with the technique of post-coding. This method consists of defining response themes and then assigning them a number. Thus, this makes it possible to process open responses in surveys with a level of precision approaching the processing of closed questions.

To perform this analysis, you can use spreadsheet software like Excel or equivalent. Then, you have to define the themes corresponding to your survey. You can also define sub-themes to gain precision in your results.

Once this step is complete, select all of the responses to your questionnaire (or a representative sample), then note the number of the corresponding theme or subtheme next to each response.

To better understand, we will see in more detail how to perform this analysis with a concrete example.

a) Define the themes of the survey

Manual verbatim analysis can be applied to any type of verbatim, whether from a survey but also to reviews on online platforms (TripAdvisor, TrustPilot, Google Reviews, Verified Reviews, Critizr, etc.).

For the purposes of this analysis, consider the example of a survey in the banking sector with an open-ended question such as "Are you satisfied with your bank branch?"

By discovering the answers to the questionnaire, define the themes that emerge and that seem most relevant to you.

For example :

Staff

1.1 Staff reception

1.2 Listening skills

1.3 Expertise & Skills

1.4 Availability

Agency

2.1 Opening hours

2.2 Access to the agency

2.3 Geographical proximity

2.4 Atmosphere

2.5 ATM

Customer journey

3.1 Making an appointment

3.2 Support

3.3 Termination

Product

4.1 Current Accounts

4.2 CB / CR

4.3 Pricing

The codification plan must however be fixed at the time of codification. For the sake of comparability, we will hesitate to change it too often. Any addition of a theme during analysis may require a new analysis of all data already classified as well as historical data. Unlike an automatic classification tool, the thematic plan is therefore not very scalable.

On an ad hoc basis and if comparability with your other surveys or sources of customer reviews is not useful, we recommend that you add current topics in your topics.

In addition to analyzing the themes raised, you can also classify the comments by sentiment (positive, negative, neutral) or primary emotion. In order to carry out a cross-analysis, it is, therefore, necessary to make the themes neutral and without valence.

Thus: The Personnel> Availability theme must include both times of availability and times of staff unavailability. It is the feeling/emotion dimension that will add the level of satisfaction or emotion of the subject.

To organize your time: Allow between 1 day and 3 days to establish a codification plan that can be used on the upcoming survey (s).

b) Carry out the codification

Now that our themes and sub-themes are defined, the real fun begins! You must now assign the corresponding number to each of the responses to the survey. To better visualize the task, here is an example with 10 customer comments:

Capture d’écran 2021-01-20 à 17.26.01.png

To summarize, manual semantic analysis of verbatim with coding allows for thorough and precise results. On the other hand, the preparation and the analysis are very time consuming because it is necessary to go through all the verbatim. To limit the coding time, we can use a random extract (eg: each answer every 10 answers is coded) but therefore the analysis is no longer exhaustive and problems of statistical representativeness may arise.

It should also be noted that the longer the verbatim, the more tedious the classification.

On the other hand, if the verbatim are multilingual, it will be necessary to ensure that the analyst will master all the subtleties of the different languages ​​of expression in order to proceed with the proper consolidation of the results by country or by language.

Therefore, depending on the volume of the survey (or other sources of advice), the length of the comments, it is not advisable to use this time-consuming and resource-intensive method and instead opt for the analysis automated.

On the other hand, the efforts made are not capitalized for future studies or data. Any new analysis will require starting from scratch.

In addition, although the results are accurate, there may be a human bias in responses related to lack of objectivity depending on each analyst. For the same data, the results may indeed vary from one person to another depending on the perception and personal experience of the person who coded the survey or analysis. This bias can be a particularly important problem if there are prejudices or corporate culture that do not spontaneously encourage listening to the customer.

Finally, these manual analyzes are difficult to compare with each other.

To organize your time: Allow between 2 to 3 minutes per verbatim to classify, i.e. approximately 1 week for 1000 short verbatim and 1 week 3 days for 1000 long verbatim (greater than 144 characters). This time is in addition to the time dedicated to designing the codification plan.

Our recommendation: Using the analysis of customer comments by manual coding can be good for getting to know what customers say and their opinion, but it is always very time-consuming. If your survey or opinion source has a significant number of comments and you want to be thorough, you will quickly find yourself overwhelmed. This analysis is therefore particularly suitable for small amounts of data.

As briefly explained previously, there is another much faster and better way to analyze your customer feedback: automated verbatim analysis.

2 / Automatic verbatim analysis

Unlike manual analysis, a semantic tool is able to analyze a large number of verbatim in no time and identify the subjects and themes that emerge from them.

You have to understand that in the vast majority of projects, setup times are negligible and that this approach allows you to go more directly to analysis and research of insights. On the other hand, in addition to the automated analysis itself, the solution provides you with visualization tools that allow you to go directly to the data and easily discover the lines of analysis relevant to your survey. By choosing the analysis grids for your sector offered by the solution, you save configuration time and you can even benefit from a sector benchmarking vision.

In this article, we'll walk you through how to use an easy-to-learn tool like CXinsights.io, which can help you speed up your customer feedback analysis ... and find insights quickly and effortlessly.

First of all, you should know that the algorithm of the CXinsights.io solution is able to identify the topics discussed and associate a felt emotion (among the 6 primary emotions: joy, surprise, fear, sadness, anger, surprise) to reveal new insights at a glance.

Capture d’écran 2021-01-20 à 17.37.26.png

If the verbatim are multilingual, the classification will allow processing of each verbatim in a transparent way to easily give results by country or by language.

The size of the comments has a marginal impact on the processing times of the automatic classification tools.

To start your analysis, all you need to do is import your data file into the SaaS platform. You can also import verbatim directly from Google Reviews, Trustpilot, Tripadvisor and other review platforms if needed.

Capture d’écran 2021-01-21 à 10.07.10.png

You just have to wait a few moments for the algorithm to automatically process your data and give you the first results.

On the CXinsights.io automatic analysis platform, you can choose a theme corresponding to your industry or even customize your entire project based on personalized themes and sub-themes that seem more relevant to you. This setup only takes a few minutes. Customization of themes, route etc. is also possible on request (custom setting requiring a few hours of additional work).

Note: unlike manual analysis, any customization effort made during project configuration can be capitalized on all data.

For example, if you create a new topic even 2 months after the first treatment, you can update all the data effortlessly in minutes.

To organize your time: Once the file has been submitted, plan a processing time of 2 to 3 minutes per batch of 5000 verbatim to classify, or about 20 minutes for 50,000 verbatim. The tool will then automatically place all verbatim in the corresponding themes and you can immediately access data navigation.

You can compare at a glance the number of comments by theme as well as the main emotion that emerges (the E-index in the screenshot below corresponds to a temperature varying from -20 ° C to + 40 ° C defined by the main emotion detected in the speech, the higher the temperature, the more positive the emotion).

Capture d’écran 2021-01-21 à 10.10.10.png

Monitoring is much more easily done than with manual analysis, the tool offers you a follow-up of the evolution of emotions in customer verbatim as well as of the E-Index.

Capture d’écran 2021-01-21 à 10.15.08.png

In the screenshot above, for example, we see that there has been a drop in joy and an increase in sadness at the same time from the month of September 2018. It is therefore important to take note mainly of the subjects evoked over this period to understand where this change comes from.

To do so, we choose the corresponding period on the platform which will then automatically identify the 3 points of success and the 3 points of improvement mostly mentioned in the comments.

Capture d’écran 2021-01-20 à 17.16.01.png

In this example, we see that there are 3 major risks that correspond to this fall in joy and increase in sadness: the churn risk, Current Operation and Ability to Solve Issues. By clicking on the “Ability to Solve Issues” button, we can quickly realize that poorly done customer support is the root of this problem and is mentioned in customer reviews. We can then prioritize the actions to be implemented to solve the problem.

Capture d’écran 2021-01-21 à 10.19.26.png

As you understood, automatic semantic analysis is essential when you are conducting large-scale surveys or receiving numerous comments each month on review platforms or social networks.

A tool like Q°emotion (editor of the CXinsights.io platform) will allow you to save precious time and very quickly identify the subjects mentioned by your customers as well as the emotions they feel.

We were only able to review some of the features of our SaaS platform, if you want to know more, you can test our solution for free by clicking here, or discover our customer cases by clicking here.

Capture d’écran 2021-01-20 à 17.26.17.png

Share this article

Similar posts

How does this British bank compare customer reviews across its branch network?

February 17, 2021

The banking industry is probably the area where customer experience has a central role to play. Indeed, with the emergence of online banking, consumers increasingly need to feel close to their bank an...

Our top 5 best sources of Customer Reviews

December 02, 2020

Feedback from your customers is a reliable and comprehensive source to enrich your knowledge of them. \nAnalyzing these comments\n will allow you to learn more about your customers and optimize their ...

Q°emotion enables you to…

Process a large
amount of data

Studies & Data

Q°emotion, a tool for ...

Improve the customer experience.

Marketing & Customer Experience

Want to test our tool?

Ask for a
test of our tool!

phone