The analysis of customer feedback through open-end questions is a crucial part of digital research.
Closed-end questions provide focused, quantified data, but it is through open-end questions that organizations discover what customers really want to say to them. A common topic that respondents mention in open-ended feedback is the site’s usability. Reading through the textual responses provides researchers insights about the issues that site visitors are experiencing. With high-traffic websites, which collect large volumes of data, the challenge is figuring out patterns of responses and determining priorities among the numerous issues customers mention. By doing this you can identify what makes the website easier to use for the majority of the visitors.
At iperceptions, we leverage our vast collections of survey data, text analytics and proven research frameworks to extract actionable insights from free text data. A prerequisite to this exercise is to understand what people mean when they say that something is hard to use, and that is what I will discuss in this post. The two key components that we need to understand are: how people perceive usability and how people describe it with language.
Perception is everything
We often use scales (i.e., numeric values) to describe the perception of how easy or hard an activity is. For example, one might say that given a 5 point scale where 1 is the easiest and 5 the most difficult, going to a grocery store is 2 and dropping a child off at daycare is 4. The values that people pick for a given activity varies from person to person and the particular values people assign to a particular activity may change with time. Still, numerical scales are useful way to communicate this type of information.
Scales and Language
What we need to establish next is how people use language to express scalar information. We might at first think that, given a five-point scale of usability, language would map 1 as very easy, 2 as easy, 3 as neither easy nor hard, 4 as hard, and 5 as very hard. We can graphically represent this distribution as figure 1.
Unfortunately, this nicely systematic distribution of adjectives does not match how people actually speak.
In a recent study, we used the survey data in which site visitors 1) rate the site’s usability on a scale and 2) describe their experience in their own words. Firstly, we identified the most common expressions relating to usability that people use in textual responses. These included; “easy”, “not easy”, “difficult”, and “hard”. We then created frequency distribution charts to see how the respondents who used these phrases in the text, numerically rated the site’s usability.
A pattern emerges for negative comments (e.g., “hard”, “difficult”, “not easy”), which differs from positive comments (e.g., “easy”). Those respondents who make a positive comment about the usability also rated the usability high on the numerical scale. The graph below is an example of the distribution of ease of use ratings by the respondents who said “easy” in the survey. The score is based on an 11 points scale, where 0 is very bad and 10 is outstanding.
In contrast, those respondents who made a negative comment about the usability vary widely in their numerical rating of usability. In fact, the variability of usability rating for these respondents is as spread-out as the rest of the survey respondents. The example below shows an example of the ease of use ratings by those respondents who mentioned “difficult”. The scale used is the same as the previous one.
This pattern might appear surprising, but it is consistent with the formal semantics of antonymous adjective pairs, such as “difficult” and “easy”. Theoretical linguists have noted that a number of antonymous adjective pairs show this type of relation to the scales of perception.[i], [ii] A positive adjective such as "easy" expresses an extreme end of a scale (e.g., 10 out of 10 on the usability scale), and anything that is not perfect (e.g., 9 and below on the same scale) is expressed with the negative counterpart, “difficult” or “hard”. So instead of the nice distribution we expected, what we found was a skewed distribution, as illustrated in figure 2 below.
Other antonymous adjective pairs that also have this pattern include “clean” and “dirty”, “safe” and “dangerous”. Human behavior is not as clear-cut as what theoretical linguists predict, so we do see respondents mentioning “easy” selecting 9 and 8 as well as 10. But what is most relevant to us is that it is the fundamental semantics of these negative adjectives that they can mean any part of the scale below the extreme positive end (i.e., the point of perfect positive). Our studies show that other negative comments, such as “hard”, “not easy”, “should be easier” pattern with “difficult”; the respondents are spread out across the entire scale of usability. We observe this pattern with multiple surveys from different industries.
Not all negative feedback is a cause for concern
For the purpose of web analytics, open-ended survey questions are extremely useful. Survey respondents are forthcoming in describing their likes and dislikes. The challenge is in measuring the importance of possible issues and insights through aggregation. Our study on usability has shown that language is skewed in providing scalar information. What respondents mean by “hard to use” vary greatly. In most cases, only half of these negative responses are cause for any concern.
To identify which half, the numerical rating provides more granular indication than the linguistic expressions. This lesson applies to different dimensions of Active Research. iperceptions’ feedback collection includes both open ended questions and numerical questions, because to complement the power of open ends it’s important to gather detailed insights with the scalar measurement from the numerical ratings.
[i] Kennedy, C. (2007) Vagueness and Grammar: the Semantics of Relative and Absolute Gradable Adjectives. Linguistics and Philosophy 30: 1-45.
[ii] Rotstein, C. & Y. Winter (2004) Total Adjective vs. Partial Adjectives: Scale Structure and Higher-Order Modifiers. Natural Language Semantics 12:259-288.