How To Avoid Selection Bias In Market Research

What Is Selection Bias?

Selection bias occurs when research participants are not randomly selected. This leads to errors in research results, which negatively impacts the validity of research findings because the sample does not accurately reflect the population of interest.

This blog post discusses a real-life example of selection bias, presents a type of selection bias, and some tips on how to avoid selection bias in your research.

Selection Bias in Politics: Truman vs Dewey

Selection bias is very common in research. It is so common that it has been seen in political settings for example the 1948 US presidential race between Turman and Dewey. During the race, a nationwide political telephone survey was conducted to determine who the majority of the US voting population were going to vote for. The political research results implied that Dewey would win the race by a significant majority of votes. Even the Chicago Tribune¹ printed a newspaper article with the headline “Dewey Defeats Truman”, suggesting Dewey had won the election. However, this was not the case when the election results were released. As you might have guessed, Truman defeated Dewey.

But what went wrong in the political research study? The telephone was still relatively new in 1948, therefore it came at a cost which meant only a minority of wealthy families owned telephones in their homes at the time. As it turned out, high-income individuals formed the bulk of Dewey’s voter base whereas middle to low-income voters were more likely to vote for Truman. The researchers failed to take this into account when selecting their data collection method. This thus resulted in a skewed response data in favour of Dewey, which led to inaccurate and unrepresentative insights.

A Type of Sample Selection Bias: Self-Selection Bias

There are different types of selection bias in sampling. Self-selection bias (also known as volunteer bias) is a common type of selection bias. Like selection bias, self-selection bias occurs when respondents are demographically or behaviourally different to the population of interest. However, with self-selection bias research respondents can decide on their own whether to participate in the research rather than being pre-selected into a sample by researchers.

The problem with self-selection bias is that certain groups of individuals with self-selecting characteristics may be attracted to taking part in particular studies. For example, research has shown that individuals that have sensation-seeking or thrill-seeking personalities are more likely to volunteer for certain research studies. This would skew the results of the research if it were to be studying these traits, making the research findings unreliable and ungeneralizable to the population.

How Reliable Are TripAdvisor Reviews?

Self-selection bias is so common that it can be seen in online reviews. This can be seen in a study² conducted using TripAdvisor hotel reviews across four states in the US. This study compared hotel-prompted online reviews with self-motivated online reviews. The study is framed using the Social Exchange Theory (SET). SET is a psychology theory that describes all social interactions as a function of perceived cost and reward. Costs refer to factors that inhibit one from acting in a certain way. Rewards refer to internal motivations that drive individuals to behave in a certain way. In the case of the TripAdvisor reviews, reward may be the satisfaction reviewers get from leaving a review and sharing their experiences, and the cost could be the burden of creating a TripAdvisor account if they do not have one already.

The researchers found that consumers’ decision to post an online review is associated with their level of satisfaction. Those who had bad or extreme experiences are more likely to post a review than those who had mild or positive experiences. This shows that there are specific characteristics that encourages self-selection in consumers which can skew ratings/reviews. In this case, those specific characteristics are bad or extreme experiences with the hotels where consumers stayed.

Consumers that were self-motivated into leaving a review did so because their perceived gain is much greater than the perceived cost of leaving a review. This poses a risk of the pool of reviews being dominated by consumer unrepresentative experiences and views of these hotels. This could ultimately have negative implications for the reputation of these hotels and can act as a deterrence for potential consumers.

How To Avoid Selection Bias

Avoiding selection bias can be rather difficult, however here are some tips on how to avoid it where possible:

• Use random methods of selecting the sample from the population of interest.

• Ensure that key characteristics of the sample reflect the characteristics of the population of interest as much as possible.

• Ensure that your data collection method is appropriate for the type of respondents needed for research.

• If after reviewing your data, you realise that some groups are overrepresented while other groups are unrepresented, this bias can be resolved by assigning weights to the misrepresented subgroups. Applying a weighted average to your data helps to take into account the proportional relevance of each subgroup. This can lead to results that more accurately reflect the study population’ s demographics.

¹ https://www.history.com/this-day-in-history/newspaper-mistakenly-declares-dewey-president
² https://journals.sagepub.com/doi/pdf/10.1177/1938965520902012