Our company has an axiom and it is very simple – good research comes from asking the right question of the right person. Of course, there is much more to market research than this truism. The design of the study, the interpretation of the findings and the presentation of results can be just as important. However, at the heart of good research are those two things – the right question and the right person.
My guess is that 80-90% of the time spent by the research team that oversees a market research project is making sure we ask “the right question”. The questionnaire is critically important and, what’s more, it is very visible and accessible. You can see it, you can discuss it and you can change it. It usually does get changed numerous times and it isn’t unusual for a questionnaire to end up as version 5 or 6 before everyone is satisfied and it is ready to launch. The attention that is given to the questionnaire is huge because in many respects it is far easier to focus on this than the respondent selection. However, the respondent selection has a significantly greater impact on research quality.
Figure 1: Effect Of Respondent Selection On Research Quality
Determining whether we are speaking to the right person is a more difficult and tedious task than making sure we have the right question. There will probably be quotas set to ensure that the right balance of people are interviewed but the decision as to which respondent is chosen at a particular company is very often left to the interviewers. The interviewers are unsung heroes who do their best to get the right person to speak to, but their task isn’t easy.
In business-to-business market research the best respondent to interview can be hidden in the depths of a company. Let’s take, for example, a business-to-business survey in which we want to speak to people who buy and specify cutting oils. Cutting oils are a typical industrial consumable used in large and small metalworking companies. In a large company there could be three or four people who have an influence on this subject, with some of them on the production line and some in a technical department. It would be unreasonable to expect that a receptionist on a company switchboard can put an interviewer through to such a person, especially if the caller doesn’t know the name of the person they are seeking. Company receptionists are inundated by unsolicited calls and turn people away if they can’t name the person they want to speak to. No name usually means no contact.
There are ways and means of getting names of contacts. A search on the internet and specifically on LinkedIn could provide some contact suggestions. However, if the aim is to conduct a survey of 200 people it will be necessary to compile a list 10 times this number – and the task of searching for 2,000 cutting oil contacts on the internet would be a Herculean, if not impossible, task.
This means that business-to-business market researchers are highly dependent on their clients to provide suitable lists of respondents. This should be a simple task as it may be assumed that every company knows its customers and potential customers and can easily download a list from a database. However, databases are only as good as the information fed into them. Even the most disciplined of companies finds it difficult to keep up-to-date telephone numbers, email addresses and names of contacts in their databases despite the relatively small numbers of people they deal with. The very largest databases in most business-to-business markets number only a few thousand contacts and many are measured in just hundreds. An “acid test” or indicator of a marketing-orientated business-to-business company is the quality of its customer and potential customer lists. Direct contact with customers via a well-organised database is critical to business-to-business marketing.
It may be assumed that names of specifiers of cutting oils (for which read any other business-to-business product) can easily be purchased from some list supplier. Lists that can be purchased often originate from subscribers to journals and magazines and could include many people of dubious interest. The lists are frequently out of date. It is a cruel but wise assumption that by far the bulk of the names on a purchased list will be of little value.
This moves us on to what has become a poisoned chalice to market researchers. Over the last five years panel companies have arisen, whose business it is to assemble names and contact details of people who are happy to take part in market research surveys. These panel companies offer small incentives such as air miles or vouchers to their members who complete questionnaires. Market research companies design questionnaires which are scripted for an online survey and completed by these panel members on a “cost per complete” basis. Crucially, however, the market research companies who buy the interviews do not know the name of the person or company that has replied. They can and do collect self-reported data from each respondent which shows which industry vertical they are in, the size of the company, the department in which the respondent works, etc., but they will not know the company name or the respondent’s name.
As you are reading this you will quite reasonably ask yourself what type of person signs up as a panel member. You may find it hard to imagine that there are many CEOs of large companies that are interested in completing market research surveys to earn a few loyalty points towards a holiday. Also, you would not imagine that there will be many hundreds of buyers and specifiers of cutting oils on these panels, if any. You may question whether there are many tradesmen on panels who, after a day sweating up the ladder, relax by ticking boxes on questionnaires. And, in the main, you would be right. The panel companies have good access to a variety of middle and junior managers who sign up to panels and complete half a dozen questionnaires before finding something else to interest them.
Without knowing precisely who has completed an interview, you are driving blind. All you see are the answers. Fortunately these give a clue as to the quality of the interview. If the answers to rating questions are all the same (we call it flat lining) then the chances are that the respondent clicked through the interview without giving due consideration to each question. A well completed questionnaire is likely to have at least some variation in views and ratings. We can also check how long it takes for a respondent to complete an online interview. If the interview is designed to be completed in 15 minutes and someone completes it in only three or four minutes, we can presume that it hasn’t been taken seriously. We can test logic by including cross-checking questions so if someone gives an answer to one question which doesn’t square with the answer to another, we can presume it is a bogus response. The proportions will vary but, on average, around 20% of responses to a panel survey need to be removed and replaced using quality checks of this type.
So here’s the thing: the client and the market research agency spend 80% of their time focusing on the questions and 20% or possibly less of their time thinking about respondents themselves. If I could choose between a poorly constructed questionnaire and a good respondent, or a good questionnaire and an inappropriate respondent, I would choose the former every time. We need to improve the quality of surveys by spending not less time on questionnaires, but more time on respondent selection.
Here are my top ten tips for improving survey quality by ensuring that the right person is interviewed:
-
Ask the sales force for their “Christmas card list” of contacts. They all have one and this is very often a definitive list of who they think it is worth speaking to. If the sales force claim they can’t provide such a list, it would be hard to believe they are doing an adequate job. Be prepared for some members of the sales force being reluctant to release these precious contacts and, if necessary, put pressure where it matters to have them released. It won’t make you the best of friends with the sales team but it will result in a better survey.
-
Look for contacts beyond those with a procurement title. In business-to-business decision-making, procurement personnel may be day-to-day contacts but key decision-makers are often located in other departments.
-
Start the list building process early in the program. Don’t wait until the questionnaire has been developed and don’t assume that the list can easily be assembled at the last minute. Building the list usually takes time.
-
If you have to build your own list of contacts, don’t be too ambitious. It is better to get 100 interviews with the right people than 1,000 interviews with the wrong people. Quality beats quantity in list building.
-
When specifying what should be included in the contact list, stay realistic and don’t set up a request so complicated it is forced to fail. You need the name of the company, its location, the telephone number of the contact, and the contact’s name. Everything else is nice to have but not vital.
-
Don’t leave the list building task to a junior person. It should be supervised by the research executive in charge of the project as it is one of the most critical components of the research design.
-
When working with panel companies, make sure that questionnaires have numerous screening questions which sort out the wheat from the chaff. Don’t rely on screening questions such as “are you the person responsible for buying metal cutting oils?” The fraudster who wants to complete a questionnaire for the incentive will answer “yes”. Consider putting your screening questions into a context which gives people a choice of different subjects. At least this approach presents someone with the option of choosing a subject other than cutting oils and so would get screened out. For example:
Which of the following products are you responsible for specifying?
- machine tools
- packaging materials
- cutting oils
- lubricating oils
-
Include some open-ended questions in panel questionnaires. It is surprising how often people catch themselves out by putting an answer to an open-ended question which identifies them as clueless. It has been known for some respondents to complete more than one questionnaire in an attempt to earn incentive points but their laziness catches them out as they copy and paste the same open-ended answers.
-
Watch for the flat liners in online surveys. People who give the same rating scores to a question may well be rushing through the questionnaire without consideration and replacements should be sought from the panel company.
-
Look at the time it takes people to complete an online interview. If it is less than half the time you consider to be a reasonable average, throw the interview out and obtain replacements.