Identifying and weeding out dishonest survey respondents

Here we will outline the types of rogue respondents you may come across, how you can identify them and some practical solutions for dealing with them and thereby cleaning up your data.

Types of dishonest respondents

Firstly the bane of many researchers lives – professional competition entrants. They will scour the internet, magazines or other media for competitions they can enter which will include any surveys with an attractive incentive. Even if they don’t know anything on the subject matter of the survey they will tick some random answers in order to enter the prize draw. Worse still, they may fill out the survey several times in order to give themselves a better chance of winning the prize.

Another common group of dishonest respondents are survey speeders – as the name suggests, they speed through the survey as fast as they can. In some cases, they may start off with good intentions and give honest answers, but for a variety of reasons they lose interest or become careless with their responses further into the survey. This is more likely to happen with longer surveys where survey fatigue kicks in. These types of respondents are also known as straight-liners i.e. they click the same answer for each option in a matrix question, therefore drawing a straight line. Other variations include drawing diagonal lines or a pretty pattern. You may also find that survey speeders will skip all the questions which are not set as compulsory.

If you are running a poll or awards survey where you are asking respondents to vote, you may come across block voting. This is where you have respondents or organisations who have a vested interest in a particular outcome and will vote for this option over and over again and may also get their friends, family or colleagues to do the same, thereby rigging the results. If they are technically minded, they may even set up an internet bot to automatically enter your survey repeatedly to ensure their preferred option comes out on top.

How do I identify dishonest respondents?

Within the Demographix analysis toolset you have an option to view individual responses. This allows you to look through all of a person’s answers and see whether they make sense and they haven’t been straight-lining or skipping lots of questions. If the survey includes open ended questions, it is worth having a read of their answers to check whether they are responding in a reasonable or expected way, or indeed leaving all the write-ins blank.

For each individual response, you will also be able to see the time it has taken them to complete the survey. Have a look at all respondents to see what the average time taken is. If certain respondents have completed the survey in a much shorter time, they may well be survey speeders.

You may also want to check for incongruent responses, for example if they say their last holiday was in the UK and later on you ask them what sites they have seen on their last holiday and they tick ‘Empire State building’, you know something is not right.

If from the outset you suspect that your survey may attract a number of rogue respondents, you can build in some trap questions into your survey. Examples you could use include ‘please choose the third response below’ or ‘please type the word hello here’ which will tell you if people are actually reading the question. To make it less obvious you can hide a trap question in a matrix as the example below shows:

What can I do to weed out dishonest respondents?

Be careful with the incentive you offer. If the incentive is widely appealing, such as cash or vouchers, this may attract more rogue respondents. The higher the prize value, the more likely you are to attract professional compers. If your survey is aimed at a specialist sector, think about whether you can offer an incentive which will appeal specifically to them rather than the wider population.

If you have identified dishonest respondents by viewing their individual responses and/or looking at the time taken to fill out the survey, you can flag their ID numbers to us and we can remove them from your survey or group them in a way so that you can filter them out. Note: you cannot delete responses yourself but we can do this on your behalf.

If you have added some trap questions into your survey, you can route respondents who don’t answer them correctly out of the survey at that point. Alternatively you can identify them afterwards and we can delete their response.

Where block voting is suspected, we can look at IP addresses of respondents and check whether there are multiple submissions from the same IP address. We are able to remove any duplicates to clean up the data.

To weed out straight liners who may be suffering from survey fatigue, consider shortening the questionnaire or use more routing where possible so that respondents only answer the questions relevant to them. This is particularly relevant now that an increasing number of people are using mobile devices to respond to surveys, meaning their attention span is likely to be even shorter. Our own research shows that mobile responses accounted for 46 per cent of all responses in the last year and are likely to overtake desktop responses in the next six months.

If you still find that the survey takes a long time to complete, you can give respondents the option to use the ‘Save & Resume’ feature, available on all Demographix surveys. This allows respondents to complete a survey in more than one session, by saving the answers already given and allowing the respondent to continue where they left off at another time.

Drop-out rates also increase with survey length. Having fewer responses will make your data less statistically robust so do give the issue of survey length serious consideration. We recommend that surveys should generally be no longer than 10-15 minutes (although there are exceptions for certain topics/audiences). Any longer than that and the quality of response is likely to be compromised, if indeed respondents even get to the end of the survey and submit their answers.

A well-structured and clear survey design will give you better chances of attracting honest responses from the first question to the last. Where you have questions with a long list of options, say in a multiple choice or matrix question, consider rotating the answer options. This should eliminate any bias to the top answers or fall out for lower rows due to respondent fatigue or survey-speeding. Also think carefully about which questions to make compulsory. If you have a survey full of compulsory questions, you may find that you turn genuine respondents into survey speeders as they get increasingly frustrated with being forced to answer every question.

Don’t feel the need to put up with dishonest respondents as this will result in diluted data. In the worst case, you may end up making important decisions based on bad data. Spending some time on the methods above can help you clean up your data to make it more reliable. Of course no method is foolproof and some rogue responses may still slip through the net but the steps above will help eliminate the worst offenders.

This article is sponsored content by Demographix.