Surveying in the Modern Age
By: Jason Wheeler, PhD
Research Manager
By: Jean Denious, PhD
Chief Executive Officer
Spammers, scammers, and bots are similar concepts: someone who “spams” a survey by retaking it multiple times, “scams” a survey by providing false information, or (more rare) someone who operates an online script or “bot” that automatically searches for accessible surveys and spams and scams them.
In our blog post on Equitable Evaluation, we underscored the importance of compensating people for their time and effort spent participating in research, including surveys. To the detriment of all involved in research, however, incentives provided for online survey participation can draw the attention of those who would exploit them for profit. And they are persistent in their attempts! Spammers, scammers, and bots come out in droves when a survey is published that offers incentives.
These bad actors give rise to distinct types of “bad data” in survey research even using the most well-built survey instruments. Furthermore, incentives constitute a significant investment on the part of funders, and the loss of these resources due to bad actors can undermine efforts to ensure participants are compensated fully and equitably.
Online bad actors are constantly working to circumvent safeguards we put in place to protect access to online surveys, so survey design and implementation must therefore continually innovate to stay one step ahead. At OMNI, we apply rigorous procedures and protections to ensure resources are not wasted, that surveys reach the intended audience, and that our participants' experiences are captured accurately in the resulting data. Here are several lessons we’ve learned during our history of serving clients through survey research.
Get your survey to those, and only those, it is intended to reach. This starts with effective outreach and recruitment approaches that are both robust and targeted, so that the ‘net’ that is cast reaches far and wide enough, but isn’t so porous as to offer easy opportunity for bad actors.
Disseminate your survey in multiple forms. Research shows that surveys do best when sent via multiple modes, for example by email, social media, text, and even paper surveys! Usually, researchers should choose the mode(s) that is most familiar or used by their intended participants. But what happens when that mode is also the favorite of those seeking to exploit your survey? Staggering the use of particular modes to send your survey can help to reduce the exposure it gets to spammers, scammers, and bots; before posting a survey to open social media channels, first strive to send surveys through other more restricted modes whenever possible.
Establish and maintain trust with clients and participants. Collecting survey data anonymously is the highest form of confidentiality, but it also limits the ability to know if the participant is a bad actor or not. As evaluators, we must include building and maintaining trust in research design, scopes of work, and other client-evaluator processes from the beginning so that clients feel safe sharing their participants with us, and participants feel confident contributing to the study.
Stay up-to-date with tech solutions. The very technology that facilitates the exploitation of survey incentives can inform the measures used to prevent it. Web and text services can allow participants to access a survey only once while maintaining their anonymity. Additionally, analysis of metadata can help researchers identify survey-taking patterns (duplicate IP addresses, little time spent on a survey response) to flag when a survey has been “infiltrated” by bad actors. The survey team can then weed out the bad data while continuing to seek out genuine participants through other modes of dissemination.
Understanding the opinions, attitudes, and behaviors of individuals and communities through survey research is a dynamic process at its core, as are the persistent threats to the integrity of survey data from those who would seek to exploit our efforts. OMNI’s survey experts thrive being on the front lines to anticipate challenges and deploy solutions as survey modes continue to change and data science becomes more sophisticated. To learn more about OMNI’s commitment to survey data integrity, contact Jason Wheeler at jwheeler@omni.org.