It remains a fact of business life today: after the Great Recession, two and even three jobs were combined into one. This occurred just as Big Data (and small) was becoming part of everyoneís jobówhether or not they had any data experience.
This isnít about CRM data, which is a huge but totally separate issue. This is about the qualitative and quantitative data thatís critical to every stage of product and campaign lifecycles. Itís statistical science, the complexity and importance of which are frequently minimized in the rush to market.
Cheap DIY survey websites created by non-research developers have been used for years by managers in an effort to have some insights, instead of none. Sounds good, right? Not so much, when the likely (but often overlooked) result of this approach is:
Many B2B marketers and consumer product managers who lack requisite background are given the task of gathering data. Thatís ok: when you give smart people smart tools they can usually figure it out and get the job done properly.
What tends to go wrong with a blind spot in data chops isnít typically about the individual; itís about the tool theyíre using. Inaccurate data from poorly designed DIY surveys is a titanic problem in business, and itís largely undiagnosed
Thankfully this is going away, as the latest self-service market research tools are debuting. Theyíre faster, cheaper, more accurate, and they actually help the user create valid surveys by automating intricate data science.
Whatís Wrong With My DIY Survey Tool?
This can be answered with a question: who built it? From the late 1990s to the mid 2000s as web browsers, servers, and bandwidth made big advances, entrepreneurs jumped in with DIY survey tools. Ironically, almost none of these players were market research firms. They were mostly Internet startups with IPO dreams. They took a database programming approach, a simplistic UI/UX, and did their thing.
Whatís wrong with that? If you need quality data, quite a bit.
While the old DIY tools have certainly improved, most of these companies didn't have any expertise in market research to begin with. They approached it as a database solution, not a data solution. Huge difference. Accordingly, the old guard DIY survey tools were never able to perform the kind of concept rotations, quota assignments, randomization of responses, questions, screens, or groups of questions & screens, and other complex processing required to collect quality data. No disrespectóthe older tools helped a lot of people over the years to move things along and shed some light. They just didnít do it very well.
Hereís an example:
You have two concepts that you want to measure (basic A/B testing); you want to present an ad concept as a simple graphic, then on the next screen ask Purchase Intent and Attribute Communication. So you have two concepts, two screens containing three questions for each concept. What you need to do is group the concept A screens together, and group the concept B screens together, then rotate/randomize which concept group is evaluated first and second.
To add one more level of complexity, letís say you want to balance customers and non-customers across each concept so you have a readable sample of each group for each concept.
With many older DIY survey tools, youíre out of luck. It canít be done. In the more advanced tools you may have the basic randomization or grouping ability, but it is locked by an Upgrade button to the most expensive service levels, and still falls short of the control you need.
Moving forward without proper survey flow control leads to something called ďposition biasĒ which registers more favorable ratings on the first concept than the second. This fundamental flaw in the older tools can present across many different types of research design.
New self-service survey tools allow you to easily group concept questions and randomize or rotate how they're presented. First concept first for some people; second concept first for other people. Itís how you avoid position bias. Database solutions donít correct for this, because theyíre generally not built on statistical science. New self-service tools give you this advanced control at the free service level because that is quality research.
Here you can see an example of this functionality in use:
Go With the Flow
It's hard to build a system that stops somebody from making mistakes. For example, no automated system can know in a sentient way the qualitative content of the questions youíre writing for your survey. Thatís why newer survey tools have actual data scientists available to answer questions. This alone is a serious enhancement.
Another strength of newer tools is managing the flow of survey creation from the perspective of a market researcher. It avoids logic flaws that can also induce bias. Yes, the older tools can perform skip patterns and randomization. But they don't necessarily allow you to group questions or group screens together so you can really manage extremely complex rotations. Thatís a key difference between data and database.
If your eyes glaze over at the mere mention of randomization or rotation against target quota cells, donít beat yourself up. That happens to everyone except market researchers. Their eyes twinkle when someone says ďcomplex research design.Ē
Which brings us back to the question of ďwho built itĒ as it relates to DIY surveys and outmoded tools. If you need one hundred people to view a concept in a certain way, and then that one hundred people needs to be balanced by gender, age, ethnicity, and other variables, you need a data solutionónot a database tool.
The intelligence of professional market research systems developed over years has now been put into a usable form. New survey tools are capable of executing complex research design; balancing randomization and rotations for concepts across your sample; and performing core functions like branching and piping. It sounds like jargon, but itís actually an industry sea change: powerful research systems becoming available to non-professional users in free versions marks a vast improvement over older DIY systems simply upgrading technical capabilities.
Add to this responsive design that allows you to build a perfect survey on any mobile device, and the new breakthroughs come into focus. Itís amazing: twenty years of advanced research methodologies are now utterly native to mobile.
Hereís a parting thought: when you crank out that next infographic, better to make it with a tool created by actual researchers. Otherwise, itís just you alone out there with a fancy database thingy, hoping for the best. That sucks.
At least you have options.