Have surveys been left high and dry as research tools?

Surveys have come in for a hard time over recent years. The pollsters generally failed to pick the Trump, Brexit and now Australia’s own ScoMo victory*.

Similar techniques to those used by pollsters are employed daily by business. Tools like SurveyMonkey have become a low-cost means of quickly gaining information from customers and other stakeholders on everything from satisfaction to their opinions on ethics.

The key word in this was ‘information’ and I studiously avoided the term ‘insights’. Many still confuse the two an they are miles apart in terms of their usefulness and application to business.

The question in the light of the brutal exposure of the failings of polling in recent years is whether surveys are of any use in contemporary research and what, if any, are the alternatives given the tightening of screws around privacy protection?

I believe that they can still play a useful role, as long as you don’t rely exclusively upon them as research tools. Surveys become more effective as you add other sources and layers of information. In ancient times, we used to simply think in the context of whether we should do qualitative research, usually focus groups, to help direct the focus and design of surveys in order to quantify the findings from those groups.

I don’t know how many times, largely due to cost considerations, the decision was to just do the survey. With zero input from customers or others, this inevitably led to survey design with inbuilt biases and omissions, increasing susceptibility to error and flawed analysis. Even with regression analysis, surveys do not pick up the nuances in conversations or response that focus groups or one-to-one ethnographic type interviewing can. So even before errant polling predictions, surveys where the victim of poor or incomplete methodology and/or budgets.

Inevitably, results like the recent Australian federal election lead to reports, always with the benefit of hindsight, of the much greater insights and accuracy derived from big data. One Australian business claimed it foresaw the result through analysis of Facebook and Twitter conversations by postcode, but was nowhere to be seen with its prediction before the result was known.

Cambridge Analytica’s role in delivering the Trump presidency was a case study on the power of big data and high-end analytics to predict outcomes. It was helped significantly by data shared by Facebook. But the controversy that erupted around its methods raised serious questions about constraints or lack of them on privacy. The European General Date Protection Regulation (GDPR) regime is likely to become typical of the sort of regulatory intervention that will take place globally in response to this.

Many businesses have little need for stealth to capture a broad customer data set. In financial services, for example, data includes date of birth, tax and social security identifiers, and a mix of income, family and even daily (including real-time) expenditure, most of which is willingly provided by the customer.

Surveys can still be valid and useful adjuncts to the insights gleaned from data capture and analytics and more immersive and collective processes like human-centred design (HCD), which has industrialised several research and implementation practices of the past. Surveys still have a unique capacity to reach customers at scale and quantify the insights gleaned from micro-groups.

Some of their shortcomings derivce from how and when they’re deployed. I have serious misgivings, for example, about the value of the traditional, catch-all annual survey.

For industries with high-frequency customer interaction, the experience of each interaction becomes blurred over time. For sectors in which there is infrequent interaction, there will be a high proportion of responses that are indeterminate due to the time lag between the customer’s most recent interaction and the survey.

The most effective surveys are either in the moment, for example an online pop-up immediately following an interaction or transaction, or within a few days of these. The ensures the memory of the interaction and, perhaps more importantly, how customers felt about the interaction is still top of mind.

Often referred to as pulse surveying, capturing these in the moment insights has been better enabled by technology, removing the hurdle to quick and accurate aggregation of information for analysis. The operational advantage of lengthy annual, bi-annual or quarterly surveys with their often mind-numbing array of questions is gone.

Surveying at its best is a dynamic, automated process, linked to real-time analytics and platforms that can rapidly optimise and instruct the activities of customer-facing and operations teams to deliver better customer outcomes.

Once in place, this type of surveying constantly enriches your view of customer. It is much more likely to support insights that are operationalised than the annual snapshot in time which, in my experience, tends to end up as a nice glossy report that drops into the bottom drawer never to be looked at again.

Reaching the nirvana of data capture, evolving your analytics and modelling capabilities and setting up automated workflows is a considerable and sometimes arduous journey. It will take several years for larger organisations and require employees to fundamentally change the way they do business.

The point is that surveying does have a role to play, despite the recent smack to its reputation. It’s just that we need to think about where surveying fits into the architecture of our research and analytics and, ultimately, the manner and purpose of its deployment.

  • For the uninitiated international readers, ScoMo is the handle for the Australian Prime Minister, Scott Morrison. The 2022 federal election is likely to pit ScoMo against newly minted opposition leader, Albo (short for Anthony Albanese). It’s just an Australian thing.