We All Have a Quant Problem: Recognizing It and Identifying a Vendor to Fix It
The analysis of quantitative data in its many forms was supposed to be the silver bullet that would help businesses identify and solve all manner of business problems. We were all convinced that Big Data would uncover new insights and solve the previously unknown issues, but even with these advanced tools the world is still a messy place that business still finds difficult to navigate. Marketing surveys still cannot accurately tell businesses what consumers want or need. Supply chains are still grossly inefficient. Even finding lasting love after extravagant analysis of people’s personal information is still leaving too many a broken heart.
It has become clear that in business research, we have a quant problem. We have many practitioners making bold promises and predictions that are not met. Skepticism is increasing and businesses are doubting the quality of the quantitative research that is presented to them. This is a problem.
There are two main reasons that have led us to where we are today. First, business research is the study of humans and human society doing business related activities. As such, the quantitative analysis of businesses and their interests should reflect the complex, adaptive and dynamic nature of human life. Unfortunately it does not and it will not for some time. Business research from its inception followed the paradigm of the natural sciences and engineering, which is inadequate for understanding humans. In an classic article, Horst Rittel and Melvin Webber famously noted:
“ … the classical paradigm of science and engineering – the paradigm that has underlain modern professionalism – is not applicable to the problems of open societal systems. … the cognitive and occupational styles of the professions – mimicking the cognitive style of science and the occupational style of engineering – have just not worked on a wide array of social problems. … We shall want to suggest that the social professions were misled somewhere along the line into assuming that they could be applied scientists – that they could solve problems in the ways that scientists can solve their sorts of problems. The error has been a serious one.”
In addition, numbers are not innate to humans. Behavioural scientists have found that humans suffer from representativeness bias, i.e. people use similarity or representativeness as a proxy for probabilistic thinking. This impacts our ability to understand and interpret numbers in several ways. We are insensitive to prior probabilities of outcomes. For example when we are told “Maria is very shy and withdrawn, invariably helpful, but with little interest in people, or in the world of reality. A meek and tidy soul, she has a need for order and structure and a passion for detail”, we are more likely to believe she is a librarian and not a teacher, when in reality she is more likely to be a teacher because there are far more teachers in the world than librarians. We are insensitive to sample size. People generally assess the probability of a result from a sample by asking how the properties of that sample are similar to the population from which it is drawn instead of considering its size. We misconceive chance and probability, believing that a sequence of events generated by a random process will represent the characteristics of a random process when the sequence is short. For example we suffer from the Gambler’s fallacy, where we believe that a series of heads in a series of coin flips should be followed by a tail. We misconceive regression to the mean. Outliers in data typically regress toward the mean in subsequent iterations, however we expect subsequent iterations to be representative of the previous iteration.
All these cognitive limitations mean that humans generally find it very challenging not only to interpret quantitative data, but also find it immensely challenging to design quantitative studies and to analyse the quantitative data they generate. Good quantitative research needs to be reflective of the complex, adaptive and dynamic nature of human life and human society. It also needs to account for, and where necessary correct for, the cognitive limitations humans have. Done well, quantitative research can still be a valuable source of insight in business research. Here is how to do it right:
It is the question, stupid!
The research question that businesses want answered should determine the type of analysis that should be used to answer that question. Many practitioners attempt to answer questions that quantitative analysis cannot answer. Quantitative analysis should be used to quantify how much and/or how many of a defined variable there is in a given population. Nothing more, nothing less. The methods and techniques used can be very complex but they must all be used in a way consistent with this basic fact of quantitative analysis.
Quantitative analysis becomes questionable when it is being used to help understand meaning or an experience. Quant analysis should not be used to understand how customers experience a new product, to understand people navigate the digital world, nor get a better picture of the experience of managing a workplace. Quant analysis should not be used to develop proposed explanations as a starting point for investigation when there is limited evidence.
To recognize good quant analysis, you should ask yourself: can a quantitative approach answer the question being asked? If it cannot, it should not be used.
How it worked there and then, may not work here and now
Human attitudes, beliefs and behaviours are influenced by the social, cultural and environmental context surrounding them. How consumers feel about the clothes they are wearing depends on what they are doing. Think about it, you will not be happy in your favourite gym clothes if you had them on in a business meeting that required formal suit and tie. It is therefore important that quant analysis understands the context the data being used was collected and that the analysis reflects this context. Data from different demographic groups, collected at different times, from different parts of the country should be analysed in a manner that reflects this complexity.
Let’s consider market segmentation, a widely used method of dividing a consumer or business market into smaller sub-groups based on a set of shared characteristics. It is argued that these sub-groups, or segments, will behave and respond in a similar manner to marketing and business strategies that target them. However, we know from behavioural science research that the attitudes, beliefs and behaviours of humans are profoundly influenced by the social, physical, and environmental context that surrounds them. How people feel about a particular type of clothing, food, car or innovation is dependent on the situation people are faced with. Our feelings and attitudes about a particular type of clothing, food, car or innovation will change depending on the context we are in. For example, we feel differently about snacking depending how full we are, the time of day, who we are with when we want to snack, and how much money we have. Segmentation of snacking needs to reflect this complexity. Market segmentation should be dynamic and adaptive.
Good business quant research should identify and reflect on the important contextual factors that influence humans and impact business performance.
Robustness
Good quant analysis is robust. That is, means analysis is statistically sound. It is not enough to simply present findings from data. Any patterns, differences and relationships that emerge from the data should be consistent and accurate. This means all relevant tests for reliability and validity should be run before conclusions are made. Too often simple cross tabulations of data are presented as research findings without any testing to ensure that any differences that are observed are statistically significant. Trends that emerge from the data are used to forecast future behaviours and are allowed to influence decision making. This is without testing to confirm that the slope of the trend line is significant nor determining whether the trend line is linear or quadratic.
Good Design is Everything!
The alternative to good quantitative research design is always bad quantitative bad research design--the latter being sadly more common. Unfortunately quantitative business research has been inundated with bad design which has led to bad findings. This has created a culture of expectations that is more comfortable with the poorly designed version. Often people see this as “good enough” or “fine for directional work.” Good quant research design should match the type of analysis needed for the research question that is being addressed. For example, if we want to generalize our findings to a population, we should have a sample size that is large enough.
If we want to measure the impact of an innovation, the design of the study should be longitudinal so that we can ascertain causality. For example, in a classic study, Robert Jensen surveyed people in the fishing industry in India weekly for four years to show that the introduction of the mobile phone increased information availability and reduced information asymmetry resulting in a much more efficient fish market with less waste and universal pricing.
Experimental Design
Many research questions facing business are best answered using experimental design. Experimental design is where participants are given or experience different conditions (or treatment) in a study. At its simplest, experimental design divides participants into a treatment group, where a treatment is imposed on a group of subjects, and a control group, where there is no treatment. The analysis of data from an experimental design focuses on comparing differences between these two groups. Experimental design allows business researchers to clearly understand the effect of something of interest and to estimate its effect size. Experiments are able to explain causation, that is, does doing X cause Y to happen, understanding the relationship between cause and effect. Experimental design is important for many businesses because it is important for them to understand whether interventions, such as advertising and product changes, have been successful and to know the effect size of the success caused by the intervention.
Tell the truth! Stick to the numbers.
Often there is a temptation to exaggerate what the data is telling us. As humans, we are susceptible to pressure from high expectations and the need to impress others. Unfortunately this often leads to people stretching findings from quantitative analysis far beyond what findings imply. This is particularly true for quant analysis as complicated analysis methods can make it difficult for most people to interpret findings. Unfortunately, many in our profession are happy to take advantage of this. This kind of shock and awe approach is all too common in strategy projects where consultancies cover up weaknesses in their research by amplifying the importance of their conclusions.
It is good to play with others
Humans are complex. Human society is dynamic and adaptive. The world of business is fast changing and difficult to predict. In such a reality, it is essential that quantitative researchers be able to be part of an interdisciplinary research team that includes a wide variety of skill sets. This should include, but not be limited to the disciplines of anthropology, psychology, sociology, language and linguistics, design, and strategic foresight.
Working in an interdisciplinary context allows good high-quality quantitative research in the following ways. First, researchers are able to identify the important contextual factors that influence and shape human attitudes, beliefs and behaviours. This leads to appropriate research questions and good research design. Second, researchers are able to use or design quantitative research methodologies that are informed by the different disciplines. Third, the analysis of data is much richer and insightful when done from an interdisciplinary perspective.
Conclusion
Just because something is broken does not mean that it has to remain broken. Quantitative business research can, must, improve. Good quant research will lead to more effective and impactful business insights and decisions. Good quant starts by putting the human at its core so that we can better shape the future.
QUESTIONS TO ASK
Can a quantitative research approach answer the question being asked?
What are the important contextual factors driving the phenomena we are interested in?
How does the quant method reflect the context?
Does the research design address/answer the research question?
How large a sample is needed?
Is the analysis robust?
How do these numbers confirm the story you’re telling me?
What are the complications in these conclusions?
Will you tell me the truth behind these numbers even if they tell a story I was hoping to avoid?
By Dr. Vurain Tabvuma
References
Jensen, R. (2007). The digital provide: Information (technology), market performance, and welfare in the South Indian fisheries sector. The quarterly journal of economics, 122(3), 879-924.
Kahneman, D. (2011). Thinking, fast and slow. Macmillan.
Rittel, H. W., & Webber, M. M. (1974). Wicked problems. Man-made Futures, 26(1), 272-280.