There is a growing assumption in many consumer insights circles that surveys need to be designed to be as similar as possible to the environment we are asking about. If we are asking about shopping behaviour, then how can the ‘shopper mindset’ be recreated. If we are wanting to test a new product then how can we create an environment as close as possible to the one in which the decision will be made? This ‘ecological validity’, as it is called, has long been something that has been used for experimental work in psychology but is something relatively new to survey design. So just how should practitioners respond to this? And why is this now interest in this issue?
There are a couple of issues in this that need untangling. First, there is often an assumption that our habitual behaviours in low involvement activities, such as shopping, are ‘non-conscious’ and as such we cannot reliably ask for recall. When we are shopping we are not zombies – we are all capable of talking about our experience of low involvement activities. I can tell you about how I ride my bicycle as although it’s a fairly automatic behaviour, we are not zombie cyclists. Of course, some people are better at talking about this than others – which is why some people make better teachers than others. Market research practitioners have long been experts at the careful crafting of surveys to help people explore and report what they do in these less reflective moments, intuitively identifying and challenging the limits of what people can self-report. Indeed, much qualitative research does this and we use conjoint and multiple regression for just this reason. Indeed, a good survey designer knows how to tease out low involvement issues to bring them into awareness/consciousness.
The second, related notion is that there is little point in asking people about their past experience and why they did something as they will not be able to offer an accurate account of why they chose certain products etc. In which case, it is necessary to recreate the environment as closely as possible and observe/ask them in that context.
To examine this assumption properly we need to understand what we are trying to get from a survey. There are often two objectives:
- We want to identify and measure mental states such as attitudes, behavioural intentions, thoughts, mental images, recollections, emotional states or reports of behaviour
- We want to understand the way in which different aspects of their experience shaped their behaviour (e.g. in shopper study was it the price promotion or the store layout that ‘drove’ their behaviour?)
To get to the first objective, then explicit surveys are an entirely legitimate tool. We are perfectly able to self-report in this way. A useful paper by Prof. John Cacioppo et al [i] pointed out that these are all measures that don’t ask people to self-report on their cognitive processes. For example, we would not (or at least should not) ask people to rate the impact of an advert on whether they purchased a product. But we can ask perfectly reasonably ask people to report on things about which they are aware (“What are your thoughts on this product?”). Of course, some cognitive and behaviourist psychologists consider that we have no internal lives but surely there is an abundance of evidence from our everyday experience that would challenge this notion.
To respond to the second objective then we will need some kind of experimental design and ecological validity to reproduce the shopper environment. This is because although we are good at reporting on our mental states, we are not so good at identifying the relative importance of different influencers on determining our attitudes and behaviour. This sort of work will typically involve an experimental design to assess the impact of different environmental cues on behavioural outcomes.
These two objectives are often confused, so it is essential that prior to assuming that ‘ecological validity’ is needed, first check that the purpose of the survey requires this.
Related to this, it is worth flagging that a classic mistake is to confuse reasons and causes. Human behaviour is very clearly produced by reasons. As Julian Baggini points out [ii], you can’t properly understand why someone has done something unless you understand the reasons for it. For example, if I turn on the light, the cause of it coming on is the movement of my hand, which flicked a switch and turned on an electrical current. But the reason for the movement is that I wanted to find my book. Similarly, the cause of an uplift in sales of cheese sandwiches may be due to advertising raising awareness. But the reasons for me buying a cheese sandwich on this occasion may be that I have greater awareness they are available but also there are many things I like about them that means the advertising is effective. Reasons and causes both explain why things happen, but in very different ways.
The implicit assumptions in the requirement for ecological validity are that we need to find causes. This assumes a model of consumers as being ‘driven’ by ‘causes’ in the environment (in a sense a Lab-Rat model). But much of our behaviour is also shaped by reasons – albeit often satisfied by quite habitual behaviour. So I could say that I habitually buy Coca-Cola. If I want to look for ‘causes’ then I may look at different aspects of the shopper experience to see which of these influenced my purchase on that occasion (or indeed whether I chose Pepsi instead). But if I do not understand my ‘reasons’ for buying Coca-Cola then the brand will not know how to properly promote it (e.g. alongside other foods/style of promotion). And indeed, as we have seen with the sugar issue, our reasons for choosing what we drink may change, with a subsequent dramatic impact on sales.
So, we always need to challenge ourselves to explain what we are trying to get to. It is very easy to make the mistake of talking about tools before thinking about the underlying objectives. If, for example, we want to identify the causal impact of different aspects of the shopper experience on behaviour then we need ecological validity alongside an experimental design. But if we want to understand the consumer experience and the reasons for shopper behaviours then we need some very familiar market research survey work.
By Colin Strong
[i] John T. Cacioppo, Stephanie Cacioppo & Richard E. Petty (2018) The neuroscience of persuasion: A review with an emphasis on issues and opportunities, Social Neuroscience, 13:2, 129-172, DOI: 10.1080/17470919.2016.1273851
[ii] Julian Baggini (2016) Freedom Regained: The Possibility of Free Will. Granta Books
Factaplus Press 