What Is Predictive Processing? & How It Explains 3 Visual Illusions

The nature of the human brain has puzzled philosophers and scientists for millennia. How does the squishy matter inside our skulls, made from fundamentally the same stuff as the rest of the universe, enable us to access the world outside our bodies? An emerging theory in cognitive science could have the answer. It’s called predictive processing – a radical inversion of our previous theories of cognition, perception, and action. And it might just be correct!

We start our story with a question. The brain is trapped inside the skull, and yet it must somehow come to discover what exists out there in the world. This is a serious problem because, as the philosopher and cognitive scientist Andy Clark writes, “all the system has direct access to is its own states” (2013: 183). So, the brain has to use its own states (the states of all its neurons, synapses, and neurotransmitters, etc.) to infer the stuff in the world that caused its current stream of sense data. That’s already a fairly substantial problem for the brain to overcome.

However, it’s even worse than this, because the sense data is underdetermined by its causes. That is, the same stuff in the world can result in many different streams of sense data (and vice versa; lots of different stuff in the world can result in similar streams of sense data). This means that the electrical signals that barrage our sense receptors (which, remember, are ultimately all the brain has access to in the first place) are ambiguous since there is no one-to-one relation between cause and effect. To make matters even worse, the world is complicated. The causes of the stream of sense data can be nested within each other and evolve over time in unexpected ways.

How, then, does the brain perceive the world so that it can respond to the environment and continue its survival, all from within the skull where there are no colors, sounds, or sensory properties of any kind? The solution comes in the form of a special trick, and it is a radical inversion of how philosophers and scientists have thought about perception.

The traditional, commonsense view of perception is one where stuff exists out there in the world, and our senses provide a transparent window onto it. Stuff in the world (light waves, sound waves, molecules, etc.) is detected by our sense receptors, which causes electrical signals to be sent upward to the brain for processing. There, the brain fills out the rough details, then the finer details, and so on, until we perceive the stuff in the world. We can call this the “bottom-up” view of perception.

An emerging theory — predictive processing — states that this traditional, commonsense view is backward. In fact, our perception of the world is determined by the opposite, “top-down” direction, in the form of the brain’s predictions about the stuff in the world. As Andy Clark writes: “It is that twist — the strategy of using top-down connections to try to generate, using world knowledge, a kind of virtual version of the sensory data… that lies at the heart of [predictive processing] approaches to perception” (Clark 2016: 25).

The brain constantly makes predictions about stuff in the world. These predictions (which are informed by prior experience) are sent down the brain’s cortex in a top-down direction. At the bottom of the cortex is the stream of sense data, which flows into the brain in the opposite, bottom-up direction. These are the prediction errors, the leftover bits of the stimulus for which the predictions failed to account.

In contrast to the bottom-up view we discussed earlier, all that needs to be sent up on the predictive processing approach are the prediction errors, as these are the signals that carry useful or newsworthy information. The brain can use these prediction errors to learn from its mistakes and adjust the predictions it will make in the future, and in the process,  minimizes the prediction errors. This makes for a remarkably efficient system and affords massive savings on neural bandwidth compared to the bottom-up view.

It’s important not to underestimate the importance of predictions. The basic principle that the brain’s predictions determine our perception of the world applies to all of our perceptions all of the time. This includes the other sense modalities and phenomena that we don’t usually think of as perceptions, such as emotions, the self, and even our tacit awareness of being alive. It also makes sense, given the brain’s biological structure, where there are four times as many top-down connections than bottom-up connections in some places.

This explains the common moniker that perception is a “controlled hallucination.” It’s a ‘hallucination’ because the brain creates it, and it’s controlled because the world corrects the errors when the brain makes mistakes. As the neuroscientist and philosopher Anil Seth writes: ‘You could even say that we’re all hallucinating all the time. It’s just that when we agree about our hallucinations, that’s what we call reality” (2021: 105). To clarify this point, we can look at three visual illusions where our visual system is tricked in ways that make the predictive nature of perception more apparent.

The Checker Shadow Illusion

Take a look at this classic illusion, the checker shadow illusion. Clearly, the squares marked ‘A’ and ‘B’ are different colors — ‘A’ is darker than ‘B’. However, this is not the case. In fact, both squares are the same color. What’s happened here is that the brain has lots of experience with objects in shadow, and it tends to be the case that objects in shadow appear darker. Even though ‘A’ and ‘B’ are both the same color and emit the same wavelengths of light into our sense receptors, we do not perceive them as the same color. The brain utilizes its prior experience of objects in shadow to generate the prediction that ‘B’ must, therefore, be lighter than ‘A’, and this is what we perceive.

Notice that this is the case even after reading the explanation above. The fact that ‘objects in shadow appear darker’ is so deeply entrenched in the brain’s experience that it overrides any conscious attempt to perceive the scene for what it really is.

This illusion alone should be enough to convince one that what we perceive is not determined by the stream of sense data. If so, we should perceive both squares as the same color. They both emit the same wavelength of light, after all. But no, there is clearly some top-down influence here, meaning we perceive ‘A’ as darker than ‘B’.

The brain is not concerned with what color the squares are in reality but rather what color it predicts they are based on prior experience. Anil Seth notes that this is not a failure of the visual system: “A useful visual system is not meant to be a light meter, of the sort used by photographers. The function of perception, at least to a first approximation, is to figure out the most likely causes of the sensory signals, not to deliver awareness of the sensory signals themselves — whatever that might mean” (2021: 108).

However, in some cases, new information can result in differences in our perceptual experience. Take this well-known illusion, the rabbit-duck illusion. The figure seems to be either a rabbit or a duck (also known as a “bistable” figure), but whichever one we perceive is determined whichever prediction the brain makes (also called “cognitive penetration”). This illusion works because the figure is ambiguous and could be either a rabbit or a duck. It’s extremely difficult to view it as both at the same time. But that extra information, the brain’s prediction that the figure is a rabbit, is enough to make us perceive a rabbit. This will remain true until the brain makes a different prediction that the figure is a duck, so we will perceive it as a duck.

Notice also that there are two figures above. Try to view one as a rabbit and one as a duck. It’s pretty tricky and usually takes a decent amount of concentration. However, now use this extra information: the rabbit is on the left, the duck is on the right, and they are both looking at each other.  Now, the brain has more information to inform its predictions, and hopefully (with a little bit of effort), both the rabbit and the duck can be seen simultaneously. Again, nothing has changed about the figures. Rather, the same figures are situated in a new context, which changes our perception of the figures.

The Dress

Let’s discuss one final famous case from the internet. If you were on the internet around this time, you would almost certainly know about “The Dress.” Someone posted a picture of a striped dress, and depending on who you asked, you were either told that the dress was “white and gold” or “blue and black” – two totally different color combinations. What could cause people to reliably come up with such different answers? Well, surprise, it seems to turn on the brain’s predictions. The photo itself is overexposed and lacks important context that would allow the brain to resolve ambiguities in the scene, so the brain’s predictions will lean more heavily on prior experience.

As it turns out, if one’s brain is more accustomed to yellow light (if, for example, one spends too much time indoors), then one is more likely to perceive the dress as blue and black. Conversely, if one is more accustomed to blue light (if one spends a lot of time outdoors), then one is more likely to perceive the dress as white and gold. As Andy Clark writes: “Our own actions and histories sculpt the onboard prediction machinery that in turn sculpts human awareness, right down to the level of what seem to us to be basic sensory experiences” (2023: 26).

Even the simple fact of whether we spend more time indoors or outdoors contributes to how our visual systems are calibrated, which clearly affects how we perceive the world. The unusual circumstances of The Dress make this very apparent, but there are undoubtedly innumerable ways in which this is true in normal and mundane cases as well. We just don’t notice them.

Anil Seth recounts a moment where the philosopher Ludwig Wittgenstein asked: “Why do people say that it was natural to think that the sun went round the Earth rather than that the Earth turned on its axis?” His conversation partner, the philosopher Elizabeth Anscombe, replied: “I suppose because it looked as if the sun went round the Earth.” In his usual evocative manner, Wittgenstein rebutted: “Well, what would it have looked like if it had looked as if the Earth turned on its axis?”

The point, of course, is that it would look exactly the same. As it turns out, the same could also be true of the brain. It does indeed look like perception is a process of bottom-up feature detection, as philosophers and scientists have assumed for millennia. But predictive processing offers a view that is precisely backward. Perception is a process of top-down prediction, and our conscious experience is a kind of controlled hallucination.

The predictive nature of perception results in some of the visual illusions we have seen in this article, but as the philosopher Jakob Hohwy writes: “illusory perceptions represent the cost of having fairly low-level, automatic mechanisms for dealing efficiently with noisy or ambiguous types of sensory input” (2013: 141). Our visual experiences of illusions are the brain’s best predictions about the causes of its stream of sense data, the same as any other perception. It’s just that in the case of illusions, there are often some weird and unusual conditions which make them possible, and which reveal how perception normally works.

This article has barely scratched the surface of predictive processing, and there is still much more we still need to learn about the brain. But perhaps predictive processing is the Copernican Revolution that cognitive science has been waiting for.

Bibliography

Clark, A. (2013). Whatever next? Predictive brains, situated agents, and the future of cognitive science. Behavioral and Brain Sciences, 36(3), 181-204.
Clark, A. (2016). Surfing Uncertainty: Prediction, Action, and the Embodied Mind. United
Kingdom: Oxford University Press.
Clark, A. (2023). The Experience Machine: How Our Minds Predict and Shape Reality. United Kingdom: Penguin Books Limited.
Hohwy, J. (2013). The Predictive Mind. United Kingdom: Oxford University Press.
Seth, A. (2021). Being You: A New Science of Consciousness. United Kingdom: Faber & Faber.