When looking at averages, it’s easy to miss things that might otherwise stand out. Real insight lies deeper: you have to dig to the bottom to find valid improvement points.
Just like in World War II, when the Allied forces had to come up with a solution as their bomber planes were constantly being shot down. Armor-plating the whole plane would have made it too heavy, so extra plating could only be put in the most crucial areas. A mathematician, named Abraham Wald, was asked to look into this problem. First, he created a diagram that outlined the plane and marked where the bullet holes were usually found on the planes that had returned.
The pink dots in the image above mark these spots. Based on this, you might say that armoring these areas would protect the bomber. On a second thought, when looking at these averages, people were missing the real weakness. Wald argued that instead of plating the areas where the planes had been shot, the plane needed reinforcement where it didn’t have bullet holes. Why? What people have forgotten is that the planes returning hadn’t been shot down. Just as in business: it’s easy to make assumptions with generalised data, but to understand the problem you have to delve deeper.
Weaknesses can't be found in averages
When looking at analytics, our dashboards are filled with averages: average bounce rate, average time on site, average conversion rate.
The problem is, averages hide variations and anomalies. One event can skew the average and your perception of the information. Analysing data is just like looking for bulletholes: by digging deeper, you may find areas that are not performing as well as you would have expected, but these won't jump out at you. They have to be uncovered.
What people have forgotten is that the planes returning hadn’t been shot down. Just as in business: it’s easy to make assumptions with generalized data, but to understand the problem you have to delve deeper.
How to identify weaknesses in your own website
- Compare paid traffic to organic traffic: are there any differences? You may be paying for traffic that isn't engaging with your content...
- Where are your visitors dropping out? Identify these points to stop losing potential new customers.
- Break your conversion rate down by device to identify weaknesses on mobile or tablet.
- For A/B tests with no significance: segment your data into mobile and non-mobile to see if trends begin to appear.