When Data Driven Decisions Harm More Than They Help

Data driven decision making is becoming the way of the world, but in some cases it does more harm than good. In a world with endless data, how do you sort the signal from the noise without getting lost in the process?

The term “data-driven” has become an increasingly popular way to describe any decision made by analyzing numbers and statistics. The rise of information technology has given us unprecedented access to volumes of data we never had before, which is beneficial in many ways – my career is built on data, so I’m definitely not advocating for a return to assumptions and gut feel decisions!

Where “Data Driven” Breaks Down

The issue arises when people lose sight of what matters, and instead of applying critical thinking to a situation they blindly follow the data with no thought to real world implications.

Let me describe a situation, see if this sounds familiar – your media agency has told you that they ran a test where the same ad was displayed but one version was red and one was blue. The blue version has a slightly higher click through rate (CTR) than the red version, so they say they’ll put more budget behind the blue version to “optimise” the campaign.

On first read through, that might make sense. But if you really think about it, there are far more questions here than answers – is the increase statistically significant, or is it a natural fluctuation in the results? How many total impressions & clicks are we talking about, what’s the expected uplift by spending more on the ‘winning’ ad?

This is a simple example that illustrates why data literacy is so important, because I’m sure we all hear some version of this conversation all the time. Without a basic understanding of data, how can anyone be expected to make well informed data driven decisions?

If You Torture the Data Enough, it will Confess to Anything.

So should data access and use be limited to specialists with a background in statistics?

Not at all – in fact, I believe the opposite is true, and a major initiative for my team is to democratise data access across our business. If you only understand stats but have no knowledge of what data means within a business, you’re not likely to be able to extract and action valuable insights.

Instead, businesses should implement some form of basic data literacy training for all employees. Just as IT security tests are standard practice for companies these days, so should data literacy training be part of your general business.

Good data literacy training doesn’t mean everyone needs to know or understand advanced mathematics, and in fact the main learning should be how to think more critically about data when it’s encountered on the job.

Equipping your people with the tools they need to ask the right questions about data, and fostering their innate curiosity to learn more, is the first step to becoming a truly data driven business, without making potentially expensive mistakes by getting lost in the noise.