Why you must experiment
Kinase founder Richard Brooks on why experiments and modelling are now essential for taking digital marketing forwards
An accurate view of marketing performance is critical for a business. Without good information, a business cannot make informed decisions on where to invest budgets, and how to optimise individual channels. The result is that a large proportion of those budgets goes to waste. And with greater automation of digital bidding and buying, good data is now more valuable than ever.
With UK companies investing more than £23bn in digital marketing annually, the value of good data cannot be overstated. Yet despite the staggering media spends at stake, businesses frequently underinvest in understanding, and fail to improve the metrics that underpin it all.
There are two common errors:
The wrong KPI is being used in budget and optimisation decisions
The KPI is correct, but the measurement of it is wrong
The good news is that neither one is that hard to fix if approached in the right way.
Aligning KPIs to Objective
There is a simple test for good alignment. State as succinctly as possible the business objective(s) and then compare them to the KPIs being used in decision making and optimisation. If the two are pulling in different directions, or produce a contradiction, then ask one powerful question: “What KPI would better reflect these objectives?”
Some of the more obvious examples would be:
Finding the ‘perfect KPI’ can be hard (if not impossible), but finding one that better reflects your objectives is often much easier.
Accurate Measurement
It is easy to define what you would like to measure. Actually measuring it can be a little harder. There may be internal systems that require development to join together and output what you require. In some cases (e.g. long sales processes, lifetime value), modelling is involved. The data may not yet even be collected or digitised. It can be a long, but ultimately valuable process, where every incremental improvement will yield a financial benefit.
When linking 1st party data back to digital marketing spend, there is an additional problem. I’ll put it succinctly: ALL ATTRIBUTION MODELS ARE WRONG.
This is meant with no disrespect. There are many good attribution tools, and the data they provide is essential for day-to-day reporting and optimisations. But that does not detract from the fact that even the best ones are fundamentally unable to provide entirely accurate data. They can never actually read people’s minds as they consider a purchase and see ads.
All attribution tools have gaps in the data points they can model. There has always been a gap when it comes to offline interactions (offline ads, shop visits, word of mouth, in store purchases…). This is now compounded by ever expanding holes in online data as cookies become ever less reliable.
With large holes in the data even the most sophisticated attribution modelling cannot provide the whole picture and will overstate the value of digital activities nearest the bottom of the funnel (though they are still far better than relying on Last Click).
The Solution
The problem is that the data in online attribution tools is wrong, but it’s the best data you have for making budget and bidding decisions.
The solution is to add experimental data to the strategy, then use the results to calibrate the more real time data sources (i.e. by updating the attribution model and knowing which biases will still need correcting for).
A good experimental approach will meet these five criteria:
It will answer a specific question. E.g. “What is the incremental impact of investing in X on multichannel revenue?”
It will be statistically robust, utilising a genuine geographic AB split and clear confidence levels.
It will minimise business disruption (the lowest possible impact in terms of duration and spend/sales impact).
It will provide a data point that can be directly compared across channels. E.g. Incremental £Sales can be evaluated across online and offline investments, removing silos in budgeting. (Actually this can go further – what generates a better ROI? Is it a) store refurbishment, or b) YouTube, or c) a Free Delivery offer).
The result will be acted upon objectively.
Having a trusted understanding of the incrementality of digital marketing spend or the true extent of digitally driven store sales has had a transformative effect on businesses Kinase have worked with. More nuanced tests can also be extremely valuable at a more tactical level.
The good news is that with the tools now available, running a good experiment has never been easier or cheaper. And with the loss of cookies and uncertain economic conditions ahead, the learnings they can generate have never been more valuable.
