Measuring social media ROI with experiments and modelling

Measuring Social Media ROI in 2023

Social media is now part of most marketing plans. Yet it is still hard to measure the return on investment (ROI) of what brands do there. Privacy rules now limit the use of cookies and other ways to track people. So marketers must get creative. They need to look past vanity metrics like likes and shares to find the real impact. The focus has moved to methods that test whether social activity really causes changes in business results. These need sound statistical designs.

Leading voices such as Les Binet stress the need for balance. Long-term brand building matters. So does short-term activation, the work that drives a quick response. Conversions are tempting to track. A conversion might be a sale, a lead or a sign-up. Attribution models try to share out the credit for each one. But these models have limits. So on their own, conversions are not a measure you can trust. The latest methods combine brand lift studies, incrementality testing and multivariate tests. Between them, they gauge the extra effect that social has. They also gauge how it shifts the way people see a brand. Correlations, where one thing tends to rise or fall as another changes, do not on their own show what social caused.

Financial ROI needs a return and a cost, not just a rise in likes or awareness. One way to work it out uses contribution: ROI = (incremental contribution before programme cost − programme cost) / programme cost. Here, contribution means the added revenue, minus what it costs to deliver and serve those sales. Programme cost covers the media, creative, tools, staff and other costs in the scope you are looking at. Count each cost once, and state the time period. Say the added revenue is $30,000. It costs $12,000 to deliver and serve those sales. That leaves $18,000 before a $10,000 programme cost. The net return is then $8,000, and ROI is 80%. These figures are made up. An ROI worked out this way is an estimate. It can only be as sure as the estimated effect on sales.

Causal Impact Analysis

Causal impact analysis is catching on as a way to measure social. Google’s CausalImpact package is one way to do it. It is not the source of all causal analysis. It estimates how much a change, or intervention, moved a measure that is tracked over time. On social, the change might be a new campaign, or posting more often. The measure is a business KPI, such as web traffic, leads or sales. The tool sets what you saw against a modelled counterfactual. That is an estimate of what would have happened with no change. It uses a Bayesian structural time-series model to work out the gap between the two. This kind of model learns the trends, seasons and links in data over time. It also draws on control series. These are other measures tracked over the same time. The change must not affect them. Their link with the result must also stay steady enough after the change. If these things do not hold, the gap need not be the campaign’s effect.

Its strengths include:

• It can allow for seasons and natural ups and downs, when the data and model support that. This makes it a stronger check than a simple before-and-after total.
• The synthetic control is built from the control series. It stands in for what would have happened with no change.
• It helps you keep improving. You can test each new change against the synthetic control as it evolves.
• It puts a number on the extra lift, with a posterior credible interval. In this Bayesian method, that is the range where the true effect likely lies, given the model and data.

To use it, you need enough past data to train the synthetic control. No fixed rule says that 50 weekly data points will do. The data must go back far enough to support the seasons and links being modelled. You can use it for measures like web traffic, coupon use and call volume. But the estimate rests on its assumptions and data. You cannot count on a positive result.

Incrementality Testing

Incrementality testing is another method that more people now use. It helps judge how much credit a channel deserves. It aims to find the extra impact of a channel by running an A/B test. The channel is turned on for one group, the treatment group. It is turned off for the other, the control group. Putting people into the groups at random helps split the channel’s effect from other ways the groups differ. Gaps can still come from chance. They can also come from spillover, where the treatment leaks into the other group. So the results should show how sure they are. You also need to check how the test ran.

On social, you can run tests with a holdout group. These can show how the channel does on sign-ups, purchases and taking part in surveys. The holdout group is the control group above. It is made up of the people held back from the channel.

Best practices for these tests include:

• Choose a holdout size with care. Weigh the cost to the business against the smallest effect worth finding. Also weigh how much the result varies, how long the test runs, and its statistical power. Power is the chance the test will find an effect of a given size, if one is there. Holding out 10% of spend is not enough by default.
• Run tests over several time spans and groups of users to check that the results hold.
• Compare results over a long enough time to catch effects that show up late.
• Use what you learn to make social programs more efficient in future.

Brand Studies

Social media plays a big part in building brands. That part is hard to capture with direct response measures alone. Well-designed brand lift studies can gauge how social activity changes the way people see a brand. The same goes for how they feel about it, and how drawn they are to it. Their strength depends on how the comparison groups and the survey sample are formed.

Multi-cell brand lift studies compare several groups. Here are some things they can measure:

• Brand awareness: They compare recall in the exposed and control groups. Recall can be aided (with a prompt) or unaided (without one).
• Brand favorability: They compare how each group rates the brand’s traits.
• Message association: They compare how often each group links slogans or campaigns to the brand.
• Purchase intent: They compare how likely people in each group say they are to buy.

Online providers have made these studies simpler to run. They can now pick samples and run surveys on the platforms themselves. These studies let marketers see how far social lifts key views of a brand. That helps them put a better number on its value.

Multivariate Testing

Multivariate testing is a controlled test that changes several factors at once, in planned mixes. It then uses statistics to estimate their effects on a result. That is not the same as looking for patterns, or correlations, across several variables in data you have just observed. On social, you can run such tests across content types, formats, calls to action and targeting settings. They can show which posts, audiences and strategies have the most impact.

On social, its strengths include:

• It finds interaction effects. That is when the effect of one factor depends on another.
• Models can help you compare content mixes for the audiences and conditions you tested.
• It helps you learn fast. Then you can keep adapting and improving your content plan.
• It can show interactions you would miss by changing one factor at a time. But the test design and sample size must support that.

MODEL is a handy mnemonic I use to design these tests:

M – Metrics: Pick the key business outcomes to measure.

O – Objectives: Predict which factors will shape the results.

D – Design: Make test posts that mix the factors in different ways.

E – Execute: Publish the test posts and measure the results.

L – Learn: Study the results to find which factors matter most.

Conclusion

To measure social media ROI, you need to move past simple conversion tracking. You need methods that can come close to showing cause and effect, and the extra value social brings. No approach is exact. But used side by side, the four methods above can give a fuller view of social’s impact. Working out financial ROI still needs the costs and returns, not just those measures on their own. Marketers should not try to credit social with every sale further down the line. They should focus on sound analytics that link what they do on social to real business results. Privacy rules now limit the old ways of tracking. So the future of measuring social lies in methods based on statistics. These can also help keep budget for a channel with real influence.