Measuring To The Beat Of Our Own Drum
*Award-winning Case Study
DRUM is used to measure immediate, incremental website visitation as a result of an offline media airing – radio or television.
Situation
As the country began reacting to the COVID-19 pandemic in March, many advertisers started pulling their media, which left a plethora of media inventory available. With more supply (available spots) than demand (brands wanting to advertise), prices decreased (shoutout to our high school economics teachers). By zigging when everyone else zagged, we were able to take advantage of these floor-level CPMs across multiple offline channels.
Objective
- Measuring offline media is just as important as measuring digital – every media dollar is held accountable at Net Conversion. We sought to develop our own offline measurement lift model to gauge the incremental lift of media that most advertisers would say is only measurable via media mix modeling.
Methodology
For the recent television and radio buys, we requested and received log files (exact timestamps of the airings) of every spot that occurred for each station and their respective markets. We married the log file with data from Google Analytics, containing every market’s website visitation down to the minute. We blocked out the noise from channels that would not have been directly impacted as a result of being exposed to a television or radio spot, like visitors from an email or banner ad, and only focused on brand acquisitions – new visitors from brand search or direct traffic.
As for determining our attribution window – how long after an airing we attributed traffic to each airing – we’ve found that the immediate lift in website traffic typically lasts around eight minutes. The example below shows the percentage lift in website visitation versus what was expected by the amount of minutes following an airing.
While the immediate lift in website traffic is clearly shown in the minutes following an airing, we also realize that there’s a halo effect and ad decay from brand awareness media like television and radio and that the total effect isn’t limited to eight minutes. The purpose is to gauge the ‘direct response’ of offline media to directionally compare the effectiveness of offline media campaigns.
CAUTION: NERDY THINGS AHEAD
To develop the offline measurement lift model, we used the ‘Prophet Model’ – a time series forecast model developed by Facebook’s Data Science team – to allow us to predict intraday website visitation based on cyclical patterns. Facebook’s Prophet model is not only very versatile with it’s tuning parameters, but also quite dynamic in its application purposes.
Although Prophet is mainly used to predict future values (“What are we expecting sales to be next week?”), we used it as a “Fill The Gap” forecasting model to estimate what would have occurred during a given time frame. In this case, the eight minute window following each airing. A common use case for the Fill The Gap model is when advertisers lose tracking on a website and need to backfill what website traffic would have occurred during the time of the outage.
By nulling the actual traffic volumes during the eight-minute window following every ad, the Prophet “fills the gap” by predicting what would have happened, which allows us to measure the incremental lift – actual visits minus predicted visits.
Results
For a nationwide television buy, we measured an incremental lift of +12% website visitation in the eight minutes following each airing. The highest lift for each of the five spots was within the first minute of the airing, and the lift gradually declined for every additional spot (the first spot had the highest lift, fifth spot had the lowest lift).
36.9
RSME
0.97
R2
Model Accuracy
RMSE: 36.9
R2: 0.97
97% of the variance in the predicted values can be explained by the actual values
Model’s accuracy based on the site visitation outside the eight-minute attribution window of each airing
Our model helps us measure media effectiveness, but thanks to the level of granularity, we were also able to determine best performing times and days to air, top performing markets, stations/networks and creative. We ain’t out here buying random spots; we compulsively need to know where every dollar of media spend is going, and more importantly, how it’s driving a lift in performance.
Since our model is not client or channel-specific, we can use it as a direct response model across any traditional media effort and ultimately turn insights into action.
