Situation

The Nassau Paradise Island Promotion Board (NPI) is responsible for promoting and driving bookings to member hotel properties in Nassau Paradise Island, Bahamas, including Atlantis and Baha Mar. In 2023, NPI faced challenges due to two significant external factors: shifts in consumer travel behavior post-pandemic and the economic impact of inflation on travel demand. Despite these challenges, their destination’s goal remained consistent: to effectively use the most effective mix of audience targeting, geographical focus, media channel mix, and creative strategies to drive demand and qualified travelers to choose Nassau Paradise Island for their next vacation.

In response to these challenges, as an analytics-first agency and NPI’s media partner, Net Conversion decided to reassess their audience-centric creative strategy. The objective was to ensure we were effectively targeting the most qualified audiences with compelling and relevant creative messages, considering the evolving consumer behaviors. Studies indicate that campaigns featuring high-quality creative can achieve up to a 35% increase in effectiveness, and creative imagery and messaging continues to grow as an influencing factor for consumer decision making. Amidst a changing advertising landscape and economic uncertainty, we have recognized the critical importance of optimizing campaigns, particularly in terms of creative, for our partners and the broader marketing industry, leading us to create our machine learning creative model for creative scoring purposes.

---

Objective

---

Knowing there are various factors impacting performance, it’s difficult to fairly measure creative performance in an ‘apples to apples’ manner.. Through advanced modeling, we can isolate the creative to understand it’s true contribution to performance. Our objective was to develop a decision tree-based model that controls for the other contributing factors, thus isolating the impact from the creative asset as it relates to user engagement (clicks). Each creative theme is scored based on it’s proportional impact to incremental clicks. From there, our objective was simple:

• Identify which creative has the best impact score that could lift website visitation and performance

---

Results

---

Of the five creatives that make up 99% of YTD spend, our team was able to identify that the ‘Better in the Bahamas” creative drives the highest lift in website clicks, followed by ‘Family – Flamingo Crossing”.

From both a modeled and non-modeling approach, “Toe Tapper” and “Airline” underperform in driving clicks to the website. There is opportunity to rotate in more of the creative that has a strong impact score with minimal spend, ie: “Couples – Staniel Cay”.

NPI’s new way to measure creative is still being tested, however by shifting creative assets we expect to boost creative efficiency and website visitation by more than +15%, using the visit rate average and current volume each creative is driving website traffic.

Implementation

---

​​

To analyze creative impact on website visitation, our advanced analytics team created the Machine Learning Creative Model Score; a tool that helps businesses evaluate the effectiveness of their creative content. The score is calculated based on a number of factors, including the date, tactic, targeting type, creative version, spend, and clicks. The score is then used to make decisions about which creative content to use in the future.

The data requirements for the Machine Learning Creative Model Score are as follows:

• Date: The date of the creative content.
• Tactic: The type of creative content, such as a banner ad or a video ad.
• Targeting Type: The targeting method used for the creative content, such as demographics or interests.
• Creative Version: The theme of the creative image.
• Spend: The amount of money spent on the creative content.
• Clicks: The number of clicks on the creative content.

The Machine Learning Creative Model Score is computed through a boosted tree model, a machine learning model that utilizes a series of decision trees for predictions. These decision trees are trained on a dataset containing creative content to forecast the performance of new creative materials. The boosted tree model assesses the significance of each field in the dataset and quantifies their impact on predicting creative content performance. Fields with higher feature importance wield more influence on creative content performance.

After creating the boosted tree model, the Machine Learning Creative Model Score is determined by processing the dataset through the model, assessing each record’s contribution to changes in clicks compared to the baseline. The baseline represents the average click rate for all creative content in the dataset, and the contribution to changes in clicks gauges how much each record affects the click rate in comparison to the baseline. The Machine Learning Creative Model Score assigns a numerical value between -1 and 1, with higher scores indicating the likelihood of creative content being more effective. This score aids in decision-making regarding the selection of creative content for future use, enabling businesses to choose content with a higher likelihood of effectiveness based on the score and other external factors, not solely on creative alone.

For NPI, we used the top five creative that makes up 99% of YTD spend. By scoring the top five creative for our partner, we were able to identify creative efficiencies and optimize campaigns based on which images had a positive impact on click performance and how they compared to each other to drive campaign performance. Additionally, our new model allowed us to identify ads where new creative should be tested.