ICOM | Data Creativity

Download Presentation Slides as PDF

Why is this an outstanding example of Data Creativity?

To develop the methodology behind Creative Performance Indexing, we applied modern predictive analytic techniques to the oldest of marketing channels and cracked a problem that had remained unsolvable for over 50 years.

Problem Definition

  1. How can we ensure in advance if a DRTV spot will outperform its predecessor?Our legacy Champion/Challenger approach pitted two pieces of creative or “spots” head-to-head in a subset of test markets where both ran with equal rotation for a period of five weeks.While sound in theory, remnant inventory is purchased for DRTV to maximize return on investment. However, a deeper analysis showed that confounding factors, like fluctuating media costs, different rotation weights and daypart mixes were having a profound effect on the outcome of the tests.
  2. What is driving performance, is it the creative (spot) or the offer?The same spot may be updated and put into market multiple times with different offers, claims and proof points. Each variation of the spot could have widely different results. This made it difficult to optimize across different spots running simultaneously because it was difficult to understand what was driving the changes in performance.Through conventional wisdom and analysis we have learned that a spot generates better results when the core strategy is centered around the offer rather than when the spot is a generic creative shell to plug and play various offers. The majority of our work was produced this way, so when the offer changes, there was generally a need for a new piece of creative, but this led to situations where the two creative concepts were being tested against each other with different . How can we understand how much of the change in performance is due to the change in creative and how much is due to the change in offer?

Initiative objectives

In order to truly understand the impact of the creative, we needed a way to isolate and control all other confounding factors.  If we could identify the top performers, we could learn what elements made them successful and replicate that success either by updating existing spots with new offers or in developing future concepts.

Description of Impact

One of our client’s products was under heavy attack by a competitor in Chicago, a market where DRTV investment consistently underperformed. A Champion creative had been running in that market for three years after 14 Challengers failed to unseat it.

Using the Creative Performance Index, we identified a legacy creative, which had been considered a mediocre performer at the time. We updated the legacy creative with the new offer and ran it head to head against the Champion. The legacy spot outperformed the long reigning champion in every week of the test, lowering the Cost per Call by up to 24% in a single week and 12% overall during the test period.

Key takeaways

  1. Only in-market testing will give you truth. The go-to approach for creative evaluation is copy testing, however eight out of 10 rounds failed to select a spot that could beat the Champion in an in-market test. Only through using data based on observed customer responses, were we able to successfully select winning concepts.
  2. Many factors conspire to distort the truth, and if you don’t control for them, they will lead you to the wrong answer. We applied this methodology retroactively to concluded tests and found many cases in which comparing only topline numbers had led us to the wrong conclusion, even when those tests concluded with high levels of statistical significance.
  3. Media and Creative cannot be analyzed separately. Effective work was shelved because of irregularities in delivery.
  4. In working to isolate the creative performance by controlling for other factors we ended up building an approach that was also highly predictive of in-market performance and allowed us to break down the week-over-week changes into their component factors. Now, rather than saying that Cost Per Call moved from X to Y, we can quantify how much of the change was due to each factor, giving a previously unknown amount of visibility into other opportunities for optimization.

Please list & briefly describe the data sources & ecosystem partners used.

Client Data, Media Agency Data, Elastic Search, SAS

What processes were involved?

Data was assembled from there primary sources:

  1. Response data was collected from client first party systems.
  2. Execution data was collected from the partner media agency.
  3. Enrichment data was collected from client first party systems third party vendors.

This data was combined using Elastic Search to produce the analytical dataset, which contained data for every spot, on every day, in everyday part, in every market for every station over a period of three years

The statistical process that was used was a flavor of Mixed Model modified to accommodate time-series data. All other specification is proprietary.

The output is an indexed rank order of every creative analyzed with the highest performing creative having a score of 100 and the lowest performing creative will always have a score of 0. When comparing creative, a “good” CPI is any score that is higher than the one it is being compared to.

What was the inspiration for the solution? Why was the solution different, unique or creative?

Many approaches and methodologies used in today’s digital marketing have their roots in Direct Response, where the stimulus to the consumer and the response were directly observable.  While still widely used, these descriptive analytical techniques are being replaced by more sophisticated, predictive analytical techniques. The focus of most of the innovation in this field is related Digital Platforms and Addressable TV, but few have pulled what is new back into what is old.

Results

While originally developed for DRTV, we have since evolved this methodology to apply to other channels and other business needs. For example, the effect of Out-of-Home creative customized to particular Airport on bookings locations and Spanish Language TV strategy variation on retail transactions.