At Facebook, research permeates everything we do, from improving our products to using AI to help keep people safe on our platform. In addition to serving the people who use our apps and services to build community, our research also informs how big and small businesses can effectively run ads on our platforms. These marketing insights can be found on Facebook IQ, a resource designed to help marketers and agencies discover actionable insights on consumer behavior, marketing, and measurement.

To learn more about the research behind these marketing insights, we sat down with Lara Andrews, Research Manager at Facebook’s Creative Shop. The Creative Shop is an internal team of creative strategists, designers, writers, and data experts who collaborate with advertisers to help them run effective campaigns that push the boundaries of creativity across the Facebook family of apps and services, including creative platforms like AR and Oculus. The group provides clients with knowledge and insights about creative best practices to help deliver better business results.

Andrews recently co-authored “A framework for improving advertising creative using digital measurement,” a whitepaper that outlines how to measure creativity. In this Q&A, we ask her about the creative measurement methodologies described in this whitepaper, her role within the Creative Shop at Facebook, what makes a great ad, and more.

Q: What are you responsible for in your job?

Lara Andrews: My role is to help clients with research methods and data insights to understand what helps drive success on our platforms and to help creative decision makers operate creatively and efficiently at scale.

Q: What do you enjoy most about your job?

LA: I studied economics and psychology and have always been interested in behavioral economics and the underlying drivers of human behaviors.

My role is at the intersection of creativity and research. I really enjoy the process of starting with a wild, crazy idea that’s very exciting and working across the Creative Shop and our cross functional partners to operationalize that.

We can set up an experiment with good reliability. I figure out what we need to know and turn it into robust research questions. The process of trying different ways to operationalize variables is an iterative process. You don’t get it right the first time. There are many interpretations of a concept. As a team, we have a clear process that moves us along an explore phase, where we use data-driven methods to locate the right creative questions to ask, followed by an experimentation phase, where we run controlled experiments to isolate those variables. Then, we work with our Creative Shop team to activate those learnings and communicate them to the industry. This process allows us to be iterative and quickly abandon ideas that fail and double down on those that demonstrate success.

I also like that it’s always changing. The Facebook platform is evolving all the time. There are always new products, and those products also change. That dynamism is both challenging and exciting. For example, we are looking at whether interactive AR or VR ads result in different outcomes. Do people feel more ownership through an interactive experience with product versus watching a video about it?

Q: What’s different about advertising today on Facebook?

LA: In the past, awards were the only way to determine ‘what works’ in ads, but now we can augment that with so much more. With today’s digital platforms, there is data available to understand what the true business outcomes are.

Q: What’s unique about the Facebook platform for creative development?

LA: For testing ad concepts, we are not limited to running focus groups where people could be swayed by other participants or the moderator. We can test ideas in the context in which they will be brought to market. By using the platform where the ad will run to test it, we can come away with a more complete understanding of the effect it will have on an intended audience.

Q: What makes a great ad?

LA: For us, a “great” ad is one that deploys creativity to achieve the advertiser’s business objectives. There are some features we can measure that are key to achieving this. We do know that on mobile, simplicity is key. You have a limited amount of time to appeal to a consumer scrolling on their phone. You need to keep it simple and relevant. You need a tightly crafted idea, with just the right amount of message. We also know people respond better when humans are in an ad. It’s about understanding the audience, knowing what’s going on with them — but ultimately, in every “great” ad there is an element of creativity that is difficult to pin down and measure.

Q: How do you measure creativity?

LA: I just collaborated on a creative measurement white paper that examines historical approaches of measuring creativity and outlines our current creative measurement methodologies. In our literature reviews, everyone uses different methodologies with different types of outcomes and there is no sense of replication, so you can’t generalize outside of the studies.

We need to have guardrails about what is appropriate to test. A lot of creatives have intuitive opinions about what will work. We do not rule out new ideas but instead use data to test different ways of expressing that idea.

This is our three-stage framework for how to approach the problem of creative testing and measurement:

The first stage is to identify the desired business outcome, how well the ad drives sales or brand awareness. We then look at how much of the success can be attributed to the creative execution of the experience.

Second, we deconstruct creative into its component parts to reduce complexity. We examine specific features of how the creative concept is expressed — the creative “degrees of freedom.” These features can be mechanical (aspect ratio, video play time), visual (presence of people, animals, indoor/outdoor setting), or thematic (how the ad evokes an emotion from the viewer). Machine learning can be very helpful in determining what features will be most effective.

The third stage is experimentation where we take creative features and model out the creative outcomes. We can run tests where everything is controlled for except specific creative variables. The modeling gives us a more complete picture of independent features and which ones made the biggest impact. It’s important to apply advertising professional domain expertise at every stage of the research process to identify a creative hypothesis and interpret our findings. We then work to provide best practices to advertisers so that they can put these insights into practice.

Q: How do creative people respond to these methodologies?

LA: They really value having a cross-functional team to work with them to think through how to operationalize their ideas and keep them consistent. We spend days with the creative team mapping out the variables they can test. The industry is becoming comfortable with the methodologies as they try them out and start to see the results.

Q: What is most surprising about your research?

LA: Even working all day analyzing creative results, I’ve learned that we’re terrible at predicting what will resonate. I’m surprised by how much we are always surprised. What does work is letting people tell us what they like.

Q: Will this change creativity?

LA: This research capability is not going to change creativity. The magic that creatives bring will always be needed and in demand. What we have is access to the data and more complete research that gives advertisers more understanding of what works and why. That means fewer dollars wasted on ideas that don’t work out. Marketers have become familiar with using analytics and see the value of it.

—

To learn more about marketing insights on Facebook, visit Facebook IQ.