Is this really the most 'creatively effective' print ad ever?

A press release landed in my inbox this morning with the boldest claim I've seen in a long time. Cadbury's latest Heroes campaign has apparently achieved the highest ever score for outdoor advertising (5.9 stars) on System1's "creative effectiveness platform".

Well, I thought, this sounds like big news for the advertising sector. But looking at the ads themselves, something didn't quite feel right about this story. Is this really one of the best print ads of all time?

Don't get me wrong, the billboards themselves are perfectly fine. Created by VCCP, the basic concept is: chocolate bars renamed after pop culture icons. Twirl becomes the rock band Queen, Fudge becomes Robin Hood. And it's all delivered in a way that's smart, simple and unmistakably Cadbury.

It's the kind of work that makes you smile at a bus stop, and hats off to VCCP for a job well done. But I have to be honest, none of it feels deserving of a "best ever" score. And those words "creative effectiveness" kept nagging at me. Because what do they actually mean in this context?

Here's what I've managed to piece together.

The methodology

As far as I can tell, System1's methodology involves showing people ads and measuring their emotional response. Happy, sad, somewhere in between. These scores gets turned into star ratings between 1 and 5.9, which apparently "predict the short- and long-term commercial potential of ads".

And there's the rub. Potential. Not actual growth. Not sales. Not market share. Just potential.

System1 will tell you they've validated this approach. They point to analysis showing higher-scoring ads correlate with better business outcomes. However, in this LinkedIn comment Jon Schultz, managing director of VML, calculated that—based on figures System1 has actually shared—their model predicts market share growth with 48% accuracy. That's marginally worse than a coin flip.

So I'd argue that what we're really saying is that these ads were most likely to generate an instantly positive response in viewers; no more, no less.

That's not nothing, of course. In fact, it's an achievement to be loudly applauded. But that's a very different thing from actually being effective. You know, the boring stuff like selling more chocolate or increasing Cadbury's market share.

Happy feelings

Don't get me wrong, I understand that being able to say "this scored 5.9 stars; the highest ever" gives people a nice, happy feeling about what they've created. And we could all do with some nice, happy feelings right now. I also think the service System1 offers is valid and valuable. Insights gleaned from this kind of research could well end up being a game-changer for brands.

But as a professional wordsmith, I am cursed with pedantry. And unless I'm missing something, there's a universe of variables between someone putting a smile on someone's face and them actually buying the product.

Media weight. Competitive context. Pricing. Distribution. Existing brand love. Whether it's raining. Whether your competitor just launched something better. Whether a Taylor Swift launch just sucked all the oxygen out of the room.

So I'm not arguing that System1 is failing to predict creative effectiveness. I'm saying that in a chaotic world, it's essentially impossible to measure an ad's effectiveness until after the event.

In other words, the ultimate test isn't happening on System1's platform; it's happening in supermarkets right now. And we won't know the answer until someone analyzes the sales data and works out whether all those happy faces translated into tubs of chocolates leaving shelves.

Given the cost of chocolate these days, that'll be a challlenge. But either way, it'll be fascinating to see what happens.

And if sales do rocket, and this actually turns out to be the most effective ad campaign ever, I'll be forced to eat my words. Along with a tubfull of Twirls, Fudges and Double Deckers, of course.