My takedown of Wikipedia's “gimme gimme gimme” ad copy drew this response from subscriber Matt:
Funny you should mention wikipedia. I've actually sat in a presentation by their data scientists and they have probably done more a/b testing on that banner than 99% of websites….. and they make bank… millions.
Awesome—didn't know that.
That made me wonder if I was too harsh on Ol' Sad Eyes Jimmy.
To find out I jumped on teh googlz to see what I could find out about their testing.
Which lead me to this:
It's a HUGE wiki page (natch) with all the results from their testing from a few years back.
I took a look at a sampling of their ads, and here's what I discovered…
Yes, they did a BOATLOAD of testing.
They tested different pictures of Jimmy.
They tested using different Wikipedia authors to deliver the message.
They tested different background colors for the banner.
But you know what they didn't test?
THEY DIDN'T TEST DIFFERENT MESSAGES.
They're all variations on the same “guilt and obligation” theme:
“You get free info from us. It costs us money. Donate to us or you're a poor excuse for a human being.”
And so every single ad they tested SUCKED.
This highlights a really common fallacy with “conversion rate optimization” and “split testing”…
You can easily end up testing things that don't make a big difference.
As marketing pro Glenn Livingston says:
“Most people think they've tested 20 ads. They've really tested 1 ad 20 times.”
If Wikipedia was smart about this, they'd hire a handful of skilled direct response copywriters to come in and churn out 50 different ads…
Taking COMPLETELY different angles.
“Make 'em cry” ads.
“Make 'em laugh” ads.
As the copywriting saying goes, “Test shouts, not whispers.”
If you test 10 ads like this, 1 or 2 will beat the others by a factor of 10X.
Now, coming up with 10, 20 or 50 completely different angles on your product can be daunting.
I know my brain sometimes starts to scream after 2-3.
There's a great tool for this, though…
This tool makes it easy and automatic to come up with great new ways to pitch your product.
You can use it for headlines, emails, sales pages…
I use it when I'm stuck and need a fresh idea.
Details are here: