Wheel … of … Fortune!

A friend of mine was telling me about an A/B test that he ran with his email list that produced really weird results.

I happened to ask him what criteria he used to create his segments.

Turns out, he's using ConvertKit, and there aren't a lot of good options for this kind of thing.

So he used first name field to create the segments using a “Wheel of Fortune” aproach, like this:

“First Name contains N, or O, or P”

When I heard that, my eyebrows shot up.

In college I was a political science major, and we were required to take a course in survey research methodology.

One of the cardinal rules in research is that any time you divide up a group of people for an experiment, the assignments MUST be random.

Otherwise you're likely to introduce some unexpected bias.

ZIP code, area code, time zone, name, social security number, birthdate, age, height, gender—anything like that can have some counterintuitive impact on your results.

I don't know if that was the case for my friend.

His story did get me curious though, and I decided to see if there were any patterns like this in one of the email lists I manage.

In this particular list, I have purchase data associated with each email address, so I can see who is spending the most money.

I dumped the whole list into Excel, and started filtering based on the first letter of the email addresses.

Now this is a different query then my friend used—I wasn't interested in replicating his experiment exactly, just seeing whether segmenting based on letters was as problematic as I suspected.

Turns out there are MAJOR patterns, at least in this customer base.

I looked at groups of customers whose email addresses start with 9 different letters.

3 of the groups were on par with the overall list average in terms of dollars spent per subscriber.

3 of the groups were MUCH lower than average.

1 of the groups was MUCH higher than average.

The difference between the lowest and highest dollar value group was 275%.

Moral of the story:

It's not hard to unintentionally skew test results, so do whatever you can to make sure the groups are randomly assigned.