So why’d Iran cease all air patrols on Friday?”
The airwing intelligence commander muttered the question without looking up from his clipboard.
I had just wrapped up my morning intelligence briefing. It was my first as a junior intelligence officer on board the USS Carl Vinson aircraft carrier.
We were steaming off the coast of Iran. Inquiring minds wanted to know why Iran hadn’t flown any military aircraft that Friday.
I moved towards the map on the wall. “Well sir, every day this week we have been patrolling here, right off their coast. And they have been flying intercepts. But on Friday we moved farther out to sea, so we are obviously less of a threat. They kept their planes on the ground.”
Wrong. And he made me aware of that in no uncertain terms…
“Listen, Lieutenant, there’s no ‘obviously’ about it. Friday’s holy day; they don’t fly. I suggest you hang around here and observe for a while before you toss out opinions, is that clear?”
Ass chewed. I had confused causality with correlation. Rookie mistake.
And you know what? As online marketers we do the same thing every day:
These hot-takes all display the same rookie tendencies I made twenty years ago on board that ship. They trumpet the results of a marketing campaign without having investigated what would have happened anyway.
They screw up the causality.
How do you prevent this mistake? You rock control groups like a boss.
That’s the subject of this ultimate guide.
I know, I know, start the “NERDS!” chant … “control groups,” right? Who needs them?
Well, you do.
I am here to tell you that control groups can make you a lot of money, fast.
As in money that you are throwing out the window today. By showing you where to cut unnecessary spend, control groups make you profits overnight.
Read on, Nerd.
Customers do stuff.
Sometimes that stuff includes buying things from you.
Whatever the reason, you may have had nothing to do with it.
Think about that for a second. This creates a problem — how do you know if your marketing is working?
But the last thing you want to do is to spend up on a campaign if you don’t have to.
Think of Adwords branded ad campaigns. Always super-high ROI. And your paid agency is oh-so-quick to take credit for that ROI, right?
But, ahem, do you really need to pay for those super-high ROI campaigns when potential customers are searching for your brand by name in the first place?
There’s no universal yes or no answer here. But control groups will help you to find the answer for your company—and save you money in the process.
How much? Dunno for you, but I can say that at Karmaloop we were spending six-figures on branded search each month. That’s a lot of moola.
A control group is a group of people that we carve out of a promotion or campaign. Not with a knife. Meaning we don’t send or show it to them.
We send it to another group: the test group.
We then compare the behavior of the control group and test group. We want to know the lift in profits, if any, of the test group over the control group.
That lift is all that matters. That’s what determines your ROI.
Note that this is a special subset of A/B testing.
In a true A/B test, there’s an A variant and a B variant. Some customers see A and some see B.
In control group testing, the B variant—the control group—sees nothing at all. This is the only way to determine the lift that our campaigns produce over what customers would have done anyway.
There are at least five killer benefits to using control groups.
News flash: Most online businesses doing six-figures and up spend heaps of money trying to get customers to buy when they would have bought anyway.
Identifying and eliminating that spend is the primary benefit of using control groups.
Consider this scenario. My wife, Sara, decides to buy a new sweater at Club Monaco. She knows the exact sweater—the model number, size, price, everything.
She loads up clubmonaco.com on her iPhone …
… And before she can even add the item to her shopping cart, she’s hit with a 20% off popup.
Sara snatches the 20% off. Club Monaco Marketing is pumped because they see her convert. They calculate the ROI. Because that ROI is positive (even after the 20% discount), they call it a win and keep running the promo.
But that calculated ROI number is too high. It takes credit for converting Sara when she was about to pull the trigger at full price!
In reality, the ROI is lower, maybe even negative, because Club Monaco didn’t have to spend all those promotional dollars to get some of those conversions.
This hidden cost is called a subsidy cost, and I’ve written about it before. Reducing it is the quickest way to add profits to your bottom line.
Enter control group testing.
To test, Club Monaco would first carve out a control group. Again, this is not actually by using a sharp implement. The control group would not see the 20% off popup. If Club Monaco watched that control group, they’d see that there are women out there who come to the site, credit card in hand, and proceed to buy at full price.
Is the size of this “will pay full price” group large enough to offset the inevitable decrease in overall conversions? Testing would tell.
Armed with knowledge observed from this test, maybe Club Monaco ditches the offer. Maybe the company changes it up. Maybe it switches to an exit intent popup.
Don’t know, don’t care. All I know for certain is that with more accurate measurement, Club Monaco will pocket more dollars.
Promotions can change a customer’s behavior even after the promotion has ended.
Control groups help to measure that (delayed) change.
Win-back campaigns are a great example of this. Most of you send win-back promotions to customers who haven’t bought in a while, right?
You know the logic. These customers are defecting. If you wait too long to lure them back, it will be too late.
You gotta give these defecting customers a reason to buy again, so you send out some coupons and offers that expire within a week.
Because of this win-back campaign, some customers do indeed return. They take the bait. At the end of the week, you measure their activity. You slap an ROI on the campaign and call it a win.
But you know what? That promotion isn’t over! Far from it. Some of the customers you’ve won back will stick around for weeks if not months after your coupons expire.
These defecting customers are now back in the fold. Some will even buy again and again at full margin later on.
This is called a halo effect. And it’s glorious.
This promotion and its glorious halo effect just juiced up your customer lifetime value.
But if you had stopped measuring your ROI at the end of that first week when the coupons expired, you’d have missed the halo effect. Indeed, your initial ROI on that win-back would have been low.
With a low ROI number, you might look at your ROI spreadsheet and dismiss the win-back in favor of a slick new Facebook Bot campaign.
You’d reallocate your time and effort and dollars away from the win-back when you should be doing just the opposite.
Control groups reveal the truth. By holding a control group out of your win-back campaign, you would see three things.
You’d then include this long-term boost in your ROI calculation. Your final ROI would be much, much higher than your initial projection.
And with a more accurate number you’d then say, hey, win-backs kill it for us.
You’d want to lock them in, optimize the offer, and generate more profits.
How the hell do you measure things like your speaking on a panel at a conference?
What about that money party you held at Sweet and Vicious with the local vodka brand?
Where are those “brand-y” things in GA?
They aren’t there—but you should still try to measure them. They are costing you something, after all.
And if you want to keep throwing parties and appearing on stage you should justify these things.
Control groups can show you the impact of these non-direct-response events.
Here’s how. When you send your local customers an invite to your next shindig, you leave a random sample off the invite list. You then watch to see how the invitees (test group) and non-invitees (control group) perform (i.e., buy, click, engage) over the next few weeks.
You calculate the lift. Voila, you can now slap an ROI on “brand stuff.” Now you know if the hipster parties are worth it.
And this ain’t just for ecommerce brands, BTW. SAAS readers: Constant Contact (5 million customers and growing) attributes their success to speaking at local chamber of commerce meetings.
Wonder how they figured that that was working?
Case studies like this one make my head spin.
Online mattress company Casper is emptying the kitchen sink on social media.
After running this gauntlet of Instagram fake photos and content and in-store events and Facebook contests, some customers do indeed buy … but how the hell does the Casper CMO Michael Behrens know which specific tactics worked??
Is he just going to write the social media team a blank check and tell them to knock themselves out?
Is he going to pay Analytics Pros five figures to try to jam all that social craziness into a decent attribution model in Google Analytics?
(Don’t answer—he may actually do both, because Casper has VC cash. But you shouldn’t, because you don’t.)
No! He’s going to run control groups wherever he can!
By removing a random sample of his customers from this social media activity, he skirts the attribution issue. All he has to do is measure the lift of the customers who ran the gauntlet over those who did not.
There’s his ROI.
Armed with this ROI, he now can tell his board that Casper needs to go all Kardashian on social. And he’s then much more likely to raise another $170 million to pour into Instagram.
Attribution is tough. Simplify it with control groups.
All these benefits are about one thing: getting a more accurate ROI number on your marketing spend.
If it isn’t clear so far, I’ll say it again. The more accurate your measurements, the more money you will make.
This isn’t an intellectual exercise.
It’s not work for the sake of work or geeking out on spreadsheets.
No, this IS your job as a data-driven marketer.
You are the Warren Buffett of your company. The head allocator of your marketing dollars.
You make investments and back horses in the race.
The more you move your and your team’s attention, time, and money away from low-ROI activity to high-ROI activity, the more cash flow you will generate.
Control groups (and testing more broadly) are your number one tool for beating the market.
Heck, this is why the former CEO of Harrah’s casinos, Gary Loveman, has stated that there are only three things that can get you immediately fired at Harrah’s:
Sold on control groups yet?
If you’ve read this far, I’m assuming the answer is “yes.” So let’s dive into the nuts and bolts of setting up these tests.
As you probably suspect, you can’t just log into MailChimp and start pounding on the big “Control Group” button.
You can still, however, configure some basic tests fairly easily.
In the second half of this post, I’m going to walk you through how you might set up a very basic control group test using email software.
My goal is to make you money, today.
Spend an afternoon working through these five steps. You’ll end up with a campaign that generates profits forever.
You can run control group tests on almost any marketing campaign. But here are some things to consider:
Consider two campaigns:
I’d want to start testing with the latter.
The more promotions and discounts in your campaign, the better — for now.
Why? Because there’s a higher potential subsidy cost living in that campaign. Which means you may be giving customers an unnecessary discount.
Testing will show you if this is the case.
Consider two campaigns:
Opt for the ongoing campaign.
The more repeatable the campaign, the better.
Why? Because you’ll get more mileage out of your test.
Finally, I’d start my testing with email campaigns. Email makes it easy (I’ll explain why), and I want you to get a quick win.
For most of you, the perfect place to start testing is with your email win-back campaigns.
They are perfect for control group testing.
If you have a win-back campaign running, then use that for this example.
If you don’t have a win-back campaign running, create a simple one now. Here is a rule for a basic “one-size fits all” win-back:
Basic win-back: If your customer last purchased 60-120 days ago, send her a 20% off coupon to come back and buy.
Easy enough, right?
BTW, nerds refer to this customer segment as an “R60-120.” “R” stands for recency (days since last purchase) of 60-120 days.
As we are all nerds here, let’s roll with R60-120.
Action step: To start control group testing, consider your own win-back campaigns. If you don’t have any, use the “R60-120 / 20%” example I gave you above. Also, get ready to make some quick cash. Decide what you are going to blow it on.
Now that we have our campaign in mind, we need to create our two groups:
There are a couple rules for creating these test and control groups.
First, your control group should be about 10% of the total group of eligible customers.
The last thing you want to do is have your revenue go to zero because you dropped 90% of your customers out of a promotion.
That’s too much.
Dropping only 10% allows you to run a decent test without (potentially) shooting yourself in the foot.
In our win-back example, that would mean that I’d first pull a segment of all customers who are eligible for the win-back:
That 10% can go up or down depending on how big your initial customer set is.
The smaller your total segment of customers, the larger that control group percentage needs to be. So if you only have only 100 customers, you need a control group of more than 10%—20% or 30%—to be confident in your results.
Conversely, if you have millions of customers in your segment, your control group can be smaller. Maybe 5%.
So start with 10% and work up or down as you see fit.
(And yes, I realize that statistics PhDs out there are ready to get all up in my nuts on this rule of thumb. But I’m a marketer first. Aim for production, not perfection. Let’s do this and not lose anyone in the process.)
Second, your control group must be random. Don’t skew your results!
What would skew the results? Well, choosing your control group on the basis of non-random stuff like:
No sorting on these things! In fact no sorting!
If you were to generate a control group like this, that group is guaranteed to be better or worse performing than the test group. Your control group test will go right out the window.
No, we need your control group to be completely random.
Fortunately, there’s an easy way to do this in your email software …
How to generate a random sample in your email software
There’s no “random sample” button in Mailchimp or Klaviyo.
But most email software does offer a down-and-dirty work-around.
When I’m passing the time generating control group segments in my email software (“NERD!”), I like to create the segment off of the presence of a letter in the email address.
Most email software allows you to build segments off of text matching. I find that if I build a segment wherein the email address does not contain the letter “a,” for example, that that often runs about 5-10% of my total emails.
If “email address does not contain the letter ‘a’” is your control group, then “everything else” is your test group.
Good enough for most lists.
Here’s how you’d do this in Mailchimp:
And here’s how you’d do this in Klaviyo:
Got it? Good.
Action step: Create a win-back control and test group in your email software. Play around with the text matching to ensure that the sample generated is about 10% of your total list size. And like I said, the results should be random.
Now the fun part!
Send your campaign to your test group. Don’t send it to your control group.
(Fun, right? Hey, it’s a slow day here at Nerd Marketing HQ.)
I’m talking about a one-off send here. Although most email service providers are capable of automation, let’s hold off on that for now. We want to see what’s working before we lock it in.
For the R60-120 / 20% win-back example, our campaign would be a simple “Come back and save 20%” email. The email should contain a coupon for 20% off that expires in a few days.
In Mailchimp, you would want to send this email to a Saved Segment, which is your Test Group:
In Klaviyo, you would target your campaign like this:
Important: Once you send that single email campaign, do two things:
Action step: Send your own win-back offer to your test group. Note the date and export the lists of test and control group customers.
Now the not-as-fun part!
You want to measure the difference in profitability (if any) of the test group over the control group. This will tell you whether the campaign was worth running.
Some tips on how to measure this profitability:
You want to measure profits here, not revenue.
Why? To keep ramming this point home, it’s because you used discounts in your campaign. You therefore need to see whether that discount margin was worth giving away as a promotion.
Moreover, you want to measure profits per customer. Obviously, your test group is much bigger than your control group. So the test group will have more total profits.
To compare apples to apples, you need the per-customer metrics.
Remember: The expiration date on your coupons is not when the promotion ended.
You probably have a glorious halo effect. You should wait longer—90 days is a good rule of thumb—to measure how your campaign performed.
Why 90 days? Because when you send a coupon, you alter customer behavior for a while. Think about it:
Pre-win-back, Mary was out of here. Gone. Post-win-back she’s ordered not once but twice. And the second time was two weeks after the promotion ended. And it was at full margin.
This is your glorious halo effect. You won’t have 100% Marys, but you’ll have some. You have to attribute them back to the original win-back, even if the coupons expired weeks ago.
So wait 90 days before counting up your profits per customer. And when you do, don’t count coupon redemptions. Count total profits per customer on all orders from the control and test groups during that 90 days.
It would be nice if your email software did all this measurement on the fly, right?
Profits per customer per segment group over 90 days?
Unfortunately, I have only seen this kind of sophistication in super-primo email software.
That said, it’s easy enough to run these numbers in a spreadsheet. And much cheaper too. I’ll walk you through it here using what I call the Convoluted Spreadsheet Method.
(OK, you actually can start the “Nerds!” chant now.)
Drew’s Convoluted Spreadsheet Method for measuring test performance
You can read through here and if you get stuck watch my video tutorial:
Piece of cake, right?
If you live in Google Sheets, it actually should be straightforward.
But if you’ve never done LOOKUPs or pivot tables before, it might take you some time. To help you, watch the video below. That might help.
Or, better yet, hire someone cheap on Upwork to do this for you. All this cranking in Google Sheets looks time-consuming, but it should take a spreadsheet nerd 10-15 minutes.
Value your time and outsource.
Profits per customer per group tells you which won: the control or test group.
Want to get fancy? To calculate the true ROI (if any) of the test campaign, take the lift of the test over the control. Plug that into the second tab of my Marketing ROI spreadsheet.
Remember: Don’t get greedy. Your ROI is only the lift of the test versus the control. It’s not the absolute ROI from within the test group.
Action step: Measure the results of your campaign using the Convoluted Spreadsheet Method. Or hire someone to do it for you.
At this point, you know whether your win-back campaign makes you money.
You know whether you have been spending too much too soon to keep customers around.
For most of you, I’d guess that the answer is yes, win-backs are worth it. The profits per customer of the win-back test group typically beat that of the control group.
If this is the case for your business, go ahead and lock in that campaign. Set it up to run automatically using your email software. Extend it to Facebook using custom audiences. Mail it out via Touchcard.co if you feel up to it.
You can be certain that the campaign will generate solid profits per customer for you for every customer you run through your funnel going forward.
If, however, your control group beat your test group, or if you want to keep testing different offer levels or timing, take a step back and rethink things.
Action step: If you have a winner, lock it in. If not, run the test again with different promotions or timing.
Sold on control group testing?
Here’s where you might go next…
If you have enough customers, you can get more sophisticated with your tests.
At Design Public, I was always combining control groups with my A/B and A/B/C tests.
So in the case of a win-back, I ran something like this:
Once I found the ideal offer, I’d play around with timing while holding the offer constant:
With enough testing, I could tell which offers and timing were best for my business.
This would allow me to build a hyper-dialed-in discount ladder that maximized profits.
Isn’t that cool? I mean, it doesn’t get any better than this with database marketing. After years of testing you could literally have a customer retention machine just cranking for you while you sleep, throwing off maximum profits left and right.
If you are starting out, however, keep it simple. Hold variables constant. Limit yourself to one control group and one test group per test until you get the hang of it.
One reason I recommend starting with email is that your email software can serve as your one-source CRM.
Create groups and segments there first. Then push them to other channels.
Out of the box, these other platforms and channels aren’t always set up for proper control group testing. But if you start with email, you can extend that testing to other platforms making your life easier.
Win-backs are a great place to start, but think about where else you might be bleeding cash.
For most online brands, I’d start with these:
Not all of these are as easy to set up, but they can be done. Weigh in on the comments on other examples you might have.
I started this email on an aircraft carrier. I’ll end it in the Department of Defense.
Former Defense Secretary Robert McNamara once said, “Measure what is important, don’t make important what you can measure.”
Think about that. Don’t blindly pursue certain campaigns just because the numbers are readily available to you. (I’m looking at you, branded paid ads.)
What’s really important? The things that will generate profits.
And you can’t figure out what those things are until you make control groups your best friend.
Use ‘em today. Watch my video tutorial. Work through the action steps in this post.
The more you use these, the more you know about your business. And the more you start to build a highly-customized marketing machine that generates profits.
Now—over to you. If you are using control groups in your current campaigns, can you comment below and let me know how? What’s working and what isn’t?
Got a comment? Email me at drew [at] nerdmarketing.com!