A/B Test Results: Cheeky opt-out text vs "X" button


This is the sleeper of pop up tests…

Why? Simply put, optimizers hate to love this technique, and audiences don’t even know how it influences them. I was listening to a recent webinar with some well-known experts who detested pop ups with guilt-ridden opt out text (e.g. “no, I hate savings”), as opposed to the old vanilla “X” button. This got me thinking: will people opt out of a discount offer on principal, or do most feel “guilted” into engaging?

I reached out to Frankie’s Bikinis, a high-volume bikini brand who speaks well to a millennial audience. Good volume is key to running a high-confidence A/B test, and in this case, I thought this particular test fit well with their branding. Another factor that led me to ask them was that her “X” button just didn’t fit anywhere on their heart-shaped pop up… maybe you have a similar design issue? Have a look:

Here’s the control (“X” close button):

And here’s the variable (“No savings for me, thanks”)

The Results

Small potatoes, right? Almost unnoticeable? WRONG…

Over a period of 22 days, the cheeky text recorded a 15.51% improvement in engagements over the control. In terms of sales conversions attributed to these promotions, the variable improved upon the control by a whopping 217%. This may not work in every type of store, but the results are compelling enough to give it a shot on your own site.

To set up an A/B Test, follow these quick steps:

  1. Log in, and go to the A/B Tests tab in your dashboard
  2. Name the test, segment it by device, and pick a date range.
  3. Select a control (“A”) promotion, then select one or multiple variable promotions available.

Tips: if you have lower traffic, don’t waste time with the small stuff. Try making a few big changes to your promotions to gauge the improvements in overall conversion rates. Also, duplicate the control promotion to make the changes, as the rules need to be the same for both the control and the variable. Lastly, let the tests run for a few days before checking the results - data takes a little while to grow and become reliable.

Be sure to review some of our other A/B Tests Of Interest here!


Wow! No brainer. Let’s present which text performs best next, yeah?