FAQ: How do I run a "pop up vs. none" A/B test?


Also called a “holdout audience”, this is a commonly requested feature from our avid A/B testers using Justuno.

Though the following explanation will give you a temporary workaround, we do plan on making this available in an upcoming overhaul of our A/B testing feature, which will not disappoint. Until then, here’s the workaround:

NOTE: because the “hidden promotion” cannot be engaged with and thus reported in Justuno, you must use reporting within Google Analytics to analyze whether or not the pop up provided a lift. Spoiler alert: it almost always does…

  1. Duplicate the desired control (or “A”) promotion - this will be your variant (or “B”) promotion
  2. Remove ALL design layers and opacity of the background layer.
  3. Move the background layer to the corner
  4. Shrink background layer size to something small, like 1px x 5px - the idea here is that the variant will show with the same rules as the control, but no one will see it, and it won’t affect the UX
  5. Set up your A/B test, and push live.