A/B Testing

Using A/B tests in Blueshift, you can see which variation of a template performs better. You can test several templates that may have different forms of personalization or entirely different content.

You can set up the A/B test from the A/B Test tab of the trigger. The type of A/B test available depends on the type of channel selected for the trigger.

Tip

During A/B testing, if you want to compare the winning variation from a previous A/B test against new variations, we recommend that you first archive the winning variation and then clone it. Compare the cloned variation against the new variations in your new A/B test. This ensures that the new A/B test is isolated from your previous A/B test and that the results from your previous A/B test do not skew the results of your new A/B test.

Types of A/B tests

A/B Test	Triggers	Description
Entire Creative	All	Test the entire template. If your templates are similar to each other, you can use the clone functionality after creating one template and edit the cloned template.
Subject Line	Only Email	Test different subject lines for the same content.
Preheader Text	Only Email	Test different preheader text for the same content.
Sender	Email and SMS	For emails, test the From name, From address, and Reply to address fields. For SMS, test different Sender numbers/Codes for the same content.

Set up an A/B test

To set up a A/B test, complete the following steps:

Select a Template from the Channel tab of the trigger.
Click Create A/B Test or go to the A/B Test tab to set up the test.
On the A/B Test tab, select the Type as required. Depending on the channel for the trigger, you can select one of the following options:
- Entire Creative
- Subject Line
- Preheader
- Sender
Click Add Variation, add a variant, and the percentage of users to allocate to it.
- The first variant is your control and cannot be edited.
- You can add up to 20 variants to test.
- For an email Subject Line A/B test, you can use the same markup to customize the subject as you would for the email template.

Tip

If you want to ensure that the winning variant of an A/B test is truly representative of the test population, we recommend using A/B Testing instead of using Automatic Winner Selection. When you set up an A/B test on a percentage of your customers, you can use Blueshift's holdout feature or you can use the bsft_control_bucket attribute to select the holdout group. After you have run the campaign and waited for a significant period of time (for example, a day), you can determine the winning template from the A/B Test report. You can then use this template in a separate campaign targeting the rest of your customers.

After you set up an A/B test, you can view the information for the test on the journey component in both the Compact and Detail modes.

Archive a variation

After you launch a campaign, you can archive any of the A/B Test variations by setting the percentage of users allocated to it to 0%. You must redistribute the users to other variations and ensure that the total percentage of users is always equal to 100%.

For a launched event triggered campaign, the variation that you archive is available in the Archived variants as soon as you Save the campaign.
For a paused event triggered campaign or paused segment triggered campaign, the variation that you archive is available in the Archived variants after you Save and then Launch the campaign.
For a launched segment triggered campaign, you must pause the campaign to archive any A/B Test variations.

Clone a variation

If your A/B Test variations are similar to each other, or if you want to copy the contents of any variation, you can Clone the variation. You can clone active and archived variations. Cloning is an easy way to revive archived variations.

When you clone a variation, the subject line or the template is copied to the new variation along with any tracking parameters and notes. You can then edit the variation and use it in the A/B test.

Measuring & Optimizing Campaigns

Once the campaign is live, you can access the detailed analysis under the Reports > A/B Test Results tab of the campaign studio. For each template, you will see the number of users, impressions, clicks as well as orders/revenue and custom goals data. This helps you measure template changes and their effect on downstream conversions and custom goals.

For more information, see A/B Test Reports.

A/B Test Email Components

When you run an A/B test on the entire creative, you can test the overall effectiveness of a template. The default A/B test report looks at the total conversion ( e.g. clicks) for the entire creative. However, if you wants to look at the conversion for a specific component, like a recommendation block, then you can use the template click heat map.

For example, let’s say you want to include a “customers who bought X also bought Y” recommendation in your email and you want to figure out what look back period to use for this recommendation so that you get the best engagement. You can create a recommendation scheme, Rec A, with a 30-day look back period and include it in Email template A. Add another scheme, Rec B, with a 90-day look back period and include that in Email template B. Run an A/B test on templates Email A and Email B. Now when you look at the click heat maps for these templates, you will be able to check the click through rates on the recommendation blocks Rec A and Rec B. With this information, you can determine which of the look back periods had a higher engagement.