Using A/B tests in Blueshift, you can see which variation of a template performs better. You can test several templates that may have different forms of personalization or entirely different content.
You can set up the A/B test from the A/B Test tab of the trigger. The type of A/B test available depends on the type of channel selected for the trigger.
|Entire Creative||All||Test the entire template. If your templates are similar to each other, you can use the clone functionality after creating one template and edit the cloned template.|
|Subject Line||Only Email||Test different subject lines for the same content.|
|Preheader Text||Only Email||Test different preheader text for the same content.|
|Sender||Email and SMS||
For emails, test the From name, From address, and Reply to address fields.
For SMS, test different Sender numbers/Codes for the same content.
To set up a A/B test, complete the following steps:
- Select a Template from the Channel tab of the trigger.
- Click Create A/B Test or go to the A/B Test tab to set up the test.
- On the A/B Test tab, select the Type as required. Depending on the channel for the trigger, you can select one of the following options:
- Entire Creative
- Subject Line
- Click Add Variation, add a variant, and the percentage of users to allocate to it.
- The first variant is your control and cannot be edited.
- You can add up to 20 variants to test.
- For an email Subject Line A/B test, you can use the same markup to customize the subject as you would for the email template.
After you set up an A/B test, you can view the information for the test on the journey component in both the Compact and Detail modes.
After you launch a campaign, you can archive any of the A/B Test variations by setting the percentage of users allocated to it to 0%. You must redistribute the users to other variations and ensure that the total percentage of users is always equal to 100%.
- For a launched event triggered campaign, the variation that you archive is available in the Archived variants as soon as you Save the campaign.
- For a paused event triggered campaign or paused segment triggered campaign, the variation that you archive is available in the Archived variants after you Save and then Launch the campaign.
- For a launched segment triggered campaign, you must pause the campaign to archive any A/B Test variations.
If your A/B Test variations are similar to each other, or if you want to copy the contents of any variation, you can Clone the variation. You can clone active and archived variations. Cloning is an easy way to revive archived variations.
When you clone a variation, the subject line or the template is copied to the new variation along with any tracking parameters and notes. You can then edit the variation and use it in the A/B test.
Once the campaign is live, you can access the detailed analysis under the Reports > A/B Test Results tab of the campaign studio. For each template, you will see the number of users, impressions, clicks as well as orders/revenue and custom goals data. This helps you measure template changes and their effect on downstream conversions and custom goals.
For more information, see A/B Test Reports.
When you run an A/B test on the entire creative, you can test the overall effectiveness of a template. The default A/B test report looks at the total conversion ( e.g. clicks) for the entire creative. However, if you wants to look at the conversion for a specific component, like a recommendation block, then you can use the template click heat map.
For example, let’s say you want to include a “customers who bought X also bought Y” recommendation in your email and you want to figure out what look back period to use for this recommendation so that you get the best engagement. You can create a recommendation scheme, Rec A, with a 30-day look back period and include it in Email template A. Add another scheme, Rec B, with a 90-day look back period and include that in Email template B. Run an A/B test on templates Email A and Email B. Now when you look at the click heat maps for these templates, you will be able to check the click through rates on the recommendation blocks Rec A and Rec B. With this information, you can determine which of the look back periods had a higher engagement.