Campaign holdout testing

You can use campaign holdouts to measure the impact of running a campaign against a baseline of doing nothing. In other words, you can compare the outcome of a campaign that you run for a group of users (test group) who receive messages from a campaign against the group of users (control group) who do not.

With Blueshift, you can set up two types of control groups:

• Global Control Group: A group of users who are held out from all campaigns and syndications
• Campaign Control Group: A group of users who are held out from a specific campaign

Global Control Group

A global control group is a percentage of all your customers who don't receive messages from a campaign and are used to measure the impact of running a campaign against a baseline of doing nothing.

Blueshift uses bsft_control_bucket to determine which customers will be a part of the global control group. In the Blueshift platform, all your customers are equally distributed in 100 buckets randomly. When your customer is added to the Blueshift platform, the bsft_control_bucket attribute is automatically attached to the customer's profile and its value is randomly set between 1-100. 

In the context of global control group, for example, if you choose to hold out 18% of all your users from all campaigns and syndications, users in bsft_control_buckets 1-18 will be a part of the global control group.

You can specify the global holdout percentage for your account from the Other Settings tab of the Account Settings page.

Campaign Control Group

You can specify the campaign control group as a percentage of the users eligible for the campaign. Setting this to 0% means that there won't be any campaign level holdout and all eligible customers receive messages from the campaign.

When the campaign is launched, Blueshift holds out customers in the campaign control group in addition to the users in the global control group from receiving messages. You can update the holdout percentage after launching the campaign. However, any percentage change would only impact new users who are eligible for the campaign. It wouldn't change the treatment for users who re-qualify for the journey. So for example, a user who was previously in the control group, will continue to be in the control group and a user who was previously in the test group will continue to be in the test group when they re-qualify for the campaign journey, even if the holdout percentages have changed.

You can specify the customers who should be a part of the control group at the time of specifying the settings for the campaign from the Properties tab. 

Here are the settings that you can use to holdout your customers from receiving messages of a campaign:

Best practices

Changing global or campaign level holdout settings while an experiment is underway can cause the same user to show up in the control group as well as the test group. This would defeat the purpose of your experiment and would lead to incorrect conclusions. Blueshift has added UI restrictions that would prevent you from repeatedly making such changes. However, the UI restrictions aren't a 100% safeguard. We suggest that you should exercise caution when you change holdout settings and be extra careful when you analyze the results of experiments where the settings were changed in the middle of the experiment.

It is also important to keep in mind that the users in the control group of a campaign may receive messages from other concurrent campaigns which could impact how they respond to the current campaign. Although users would be selected randomly, the impact of other campaigns may not be equally negated from the control and test groups. Hence, when you run your analysis, you should choose a time window that is sufficiently long to cancel out the impact of other campaigns on the control and test groups.

Holdout reporting

The Holdout report for a campaign helps you to measure the impact of running a campaign against a baseline of doing nothing. It compares the performance of the test group to the performance of the campaign control group. For more information, see Campaign holdout reporting.

Reporting - User metrics

The campaign 'send summary' report on the campaign detail page shows a count of users who were held out and the count of users who received a message.

If you want to know which individual user was held out (a user who was a member of the control group) vs which individual user was messaged (a user who was a member of the test group), use the campaign activity report. In the User Campaign Activity Report, you will notice a new campaign activity (in addition to sent, open, click etc.) called 'holdout', that indicates that the user was held out (instead of being messaged). For the holdout activity, you will find the following extended attribute:

• is_global: True indicates that the user is in the global control group. False indicates that the user is in the campaign control group

Ensure that you enable the holdout activity to receive these events via webhook. For more information, see Campaign Activity Export via Webhooks.

Things to know

• Holdout setup is honored only when a campaign is run in the normal mode. It is not honored when a campaign is run in the test mode.
• Similarly, users aren’t held out during test sends.
• When a user is eligible for messaging in a campaign journey, Blueshift determines whether the user can be messaged depending upon whether they are in the test group or in the control group. If the user is held out, they exit the journey at that point itself. In other words, the user is not evaluated for other subsequent triggers in the campaign journey once they are held out.
• Holdout activity will be reported every time a user is held out as per the holdout criteria. Thus you may see multiple ‘holdout’ actions for the same user for a given journey if the user re-qualifies for the journey multiple times (e.g. in a recurring campaign).
• Holdout is considered to be a successful journey completion. So if a user is held out of a segment triggered campaign that is set to never re-qualify or an event triggered campaign that has once in a lifetime journey criteria, the user will never re-qualify for that campaign again.
• User merges can impact your holdout analysis. For example one of the user identities may have been in the control group and the other in the test group. When these identities are merged into a single profile, then all the activities across both the profiles would get merged as well. This means that there may be some users in your experiment who are a part of both test and control groups at the same time.
• Blueshift tries to maintain a user’s campaign membership throughout the lifetime of the campaign. So a user who is in the test group continues to remain in the test group even if they re-qualify for the campaign (for example, if it is a recurring campaign). The same applies to users in the campaign control group and the global control group. The only times a user’s membership can change during the course of a campaign are (1) when you update the global holdout settings and (2) when a user’s profiles are merged. Therefore, we recommended that you do not change the global holdout settings often. If you absolutely need to make such a change, you should make a note of the date and only use a time window that excludes this date when you do a holdout analysis.