You can use campaign holdouts to measure the impact of running a campaign against a baseline of doing nothing. In other words, you can compare the outcome of a campaign that you run for a group of users (test group) who receive messages from a campaign against the group of users (control group) who do not.

Note

To enable holdout analysis for your account, please reach out to support@getblueshift.com.

With Blueshift, you can set up two types of control groups:

  • Global Control Group: A group of users who are held out from all campaigns and syndications
  • Campaign Control Group: A group of users who are held out from a specific campaign

Global Control Group

You can specify the global control group as a percentage of all users on your platform. Blueshift uses bsft_control_bucket to determine which users will be a part of the global control group. In the Blueshift platform, all your customers are equally distributed in 100 buckets randomly. When your customer is added to the Blueshift platform, the bsft_control_bucket attribute is automatically attached to the customer's profile and its value is randomly set between 1-100. In the context of global control group, for example, if you choose to hold out 18% of all your users from all campaigns and syndications, users in bsft_control_buckets 1-18 will be a part of the global control group.

Once you set up a global control group for your account, you can disable it. However, you cannot update the global holdout percentage.

You can specify the global holdout percentage for your account from the Other Settings tab of the Account Settings page. 

control_groups.png

Campaign Control Group

You can specify the campaign control group as a percentage of the users eligible for the campaign. Setting this to 0% means that there won't be any campaign level holdout and all eligible customers receive messages from the campaign.

When the campaign is launched, Blueshift holds out customers in the campaign control group in addition to the users in the global control group from receiving messages. You can update the holdout percentage after launching the campaign. However, any percentage change would only impact new users who are eligible for the campaign. It wouldn't change the treatment for users who re-qualify for the journey. So for example, a user who was previously in the control group, will continue to be in the control group and a user who was previously in the test group will continue to be in the test group when they re-qualify for the campaign journey, even if the holdout percentages have changed.

You can specify the customers who should be a part of the control group at the time of specifying the settings for the campaign from the Properties tab. 

jb_holdout.png

Here are the settings that you can use to holdout your customers from receiving messages of a campaign:

  • Honor Global Control Group: Set the value of this setting to Yes to exclude the customers who are a part of the global control group (described above).  You can set this to No if you don't want to use the global control group in this campaign and send a message to all the customers. For example, in cases where you are running a one time campaign to notify all your customers of something urgent, you can set the value of this setting to No.  
  • Campaign Control Group: Specify the percentage of eligible customers who should be excluded from receiving messages of a campaign. When a user enters a journey who is eligible for messaging (after they meet all the trigger criteria, global inclusion segments, global control group restrictions etc.), Blueshift generates a random number to determine whether the Blueshift platform should send messages to the user or hold out the user. For example, if you specify that 10 % of the customers should not receive messages from a campaign and you also have a global holdout = 20 %, then 10% of the 80% users (the total number of eligible users minus global control group users) will be held out as a part of the campaign level holdout group. So if there are 100 users who are eligible for the campaign, Blueshift platform holds out 20 users who are members of the global control group and 8 users who are members of the campaign control group, leading to the Blueshift platform holding out a total number of 28 users.
  • Campaign control group is mutually exclusive: Set the value of this setting to Yes if you want to ensure that users in the control group are not members of the control group of any other campaign. This ensures that the same user doesn’t get held out from multiple simultaneously running campaigns. 

    If you set this value to No, then some control group users may get held out only from this campaign, while some others may get held out from multiple campaigns. In other words, this would introduce another variable in the control group, which can impact the results of your experiment. Hence, we don't recommend this setting. It’s best to leverage the global control group setting if you want to hold out the same user from multiple campaigns at the same time.

Best practices

Changing global or campaign level holdout settings while an experiment is underway can cause the same user to show up in the control group as well as the test group. This would defeat the purpose of your experiment and would lead to incorrect conclusions. Blueshift has added UI restrictions that would prevent you from repeatedly making such changes. However, the UI restrictions aren't a 100% safeguard. We suggest that you should exercise caution when you change holdout settings and be extra careful when you analyze the results of experiments where the settings were changed in the middle of the experiment.

It is also important to keep in mind that the users in the control group of a campaign may receive messages from other concurrent campaigns which could impact how they respond to the current campaign. Although users would be selected randomly, the impact of other campaigns may not be equally negated from the control and test groups. Hence, when you run your analysis, you should choose a time window that is sufficiently long to cancel out the impact of other campaigns on the control and test groups.

Holdout reporting

The Holdout report for a campaign helps you to measure the impact of running a campaign against a baseline of doing nothing. It compares the performance of the test group to the performance of the campaign control group. For more information, see Campaign holdout reporting.

Reporting - User metrics

The campaign 'send summary' report on the campaign detail page shows a count of users who were held out and the count of users who received a message.

campaign_reporting_holdouts.png

If you want to know which individual user was held out (a user who was a member of the control group) vs which individual user was messaged (a user who was a member of the test group), use the campaign activity report. In the User Campaign Activity Report, you will notice a new campaign activity (in addition to sent, open, click etc.) called 'holdout', that indicates that the user was held out (instead of being messaged). For the holdout activity, you will find the following extended attribute:

  • is_global: True indicates that the user is in the global control group. False indicates that the user is in the campaign control group

Ensure that you enable the holdout activity to receive these events via webhook. For more information, see Campaign Activity Export via Webhooks.

mceclip0.png

 

Things to know

  • Holdout setup is honored only when a campaign is run in the normal mode. It is not honored when a campaign is run in the test mode.
  • Similarly, users aren’t held out during test sends.
  • When a user is eligible for messaging in a campaign journey, Blueshift determines whether the user can be messaged depending upon whether they are in the test group or in the control group. If the user is held out, they exit the journey at that point itself. In other words, the user is not evaluated for other subsequent triggers in the campaign journey once they are held out.
  • Holdout activity will be reported every time a user is held out as per the holdout criteria. Thus you may see multiple ‘holdout’ actions for the same user for a given journey if the user re-qualifies for the journey multiple times (e.g. in a recurring campaign).
  • Holdout is considered to be a successful journey completion. So if a user is held out of a segment triggered campaign that is set to never re-qualify or an event triggered campaign that has once in a lifetime journey criteria, the user will never re-qualify for that campaign again.
  • User merges can impact your holdout analysis. For example one of the user identities may have been in the control group and the other in the test group. When these identities are merged into a single profile, then all the activities across both the profiles would get merged as well. This means that there may be some users in your experiment who are a part of both test and control groups at the same time.
  • Blueshift tries to maintain a user’s campaign membership throughout the lifetime of the campaign. So a user who is in the test group continues to remain in the test group even if they re-qualify for the campaign (for example, if it is a recurring campaign). The same applies to users in the campaign control group and the global control group. The only times a user’s membership can change during the course of a campaign are (1) when you update the global holdout settings and (2) when a user’s profiles are merged. Therefore, we recommended that you do not change the global holdout settings often. If you absolutely need to make such a change, you should make a note of the date and only use a time window that excludes this date when you do a holdout analysis.

 

Was this article helpful?
1 out of 1 found this helpful

Comments

0 comments

Please sign in to leave a comment.