Predictive Studio - Score Definition

Creating a Predictive Score definition

Blueshift’s Predictive Studio allows you to define and customize a Predictive Score based on your Marketing objective. The underlying AI model learns a user’s likelihood (Predictive Score) to convert on:

• The specific channel such as email, push, SMS, or in-app based on past behavior
• The specific events in the future, based on past behaviors

 

Input for a channel engagement model

To create a model that predicts engagement on an event, click + Predictive Model on the index page of the Predictive Studio and then choose one of the following:

• Email
• Push
• SMS
• In-App

Next, specify what you would like for the email score:

Goal Actions: Actions for which you’d like to create a Predictive Score. If a user performs the action on the message that we send, we count it as conversion. For example, you can add open and click as actions that should be counted as conversions for the email channel. At the moment, here are the actions that you can choose (based on the channel you select to create a score):

• Email: Open, Click
• Push: Click
• SMS: Click
• In-App: Open, Click

 

Goal Look-Forward: Predictions will be made and remain valid for this time period. For example, predict channel engagements in the next 7 days.

The default is 7 days. Shorter windows(a day or few days) could lead to more accurate predictions, while longer windows could introduce noise/irregularities, which leads to lower predictive accuracy.

Consider setting this value based on the historical conversion rate. For example, n number of conversions in the prediction window/number of active users in lookback days * 100 is at least 1%.

 

Activity Look Back: Audience for the Predictive Scores will include those users who had at least one action on the channel in the lookback period. The default value is 30 days (monthly active users).

Consider setting this value based on the historical conversion rate. For example, n number of conversions in the prediction window/number of active users in lookback days * 100) is at least 1%.

After you create the model, our platform generates a summary and visualization.  It is empty in the beginning since we refresh the model daily. So we recommend that you check back later. Once it's ready, you can take a look at it and here's how it looks:

For more information, see Model Visualization. 

 

Input for an event engagement model

To create a model that predicts engagement on an event, click + Predictive Model on the index page of the Predictive Studio and then choose Event.

 

Next, specify what you would like for the event score:

Goal Events-  Events for which that you’d like to create a Predictive Score. Presence of at least one of these events will be counted as a conversion. For example:  purchase, subscription_renewal, etc. You can choose any combination of events that you are tracking with Blueshift.  You can read more on events here.

 

Goal Look-Forward - Predictions will be made and remain valid for this time period. For example: predict purchase intent in the next 7 days or subscription renewal in the next 30 days.

The default is 7 days. Shorter windows (a day or few days) could lead to more accurate predictions, while longer windows could introduce noise/irregularities, which leads to lower predictive accuracy.

Consider setting this value based on the historical conversion rate. For example, n number of conversions in the prediction window/number of active users in lookback days * 100 is at least 1%.

 

Activity Look-Back - Audience for the Predictive Scores will include those users who had at least one impression(event or campaign action) in the lookback period.  The default value is 30 days (monthly active users). This input should be equal to the length of the conversion funnel.

Consider setting this value based on the historical conversion rate. For example, n number of conversions in the prediction window/number of active users in lookback days * 100 is at least 1%.

In most use cases, the conversion is heavily correlated with the recent engagement, so 30 days(monthly) or 90 days (quarterly) usually suffices.

E-commerce funnels are usually short, so 7 to 14 days work for most e-commerce purchase intent use cases. Subscription businesses can set this input based on their billing cycles to predict subscription churns. You can add funnel events using the Advanced Options button.

 

[Optional] User attributes - You can choose to include user demographics (i.e. age, location, etc), preferences(attributes of interest from the catalog), memberships (subscription type) and lifetime attributes (lifetime orders, revenues, etc). We use the latest value of the user attributes.  Warning: Be careful and don’t pick a user attribute which is updated as a result of the conversion.  Otherwise this would cause leakage(overfitting) and the model wouldn’t generalize well on unseen data. An example of one such user attribute is “total_orders” for predicting purchase conversions. If we increment total_orders after conversion, for one-time purchasers, the model could confidently learn a simple rule that users who convert must have total_orders>0, which is not what we want.

Click here for more details on user attributes. See this article for more details on Blueshift’s user attributes. You can see the summary on the right side. 

After you create the model, our platform generates a summary and visualization.  It is empty in the beginning since we refresh the model daily. So we recommend that you check back later. Once it's ready, you can take a look at it and here's how it looks:

 For more information, see Model Visualization.

 

Data requirements

The most important thing while setting up a Predictive Score is ensuring that the historical conversion rate, ( i.e number of conversions in prediction window/size of the modeled audience * 100), is at least 1% and at least 1,000+ users have converted in the past prediction window. The AI model learns to make predictions based on historical conversions. Too few conversions or very low conversion rates may lead to low performance or stats resulting in an insignificant model. 

If data requirements are not met, Predictive Scores will fail and there will be an alert on the studio dashboard and an error on the index screen (described below). Consider tuning the prediction window, goal lookback days, user attribute filters to make it work.

Once data requirements are met, an AI model will be created. AI Model summary details are talked about in the Model Viz section.

Output

A Predictive Score between 0-100 is generated for every user in the scoring audience. The Predictive Scores are available on customer profiles, which can be used for lookups, segmentation, syndications and exports.

Predictive Studio UI tips

In addition to the options that you can use to create a score, we provide its status on the UI. You can take a look at the score's status and take actions on it, such as tweaking its definition or resolving its errors. A score’s status is indicated using a color. Here’s how a score is color coded and what it means:

• Green: Our platform is computing scores as expected, and you can use the scores based on your requirements.
• Red: Our platform encountered an error while computing the scores. For example, if there's an error with a score, we show an information icon and the type of error that appears. 

  You can click on the report and find more information about the error on the model summary tab of the details page.
  
  Here's a list of errors that you can see and possible steps to resolve them.

  • Data error: Our platform stops refreshing the scores if the data that we get does not meet the criteria that you specify when you define it. Here are the possible cases when it happens:

    • Insufficient data: This error can occur if the data that we get is insufficient for our platform to compute a score. For example, you create an SMS engagement score, however, the customers you choose in the segment don't receive enough SMSes. In such a case, our platform cannot compute an SMS engagement score. To resolve such an issue, choose a lookback period in which your customers receive an adequate amount of SMSes.

    • Insufficient conversion rate:  This error can occur if the conversion rate of a campaign is too low. For example, if you are using an email channel to run your campaigns, and in the email messages you send a list of products to your customers. However, if the conversion is less than 1% of the total number of users to whom you send a message, we cannot generate an accurate score. This happens since we don’t have enough conversions to train our models. To resolve this, try to add filters in your campaign so that it targets customers who have purchased your products earlier. This will allow our platform to use a customer set with more than 1% conversions. For more information on campaigns, see Campaigns.

    • Low predictive power: This error can occur if the data that our platform receives is noisy. For example, if you choose a segment that targets customers in USA. However, our platform also receives a lot of data of your customers who reside in Canada. In such a case, our platform cannot properly compute a score and pauses the score refresh. To resolve this, ensure that the data that our platform receives is clean and consumable. This can happen due to improper integration, so we recommend that you review our developer documentation if data from your service is sent to us using our APIs. Or, you can review documentation on S3 imports and exports if you send your service's data using S3.
      
      
  • Internal error: This happens when our internal systems run into an issue. To resolve this, reach us on support@blueshift.com.  
    
    
• Light grey: This means one of the following:
  • Our platform is just getting started with computing the scores, so the scores aren't ready yet. Check back again later to see its status. 
  • Our platform has paused computing the scores. Our platform pauses refreshing scores if you created or updated a score definition more than 30 days ago, and none of the entities (such as campaigns and templates) use it. To resolve this, you can either restart the computation, update the definition and use it in a campaign or template, or you can reach us on support@blueshift.com.

    

In addition, we use Dark grey for scores that are archived. We stop computation for archived scores, and you can use the filters to see them. For more information on archived entities, see Archive. You can activate an archived score using this drop-down menu. 

 

FAQs

Can I choose a goal based on an event attribute, like product sku?

Yes, the studio doesn’t support this option yet, but we can customize the goal inputs on the backend. Currently, we support goal event filters on product attributes(i.e category, brand, sku) if product sku is available on the event.

Can I filter my audience based on event counts and event attributes?

Yes, the studio doesn’t support this option yet, but we can customize filters on the backend. Currently, we support event filters based on event count(e.g number of purchase > 0), event recency(e.g last purchase < 7 days) and product attributes(e.g category, brand, sku) if product sku is available on the event.