Data Validation

1. Most Recent Event
2. Event Summary Counts

3. Event Archive
4. S3 Error Reports
5. Segment Exports

1. Most Recent Event

The event debugger API endpoint returns the most recent events our server has seen of each event type. Use the following endpoint to fetch the most recent event:

curl -u <EVENT_API_KEY>: https://api.getblueshift.com/api/v1/event/debug

2. Event Summary Counts

You can view your event stream in the clickstream page. You will also be able to look at errors and download the error logs.

To fetch this information via API

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{
  "timestamp_start": <start_time>,
  "timestamp_end": <end_time>
}' -u <EVENT_API_KEY>: https://api.getblueshift.com/api/v1/event/summary.json

 Ex:

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{
  "timestamp_start": "2018-10-16T00:00:00Z",
  "timestamp_end": "2018-10-28T23:59:59Z"
}' -u <EVENT_API_KEY>: https://api.getblueshift.com/api/v1/event/summary.json 

Sample Response:

Please find more information on each section below

{
  "events_chart_stats": {
    "archives": {
      
    },
    "error": {
      "invalid_user_attributes": [
        0,
        0,
        0
      ],
      "missing_identifier": [
        0,
        5,
        0
      ]
    },
    "events": {
      "add_to_cart": [
        100,
        110,
        200
      ],
      "subscription_update": [
        50,
        60,
        80
      ]
    },
    "timelapse": [
      "10/16",
      "10/17",
      "10/18"
    ]
  },
  "stats": {
    "add_to_cart": {
      "archived": false,
      "count": 310,
      "display_name": "add_to_cart",
      "properties": {
        
      },
      "type": "standard"
    },
    "subscription_update": {
      "archived": false,
      "count": 190,
      "display_name": "subscription_update",
      "properties": {
        
      },
      "type": "custom_event"
    }
  },
  "status": "ok",
  "timestamp_end": "2018-10-18T06:59:59.999Z",
  "timestamp_start": "2018-10-16T07:00:00.000Z"
}

 The response body contains the following data:

a. Event Count

Returns the count for each event received by Blueshift. The “stats” key contains the details on each event. You need to filter for type == “standard” or “custom_event”.
Ex:

{
  "stats": {
    "subscription_update": {
      "archived": false,
      "count": 190,
      "display_name": "subscription_update",
      "properties": {
        
      },
      "type": "custom_event"
    }
  }
}

b. Event Distribution Count

Returns event count distribution by day. The timelapse attribute contains the dates

Ex:

{
  "events_chart_stats": {
    "events"{
      "customer_subscription_current_add": [
        0,
        0,
        3
      ]
    },
    "timelapse": [
      "10/16",
      "10/17",
      "10/18"
    ]
  }
}

c. Event Processing Errors

Returns the error count and also the distribution of error type by day.
Ex:

{
  "stats": {
    "missing_identifier": {
      "archived": false,
      "count": 7,
      "display_name": "Missing Identifier",
      "properties": {
        
      },
      "type": "event_error"
    }
  }
}

Errors:

Please find more information on different event processing errors below

Error Types Description
es_index_error Indexing Error, This usually happens when there is a data type mismatch between the event payload and our backend system. Ex: If you passed in string value for date
event_name_missing The event is missing the required attribute ('event')
event_verification_error This is applicable only if the event verification feature is enabled. The event signature verification has failed
invalid_customer_id The customer_id is invalid Ex: '0', customer_id greater than 64 characters
invalid_device_id The device is invalid, does not match the device_id format for the iOS or Android device
invalid_email Email is invalid
invalid_event_attributes There are some invalid attributes in the event payload Ex: event name is greater than 100 characters, some attribute contains script tag etc
invalid_product Product not found in Catalog
invalid_user_attributes Payload has invalid user attributes. Ex: invalid gender value etc
missing_identifier Missing Customer Identifier (cookie, device_ids, email, email_hash, retailer_customer_id)
missing_product A whitelist product/catalog event is missing the product/catalog ids in the payload
parsing_error Blueshift was unable to JSON parse the payload
transaction_indexing_error There was an error updating the transaction record
unknown_error Some unexpected processing error, please contact Blueshift
user_lock_error We were unable to acquire lock on the user.
user_not_found_error The user was not found in our system

 3. Event Archive

Once an event has been processed it gets archived to S3, all the processed events can be fetched from the following location in S3

Bucket: bsft-customers
S3_path: <site>/events/<year>/<month>/<day>

4. S3 Error Reports

Any event processing errors will be exported to the following s3 locations:

Bucket: bsft-customers
S3_path: <site>/events/errors/<year>/<month>/<day>

You can find your s3 credentials on your Account Profile page. Only admins will be able to view the s3 credentials.

Below is a sample error event with the detailed error message:

{
  "event": "subscription_update",
  "site": "blueshiftreads.com",
  "user_agent": "Faraday v0.9.2",
  "timestamp": "2018-08-22T07:29:58Z",
  "account_uuid": "1ecf5042-7738-44d7-a09c-bb89ff8f5327",
  "event_uuid": "1c0108b0-a5dd-11e8-ad27-0242ac110005",
  "timestamp_epoch": 1534922998,
  "browser_type": "Faraday",
  "browser_version": "",
  "browser_platform": null,
  "error_message": "Missing Customer Identifier (cookie, device_ids, email, email_hash, retailer_customer_id)",
  "error_type": "missing_identifier"
}

5. Segment Exports

You can validate your segment counts/users by doing segment exports (ad-hoc or a scheduled export), which you can then use to validate the counts and compare the users with your backend systems.

Was this article helpful?
0 out of 0 found this helpful