How the Campaign Optimizer Agent works

The Campaign Optimizer Agent workflow has two parts: generating content and running the test. For details on how variants are created, please take a look at the Understanding optimizer concepts page.

This page explains how tests are executed and evaluated.

When the Campaign Optimizer Agent is enabled on a trigger, it runs a structured A/B test between the current control and a new variant. The agent sizes the test, starts with a small split, and evaluates lift and statistical significance across two phases: optimization (adjusts allocations) and observation (holds allocations).

The following flowchart shows how the Campaign Optimizer Agent moves from opportunity generation through testing, observation, and decision-making.

Open in new tab Download

The Campaign Optimizer Agent logic

The Campaign Optimizer Agent starts with a small share of traffic, adjusts it based on performance, and decides whether the variant continues or is replaced. The process runs in two phases.

Phase 1 Optimization

Check current send volume and performance.
Compute the number of days to run the test.
Start the variant with a small share of traffic.
Check lift and statistical significance hourly to adjust allocations.
At phase end:
- Move to Observation, or
- Reject the variant and create a new one.

Phase 2 Observation

Traffic is locked — no allocation changes.
Continues until one of the following:
- Significance reached:
  - Positive lift → variant becomes the new control.
  - Negative lift → variant fails.
- 90-day limit reached (Optimization + Observation) without significance → variant fails.
After Observation ends — regardless of outcome, the Campaign Optimizer Agent generates a new variant and kickstarts the next cycle.

Decision scenarios

When interpreting test results, keep these points in mind:

Significance tells you whether the result is reliable.
Lift shows which option is performing better:
- Positive lift → the test outperforms the control.
- Negative lift → the control outperforms the test.
The Campaign Optimizer Agent uses both significance and lift direction to decide the outcome.

These decision scenarios apply at the end of the optimization phase, when the agent reviews lift and significance to determine the next step.

Scenario	Significance of results	Lift	Behaviour
1	Significant	Positive	Continue in observation for 2 more days. If still significantly positive → mark as success.
2	Significant	Negative	Reject immediately after optimization. No observation; mark as failure.
3	Not significant	Positive	Continue in observation (max 90 days including optimization). If significance is reached → follow scenario 1 or 2. If not → mark as failure.
4	Not significant	Negative

Applies to all scenarios: Once a test concludes, the Campaign Optimizer Agent generates a new variant and queues it for approval.

Campaign Optimizer Agent statuses

Status	What it means
Generating content	The Campaign Optimizer Agent is creating and ranking test variants for approval.
Approval pending	Variants are awaiting user approval before testing can start.
Approval denied	Variants were rejected during approval; this set will not proceed to testing.
Approval expired	If the test is in Approval pending and you switch the mode from Optimizer Agent to Manual, the status changes to Approval expired. This ends the approval process.
Test running	An approved test is in progress (optimization or observation).
Success	The variant achieved significantly positive lift and is promoted over the control.
Failure	One of the following occurred: The variant did not meet success criteria (negative lift or no significance within the limit); control is set to 100%. The test was in Generating content or Test running and you switched the mode to Manual; the status changes to Failure.

Changing from Campaign Optimizer Agent to Manual

If you change the mode in the A/B Test setup from Optimizer Agent to Manual, the current optimization process ends immediately. The status updates based on the stage it was in:

Approval pending → Approval expired
Generating content → Failure
Test running → Failure

If you later change it back to Optimizer Agent, the process starts again with new content generation.

An example walkthrough: end-to-end test

The flowchart below illustrates how the Campaign Optimizer Agent runs a subject line A/B test for the fictional e-commerce brand Trendify:

Control: “Did you forget something?”
Variant A: “Your cart is about to expire – Complete your order!”

The chart shows the full lifecycle — from setup through optimization and observation — leading to either success (variant promoted) or failure (control retained).

Next up

Set up the Campaign Optimizer Agent for a campaign — start tests from Discover opportunities or directly within a launched campaign.