A/B tests

An A/B test sends 2 to 4 versions of a message to a sample of the audience. When the wait ends, Pushtifly compares the variants and sends the winner to the rest.

• Sample: 10 to 50% of the audience (20% by default), split equally among the variants. With 2 variants and 20%, each one reaches 10% and the winner the remaining 80%.
• Wait: 30 to 1,440 minutes (120 by default). It's the time to measure before choosing.
• Metric: open rate (open_rate, default) or click rate (click_rate), over accepted deliveries. In a tie, the one with more clicks wins and, if it's still tied, the first one.

Each device always falls into the same group of the test, so nobody receives two variants or a variant and the winner.

A/B tests are a plan feature. They only allow an immediate send: they can't be scheduled.

When there's no clear winner

If any variant has fewer than 100 accepted deliveries, or none gets opens (or clicks), the test ends without a clear winner (inconclusive) and nothing is sent to the rest. You can pick the winner manually or cancel the test.

You can also pick the winner manually before the wait ends.

From the panel

In A/B tests, choose the app, the topic (optional), the metric, the sample and the wait, and write the variants. The list shows the status, the accepted and opened counts and the rates of each variant, plus the Choose as winner and Cancel test buttons.

Each send's details show whether it's a variant or the winner's send of a test.

From the API

Use a secret key with notifications:send. The body is the same as POST /v1/notifications plus abTest; each variant can change content, deepLink, data, actions and media.

curl -X POST https://api.pushtifly.com/v1/ab-tests \
  -H "Authorization: Bearer sk_live_YOUR_KEY" -H "Content-Type: application/json" \
  -d '{
    "name": "Weekend deal",
    "target": { "type": "topic", "topic": "offers" },
    "category": "marketing",
    "content": { "title": "Deal", "body": "20% off" },
    "abTest": {
      "samplePercent": 20, "waitMinutes": 120, "metric": "open_rate",
      "variants": [
        { "content": { "title": "Today only", "body": "20% off everything" } },
        { "content": { "title": "Your coupon is waiting", "body": "Use it before midnight" }, "deepLink": "myapp://coupon" }
      ]
    }
  }'

It responds 202 with the test in testing status. Check the result with GET /v1/ab-tests/{id}:

{
  "id": "01K…", "status": "completed", "metric": "open_rate",
  "samplePercent": 20, "winnerPercent": 80, "decidedBy": "auto", "winnerVariant": 2,
  "variants": [
    { "variant": 1, "label": "A", "audiencePercent": 10, "stats": { "accepted": 980, "opened": 72, "openRate": 0.0735, "clickRate": 0.0112 } },
    { "variant": 2, "label": "B", "audiencePercent": 10, "stats": { "accepted": 1004, "opened": 118, "openRate": 0.1175, "clickRate": 0.0199 } }
  ],
  "winner": { "notificationId": "01K…", "status": "sending" }
}

Status	Meaning
testing	The variants were sent and it's measuring.
sending_winner	The winner was chosen and its send is being prepared.
completed	The winner was sent to the rest (decidedBy: auto or manual).
inconclusive	No clear winner (inconclusiveReason: insufficient_sample or no_signal).
cancelled	Cancelled: no winner is sent.

To choose manually: POST /v1/ab-tests/{id}/winner with { "variant": 2 }. To cancel: POST /v1/ab-tests/{id}/cancel. Variants already sent can't be undone.

Each variant and the winner's send are normal sends: they appear in /v1/notifications with abTest: { id, variant } (variant 0 is the winner), and their statistics and deliveries are queried and exported like any other send's.

The app's marketing limits also apply to the variants and the winner.