Create multivariate and A/B tests

Quoted on this wiki

Every place a page here uses this source, in the order the words come in it.

1. Mistakes to avoid “There are some common mistakes to avoid creating the appearance of differences based on the messaging channel if audiences are not filtered correctly.” For example, if you send a push message to a wide audience with a control, the test group sends messages only to users with a push token. However, the control group includes both users who do have a push token and users who don’t. In this case, your initial audience for the campaign or Canvas must filter for having a push token (Foreground Push Enabled is true). The same must be done for eligibility to receive messages on other channels: opted in, has a push token, or subscribed.

   In Campaign audience definition

2. Step 2: Compose your variants “An A/B test examines the effect of changing one variable, whereas a multivariate test examines two or more.” For some ideas on how to get started differentiating your variants, refer to Tips for different channels.

   In Campaign variant testing

3. Choosing a segment “Since different segments of your users may respond differently to messaging, the success of a particular message says something about both the message itself and its target segment.” For instance, while active users may have equal response rates to “This deal expires tomorrow!” and “This deal expires in 24 hours!”, users who haven’t opened the app for a week may be more responsive toward the latter wording since it creates a greater sense of urgency.

   In Segment conversion rates