How do I create an A/B Test and Message Variant?

Create Message Variant

Message Variants are necessary to conduct A/B Testing. A message variant contains alterations in the original message to the envelope, stationery, or content of the message. Once you have created a variant you will be able to perform an A/B test to measure audience response to the two messages. For the most accurate results, change one element at a time in a message variant.

Create a Message Variant as follows:

1. Click Create a Variant on the Message Variants page. This is the last page in the message creation flow.

2. Enter a label for the variant. This label is helpful in detailing the changes that this variant will encompass.

3. Click Next.

4. At this point, you should change only one element amongst the envelope, stationery, and body of the message. This process is the same as creating a new message.

Run A/B Test

A/B Testing will allow you to test the effectiveness of your emails, ensuring the best response from your constituents. To run an A/B Test you must create two versions of your Email Message: the original, and a variant (alternative version). For the best, and most accurate results, you should change one element in your variant message. During the configuration of your test you will select subset audience sizes. These audience members will be chosen randomly from your entire audience. After you run the A/B test, you can analyze the results through the Variant Performance Report to determine which message to send to the remainder of your audience.

New for Version 4.0.9: When a message has variants, the Email Campaigns message list Status column now shows the number of variants in each delivery status.

Run an A/B Test as follows:

1. Click Run an A/B Test at the upper left of the Message List page.

2. Select the message that you would like to test. The message must have a variant, and both message variants must be in the approved state before you can run the A/B Test.

Note: New for Version 4.0.9: Approved message variants may be tested even when the original message is not approved.

3. Click the A/B Test link next to the message to test.

Select A/B Test Details

1. Review your test instructions. You will need to configure the A/B Test parameters, as well as set the schedule for when to run the test. The test is scheduled to run immediately if no specific time and date are chosen.

Enter Test Audience Size

1. To set up the size of the audience that should receive each test message, click the Setup button under the A/B Test Setup header.

2. For each message, to set the sample size: (although two fields display, you may use only one field to set the audience size).

As a Sample Percentage, type in the percentage of constituents from your Target List that should receive the message.

As a Sample Number, type in the number of audience members from your Target List that should receive this message.

3. Click Finish.

Change Test Delivery Schedule

1. To schedule a test delivery (other than immediate delivery), click the Edit button under the Schedule header.

2. Select the specific date and time for your test delivery from the drop-down lists.

3. Click Finish.

Note: This scheduled date and time does not take effect until you click the A/B Test button in Step 2 of the A/B Test page.

Start A/B Test

1. When you have completed the configuration of your test, click the A/B Test button on Step 2 of the A/B Test Review page. Your message variants will be sent one after another.

You can analyze the results of your A/B Test through the Variant Performance Report after your sample audience has received your message.