A/B testing subject lines - Clarifications

Stephen Gouws

Hey,


So the documentation isn't 100% clear on A/B testing within Luminate, at least as far as I can see.


If for example, I wanted to create an A/B test of a certain email and split it exactly 50/50, with just a subject line change, is it as simple as: create a variant of the original approved message and simply adjust the subject line in the Envelope settings and then setup A/B testing for that message targetting 50% of the audience?

I would then do the same for the original variant, also targetting 50% of the audience.

1. Does Luminate ensure that 2 messages within a campaign, that each have equal A/B test share cannot send to the same constituents - meaning if the A/B test is set up in the manner described above, will a constituent who receives 1 variant be disqualified from receiving the other automatically?
2. How does scheduling work? It does not seem possible to schedule the A/B tests on the calendar.
3. If No.2(above) is true. How should I manage the send, simply click 'start test' when the 2 variants should go out?

Any help would be amazing!


Thanks and Regards,

Stephen

 

Allison King

Hi Stephen,


You're on the right track.


For A/B testing, first you would create the original message and the variant with the different subject line. You'll see in your Email List that the message has two versions. Make sure both version are approved. When you click A/B Test, you'll be able to assign a percentage to the original and the variant. In your case you would put 50% for each. On the second step of this process, under Related Actions, you can set a schedule so the messages will not send immediately.


The best practice is to keep your audience size small (well under 10,000 total audience size) for the testing and then send out the "winner" to the larger group. Luminate Online matches each email ID to the constituent ID, so the same email will never be sent to the same constituent record (unless you're using recurring delivery and override this setting).


For more information on A/B Testing, check out this great article in the Knowledge Base: https://kb.blackbaud.com/articles/Article/67299. We also cover A/B Testing in the Luminate Online: Optimizing Your Email Performance course. 

Philip Nawrocki

Hey Stephen,


I made myself a cheat sheet for this.


Here are my steps:


1.    Variant sample test

       a.    Create e-mail and its variant. (This is where you would change your subject line)

       b.    On the Messages screen, click A/B test.

       c.    In Step 1. Choose the percentage you want to test with. (Usually this is a 10/10/80 test. So 10% to original message and 10% to variant, then 80% to "winner" of the test)

       d.    Then do the rest of the setup steps

       e.    Once a sufficient amount of time has passed, run the Variance Performance Report to find the message that should be sent to the rest of the audience.

              i.    This is only something you will be able to know. You are setting up the parameters for the test and will know who the 'winner' is.

       f.    Click, Setup Delivery on the "winning" message and add the audience. (Same Target Audience you used in the variants AND same suppression groups.)

              i.    Do audience calculation and you will see in the filters applied area there is a line for 'has been sent this message already'. This filters out the constituents that received the 10/10 send and will match your audience from the initial A/B test.


It definitely gets easier as you continue to work with it.


Thanks,


Phil


 

Stephen Gouws

Thank you both soo much, this helps a ton!

Also, this resource was super helpful for anyone coming across this post in the future: https://convio.adobeconnect.com/_a836067539/messagevariants/


Philip, thanks so much for the steps there! Awesome stuff, I really appreciate!!


You both have yourself an awesome day!


Regards,

Stephen