A/B email testing - segregating email house file into two sepereate groups

brad.makaiau

I'm trying to do some A/B email testing to figure out which variation of our sender name and subject line works best. Im tacking the open rates and clicks.

Can someone tell me how I can split our opt-in groups into two seperate test groups?

Matt Burghdoff 2

Hi Brad,

Very easy to do.  While editing your email, create a variant of that email (step 8 - Create a new Variant).  Once both versions are approved, select option "A/B Test" on page Email --> Campaigns --> Messages for the specific email that you want to send.

This ONLY splits the mailable audience into the chunks you define for purposes of mailing out that specific email, though.  Say you have a Convio group built out named "Email these People" with 1,000 members and choose to do a 50/50 split.  Your "Email these People" group will only be divided for the purpose of that specific email send, not permanently cut apart into "Email these People 1" and "Email these People 2."

Hope that helps.

brad.makaiau

Matt Burghdoff:

Hi Brad,

Very easy to do.  While editing your email, create a variant of that email (step 8 - Create a new Variant).  Once both versions are approved, select option "A/B Test" on page Email --> Campaigns --> Messages for the specific email that you want to send.

This ONLY splits the mailable audience into the chunks you define for purposes of mailing out that specific email, though.  Say you have a Convio group built out named "Email these People" with 1,000 members and choose to do a 50/50 split.  Your "Email these People" group will only be divided for the purpose of that specific email send, not permanently cut apart into "Email these People 1" and "Email these People 2."

Hope that helps.

Awesome! Thanks Matt!

Michael

Matt Burghdoff:

Hi Brad,

Very easy to do.  While editing your email, create a variant of that email (step 8 - Create a new Variant).  Once both versions are approved, select option "A/B Test" on page Email --> Campaigns --> Messages for the specific email that you want to send.

This ONLY splits the mailable audience into the chunks you define for purposes of mailing out that specific email, though.  Say you have a Convio group built out named "Email these People" with 1,000 members and choose to do a 50/50 split.  Your "Email these People" group will only be divided for the purpose of that specific email send, not permanently cut apart into "Email these People 1" and "Email these People 2."

Hope that helps.

One warning:  keep in mind that unless you have a very very large email file, be careful that you pay attention to statistical significance...  With Matt's example, if you do an A/B test with 1000 people and find that variant 1 is better, then your e-mail list grows to, say, 1250, then your results may not be valid -- and at the very least the chance is equal to a coin toss for them to still be the same.  Only if you have a large sample size will doing lots of segmented testing give you much more than information that you can use *right now* then discard...  and by the time you've collected the data, you'll probably have already used the asks/text/placements/whatever..  Anyway, point is, you have to pay attention to your sample sizes or your results won't be very meaningful.