A/B Testing Coaching Email

Brian Mucha

Our TR admins are interested in doing A/B tests of coaching email. Of course this functionality is not present in TR the way it is in broadcast.

 

So the biggest problem with doing that is creating the two random target groups. Support suggested pulling a list of participants, splitting them manually and doing a Group Assignment upload. I had thought about that, but what a pain!

 

I started thinking how I could build a query driven random split. I couldn't think of a way to make this truly random, but I did stumble on System > Contact Range. Like so much in Luminate there is no documentation that I could find on what this actually is, and the wording is pretty vague.

 

 

Turns out that this allows you to key on the last X digits of the Cons Id! I can't imagine what this was created for, but it's perfect. I created a query that makes a rebuildable group for everyone who has a cons Id that ends in an even number, and another for the odd numbers.

 

 

Even/odd is not exactly random, but I think this pseudo-random selection should be good enough to not unintentionally introduce bias in the tests. (I'm assuming this, rather than simply using 0-4 and 5-9, will break up families/groups that register together.)

 

Here’s the results when run against our whole house file.

 

Cons with Ids ending in even numbers: 114,305 (50,423 usable)

Cons with Ids ending in odd numbers:  129,078 (53,091 usable)

 

Hmmm - 47% even and 53% odd.

 

Not quite an even split. I am surprised that it is not closer. Can anyone think of why it would lean so much toward odd numbers?

 

When this is applied to the far smaller numbers of TR participants I'm sure the difference will seem less significant. And unless anyone can point out a flaw I think this is still a usable technique.

 

 

Jori Taylor

Brian,

 

Do you have a lot of "inactive" records, which could coincidentally be odd cons IDs?

Brian Mucha

Jori Taylor:

Brian,

 

Do you have a lot of "inactive" records, which could coincidentally be odd cons IDs?

Thanks for the reply Jori.

 

I'm not sure why inactive users would lean towards odd, but I updated my queries to only include active users that accept email.

Segmenting - Even Contact Ids

44,965 members



Segmenting - Odd Contact Ids

47,545 members

 

Seems like we simply have more odd Cons Ids. Strange.

Noah Cooper

Brian Mucha:

Thanks for the reply Jori.

 

I'm not sure why inactive users would lean towards odd, but I updated my queries to only include active users that accept email.

Segmenting - Even Contact Ids

44,965 members



Segmenting - Odd Contact Ids

47,545 members

 

Seems like we simply have more odd Cons Ids. Strange.

Brian,



Just curious -- at the point at which you're using a Query and a Group you manually create, why even use Coaching Emails anymore, and not Email Campaigns? In my mind, the advantage of using Coaching Emails has always been the way it simplifies audience selection, i.e. you can without having to run a Query select current or past participants, captains or non-captains, etc. If you're manually creating the Groups, though, you lose that.

Brian Mucha

Noah Cooper:

Brian,



Just curious -- at the point at which you're using a Query and a Group you manually create, why even use Coaching Emails anymore, and not Email Campaigns? In my mind, the advantage of using Coaching Emails has always been the way it simplifies audience selection, i.e. you can without having to run a Query select current or past participants, captains or non-captains, etc. If you're manually creating the Groups, though, you lose that.

Thats exactly why I'm looking for a query driven solution.

 

The idea here is that they can select whatever target groups they desire as usual, and then use my even/odd groups as the do-not-email group. They just send one supressing the evens, and then send the control supressing the odds.

Noah Cooper

Brian Mucha:

Thats exactly why I'm looking for a query driven solution.

 

The idea here is that they can select whatever target groups they desire as usual, and then use my even/odd groups as the do-not-email group. They just send one supressing the evens, and then send the control supressing the odds.

Ah OK, that makes sense. I was thinking you were using the even/odd groups as the target, not the do not mail group.

Noah Cooper

Noah Cooper:

Ah OK, that makes sense. I was thinking you were using the even/odd groups as the target, not the do not mail group.

And as to why you have more odds than evens -- note that when a new constituent record is created, its cons_id is not necessarily exactly 1 larger than the previous constituent created. The sequence does not always increment by 1. This is why if you look at your list of cons_ids you might see something like:

 

1000000100000110000021000003100114110011611001162100118110012811001282100128310012841001285100128610012871001288100128910012901001291100129210012931001294100129510012961001297100129810012991001300

 

Like you said, though, the resulting percentage of odd vs. even is still close enough to 50% that it seems like it should work.