A/B Testing over different days

Aaron · December 2009

We'd like to try sending our message over different days, not different content. Is there a way to do an A/B test (or the equivolent) on two different schedules? I'd like to send the message today to 1/2 the list, and tomorrow to the other half. Do I have to create a pair of groups to do this, or is there a way to make this work using a function like A/B testing?

Sally Heaven · December 2009

I don't think you can use the A/B functionality to schedule messages for two different days, but you could easily configure two different Deliveries of the same message and just schedule those Deliveries to be sent on different days at the same time of day so you . To create two different Groups randomly, you can use Contact Range in Query. Instructions for doing that are in this thread: http://community.customer.convio.com/message/6522#6522

Aaron · December 2009

Sally Heaven:
I don't think you can use the A/B functionality to schedule messages for two different days, but you could easily configure two different Deliveries of the same message and just schedule those Deliveries to be sent on different days at the same time of day so you . To create two different Groups randomly, you can use Contact Range in Query. Instructions for doing that are in this thread: http://community.customer.convio.com/message/6522#6522

Well that looks like a workable solution, if a bit ugly.

Anyone done any testing on the thoery behind this? Do the numbers stay pretty uniformly distributed in real datasets? It would be interesting to see if using the last 2 digits made the distrobution better or worse than using just the last digit.

TJ Spinks · December 2009

It is possible to use the A/B Test feature to send a message out in stages even over days.

Click Run an A/B Test

Provide a Sample percentage

Click Next

Click Schedule

By default the delivery will initially be configured for immediate scheduling however you may schedule the delivery to occur when desired. repeat the process for subsequent A/B Test deliveries.

The audience target group list should remain the same for either delivery. The Convio product will gather the specified percentage from the deliverable audience. Subsequent deliveries will exclude the constituents that have already been targeted.

Sally Heaven · December 2009

TJ Spinks:
It is possible to use the A/B Test feature to send a message out in stages even over days.

Click Run an A/B Test

Provide a Sample percentage

Click Next

Click Schedule

By default the delivery will initially be configured for immediate scheduling however you may schedule the delivery to occur when desired. repeat the process for subsequent A/B Test deliveries.

The audience target group list should remain the same for either delivery. The Convio product will gather the specified percentage from the deliverable audience. Subsequent deliveries will exclude the constituents that have already been targeted.

Hey, that's even easier - thanks, TJ!

Re: stats. A few years ago there was a lot of chatter about "what's the best day to send an email." The conventional wisdom was that Mondays and Fridays were bad, and that Tuesday/Wednesday/Thursdays were the best days.

Then some people started testing sending email on Mondays and Fridays, and lo and behold, they got good results. Some thought it was because most other nonprofits were NOT emailing on those days, so you had a pretty clear playing field. The word got around that "Monday is a great day to send an email" and everyone rushed to start sending emails on Mondays. Imagine a boat where everyone keeps running toward one side of the deck, then the other side.

The best advice that I have seen is that trends are informative, but your list may perform differently than the average for a variety of reasons. Maybe Mondays are good for your organization, maybe Wednesdays are better, or maybe they are equal. But one thing is certain - if you email when something big is happening, then your messages will perform better. For example, if you work on judicial issues, and send an email when a Supreme Court Justice retires, that message will get great traction even if you send it on a Saturday - because your list is interested in that type of thing.

Anyway, I think that it's a good test that is worth running to measure how your own list performs.

Something else I've been wondering about is splitting up a big send by time zones - creating Groups of states that are in the Eastern time zone, Central, Mountain, and Pacific, for example, and timing the sends to arrive in people's inboxes around 9:30am local time. Has anyone ever tested whether the open/click-through/action rates vary, or don't vary, if the message arrives around 9:30am local time instead of, say, 6:30am local time?

A/B Testing over different days

Comments

Categories