Already 2010 is feeling like the year of optimization. Everywhere I look, I’m seeing conversations about A|B and MVT testing, optimizing conversion flows, and understanding statistical significance.
When I first started running A|B tests, everything I did was on faith. I had good intention, I measured all the key indicators, but I had no idea how to tackle the question of “yeah but, is it statistically significant?” Then I began to crawl as I experimented with online calculators and eventually I moved on to building out my own formulas in Excel but still there was little confidence in myself, let alone the test results.
Eventually I began to experiment with testing tools like Google Web Optimizer, Amadesa, and Omiture Test & Target. This seemed to make life so much simpler as all the questions I was being asked were answered right in the testing application. Is it significant? Amadesa says they are 98% confident in the results. What is the lift we are seeing? Google Web Optimizer says its 8.5% and as a bonus it gives the confidence interval.
While I think it is extremely valuable to have your testing and optimization platform provide the key statistical measures that relate to your test, I think it is just as important to understand the math behind the reports, after all, you can’t call yourself a “car guy” or a “car girl” if you drive on the gauges alone and you don’t understand how the underlying systems work.
Let’s walk through an example campaign to understand how Omniture Test & Target calculates the statistics behind the results.
For our campaign, lets assume the following facts:
- Our campaign has two treatments, a control and one alternative.
- The control has had 4,008 visitors
- The alternative has had 4,003 visitors
- The control has had 377 conversions
- The alternative has had 355 conversions
#1 – Conversion Rate

Conversion rate equals the number of conversions divided by the number of starts, in this example we are using visitors but this can be visits, impressions, unique starts, etc. depending on how you measure site conversion.

Conversion Rate (control) = 377 / 4,008 = 9.41%
Conversion Rate (alternative) = 355 / 4003 = 8.87%
#2 – Standard Deviation
Standard Deviation shows how much variation (measures the spread or dispersion of a set of data) there is from the “average” (mean). As conversion rate is a binomial distribution, either a visitor converts or does not convert, the binomial distribution for variance is used:

Variance (control) = 9.41(1 – 9.41) = 0.09
Variance (alternative) 8.87(1 – 8.87) = 0.08
To calculate Standard Deviation from the variance, we take the square root of the variance:

Standard Deviation (control) = SQRT(0.09) = 0.29
Standard Deviation (alternative) = SQRT(0.08) = 0.28
#3 – Standard Error
The Standard Error is the estimated Standard Deviation of the error; the “noise” in the result. The Standard Error is calculated in order to calculate to Signal-to-Noise ratio.
To calculate the Standard Error for the Control:

Standard Error (control) = SQRT(0.09 / 4008) = 0.005
To calculate the Standard Error for the alternative:

Standard Error (alternative) = SQRT((0.09 / 4008) + (0.08 / 4003)) = 0.006
# 4 – Signal-to-Noise Ratio
To calculate the Signal-to-Noise ratio:

Signal-to-Noise = (9.41 – 8.87) / 0.006 = 0.84
OK….stay with me…..we are almost there.
#5 Finally We Arrive At Confidence
We will make use of the Signal-to-Noise ratio to calculate confidence using the Student’s T-Test.

Student’s T-Test = 1 – TDIST(0.84,(4003 + 4008 -2),2) = 0.60

As reported by Test & Target, we are 60% confident in the current results.
Extra Credit: Confidence Intervals

The Confidence Interval shows how much your test results can vary and still be within a predetermined confidence level. Standard confidence levels are 90%, 95%, 99%, and 99.5%. Omniture Test & Target uses the 95% confidence level.
To calculate the Confidence Interval:

Confidence Interval = 1.96(0.28 / SQRT(4003)) = 0.008
1.96 is a constant in this formula. 1.96 is equal to z*, which is taken from a Standard Normal Critical Values table based on 95% Confidence Level. The Standard Normal Critical Values Table can be found in any introductory level statistics book.
Now that we have determined our Confidence Interval, we can calculate the +- of our test results:
High Bound = 8.87% + 0.008 = 9.75%
Low Bound = 8.87% – 0.008 = 7.99%
Giving us the Confidence Interval as reported in Test & Target of 7.99% to 9.75%, meaning given the current volume, we are 95% confident that our conversion rate will fall between 7.99% and 9.75%.
The formulas in this post have been provided by Omniture consulting. The screenshots have been taken from Omniture Test & Target and have been modified for the purpose of this example.
Omniture Summit 2010
Omniture
Leaders of the Analytics World
550 East Timpanogos Circle
Orem, UT 84097
Dear Omniture,
I would like to take this opportunity to express my heartfelt thanks to you for your amazing Summit 2010 held in Salt Lake City, Utah. Many conference attendees have also asked me to pass on their sincere appreciation for your efforts in this important undertaking.
Your skill in putting together such an amazing panel of industry experts and luminaries was very much appreciated by those present at the conference. Seth Godin and John Battelle injected their energy and excitement into all of us and we left Salt Lake City with a renewed passion for leading the next digital decade.
On both a professional and a personal level, I really appreciated the time that Adobe leadership took to share with us their vision of the future. Hearing from Shantanu Narayen and Josh James brought a sense of excitement about what the future has in store. I couldn’t have been more impressed with Brett Error, his concluding session is always looked forward to and his willingness to sit down with me individually was more than I ever expected.
As a member of Omniture’s Customer Advisory Board (CAB), I was able to spend two days before Summit started interacting with an amazing Product Management team led by Bill Ingram. Bill and his team should be applauded for the amazing work they are doing and the level of customer involvement they are maintaining. I loved the two days of CAB meetings, lunches, dinners, and causal conversations. Had those two days encompassed my entire Summit experience, I would have left very satisfied.
I was also able to attend Mind Meld organized by Matt Langie. Matt did a masterful job in bringing together a select group of industry leaders to discuss topics that are extremely important to the future of our industry. I was amazed at the level of talent present at Mind Meld and I’m sure I took much more from the group than I had to offer. Thanks Matt for putting together this wonderful event.
Omniture’s desire to be connected with their customers was a theme woven throughout the conference. Having a customer representative speaking in every breakout session was extremely valuable. The only complaint I had was the stress of choosing which session to attend. There were some tough choices with so many amazing sessions happening simultaneously.
I would be amiss if I didn’t mention Brent Watson. I was happy to see a technical flare being brought to the Omniture Summit. His hard work in developing a technical track and offering highly technical labs was a huge success and I can only see it growing year after year.
There were so many events, I wish I could have cloned myself to be everywhere at once. I stopped by the Engineering Services lab run by Matt Moss and got to hear about all the amazing projects they are working on with clients. I’m sure if I had the time, I would have heard similar stories from the other Client Services labs.
I’m sure I’m leaving things out that are worth mentioning, perhaps next year I will take my HD Flip camera and create a documentary of Summit from an attendees point of view. That would be some great cinema.
Again, thanks so much for an amazing conference, I left Salt Lake City yesterday with a tear in my eye knowing it was over. I have no doubt that it would not have been the success that it was without the presence of such an amazing group of Omniture and Adobe employees.
Please keep in touch and I’m in the area, so I wouldn’t mind dropping in for a visit every now and then.
Very sincerely,
jason thompson
humble analyst