Already 2010 is feeling like the year of optimization. Everywhere I look, I’m seeing conversations about A|B and MVT testing, optimizing conversion flows, and understanding statistical significance.

When I first started running A|B tests, everything I did was on faith.  I had good intention, I measured all the key indicators, but I had no idea how to tackle the question of “yeah but, is it statistically significant?”  Then I began to crawl as I experimented with online calculators and eventually I moved on to building out my own formulas in Excel but still there was little confidence in myself, let alone the test results.

Eventually I began to experiment with testing tools like Google Web Optimizer, Amadesa, and Omiture Test & Target.  This seemed to make life so much simpler as all the questions I was being asked were answered right in the testing application.  Is it significant?  Amadesa says they are 98% confident in the results.  What is the lift we are seeing? Google Web Optimizer says its 8.5% and as a bonus it gives the confidence interval.

While I think it is extremely valuable to have your testing and optimization platform provide the key statistical measures that relate to your test, I think it is just as important to understand the math behind the reports, after all, you can’t call yourself a “car guy” or a “car girl” if you drive on the gauges alone and you don’t understand how the underlying systems work.

For our campaign, lets assume the following facts:

1. Our campaign has two treatments, a control and one alternative.
2. The control has had 4,008 visitors
3. The alternative has had 4,003 visitors
4. The control has had 377 conversions
5. The alternative has had 355 conversions

#1 – Conversion Rate

Conversion rate equals the number of conversions divided by the number of starts, in this example we are using visitors but this can be visits, impressions, unique starts, etc. depending on how you measure site conversion.

Conversion Rate (control) = 377 / 4,008 = 9.41%

Conversion Rate (alternative) = 355 / 4003 = 8.87%

#2 – Standard Deviation

Standard Deviation shows how much variation (measures the spread or dispersion of a set of data) there is from the “average” (mean).  As conversion rate is a binomial distribution, either a visitor converts or does not convert, the binomial distribution for variance is used:

Variance (control) = 9.41(1 – 9.41) = 0.09

Variance (alternative) 8.87(1 – 8.87) = 0.08

To calculate Standard Deviation from the variance, we take the square root of the variance:

Standard Deviation (control) = SQRT(0.09) = 0.29

Standard Deviation (alternative) = SQRT(0.08) = 0.28

#3 – Standard Error

The Standard Error is the estimated Standard Deviation of the error; the “noise” in the result.  The Standard Error is calculated in order to calculate to Signal-to-Noise ratio.

To calculate the Standard Error for the Control:

Standard Error (control) = SQRT(0.09 / 4008) = 0.005

To calculate the Standard Error for the alternative:

Standard Error (alternative) = SQRT((0.09 / 4008) + (0.08 / 4003)) = 0.006

# 4 – Signal-to-Noise Ratio

To calculate the Signal-to-Noise ratio:

Signal-to-Noise = (9.41 – 8.87) / 0.006 = 0.84

OK….stay with me…..we are almost there.

#5 Finally We Arrive At Confidence

We will make use of the Signal-to-Noise ratio to calculate confidence using the Student’s T-Test.

Student’s T-Test = 1 – TDIST(0.84,(4003 + 4008 -2),2) = 0.60

As reported by Test & Target, we are 60% confident in the current results.

Extra Credit: Confidence Intervals

The Confidence Interval shows how much your test results can vary and still be within a predetermined confidence level.  Standard confidence levels are 90%, 95%, 99%, and 99.5%.  Omniture Test & Target uses the 95% confidence level.

To calculate the Confidence Interval:

Confidence Interval = 1.96(0.28 / SQRT(4003)) = 0.008

1.96 is a constant in this formula.  1.96 is equal to z*, which is taken from a Standard Normal Critical Values table based on 95%  Confidence Level.  The Standard Normal Critical Values Table can be found in any introductory level statistics book.

The formulas in this post have been provided by Omniture consulting. The screenshots have been taken from Omniture Test & Target and have been modified for the purpose of this example.

Step 1: Update your s_code.js file with the Omniture TnT Integration Plugin – Version 1.0

/*
* TNT Integration Plugin v1.0
*/
s.trackTNT = new Function("v", "p", "b", ""
+ "var s=this,n='s_tnt',p=p?p:n,v=v?v:n,r='',pm=false,b=b?b:true;if(s."
+ "getQueryParam){pm=s.getQueryParam(p);}if(pm){r+=(pm+',');}if(s.wd[v"
+ "]!=undefined){r+=s.wd[v];}if(b){s.wd[v]='';}return r;");

Step 2: Determine if you will include the integration call in your standard SiteCatalyst page tag or if you will require to use a Custom Link call.

Is your SiteCatalyst code block located after the last mBox call on your page?

YES!  Simply add the following before you call s.t():

var tntInput = s.trackTNT();

s.eVar47 = s.tnt = tntInput;

Is your SiteCatalyst code block located before the last mBox call on your page?

YES! This will take a bit more work but, if you ask me, worth it.  I have our SiteCatalyst code directly following the <body> tag.  This has greatly improved our data accuracy.  Moving the code to accommodate an integration with Test & Target was simply out of the question.

To send the Test & Target information to SiteCatalyst using a Custom Link call add the following after all mBoxes on the page:

<script>
var tntInput = s.trackTNT();

trackTNT(s_account, tntInput);

function trackTNT(s_account, tntInput){
	var s=s_gi( s_account );

	s.linkTrackVars="tnt,eVarN"
	s.linkTrackEvents="None"

	//variable for TNT classifications
	s.eVarN = s.tnt = tntInput;

	s.tl( true , 'o' , 'For tracking TNT' );
	}
</script>

Step 3: Classify the Test & Target Campaign and Recipe ID. Omniture has a behind the scene integration to automatically classify your TnT integration. However, I have been told it is limited to one (1) report suite. I found it easy enough to simply do this myself.

Step 3a: Create a Custom Conversion variable to capture your Test & Target integration IDs

Step 3b: Classify your conversion variable to include a text based classification called ‘Treatment’

Step 3c: This part is a little hacky but hey, it works.  Log into Test & Target and view your campaign.  The TnT plugin will return a value like ‘4503:0:0,’.  To get the values hover your mouse over the preview icon for each treatment

Now, look down in your status bar and you will be able to see the values you need to write down.

Now you can compile a list of Ids and treatment names

4503:0:0,       Control

4503:1:0,        Improved Headline

Step 3d: Download a SAINT template for your TnT classification.  Map each treatment id to your friendly treatment name.  Upload.

Step 4: Analyze your results.  Test & Target is great for giving you a dashboard view of your treatments and how each one is converting (with statistical significance).  However, you still are not able to fully understand how your tests are changing user behavior inside your site.  This is where SiteCatalyst and more powerfully, Discover, comes into play.  Use SiteCatalyst and Discover to slice your visitor population to better understand how your Test & Target campaigns are influencing user behavior beyond the conversion check point.

And by uploading a friendly treatment name via SAINT, this job is made much easier.

]]> http://emptymind.org/integrate-test-target-with-omniture-sitecatalyst/feed/ 2