If you haven’t heard of Visual Website Optimizer then I suggest you go check them out now. Do it. Go! Check them out, then come back and read the rest of this post.

Visual Website Optimizer (VWO), currently in beta, is poised to be a real player in the optimization space. Their elegant design and drop-dead simple approach to designing, launching, and monitoring tests is just sexy, I don’t know how else to describe it.

I have been part of the beta team testing VWO for over a month now and, as you can probably tell, I am impressed. So impressed, that I wanted to pull my VWO data into Omniture SiteCatalyst and Omniture Discover to extend the value of the product even further.

The integration between VWO and Omniture was super simple.

Step 1: If you haven’t already done so, designate one of your Custom Conversion variables as your optimization tracking variable.

Step 2: Copy the following code snippte and paste it into the doPlugins section of your s_code.js file.

var _combination = _vis_opt_readCookie(‘_vis_opt_exp_’+_vis_opt_experiment_id+’_combi’);

if(typeof(_vis_opt_comb_name[_combination]) != “undefined”){
s.eVar1=window._vis_opt_comb_name[1] + ‘:’ + _vis_opt_comb_name[_combination]
}

Step 3: That’s it!!!

NOTE: Using a similar approach, you can very easily integrate VWO with Google Analytics using ‘pageTracker._setCustomVar’.

I’m running an A|B test on the ‘Contact Jason’ section of my blog.

I’m testing if it’s best to show a photo of me or to show an icon of a telephone. When the integration code executes, my conversion variable is populated with the following:

s.eVar1=”Contact Icon:Photo”

‘Contact Icon’ is the site section that I’m testing and ‘Photo’ is the treatment version. For reference, you can see these values inside the details section of your test within VWO.

Although Visual Website Optimizer is still in it’s infancy, I see good things in their future. They really seem to value customer engagement and their product is sexy as hell.

P.S. I have a few beta invites to VWO, if you are interested, email me via my contact form.

Taking timeout with the kids to make some custard.

I was honored to participate in a webinar with Ian Michiels from the Aberdeen Group. Ian and I presented on digging deep below the surface to reveal hidden “gems” or opportunities to increase online conversions.

For more information on Omniture’s webinar series and to view additional webinars, please visit Omniture.com

Omniture
Leaders of the Analytics World
550 East Timpanogos Circle
Orem, UT 84097

Dear Omniture,

I would like to take this opportunity to express my heartfelt thanks to you for your amazing Summit 2010 held in Salt Lake City, Utah. Many conference attendees have also asked me to pass on their sincere appreciation for your efforts in this important undertaking.

Your skill in putting together such an amazing panel of industry experts and luminaries was very much appreciated by those present at the conference. Seth Godin and John Battelle injected their energy and excitement into all of us and we left Salt Lake City with a renewed passion for leading the next digital decade.

On both a professional and a personal level, I really appreciated the time that Adobe leadership took to share with us their vision of the future. Hearing from Shantanu Narayen and Josh James brought a sense of excitement about what the future has in store. I couldn’t have been more impressed with Brett Error, his concluding session is always looked forward to and his willingness to sit down with me individually was more than I ever expected.

As a member of Omniture’s Customer Advisory Board (CAB), I was able to spend two days before Summit started interacting with an amazing Product Management team led by Bill Ingram. Bill and his team should be applauded for the amazing work they are doing and the level of customer involvement they are maintaining. I loved the two days of CAB meetings, lunches, dinners, and causal conversations. Had those two days encompassed my entire Summit experience, I would have left very satisfied.

I was also able to attend Mind Meld organized by Matt Langie. Matt did a masterful job in bringing together a select group of industry leaders to discuss topics that are extremely important to the future of our industry. I was amazed at the level of talent present at Mind Meld and I’m sure I took much more from the group than I had to offer. Thanks Matt for putting together this wonderful event.

Omniture’s desire to be connected with their customers was a theme woven throughout the conference. Having a customer representative speaking in every breakout session was extremely valuable. The only complaint I had was the stress of choosing which session to attend. There were some tough choices with so many amazing sessions happening simultaneously.

I would be amiss if I didn’t mention Brent Watson. I was happy to see a technical flare being brought to the Omniture Summit. His hard work in developing a technical track and offering highly technical labs was a huge success and I can only see it growing year after year.

There were so many events, I wish I could have cloned myself to be everywhere at once. I stopped by the Engineering Services lab run by Matt Moss and got to hear about all the amazing projects they are working on with clients. I’m sure if I had the time, I would have heard similar stories from the other Client Services labs.

I’m sure I’m leaving things out that are worth mentioning, perhaps next year I will take my HD Flip camera and create a documentary of Summit from an attendees point of view. That would be some great cinema.

Again, thanks so much for an amazing conference, I left Salt Lake City yesterday with a tear in my eye knowing it was over. I have no doubt that it would not have been the success that it was without the presence of such an amazing group of Omniture and Adobe employees.

Please keep in touch and I’m in the area, so I wouldn’t mind dropping in for a visit every now and then.

Very sincerely,

jason thompson
humble analyst

Site measurement variables wanted! Would love to find gently used, in good condition site measurement conversion variables. If you have any that fit this description, please contact me with your asking price.

Five years ago, I remember sitting around a conference room, laughing at the thought that anyone would ever need more than 20 custom conversion variables. Oh, how wrong I was. Today, with one of my Omniture implementations busting at the seems, I’m looking everywhere for a spare variable to capture new data elements.

I have already gone through 3 audit rounds, where I systematically weighted each and every one of the 50 custom variables that we are actively using, at the end, the lowest weighted ones were on the chopping block, making way for newer tracking requirements.

Am I alone? Are there others out there that are facing a similar dilemma and if so, how are you managing your variable allocation and what do you do when you simply run out?

I think I have developed an unhealthy obsession with trying to understand how companies can harness the power of social media to better support their customers and create a sense of community and brand awareness. I have a laundry list of questions and I thought who better to start with then Ben Gaines, a.k.a. @OmnitureCare, the face of support and community at Omniture.

1. How did you get started using Twitter at Omniture?

BG: I was on my way to lunch with a manager and friend in the ClientCare organization, and he sort of nonchalantly mentioned that our social media strategist thought it would be a good idea to have someone out there, and that they would like me to do it. I had plenty on my plate, but it seemed like a great way to see what was going on in the user community and to help answer questions before they came in as support tickets, so I agreed. The funny thing is that I had never—not even once—used Twitter before setting up OmnitureCare. It was a learn-as-you-go process in many ways, but this was actually a good thing: it allowed me to play around and see what worked (and what didn’t). There were no pre-established rules.

2. Are you proactively looking for customers or potential customers with questions, if so what terms do you search for?

BG: Absolutely. Very few users know, before interacting with me, that Omniture is actively out there and listening—they don’t know to contact me directly—so I monitor brand mentions and butt in if I think I can help. That’s actually one of the coolest things about Twitter; people throw questions or complaints out there with little or no expectation of follow-up, and then they’re thrilled when they find out that we’re listening and want to help. So, in TweetDeck, I search on a bunch of keywords: Omniture (which includes #omniture), OMTR, SiteCatalyst, SearchCenter, Test&Target, etc. The tricky ones are products whose names are normal words in the English language, like Discover. You can’t search on them, because 99.999% of uses would be unrelated to Omniture, and it would be overload. So I have to hope that people say “Omniture Discover” instead of just “Discover.”

3. Do you search competitor terms? And if so, how do you approach them?

BG: I suspect that there is a lot of this going on, and I think it’s a good thing. Companies can’t hide anymore, which I’ve had employees of competitor companies jump in on Twitter conversions that I’m having in a very non-threatening way, and it’s actually kinda cool. It’s nice to see that we can be friendly and chat about web analytics overall, even amid fierce competition. I don’t see it as my role to watch them like a hawk—I’m out there to support Omniture users and build a sense of community, not to gather competitive intelligence. Sometimes I will monitor competitor terms (depending on my mood, I guess!), but it’s mostly just to see how they’re interacting with customers.

4. Do you turn text alerts on for anyone you’re following? If so, why?

BG: I used to do this, but it got to be too much—I kept getting excited that someone was texting me, only to find that it was another tweet that I had already seen. I’m in front of my computer all day at work and much of the time at home, so I think I’m sufficiently connected as it is. And even though I spend most of my time on Twitter talking about Omniture, I do keep my “All Friends” TweetDeck column around so I can see what people are discussing beyond Omniture and Web analytics.

5. How much time do you spend each week using Twitter?

BG: I would estimate that I am somewhere in the 30-40 hour range. That’s time spent actively tweeting or reading tweets, as well as researching questions that come to me via Twitter (or by e-mail after a conversation got too complex for Twitter). You aren’t the first person to ask me, and it’s a surprisingly tough number to estimate because I’m constantly in and out of TweetDeck for iPhone when I’m not at my desk and it’s always running while I’m at home, where I check in from time to time, depending on what I’m doing. And that number also does not include other social media aspects of my job—writing blog posts, participating in the Yahoo! Group for web analytics, etc. I keep meaning to gather some real data on this, but my brain is split across enough tasks that I never remember to start keeping track when I wake up on Monday morning. I can put it to you this way, though: checking Twitter is the last thing I do before I go to bed, and it’s the first thing I do when I wake up, so I’m involved at least intermittently literally throughout the day.

6. Does Adobe have an internal policy related to social networking that you follow?

BG: Omniture had one prior to the acquisition, and Adobe will soon release one as well. My role, of course, is slightly different from that of my colleagues who are out there on blogs, Twitter, Facebook, etc. I don’t have details on it yet, but I’m told that it will be similar to the policies implemented by other large companies.

7. You have become the “face” of OmnitureCare. Have you put a plan into place in the case you decided to leave Omniture?

BG: It’s worth mentioning that having a “face” on our social media presence has been a really powerful thing. It’s difficult to build trust and a sense of community without that. People need to know that they’re dealing with a real person, and that couldn’t be truer in the case of OmnitureCare. At present, I’m the only one who has ever logged in to that account.

We’ve put quite a bit of thought and planning into “next steps” for our social media strategy. What we’ve been able to do thus far has been valuable enough to users that it only makes sense to develop our thinking and try to do even more and do it better. I know you’ve seen @OmnitureUXD out there—Jessica is doing a fantastic job (once again, note that she is a real person and not a nameless, faceless corporate entity) and we are working on plans to add additional social media resources. There is definitely a contingency plan in case a bus runs over me tomorrow or something. We wouldn’t leave the community hanging.

8. How has using Twitter benefited Omniture? Anything that is measurable?

BG: A positive and a negative of Twitter is that its effect on customer relationships, technical support, etc. is difficult to measure in many ways. For example, of course we can tell when a Twitter conversation answers a user’s question and, therefore, saves that person from having to call ClientCare or their Account Manager. What’s tricky is that 50 other people—or more—may have seen that conversation and, as a result, not needed to pick up the phone themselves at some point in the future.

One thing that we can definitely measure is the amount of feedback and enhancement requests that we’ve received. I don’t have the number in front of me, but I know of at least one soon-to-be-released feature in SiteCatalyst that originated as a suggestion on Twitter. That’s one demonstration of powerful benefit to Omniture and to its entire user base.

And I think that’s the key. Ultimately, Twitter benefits Omniture inasmuch as it benefits our customers. Being out there talking, answering questions, and responding to complaints shows that we are invested in our users. We want to do whatever it takes to make them successful, because when they get value out of their relationship with us, everyone wins. And if that means I stay up late discussing dashboards and eVars, then that’s what I’ll do.

9. What makes you most excited about using Twitter for customer support?

BG: Twitter’s “reach” is awesome. As I suggested above, the fact that hundreds of users can learn about the product from a single person’s question means that support and education scale better than ever before. I love when I answer a user’s question and then I see that answer re-tweeted by people who weren’t even involved in the conversation.

You also can’t beat Twitter’s immediacy. Users can fire off questions and get a response from me (assuming I know the answer off the top of my head!) within seconds—and almost always within a few hours. On top of that, using Twitter means that the person asking the question didn’t need to interrupt his or her day to pick up the phone or log in to chat with a support agent.

There are certainly questions and issues that are more complex than is ideal for Twitter, and I do recommend that users contact ClientCare in those cases. But generally, we’ve found that technical support in 140 characters or less is a beautiful thing.

(Can you tell that I could write a book in answer to this question?)

10. What advice would you have for other companies who are attempting to use Twitter and other social networking sites as a customer support tool?

BG: Okay, I’m going to try to do this without straight-up plagiarizing Jeremiah Owyang’s presentation at Omniture Summit in 2009. Unfortunately for my attempt to be original, everything he said has been 100% true in my mind. His blog is a great resource. Anyway, here goes.

I’m naturally drawn to social media. Even though I had never used Twitter before launching OmnitureCare, I was a relatively early adopter of Facebook and blogging. That was huge because the concepts, the norms, and the ethos of social media were already a part of my attitude when I started. They can be difficult to teach—I have seen really bright, really passionate people struggle with customer support via social media because they just don’t quite understand what works and what does not. For whatever reason, I’ve been able to sense how to approach customer support via social media without much formal training. So the advice here is this: find the right person to represent your company out there. He or she doesn’t just know and love your products—which, don’t get me wrong, is hugely important—but also knows and loves social media. But you probably have some of these people in your organization already. After all, Omniture didn’t go out and seek to hire a community manager. I happened to be around already and was interested in the role.

Companies also need to empower their social media representative to affect change on behalf of customers. I have noticed that customers can sense it almost immediately when someone doesn’t actually have any ability to help. If you’re out there just to say that you’re out there, with no actual intent to improve your customers’ experience, that’s okay—but don’t promise support if you aren’t planning to back it up. Your community of customers will not take kindly to this sort of a run-around.

Finally, trust your social media representative. Hopefully, you have found someone who has a good sense of what should and should not be said—if not, making the person the public face of your support team is probably a bad idea. Let the person run with it. They might make mistakes from time to time—I certainly have—but in most cases, it seems to me that users are more forgiving of the occasional error than they might be when using more traditional communication channels because they understand the challenges of communicating effectively via social media. Let the person set goals, SLAs, etc. once they’ve gotten the lay of the land, and support those goals.

Someone who is passionate about social media, has good communication skills, knows your products and organization, and is empowered to truly help can do marvelous things for your company using social media. That much is crystal clear to us.

Already 2010 is feeling like the year of optimization. Everywhere I look, I’m seeing conversations about A|B and MVT testing, optimizing conversion flows, and understanding statistical significance.

When I first started running A|B tests, everything I did was on faith.  I had good intention, I measured all the key indicators, but I had no idea how to tackle the question of “yeah but, is it statistically significant?”  Then I began to crawl as I experimented with online calculators and eventually I moved on to building out my own formulas in Excel but still there was little confidence in myself, let alone the test results.

Eventually I began to experiment with testing tools like Google Web Optimizer, Amadesa, and Omiture Test & Target.  This seemed to make life so much simpler as all the questions I was being asked were answered right in the testing application.  Is it significant?  Amadesa says they are 98% confident in the results.  What is the lift we are seeing? Google Web Optimizer says its 8.5% and as a bonus it gives the confidence interval.

While I think it is extremely valuable to have your testing and optimization platform provide the key statistical measures that relate to your test, I think it is just as important to understand the math behind the reports, after all, you can’t call yourself a “car guy” or a “car girl” if you drive on the gauges alone and you don’t understand how the underlying systems work.

For our campaign, lets assume the following facts:

1. Our campaign has two treatments, a control and one alternative.
2. The control has had 4,008 visitors
3. The alternative has had 4,003 visitors
4. The control has had 377 conversions
5. The alternative has had 355 conversions

#1 – Conversion Rate

Conversion rate equals the number of conversions divided by the number of starts, in this example we are using visitors but this can be visits, impressions, unique starts, etc. depending on how you measure site conversion.

Conversion Rate (control) = 377 / 4,008 = 9.41%

Conversion Rate (alternative) = 355 / 4003 = 8.87%

#2 – Standard Deviation

Standard Deviation shows how much variation (measures the spread or dispersion of a set of data) there is from the “average” (mean).  As conversion rate is a binomial distribution, either a visitor converts or does not convert, the binomial distribution for variance is used:

Variance (control) = 9.41(1 – 9.41) = 0.09

Variance (alternative) 8.87(1 – 8.87) = 0.08

To calculate Standard Deviation from the variance, we take the square root of the variance:

Standard Deviation (control) = SQRT(0.09) = 0.29

Standard Deviation (alternative) = SQRT(0.08) = 0.28

#3 – Standard Error

The Standard Error is the estimated Standard Deviation of the error; the “noise” in the result.  The Standard Error is calculated in order to calculate to Signal-to-Noise ratio.

To calculate the Standard Error for the Control:

Standard Error (control) = SQRT(0.09 / 4008) = 0.005

To calculate the Standard Error for the alternative:

Standard Error (alternative) = SQRT((0.09 / 4008) + (0.08 / 4003)) = 0.006

# 4 – Signal-to-Noise Ratio

To calculate the Signal-to-Noise ratio:

Signal-to-Noise = (9.41 – 8.87) / 0.006 = 0.84

OK….stay with me…..we are almost there.

#5 Finally We Arrive At Confidence

We will make use of the Signal-to-Noise ratio to calculate confidence using the Student’s T-Test.

Student’s T-Test = 1 – TDIST(0.84,(4003 + 4008 -2),2) = 0.60

As reported by Test & Target, we are 60% confident in the current results.

Extra Credit: Confidence Intervals

The Confidence Interval shows how much your test results can vary and still be within a predetermined confidence level.  Standard confidence levels are 90%, 95%, 99%, and 99.5%.  Omniture Test & Target uses the 95% confidence level.

To calculate the Confidence Interval:

Confidence Interval = 1.96(0.28 / SQRT(4003)) = 0.008

1.96 is a constant in this formula.  1.96 is equal to z*, which is taken from a Standard Normal Critical Values table based on 95%  Confidence Level.  The Standard Normal Critical Values Table can be found in any introductory level statistics book.

The formulas in this post have been provided by Omniture consulting. The screenshots have been taken from Omniture Test & Target and have been modified for the purpose of this example.

Whether you are validating the deployment of a new measurement solution, testing enhanced page tracking, or debugging an issue with your site, having quick access to the data your site measurement tool is collecting is a must.

I have used debugger tools provided by the web analytics vendors, I have used network analyzers like Charles, Ethereal, and Wireshark and tools built specifically for web analytics like WASP.

However, I find myself always going back to HttpFox. It’s Clean. It’s simple. It’s so easy to use, after I was introduced to it by my friend Michael Sanders, I went on to train everyone from quality assurance engineers to CEOs on how to use HttpFox to answer the question “what variables are we capturing on this page?”

How do I use HttpFox to validate my web analytics page tags?

1. Add the HttpFox Firefox extension to your browser.

2. Once you have installed the extension and have restarted Firefox, you will notice a new icon in your status bar.

3. Activate HttpFox by clicking the icon in the status bar.

4. Click the Start button and navigate to a page on your site. You will see a long list of records rolling in, don’t worry, in the next step we will talk about searching for the one record that matters.

5. Depending on what measurement solution you are using, the search string will be different. Here is a list of search strings that I use most often:

Google Analytics = _utm.gif
Yahoo Analytics = a.analytics.yahoo
Omniture = /b/ss/

6. Click on the Query String tab and you can quickly see all the name-value pairs that are being sent to your analytics provider.

With all the chatter on Twitter about the upcoming Omniture Summit, I began to reminisce about last year’s summit. Last year at this time, my colleague Hila Strong and I were just finding out that we had won the inaugural Omniture Developer Challenge. I thought now was as good a time as any to celebrate the anniversary of our accomplishment.

The Break Even Calculator calculates the $ value of visitors at a funnel conversion event. This value is then compared before and after a site revision or marketing event.

This can be done using a simple calculated metric in SiteCatalyst, however the Break Even Calculator goes one step further and calculates a break even point. What your conversion rate SHOULD be for revenue to break even or increase.

The Break Even Calculator will work for any conversion funnel on your site. As part of the installation process, start events and completion events can be defined. These events could be Omniture pre-defined events, custom events, or a combination of both. The conversion rates will be calculated using the two events selected by the end-user.

If you are interested in using the calculator or simply interested in seeing what we developed, feel free to download the code base and documentation from the product page.  You are more than welcome to extend the application to best meet your business needs. If you find the application useful, extend the application, or have general feedback, I’d love to hear about it.

I set out to test the data accuracy of Omniture’s ClickMap tool and the heat map tool offered by CrazyEgg. I quickly found it difficult and frustrating to run a clean test. So, I decided to simplify the test and measure the perforce of each tool separately.

I also want to be clear, that the point of this experiment is not to debate Omniture ClickMap vs. CrazyEgg. They are two separate products, with different pricing and distribution structures.

I see ClickMap and CrazyEgg as being complimentary tools. ClickMap is great for measuring ongoing site behavior, it can easily be tied to site conversion, and is great for measuring sites with dynamic content. CrazyEgg is best suited for structured tests, the run for a pre-determined period of time and for measuring exact click locations to better understand creative placement.

NOTE: This test covers the out-of-the-box deployment of ClickMap. I did not test the impact of dynamic pages. For sites that use highly dynamic pages, Omniture recommends using s_ObjectID tracking. Speak with your Omniture representive about updating your code to automatically assign unique IDs for link tracking. Also refer to my post Omniture ClickMap & s_ObjectID

I tested the following link types:

1. CSS styled drop down menus
2. Text links that open in a new window
3. Text Links that open in the same window
4. HTML Form button
5. Photo wrapped with an href
6. A button with no destination link
7. Flash Banner

Drop Down Menus

I was pleasantly surprised to see that ClickMap accurately, I clicked each link 5 times and the overlay reported 5 clicks on each link, tracked clicks on drop down menus. Again, the accuracy will depend on the technology you use to create your drop down menus but for CSS styled menus, ClickMap works great, with one exception. If the link in the drop down menu opens in a new window (i.e. target=”_blank”), ClickMap did not record a click on this link.

Text Links (New Window)

I tested two different link types.  One I call simple links, a standard link with no query string parameters and complex links, a link that contains query string parameters. In this test, the links open in a new window (i.e. target=”_blank”).  Again, 5 clicks equaled 5 clicks in the ClickMap report.

Text Links (Same Window)

The same two links from the previous test, this time the links opened in the same window, and with the previous test 5 for 5 on the click tracking.

HTML Form Button

I deployed a simple HTML form with a button that navigates to a new page.  I had heard complaints that buttons were not accurately tracked with ClickMap but I found in my test 5 clicks were recorded 5 times.   It should be noted that the action of the button is a URL and not a function, if the action was a function, my guess is that the button would not be properly tracked with ClickMap.

Photo Link

Again, another 5 out of 5!  Go DWill!!!

A Big Button That Does Nothing

As expected, ClickMap did not record clicks on the big button that goes nowhere.

Flash Banner

ClickMap did not record any clicks on the Flash banner.

NOTE: Click tracking on Flash applications is not supported by the out-of-the-box ClickMap solution, however Omniture does have a Flash tracking solution which does support ClickMap tracking, please contact your Omniture representative.

By no means was this test scientifically rigorous, there are many different factors that can impact the accuracy of the ClickMap data, however what I hoped to prove was that ClickMap is directionally correct.

In the end, ClickMap should be used as an indicator, as to which areas of your site navigation are working and which areas warrant further analysis. No conclusions should be reached based on the results of ClickMap data alone, however as part of a more formal analysis and for use as a directional indicator, ClickMap is a very solid tool that should not be dismissed.