#22 – Start A/B or Multivariate Testing

This is my series of 30 brief interactive marketing tips. I’m posting each business day at 8:30 a.m. EST. Catch them all by subscribing to my blog or bookmarking the site. All of the posts will be tagged “30 interactive marketing tips” and listed on this page.

#22 – Start A/B or Multivariate Testing

All of the creative genius or market research in the world can’t predict the way your customers and prospects will react to your site in the moment. At the same time, looking at web analytics data will only tell you how things happened in the past and it can be difficult to distinguish exactly what influenced results when multiple variables are in play.

Instead of trying to guess, test. Pick factors on your site to test and create variations of those factors (e.g. headlines, images). Use any of the a/b and multivariate testing tools on the market to facilitate the process. For a list of testing vendors, check out my page on free a/b and multivariate testing resources.

The Surprisingly Important Element of Successful Testing

Source: http://bit.ly/aPzFlC

I’ve had a lot of positions at my current company (ClickEquations, formerly Commerce360), but one of the ones I most enjoyed was a multivariate testing project I did with a large entertainment provider.

In preparing for the project, I read a ton about testing, what to test, past tests that worked, etc. There is no shortage of information about what to test or how to test it (see. www.WhichTestWon.com, for example).

What surprised me about the experience was that very little time was spent talking about the most important elements of a successful test: people.

Specifically, the people you need to help you with a test.


CRO & Big Companies

If you manage your own website, launching a test happens in an afternoon and you’re making the calls. If you’re heading up a conversion rate project for a website with hundreds of thousands of visitors a day, it quickly becomes a project convincing people to help you test with some conversion rate optimization attached.

That was the case for the company I worked with, as it was one of their first forays into testing. Getting momentum required buy-in from several key stakeholders:

  1. Champion – We had an awesome internal advocate who prioritized the project and rallied the troops. He served as the central point of contact to coral everything on the client side while my coworker (the account manager) and I got to focus on the testing.
  2. Executive Sponsorship – The client had to lay out money for both software (we used SiteSpect) and our consulting fees. Working on our project also meant diverting hours for contributing departments (see below) from other projects to testing.
  3. Development Buy-In – Every testing project involves some changes to your site in one way or another: DNS changes, tags in footers, Mboxes, etc. The bigger the site, the more cautious the IT and development team. You need to get in their sprint cycle. I recommend you choose a technology that allows you to ask for help from development once and then permits changes on the fly afterward.
  4. Creative – Someone has to create all of those test variations and in bigger companies, they generally have to be sensitive to brand guidelines or other restrictions. More importantly, you need an ally on the creative team who is will to let you butcher their work and possibly test something that, from a design perspective, they may not necessarily agree with.
  5. Analytics – Chances are, there is already someone in the company doing analytics or, as was the case for my client, statistics. You’re going to need access to analytics data to design an effective test and they might even review it with you.


Converstion Team Optimization

Testing can fundamentally change how a business operates and where decision making power lies. It also involves new and different business processes. That is to say, testing means change. People, especially bigger companies, are often slow to change.

In my experience, a great Champion and Executive Sponsor set the tone for the rest of the team. They approve budgets, timelines and the allocation of resources. A great process and communication help win over the rest of the team, but that’s for another blog post.

The State of Search Analytics

This post originally appeared on Search Engine Watch, where I will be a (semi-regular) contributor

You can’t make money in web analytics just by looking at reports.” With that, SES Advisor and New York Times bestselling author Bryan Eisenberg set the stage for the state of analysis in search marketing.

As budgets flow from offline to online, they’re moving disproportionately to search. Search has always excelled at accountability, but as the spend increases the bar is being raised.

We all have and know what web analytics are. Data quantity isn’t the issue, action quantity is.

Matt Bailey (SiteLogic) summed it up perfectly with the word “velleity”. It means “the desire to do something, but not enough of a desire to take action.” It describes the feeling of marketers who have been burnt by reporting that isn’t actionable.

Over the course of the Analytics, Conversion and Attribution sessions during the first day at SES Chicago, the speakers each addressed three of the core challenges in fighting velleity and taking action on your data:

  1. Prioritization
  2. Segmentation
  3. Process

1. Prioritization
Web analytics has come a long way from the days of log file analyzers used by the IT department. Naturally, people tend to want to make tools bigger, faster and more complicated, however “if you have a tool, everything looks like a nail,” says Jim Sterne of eMetrics and the Web Analytics Association.

The problem is “what should I look at?” and “what do you actually analyze?”

Jim suggests you bucket and prioritize 4 ways

  1. Revenue that is at risk
  2. Anything the boss’s boss’s boss asks for
  3. Requests that do not overwhelm the department
  4. Analysis that requires an analyst rather than simply being self-served

Your job is to find a specific process and optimize that, for example a shopping cart, experience from a search campaign or lead generation page. Go to your data and ask “How can you help me optimize the process?”

In another session, Jim reminded us that traditional web analytics tools tell you “where the problem is and which one is most significant at the moment.” To really understand why, you have to augment those tools with usability testing and surveys.

In a nod to his upcoming book, Jim also shared a framework for prioritizing analysis for one of the most discussed topics at the conference–social media:

  • Reach – How many people could possibly see my message (blog mentions x blog readers)
  • Frequency – How often is my message being discussed, ex: comments on a blog post
  • Influence – The more authority the author, the bigger the halo effect and potential viral spread
  • Sentiment – Having lots of people talk about you is great… unless they say bad things. The tools aren’t great yet, so hire interns.
  • Outcomes – Did they take the action you wanted them to?

2. Segmentation
Of course, even if you prioritize your analysis, there is no such thing as an average user. People who come to your website aren’t looking for the exact same thing. They all have different needs and wants. We can’t treat visitors the same. Unique visitors aren’t all the same. Segmentation matters. We have to look at what people wanted when they came to the site.

But, as Matt Bailey (SiteLogic) says “analytics dashboards tend to be as usable as Ikea furniture instructions.” The cure for dashboards that suffer from average-itis is to segment.

Matt shared his 7 favorite tips:

  1. Create Keyword Buckets “Segments” – Start with the big buckets, e.g. digital cameras and then develop smaller buckets, e.g. digital cameras, professional digital cameras, etc.
  2. Segment Based On Acquisition (Channels) – Ask “which source is bringing the best traffic?” Matt shared an interesting a pattern that drives engagement: the more engaged a visitor was with a message about your company or product before they got to your site, the more likely they’ll be engaged after. Twitter is at the bottom of the this inverted pyramid: Blogs & Articles, YouTube, Forums, Search, Social News and Twitter.
  3. Segment Your Bounce Rates – Typically if it’s high, it’s a word that means something totally different in another industry. You have to look at the context. Sometimes it’s not just their intent, sometimes it’s your design (ex: Fluid width design on a large screen monitor)
  4. Segment Your Content - Divide your analysis by Persuasive pages (which ones drive conversions), entry pages, time on page and search behavior
  5. Segment Behavior – Divide the behavior on your site, but make it more accessible by giving the data friendly names for the audience your presenting to (ex: someone with “O” in their title would understand)
  6. Segment Entry Points – You could have the right ranking on the wrong page
  7. Take Action – After you segment, take some action!

In his 21 Secrets of Top Converting Websites (which he will deliver as the keynote of SES London), Bryan emphasized how important it is to also analyze trends by segments in search. For example, focus on “what’s changed” reports of keywords that are rising and falling the fastest by CPC or Revenue, instead of just analyzing the top 10.

3. Process
Of course, prioritization and segmentation are both part of a larger web analytics process. As Bryan reminded us, “to do web analytics correctly, you have to make a to do list regularly.” An actionable list addresses:

  • What marketing efforts or parts of your site have challenges
  • What you think needs to be improved
  • What things you want to test
  • What efforts you should do less of
  • What efforts you should do more of

The core of an actionable web analytics process is data driven decision making. Nothing does that better than testing, using the tools that fit your questions and budget:

Part of a successful process is optimizing for your conversion rate, which often hovers around 2%. Dr. Phil Mui challenged us to expand by process by asking “How do you measure success for the rest of the 98%?”

First, map out the micro-conversions that fall into that 98% for you business type, for example:

  • Ecommerce – product research account signups, contact us
  • Non-profits – volunteer leads, promotion of the cause
  • Video Sites – Account registration, newsletter signups, premiums account signups
  • Blogs – Links to your blog, comments on your posts

It’s also important that you optimize holistically in your process. Don’t just test elements on your landing page; also test which landing page choice makes sense, for example: a category page vs. a product page.

You can’t forget to test what people see before they get to your site. Optimize search creatives for conversion. Test multiple ads and measure which one drove conversion.

Regardless of the tools and process you use, the message every speaker conveyed is that more action is better than more data, so measure, focus, optimize and repeat.

I No Longer Work at Commerce360

By WTL Photos - http://www.flickr.com/photos/wtlphotos/

By WTL Photos - http://www.flickr.com/photos/wtlphotos/

I’m happy to report that I no longer work at Commerce360.  I had a good time while it lasted, but it was time for something different.

And that something different is ClickEquations.  We’ve re-branded Commerce360 as ClickEquations to focus on our advanced pay per click software.  (Sorry, it was a little late for April Fool’s but I needed a dramatic intro :-) )

This is a bit of a personal post, which I almost never do, but worth reading if you care about paid search.

A Trip Down Memory Lane

Back in 2007, I left Refinery (now G2 Philly, a division of Grey Interactive) to join Commerce360.  At the time, I was still a full time web analyst with strategic aspirations.

Commerce360 was then focused on being the next generation of agency: a team of smart people optimizing across channels based with data driven decisions.  I joined the team as an analyst on a course toward full time strategy.But, as I soon learned is the norm for startups (this is my first), we shifted focus entirely on paid search and SEO.

The Birth of ClickEquations

clickequationsThe vision of the company was always to have smart people supported by killer technology.  Search is among the most data intensive channels out there, so it was the obvious place to start looking for a tool that could do the heavy lifting while we focused on strategy and optimization.

After surveying the market, we just couldn’t find anything good enough, including Omniture Search Center.  Too much money had been spent building tools focused on a search engine-centric view of managing paid search instead of a customer and practitioner centric view.  It was a completely flawed way of attacking the problem and even the “best” of what was on the market was an expensive and clumsy solution at best.  So we hired a development team and began building our own tool: ClickEquations.

A Slight Career Detour

Search marketing is strategic, but it’s not the same as developing cross-channel strategy. Without a pure strategist role, I ended becoming a strange hybrid: part Strategic Account Manager (client relations), part multivariate tester, part guy-who-does-random-things.

bewitchedWhen people ask why I got into advertising and marketing, I give them the same answer: Bewitched.  I used to watch the show as a kid and was strangely fascinated with the ad lifestyle (portrayed as a sanitized and more kitsch version of today’s Mad Men). I figured it was 3 martini lunches and everyone got to do fun pitches all of the time, right?

Without nose twitching magic powers, Account Management is a fairly high pressure job.  You’re the middle man between clients with high expectations (sometimes disproportionate to what they’re paying) and limited budgets and a services team with limited time and all of the pressure for results.

On the plus side, it’s a great way to learn a lot of businesses quickly, master contracts (write 20 contracts in 6 months and you pick up a few things) and practice the fine art of expectations management:  “Yes, we can do that, but we’ll have to push this off and cut that down by 20%”.

I worked on SEO and paid search engagements with clients from startups to large corporations.  Perhaps the most rewarding project for me was a multivariate testing engagement with Comcast.net, one of the most visited sites on the Internet in the US. It was my first opportunity to dive deeply into testing on a site with both large enough traffic to get statistically significant results from large, full factorial multivariate tests and with a client who trusted us to take most of our recommendations and make all of the pieces line up.  More on this in a future post…

Continue reading

The Secret To A Web Analytics Career: Stop Learning Web Analytics!

This post originally appeared on Corry Prohen’s blog.  You might also enjoy my 10 tips on learning web analytics article and my advice about writing an analytics resume.

Eventually, your web analytics career is going to hit a wall.  Learning interactive marketing as a web analyst, you start with the numbers and then seek context.  When you stop to think about it, it’s really an inside out view, isn’t it?

You can master tags, logs and the many intricacies of implementation and that will get you so far.  If you want to go from good to great, then you’re going to have to stop learning analytics.

Advance Your Career with Action

Measurement craves action.  That’s why site testing (a/b and multivariate) has been such a huge hit.  Your success as a web analyst is defined by the impact of the changes your work inspires: more leads, higher revenue, greater customer satisfaction.

Start your search for by following the money.  In most companies, the budget is often biggest in one of these 4 channels:
1. Paid Search
2. Email Marketing
3. Organic Search / SEO
4. Landing Pages

1. Paid Search

eMarketer predicts that paid search spend will hit $10 Billion by 2010.  Each year, more money migrates from offline advertising to online buys, often starting with paid search (aka pay-per-click or PPC advertising).

Who’s to blame them?  PPC advertising, most often on AdWords and Yahoo Search Marketing, is among the most measurable marketing investments you can make.  That’s exactly why web analysts need to get up to speed… fast.

The basic structure of paid search advertising is simple: you select words, bid on them, write a text ad and send them to a landing page.  The reality is far more complex.  To get started, I recommend you start with a simple 5 point PPC questionnaire:
1. How are we performing against our yearly and monthly goals?
2. Which campaigns are driving 50% of our cost?  Which ones are driving 50% of our revenue (or similar KPI, such as leads)?
3. What are our ad groups and how are they performing?
4. What is our impression share for the top campaigns?
5. How is competition affecting our brand campaigns?

This is a mix of simple questions, but you’d be amazed how eye opening they can be for a web analyst.  I threw in “impression share”, because it’s a metric that few outside of paid search understand.  It once again highlights the need to move outside of just one tool.

You can get a good overview of paid search with this guide.  For free advice on measuring and optimizing paid search, check out the ClickEquations blog.

2. Email Marketing

Email is a rather unsexy channel.  Most people think of it as outdated at best or spam at worst.  The truth is that email is very much a part of online marketing in a basic way (support email, order confirmation) and more advanced uses (personalized offers, abandoned cart recovery).

As an analyst, you can lead the charge to maximize email ROI by asking:
• How does email stack up against other channels?
• What’s the most effective way to grow our subscriber list?
• What have we learned from past tests?  How can we structure future tests to boost results?
• Which segments of our list are most valuable?
• What kind of from and subject lines boost open rates?

I usually turn to the Email Experience Council to find resources for email marketing.  Their Email Stat Center is a really great collection of research and some stats for comparison.  The Email Benchmark Guide is a good starting point.  For more in-depth training, there is an Email Marketing Summit coming up shortly.

Continue reading

Miracle Website Diet! 8 Web Analytics Resolutions

It’s a New Year which means… resolutions!  I’m trying to skip the meaningless, aspirational goals that I usually dream up.  Instead, I’m digging down into the things that I can really commit to.  I’m making 8 resolutions for Digital Alex and ClickEquations.com (which I’m responsible for).

  1. I will integrate my data.  People come to my site.  Then, they become leads, customers and retained customers.  I really want to make decisions about where to spend my money based on who is the most valuable.  That means I have to connect the clicks to the customers with extra site and campaign tagging.
  2. I will QA my data at least once a month.  I’m going to totally level with you here.  I’m guilty of some web analytics sloth.  Sometimes I’ll QA data once, make the fixes and only deal with spot issues.  We all know data is dirty and I just can’t assume it’s right.  Bad data happens!  This is especially true on B2B sites (see above).  I resolve to do a more thorough review at least monthly and act swiftly.
  3. I won’t have an ego about my website. In the face of mounting evidence to the contrary, I concede that I do not know everything.  That’s why it’s probably better for me to rely on what customers care about what the data says.  Multivariate testing and voice of customer, here I come!
  4. I will prioritize speed over perfection.  I’ve been accused of a “shoot first, aim later” style and, frankly, I’ll embrace the label.  In my opinion, it’s better to ask forgiveness instead of permission.  To me, it’s all about speed.  The same is true in online marketing, design and analytics.  It’s easy to obsess to the point of inaction.  I’m not saying to be sloppy, but I am say that something launched at 80% is better than nothing launched at 95%.  Speed has a tremendous value.  I will analyze until the answers are good enough and then correct. Continue reading

Most Popular Blog Posts of 2008

‘Tis the season to reflect on the most popular blog posts of 2008.  Following in my grand (1 year old) tradition, I scoured my Google Reader for those blog posts that I starred throughout the year as useful and entertaining.

These are the most popular blog posts to me, but I’d also like to know which ones are your favorites.  Feel free to comment and share either ones you’ve written on your blog or ones you’ve read on other people’s blogs.

Special thanks to Dennis, Li, Craig, Matt, Manoj, June, Rich, Lu, PPC Hero, Bryan, and Avi.

Digital Alex (most popular blog posts on my site per analytics data)

Web Analytics Blog Posts

Website Testing + Targeting Blog Posts


Paid Search Blog Posts

Copywriting Blog Posts

SEO + Social Media Blog Posts

Startup Blog Posts

Web Design + Visual Inspiration Blog Posts

Careers & Salary Blog Posts

General Marketing + Strategy Blog Posts

Presentation Blog Posts

A few of these blog posts aren’t from 2008, but I discovered them in 2008.  So, they’re still the most popular blog posts of 2008 to me :-)

Which Pages Are People Testing?

A/B and multivariate testing are staring to get their due.  This whitepaper from ZeroDash1 is a good state of the union for the MVT market.  I thought this graphic of which page types people test was illustrative:

homepage testing

Most of the traction in multivariate testing is on landing pages, with the rest evenly split between home pages, conversion/purchase pages and product pages.

I’m not terribly surprised because:

  1. People who are testing are more likely to have landing pages, which are generally more advanced than a small webmaster has.  Plus, those people are more likely driving acquisition channels to these pages, where conversion makes a big difference in profitability.
  2. Testing, like analytics, is best used first where you can follow the money.  Campaign landing pages are often the leakiest part of the bucket.

69% of people think testing is worth it and 22% can’t live without it, which is pretty compelling data.  Of course, you need to translate the value into dollars and cents to your boss…

There’s a good graph in there with ideas of things to test, so download the whitepaper.

101 Great Blogs You Must Subscribe To (and How to Use Them)

Pop quiz: You’re pressed for time and need an answer to your internet marketing problem fast. Where do you turn? Search engines can be imprecise, consultants can be pricey and if your coworkers knew how to do it, you wouldn’t need an answer.

I’m faced with that situation every week. And that leads me to my confession:

I subscribe to a lot of blogs, but I don’t actually read them regularly.

Instead I use them as my own personal search engine. I’ve amassed over a hundred trusted sources in my Google RSS Reader. I’ve tagged them all to categorize them. Then, when I’m faced with a question, all I have to do is search all the blogs or an applicable subset, like this:

great blogs

I’m much more likely to get relevant and actionable results that way. To get you started, I pasted links to all of the blogs from my reader below. They fall into the following 16 categories:

  • Web Analytics
  • Copywriting
  • Blogging and Social Media
  • Competitive Intelligence
  • Conversion Rate
  • Comparison Shopping Engines
  • Data Mining
  • Excel Tips
  • Industry News
  • Information Presentation
  • General Marketing, PR & Strategy
  • Paid Search
  • Public Speaking
  • Search Engine Optimization
  • Testing – A/B, Multivariate, Taguchi, etc.
  • Data Visualization

Am I missing any? Please leave a comment.

Web Analytics


Blogging and Social Media

Competitive Intelligence

Conversion Rate

Comparison Shopping Engines

Data Mining

Excel Tips

Continue reading