You’ve got a measurement tool and you’ve got a website. Sometimes it feels like the two will never meet.
Layer in your paid search campaigns, email tactics, and search engine optimization projects and you can see how translating data into dollars can be a little cumbersome.
I agree that process is key to improving your website and advertising. I want more to be written, improved upon and advocated.
Thus, in this post I’m going to lay out my thoughts for a basic website and marketing analysis method that has worked for me and my clients.
My process is built on 5 steps:
Step 1 – Data Preparation
Step 2 – Analysis
Step 3 – Report Preparation
Step 4 – Communication
Step 5 – Action
Each step in the method feeds into the next, as detailed in this graphic (click here to see a larger, clearer image):
Step 1 – Data Preparation
I recommend 3 steps to get your data in order:
* Gather the data
* Validate the data
* Share raw data with the team
1a. Gather The Data
Some of the data you need will be readily accessible within your site measurement tool(s). Chances are, however, that in order to analyze the data in a simple way that directly answers your business questions you’re going to have to gather the metrics from a variety of sources and combine them into one or more dashboard(s).
The trick with data gathering is to automate and document as much as possible.
Automation is key, because assembling data is a totally thankless task (trust me, it’s how I started in analysis) and it opens up the process to more human error.
Essentially, you need figure out a way to get out of your web analytics tool and into. The most basic way is to download the data as an Excel or CSV file and clean it up with references or programming, such as Visual Basic.
Your particular web analytics tool might also has some features to simplify this process. Omniture has the Excel toolbar. WebTrends has the open database connection (ODBC).
No matter how you gather the data, you also need to document the data you gather. At some point, you’re going to need to retrace your steps. This is exactly the point Murphy’s Law will kick in and you’ll forget where you got the exact information you need. Either that, or someone else will be doing the work for you and he or she will have to bug you to ask “Where did get this number? How are you calculating that metric?” and so on.
There are three key things anyone needs to look back at your web analytics dashboards and deconstruct them:
- Name of the tool
- Date range
- Specific report source
You can put it in the footnotes, endnotes or just as a comment in a cell in your spreadsheet. For example, Omniture – September 2007 (9/1/07 – 9/30/07), Finding Methods.
1b. Validate the Data
I pretty much assume the data are wrong. Call me an analysis cynic, but I think a healthy dose of skepticism prevents complacency. It doesn’t matter what tool you’re using; they all have their limitations.
Most of the time you can validate your data with a little common sense. For example, you would expect that your direct (typed/bookmarked) traffic and branded keywords are going to convert the best. Experience teaches you how specific channels are likely to perform at each stage of maturity.
There are a few more ways to see if your data “smells right”:
* Historical Performance – History isn’t always the best predictor of future performance, but you can generally gauge whether trends follow recent changes or seem off.
* Seasonality - Year over year trends can help you get a sense of whether changes are expected or unexpected.
* Other Data Sources – Backend ecommerce platforms, paid search conversion tools and affiliate reports are all good, alternate sources to look at the same data.
This will also familiarize you with the different ways tools measure (for example upon sale vs. upon shipment). Trust me, someone’s going to ask this question and it’s better to have the answer already sorted out and noted in your report.
1c. Share Raw Data With The Team
I know it seems weird and against the grain of actionable analysis to share raw data with your team, but it’s a useful step. No matter how great your are at gathering an validating the data, it’s still very possible to miss something.
Take 30 minutes to look over the data together and note anything that’s surprising or obviously off. Another set of eyes always catches something.
Frankly, I also find this step is important from a project management standpoint. It forces a deadline on the team. It also makes you think through all of the timing and duration in the other steps in the process:
- When is the data available?
- What’s a reasonable amount of time to gather it?
- How long does it take to put data into a report form?
- How much time is necessary for good analysis?
- When does the data need to be reviewed? (and, conversely, when does it become meaningless?)
I’m specifically talking more in-depth analysis here. Obviously, you’re going to be checking the data with bookmarks, dashboards and the like on a daily or weekly basis.
Step 2 – Analysis
I’ve found that analysis tends to take one of three forms:
1. Reactive - Some change is made to your site and it naturally stimulates questions. This is particularly true for anything new or changes that required substantial time or money investment. Questions always follow expenditure and so should analysis.
2. Discovery - You gather the data and start digging. Unusual data or curiosity take you through the analysis. Sometimes it’s just exploring something you’ve never analyzed in the past.
3. Business Rules – You’ve set some website goals or marketing targets that define success and measure against them. The analysis in this case addresses why you’ve succeed or failed in meeting your targets and what you can learn from what works or address the issues that are keeping you from reaching the goal.
There are 3 steps that are part of every analysis:
* Compare data to goals and benchmarks
* Form preliminary hypotheses
* Perform exploratory analysis to test hypotheses
2a. Compare Data to Goals and Benchmarks
“Is that good or bad?” is probably the most common website and marketing analysis question ever asked. Usually the answer is, “It depends.”
If you have goals, it’s much easier to judge performance. Either you’re measuring backward, such as comparing the previous month’s results vs. goals or you’re measuring progress—likelihood to reach goals in the future.
I think analysis against Business Rules (or outcomes) is the most productive and easily understood by analysts and non-analysts alike. Goal setting itself is immensely productive, because it forces you to prioritize, think through assumptions and consider the key metrics that help you measure and diagnose your progress.
Competitive intelligence tools like Hitwise, comScore , FireClick, Compete Search Analytics can help you get some external benchmarks. Just remember that each tool’s data collection methodology has its limitations and price.
In the absence of goals, internal benchmarks are usually the next recourse. Your own site or marketing’s historical performance is the best comparison.
This is where segmentation starts to illustrate differences–pay-per-click vs. affiliate marketing, results among ad groups, different email newsletter formats, etc. Differences will quickly bubble to the surface, which leads me to my next point:
2b. Form Preliminary Hypotheses
Teasing out patterns or anomalies leads to more questions. Start with the trend or quirk at hand and form a preliminary hypotheses.
This doesn’t have to be a formal process where you write it down. Mostly, you’ll just think about it. The point is translate something you notice into a theory about why it occurred in a form like “I think ___ happened because of ___”, such as “I think the t-shirt ad group has a low ROI because the creative is underperforming.”
A good hypotheses states the issue and considers the underlying cause. It’s specific and data based.
2c. Perform exploratory analysis to test hypotheses
Once you have your hypotheses, you have to test them. This is the essence of data-driven analysis. You look to the numbers to poke holes in or support your theories.
This part is part science, part art.
For the science part, you need to identify the metrics that best test your hypothesis. In the aforementioned t-shirt ad group example, you’d be looking at clickthrough rate (CTR) to start with – Comparing the CTR against other creatives in the ad group, ad variations in other ad groups, among similarly themed ads elsewhere, etc.
The art part is in the interpretation of the numbers and understanding metrics in context. You have to be able to separate the signal from the noise. For example, is it really the creative that’s bringing you down or are your bids too low? If it is creative, is it because you’re not matching keywords to ad titles?
This is the part where you have to try to translate piles of numbers that tell you what happened into analysis that explains why things happened.
Sadly, no one tool can combine all of the pieces of data and tell you exactly why the customer behaved the way she did. This is where the smart analysts you hired can explain the difference.
Step 3 – Report Preparation
Winston Churchill once quipped, “However beautiful the strategy, you should occasionally look at the results.”
No matter how fun or intellectually stimulating your analysis, it means nothing without action and results. That means you have to share the information.
There are 2 key steps in preparing a report
* Assemble data with insights and actions
* Build team consensus on actions
3a. Assemble Data with Insights and Actions
You have to pick your poison when it comes to presenting data – Excel spreadsheet or PowerPoint.
PowerPoint decks tend to be better for a wider, non-technical audience because they’re linear and structured. Spreadsheets are great for more intermediate reports or those targeted at more analytical and numbers types.
They’re not mutually exclusive. Sometimes I put together a deck and back it up with the numbers in a spreadsheet.
Whichever format you chose, the point here is not to spit back a bunch of numbers. Anyone can get that from a tool. The goal is to tell a story–the story of your analysis.
To do that, you need to a.) help people see the same things you see and b.) pair each notable data point with insight and action.
Graphs and tables help you bring attention to the points that caught your eye. It’s particularly useful to pull them out and annotate with marks to highlight important dates or trends.
When discussing your analysis, avoid narrating your data. Instead, explain why it’s important, what it means to the business/organization (insight) and what needs to be done about it (action).
Data: “Conversion rate is up 10% and average order value is down $15″
Insight: “Conversion rate is up 10%, the fastest increase in 6 months, but the average order value is down $15. Both of these trends are directly related to the recent 25% off promotion. While we had more sales, overall revenue dropped due to the decrease in AOV”.
Action: “Test additional offers at different discount levels or non-discount promotions to all customers or different segments.”
That’s a simplified example, but it answers questions and problems with solutions instead of more questions. Even if people disagree, it’s a place to start the conversation.
That brings me to my next point:
3b. Build Team Consensus on Actions
You’ve got to work with your immediate stakeholders if you want the best ideas possible and to get your ideas in motion. Once your report is assembled, meet with the core team to walk through your insights and recommendations. Since you’ve already shared the raw data (step 1c), you should be able to focus on the interpretation.
Everyone’s experience will color the interpretation and help you make sure your analysis is well thought out. There’s nothing worse that presenting an interpretation that gets shot down because you weren’t thorough enough.
It’s also important that your team agrees on the right course of action. Getting executive buy-in is useless if the people who actually need to make it happen don’t agree with the direction. Teamwork in choosing the right direction makes it easier to foster teamwork in pursuing that direction.
Step 4 – Communication
The truth is that sales is an important part of site and marketing analysis. You have to persuade all of the gatekeepers and opinion leaders to dedicate time, resources and budget to pursuing your action. As you can imagine, this is no small feat.
There are two key steps:
* Present to client(s)
* Provide exec friendly leave-behind
4a. Present to Client(s)
This is the moment of truth. Your audience is nowhere near as close to the site, data and analysis as you are. Time is short.
Unfortunately, there’s no one perfect style for data presentation. You have to read your audience and adapt to their particular needs and priorities. There are several common types:
* The Bottom-Liner – This gal just cares about the results. She knows what the business is trying to achieve this year or quarter. Her priority is to understand how the site and marketing are contributing.
Cater to the Bottom-Liner by having all of your KPIs squared away and relate them back to the business goals.
* The Warm and Fuzzy -Chatty and friendly, the Warm and Fuzzy loves the back and forth in the meeting. Sure, it feels fun, but is he going to get the job done?
Keep the Warm and Fuzzy on task with an agenda ahead of time, clear next steps and follow-up notes. Get him to commit to action and deadlines. When he completes them, make him feel good.
* The Naïf – The Naïf is new to interactive or website analysis. She’s well meaning, but she needs time to get up to speed on everything, possibly time you don’t have.
Train the Naïf. Make her comfortable by avoiding jargon as much as possible. Put in definitions and glossaries to help translate the analysis-speak. Offer to have a 1 on 1 session to train her on the topic(s).
* The Gunslinger – We’ve all met the gunslinger. He’s brash, opinionated and loves to shoot holes in your presentation (hence the name). The fact is that you’re just not going to change this guy’s personality. You have to work with it or work around it.
Arguing with the Gunslinger is not going to get you anywhere. Instead, come to the presentation with the idea that you’re all working toward the same goal with this analysis. Ask for his interpretation of the data, e.g. “Here’s what I think this trend means and why. What do you think could explain this?”
Accept his points where they’re right. If he pins you down with a particular question, be honest and say “I don’t have that information right now, but I’ll look into it for you.” You have to earn credibility slowly over time until you get the benefit of the doubt.
If it makes sense, give the Gunglinger early access to your work before everyone else sees it. That way, he feels appreciated and you have an opportunity to address his concerns before you’re in front of a group.
* The Disinterested – This is possibly the most poisonous of all audience members. She either doesn’t get or doesn’t care about improving the site and marketing. Whatever the issue, it’s now an obstacle between you and results.
Work around the Disinterested. Find out who the real doers are and figure out a way to loop them into the process. You can’t cut the Disinterested out of the process, but you need to do her job for her without pissing her off.
Analysis Presentation Tips
Even though each audience is different, I think there are some fundamentals of presenting analysis that help set you up for success:
* Start General, Get Specific – You need to set the framework for your presentation. Tell people what you’re going to tell them, tell them, tell them what you told them. Start at the higher level (performance vs. goals) and then dig into the details (diagnosis/analysis).
* Provide Context – Your site doesn’t exist in a vacuum. Seasonality, industry and competitive changes all influence the results. Try to document as many of the influencing factors as possible to help provide context to your numbers.
* Remember to Breathe – Don’t rush through your presentation. Pause. Give people time to process what you’re saying. If they’re still trying to figure out what happened 3 slides ago, you’ve lost them on the point you’re making now. Stop and ask, “What do you think of that? Is that how you interpret these data?”
4b. Provide an Exec Friendly Leave-Behind
No matter how eloquent your presentation, it must be able to stand alone without you there. Not everyone who needs to hear your points will be able to make the meeting. Even the audience who was there will need to go back and review. That’s why you need a good leave-behind.
The document you provide should have an executive summary–1 to 2 pages that summarize the key points and next steps. If you can only say a few things, what is most critical?
I find it effective to organize your next steps in a table that relates the data to insight and action. To help client’s make decisions, prioritize the options by likely impact and resources required. Trade-offs are definitely going to happen, so it’s in your best interest to make sure they’re picking the right things.
Just remember to document your sources, assumptions and any data caveats in case you have to retrace your steps.
Step 5 – Action
Don’t leave that room until you get your audience to talk through and agree upon the next steps. You have a captive audience, so this is the best time to commit before a million other meetings take you from top to bottom of mind.
There are 2 key steps:
* Assign responsibilities and due dates
* Revise goals
5a. Assign Responsibilities and Due Dates
Half of the issue with getting site and marketing changes done is working through the logistics of who will do it and when.
I’ve found that identifying the person and following-up with them directly (or at least making the offer to do so) can go a long way toward getting things done. It helps to prevent things from being lost in translation and opens you up to questions when needed.
Due dates are a tougher matter, because most everyone is busy and it’s a matter of prioritization. You can facilitate the process by relating the changes to business outcomes, that is to say, helping people understand the return on their investment.
It’s also important to highlight contingencies. If X must happen before Y, you’ve got to make that transparent verbally or visually, maybe both.
Give people the big picture timeline and break it down into project components. Smaller chunks are easier to manage and monitor (and easier for procrastinators to start).
5b. Revise Goals
Forecasting is messy business. You have to make a raft of assumptions that invariably prove to be off in the short term, guesses in the mid-term and unreliable in the long term. And that’s fine. It doesn’t need to be perfect.
What forecasting does need to be is based on the best information at hand. As your data prove or disprove your assumptions, you should adjust accordingly.
Improving your site and marketing takes a bit of rigor. Process helps you move from one-off and sporadic digging to regular review and improvement.
I know this article is mammoth, but once you go through the process a few times, it becomes second nature.
What works for you? Please comment or email me, alex @ alexlcohen . com