Mar 52009

Blinded by Data?

Are you data-driven, or mired in data?

Sometimes it's hard to tell. Oftentimes it's a bit of both.

The critical question: what fraction of the data you look at each day is actionable? If you look at the same reports each day or each week, ask yourself: are there steps I could take based on what this data might reveal that would make the numbers improve going forward? If the answer is "no" then does it make sense to spend time looking at that report?

As staffing gets leaner in retail we all need to figure out how to do more with less. Part of the solution is to spend less time looking at numbers we can't impact and more time taking steps that will positively effect the numbers that matter.

Habits are hard to break. Here are some questions that might help separate the actionable from the non-actionable and the KPIs from the not-so-KPI:

  • Has looking at this data ever made me do something differently?
  • Are the Key Performance Indicators I study actually KPIs? Two examples of common KPIs that aren't:
    1. Site Conversion Rate: If your site conversion rate goes down, is that bad? Or, does it simply reflect either the addition of some lower converting traffic (perhaps due to an increase in competitive search traffic) or decrease in higher quality traffic (like the end of a catalog mailing cycle). Much time can be wasted chasing explanations to these changes that don't result in any meaningful action.
    2. Average Order Size: If the AOV drops does that mean you're selling fewer big ticket items, or more low ticket items? A decrease can be good or bad, and chasing the source of fluctuations is time consuming.

    We'd suggest that if changing a metric in one direction can be either good or bad, then that metric isn't really an indicator of anything. Instead, go either to the next level higher (Site Sales), or the next level lower (AOV by category or CR of top search keywords) to find the numbers that are less ambiguous.

  • Do I look at this report because it's cool, or because it's useful? I remember long ago looking at some website analytics reports (heat maps, next page/ previous page, etc) thinking: "Wow, that's cool, but what would one DO with this info?"
  • How much time should we spend benchmarking? Being consumed by what competitors are up to makes sense only to the extent that you can react to it intelligently. If their paid search ads are higher on the page than yours you can be pretty well assured that they're generating more traffic. But, you don't know whether they're overspending for that traffic. It might be wise to see if perhaps their prices are lower, or their selection is more comprehensive -- some reason to believe they generate more sales per click and hence can afford to spend more per click. That knowledge could generate action on the part of merchandising. Absent that, since you can't know what they're spending, why pay attention to where they are?
  • How many meetings do you attend that do not produce changes in direction? Any way to eliminate those meetings?

Data should be used to take positive action. If we think about the dashboard metaphor: dashboards provide data that helps you drive, but if you stare at the dashboard you'll wreck. Right now, its in all of our interests to avoid wrecks!



6 Responses to "Blinded by Data?"
Will says:
Well said George!
Thanks Will, and may we all get to chase fewer wild geese!
Marc Adelman says:
So here is a new thing I've been working on: The Question: How should decision makers react to changing KPIs? One of my main issues with KPIs is that they are indicators not story tellers. They say "Hey, something is happening here!" but not, "Hey, here is the whole story to what is going on here!". Many people fall into reading the whole story from the KPIs and worse, a single KPI. As you stated, "Our AOV is down - that is Bad." Since the overall performance is the combination of all KPIs, can creating a system of viewing a composite score that applies appropriate weight for each KPI gives us a better immediate view into performance change and the need to act? What I've been stewing up here is a composite KPI score that groups campaigns in ranges of performance. For each KPI, I apply a score of 0 or 1 (0 if the KPI is above the average for that KPI, and 1 if it is below the average). Different KPIs can be given a number score up to 3 depending on the weight of the KPI. Then all of the number scores for the KPIs are totaled. This gives a 0 - 10 score for the campaign/ marketing channel/adgroup etc. 0-3 are Top Performers, 4-6 are Middle of the Pack, and 7-10 are Bottom Performers. This allows for a quick view of the relative holistic performance of the campaign and allows deeper focused penetration into problem areas. This gets decision makers away from pivoting the decision making process off of one or two changing KPIs. Not sure if I explained this clearly. Let me know what you think.
I like your approach, Marc. What we're really looking for is a way to algorithmically identify warning signals in a way that's useful. Flagging anomalies is trivial, the problem is the warning bell goes off all the time and you spend the whole day chasing problems that turn out not to be problems. Given the spiky nature of PPC marketing, figuring out how to spot real problems or opportunities without having a constant barrage of false alarms is a challenge. The alarm system we've created involves calculating historical averages and standard deviations factoring in day of week fluctuations. We then "tune" the system by setting the number of variances from the norm that we're willing to accept as "statistical noise". This was a hassle to build, but has allowed our analysts more time to focus on what matters.
Marc Adelman says:
George, That makes perfect sense. I've been playing the broken record game here, trying to get a full time programmer to be dedicated to Online Marketing. It's processes like the one you shared that makes me pine even more to get this resource on staff. Thanks for the inside view.


Check out what others are saying...
[...] bookmarks tagged ambiguous Blinded by Data? saved by 7 others     teenxtream bookmarked on 03/06/09 | [...]

Leave A Comment