A few weeks back one of RKG’s sharp analysts provided insight into the vexing problems associated with thin data. Given the conversion rates in retail PPC, it becomes very difficult to separate signal from noise when the number of click-throughs is relatively small. This has some important consequences for paid search management and how we think about and react to data.

The most important metric to measure and predict in paid search is the value per click associated with a given ad. Value could be leads, sales dollars, margin dollars or some other metric that corresponds even more closely to “goodness” to the advertiser. Tying the cost per click paid to this value per click for each ad maximizes value within an advertiser’s ROI needs.

However, at the ad level, we don’t often have enough data to separate the signal from the noise. Analyzing statistical noise is a waste of time; reacting to statistical noise can be a fatal error. The solution is to add data until the signal is reasonably clear.

Please forgive a photography metaphor. If thin data is analogous to insufficient light, we can say that there are essentially two ways to generate a clear picture:

1. by widening the aperture; or
2. by lengthening the time of exposure

(the third option of a flash would be akin to adding artificial data — not a good idea).

Widening the aperture means aggregating data from a wider set of closely related ads; lengthening the exposure means incorporating data from farther and farther back in history. Both solutions are fraught with peril and consequences that must be addressed, and the proper combination of the two approaches may vary across the portfolio of ads. This is where data modeling comes in for paid search, and where the Frankencamera might reign in digital photography.

But my point is not to wade into the weeds of details right now. The point is simply this: every program no matter how big or small must wrestle with thin data at some level.

Small programs may have so little conversion data each week (less than 50 or 100 conversions on competitive search ads) that the performance of the whole is noisy week-to-week.

For larger programs, the performance of the whole may stabilize into a clear signal week to week or even day to day, but there will be categories of keywords, or even whole engines (Ask, MSN) that require a broader data set to view sensibly. This does not reflect a failure of analytic power, but simply fundamental laws of statistics.

The more sparse the data the less value there is in fine tuning. Tiny programs may not benefit from day-parting or many other advanced techniques that are materially helpful for larger programs. It’s one of the reasons that a “how to” book for paid search is hard to write: what is imperative for some is a waste of time for others.

But even the biggest players in paid search can get lost in the randomness of statistical noise, forcing conversations about nothing and reactions to non-issues. We’ve sometimes contemplated whether a paid search reporting tool should even provide a keyword-level performance report for a single day, or whether it would be better service to send up an error message to the effect: “We’re sorry, nothing good will come of looking at this data. Please request a different report.”

Then again, some of the heaviest hitters find tremendous value in hourly performance monitoring of head terms during their peak seasons, so…

Anyhow, consider the expected variance in performance for an ad, keyword, group or program before making unwarranted adjustments. Adjust the aperture and exposure to get a clear picture of performance and the correct next steps to take.