Mar 222013

What eBay's Test Results Teach Us

A number of people in the industry have asked me what I think about the eBay research study on the incremental value of paid search. While it's flattering to be asked, I'm not sure I have a ton to add to what others have already written.

First, as Chris Zaharias points out: it's really important to read the actual study, not other people's summaries of it.

eBay is actually very careful to point out that they are a fairly unrepresentative advertiser in many respects and that their findings may not reflect what others would find. This is exactly right. Two physicists should get the same results using the same methodology, but that's because physical laws seem to be fairly universal; not so with marketing.

What eBay found:

In a carefully run geo-targeted series of control tests eBay found that advertisements on their brand name drove no incremental revenue. More surprisingly: they found that very little revenue attributed to non-brand advertisements was incremental to their business. Given those findings the ROI of non-brand paid search as measured by incremental revenue generated was terrible, leading them to cut back tremendously on paid search spend.

They also found that non-brand paid search did drive a higher percentage of incremental purchases from new or infrequent buyers than did other marketing channels, and suggested that smart marketers should focus efforts on new customer acquisition.

Here's the thing: you're not eBay.

In so many respects, eBay is an outlier.

  1. Market share. Within the world of "auctions", and indeed in many many verticals of merchandise (think antique post cards) eBay's market-share is orders of magnitude bigger than almost any other advertiser. The notion that potential customers will check out the offering of the dominant player in a space before buying regardless of whether they have an ad running is pretty easy to understand. If you have 50% market share (fraction of total sales) in your vertical, and gigantic market penetration (the fraction of customers in the market who have bought from you), the incremental value of non-brand paid search might be low, too. While eBay found little evidence that category mattered for it, those finding might be skewed by heavy volume within each category of "auction, used, antique" prefixes.
  2. Scale. Beyond vertical specific market share, the sheer scale of eBay's business means they have huge consumer awareness. This in turn means advertising may be less necessary, particularly when measured in a short window of time. How much would Coke's sales slump if they didn't spend a dime on advertising for a couple of months? Sales might suffer over time, however. There is a great deal of fascinating research on this topic revealed in a book by Byron Sharp I'm reading on the recommendation of a friend.
  3. Customer base. I'll venture to guess that eBay's most loyal customers are also a somewhat different breed. These are people who love to shop. These are people who love to find deals. These are people who will shop around simply for the joy of doing so. Many advertisers would do well to dynamically bid less for loyal eBay shoppers because they're going to keep looking for a better deal. eBay is more "inevitable" than others would be simply because of who they attract as customers. I, on the other hand, hate to shop. I will buy from the first store that has what I want at what strikes me as a reasonable price. If you don't have a link on the page I'm not likely to look elsewhere. I'm an outlier on the other end, but that notion of range is important.
  4. Brand name differentiation. Someone typing "eBay" is going to end up at Will someone typing "drugstore" necessarily end up at, or might some of them be looking for any drugstore? Will someone searching for "Sony" always go to, or might they be delighted with Best Buy? It is worth testing the incremental value of brand search, but understand that all brand search is not created equal.
  5. Paid search strategy and execution. Larry Kim humorously ripped eBay's ppc efforts to shreds and made a compelling case that better strategy and execution would yield better ROI for them. I've met eBay's paid search folks, and I know they're a really smart group of people. However their reliance on automation for keyword generation, copy writing, landing page choices, etc mean they had to pay more for traffic because of poor quality scores, their landing pages convert traffic less well than human driven choices, and the weight of poorly targeted keywords make the aggregate performance look awful. Smart targeting, well crafted/tested copy, good landing pages would demand much less "I" for a similar "R" in ROI. Buying the dictionary doesn't work well in paid search, just like mailing the phone book leads to disaster in direct mail. As a brand building strategy sending the message "You can get anything at eBay" might be a stroke of genius, but don't then expect the short-term ROI to look good on a branding exercise.

What should advertisers learn from eBay's rigorous test? It pays to test your assumptions. eBay identified a ton of wasteful spending by challenging the attributed sales from the channel to find out what was incremental. Good for them.

Doing the same test will you find the same results? There is absolutely no reason to think you will, but testing assumptions is the only way to know for sure.

The notion of "inevitable" visits in the course of shopping is a profound and interesting one. My wife likely spends 80% of her clothing budget at 5 or 6 different stores (online and off). When she's in the market for a bathing suit, she will almost certainly visit whether they advertise to her or not. She likes to shop more than I do, but her consideration set is relatively small. Should Land's End target ads to her to capture wallet share from others, or would they be wasting money? Re-targeting efforts can look artificially productive because of this behavior exactly. Indeed, many marketers have it backwards by pushing offers to their most loyal customers who will buy from them without the offers.

I applaud eBay for applying the scientific method to their marketing efforts and encourage others to do so as well. We can all raise our game by testing assumptions, but we must be very careful interpreting the results of our own tests and should never assume that what one company learns will translate directly to your company.

Other good articles on the topic from Harvard Business Review, Quartz, SEL.


8 Responses to "What eBay's Test Results Teach Us"
sam mazaheri says:
Great, well balanced article. I was also impressed by how cautious ebay was to apply their findings to other advertisers.
Thanks Sam. eBay may have an axe to grind with Google, and it's interesting to note that they only talked about Google (I don't think they tested Bing), but they do deserve credit for pointing out that their findings may not generalize.
Michael says:
George - Thank you for this thoughtful post. For a while I've been considering ways to test the incremental ROI of Paid Search on Brand and Brand+ terms. Do you feel the methodology eBay employed is the gold standard, or would you suggest another setup? My question assumes that Brand Search between Bing and Google follows the same pattern so that one may serve as a reasonable control for the other - something one could verify prior to testing. Thanks, Michael
Thank you Michael! Geo-testing is a great approach as it creates many test cells. It is cumbersome for that same reason. We did some tests with a few clients that were simple "on-off-on-off" tests that gave us a pretty good sense of incremental lift (or lack thereof in some cases) of "brand" search. Reading these tests requires normalization for day-of-week and promotional effects, and you'll need to look at the totality of brand paid search plus brand organic plus any brand traffic from PLAs, and if affiliates show up on your brand name you might need to include traffic flowing to those affiliates as well. You'll also need to not just pause brand ads but create exact match negatives for them as well -- make sure to include the PLA campaign(s). If you have sufficient traffic -- and I'm sure you do -- you might pick just a couple of brand phrases to pause and negative out to make it a bit less cumbersome. If helpful, I'd be happy to jump on a call to discuss. George
Ben says:
Hi George, thanks for this incredibly informational post. We agree that testing is so important in any successful, ongoing campaign. It was interesting to read your take on eBay's recent experiment.
Hi George, I agree that eBay's dire results from PPC are more a reflection of their poor execution, rather than the PPC industry in general, and that eBay are very different to the average advertiser, as you point out. On the value of brand ads, I think it's illogical for eBay (and other businesses) to simply write off the practice of showing ads for your brand terms, if the sole purpose of doing so is to provide a mere navigational link. I have identified 13 different ways businesses can use their brand ads creatively and intelligently to add value to their business: If businesses just bid on their brand terms so that they simply 'appear' in the paid listings when someone searches for their brand, then I agree with eBay, they will probably notice little incremental value from doing so. But if a more creative approach is taken to capitalize on yet another opportunity to connect and engage with their audience (for example using strategies such as those I suggested above), I'm sure a different result is possible. Cheers, Alan
Agreed, Alan. We've made similar points in the past that there are good reasons to run brand ads beyond incremental revenue, and the price is usually easy to justify. Thanks for your comment and fine article link.

Leave A Comment