Aug 202009

Google Finds Position Bidding Counter-Productive Too

Hal Varian, Google's chief economist posted some interesting findings about the relationship between position and conversion rates. Namely, that there isn't much of a relationship.

We reached that conclusion in 2003, and co-authored a stats paper on the subject in 2005.

It is funny to see the position bidding mavens jumping up and down crying foul, because this undermines the whole premise of their system. But the numbers don't lie.

It is terrifically difficult to do this research and it requires a mountain of data to reach the right conclusion. It's funny that Hal only mentions QS as a complicating factor. The much larger factor is that if the advertiser's bidding system functions properly, it bids down poor performing KW and bids up great performing KW, making the top of the page look artificially better, if you're not careful with your study.

Our research suggests it's not just conversion rates that remain constant but AOV as well, indicating that the people who click on ads behave largely the same way regardless of where the ad was on the page.

Our research also suggests that the slightly worse performing positions Hal mentions are at the top of the page and the slightly better are at the bottom of the page. We reason that some fraction of folks click the first link or two by reflex without reading the copy and are therefore more likely to be disappointed once they get there. The folks clicking on the bottom links have either picked those ads for a reason -- they recognize and like the company, or liked the copy -- or they've already visited the sites from higher ranked ads and didn't find what they want, so the competition has already fallen away.

In any case:

  1. It's pretty obvious why Google doesn't want to go on record saying position 1 converts less well than other positions -- a race to the bottom would hurt them; and
  2. Trying to avoid position 1 really is silly. The traffic value differential is very small, much smaller than the error bars around the calculation of value for a given ad, hence immaterial to performance.

Let the value of traffic drive your bids, not the position. Position bidding both wastes money AND misses opportunities.


18 Responses to "Google Finds Position Bidding Counter-Productive Too"
A sound article as ever George. I've read many of the discussions online about Varian's research. I've never known a piece of research to be so badly understood as this, even though in my opinion it's extremely clearly written.
Thanks Jonathan, Some of the posts are really remarkable. One guy interpreted the research to say: the bids and QS don't matter because the conversion rates are the same. Wowzers. I guess you don't need a license to blog!
Hi George, Perhaps you'd care to comment on Engine Ready's study of conversion rate by position, as it paints the opposite picture of your own and Google's data: Also, David Rodnitzky makes some good points on Blogation about how large data sets can be true while masking a very different picture at the individual advertiser level: I'm no math whiz, but I've seen enough anecdotes over the years saying the contrary of your study to be wary of deeming your and Hal's data as truth and not merely hypothesis at this point. Of course, none of that changes the fact that bidding to position rarely makes sense. Good to see the esteemed Sir Beesticles commenting as well!
Chris, thanks for joining in the discussion! I don't buy Engine Ready's all. Does anyone really believe that the conversion rate of position 7 is 4 times that of position 1?!? Absurd. The fact that they mixed lead gen folks in with retail and everything else, and probably didn't even exclude brand phrases screams of a haphazard study. This is an incredibly difficult subject. We consulted with several different stats PhDs in addition to our own and we studied far more than 2 million clicks. We were very careful to compare keywords only to themselves in different positions for the same retailer. Hard stuff. As a rule I don't trust anyone's research unless they detail their methodology, and even then question the results when motives are involved. If you think about it, Google's interest would be to suggest that the top positions have higher conversion rates than the bottom positions, and a poorly done study could easily show that. I applaud them for having the integrity to say otherwise. With respect to the folks who say that what a large data set shows may not be true for an individual client doesn't make sense to me either. Are they saying that if you look at much less data you reach a better conclusion? Doesn't make sense to me.
Hey George, fun topic. Thank you for commenting on Hal’s post. Chris, good to see you commenting and spurring a discussion. I was reluctant to join in... given how strongly my statistics professors would encourage me to stay far away from this topic. Although, it is fun to throw around the word heteroskedastic on occasion. With that said... I would assert that this is really more of a finance topic. The DM folks figured out a long, long time ago that data driven marketing is centered on maximizing contribution margin (after advertising expense.) An economist would also be happy to jump in and initiate a discussion on the interaction of marginal profit and cost curves. Which was my interpretation of Hal’s statistics “speak.” I subscribe to bidding strategies focused on generating the maximum total profit per sku. Our customers who have come over to this way of thinking have seen their revenue and bottom line profits increase dramatically. Granted, this practice is much easier said than done. Which is why our industry really needs the tools and technology to make this quicker and easier.
Stephen, thanks for jumping into the fray! I respectfully disagree with your assertion that this is related to the marketing/finance objectives. We set bids based on the anticipated value of the traffic for the given ad and the fraction of that value the advertiser is willing to spend. Our research and Hal's relates only to the first piece of this puzzle. If the value of the traffic depended not just on the keyword, match-type, season, time of day, etc, but also on the position of the ad at that moment, setting bids to hit any target would be a much different game. We certainly agree that bidding based on margin data is important, but can be more complicated than it appears. Some of our clients, for example, get discounts from manufacturers for hitting certain sales volume thresholds, hence cost of goods isn't always a knowable number. Return rates and other factors also complicate the picture in some cases. Certainly, measuring the "true value" of orders is preferable to just the price. As a former direct marketer, I'm not sure I concur that it's always best to market for maximum profitability, either. Ignoring lifetime value considerations, spillover to call centers, cookie breakage etc, can sometimes lead retailers down the path of the "death spiral". There are always tensions to balance between top line and bottom line, and while I agree that erring on the side of the bottom line usually makes sense, I've seen folks focus too much on the short term and hurt their business in the long run.
David says:
George, I don't have any statistical data to support this theory, but wouldn't it logically make sense that the number of tokens in a keyword phrase would have a significant impact on conversion rate variability by position? For example, if you bought the word "search" and showed up in #1 position, you would inevitably get tons of people who either a) inadvertently clicked; b) clicked out of curiosity. Your conversion rate on virtually anything you were selling would be phenomenally low. If someone, however, managed to click on your listing in position #7, one would assume that your ad text must have been relevant to their intent, because to find a listing in #7 position actually requires work on the searcher's part. Conversely, if you are selling a commodity product and the search is a multi-token keyword phrase with high purchase intent, one would assume that clicks on the first ad might have the same conversion rate, but an astronomically higher eCPM. In other words, if there's a search for "The Search by John Battelle New Paperback" and all the ads are selling the book for relatively the same price, you would see the first position likely grab the lion's share of conversions. Indeed, positions #5 and below might not receive any clicks at all. So in the first case, you have a generic query where conversion rate is dramatically lower in early positions; in the second place, you have a very targeted query where conversion rate might be the same, but actual yield, revenue, eCPM, etc, would be almost entirely clustered in the top positions. Again, I don't have data to support this, but intuitively it makes sense to me.
David, your notion is quite fascinating. Obviously, the more specific the phrase, or perhaps: the better the user search reveals buying intent, the higher the conversion rate, period. Your argument then is that the higher conversion rate means there is a greater imperative to be near the top of the page. Our research showed that ads at the bottom of the page have slightly higher conversion rates. We posited that this is probably because the user: recognizes the retailer's name and trusts them, or liked the ad text/offer more than the others, or they tried out the other offers and didn't find what they wanted You make the case that #3 is less likely to happen in cases where the search is very specific, and the advertisers are all very likely to have what they want. Really interesting notion, but I don't think it's the case. If it was the case, advertisers would find that bidding more aggressively on these types of terms would lead to lower cost to sales ratios than "normal" bids. Higher CPCs would be off-set by higher Sales per click. We've tried these kinds of pushes thousands and thousands of times and have not found it to work out that way. More aggressive bidding always leads to lower efficiencies. Sometimes the increase in volume makes that trade off worth while, but usually not. I don't doubt that there are anecdotal cases that seem to prove your point, but I'd bet that those cases are a product of random statistical noise. Like the hot streak at the blackjack table that the gambler believes was caused by switching his pinky ring from one hand to the other, thin, spiky data can trick us into believing funny things.
John Shea says:
George, I'm confident that it was your inquiry to this end that motivated Hal and team to publish these results. By the reactions to the post scattered across the blogosphere (I'm still making my way through them) it's clear that it's a topic of great interest. So, thanks for bringing this up with such enthusiasm. The point about AOV remaining constant across positions is very interesting and provides a layer of depth not covered by the Google study. Have you guys published any of that research? Be well, - J
John, Thanks for passing my questions to Hal. If our interest really motivated his post we'll call that a feather in our cap :-) We have not published our research on this topic. To be quite frank, this question is so central to our philosophy of bid management that it's been an obsession of ours for years. We've studied the data every way we can think of and done so regularly over the last 6 years. We've challenged our friends in academia to work on this problem. Every way we've looked at this points to the same conclusion, but we haven't been confident enough in our methodology to go public with a formal study. I'm much more confident in our results now that I hear Google found the same thing with respect to conversion rates. You folks have more data than any of us, and have the smarts to do this study really well. Thanks to Hal for publishing and thanks to you for chiming in!
Marc Adelman says:
Very intriguing conversation! A few points 1. It is important to understand the findings from RKG's study in context to the e-commerce industry. 2. Engine Ready’s Study was a cross industry study and only gives over data in aggregate - not segmented per industry. 3. If we are looking at the possible relationship between Position and Conversion Rate, then we must isolate the conversation per industry due to the actual difference of definition per industry for a conversion. B2b, e-commerce, and lead generation have different definitions of conversion & different values and desired user behaviors associated with the conversion. This alone, (leaving out the other "PPC Personality" variables that Engine Ready mentions as additional statistical noise per industry that "may" vary the results when seen on a industry level) detracts from the value of the overall findings if you do choose to view the relationship between Position and Conversion Rate from cross industry data. 5. I believe that there are too many variables(competition's strength of brand, competitive pricing, adcopy message, day - geo targeting, actual product -Brand & Item - product availability, etc.) even if we isolate industries, to draw real data driven conclusions to the exact relationship between Position & Conversion Rate. There is just too much noise and not enough static variables for to feel anything but a nice fuzzy feeling about the data. 4. In the e-commerce ppc world, if bid rules are created to drive towards revenue/efficiency targets, then position should for example, be at most a secondary weighted value that indicates possible range of bid change. On the atomic level of individual KW performance(if one is using revenue/efficiency targets) - 1st page position is almost irrelevant.
David says:
George, thanks for the response - this is a really great discussion and I'm glad that Chris a/k/a SearchQuant alerted me to it. Ultimately, conversion rate by position is a bit of a red herring in and of itself. Since Google optimizes results based on CPM (CTR X Max CPC - with addition QS factor), the goal is to actually figure out your optimal eCPM (earnings per thousand impressions) by position. Thus, you need to consider the CPM and RPM by position. Since both CPM and RPM decline as you get lower in the rankings (since you get less clicks), the "sweet spot" is the positon at which the differential between CPM and RPM is at its greatest, thus maximizing your profit per thousand impressions. If you accept this to be true, the conversion rate might well be the same by position, but the differential between volume and cost per click are not, which means that even if conversion rate is the same, there is still value to experimenting by position to find the efficient frontier.
David, thanks for the excellent commentary. I totally agree that the volume versus percentage game must be tested depending on the objectives of the advertiser. However, whether you look at this as EPM versus CPM or Earnings per Click vs Cost per Click, the notion that lower gross profit percentage per click or impression might be made up for by additional volume is the same. Testing should be done, but there are costs to testing as well. Experimenting with bids that are likely too high and likely too low can waste more opportunity than is gained by finding the answer, particularly given that the answer one finds is only the answer for that moment, as the "frontier" continually shifts :-) Google's Bid Simulator is fairly revealing in this respect. As we study this tool and the shape of the curves we find very few instances when bidding more than the Earnings per Click suggests would help. It will be interesting to see whether enough folks take advantage of the bid simulator to make it useless. Like Schroedinger's Cat, by observing and reacting to the data we change the landscape we were trying to game.


Check out what others are saying...
[...] The Folly of Position Bidding Managing thin data [...]
[...] by time of day, by season, by stock position, and many other factors depending on the vertical. RKG, Google, and others who have studied the data carefully have found that the value of traffic does [...]
[...] by time of day, by season, by stock position, and many other factors depending on the vertical. RKG, Google, and others who have studied the data carefully have found that the value of traffic does [...]
[...] different advertisers.  While we’ve long argued against position bidding largely because, as Google has confirmed, traffic value doesn’t vary by position, we do receive requests from clients to hit a minimum [...]
[...] previously found that conversion rate doesn’t vary significantly by position and Google has since said as much as well, it will be interesting to take another look, particularly at the top vs side piece.  Often for [...]

Leave A Comment