An Empirical Study of Search Engine Advertising Effectiveness
August 15, 2006This article is best viewed in PDF due to the embedded equations:
An Empirical Study of Search Engine Advertising Effectiveness (pdf)
An Empirical Study of Search Engine Advertising Effectiveness
Sanjog Misra, Simon School of Business University of Rochester
Edieal Pinker, Simon School of Business University of Rochester
Alan Rimm-Kaufman, Rimm-Kaufman Group LLC
1. Introduction: Jupiter Research projects that online search advertising expenditure for 2006 will be approximately $6.5billion. This is about 40% of all online advertising and is expected to reach $11.1 billion, by 2011. In recent years there has also been a rapid move away from online display advertising (a.k.a. banner ads) and towards search related advertising. This method of advertising has become attractive for several reasons. Search engines like Google and Yahoo! attract very heavy traffic making them useful marketing channels. The search engines are also attractive because, through the search phrases, they elicit information about what potential consumers are interested in allowing ads to be targeted to these interests. In general Internet marketing has attracted interest because of the ability to more closely measure its impact than traditional marketing efforts such as TV, print and direct advertising. The same advantage holds for search engine marketing. Ironically, despite the vast amount of data being continuously collected electronically about consumer responses to search engine advertising, many advertisers fail to use these effectively, and some fail to use these data at all. This reduces the economic benefit advertisers can derive from this marketing channel.
Search marketing advertisers create ads that are then mapped to search phrases. These ads have links that associate them to a set of webpages. The advertisers place maximum bid amounts for each search phrase signifying their willingness to pay for a click on the ad associated with that phrase. The bids themselves (in the case of Yahoo!) and the product of bids and clicks (in the case of Google2) determine the position of the ad on the search page. Some of the clicks that occur lead directly to sales. Ultimately advertisers would like to know the value of each click and calibrate their maximum bid amounts accordingly. Along these lines it would be useful to understand how position impacts click through rates (CTR) and ultimately sales. Advertisers would also like to know how ad characteristics might influence CTR and sales as well. Unfortunately despite the enormous detailed data available considerable challenges remain to answering these questions.
In this presentation we will empirically explore some of these questions and discuss some of the related statistical and data challenges. As work in progress our aim is to give some answers to the main problems faced by search engine advertisers and to identify problems of interest to researchers. We will present an analysis of a rich data set of advertising done by a single firm on Yahoo! and Google over the course of several months.
This form of advertising is also known as “keyword advertising”, “pay per click advertising” and “search engine marketing.” Essentially, these ads are place by the search engine based on a match between the search phrase entered but the consumer and the keywords associated with the ad
Google’s ranking mechanism is actually less transparent than depicted here.
Description of Data Set: We obtained a data set comprised of paid search impressions, clicks, and orders for an online specialty retailer selling automotive parts and accessories. For the advertiser in question we had the following data: the mapping of search phrases and ads that they ran on Yahoo and Google for three months in 2006. For each search phrase we had the number of daily impressions, number of clicks, number of orders, average display position, and cost per click on both Yahoo! and Google. Summary statistics appear in Table 1 and give an indication of the size and complexity of the management problem faced by the advertiser.
3. The Empirical Investigation: In what follows, we discuss the impact of advertisement characteristics on the click through rate and the order rate.
3.1 Clicks: The click-through-rate is defined as the ratio of clicks to impressions (number of searches where the ad was displayed) and is a standard measure used in industry parlance. Borrowing from this construct, we model clicks using a binomial regression conditioned on the number of impressions. In other words, ()~,itititCBinpI.
itC anditI are the number of clicks and impressions (respectively) of advertisement i on day t. Note that the “success” probability,itp, is the click-through-rate and we model that using a logistic transform, ()()()()exppos;1exppos;iiitiitiiitiXpXβϕωβϕω′+=′++.
In the above equation, iX is a vector of advertisement specific characteristics and is the position of advertisement i on day t. Note that we allow position to have a heterogeneous and nonlinear impact by positspecifying:()212;iiiizzϕωωω=+. This heterogeneity is critical because each ad may demonstrate a different sensitivity to its position. We assume that the iω’s are distributed normally (independently) with means []12,ωω and variances Consequently, the likelihood for the clicks data can be written as 2212,vv
()()()()()()()11exppos;11exppos;1exppos;itititCINTiiitiiitiiitiiiitiXLdFXXβϕωωβϕωβϕω
with ()()(2111222;,;,iiiFvωωωωω=Φ×Φ,Φbeing the normal distribution function. We use Monte-Carlo methods to approximate the integral in the likelihood and then maximize (Simulated Maximum Likelihood) to obtain the parameters of interest. Since we have two data sets (Yahoo and Google) we repeat the analysis for each case. We discuss each in turn.
Yahoo: The ad characteristics()iX we used included product category dummies and dummies for the number of words in the advertised phrase. (Note that due to match type algorithms at the engine, the numbers of words in the user’s search query may exceed the number of words in the advertised phrase. For example, an advertiser buying clicks on the search phrase “blue widget” who opts for one of the looser non-exact match algorithms could see their ad displayed on searches for “blue widget”, “inexpensive blue widget”, and “blue green widgets”.) Since the category effects were included simply for control purposes we will not discuss them here. The more interesting results pertain to the effect of the number of words in the advertised phrase and position on the click through rate. In the Yahoo data, the impact of the number of words was monotonically increasing. In other words more specific ads displayed in more complex search environments had a higher probability of being clicked. This is not surprising, since, there is a typically a strong matching of user intent when the number of words in the search phrase is large. In a sense the number of words serves a sa proxy for the specificity of the user’s search. The effect of position of clicks was also in accordance with intuition. We found that [][]12,0.09,.013ωω=− implying that lower positions (remember that 1 is a higher position than 2) have lower click through rates but the effect is diminishing. There was also significant heterogeneity in this linear effect, evidenced by the estimates, [][]12,0.125,0.001vv=. Together, these results suggest that position has a significant impact on clicks but this effect varies significantly over ads.
Google: The results for the Google click through data were qualitatively similar. The category effects were all of the same sign as in the Yahoo results, as were the effects of words and position. This lends credibility to the model and our analysis since the data come from two very different sources. Again, as with Yahoo, we found that ads displayed in the context of more complex searches tend to have higher click-through rates. However, the effect seemed to be stronger in the Google results. A similar pattern was observed with the position effect. We found that [][]12,0.173,.0ωω=− suggesting that the effect of position on clicks was twice as large in Google as in Yahoo. In contrast, we found that, [][]12,0.079,0.0003vv=, implying that the level of heterogeneity in these effects was substantially lower than in Yahoo. Note that the direction of the effects is the same in both data. What is different in the average magnitude of the effect and the spread across ads. We note that while our coefficients for CTR were decreasing with position, the first few positions (between zero and three) are often rendered at the top of the page, while the following positions are rendered on the right side of the page. Thus, while traffic and CTR are typically monotonically decreasing with position, it is important to recognize that some users on certain renderings of search results pages find position three more prominent than position two. This non-monotonicity in display location may explain some of the variance in our coefficient estimates. Given that Google varies the location of the various ad positions more often than Yahoo, we find the lower level of heterogeneity noteworthy. This could suggest that Google has a more precise understanding of the economic value of each page location than Yahoo, and allocates ad inventory accordingly.
There could be a number of explanations for our results. First, it is well known that the Google and Yahoo business models are very different. This is particularly true when it comes to the definition of the “position rank” construct. While Yahoo sets the position of the ads based solely on the bids of the advertisers, Google uses a more complex combination of bids and the number of clicks (i.e. total revenue). This of-course calls into question the Google analysis and requires that we explore alternative mechanisms (possibly simultaneous equation methods) to model the Google phenomenon. This is one object of our ongoing research. Nevertheless, the similarities (and differences) provide some new insights into the nature of click through rates and its antecedents. 3.2 Orders: The model of Orders was very similar to the earlier model described for clicks. In this case Orders were modeled using a Binomial regression conditioned on clicks. We will skip details for brevity.
Yahoo: The effect of the number words on order exhibited some interesting results. We found that searches with a few words (one or two) and those with many (five) were more likely to result in orders than searches with intermediate number of words (three or four). We conjecture that this is a result of the nature of the match between the consumer’s search and the ad. To elaborate, imagine a consumer who searches for a product and uses five words to describe what she needs. If a given ad were to show up in that search context it is likely that this ad is particularly good match. In such a case, conditional on clicking that ad, the consumer is also likely to buy. Now imagine the other extreme. The consumer uses a single word to describe the search and consequently the natural search results will be poor matches. Now, under the scenario that the consumer finds a relevant ad and clicks on it, an order is likely simply because the outside alternative is poor. The 2-3 word case represents a situation where the natural search results are stronger and the consumer envisions a higher payoff from continuing the search and decides not to buy. While, we believe this to be a plausible story further behavioral research is required to validate our conjecture. The Yahoo results also reveal that there is no significant impact of position on the order rate. This implies that once the effect of position on clicks is accounted for there is no residual impact on the likelihood of an order. We note that this advertiser included SKU designations in their ad portfolio, such as “ACME1234.” These highly precise single word phrases may have significantly higher conversion than generic single word phrases, such as “WIDGET”. The effectiveness of using number of words as a proxy for the precision of an ad is therefore dependent upon the context.
Google: The Google results from the order model are in stark contrast to the Yahoo results. First, there is no discernable effect of the number of words. Second, there seems to be a position effect. Continuing our discussion from the Clicks results, it is difficult for us to disentangle the true results from spurious ones because of the nature of the Google “position” construct. We hope to do so in our ongoing research. 4. Looking forward: The second and third largest search engines, Yahoo and MSN, respectively, have joined Google in ranking ads by a mix of maximum bid, click through rate, and subjective factors, or soon will do so. This makes position-based bid approaches much more difficult to control. Advertisers must rely on instantaneous and average position data to manage bids in their attempt to hold a certain position on the page. This study, conducted in the waning months of the older Overture (now Yahoo) platform, offers a rare glimpse into the tradeoff between position and bid, and hopefully informs the newer more complicated environment.
PDF: An Empirical Study
of Search Engine Advertising Effectiveness (pdf)
Sanjog Misra, Simon School of Business University of Rochester
Edieal Pinker, Simon School of Business University of Rochester
Alan Rimm-Kaufman, Rimm-Kaufman Group LLC
1. Introduction: Jupiter Research projects that online search advertising expenditure for 2006 will be approximately $6.5billion. This is about 40% of all online advertising and is expected to reach $11.1 billion, by 2011. In recent years there has also been a rapid move away from online display advertising (a.k.a. banner ads) and towards search related advertising. This method of advertising has become attractive for several reasons. Search engines like Google and Yahoo! attract very heavy traffic making them useful marketing channels. The search engines are also attractive because, through the search phrases, they elicit information about what potential consumers are interested in allowing ads to be targeted to these interests. In general Internet marketing has attracted interest because of the ability to more closely measure its impact than traditional marketing efforts such as TV, print and direct advertising. The same advantage holds for search engine marketing. Ironically, despite the vast amount of data being continuously collected electronically about consumer responses to search engine advertising, many advertisers fail to use these effectively, and some fail to use these data at all. This reduces the economic benefit advertisers can derive from this marketing channel.
Search marketing advertisers create ads that are then mapped to search phrases. These ads have links that associate them to a set of webpages. The advertisers place maximum bid amounts for each search phrase signifying their willingness to pay for a click on the ad associated with that phrase. The bids themselves (in the case of Yahoo!) and the product of bids and clicks (in the case of Google2) determine the position of the ad on the search page. Some of the clicks that occur lead directly to sales. Ultimately advertisers would like to know the value of each click and calibrate their maximum bid amounts accordingly. Along these lines it would be useful to understand how position impacts click through rates (CTR) and ultimately sales. Advertisers would also like to know how ad characteristics might influence CTR and sales as well. Unfortunately despite the enormous detailed data available considerable challenges remain to answering these questions.
In this presentation we will empirically explore some of these questions and discuss some of the related statistical and data challenges. As work in progress our aim is to give some answers to the main problems faced by search engine advertisers and to identify problems of interest to researchers. We will present an analysis of a rich data set of advertising done by a single firm on Yahoo! and Google over the course of several months.
This form of advertising is also known as “keyword advertising”, “pay per click advertising” and “search engine marketing.” Essentially, these ads are place by the search engine based on a match between the search phrase entered but the consumer and the keywords associated with the ad
Google’s ranking mechanism is actually less transparent than depicted here.
Description of Data Set: We obtained a data set comprised of paid search impressions, clicks, and orders for an online specialty retailer selling automotive parts and accessories. For the advertiser in question we had the following data: the mapping of search phrases and ads that they ran on Yahoo and Google for three months in 2006. For each search phrase we had the number of daily impressions, number of clicks, number of orders, average display position, and cost per click on both Yahoo! and Google. Summary statistics appear in Table 1 and give an indication of the size and complexity of the management problem faced by the advertiser.
3. The Empirical Investigation: In what follows, we discuss the impact of advertisement characteristics on the click through rate and the order rate.
3.1 Clicks: The click-through-rate is defined as the ratio of clicks to impressions (number of searches where the ad was displayed) and is a standard measure used in industry parlance. Borrowing from this construct, we model clicks using a binomial regression conditioned on the number of impressions. In other words, ()~,itititCBinpI.
itC anditI are the number of clicks and impressions (respectively) of advertisement i on day t. Note that the “success” probability,itp, is the click-through-rate and we model that using a logistic transform, ()()()()exppos;1exppos;iiitiitiiitiXpXβϕωβϕω′+=′++.
In the above equation, iX is a vector of advertisement specific characteristics and is the position of advertisement i on day t. Note that we allow position to have a heterogeneous and nonlinear impact by positspecifying:()212;iiiizzϕωωω=+. This heterogeneity is critical because each ad may demonstrate a different sensitivity to its position. We assume that the iω’s are distributed normally (independently) with means []12,ωω and variances Consequently, the likelihood for the clicks data can be written as 2212,vv
()()()()()()()11exppos;11exppos;1exppos;itititCINTiiitiiitiiitiiiitiXLdFXXβϕωωβϕωβϕω
with ()()(2111222;,;,iiiFvωωωωω=Φ×Φ,Φbeing the normal distribution function. We use Monte-Carlo methods to approximate the integral in the likelihood and then maximize (Simulated Maximum Likelihood) to obtain the parameters of interest. Since we have two data sets (Yahoo and Google) we repeat the analysis for each case. We discuss each in turn.
Yahoo: The ad characteristics()iX we used included product category dummies and dummies for the number of words in the advertised phrase. (Note that due to match type algorithms at the engine, the numbers of words in the user’s search query may exceed the number of words in the advertised phrase. For example, an advertiser buying clicks on the search phrase “blue widget” who opts for one of the looser non-exact match algorithms could see their ad displayed on searches for “blue widget”, “inexpensive blue widget”, and “blue green widgets”.) Since the category effects were included simply for control purposes we will not discuss them here. The more interesting results pertain to the effect of the number of words in the advertised phrase and position on the click through rate. In the Yahoo data, the impact of the number of words was monotonically increasing. In other words more specific ads displayed in more complex search environments had a higher probability of being clicked. This is not surprising, since, there is a typically a strong matching of user intent when the number of words in the search phrase is large. In a sense the number of words serves a sa proxy for the specificity of the user’s search. The effect of position of clicks was also in accordance with intuition. We found that [][]12,0.09,.013ωω=− implying that lower positions (remember that 1 is a higher position than 2) have lower click through rates but the effect is diminishing. There was also significant heterogeneity in this linear effect, evidenced by the estimates, [][]12,0.125,0.001vv=. Together, these results suggest that position has a significant impact on clicks but this effect varies significantly over ads.
Google: The results for the Google click through data were qualitatively similar. The category effects were all of the same sign as in the Yahoo results, as were the effects of words and position. This lends credibility to the model and our analysis since the data come from two very different sources. Again, as with Yahoo, we found that ads displayed in the context of more complex searches tend to have higher click-through rates. However, the effect seemed to be stronger in the Google results. A similar pattern was observed with the position effect. We found that [][]12,0.173,.0ωω=− suggesting that the effect of position on clicks was twice as large in Google as in Yahoo. In contrast, we found that, [][]12,0.079,0.0003vv=, implying that the level of heterogeneity in these effects was substantially lower than in Yahoo. Note that the direction of the effects is the same in both data. What is different in the average magnitude of the effect and the spread across ads. We note that while our coefficients for CTR were decreasing with position, the first few positions (between zero and three) are often rendered at the top of the page, while the following positions are rendered on the right side of the page. Thus, while traffic and CTR are typically monotonically decreasing with position, it is important to recognize that some users on certain renderings of search results pages find position three more prominent than position two. This non-monotonicity in display location may explain some of the variance in our coefficient estimates. Given that Google varies the location of the various ad positions more often than Yahoo, we find the lower level of heterogeneity noteworthy. This could suggest that Google has a more precise understanding of the economic value of each page location than Yahoo, and allocates ad inventory accordingly.
There could be a number of explanations for our results. First, it is well known that the Google and Yahoo business models are very different. This is particularly true when it comes to the definition of the “position rank” construct. While Yahoo sets the position of the ads based solely on the bids of the advertisers, Google uses a more complex combination of bids and the number of clicks (i.e. total revenue). This of-course calls into question the Google analysis and requires that we explore alternative mechanisms (possibly simultaneous equation methods) to model the Google phenomenon. This is one object of our ongoing research. Nevertheless, the similarities (and differences) provide some new insights into the nature of click through rates and its antecedents. 3.2 Orders: The model of Orders was very similar to the earlier model described for clicks. In this case Orders were modeled using a Binomial regression conditioned on clicks. We will skip details for brevity.
Yahoo: The effect of the number words on order exhibited some interesting results. We found that searches with a few words (one or two) and those with many (five) were more likely to result in orders than searches with intermediate number of words (three or four). We conjecture that this is a result of the nature of the match between the consumer’s search and the ad. To elaborate, imagine a consumer who searches for a product and uses five words to describe what she needs. If a given ad were to show up in that search context it is likely that this ad is particularly good match. In such a case, conditional on clicking that ad, the consumer is also likely to buy. Now imagine the other extreme. The consumer uses a single word to describe the search and consequently the natural search results will be poor matches. Now, under the scenario that the consumer finds a relevant ad and clicks on it, an order is likely simply because the outside alternative is poor. The 2-3 word case represents a situation where the natural search results are stronger and the consumer envisions a higher payoff from continuing the search and decides not to buy. While, we believe this to be a plausible story further behavioral research is required to validate our conjecture. The Yahoo results also reveal that there is no significant impact of position on the order rate. This implies that once the effect of position on clicks is accounted for there is no residual impact on the likelihood of an order. We note that this advertiser included SKU designations in their ad portfolio, such as “ACME1234.” These highly precise single word phrases may have significantly higher conversion than generic single word phrases, such as “WIDGET”. The effectiveness of using number of words as a proxy for the precision of an ad is therefore dependent upon the context.
Google: The Google results from the order model are in stark contrast to the Yahoo results. First, there is no discernable effect of the number of words. Second, there seems to be a position effect. Continuing our discussion from the Clicks results, it is difficult for us to disentangle the true results from spurious ones because of the nature of the Google “position” construct. We hope to do so in our ongoing research. 4. Looking forward: The second and third largest search engines, Yahoo and MSN, respectively, have joined Google in ranking ads by a mix of maximum bid, click through rate, and subjective factors, or soon will do so. This makes position-based bid approaches much more difficult to control. Advertisers must rely on instantaneous and average position data to manage bids in their attempt to hold a certain position on the page. This study, conducted in the waning months of the older Overture (now Yahoo) platform, offers a rare glimpse into the tradeoff between position and bid, and hopefully informs the newer more complicated environment.