THE RKGBLOG

Quality Score: the Great Misnomer

One of the great misnomers in our space is the phrase “Quality Score”.

The name implies that it is a measure of the ad copy’s merit when in fact it measures more than this. I’m not talking about “relevancy” or landing page quality, I’m talking about “commercial intent.” This last piece has been neglected in our industry’s discussions about QS.

It is simpler to think of QS as something wholly under an advertiser’s control. We control the ad text associated with each keyword which directly impacts the CTR and the relevancy of that copy to that keyword, and we control the landing pages. The only excuse for poor Quality Scores seems to be advertiser malfeasance.

Indeed, QS is a favorite target of outsiders who don’t know much about the paid search game. It is easy to download an account and run a pivot table on the number of keywords by QS and conclude: “25% of your KWs have a QS of less than X; that’s money left on the table, you need to fix that, harrumph harrumph!”

The trouble with this simplistic view is:

  1. A higher QS may not be in the advertiser’s best interest. More compelling copy may increase CTR but may simultaneously decrease conversion rate to a degree that the trade off isn’t worthwhile. Drawing in “bottom feeders” with promotional copy can sabotage performance.
  2. A higher QS may not even be possible. If the commercial intent of users searching with that phrase is low, there may be no wording under the sun that can improve the QS beyond a certain point.

It is this last piece that the industry outsiders really don’t get.

Consider a user search on “motorcycle”. The user’s intent isn’t particularly clear. They may want motorcycle gear, they may want information on nearby races, they might want videos, or simply sites for motorcycle enthusiasts.

Sidebar: check out the local dealership Google map eating up prime real estate! Is this a test or are others seeing this?

The fact that none of the ads appear above the organic listings indicates that Google doesn’t see much commercial intent here, and therefore none of the advertisers meet the minimum Bid * QS threshold needed to secure a promoted listing. I’d wager that the QS of NONE of these ads is above an 8 and wouldn’t be surprised if none were above a 5.

The advertisers can flip copy until they’re blue in the face and it won’t impact those scores.

Note: this doesn’t mean the ads will perform badly. Indeed, these could be tremendously valuable ads if those with commercial intent use the ads and purchase, and the price paid for the traffic makes sense.

Some will argue that these low QS ads should be eliminated from the account regardless of performance because their QS drags down the AdGroup, Campaign and Account Quality Scores. I’d argue that for a fully developed account with a thorough keyword list, smart ad groups and good copy that’s almost always a mistake, BUT I’d also argue that Google shouldn’t force us to make that choice.

Advertisers are willing to pay for the traffic because at least some of the users have found them valuable, we shouldn’t suffer any negative consequences however minor for choosing to do so.

A PROPOSED SOLUTION

Using QS as the metric to encompass all of these different notions of user experience makes all kinds of operational sense. Parsing the rankings into categories for CTR, landing page, relevancy and commercial intent would be helpful to us geeks, but might make other people lose their minds.

A simple metric to help advertisers understand at a glance whether the QS is below par because of something within their control might simply be to provide a benchmark against all other ads running on that same keyword. An advertiser that sees they have a QS of 5, but that score is in the 95 percentile of all ads on that phrase might save time for high-value projects instead of trying to fix something that isn’t broken.

Granted, it might complicate life for the average consultant who may now actually need to do some work to find something to point out, but I for one don’t mind them having to look a bit deeper before throwing rocks.

Technorati Tags:

Comments
24 Responses to “Quality Score: the Great Misnomer”
  1. Hi George, great article as always!

    I’ve been using the new Analyze the Completion tool to make CTR comparisons, share of clicks, etc. to understand if I can even improve on some of my keywords, but it is very cumbersome and full of noise.

    I recently wrote an article about it: http://www.wordstream.com/blog/ws/2010/11/03/adwords-analyze-competition

    Why can’t Google just give us this data through the API at the keyword/ad level?

  2. Thanks for your comments, Chad.

    For those of us working at scale, getting the data through the API is imperative. No question they can do it, we just need to get enough people to ask.

  3. George, you hit the nail right on the head. Thanks for cutting through the nonsense!

  4. Another great post George. I once got my mind spinning, tweaking ad copies and landing pages to get higher QS. I was successful on some, and not on others. There are instances when no matter how much we tweak, the QS just won’t budge.

    Interesting to note though is that I have low QS keywords that are out-performing the ones with higher QS in terms of conversions and ROI.

    I totally agree with this statement, “An advertiser that sees they have a QS of 5, but that score is in the 95 percentile of all ads on that phrase might save time for high-value projects instead of trying to fix something that isn’t broken.”

  5. Matt McGee says:

    Hey George – the map in the sidebar thing is what Google announced a couple weeks ago re: new local SERP displays. See this on SEL:

    http://searchengineland.com/new-place-search-shows-googles-commitment-to-local-53990

    Matt

  6. Kenny says:

    Thank you for this – totally true. If you are not toyota.com or some flavor thereof, guaranteed your QS and CTR on the word “toyota” are going to stink no matter what you say in your ad. However, the volume on a word like that is so huge, that even at a CTR of under 1%, the traffic available is huge, and if it converts well, then great. Advertisers miss out by focusing on QS and thinking that a word with a poor CTR like that will negatively impact their CPCs so much that they will just pause them or avoid them altogether. Often a big mistake…THAT’s what can can be money left on the table.

  7. Matt, thanks for setting me straight! I read SEL religiously, but missed this one somehow…or read it and didn’t remember…hoping it was the former :-)

  8. Andrew, Jun, Kenny, thanks so much for sharing your thoughts and experiences. If we all shout loudly enough maybe…

  9. Vale says:

    When you upload you new campaign/keyword using Adwords Editor, your keywords are all assigned a QS. This is done before you get a single click. My understanding of this initial QS is an average for that keyword based on what other advertisers are getting. As we know QS’s main factor is the CTR, if the average competitor has a low CTR, then I can assume there may be low commercial intent. I used this to gain some insight into the keywords I choose and even competitors.

  10. Thomas says:

    George, Matt,

    Interesting thing about the map on the right (above adwords) is, that it covers the adwords-ads, when scolling down. This is happening with Firefox, not IE. I hope, this is a bug?!

    Best,
    Thomas

  11. Interesting point, Vale. Again, I wouldn’t assume that low commercial intent means the KW will perform poorly, that’s often not the case, but it certainly may inform decisions about initial bidding and that the copy needs to be particularly clear as to what the user will find when they get to the destination page.

  12. Great observation, Thomas. If it doesn’t happen in Chrome I’d assume it’s a bug :-) I’m told that Google product folks read RKG Blog, so hopefully they’ll see this.

  13. Hi George,

    Long-time reader, first-time commenter. Another excellent article with some great points. Keep up the great work!

    As for the map covering the ads, I think it’s designed that way for this reason: if the map disappeared completely from view, the map icons labeled A,B,C,etc down the page would be of no benefit. The map has to scroll with the user so they can see where each listing is on the map. At least I think this is the intent (makes sense to me). Apparently Google calculated that covering the ads on the side was worth it.

    -Tyson

  14. Tyson, thanks for reading and for your kind words.

    You make a keen observation. I bet you’re right. No question that Google has done the math on this. Likely we see this behavior when the commercial intent is low and the likelihood of wanting local info is high.

  15. Bradd Libby says:

    Google’s Bid Simulator provides estimates of both ‘Top Impressions’ (the number of ad impressions that appear above the organic results) and of just ‘Impressions’ (the total number of times an ad is shown).

    The ratio of these two quantities sounds like it could be used as a rough measure of commercial intent.

  16. REALLY interesting notion, Bradd! I like that idea a great deal!

  17. billy wolt says:

    I am beginning to think that QS is irrelevant from a cpc, ctr, and position perspective.

    I have seen some funny things that defy the QS rule (high ctr/low qs/high position, low ctr/high qs/high position, low ctr/low qs/high position, high ctr/high qs/low position(even when not close to max cpc))

    (*high position 1-3, low position bottom of page 1 or on page 2)

  18. Marc Adelman says:

    George,

    Another great post about that “too cool for school” Quality Score. Most performance metrics in the SEM toolbox are the direct results of click and conversion performance of KWs and Ads. Quality score is not a direct performance metric like conversion rate (orders/clicks) or CTR (clicks/impressions) which are irrefutable.

    Instead, QS is a composite and weighted metric that descends from “Google Heaven”. It does not factor in actual ecommerce performance like Conv%, Cost/Sales, Sales and Orders – which are much closer to what the advertiser wants to take to the bank. Google decides the weight and data points factored in and throws us mortals a few bones through some videos and explanation on an Adwords Help page into the inner workings of this metric.

    They key for me, has been learning how to understand the true properties of QS as it pertains to its use in KW optimization. If you search Google (of course) for Adwords Quality score, you might find a video result from GoogleBusiness titled “Introduction to the Google Ad Auction”. Interesting that the title was not “Introduction to Google Adwords Quality Score”. In this video, the focus on explaining QS is the part it plays in the formula for Adrank. It is clearly a multiplier. (Ad Rank = CPC bid × Quality Score).

    Therefore, if we view QS like a direct performance metric, and optimize based solely on a Good = high QS, Bad = low QS basis, we might be taking actions that don’t factor in the true *qualities* of quality score.

    If we understand QS in the following way, it can lead to a direct impact on performance.

    QS is a multiplier. It is more of a sub-factor in larger vital equation in the Ad auction and not a direct metric that defines a black and white, good and bad quality of a given KW. Additionally, this multiplier is a reward per se, for the positive elements of your KW, adgroup, landing page, account, and other factors as listed in Google’s breakdown of the QS formula. The higher quality of all of these factors rewards a high quality score and leads to a greater multiplier of your KW bid. What this can mean in practice, is that IF your KW has other performance metrics pointing to a predictable possibility for increase in overall performance, quality score can then be factored as a multiplier or really an accelerator to the KWs ability to move up in an auction.

    Sorry for the long winded comment – I haven’t commented in many moons, so I guess it just all came pouring out. Wishing the best to all at RKG.

  19. Good to have you back, Marc! Great commentary!

  20. Rob Willox says:

    If it is indeed true that appearing above the organic search results is a measure of commercial intent as it may well be Bradd’s point about Google’s Bid Simulator is a good one to consider.

    We all tend to get hung up on particular and specific aspects of kw selection and QS and its effects on the areas we know it effects but as Marc highlights QS is a multiplier and not an absolute measure of an individual kw’s actual or expected performance.

    On the other point regarding the new Places integration into serps, have just checked in Chrome and it remains at the top of ads as you scroll but on both FF & IE it scrolls with the page. On that basis, and assuming Google’s own browser is showing Google’s preferred option, it tends to suggest it is a bug in the others or Google has got the code wrong when rendering the page.

Trackbacks
Check out what others are saying...
  1. [...] This post was mentioned on Twitter by Rimm-Kaufman Group, Chad Summerhill, SEM Software, Search Marketing, Negative Keywords and others. Negative Keywords said: Quality Score: the Great Misnomer: One of the great misnomers in our space is the phrase “Quality Score”. The n… http://bit.ly/a9R18d [...]

  2. [...] Quality Score: the Great Misnomer, Rimm Kaufman [...]

  3. [...] Not really, and don’t call me Shirley. Quality Score does not measure the quality of the copy; it is a mixture of CTR and commercial intent, neither of [...]

  4. [...] a GREAT new feature in the UI that should help paid search managers of all shapes and sizes. About 18 months ago I gave my friends at Google some grief over calling Quality Score “quality” score, [...]