Feb 242011

Paid Search Success Metrics by Length of Keyword

Most folks with paid search experience understand the “anatomy” of an account:  the head consists of broader, more general terms that offer great volume, while the tail consists of specific, narrower terms that offer great conversion.  If we assume that keyword length is an imperfect-but-still-worthy proxy for the specificity of a phrase, then we’d expect to see a mathematically positive relationship between keyword length and conversion.  Intrigued by this notion, I dug into RKG data to see what relationships are playing out for our clients.


I began with a sample of our clients, determined the linear correlation between conversion rate and length of phrase, then tested that correlation for statistical significance using the t-test for correlation.  I performed these tests within each client separately – since the distributions of conversion rates and keyword lengths are unique to each client, such differences would cloud my results if not accounted for.


The results surprised me at first, but, upon further reflection, make perfect sense.  For the great majority of clients, I found the correlation between keyword length and conversion to be positive and weak, but statistically significant.

A positive correlation signifies that the two variables move in the same direction together, so in this case, longer phrases are associated with larger conversion rates, while shorter phrases are associated with smaller conversion rates.  The absolute value of the correlation measures how dependent or independent the two variables are.  A correlation of 0% means that the two variables are truly independent, while a correlation of 100% (or -100%) means the variables are completely dependent.  Lastly, statistical significance is a stamp of approval to say “this conclusion was not the result of random noise in data, but instead reveals a true, underlying pattern”.

The correlations ranged between 1 and 7%, indicating a positive relationship as I posited, but on a much weaker scale than I imagined.  Correlation between 1 and 7% means that there is some dependence between keyword length and conversion, but that it’s at a minor level.  However minor, though, the correlations were statistically significant (at alpha =0.05 for those who like the details!), so the dependence really is there.

The fact that the correlations are somewhat weak is far less surprising when we consider all the other factors that contribute to conversion rate:  selection of products, price competitiveness, site navigability, appropriateness of landing page, and on and on.  Put in this context, it makes sense that specificity of phrase (as approximated with keyword length) is a contributing factor to conversion, but not a major one.

It’s also important to point out here that I sliced the data by engine, by match type, and by advertiser brand/non-brand.  I found that only non-brand terms on broad match exhibited significance, and that held true in both engines.  Brand terms and exact match terms did not display statistical significance, and thus for these terms we can’t be confident that there exists a real dependence between keyword length and conversion.

A Second Analysis

These conclusions were interesting enough to prod me to dig further.  Although paid search marketers are arguably less able to influence average order value (AOV) than we are conversion, it’s still an important component of success.  Conversion rates coupled with AOV determine the value of a click, so I thought it worthy of a correlation test all its own.

For the great majority of clients this time, I found correlations that were significant, relatively strong, and negative.  In this case, negative really means indirect – so in this case, longer phrases are associated with smaller AOVs while shorter phrases are associated with higher AOVs.  It also turned out that AOV was significantly correlated with keyword length for more than just non-brand keywords on broad match – brand keywords on broad match were significantly correlated too, as were brand keywords on exact match on Google only.

For the non-brand keywords, correlation rose to 15%, and for the brand keywords on broad match, correlation shot up to 20-30%.  I think the rationale behind these results is that more specific (and thus longer) keywords tend to refer to a singular product, or at least a narrower scope of products.  The user searching on this type of phrase, for example “Merrell whiteout 8 waterproof boots”,  is likely looking to buy only that product, whereas a user searching on a shorter, more general phrase, for example “Merrell”, is more likely to appreciate Merrell shoes in general, and thus offers a better chance of buying multiple pairs.

Also keep in mind here that I calculated correlations and significance within each client, so the negative relationship between keyword length and AOV does not imply that on average, "Macy's" has a higher AOV than "Bloomingdale's", but that "Macy's" has a higher AOV than "Macy's cocktail dresses".


Although this study shows that longer keywords tend to have higher conversion rates and shorter keywords tend to have higher AOVs, it would be dangerous to rely on keyword length to be the main driver for either of these metrics.  It all comes back to the cardinal rule of statistics: correlation does not imply causation.  That is, longer keywords in and of themselves don’t boost conversion, but rather both length of keyword and conversion are related to the “lurking variable”, specificity.

So if you’re looking to add some high-converting terms, don’t think long, think specific.  And vice versa – if you’re looking to add some high-AOV terms, don’t think short, think broad (but still relevant!).  If you do that, then the underlying relationships between keyword length and conversion or AOV should naturally follow.


16 Responses to "Paid Search Success Metrics by Length of Keyword"
Matt Lambert says:
Its very difficult to draw conclusions in these things, in my experience. I had a customer that insisted on a single word phrase having been historically successful, although no stats existed. After putting tracking in, we showed the cost per lead as being higher than multiple worded key phrases, but, whenever we switched off the single word campaign, their phones stopped ringing. I knew they were correct as whenever I paused the campaign out of intrigue, they called to let me know the phone had stopped and they knew it was me! In this particular instance, the plural of the word worked, but the singular didn't. I've given up trying to predict things since then. We sometimes would like to know why, but it's more important to know 'if'.
Great post! One thing I would like to point out is that if you are comparing the length of a keyword versus its paid search success, can you look at broad match keywords? I only say this because you are unsure which query is actually triggering those terms. The actual search query may be more long tail than you think (and increasing a shorter term's AOV & conversion rate). This would lead me to believe that you would only want to look at exact match terms when analyzing long tail.
Hi Jen, Nice post. You analysis was the same as my findings I did some time ago. I plotted the mean conversion rate vs word length and found something interesting. For keywords lengths up to 3-4 there is generally a positive coefficient but if you look at 4,5,6 the coefficient becomes negative. My hypothesis is that users typing very long queries look for something so specific that they dont convert as well. I also did a similar analysis for CTR but the results were statistically inconclusive. Also, as you pointed out the connection is weak (but statistically significant). I think this highlights an important aspect of search: The long tail story is very nuanced as the ability to target an in-market customer with a very long tail keyword is diluted by the search engine algorithms that broad match your and your competitor's keywords to a query. Regards Sid
Matt - Thanks for weighing in. I agree there's a lot about paid search performance that we can't predict in advance, which is why RKG is a great advocate of trying out a complete set of keywords, tracking performance as finely as possible, and acting on that performance feedback quickly. Francis - Thanks for pointing out that nuance. Fortunately, I did break my analysis out by match type, as well as by engine and advertiser brand/non-brand. The results are illustrated in the 2 charts above. I found that exact match keywords did not exhibit a significant relationship between conversion and keyword length. Siddharth - I'm glad to hear your analysis yielded similar conclusions; it's reassuring for any statistician to hear that! Also I think your hypotheses are reasonable and insightful. Thanks for chipping in.
Jen, Sid, I'd add one other thought here: that there is probably a degree of selection-bias in the results. Poorly performing ads (that can't be saved with ad copy, landing page changes, etc) are bid down by high-quality bid management systems to the point that they essentially vanish from the data. Similarly, strong performers are bid more aggressively and hence generate a disproportionate share of the traffic. These statements are true regardless of the length or number of KW present. In a sense, we should add the caveat: the relationship is weak among strong performing keywords. The really generic one and two word phrases probably don't survive long enough to impact these results. Just a thought!
Terry Whalen says:
Jen, great post - thank you. I must say I love the RKG blog. You folks write so clearly and you look at relevant data. It is a pleasure (Sid, ditto goes for you!)! My two cents is that it'd also be interesting to look at cost/conversion (or ROAS, or whatever cost-based metric makes sense for each client), since to some degree cost/conversion is even more important than conversion rate. More-specific keywords, which often correlate to higher keyword lengths, can have higher average CPCs and thus negatively affect cost/conversion. At the end of the day, I care more about conversions and cost/conversion than conversion rates. Thoughts?
Great point, Terry. I'll have to look at a measure of efficiency (ROAS, A/S, CPO, etc.) when I re-visit the data.
Jen, Terry, since we control the bidding one would hope to find that the efficiency metric is largely independent of any consideration other than the client's targets, which might vary due to conversion value differences. The beauty of studying the conversion rates or sales/click absent costs is that it tells us about the consumer behavior, not the degree to which our bid management efforts are rational. I've yet to hear a good argument as to why someone would be willing to pay more for a conversion based on the length of the keyword or volume of traffic on it.
Terry Whalen says:
Hmm, it's very possible that I'm missing something - Having said that, I'm more interested in the account behavior than I am the consumer behavior. I kind of only care about the money part. So, if there was a strong correlation between keyword length and conversion rate, I would think it was interesting, but I'd also want to study whether these longer, higher-converting keywords tend to be bid up more often; in theory, if folks or machines are doing good bid optimization, bids should correlate with conversion rates (average order sizes, etc.); but I'd be interested in whether for some reason the longer-tail keywords tend to get bid up higher than they should be, or whether there is some inefficiency or imbalance with respect to longer keyword lengths and CPA or ROAS or ROI. Thoughts?
Terry, I totally agree that studying the effectiveness of the bid management methodology is a valuable exercise. We do these kinds of self-checks all the time. The problem is: if we find our system is handling things well, the study won't say anything that's useful to others. If we find that our system is handling it badly...it also doesn't provide information that's useful to anyone else. From a blogging perspective a study showing that our bid management system works well just doesn't seem very valuable. To our thinking, the key questions center around consumer behavior and those issues that impact traffic value. These are factors that everyone should fold into their bidding algorithms. Whether they have been folded in or not yet is irrelevant in terms of what we should do today.
Wiesner Vos says:
Hi Jen Thanks for an interesting post. I just have one small statistical comment. You state that "a correlation of 0% means that the two variables are truly independent". That is not strictly speaking true. Generally, if variables are independent it would imply that they are uncorrelated; however, if two variables are uncorrelated it does not necessarily imply independence. Wiesner
terry whalen says:
Right - thanks for spelling it out for me - *now* I get what you were saying. Makes total sense!
Yes Wiesner -- Very good distinction. Luckily this doesn't really impact the study, but it's always better to be precise and accurate!


Check out what others are saying...
[...] This post was mentioned on Twitter by Ryan Beale, SEM Software, Keyword Grouper Guy, search tool madman, PPC Marketing and others. PPC Marketing said: Paid Search Success Metrics by Length of Keyword: Most folks with paid search experience understand the “anatomy... http://bit.ly/gdU2V3 [...]
[...] Paid Search Success Metrics by Length of Keyword, Rimm Kaufman [...]
[...] Paid Search Success Metrics by Length of Keyword – Rimm-Kaufman Group [...]

Leave A Comment