| Title: | Can Agencies be Rated? |
| URL: | http://www.rimmkaufman.com/rkgblog/2009/12/28/can-agencies-be-rated/ |
| Printed: | March 16, 2010 |
| Source: | The Rimm-Kaufman Group Blog, info@rimmkaufman.com |
- December 28, 2009
- 10 comments
Sometime soon Forrester will release the latest version of their SEM agency rankings, and once again the Rimm-Kaufman Group will fail to make the cut. The reason? Forrester only ranks those agencies that offer both paid search and organic search optimization. They exclude specialists like RKG, didit and Efficient Frontier on the paid search side and Net Concepts, Audette Media and other fine specialty shops in SEO.
But even if they did include the specialists, on what basis would they make their judgments? Do they look at ad level data from each agency’s clients to find out whether the programs are well-managed? Do they determine whether within that granular data the efficiency targets set by the clients have been reached? Do they aggregate the low traffic granular data to determine whether the “tail” is being managed properly? Do they evaluate the depth/coverage of KW lists, or the match between KW – landing page – and ad text across the portfolio? Do they find out whether under each agency’s management competitive (aka non-brand) search has grown as a percentage of total site revenue? Do they look at any data at all? Would they even know how to evaluate that data if they did see it?
Client satisfaction surveys might say something, but we’ve certainly run into advertisers who are delighted with their current agency even though their program is demonstrably broken.
We’ve sometimes griped that when we take over management of an account the engines should wipe the account and domain Quality Score history clean and start with an assumption that the QS will be much better going forward given our agency’s track-record.
But do the engines even know which agencies are good or bad? The engines could certainly speak to which agencies churn through clients. They probably have a sense of which agencies’ analysts are consistently knowledgeable, and stick around for a good while. And, I suppose there might be some sort of average QS metric that could be measured across an agency’s accounts that would provide some insight into how well they write copy — not a great metric, as an agency writing “Free puppies with every order” may generate terrific Quality Scores while doing great disservice to their clients who don’t offer free puppies. The engines could also track the number or percentage of an agency’s client-facing staff that have passed a certification exam.
None of these are really adequate measures of excellence, though.
The engines suffer the same basic problem that Forrester suffers: absent granular conversion data, it’s hard to evaluate a program or the agency that put it together.
As an agency we get to see all the gory details when we take over programs built by other agencies. We get to see the sometimes awful KW lists, ad copy and landing page choices. We see the in-artful use of match-types and negatives. We often get to see the granular performance data showing incompetent bid management — recently, we took over an account where all the bids were in increments of 25 cents! :-) As a result we have a very good sense of which agencies are awful (some of them are rated highly every year by Forrester!). We see many of their former clients because they churn through them, and the programs are in consistently lousy shape. We wish there was a way to “out” them as a service to advertisers and other competent agencies, but haven’t figured out a way to do so that wouldn’t get us sued.
By the same token, we occasionally see programs that are in quite good shape, and form positive impressions of those agencies as a result. We don’t see this often for the simple reason that advertisers whose programs are in good shape rarely shop around. We have favorable impressions of the agencies mentioned above in the paid search category, but not many others. There are sizable agencies whose programs we’ve never gotten to see. Likely, this means they’re very good, but we just don’t know for sure.
So, while we could put together a list of agencies to avoid at all costs, we’d be hard pressed to rank the best of our competitors simply because we don’t get to look under the hood of their programs very often. I suspect the SEO firms would say the same thing.
If we can’t rank each other, and the engines can’t rank the agencies very well certainly no third party can do the job adequately. The level of depth that needs to be studied and the degree of expertise needed to reach the right conclusion simply isn’t there.
It all comes back to reputation among clients, and the depth of knowledge of those clients. A reference who thinks their agency is great is one thing; references who knows their stuff and think their agency is great mean quite a bit more.
If you like this post, consider subscribing to our RSS feed. You can also have new posts sent to you via email.
Related Posts
- PPC Agencies: Ad Brokers or Client Advocates? Whose corner is your agency in: yours, or the Engines'?...
Comments
Your Comment
Trackback
http://www.rimmkaufman.com/rkgblog/2009/12/28/can-agencies-be-rated/trackback/Blogs Citing This Post
- Trackback: uberVU - social comments on December 28, 2009
Reminds me of the politics behind b-school rankings.
Do you have a link to Forrester’s rankings from previous years or a description of their criteria?
BTW – your post is #1 in Google for Forrester sem ranking … http://www.google.com/search?hl=en&source=hp&q=Forrester+sem+ranking&aq=f&oq=&aqi=
Great metaphor, Steinar, you’re right.
Forrester charges money for their ‘insights’. For a list of their “Wave” reports look here: http://www.forrester.com/rb/search/results.jsp?N=0+100010+133001&No=25
I’m told on good authority that there is no “payola” going on, that the fact that the agencies at the top of their list are often Forrester partners and sponsor Forrester research is coincidental. Forrester would lose all credibility if its rankings were so easily bought. I’m just not sure how much credibility they should have in the first place.
RKG doesn’t provide SEO services, but we do think our content helps our own cause somewhat :-)
George
Would you actually really want a client that did something solely based on Forrester rankings?
I’d be pleased to partner with you guys anytime, right now SEM is more in favor as people are used to the concept of migrating budgets. Once CFO’s understand search marketing’s economics more fully, that will change.
You’re a wise man, David. I should be careful what I wish for! We have had clients ask us about the rankings, but generally the context is “Why is Forrester so far off base?” As I wrote in this piece, Forrester might be faulted for pretending to have a clue, but no one should expect them to be able to generate ratings that actually mean anything.
In my opinion this is simply answered with this word you spoke: The level of depth that needs to be studied and the degree of expertise needed to reach the right conclusion simply isn’t there.
George,
This article is spot-on, and for me, timely. I am currently doing some consulting for a large company around a variety of initiatives, one of which is an RFP-ish process. The Forrester Wave report was included in some on-boarding documentation that I received and I am struggling with how to gently mention that I think it is bunk.
Your article and the comments of your readers have given me some ideas.
What is your opinion on the underlying reasons why Forrester doesn’t separate the two disciplines (Paid Search and SEO) into distinct ranking reports and then push for answers to more meaningful questions, like those you suggest in the article?
Thanks for being down-to-earth and insightful. Your articles are a pleasure to read.
Thanks for your comment, Ralf!
Byron, thanks for the kind words. I corresponded with Shar Van Boskirk on that topic, making the case that: 1) the disciplines and practices are completely different and require different types of expertise and 2) the best practitioners of each seem to be the specialty shops.
Her response was essentially that the two are so inter-related they have to be done together. I replied tartly that she’d been sold on a sales pitch from the all-in-one shops that just isn’t so. Haven’t heard back from her since :-)
George
Thanks for the response, George.
Shar is absolutely right. The two are inter-related. In fact, most effective online marketing plans leverage data back and forth between the two disciplines.
What doesn’t make sense is the assumption that, because functionally the two disciplines are linked, it is only possible to assess the expertise of a search marketing agency when they do both.
I’ve worked in plenty of environments where top-notch services were delivered by separate organizations. These organizations had in there toolkit not just the ability to get it done with paid search or SEO, but also the ability to play nice and partner effectively with the other agency. I consider that significant and a definite competitive differentiation.
Color me flummoxed. Forrester just doesn’t make sense on this one. Perhaps I’ll write a note to Shar as well.
Thanks again for your thoughts and the quick response.
Bryon
Byron, you’ve said it better than I did. Certainly, information sharing between the SEO and paid search teams is valuable, but it’s also trivially easy. There are no reasons different companies can’t share that information just as well as that information can be shared within a single organization.