Oct 182010

Be Careful with those "Assists"

Tracking "assists" is a sloppy and potentially dangerous mechanism for handling attribution.

The reason is that many "attribution systems" apply no rigor to defining an assist, and as we all know: assistance comes in all different shapes and sizes...and values.

"The Great One" (aka Wayne Gretzky) had more "points" in his career than any other player in NHL history...by a long, long way. 2,857 points, comprised of 894 goals and 1,963 assists. Interestingly, in hockey an assist is considered just as valuable as a goal for the purposes of individual statistics. In the judgment of an official, if a pass was instrumental in making the goal possible, the passer should get just as much credit as the guy who sticks it in the net.

Two facts are worth mentioning: first, while the player gets a point for his stat sheet, the team only gets one goal; second, there is judgment involved here, and not every pass preceding a goal is credited with an assist.

Rudimentary attribution systems are guilty of 4 grievous sins:

  1. Double-counting. Assists are often treated as some type of a win in addition to the actual sales driven. Let's say all online marketing channels combined to drive 2,000 orders yesterday (or last week or last month or in the last hour). Now attribution is turned on and shows that these ads also had 1,000 assists. Great, but there are still only 2,000 orders. If we assign a real value to an assist without simultaneously reducing the value of the other touches on the order we will be guilty of double-counting -- exactly what attribution management seeks to avoid;
  2. Self-assists. Particularly egregious attribution systems credit self-assists! We know that the paid search buying cycle argument is largely hooey, but what we do see in paid search fairly regularly is people clicking multiple times on the same ad prior to placing an order. The ad becomes a navigation aid. The person remembers how they found the site that had the cool product they wanted and repeats that behavior. Crediting an ad for both driving the order AND assisting the sale is ridiculous. Even Gretzky didn't get credited for assisting himself.
  3. Equivalency. All preceding touches are treated as equal regardless of order or timing. Even worse, ad impressions and actual visits to the site are treated as equally valuable, which is absurd; and
  4. Channel agnosticism. Different marketing channels behave...um...differently. A last touch on a competitive non-brand search term is fundamentally different than a last touch on an affiliate coupon site, and the statistics bear this out. Given the same preceding touches the paid search ad should get more credit than the affiliate ad on average because of the behaviors these touch-paths describe. This is also true of the distinction between brand search and non-brand search (paid or organic). The behaviors that lead to exposure to ads from different marketing channels mean that those channels cannot all be treated the same way.

All assists are not created equally.

For direct marketers, fractional allocation of credit is required to fully understand the value of different marketing channels. Controlled testing of what can be tested (display ads, direct mail, email) is also essential.

For brick and mortar chains the picture is less clear, but there is no reason not to measure carefully the effects that can be measured. We're working with folks on some very interesting tests to at least get a sense of the online impact of TV and circulars and the offline impact of online marketing.


5 Responses to "Be Careful with those "Assists""
Matt Liilig says:
The problem I have with weighted attribution is that there are limitless variables that can be factored into a weighted algorithm. For example, an impression should be weighted less than a click, a click that took place 25 days ago should be weighted less than a click that took place two days ago, a display click should be weighted less than a sponsored search click, an ad that was clicked on 5 times should be weighted less than an ad that was clicked on 100 times, etc., etc., etc. One clould literally come up with an endless amount of reasons why one ad should be weighted more or less than another. At least by offering an equal attribution method advertisers can stop shooting themselves in the foot and stop blowing their ad budgets. For example, a search advertisers might normally look at clicks and direct conversions for search search keywords. If they see that a particular keyword gets decent clicks but low direct conversions, they'll typically stop spending on that keyword (due to a low ROI). But as soon as you add the Assist factor in, it usually changes the mindset of the advertiser (even with equal attribution). While the keyword may not convert well on it's own, a high number of Assists will show that it's a good branding keyword for driving conversions to other keywords ("other keywords" being the key of course. In a basketball analogy, it's simple. In it's current state (last click conversion) if Player A scores/converts 30 points and Player B scores/converts 2 points, the coach benchs player B and keeps playing A. Adding in Assists, we see that while Player B only converted for 2 points, he also added 10 Assists (attributing to 20 team points). Player B just became munch more valuable than you originally though. Now the coach plays both player A (the converter) and Player B (the Assister). Steve Nash won two NBA MVP trophies not because he was a great scorer/converter but because he was a great assist man. He was valuable because he helped his team convert. Were his Assists weighted differently? Were his Assists early in the game weighted less than his Assists later in the game? No, they are weighted equally as 1 each. We're some of his Assists weighted less because more of his passes went to the center versus the forward? No, each was still worth 1. Just a different way to look at it. I do believe that weighting is important, but it doesn't make equal attribution worthless.
Thanks for your comments, Matt. Extra credit for using a basketball metaphor (I coached HS basketball in a former life)! I agree that something rudimentary can be better than nothing, but it can also lead to a false sense of security. If you count every pass made prior to a basket as an assist, you'll have a team full of all-star play-makers but possibly a losing record. You'll see that the guy who in-bounds the ball under the other team's basket after they score ends up with 30+ assists/game even though he may be a stiff. You'll credit a guy with an assist from a possession three possessions back if you have a couple of unsuccessful possessions. Yes, doing the statistics to determine smart weighting is incredibly complicated -- but getting the right answer is also valuable. We find that our modeling software produces fractional allocation of credit that makes sense from a marketer's perspective. The engines (sorry :-)) and shameless agencies who work as ad brokerages rather than serving their clients have an incentive to artificially inflate the value of all advertising. Looking out for our clients' interests demands that we take a more rigorous approach.
Matt Liilig says:
Good dialog George! I would actually argue that this engine (Yahoo...now Yahoo + Bing) are actually very interested in serving the best interest of the client. Yahoo! was the first engine to introduce the Assist attribution concept back in 2006 to help our advertisers understand the effect of display on search. We wanted to make sure that our advertisers were giving credit to where credit was due. We did studies way back when with ComScore that showed how a combination of search + display drove the highest ROI vs. using either channel alone. The problem was that the advertisers had no attribution tools to see this data for themselves. Hence the launch of the Assist metric which allowed advertisers to see that particular ads (search or display or email or whatever) played a contributing role in helping their other ads convert. Essentially, it allowed them to weed out the real poor performing ads and maintain or increase spend on the better preforming ads. And notice how we've always allowed you to track multiple channels (search, display, social, email, etc) while Google's new Search Funnels only provide the Assist metric for search (makes sence when 90% of your revenue comes from search but we all know display plays a big role in the path to conversion via impression or click). You stated above, "If you count every pass made prior to a basket as an assist, you’ll have a team full of all-star play-makers but possibly a losing record." Not necessarily true. A player with one assist and one conversion isn't as valuable as a player with 5 assists and 10 conversions. But every Assist should get credit because it led to a conversion. The Assist simply gives the advertiser more insight to help them make a smarter budgeting decision, that's all. The advertiser could just as easily find that their campaigns not only have low conversions but also a low number of Assists therefore helping them make an easy decision about pulling the budget for the campaign. That's good info to have becasue they can now reinvest that money somewhere else. The advertisers we're worried about most are the ones that lower or pull the budget for an ad thinking it was a poor performer when it reality it was a good ad for driving conversions to other ads (display to search for example). Advertisers miss value the branding effectiveness that their ads play. In the end. It's a win/win for all.
Good stuff, Matt, thanks! How well any variety of advertising works depends a great deal on the industry, the type of business, the business itself, and the type of success metrics used. Within retail shifting from last touch attribution to smart fractional attribution tends to move a big chunk of credit away from affiliates, CSEs, and email and towards paid search and to a lesser degree display. Control tests of display will prove that the incremental value of a display impression is MUCH MUCH less than the value of any other type of interaction. We advocate using those control tests to help tune the attribution modeling. Every publisher I've ever worked with -- from print magazines, to newspapers, to TV networks to search engine -- claims that they have the advertisers' best interests in mind. Smart advertisers don't believe everything they hear. This is not a dig against your great company, nor any other, simply that at the end of the day companies look out for their own interests first and foremost as they should.


Check out what others are saying...
[...] This post was mentioned on Twitter by Rimm-Kaufman Group, Metatags, search tool madman, Meta Keywords, Search Marketing and others. Search Marketing said: Be Careful with those “Assists”: Tracking “assists” is a sloppy and potentially dangerous mechanism for handling a... http://bit.ly/cqNorZ [...]

Leave A Comment