As part of the Lane’s Gifts settlement, the Circuit Court of Miller County, Arkansas ordered an independent review of Google’s click fraud detection system to determine if they were reasonable. The review was conducted by Professor Alexander Tuzhilin of NYU. Tuzhilin’s 47 page report, The Lane’s Gifts v. Google Report, came out last month.

The punchline: Tuzhilin, after being given “insider” access to Google people, practices, and report, found Google’s efforts to detect and combat clickfraud to be “reasonable.”

Beyond this punchline, there are many other interesting tidbits in the report:

• Even if the efforts are “reasonable”, Tuzhilin indicates there’s no direct scientific evidence that those efforts work well:
  

  It is hard for me to arrive at any definitive conclusions beyond any reasonable doubt based on Points 1 through 6 above [ 1 to 6 refer to six pieces of indirect evidence]; that Google’s invalid click detection methods “work well” and remove “most” of the invalid clicks — the provided evidence is simply not hard enough for me, and I am used to dealing with much more conclusive evidence in my scientific work.

• Tuzhilin believes Google waited too long (until winter 2005), to fix the “double click” issue, and as a result enjoyed significant economic benefit from this delay at the cost of advertisers. (p. 30)
• (p.16) Tuzhilin makes the excellent point that clickfraud itself isn’t clearly defined. While outright automated fraud robots are clearly invalid, many more human clicks fall into a gray zone. What if a human clicks with no intent to purchase, is that valid? Tuzhilin writes
  

  …In some cases [ beyond clear fraud ] the true intent of a click can be identified only after examining deep psychological processes, subtle nuances of human behavior, and other considerations in the mind of the clicking person … It is simply impossible to identify true clicking intent for certain types of clicking behavior, and therefore, [ impossible to ] classify these clicks as valid or invalid. (p. 16)

• Because of of this,
  

  Given a particular click in a log file, it is impossible to say with certainty if this click is valid or not in all the cases. (p.18)

• Most of the Google fraud filters are surprisingly simple, and haven’t evolved much over time. Newer filters have done little to improve the system. (p. 28, p. 32)
• (p.14) Tuzhilin points out none of the three types of parties with data relevant to click-fraud — Google, advertisers, and vendors — have sufficiently comprehensive data for detecting fraudulent clicks. True. Tuzhilin does not discuss that even if Google had full conversion data from all advertisers, they’d still have an incomplete dataset for clickfraud analysis, as one would assume any sophisticated fraudster would conduct fraud across the all the engines. Google can’t see what is happening on Yahoo, MSN, etc.
• The phrase “massive clickfraud” appears several times in the report, usually right next to a sentence indicating that if Google disclosed the detail of its fraud detection methods, unethical users would exploit that knowledge (p. 21 and elsewhere) I wonder if this choice of words, “massive clickfraud”, originated with Tuzhilin or with Google. The phrase brings to mind a dam about to burst. The thought that a speck of knowledge could unleash massive fraud is disturbing … online security experts state that security through obscurity doesn’t work.
• Tuzhilin makes a “long-tail” argument that most fraud today is probably simple fraud, so simple filters can catch most of it (p. 29). Speaking with no data, my hunch is that I disagree. I’d suspect the most advanced fraudsters — those doing the most damage to the PPC system — use very sophisticated to distribute the bogus clicks broadly across IPs, time, and phrases.
• Tuzhilin does point out that Adsense is more prone to fraud than core Adwords, a topic we’ve written and spoken about multiple times. Hurrah. Many of our clients avoid Adsense altogether, finding the performance simply isn’t there, be it due to targeting, fraud, or a combination of other factors.

After reading this report, I have a better appreciation of Google’s fraud detection methods.

I come away feeling that fraud on the non-Adsense network is likely small, and agree with Tuzhilin that Google is making “reasonable efforts” to fight PPC click fraud.

And I agree with Eric Schmidt’s (often misquoted) comment about the self-correcting nature of clickfraud, the truest test of click quality rests in the advertiser’s conversion data.

If the clicks sell enought to justify their cost, the channel still works.

Very interesting report — worth downloading and reading.