Sep 132013

Google Query Data Disappearing at an Unprecedented Rate, a Breakdown

RKG and plenty of others in our industry have chronicled the rise of (not provided) queries since they first began to show up in analytics packages in late 2011 when Google began conducting secure searches by default for logged in users.  Since that time, we have seen the occasional spike in the share of Google organic search traffic that did not pass a search query -- for instance when new versions of web browsers like Chrome and Firefox were released and began defaulting to secure search themselves.

The spike in not provided share that has taken place since late July of this year, however, is unprecedented.

In mid-July, on average roughly 43% of Google organic search visits did not pass the user's search query to the website receiving the visit.  By the end of September, that figure may very well double.  Currently, over 70% of search queries are not being provided:

Is this Related to iOS 6?

The timing of the initial not provided jump coincides with the return of some referrers for iOS 6 devices, but this appears to be only one piece of the puzzle.  As we've noted in previous posts, until the very end of July, no referrer information at all was being passed for around 75% of Google organic searches conducted on iOS 6 devices.  Without the referrer, those clicks were viewed as direct site visits by web analytics, rather than organic searches.  This had the effect of hiding about 14% of Google organic searches in Q2.

RKG's latest data suggests that referrers are now missing for only around 40% of iOS 6 Google searches.  That translates to about 6-7% more Google organic searches being accurately identified as such.  But, since iOS 6 defaults to secure search, the return of those visits should be adding to the not provided bucket.

Looking at a daily view of not provided share, we can clearly see when the iOS 6 clicks make their return:

Between July 28th and August 1st, not provided share increased 7 percentage points, from 43% to 50%.  Not provided share then remained fairly flat for the next two weeks.  For context, here is what the return of iOS 6 visits looked like over the same period:

Looking beyond the intra-week usage patterns, there was a one time spike in iOS 6 share of Google organic visits at the very end of July.  Since then, iOS 6 share has been stable, if not trailing off a bit as more users make the move to iOS 7, ahead of its official release date next week.

Google "Racing to Encrypt"

Three months after it helped to break the NSA surveillance story by publishing the documents Edward Snowden leaked on the PRISM program, the Washington Post asserted last week that:

Google is racing to encrypt the torrents of information that flow among its data centers around the world in a bid to thwart snooping by the NSA and the intelligence agencies of foreign governments...

The story, written by Craig Timberg, is somewhat vague on the timing of the changes.  It notes that the "encryption initiative" was approved last year and, but accelerated in June.  The article later adds that, "the project is expected to be completed soon, months ahead of the original schedule."  Although imprecise, that picture fits pretty well with what we are seeing with not provided searches.

I should note that Timberg doesn't offer a clear mechanism by which more search queries would become not provided to websites under Google's new initiative, but rather he focuses on Google increasing encryption on the communications between its own servers.  Still, the data suggest that in tandem with its internal overhaul, Google is pushing more users to secure search, regardless of their browser version or logged in status.

Not Provided by Browser Version

Looking at Google organic search data by browser version, we see that the percentage of not provided queries has risen much faster from early July to early September for some browsers:

Browsers Listed in Order of Current Traffic Share

Traffic from Internet Explorer is spread out over several versions, but all of them show not provided share more than doubling in two months.  MSIE 8.0 is currently the fourth largest Google organic traffic generator and it has seen not provided share nearly quadruple from 17% in July to 65% in September.

If we exclude Internet Explorer, we find that not provided share rose 23 percentage points from July to September.  With Internet Explorer, not provided rose 28 percentage points.  Between IE closing the not provided gap with other popular browsers, and iOS 6 referrers showing up more often, we can account for about half of the total increase in not provided share from July to September.

That still leaves not provided rising a good deal faster than the long-term trend.  The data from Chrome provides an interesting example of this.  Beginning with version 25, officially released this February, Chrome has defaulted to SSL search for searches conducted via the "omnibox" or address bar.  At the time, not provided share rose on the order of 5% or so overall.

By early July, Chrome 27 was the most popular version and 75% of its Google queries were not provided.  Chrome 29 is now the biggest traffic driver and its not provided share is 91%.  Among the traffic still coming through on Chrome 27 though, 92% of queries are now not provided.

In short, the surge in not provided is coming from the same browsers, or at least browsers with similar search defaults, that were generating the bulk of traffic a couple months ago.

Less Query Data, Less Representative Data

The flip side to all of this is the question: Where are the queries that are still being passed via referrer coming from?  It's anecdotal, but I can't seem to conduct a Google search that does pass the query regardless of my browser, device, or whether I am logged in.

Going back to the chart above, it is interesting to see that there are a number of browsers where not provided share is still close to zero.  Among the most popular browsers, these cases are almost all older Safari or iOS versions.

This raises the obvious question of how representative the remaining search query data is.  Taking a look at the share of Google organic traffic overall and the share of Google searches that do provide the organic search query, there is a clear contrast:

Safari 5.1 is generating just 3% of Google organic visits overall, but it represents 12% of visits that pass the search query.  Chrome 29 accounts for 28% of total visits, but just 8% of those that include the query.

We know that user behavior can differ significantly by device or even browser version, so trying to extrapolate the remaining referrer-based search query data to the larger picture could very easily be leading to a skewed perspective on Google organic performance.

The End of Days for Referrer-Based Organic Query Data, Now What?

Most of those following the rise of not provided have likely expected query data from referrers to go away eventually.  What is surprising is how swiftly that appears to be happening.  All is not lost, however.

From the early days of not provided, RKG and others have provided guidance on accounting for the impact of hidden queries.  The best resource for query data is Google's own Webmaster Tools, which recently announced they will be expanding their data set from 90 days to a full year.

GWT data can be integrated into Google Analytics and third party tools can help provide insights into query trends, making the data more actionable.

A final thought: some have pointed to the contradiction of Google continuing to pass queries for paid search clicks while increasingly hiding the same information from organic search as an indication their motive is to force marketers to pay to play.  While paid search data can be valuable to SEO efforts, I've always had a hard time believing that the increase in not provided queries would really lead to very different strategies on the paid side.  I think we can give Google the benefit of the doubt here.


13 Responses to "Google Query Data Disappearing at an Unprecedented Rate, a Breakdown"
Adam Dince says:
Great post! Thanks Mark. I'm still dumbfounded about why the encryption can only apply to organic and not to paid. My only hypothesis is that Google wants brands to spend more on broad match terms to get a better understanding of the types of exact matches that are driving traffic to their sites. Oh well. Smart SEOs find other alternatives to Google's referral data. Cheers, Adam
Brad Spencer says:
Despite what the Washington Post said, I don't think many credible sources would believe that the move to encrypt search terms has anything to do with the NSA. It is being sold as a privacy initiative, but ultimately search terms are a valuable resource that Google has been giving away for years and is now pulling back on. They may eventually sell the data or bundle it somehow to further their paid search departments.
Thanks for weighing in Adam and Brad. The NSA connection may be a red herring, but like I noted in the post, I'm inclined to give Google the benefit of the doubt for now, but we'll see what develops over the next few months. What they choose to do, or not to do, with the query data they have will reveal a lot about their motivations.
Adam Dince says:
Hey Mark, While I'd like to be as optimistic as you are--I'm not. If Google was really encrypting query data for security purposes--AdWords keyword data would also be encrypted. Let me provide a simple analogy. If I'm a football player and want to protect myself from injuries, would it make sense for me to put on my pads and leave off my helmet? Vice-versa? No--of course not. In the same way it makes no sense that Google would encrypt organic queries and not paid. Google has been encrypting keywords for almost 2 years now. If it was really about privacy, they would have found a way to ensure PPC keywords were protected too. Google's flawed reasoning might sound great to the general public that doesn't understand how it monetizes its search engine, but those of us who have been doing this for years can read clearly between the lines. Cheers, Adam
Good points, Adam. If Google continues to pass queries for the paid side, it would suggest that user privacy isn't their paramount concern. Again, I think we'll have to wait to see what moves they make in the coming months. I wouldn't be surprised to see paid queries disappear from referrers, and it wouldn't be as big of a loss to advertisers as losing organic query data is to SEOs. If they were to stop passing queries for paid search, I think it's likely advertisers would still have the ability to pull detailed query reports, but there would still be increased privacy for users. It would be nice if they did similar for organic.
Nick Potter says:
I've also noticed some referring sites not appearing in Google analytics on a large scale over the past few months. I only understood what was happening when i checked with the server side stats. Has anyone else seen this?
Google is in the process of reinventing itself as is evidenced not only by withholding data but also by the barrage of new "tools" which they have flooded the market with in the past few months. This means that the industry at large must also reinvent itself. The message I seem to be getting is that highly targeted keyword schemes will fail to produce results in the very near future. Those SEO companies that stay on top of their game will produce results for their clients. Those that don't will disappear. It a process of forced "natural selection". In my opinion, it is long overdo. As far as query data from some browsers still being passed is concerned, old browser will fall away so there is no need for them to focus vast amounts of energy integrating them into the new paradigm. What concerns me is the amount of additional power this puts in the hands of a single organization. On another note, this represents opportunity for the brightest of the bright.


Check out what others are saying...
[...] Google Query Data Disappearing at an Unprecedented Rate, a Breakdown, RKG Blog [...]
[...] RKGBlog Artículos relacionados:Groupon un sistema de compras que obtiene éxito mundial.Patente de Google [...]
[...] Mark Ballard posts “Google Query Data Disappearing at an Unprecedented Rate, a Breakdown” at RKG Blog. [...]
[...] Google Query Data Disappearing at an Unprecedented Rate, a Breakdown [...]
[...] that keyword-specific search data is now almost entirely gone. Google has been slowly shifting to a new model that keeps the details of what site visitors search [...]

Leave A Comment