Google Query Data Disappearing at an Unprecedented Rate, a Breakdown
RKG and plenty of others in our industry have chronicled the rise of (not provided) queries since they first began to show up in analytics packages in late 2011 when Google began conducting secure searches by default for logged in users. Since that time, we have seen the occasional spike in the share of Google organic search traffic that did not pass a search query — for instance when new versions of web browsers like Chrome and Firefox were released and began defaulting to secure search themselves.
The spike in not provided share that has taken place since late July of this year, however, is unprecedented.
In mid-July, on average roughly 43% of Google organic search visits did not pass the user’s search query to the website receiving the visit. By the end of September, that figure may very well double. Currently, over 70% of search queries are not being provided:
Is this Related to iOS 6?
The timing of the initial not provided jump coincides with the return of some referrers for iOS 6 devices, but this appears to be only one piece of the puzzle. As we’ve noted in previous posts, until the very end of July, no referrer information at all was being passed for around 75% of Google organic searches conducted on iOS 6 devices. Without the referrer, those clicks were viewed as direct site visits by web analytics, rather than organic searches. This had the effect of hiding about 14% of Google organic searches in Q2.
RKG’s latest data suggests that referrers are now missing for only around 40% of iOS 6 Google searches. That translates to about 6-7% more Google organic searches being accurately identified as such. But, since iOS 6 defaults to secure search, the return of those visits should be adding to the not provided bucket.
Looking at a daily view of not provided share, we can clearly see when the iOS 6 clicks make their return:
Between July 28th and August 1st, not provided share increased 7 percentage points, from 43% to 50%. Not provided share then remained fairly flat for the next two weeks. For context, here is what the return of iOS 6 visits looked like over the same period:
Looking beyond the intra-week usage patterns, there was a one time spike in iOS 6 share of Google organic visits at the very end of July. Since then, iOS 6 share has been stable, if not trailing off a bit as more users make the move to iOS 7, ahead of its official release date next week.
Google “Racing to Encrypt”
Three months after it helped to break the NSA surveillance story by publishing the documents Edward Snowden leaked on the PRISM program, the Washington Post asserted last week that:
Google is racing to encrypt the torrents of information that flow among its data centers around the world in a bid to thwart snooping by the NSA and the intelligence agencies of foreign governments…
The story, written by Craig Timberg, is somewhat vague on the timing of the changes. It notes that the “encryption initiative” was approved last year and, but accelerated in June. The article later adds that, “the project is expected to be completed soon, months ahead of the original schedule.” Although imprecise, that picture fits pretty well with what we are seeing with not provided searches.
I should note that Timberg doesn’t offer a clear mechanism by which more search queries would become not provided to websites under Google’s new initiative, but rather he focuses on Google increasing encryption on the communications between its own servers. Still, the data suggest that in tandem with its internal overhaul, Google is pushing more users to secure search, regardless of their browser version or logged in status.
Not Provided by Browser Version
Looking at Google organic search data by browser version, we see that the percentage of not provided queries has risen much faster from early July to early September for some browsers:
Traffic from Internet Explorer is spread out over several versions, but all of them show not provided share more than doubling in two months. MSIE 8.0 is currently the fourth largest Google organic traffic generator and it has seen not provided share nearly quadruple from 17% in July to 65% in September.
If we exclude Internet Explorer, we find that not provided share rose 23 percentage points from July to September. With Internet Explorer, not provided rose 28 percentage points. Between IE closing the not provided gap with other popular browsers, and iOS 6 referrers showing up more often, we can account for about half of the total increase in not provided share from July to September.
That still leaves not provided rising a good deal faster than the long-term trend. The data from Chrome provides an interesting example of this. Beginning with version 25, officially released this February, Chrome has defaulted to SSL search for searches conducted via the “omnibox” or address bar. At the time, not provided share rose on the order of 5% or so overall.
By early July, Chrome 27 was the most popular version and 75% of its Google queries were not provided. Chrome 29 is now the biggest traffic driver and its not provided share is 91%. Among the traffic still coming through on Chrome 27 though, 92% of queries are now not provided.
In short, the surge in not provided is coming from the same browsers, or at least browsers with similar search defaults, that were generating the bulk of traffic a couple months ago.
Less Query Data, Less Representative Data
The flip side to all of this is the question: Where are the queries that are still being passed via referrer coming from? It’s anecdotal, but I can’t seem to conduct a Google search that does pass the query regardless of my browser, device, or whether I am logged in.
Going back to the chart above, it is interesting to see that there are a number of browsers where not provided share is still close to zero. Among the most popular browsers, these cases are almost all older Safari or iOS versions.
This raises the obvious question of how representative the remaining search query data is. Taking a look at the share of Google organic traffic overall and the share of Google searches that do provide the organic search query, there is a clear contrast:
Safari 5.1 is generating just 3% of Google organic visits overall, but it represents 12% of visits that pass the search query. Chrome 29 accounts for 28% of total visits, but just 8% of those that include the query.
We know that user behavior can differ significantly by device or even browser version, so trying to extrapolate the remaining referrer-based search query data to the larger picture could very easily be leading to a skewed perspective on Google organic performance.
The End of Days for Referrer-Based Organic Query Data, Now What?
Most of those following the rise of not provided have likely expected query data from referrers to go away eventually. What is surprising is how swiftly that appears to be happening. All is not lost, however.
From the early days of not provided, RKG and others have provided guidance on accounting for the impact of hidden queries. The best resource for query data is Google’s own Webmaster Tools, which recently announced they will be expanding their data set from 90 days to a full year.
GWT data can be integrated into Google Analytics and third party tools can help provide insights into query trends, making the data more actionable.
A final thought: some have pointed to the contradiction of Google continuing to pass queries for paid search clicks while increasingly hiding the same information from organic search as an indication their motive is to force marketers to pay to play. While paid search data can be valuable to SEO efforts, I’ve always had a hard time believing that the increase in not provided queries would really lead to very different strategies on the paid side. I think we can give Google the benefit of the doubt here.