It's been a year since Google announced secure search, and while we've covered a few creative methods to somewhat account for (not provided) data, a sound solution still has not surfaced in the search marketing community. For more on the background and details please read Danny Sullivan's Google Put A Price On Privacy, which is a strong article on this topic.
Now that Google secure search is established, we've seen hidden queries (represented by (not provided) and other labels in various analytics platforms) continue to rise. Currently the segment makes up about 25% of all organic search traffic for RKG clients. There's no sign of the increase slowing, either. Major factors propelling the rise in (not provided) include the following:
- The increased adoption of Google Plus, now above 400 million registered users. Growth of the social channel represents one of Google's foremost goals internally. Since new Google Plus accounts mean more signed-in users performing searches, this is probably single-handedly the largest factor in the rise of the (not provided) segment.
- Firefox defaulting to secure search in the latest version of the browser.
It's a shame Google doesn't pass referral data between secure SSL servers, per guidelines for the classic SSL to SSL standard. That change would enable sites to optionally deploy a secure version of their sites by default, keeping data private while retaining valuable query data. Since Google isn't supporting this, we're left with very few solutions to source the lost referral data.
Methods to Recapture Lost (not provided) Data
The first and best option is to make use of Google Web Tools search query data. The good news here is that none of the referral data in GWT shows up as (not provided). The bad news, however, is that this is only a sample of your entire query landscape. It is not definitive nor comprehensive. A second downside is that this data rotates every month, and the API does not make it available, meaning you'll have to either manually export it each month, or get more creative if you desire trending.
Because of these limitations, resourceful marketers need to look for other techniques.
One of the most obvious methods to replace lost (not provided) queries is by leveraging scraped data. Please note that RKG does not condone the practice of scraping Google's SERPs. The practice is in violation of Google's Terms of Service. This information is provided only as a guide to those who use third-party scraped data, or are willing to violate Google's TOS by scraping.
The technique for reclaiming a portion of the lost data is described below. For the purposes of this description we're using the test site ShoeDigest.com, and therefore don't have a problem sharing data transparently:
- In your analytics platform, segment organic search traffic and add the URL or landing page dimension. See all that (not provided) in there? At least you have the URLs.
- Go to your competitive insight tool of choice, for example SEMRush, and generate an organic search report for your domain. Export that into Excel (or CSV). Next, sort the data by landing page.
- Now, pair the SEMRush landing page URLs with the (not provided) URLs from your analytics. What you'll end up with is some of the recovered data, sorted by landing page.
Note: you won't be able to get clicks, only the query data. To take this process a step further and estimate clicks, follow Ben Goodsell's advice outlined in his SearchEngineWatch column.
Limitations Using Third-party Competitive Tools
There are several limitations with this approach.
- First, any scraped data is necessarily not comprehensive. It can really only be used directionally, no matter how "complete" the data is claimed to be. That's because a scraper is reliant upon the source it's attempting to scrape, therefore it can never be definitive.
- Secondly, the data will only be as fresh as the last time it was captured. Using any third-party data means being dependent upon their data collection schedule and timelines.
- The final limitation, and probably the most glaring, is that anyone has access to the same data you do. If the only requirement to access scraped data is to sign up for an account, there is nothing stopping your competitors from leveraging the exact same data set you have access to.
Even with those limitations, if you're serious about reclaiming some lost query data, this is currently a viable option.
Secure Search and the Future of SEO
There is no doubt that loss of referral data makes SEO harder. However, there's a counter to that. While Google is harder to game using pre-Panda tactics like heavy internal and external anchor text, and thin, overlapping keyword-themed pages, the level of entry and competition is now higher. As I've written before, the combination of Panda, Penguin, and secure search have created a flight to quality in SEO that has never been seen before. The best online marketers don't see limitations in this new era: they see opportunities.
The bar has risen and it's up to the smartest and most diligent of search marketers to understand this and execute on strategies and tactics that not only work, but are sustainable and (as much as possible) 'future proof' to algorithms we haven't yet seen. The increase of (not provided) creates an economy where the best services and technologies will only increase in demand.
Google's secure search also creates an opportunity for Bing, and even Blekko or DuckDuckGo to differentiate by offering all the data SEOs want. And for SEOs, never has it been more important to leverage referral data from Bing.
I look forward to your thoughts and comments! What are you doing to deal with this problem?
Update: I failed to mention iOS 6 changing to Google secure search. This is big and adds to the challenges ahead!