Sleuth Your Online Competition Using Free Web Tools
Your phone buzzes just after lunch.
Your boss is shouting. “Some new website appeared today out of nowhere and they’re advertising heavily against us! Who are they? Find out everything you can on them and report back by day’s end!”
Today’s web provides easy tools for competitive research. This column provides a roadmap for sleuthing a competitor in a few of hours, at no cost, using just a web browser.
First, ready your browser. If you aren’t already using it, install Firefox (www.download-firefox.org), because you’ll need plug-ins which aren’t available in IE. Next, get these plugins: Google Notepad (http://www.google.com/tools/firefox/), SEO For Firefox (http://tools.seobook.com/firefox/seo-for-firefox.html) and Quirk’s Search Status (http://www.quirk.biz/searchstatus/).
For help installing plug-ins, check out Mozilla’s FAQ at http://www.mozilla.org/support/firefox/faq.
Turn on Google Notebook by clicking its icon on the bottom right of the browser. This handy plug-in lets you rapidly clip and annotate the URLs you visiting. Notebook is a convenient way to document your findings.
Begin by visiting our competitor’s site. Surf around, clipping interesting URLs. Pay particular attention to their press releases, which often provide valuable information on financials and financing. Scope out their jobs page to determine how actively they are hiring, at what level, and with what skills. A site’s IT job openings typically provide a clear roadmap of a firm’s technology choices. Visit their executive bios page, and record the names of their management team.
Read their company blog. Recent posts are most important, but also study their earliest blogging efforts, as these can be revealing. See if any of their execs have personal blogs.
View the HTML source of key pages (Control-U in Firefox), as sometimes webmasters leave redacted text in the source, just commented out. I once stumbled upon key pricing information from a competitor this way.
In the source, check out their META “keyword” tags to see what search terms they deem important.
Browse their site’s robots.txt file to see what content they’d prefer to keep off of the engines. For example, check out the current administration’s lengthy exclude file at http://www.whitehouse.gov/robots.txt.
The WayBack Machine (http://www.archive.org/web/web.php) lets you visit historic versions of a site. Using WayBack, watch how your competitor’s branding, management team, and mission statement have evolved over time. (Hint: to remove your own site from WayBack, disallow “ia_archiver” in your own robots.txt).
Next, head over to the social networks. Search for your competitor’s brand name on Facebook (http://www.facebook.com/), LinkedIn (http://www.linkedin.com/), Flickr (http://www.flickr.com), YouTube (http://www.youtube.com/), Delicious (http://del.icio.us), and Google Groups (http://groups.google.com/). And search each of these sites for the names of your competitor’s management team. The social networks often turn up interesting history, gossip, relationships, pictures, and videos. You may also discover relevant usernames – for example, a user posting pictures from your competitor’s social events — which can then be cross-searched on the other sites. Remember to log your findings with Google Notebook.
The news outlets provide great information. If your competitor is publicly traded, study their financials at Yahoo Finance (http://finance.yahoo.com/). Read their annual reports, and don’t overlook the comments to their financials. Next, search for your competitor using the free area of Hoovers (http://www.hoovers.com); sometimes Hoovers reveals associated entities, even for privately held firms.
Search for your competitor and their key execs across the news wires using Google News (http://www.google.com/news) and across the blogosphere using Technorati (http://www.technorati.com).
If your online competitor mails a catalog, determine their list manager and pull their data card from the manager’s website. From their data card, you get a rough sense of a firm’s size, growth, and customer demographics. It is now time for some technical digging.
Use WhoIs (http://whois.domaintools.com) to determine the technical and administrative contacts for your competitor’s domain name. It is often revealing to do a general Google search on WhoIs contact addresses. You can also learn when the domain name was registered, and when it is due to expire. DomainTools shows what web server your competitor is running, which provides a small clue into their tech strategy. And DomainTools provides your competitor’s IP address. Note this number as we’ll use it subsequent steps.
With the SEO For FireFox enabled, run Google searches for your competitor’s brand and for your competitor’s URL. On the search results page, the SEO plug-in provides counts and links to much SEO-relevant info, including your competitor’s back links.
The Quirks Plugin (the small “Q” on the bottom right of the browser) also provides interesting information your competitor’s indexation and inbound links.
Run a Google search on your competitor’s IP address as well, as sometimes this dredges up interesting entries in forums and server logs.
Type your competitor’s IP into the handy tool at SEO Logs (http://www.seologs.com/ip-domains.html). This tool reveals other domains hosted on the same box, which can reveal affiliated companies or projects not generally known to be linked to your competitor. If your competitor is using shared hosting, other domains on the same server could be entirely unrelated. However, the very fact a firm entrusts their web site to a cheap shared hosting provider provides clear indication they are very small.
Moving from white-hat research towards gray, you can scan your competitor’s IP address to see what ports are open.
If you don’t have access to nmap (http://insecure.org/nmap/), there’s a limited web-based port scanning tool at T1 Shopper (http://www.t1shopper.com/tools/port-scanner/). It is unlikely anyone is running open or anonymous FTP these days, but you can certainly check if so inclined. Of course, only access data which a site provides openly to everyone.
You’ve now amassed a great deal of information. Write a short summary of what you learned, along with supporting URLs, and put that into your Google Notebook. You can then choose to whom you give access to your findings.
With a few hours of web digging, you can obtain deep insight into a competitor’s branding, business strategy, reputation, history, financials, and key employees.
All these data are public. It is your aggregation and analysis which weave these scattered facts into a cohesive business story. The result can be comprehensive, often scarily so.
Consider running the same process on your own firm. The results can be sobering.
The web annihilates privacy. May this realization encourage us all to conduct ourselves with the utmost integrity. And good luck in your sleuthing!
