We've been designing a robust, advanced, and powerful SEO crawling tool over the last several months. SEO Scalpel is designed to focus in on very defined, specific criteria on a site and deliver it in CSV files for deeper analysis.
A partial list of SEO Scalpel's functionality is as follows:
- Efficient and fast crawling
- Scales up to millions of URLs per session
- User agent configuration
- Option to store a local copy of fetched URLs
Literally anything located in source code can be fetched individually, including:
- Any meta data field
- Any HTML tag, e.g. h(x), em, strong, etc
- Any href attribute, e.g. nofollow
Additionally, SEO Scalpel can fetch "thin" pages on a domain by analyzing content length (as defined as written word text, rather than code, comments, etc.) and comparing that to overall page weight.
And much more....