THE RKGBLOG

Robots-Nocontent

Yahoo has introduced a new tag, “Robots-Nocontent”, which allows web masters to keep the Yahoo spider from reading certain portions of a web page.

The robots.txt protocol has long allowed webmasters to keep spiders off entire pages.

This new tag gives webmasters intra-page exclusion control.

Webmasters can now mark parts of a page with a ‘robots-nocontent’ tag which will indicate to our crawler what parts of a page are unrelated to the main content and are only useful for visitors. We won’t use the terms contained in these special tagged sections as information for finding the page or for the abstract in the search results.

The most obvious use of this tag is wrapping navigational and menu elements. On many websites, identical navigation category links appear on every page. These links help users get around the site, but could confuse a spider about a page’s meaning.

The code is just a CSS class applied to a DIV or P or SPAN tag:



Because this paragraph is styled with the robots-nocontent class, the Yahoo spider should ignore it.


I can’t see any downside to this useful convention; hopefully Google will soon follow suit.

Technorati Tags: , ,

  • Alan Rimm-Kaufman
    Alan Rimm-Kaufman founded the Rimm-Kaufman Group...
  • Comments
    One Response to “Robots-Nocontent”
    1. Bookworm SEO says:

      I really enjoyed this post Alan. The new tag will be a useful tool and your post rewards those SEOs with the dedication to read up on new technologies on an ongoing basis. I’ve browsed around your blog and found plenty of other quality posts, so I thought I’d let you know that I like your blog. (On a related note, would you care to trade blogroll links? MontrealSEO.ca)

      Regarding one of those other quality posts – #3 on pricing – while my own focus is on the SEO side of things, I’ve also had people approach me with a rev-share offer. Interesting to see the downside to that from a client point of view. The flip side is that for new brands and companies, that low-hanging fruit issue is not going to be as relevant.