THE RKGBLOG

The Best Way to Handle 404 Pages for SEO and Users

GitHub 404

There are a few common misconceptions related to 404 pages and SEO that I’ll attempt to clarify.

While it’s common (and technically correct) to call 404 pages “errors,” I am not a fan of that terminology. 404s are the expected result when a website is unable to return a given request, if there is no URL to deliver. If we wanted to be picky about it, we’d say 404 pages are not errors and can even be (blasphemy!) an acceptable user experience.

Calm Down. 404s Are Ok.

404s are fine. There’s no need to panic. There’s no need to “redirect every 404 to the home page” (eek!), the category parent, or your shopping cart.

All too often I hear SEOs recommending that every 404′d URL be redirected somewhere – anywhere – just so long as “the juice is captured.” Nonsense.

I think people tend to freak out about 404s because they sense that Google will negatively score a site if they exist. Or, they worry that link equity is lost when quality links are pointing to a 404 page (a valid concern, which I’ll elaborate on below).

Generally speaking (though there are exceptions), 404 pages are a very normal part of running a website. There’s no reason to fret over them and there’s certainly no proof (at least that we’ve seen) of Google or Bing penalizing a site for 404 pages.

When 404s Can Be “Bad”

There are cases, however, where a site could be scored negatively as a result of an abnormally high occurrence of 404 pages. Does your site have 10,000 unique pages but 45,000 404s this month? That could be an indicator of a bigger problem with crawling and technical SEO. That’s also probably not a very good user experience, which could be reflected in poor user engagement and falling traffic, which could be reflected again by a search engine lowering visibility of your URLs in search results.

A high occurrence of 404s – when they are spiking and continuous – is not a good thing. But forget about SEO for a moment. Is that a good thing for users? Obviously not, and remember: search engines follow users.

Another potentially negative consequence of 404 pages is when a URL has valuable links. I’m not so rabid about “link equity” that I’m going to recommend every URL with links needs to be redirected. If there’s not a good page match for a 404, don’t redirect it. That said, if the links are precious and difficult to secure, think about contacting the site and having them update their link, or creating a new piece of content that’s relevant for the existing link, or finding a relevant page that will still ensure a good user experience and 301 the page.

Last but not least, 404s can be bad news when they act like a 200. A soft 404 error page is not a good thing, because search engines will continue to index these. Ensure your server is configured to return the proper status code for each type of page.

There’s an interesting technical SEO problem with regards to inventory and expiring products. But I’ll save that for another day.

Best Practices for Redirecting 404 Pages

Here is a quick list of best practices for redirecting 404 pages:

  • 404s should not always be redirected.
  • 404s should not be redirected globally to the home page.
  • 404s should only be redirected to a category or parent page if that’s the most relevant user experience available.
  • It’s okay to serve a 404 when the page doesn’t exist anymore (crazy, I know).
  • If you have valuable links pointing to 404 pages, do one of the tactics I outlined in the above section.
  • Don’t panic. 404s are normal.

Helpful Links

Bing’s 404 page best practices
http://www.bing.com/webmaster/help/404-pages-best-practices-1c9f53b3

Google’s 404 page best practices
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93641

Cool 404 examples
http://econsultancy.com/us/blog/9525-16-creative-404-pages-to-inspire-you-to-overhaul-yours

Another good resource on 404 pages and SEO
http://www.fallingupmedia.com/blog/404-page-examples-seo/

  • Adam Audette
    Adam Audette is the Chief Knowledge Officer of RKG.
  • Comments
    27 Responses to “The Best Way to Handle 404 Pages for SEO and Users”
    1. Rob Woods says:

      Looking forward to the next installment on the “interesting technical SEO problem with regards to inventory and expiring products”. I’ve been dealing with those issues for years and really looking forward to hearing your take on them. I hope we see that one soon :)

    2. Michael says:

      Hey Adam,

      I love funny 404 pages and will definitely check out your links. It drives me nuts when I get a 404 redirect to the homepage. I’d rather experience the 404 then decide where I want to go if I’ve entered the web address in error.

    3. Adam Audette Adam Audette says:

      Rob, great to have you here! Yeah that topic seems to be right up your alley. Now I’m curious to get your take on it, too. Will put together the post soon and we can discuss.

      Michael – great point. Me too!

    4. Dan Shure says:

      Adam

      HUGE +1 to the “expiring inventory” issue… would LOVE to hear more about that!

      Thanks for the post – I know I’ll be sharing this with clients who might freak out a little about 404′s :)

      -Dan

    5. Adam Audette Adam Audette says:

      Dan – cool. I’ll get on it! (Glad you like the post.)

    6. AJ Kohn says:

      We need more commonsense posts like this one Adam. I despise global 404 redirects. My test is always domain.com/foobar and when it does a redirect to home I vomit a little bit in my mouth.

      Frankly, given the amount of link rot wouldn’t a site without any 404s be more of a red flag? If I were a search engineer I know I’d be on the look out for that (and I’m not nearly as smart as they are.)

      The expired inventory issue is a complicated but fun one. Lots of ways to do it really but eager to hear your thoughts on it.

    7. What I “worry about” are the soft 404s and the “not found” in Webmasters Tools. I figure they are telling me because they care. So I try to fix and avoid those type of issues. I don’t think having a 100% updated site is bad but I do think just random 404 redirecting to the homepage is a pretty bad idea.

    8. Adam Audette Adam Audette says:

      Thanks for weighing in, AJ. Yep, agreed in regards to link rot and signals like abnormally low (or no) 404s. Seems like an easy flag. Then again, I’m not as smart as Amit Singhal and gang either!

      Matt – I addressed soft 404s in the article. They are a valid concern. GWT isn’t something you should take as gospel. We’ve seen errors reported in there from Googlebot’s own malformed crawls, quite frequently. It’s nice to use for a quick check, but if you really want to know what Googlebot (or any spider) is doing, grab the raw log files and mine ‘em with Sawmill, Splunk, or some nice sed and awk scripts.

    9. Grace Eline says:

      Hey Adam,

      I got many 404 errors from Google Webmaster.I have no option to resolve this problem.But after read your post,I got the solution.I’m seriously thankful to you.

    10. Gridlock says:

      “It’s okay to serve a 404 when the page doesn’t exist anymore”

      If you’re going to prioritise web standards, user experience and SEM in that order then shouldn’t you be saying “410″ (‘Gone’) here?

    11. Adam Audette Adam Audette says:

      @Gridlock – 404s and 410s are basically treated the same way. The difference is really subtle and probably too nuanced to worry about. I like 410s – gone – because (at least technically) they’re supposed to mean: this page is gone and will never come back, and it’s intentional. 404s though basically mean the same thing, with a nuance: this page is gone and may come back or may not, but we’re not saying anything about that. We’re just saying it’s gone.

      When I’ve talked to folks from Google about this they don’t spend much time fiddling with 410s vs 404s. End result is pretty much the same.

    12. Tom says:

      regards to inventory and expiring products
      Thanks Adam… very nice post on 404. By expiring products, are you referring also to include sold properties on a real estate site.. Wow.. on any site near a huge city, the 404 representing SOLD… expired inventory climbs quickly to a thousand + and google is so slow to remove them… it seems like they have to hit them 3-4 times before they decide they are really gone… and they then remove them… and that may be a good thing, since It would be a hell of a note if they removed them after finding the 404 for the first time… that way if your server went down for 24 hours while google was deep crawling you could wake up in the morning and find half of your site de-indexed…. thank god they try a few times before removing the page. Do you have any idea how many times they record the 404 hit before removing it from the index?

      What is worse, is that if you are using google toolbox, they totally obscure all the real faults… since without a check you cannot be sure if the 404 is a sold property or a truly bad page and who has time to check on 50 or 60 new ones per day.. as that is how many a huge city can sell..
      Do you know if google will remove them from the 404 toolbox if they find them good on the next pass? Or must you do it your self…. periodically I go thru and mark all the /idx mls homes as sold… since 99% will be sold….

    13. Adam Audette Adam Audette says:

      Tom – every site is different and in your example (real estate listings) properties that sell continue to be of interest anyway. It’s not akin to ecommerce sites with a product going out of stock. Real estate listings have market values and other details that people want, regardless of the property being for sale or currently sold. I’d focus on creating relevant messaging for sold properties and continue to serve them.

      You said wouldn’t it be bad if your site went down and Google crawled it and took down all your pages. That’s not accurate. If your site goes down it will result in 500 errors not 404s, those are quite a different beast.

      I wouldn’t worry about Google Webmaster Tools in this regard. Let it report what it reports and use it to find possible issues, etc, but don’t take it as the gospel. The tool can be really useful but also has its issues and isn’t a reflection necessarily of the “SEO health” of a website.

    14. CarrieE says:

      Thank you, Adam, for sharing your insight. It is reassuring to read your expert view & analysis.

    15. Adam Audette Adam Audette says:

      Carrie – it’s great to see you here! You’re welcome, really glad to know this is useful. Sometimes reassurance / validation is most important.

    16. Rob says:

      I recently launched a site that I had to move from one hosting account to another but 404 errors for the pages on the old hosting IP address are still appearing. I removed every thing to do with the domain from the old hosting after moving it but the pages still appear in results. It’s been about 3weeks since the migration and I think it’s affecting search results for the site on the new hosting because the pages show up on in a certain place in SERPs one day and they’ve gone the next. The home page was appearing on page 4 of SERPs regularly until yesterday morning when it vanished but the result for the link to the old hosting was there instead. How do I sort this out?

    17. Tom says:

      I certainly agree that you should never ever do a blanket 301 to the index page, but for every single url that can be sent to a related page on the new site should be done. Since every page you do not 301 will disappear… along with any links to that page.
      So you say.. how the heck do I 301 a real estate website with 100,000 pages indexed in google…that would be pretty daunting especially if the new site is laid out totally different from the old site…. but I solved that problem with an auto 301 generator that only generates a 301 when a specific page is requested. In other words if the page is not requested then the 301 would never be set… Tons of 301 may actually slow the server.. making decisions. So the idea is to build the necessary 301 on the fly… no request that file no 301 will generate. I find that using 301 as much as legally possible speeds the indexing of the new site by as much as 50% and I have experimented both ways… but for God sake.. never Ever blanket 301 to the index page…. that means you are feeding google the same page (index) page 100,000 times… sounds like a lot of duplicate content to me. Very informative post… as very true ALL 404s are not bad 404s.

    18. Daniel Valaperta says:

      Hi Adam,

      Thanks for the thought provoking post. I am working with a real estate website and trying to decide if we should 404 sold properties or 301 them.
      Before reading your post and its comments, I was convinced a 301 was better for user experience and SEO. Now I am not sure what the best decision is.
      The majority of articles/blogs that I have read seem to think the best course of action is a 301 with a message to the user saying something like “the property is no longer listed. We are taking you back to the search page where we hope to show you something else you might like.”

      “Real estate listings have market values and other details that people want, regardless of the property being for sale or currently sold. I’d focus on creating relevant messaging for sold properties and continue to serve them”
      I completely agree with this…however, If it is not possible for us to provide details of sold homes and continue to serve them, do you think we are better off doing a 301 with a message to the user? Or is a 404 still the way to go?

      Any thoughts and insight are very much appreciated. Many thanks,
      Dan

    19. Adam Audette Adam Audette says:

      Daniel,

      I’d think about what’s best for your users, not the potential SEO value. For example, if you message and then redirect that will likely negate any value from a straight 301 anyway (which I don’t think you’d want to do b/c it’s a harsh user experience).

      I can’t give you specific advice on this without diving into it myself. Think first about what the best user experience would be, then ensure it doesn’t hurt you from an SEO perspective. I don’t think there’s a perfect solution here. Very situational.

    20. Tom says:

      I am not a lawyer.. LOL… but I can give you some specific advice from Google.. about doing 301 to a 404… Google says NO DON’t DO it.. NEVER… They understand sold properties will come up 404 and eventually they will remove them from the index… but If you need I can look it up as MATT CUTTS has said many times DO NOT 301 a sold property to another page… he does recommend sending the 404 in the header then show a 404 page with other similar properties after you clearly note that the particular property is not available.. some folk even repeat their site map below the “No longer available.”

    21. Shawn says:

      Hi Adam,

      Thanks for posting this wonderful piece. I am in a dilemma with regards to soft 404 errors. I have started getting these soft 404 error messages only recently since a couple of months.

      I have a large site that publishes business listings city-wise and within that category-wise. Two potential issues lead to lot of 404 errors that I handle by doing an intelligent targeted 301 redirect.

      a) Every month lot of old business listings are removed and new ones added up that results in 404 errors. I simply do a 301 redirect to the corresponding city home page for the city specific listing that was removed. i.e if a listing that was in “City A” is now removed, then I do a 301 redirect to the home page of City A for that url. This is resulting in many soft 404 error messages via Google Webmasters Tool. I checked via “Fetch as Google” and it shows a proper 301.

      b) In many cases, lot of categories too are redundant and they are removed, so I simply do a 301 redirect to go to a higher level category till it finds a 200 Status page.

      What do you think should be done? Should I treat them as 404 or continue with the existing settings and ignore the soft 404 alerts. These settings have been in place since almost 10 years now.

    22. Tom says:

      Shawn: Google has recently been monitoring a lot of things the call soft 404… They clearly demand that you should never 301 a 404 if the cause of the 404 is a sold or no longer available… if the 404 is not a sold or should be removed then you are allowed to 301 to a similar page with similar material..

      You just said you are 301 to the same home page for all of them.. that is so clear to google what you are doing… of course the home page is not a similar page and google sees it as duplicate content… you will be better off to let the 404 stand and eventually after 3 to 6 hits google will finally remove it.. but meanwhile you are pissing them off.. by not forwarding to a similar page.. you probably have a ton of these 301 now and still building… so that is why I say you are digging your own grave.. stop and let the 404 work as it should…

      let it go hard 404 so they can remove… its only in the last year they have come down on this.. in an attempt to make the visitor get a better experience.
      Its no difference from those silly folks who transfer a domain and forward all 200,000 of their pages to the index page of the new site… bad bad decision… either 301 to a similar page or lose them… since they will lose all pr when sent to the index page anyway… also google sees the 200,000 pages leading to your index page as dupe content since all 200,000 see the same index page identical content… hope some of this makes sense to you before google git you.. LOL.

    23. Adam Audette Adam Audette says:

      Shawn, this sounds like it could be a server configuration issue. Crawl the site as Googlebot and Firefox and compare the results. Funny that the fetch as googlebot showed a correct 301. Maybe then what’s happening is an error in GWT. We’ve seen errors in the crawl report they provide. If you can confirm the correct status code of 301 is there regardless of user agent, then it’s most likely a GWT bug.

    Trackbacks
    Check out what others are saying...
    1. [...] If we wanted to be picky about it, we'd say 404 pages are not errors and can even be (blasphemy!) an acceptable user experience.  [...]

    2. [...] If we wanted to be picky about it, we'd say 404 pages are not errors and can even be (blasphemy!) an acceptable user experience.  [...]

    3. [...] Adam Audette discusses “The Best Way To Handle 404 Pages For SEO And Users” at the RKGBlog. [...]

    4. [...] “might” be a link, whether in an <a> tag, javascript, plain text, etc. Google states that 404 pages aren’t a big deal and are a natural part of the web. But we believe these links can offer really good insight into a [...]