The Best Way to Handle 404 Pages for SEO and Users
GitHub 404
There are a few common misconceptions related to 404 pages and SEO that I’ll attempt to clarify.
While it’s common (and technically correct) to call 404 pages “errors,” I am not a fan of that terminology. 404s are the expected result when a website is unable to return a given request, if there is no URL to deliver. If we wanted to be picky about it, we’d say 404 pages are not errors and can even be (blasphemy!) an acceptable user experience.
Calm Down. 404s Are Ok.
404s are fine. There’s no need to panic. There’s no need to “redirect every 404 to the home page” (eek!), the category parent, or your shopping cart.
All too often I hear SEOs recommending that every 404′d URL be redirected somewhere – anywhere – just so long as “the juice is captured.” Nonsense.
I think people tend to freak out about 404s because they sense that Google will negatively score a site if they exist. Or, they worry that link equity is lost when quality links are pointing to a 404 page (a valid concern, which I’ll elaborate on below).
Generally speaking (though there are exceptions), 404 pages are a very normal part of running a website. There’s no reason to fret over them and there’s certainly no proof (at least that we’ve seen) of Google or Bing penalizing a site for 404 pages.
When 404s Can Be “Bad”
There are cases, however, where a site could be scored negatively as a result of an abnormally high occurrence of 404 pages. Does your site have 10,000 unique pages but 45,000 404s this month? That could be an indicator of a bigger problem with crawling and technical SEO. That’s also probably not a very good user experience, which could be reflected in poor user engagement and falling traffic, which could be reflected again by a search engine lowering visibility of your URLs in search results.
A high occurrence of 404s – when they are spiking and continuous – is not a good thing. But forget about SEO for a moment. Is that a good thing for users? Obviously not, and remember: search engines follow users.
Another potentially negative consequence of 404 pages is when a URL has valuable links. I’m not so rabid about “link equity” that I’m going to recommend every URL with links needs to be redirected. If there’s not a good page match for a 404, don’t redirect it. That said, if the links are precious and difficult to secure, think about contacting the site and having them update their link, or creating a new piece of content that’s relevant for the existing link, or finding a relevant page that will still ensure a good user experience and 301 the page.
Last but not least, 404s can be bad news when they act like a 200. A soft 404 error page is not a good thing, because search engines will continue to index these. Ensure your server is configured to return the proper status code for each type of page.
There’s an interesting technical SEO problem with regards to inventory and expiring products. But I’ll save that for another day.
Best Practices for Redirecting 404 Pages
Here is a quick list of best practices for redirecting 404 pages:
- 404s should not always be redirected.
- 404s should not be redirected globally to the home page.
- 404s should only be redirected to a category or parent page if that’s the most relevant user experience available.
- It’s okay to serve a 404 when the page doesn’t exist anymore (crazy, I know).
- If you have valuable links pointing to 404 pages, do one of the tactics I outlined in the above section.
- Don’t panic. 404s are normal.
Helpful Links
Bing’s 404 page best practices
http://www.bing.com/webmaster/help/404-pages-best-practices-1c9f53b3
Google’s 404 page best practices
http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93641
Cool 404 examples
http://econsultancy.com/us/blog/9525-16-creative-404-pages-to-inspire-you-to-overhaul-yours
Another good resource on 404 pages and SEO
http://www.fallingupmedia.com/blog/404-page-examples-seo/
Looking forward to the next installment on the “interesting technical SEO problem with regards to inventory and expiring products”. I’ve been dealing with those issues for years and really looking forward to hearing your take on them. I hope we see that one soon :)
Hey Adam,
I love funny 404 pages and will definitely check out your links. It drives me nuts when I get a 404 redirect to the homepage. I’d rather experience the 404 then decide where I want to go if I’ve entered the web address in error.
Rob, great to have you here! Yeah that topic seems to be right up your alley. Now I’m curious to get your take on it, too. Will put together the post soon and we can discuss.
Michael – great point. Me too!
Adam
HUGE +1 to the “expiring inventory” issue… would LOVE to hear more about that!
Thanks for the post – I know I’ll be sharing this with clients who might freak out a little about 404′s :)
-Dan
Dan – cool. I’ll get on it! (Glad you like the post.)
We need more commonsense posts like this one Adam. I despise global 404 redirects. My test is always domain.com/foobar and when it does a redirect to home I vomit a little bit in my mouth.
Frankly, given the amount of link rot wouldn’t a site without any 404s be more of a red flag? If I were a search engineer I know I’d be on the look out for that (and I’m not nearly as smart as they are.)
The expired inventory issue is a complicated but fun one. Lots of ways to do it really but eager to hear your thoughts on it.
What I “worry about” are the soft 404s and the “not found” in Webmasters Tools. I figure they are telling me because they care. So I try to fix and avoid those type of issues. I don’t think having a 100% updated site is bad but I do think just random 404 redirecting to the homepage is a pretty bad idea.
Thanks for weighing in, AJ. Yep, agreed in regards to link rot and signals like abnormally low (or no) 404s. Seems like an easy flag. Then again, I’m not as smart as Amit Singhal and gang either!
Matt – I addressed soft 404s in the article. They are a valid concern. GWT isn’t something you should take as gospel. We’ve seen errors reported in there from Googlebot’s own malformed crawls, quite frequently. It’s nice to use for a quick check, but if you really want to know what Googlebot (or any spider) is doing, grab the raw log files and mine ‘em with Sawmill, Splunk, or some nice sed and awk scripts.
Hey Adam,
I got many 404 errors from Google Webmaster.I have no option to resolve this problem.But after read your post,I got the solution.I’m seriously thankful to you.
“It’s okay to serve a 404 when the page doesn’t exist anymore”
If you’re going to prioritise web standards, user experience and SEM in that order then shouldn’t you be saying “410″ (‘Gone’) here?
@Gridlock – 404s and 410s are basically treated the same way. The difference is really subtle and probably too nuanced to worry about. I like 410s – gone – because (at least technically) they’re supposed to mean: this page is gone and will never come back, and it’s intentional. 404s though basically mean the same thing, with a nuance: this page is gone and may come back or may not, but we’re not saying anything about that. We’re just saying it’s gone.
When I’ve talked to folks from Google about this they don’t spend much time fiddling with 410s vs 404s. End result is pretty much the same.
regards to inventory and expiring products
Thanks Adam… very nice post on 404. By expiring products, are you referring also to include sold properties on a real estate site.. Wow.. on any site near a huge city, the 404 representing SOLD… expired inventory climbs quickly to a thousand + and google is so slow to remove them… it seems like they have to hit them 3-4 times before they decide they are really gone… and they then remove them… and that may be a good thing, since It would be a hell of a note if they removed them after finding the 404 for the first time… that way if your server went down for 24 hours while google was deep crawling you could wake up in the morning and find half of your site de-indexed…. thank god they try a few times before removing the page. Do you have any idea how many times they record the 404 hit before removing it from the index?
What is worse, is that if you are using google toolbox, they totally obscure all the real faults… since without a check you cannot be sure if the 404 is a sold property or a truly bad page and who has time to check on 50 or 60 new ones per day.. as that is how many a huge city can sell..
Do you know if google will remove them from the 404 toolbox if they find them good on the next pass? Or must you do it your self…. periodically I go thru and mark all the /idx mls homes as sold… since 99% will be sold….
Tom – every site is different and in your example (real estate listings) properties that sell continue to be of interest anyway. It’s not akin to ecommerce sites with a product going out of stock. Real estate listings have market values and other details that people want, regardless of the property being for sale or currently sold. I’d focus on creating relevant messaging for sold properties and continue to serve them.
You said wouldn’t it be bad if your site went down and Google crawled it and took down all your pages. That’s not accurate. If your site goes down it will result in 500 errors not 404s, those are quite a different beast.
I wouldn’t worry about Google Webmaster Tools in this regard. Let it report what it reports and use it to find possible issues, etc, but don’t take it as the gospel. The tool can be really useful but also has its issues and isn’t a reflection necessarily of the “SEO health” of a website.
Thank you, Adam, for sharing your insight. It is reassuring to read your expert view & analysis.
Carrie – it’s great to see you here! You’re welcome, really glad to know this is useful. Sometimes reassurance / validation is most important.