> For the complete documentation index, see [llms.txt](https://learn.sitecove.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://learn.sitecove.com/how-to-guides/search-engine-optimization-seo/technical-seo/how-to-fix-crawl-errors-in-google-search-console.md). # How to Fix Crawl Errors in Google Search Console Crawl errors in **Google Search Console (GSC)** can prevent search engines from properly indexing your website, leading to **lower rankings and reduced organic traffic**. Identifying and fixing crawl errors is essential for **technical SEO** and ensuring that your content is fully accessible to Googlebot. This guide will explain **what crawl errors are, how to identify them in Google Search Console, and how to fix them** for optimal website performance. *** #### What Are Crawl Errors? A **crawl error** occurs when **Googlebot** tries to access a webpage but encounters an issue, preventing successful indexing. Crawl errors fall into two main categories: **1. Site Errors (Affecting Entire Website)** * **DNS Errors** – Googlebot can’t communicate with your website’s server. * **Server Errors (5xx Errors)** – The server is overloaded or unavailable. * **Robots.txt Issues** – Googlebot is blocked from crawling your site. **2. URL Errors (Affecting Specific Pages)** * **404 Errors (Not Found)** – The requested page is missing or deleted. * **Redirect Errors** – Broken or incorrect redirects (e.g., infinite loops, chains). * **Blocked URLs** – URLs disallowed by robots.txt. * **Soft 404s** – Pages that return a 200 OK status but contain “Page Not Found” content. **Fixing crawl errors helps improve search rankings, indexing, and user experience.** *** #### How to Identify Crawl Errors in Google Search Console Follow these steps to find crawl errors in **Google Search Console (GSC):** **1. Access the Crawl Error Reports** 1. Go to [Google Search Console](https://search.google.com/search-console/). 2. Select your website. 3. Navigate to **Indexing → Pages**. 4. Review errors under **“Why pages aren’t indexed”**. **2. Types of Crawl Errors You Might See** * **Not Found (404)** – Googlebot can’t find the page. * **Server Errors (5xx)** – Server failed to respond. * **Redirect Errors** – Invalid or excessive redirects. * **Blocked by robots.txt** – Page restricted from crawling. * **Duplicate Content Without Canonical** – Confusion about the preferred version of a page. **Tip:** Use the **URL Inspection Tool** in GSC to test if a page is properly indexed. *** #### How to Fix Common Crawl Errors **1. Fix 404 Errors (Page Not Found)** **Cause:** Deleted or moved pages with no redirect. **How to Fix:** * Redirect missing pages using a **301 redirect**. * Reinstate deleted content if needed. * Update internal and external links pointing to 404 pages. * Remove broken URLs from sitemaps. **Example Redirect (in .htaccess file):** ```apache Redirect 301 /old-page.html https://example.com/new-page.html ``` **Tools to Find Broken Links:** * **Screaming Frog SEO Spider** * **Ahrefs Site Audit** * **Google Search Console → Indexing → Pages** *** **2. Fix Server Errors (5xx Errors)** **Cause:** Server issues, high traffic load, or configuration problems. **How to Fix:** * Check server logs for **error messages**. * Upgrade **hosting plan** if the server is frequently overloaded. * Reduce heavy scripts or use **caching and CDNs**. * Restart the server if necessary. **Test Server Response:** * Use **Google Search Console → Pages** to find affected URLs. * Check site status with **Google PageSpeed Insights**. *** **3. Fix Redirect Errors** **Cause:** Broken, looping, or excessive redirect chains. **How to Fix:** * Ensure redirects **point to the final destination**, not through multiple hops. * Replace **302 (temporary redirects)** with **301 (permanent redirects)** if the change is permanent. * Fix **redirect loops** in .htaccess, Nginx, or CMS settings. **Example of Proper 301 Redirect (in .htaccess):** ```apache Redirect 301 /old-page https://example.com/new-page ``` **Tools to Check Redirects:** * **Redirect Checker (HTTP Status Code Tool)** * **Screaming Frog (Redirect Chains Report)** *** **4. Fix Blocked URLs in Robots.txt** **Cause:** Important pages are accidentally blocked by `robots.txt`. **How to Fix:** * Open **robots.txt** (`https://example.com/robots.txt`). * Remove blocking rules for important pages. * Use **Google Search Console Robots.txt Tester** to verify. **Bad Example (Blocking Entire Site):** ```txt User-agent: * Disallow: / ``` **Good Example (Allowing Indexing):** ```txt User-agent: * Allow: / ``` *** **5. Fix Soft 404 Errors** **Cause:** Pages return a **200 OK** response but display a “Page Not Found” message. **How to Fix:** * Configure the server to return a **proper 404 status code**. * Redirect users to relevant content with a **301 redirect**. * Ensure pages with useful content **aren’t mistakenly labeled as soft 404s**. **Test with Google Search Console → URL Inspection Tool**. *** #### How to Prevent Future Crawl Errors **1. Regularly Monitor Google Search Console** * Check **Crawl Stats** and **Index Coverage Reports** weekly. * Fix issues as they appear to avoid long-term damage. **2. Use a Clean URL Structure** * Avoid **deeply nested URLs**. * Use **SEO-friendly, human-readable URLs**. **3. Keep XML Sitemaps Updated** * Ensure your **sitemap.xml** only includes valid URLs. * Submit updates to **Google Search Console → Sitemaps**. **4. Optimize Internal Links** * Remove **broken or outdated links**. * Link to **related, valuable pages** to help Google crawl efficiently. **Use Internal Link Checkers:** * **Ahrefs Site Audit** * **Google Search Console** --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://learn.sitecove.com/how-to-guides/search-engine-optimization-seo/technical-seo/how-to-fix-crawl-errors-in-google-search-console.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.