Google Search Console:- Crawl & Coverage Errors Fixing Guide

SHARE:

Google Search Console Crawl and Coverage Errors Fixing Guide. Learn how to fix all crawl and coverage errors in search console step by step tutorial.

Hello guys, small business owners, and marketing directors, Babita Singh is here, and this is your powerful Website SEO Enhancement Step by step tutorial for you.

Today, we're gonna be talking about how to fix coverage & crawl issues in Google search console for your website. this means that you might be getting some alerts from Google search console also is known as Google Webmaster Tools on how to improve your SEO

Google Search Console:- Crawl & Coverage Errors Fixing
Therefore, you'll get an email when your website is verified with Google search console that says, new coverage issue detected for your Domain name and then it'll have a blue button in there that says fix coverage issues and if you click that button it'll take you into your Google search console.

If you don't know what is these issues and why you facing it, here is a complete guide for you which lets you know all about different types of issues and errors.

READ MORE:-
In the old Google Webmaster Tools, you used to be able to submit a fetch as Google request and that is a little bit hidden in the new Google search console, so I'm gonna show you that as well as what to do about these issues. this is actually a pretty common scenario for a URL to have a crawl issue and it's pretty easy to fix it or resubmit it to Google.

Once you're in your Google search console dashboard, you gonna see some warnings such as "Search Console has identified that your site is affected by 1 Coverage issues" and then it has a button that says done fixing, validate fix. just see screenshot below...

Google Search Console:- Crawl & Coverage Errors
But, validating the errors is the final step for the process after fixing all errors so first, you need to fix all errors and then start the validation process.

Here you may face 7 different types of Coverage & Crawl Errors, and we gonna learn how to fix these all one by one. so Get Started fixing these errors now.

Site Error >> DNS Errors>>Server error (5xx):- 

Today I am going to show you a few different methods that will help you to fix server errors, When you submit the sitemap correctly, then you will see index coverage errors, but it could be for different pages, now these pages become the most important for you to tackle first.

When the Search console report shows a warning of server error (5xx), press on it, and you will get to know all affected pages in terms of having server error. When Google loads a web page it sees the source code and it analyzes the entire HTML document.

So that means the last crawled information is important for us to keep an eye on because it may be the case that Google has visited your website perhaps when your server was being uploaded as in rebooted and then Google requested the page thus encountered an error.

Google Search Console:- Server Errors Fixing
Google Search Console also provides us some tools for you to fix these errors, such as URL  Inspection Tool, which is used to inspect your URLs, when you click on this tool it will give you feedback that "URL not on Google, indexing errors" because it just couldn't understand it because the server gave response saying "Server Error 5xx".

For Fixing these Errors Follows these Tips and Methods:-
  • Only submit a single sitemap "Parent or Child":- If you have submitted a parent sitemap and then you do have child sitemap then only submit the child ones and remove the parent sitemap. 
  • Test live URL if there is any page fetching issue:- Sometimes you may see crawl allowed but Googlebot couldn't fetch your page due to server error 5xx, here you need to test your page live, by going to Inspection tool upper right corner, and most of the time your issue will be resolved.
  • Dynamic page requests:- This may cause excessive load times; check your dynamic page requests and reduce this excessive page loading if needed.
  • Site crawling and indexing management:- sometimes Google Search Console management system intentionally prevent Googlebot from reaching the sites for testing for development purpose, so here you need to Consult your developers set up your site for optimal performance.
  • Resolve DNS Errors on Web-hosting Site:- Make sure that your web hosting server is not down or overloaded, and if your site inadvertently blocking Google then follow these steps.
    • Go to your Web-hosting Site, and Log in to your cPanel account, and follow these steps...
    • Click on the "file manager" option, under your cPanel Account.
    • Find the section "public.html" in the file manager.
    • Select the website URL where error detected.
    • Click on the site URL and find the file ".htaccess", if this file not shown to you, go to the setting at top right corner and check the option "show hidden files" and it will be shown to you.
    • Now click on the file ".htaccess", and download it for backup then delete it from cPanel.
    • Now go to Google and search for "wordpress.org htaccess", and click on the very first link, here you will find BasicWP Code, copy this code from this site.
    • Paste the copied code on a notepad and save this notepad file as .htaccess format.
    • Go to your Cpanel again upload this saved file, and then go back.
    • Now recheck your .htaccess file that your saved content showing or not.
    • Now close your cPanel, and when Google will crawl your file next time your Server Error 5xx will be removed.

Robots Errors>>Indexed, though blocked by robots.txt:-

Therefore, when you start a blog on Blogger or on WordPress and submit your website's sitemap on Google Webmaster Tool, and after few days you receive a mail from Google Search Console highlighting the issue with a warning that "Indexed, Though Blocked by robots.txt", and asking you to fix this issue. 
Google Search Console Robots Errors Fixing
Your site is affected by this issue or error because the Google search engine has found your page, but there is some instruction on your site in the form of "HTML code" that indicates to ignore the site for indexing.

So, Google follows these instructions and doesn't index your site in Google search results for the people to read, so this is important to customize your instruction in your robots.txt file according to your site need and your preference.

So if you get an email with a warning "Indexed, though blocked by robots.txt" on your blog URL then you need to follow these steps for fixing these errors on your Blogger or WordPress site.

Fixing Robots Errors on Blogger:- 

Therefore you need to follow these steps for fixing "Indexed, though blocked by robots.txt" error in blogger site...
  1. Log in to your blogger account and select your Blog affected with this error.
  2. Click on the "setting" tab under the left sidebar and further Select the "Search Preference" option.
  3. Now go to the "Crawlers and indexing" section and click on the "Edit" link opposite to "Custom robots.txt" text.
  4. To Enable custom robots.txt content, select the "Yes" option.
  5. Now COPY the code given below, and PASTE it into the box you saw here on blogger. also, replace your site name with your site name and finally click on save changes button.
User-agent: Mediapartners-Google
Disallow:
User-agent: *
Disallow: /search/labels/ 
Disallow: /category/
Disallow: /tag/
Allow: /
Sitemap: https://www.yoursite.com/atom.xml?redirect=false&start-index=1&max-results=500
Google Search Console:- robots Errors Fixing

Fixing Robots Errors in WordPress:-

When you run a site on WordPress and you get the mail from Google Search Console with the same warning, then your site might be affected by duplicity content problem, and you need to fix it. so now follow these steps to fix this issue for WordPress site.
  1. Go to your WordPress site dashboard and select the "plugin" option under the left sidebar.
  2. Install the "Yoast Plugin" if you already not installed it yet.
  3. Now go to the plugin on the dashboard and select the "search appearance" option.
  4. Now click in the "Taxonomies" tab, here go to the "categories" section and select the "NO" option, then go to the "Tags" section and select the "NO" option again, and then go to the "Formats" section and make it disabled and click on the "save changes" button.
  5. Now click on the "Archives" tab and select the "Disabled" option.
  6. Finally, save the changes.
Now when you completed these fixing on your Blogger or WordPress site then go to your Google Search Console dashboard and then click on the Coverage Tab under the right-side menubar.

Now click on the error or warning option under coverage reports, and then scroll down and then click on the warning type "Indexed, though blocked by robots.txt", now you find all affected pages so the last job is to start validation process by click on the "Validate Fix" button.

URL Errors>>Submitted URL not found (404):- 

There are several reasons you may get a warning from Google Search Console "Submitted URL not found (404)" after submitting your sitemap.xml file. and search console ask you to fix it.
Google Search Console 404 Errors Fixing
Your site has this error because of changing or deleting your Page or Post URLs, and it happens when...
  • You write a post and then delete it after a few days
  • If you make any changes in the permalink or URL of your post.
  • When you sift your complete site from Blogger to WordPress and this process automatically change all the URLs on your Website.
When you get these crawling errors and you don't fix it, you will not get a high ranking in Google search results, although you write several keywords specific post or use multiple SEO Strategies, your website won't get enough search results. 

So it is very very important to fix these crawling errors for getting optimal search results in Google. So now we will learn how to fix this 404 Error. There are only two ways to fix these errors...
  • Remove all affected URLs permanently:- When you already deleted your post or changed your permalink, there is no use to index old or deleted URLs in the search engine, and this will harm your site ranking as well, so you can remove these URL from Search Engine Using "Removal" tool in Google Search Console. here you can follow these steps for Blogger & WordPress to do so...
    • Go to your Google Search Console Dashboard, and click on the "Coverage tab" in left-hand side menubar.
    • Now click on the "Error" option on the coverage report.
    • Now scroll down, and find the "Not found (404)" error, and click on it.
    • Now here you will find all affected pages with 404 Error, copy these all URLs on a Notepad.
    • Now go to the "Removals" tab, under the Index section in the left-hand side menubar.
    • Now click on the "New Request" button, and there is another window pop-up, here paste you Affected URL one by one, choose your options and click on the "Next" button.
    • Now your all affected URLs temporarily invisible to the search engine.
  • Redirect These URLs with 301 Redirect technique:- Deleting your entire links from Google Search Console won't be a good idea, because deleting all shared links means, if someone clicks on those links they won't find anything in search results and you loose that possible visitor too for your site. so if you are wise enough use 301 redirect technique, which helps you in...
    • redirecting your possible visitors from broken pages to newly updated pages.
    • utilizing your previous traffic to new sites or pages. 
    • Multiply your traffic sources and upgrade your site in search results.
    • Multiply your revenue with increased traffic

301 Redirects:-

When you remove your broken links or the URLs affected by 404 Error, you also lose the link juice or traffic you can get by redirecting these to new pages or posts, and if you sift your entire site from Blogger to WordPress, you may have hundreds of pages and posts which already getting huge traffic to your blogger site.
Google Search Console 301 redirects
And if you delete these pages from Google Search Console you will lose all the traffic you already get to your site. that is why I never recommend anyone to remove any broken link from search console, this feature should be used only if you don't have other options.

So now you need to learn how to redirect your broken URLs to new posts or page in Blogger as well as in WordPress site so that you can utilize all broken links for getting optimum search results and indirectly you get higher google ranking by getting best traffic from all shared links.

Here you have 3 basic strategies to implement for reconnecting your broken links again...
  1. If you changed your old permalink, then redirect it to a new updated page or post.
  2. If you deleted your post permanently, then redirect your old link to your site's homepage or any post which is relevant to the old post.
  3. Create specific posts or pages for those broken links and redirect those links to these new posts or pages.

301 Redirect Method for Blogger:-

If you run your site on Blogger and you got a warning for 404 Errors, this means your site has some broken links which need to reconnect or remove. so not we gonna learn how to use the 301 redirect method for reconnecting your these pages on blogger.
Here you need to follow these steps for using the 301 redirect method for your broken links.
  • Go to your Google Search Console Dashboard and click on the "Coverage" Tab at left-hand side menubar.
  • Click on the "Error" option on the Coverage report and Scroll down.
  • Find the error "Not found (404)" and click on it.
  • Now here you will find all affected pages URLs with this error, now copy these all URLs and paste it on a notepad.
  • Now go to your blogger site dashboard and click on the setting option at left-hand side menubar.
  • Now click on the "Search Preference" option under the drop-down menu
  • Now go to the "Errors and redirections" section and click on the "Edit" link opposite to the "Custom Redirects" option.
  • Click on "New Redirect" option and a new box will be opened
  • Now go to your saved Notepad, copy your first URL after slash and paste this on blogger the "New Redirect"
  • Now redirect it to new URL, that may be URL Site's homepage (type only slash icon ), you can redirect it with updated URL (type updated post or page URL after the slash icon), or you can redirect it to a new post relevant previous link (type new post URL after the slash icon).
  • Now tick on the "Permanent" box and then save it.
  • Follow the process with all affected URLs one by one and redirect these according to your preference.
Google Search Console 301 redirect in blogger

301 Redirect Method for WordPress:-

Now, if you have your site on WordPress or you Sifted your site from Blogger to WordPress, then you need to redirect your all old URLs to New Updated URLs.

Here you can follow this step by step process for redirecting your all broken links to New web pages. Here you need to install a plugin "Redirection" on your WordPress site and then activate it, now...
  • Go to Plugins option on the dashboard under left-hand side menubar
  • Under Plugins Section you will find the "Redirection" plugin, click on setting option just below this plugin and a new window will be opened.
  • On this plugin page click on the "Redirects" tab, and then click the button "Add New".
  • Now copy your first URL from saved Notepad and paste it to the "Source URL" option.
  • Now paste your updated post link, or new post link or home-page Url whatever you what for that particular redirect under the "Target URL" option.
  • Now under the group option select "Redirection" and click on the "Add Redirect" button.
  • Follow the process for all affected URLs one by one.
  • Now your URL redirection added but not activated, so scroll down to your added URL select it by right check (✔) then go to the drop-down menu just above this list and select enable the option and then click on the "Apply" button, and it's done.
Google Search Console 301 redirects in wordpress
Now you have redirected your all old or broken links to new post or pages, and when any user click on your old links he automatically redirected to your new post and your 404 error will be removed.

URL Errors >>Submitted URL Seem to be a Soft 404:- 

When you get a warning that "Submitted URL Seem to be a soft 404" from Google Search Console, it means after crawling your all pages by Googlebot they found some pages where they found no content, or these pages showing no content at all.

Here your pages unable to load any elements such as images, text, and videos, etc. And Google unable to read your content, so in this condition when Googlebot doesn't find any content on your pages, they send you an error warning of Soft 404. and you need to fix it.

Google Search Console Soft 404 Errors Fixing
Now let's start fixing these errors so that Google can read you all content easily. so for doing so you should follow these steps for both Blogger and WordPress Sites...

If you have published the page but you have not written any content yet, then write some content on the page and then...
  • Go to your Google Search Console Dashboard
  • Click on the error option on the coverage error report. 
  • Now click on the issue "Submitted URL Seem to be a Soft 404" option 
  • Here you will find all the affected pages just Click on the Validate Fix button, and you Soft 404 Error will be removed soon.
If you already have written content on these pages, still you get a warning with the error "Soft 404", this means Google unable to fetch your content easily. so now you have to follow this process...
  • Click on the error "Submitted URL Seen to be a soft 404" under coverage reports errors option, and find all the affected pages with this error.
  • Now click on your 1st page URL and inspect it with URL Inspection Tool with live URL Testing and Google will fetch your content again from your page.
  • Now, most of the time Google will show you a prompt "URL is available to Google", so if you get this prompt just click on the link "Request Indexing", and the issue will be resolved.
Still, you didn't get rid of the error, then you only 2 ways to resolve the issue...
  1. Copy you all content from the page, and paste it on a new page. then delete your previous page, now submit your new URL under URL Inspection Tools and request for indexing.
  2. Redirect your previous link to any other page, where you have relevant content or redirect it by 301 on your website's Homepage or about us page.

URL Errors>>Submitted URL Marked "No Index":-

When you get a warning from Google Search Console that "Submitted URL Marked No Index", it means you have not enabled "Custom robots header tags" yet, in the search preference section under the setting tab on your blogger dashboard.

Google Search Console No-Index Errors Fixing
So, here you need to enable your "Custom robots header tags" for removing this error permanently from Blogger, now follow the steps for doing so...
  • Go to the Blogger Dashboard and then click on the Setting Tab.
  • Now select the "Search Preference" option under the drop-down menu.
  • Now go to the "Crawlers and indexing" section and then click on the "Edit" link opposite to the "custom robots header tags".
  • Now select the "Yes" option right check (✔), and drop-down form will be opened.
  • Here right check (✔) the given option in all sections.
    • Homepage:- First line "All", Second Line "noodp"
    • Archive and Search pages:- First Line "noindex" Second Line "noodp"
    • Default for Posts and Pages:- First line "All", Second Line "noodp" 
  • Now click on the button "Save changes", and you have done. 
Google Search Console No-Index Errors Fixing
If you run a site on WordPress and then you get a warning related to the "
Submitted URL Marked No-Index" error, then you should follow these instructions for getting rid of this error...
  • Here you need to install a plugin "Yoast SEO" for your SEO Settings.
  • After proper Yoast SEO plugin installation, click on the plugin link under the Wordpress Dashboard right-hand-side menubar.
  • Under drop-down menubar select the "search appearance" option.
  • Now you are on the Yoast SEO dashboard, here you need to select "content types" tab.
  • Here you need to do settings for single Post URLs. so select the "Yes" option for "show posts in search results" query.
  • Select the "show" option for the "Date in Google Preview" option.
  • Now select the "show" option for the "Yoast SEO meta Box" option.
  • Now click on the "save changes" button. and you have done all settings.
Now, after doing these all settings in Blogger or WordPress site, you need to go to your Google Search Console Dashboard and click on the "Error" option under coverage report, and then scroll down and find the error "Submitted URL Marked No-Index".

Click on this Error statement and, it will show you all affected pages. then Live Test one by one all affected page URLs using Inspection Tool, and when you see a prompt that "URL is available to Google" you just submit a request for indexing, and the Issue will be resolved soon.

URL Errors>>Submitted URL returns unauthorized request (401):-

The 401 error occurs when your request has not been applied because it lacks valid authentication credentials for the target resource. basically, this just means you can try accessing the resource on your website using the correct credentials.

Google Search Console 401 Errors Fixing
It's often a temporary problem however if you can't get this error to go away check out these five different solutions...

1. Flush your domain name system records:-  

Flushing your DNS involves deleting temporary data from your computer that way the next time you try to access that problematic URL, it'll make a completely new request and re-authenticate.
  • If you're a Windows user, Open the command prompt, and type the command "ipconfig/flushdns". Your DNS windows will show you a success message and you can access your website again.
  • If you are a Mac user, follow the similar process, but you need to enter the command according to which type of OSX you're running, If you are a user of these iterations (Sierra, High Sierra, and Mojave) you can use sudo killall -HUP mDNSResponder.

2. Clear your firewall and browser caches:- 

Your browsers can have a datastore so that they don't have to load websites from scratch every time you access your website, this process called caching. sometimes it might inadvertently be storing wrong login details.

If you using a firefox browser you can follow this process...
  • Go to the Menu tab on the upper right corner and click on the "options" gear. 
  • Click on the "privacy and security" option under the left side menubar.
  • Scroll down below and then click on the link "Clear Data", under the "Cookies and Site Data" section.
  • Here only right check (✔) cached web content, and click on the "clear" button, and now all cached will be gone.
Google Search Console:- 401 Errors Fixing
At this point when you try accessing your website but the problem keeps remains, that means the error may be due to your firewalls cache. For fixing this error you need to follow the process...
  • If you're a Cloudflare user go to your dashboard and navigate the caching. 
  • Here you have an option to "purge everything" which includes your entire firewall cache, click on that.
  • Here new window will pop-up and here click on the "purge everything" button.
Google Search Console:- 401 Errors Fixing in wordpress
This purging won't affect your firewall's functionality, but you might run into longer loading times when you visit next time to your site but that should fix itself after your cache is rebuilt.

If you're using a different firewall tool you'll want to look through the documentation and check, if it enables you to clear your cache manually. for some services, it might be necessary for you to contact support instead.

3. Test for conflicts between your WordPress Plugins and Theme:- 

Your site's theme and plugins compatibility issues can sometimes trigger a 401 error. deactivating either element in order to determine the source of the conflict, you need to access the Wordpress dashboard and follow the process...
  • Here simply click on plugins on the left side menubar.
  • On plugins window just select all plugin and click the deactivate link.
  • For a theme go to the "Appearance" tab in the left menubar and select the "Themes" option.
  • Now change from your active theme to another theme and you're all good.
Now visit your website and check everything working fine or not, then reactivate each plug-in one by one and wait for the issue to reoccur between your plugins and your theme.

You should be able to narrow down the problematic element however a 401 error may prevent you from accessing your dashboard, in this case, you'll disable your plugins and theme using an FTP client of your choice.

Google Search Console:- Crawl & Coverage Errors Fixing
Once you've connected to your server make sure you're in your root WordPress folder and from there...
  • Go to WP dash content and then go to plugins.
  • To deactivate your plugins simply change the name by adding underscore disabled to the folder that has all the plugins files.
  • Now your WordPress Dashboard will be shown and now follow the previous process that was deactivating and re-activating plugins one by one and find which one is causing the issue
  • If your WordPress Dashboard disabled due to your theme, then go back to WP dash content again and click on the themes folder. 
  • Now just find your theme and just like the plugins we're going to change the name of the theme by adding underscore disabled and now it will go back to a default theme and allow us to find out if our theme was causing the issue.

4. Disable password protection for your WordPress Directories:- 

Your WordPress directories include a .htaccess file that can enable you to implement several features on your site, such as password protection, which can add an extra layer of security in your WordPress site.

At this stage, you shouldn't include a recovery process because if you forget your credentials you're locked out unless you disable the feature entirely.
Google Search Console:- Crawl & Coverage Errors Fixing
Disabling the credentials, you need to locate the .htaccess file within your WordPress directory, here you can see here at the bottom is a snippet of code that basically says password-protected area if you go ahead and just delete this snippet altogether you will disable password protection.

Additionally, every relevant directory should have a second file right underneath this file with name .htpassword, deleting this will also remove password protection but only for that specific folder whichever approach you to decide to take.

Now try to reaccess in your site once you make the necessary changes if that doesn't work the source of your 401error, you need to go for the final approach below.

5. Contact your hosting provider:-

If everything on this list fails and the 401 error persists for a while without resolving itself which again makes sure you check because it may just fix itself your best bet is to get in touch with your hosting provider.

At this stage, you've tried everything which means there may be a server related issue causing the problem your provider support team has access to better diagnostic tools which means they can help you better.

Read More:-
hopefully, we were able to help you resolve the all coverage error on your website, and these tips and processes will help you a lot with getting rid of your Google Search Console errors and warnings.

I hope you enjoyed this post "Google Search Console:- Crawl & Coverage Errors Fixing Guide" and found it valuable and useful. and this gonna help you for fixing all coverage & crawl related issues.

Still, you have any trouble, or confusion in fixing your errors in Google Search Console, please let me know by commenting below, I will come back to you with an actionable solution.

Please do share this post with your friends so that they also can optimize your Site SEO, and subscribe to us for getting free digital marketing Tools & tips directly into your E-Mail Box.
Babita Singh
Thanks, and make it a great day
Babita Singh

COMMENTS

BLOGGER
Name

Blogging Tools,8,Copywriting Tools,1,Digital Marketing Tools,6,Gmails Power Features,2,Google Tools,10,Post Scheduling Tools,2,SEO Tools,6,Social Media Marketing Tools,2,Website Analysis Tools,5,
ltr
item
Digital Marketing Tools:- Google Tools For Online Marketers: Google Search Console:- Crawl & Coverage Errors Fixing Guide
Google Search Console:- Crawl & Coverage Errors Fixing Guide
Google Search Console Crawl and Coverage Errors Fixing Guide. Learn how to fix all crawl and coverage errors in search console step by step tutorial.
https://1.bp.blogspot.com/-oQ3ek5K6hIA/Xs9t0Ac96wI/AAAAAAAACZ0/Vpiug7ZbGrkkTCpotsAhoJA6y_Njdt7IgCPcBGAYYCw/s1600/Google%2BSearch%2BConsole%25284%2529.webp
https://1.bp.blogspot.com/-oQ3ek5K6hIA/Xs9t0Ac96wI/AAAAAAAACZ0/Vpiug7ZbGrkkTCpotsAhoJA6y_Njdt7IgCPcBGAYYCw/s72-c/Google%2BSearch%2BConsole%25284%2529.webp
Digital Marketing Tools:- Google Tools For Online Marketers
https://www.prodigitalmarketingtools.com/2020/05/google-search-console-crawl-coverage-error-fixing.html
https://www.prodigitalmarketingtools.com/
https://www.prodigitalmarketingtools.com/
https://www.prodigitalmarketingtools.com/2020/05/google-search-console-crawl-coverage-error-fixing.html
true
7186187585424453050
UTF-8
Loaded All Posts Not found any posts VIEW ALL Readmore Reply Cancel reply Delete By Home PAGES POSTS View All RECOMMENDED FOR YOU LABEL ARCHIVE SEARCH ALL POSTS Not found any post match with your request Back Home Sunday Monday Tuesday Wednesday Thursday Friday Saturday Sun Mon Tue Wed Thu Fri Sat January February March April May June July August September October November December Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec just now 1 minute ago $$1$$ minutes ago 1 hour ago $$1$$ hours ago Yesterday $$1$$ days ago $$1$$ weeks ago more than 5 weeks ago Followers Follow THIS PREMIUM CONTENT IS LOCKED STEP 1: Share to a social network STEP 2: Click the link on your social network Copy All Code Select All Code All codes were copied to your clipboard Can not copy the codes / texts, please press [CTRL]+[C] (or CMD+C with Mac) to copy