Recommended: PC Feels Slow? A Free Scan Shows What's Dragging Windows Down →Recommended: Crashes or Glitches? A Free Driver Scan Usually Finds the Culprit →Recommended: Fix Windows Errors and Clear Junk Files in Minutes - Free Scan →
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Some links on this page are affiliate links: if you buy through them we may earn a commission, at no extra cost to you.
Google Search Console is a powerful tool for website owners. It’s like a direct line from Google telling you about the health of your website from their perspective. One of the most important things it reports on is crawl errors. These happen when Google’s automated program (called Googlebot) tries to visit a page on your site but can’t access it for some reason. Seeing a chart full of red error lines in Search Console can be a bit alarming at first – I know I felt a jolt of panic seeing those early reports – but understanding what they mean and how to fix them is key to a healthy website and good performance in Google Search.
Fixing crawl errors ensures that Googlebot can access and index your important pages, which is fundamental for SEO (Search Engine Optimization). It also helps provide a better experience for users who might encounter broken links. This guide will walk you through how to find these errors in Google Search Console and fix the most common types.
General shopping ads
What Are Crawl Errors and Why Fix Them?
A crawl error simply means Googlebot tried to reach a specific URL on your website and failed. This failure can happen for many reasons (which we’ll cover).
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Fix the driver behind crashes, sound loss and screen glitches3Repair Windows errors before they cause bigger problemsSpecial offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Why is fixing them important?
- Indexing & Ranking: If Googlebot can’t crawl a page, it can’t index it. If it’s not indexed, it can’t rank in search results.
- Crawl Budget: Google has a limited amount of time/resources to spend “crawling” your site (your crawl budget). Errors waste this budget on broken pages instead of discovering new or updated content.
- User Experience: Many crawl errors correspond to broken links (pages returning a “Not Found” message), which frustrates visitors.
Let’s dive into Search Console to find these issues.
Shopping ad
Sale
FYY Electronic Organizer, Travel Tech Pouch Bag, Cable Organizer Black
- Dimensions: 7.5" x 4.3" x 2.2". Compact size and lightweight make it easy to carry and put into your backpack, handbags or laptop bag without taking much space. Suitable for family use and daily organization. Note: Small mesh pockets are ideal for charging cords no longer than 3ft; longer cables (over 3ft) fit better in the larger compartments
- Quality Material: This electronic organizer travel case made of high quality durable waterproof oxford and soft sponge inside to secure your gadgets in place and deliver a quick access whenever you want. Water-resistant fabric protects your gear from unexpected splashes, keeping all your electronic essentials safe and secure
- Double Layers Design: This tech pouch features a double-layer interior design with 8 compartments, including multiple see-through mesh pockets and ample space to store your cords, cables, USB drives, cellphone, charger, mouse, flash drive and more, keeping all accessories neatly organized and tangle-free
- Practical and Convenient: Comes with a comfortable hand strap for easy carrying; You may carry it in your hand when heading out. Durable and smooth zipper closure keeps your favorite device securely, convenient for you to have quick access to the items inside the case
- Portable and Lightweight: The small size and lightweight design durable cable organizer pouch is a perfect choice when going on holiday, business trip, travel, office. Enjoy hassle-free travel without wasting time on tangled accessories. Great gift for yourself also a nice share with families and friends. (No include cords, electronic accessories)
Step 1: Find Crawl Errors in Google Search Console
Crawl errors related to Googlebot being unable to access pages are primarily reported in the Pages section of Google Search Console.
- Log In: Go to Google Search Console and log in to your account.
- Select Your Property: Choose the website you want to check from the property dropdown in the top left.
- Go to the Pages Report: In the left-hand menu, click on Indexing, then click on Pages.
- Review Error Details: This report shows you how many pages on your site are indexed and, more importantly for this task, how many pages aren’t indexed and the reason why. Look for reasons listed under the “Why pages aren’t indexed” chart that indicate crawling or access problems, such as:
- Not found (404)
- Server error (5xx)
- Soft 404
- Blocked by robots.txt
- Blocked by authorization request (401)
- Redirect error
- Check Sitemaps Report (Optional but Recommended): Also check the Sitemaps report (under Indexing). If your sitemap file submitted to Google contains URLs that are causing errors, this report might flag issues related to those specific URLs.
Focus on the errors listed in the Pages report, as these represent pages Google is actively trying to crawl but failing.
Shopping ad
Sale
Ordilend Keyboard Cleaner & Laptop Cleaning Kit, All-in-1 for Computer PC
- 【UPGRADED LAPTOP CLEANING KIT 】 The macbook cleaning kit computer screen cleaner comes with a number of accessories including a retractable large brush, polishing cleaning cloth X 2, keycap puller, metal pen tip, flocking sponge, thin soft brush, soft plastic lens cleaning pen, 5 replacing cloth, large cleaning microfiber cloth. You deserve the comprehensive computer cleaning kit keyboard vacuum at a low cost
- 【PROFESSIONAL KEYBOARD CLEANING KIT】 The laptop screen cleaner keyboard cleaner can pull out the keycaps of gaming keyboards and mechanical keyboards. A retractable keyboard brush works on laptops and keyboards, while the mini high-density brush is great for deep cleaning between keys for cleaning between flatter keys on a laptop, the metal pin tip gently removes any stains. This electronic cleaning kit macbook cleaner totally meets professional cleaning needs
- 【OFFICE DESK ACCESSORIES】This keyboard cleaner kit is easy to use and can clean your keyboard and electronic screen with just one swipe. Wiping with the 2mm thicken widen polishing cleaning cloth designed at a right angle for better fitting screen corners of computers with our recyclable cleaning spray, The laptop cleaner kit for macbook effectively absorbs stubborn stains, leaves no discoloration, no streaks, and no fiber shedding on the screens
- 【MULTIFUNCTIONAL TOOLS 】Mini soft brush and soft plastic lens cleaning pen are specially designed for DSLR camera screen, lens, and other delicate surfaces. 5 more cleaning cloths of it supplied for replacement. The flocking sponge is an excellent tool for cleaning earbuds charging cases, And the earbud cleaning kit is ideal. This electronics for college students is equivalent to 10 other electronic cleaning kit
- 【PORTABLE DESIGN & CLEANER TOOL】 The office supplies is compact in design, easy to carry, and you can easily take it anywhere. It's convenient to keep one in a drawer, one in your car, or in your bag and dorm. It is easy to use and can clean your keyboard and electronic screen with just one swipe. Is the college essentials cleaning tool for your friends, family, colleagues and students
Step 2: Understand the Error Types
Before you fix, you need to know what each error type generally means:
Recommended Free Tools
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
- Not found (404): The server responded that the page at that URL does not exist. This is often due to a broken link or a page that was deleted or moved without a redirect.
- Server error (5xx): Googlebot tried to access the page, but your website’s server had a problem. This could be due to server overload, maintenance, or errors in your website’s code interacting with the server.
- Soft 404: The server responded that the page exists (returned a 200 OK status), but the content is missing, empty, or appears to be an error page to Google. This can confuse search engines.
- Blocked by robots.txt: Your robots.txt file (a file on your server that tells search engine bots where they can and can’t go on your site) is preventing Googlebot from crawling this specific URL.
- Blocked by authorization request (401): The page requires a login or authentication to access it. Googlebot cannot log in.
- Redirect error: Googlebot encountered a problem while following a redirect from one URL to another (e.g., a redirect chain that’s too long, a redirect loop where URLs redirect back to each other, or a redirect to a bad URL).
Step 3: Investigate Specific URLs Using the URL Inspection Tool
Shopping ad
BAGSMART Large Electronic Organizer Travel Case for Tech Accessories, Black
- Compatible Space: This electronics organizer bag features 2 zippered mesh pockets fits phones, standard power banks, 4 elastic loop pouches for small items, 2 elastic loop pouches for wireless headphones and small chargers. Elastic loops for phone charging cable. And specific slots for SD cards. Please check the size to ensure it meets your needs
- Lightweight Travel Accessories: The size of the electronic organizer travel case is 10.6" L x 7.5" W x 1.2" H, Compact but substantial size fits your intended bag or space. Suitable for traveling use and daily organization
- Keep Everything Organized: This compact travel organizer features dedicated compartments for your phone charger, cables, and tech accessories, keeping them tangle-free and ready to go. You can find travel accessories quickly, no chasing cords in your pack anymore
- Durable Travel Essentials: Features double zippers for easy access, elastic loops with non-slip grips for daily protection. Organizer for office use and traveling, (Not including cords, electronic accessories). It can serve as a travel checklist. Before you leave a place, just open the case and check if everything is there, preventing you from leaving things behind
- Versatile Use: Its practicality and convenience make it a travel essential bag. It is suitable for a weekend trip, business trip, and travel. This organizer pouch is suitable for office, business, daily use and can be given as a gift for friends, family, or men, for birthdays, Valentine's Day, Christmas Day, Father's Day
Once you’ve identified an error type, you need to look at specific examples to understand why it’s happening for that particular URL. The URL Inspection tool is your best friend here.
- In the Pages report, click on the specific error type you want to investigate (e.g., “Not found (404)”).
- You’ll see a table listing example URLs experiencing this error. Click on a specific URL from the list.
- Click the Inspect URL button or icon (it looks like a magnifying glass). This opens the URL Inspection tool for that specific page.
- The tool will perform a live test and show you details about Google’s last crawl attempt, index status, and the exact error Googlebot encountered. This information is vital for figuring out the root cause. It might show if the URL was found in your sitemap or linked from another page on your site.
How to do it: Use this tool on several examples for each error type to see if there’s a pattern (e.g., all 404s are from an old section of the site, all 5xx errors happen at certain times).
Shopping ad
ColorCoral Cleaning Gel Universal Dust Cleaner for PC Keyboard Car Detailing Office Electronics Laptop Dusting Kit Computer Dust Remover, Computer Gaming Car Accessories, Gift for Men Women 160g
- Universal fit: ColorCoral cleaning gel, simple and convenient cleaning kits for PC/laptop keyboard and other rugged surface, such as the car vent, camera, printer, telephone, calculator, Instrument, speaker, air conditioner, TV and other appliances
- Safe cleaning gel: The keyboard cleaner gel is made from natural gel, no sticky to hands, smells sweet with lemon fragrance, no stimulation to skin
- Easy dust cleaning: Make sure your hands are dry and clean, knead the cleaning gel into a ball, press the cleaning gel slowly into the keyboard, car vent and rugged surface till the cleaning gel could touch the bottom and then pull out, the dust would be carried away with the cleaning gel
- Reusable: The keyboard cleaning gel could be used repeatedly till the color turn to dark or it become sticky, then you have to replace the cleaning gel with a new one. After cleaning, please stock the cleaning gel in cool place. (Note: Don’t wash the gel in water.)
- In the package: 1 can of universal cleaning gel, we provide the cleaning gel with brand new, if you find the package broken, the cleaning gel dirty, or any other quality issues, please email us through message, we provide you new one soon
Step 4: Fix the Most Common Types of Crawl Errors (Implement Solutions)
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Based on the error type and your investigation with the URL Inspection tool, here’s how to fix the most common issues:
Shopping ad
Sale
HOTO Pocket-Size Laser Measuring Tool, EDC Gadget Birthday Gift for Men Dad
- Award-Winning Compact Design & EDC-Ready Gift Choice: Weighing only 0.09 lb and sized like a credit card, this compact laser measure is designed for everyday carry. It slips easily into a pocket, tool pouch, or bag, and can attach to a keychain for quick access wherever you go. Its minimalist design, premium tactile finish, and practical one-button measuring make it a useful EDC gadget for DIYers, real estate agents, homeowners, and tech enthusiasts. A thoughtful gift for men on any occasion
- One-Button Easy Measuring, Simple to Use: Designed with simple one-button operation, this compact laser tape measure makes quick measuring easy without complicated controls. Just press to measure room dimensions, furniture spacing, window height, wall décor placement, and everyday distances around the home. Ideal for users who want a smart, pocket-size measuring tool that fits naturally into an everyday carry (EDC) setup and feels intuitive, modern, and easy to use
- Fast & Accurate Indoor Measurements with Class 2 Laser: Measure distances from 0.16 ft to 98 ft with up to ±1/16 in / ±2 mm accuracy. With quick measurement response in about 0.2 seconds, HOTO helps you check spaces efficiently for home renovation, furniture layout, moving, decorating, craft projects, and DIY planning. Built with a Class 2 laser for everyday indoor measuring; use as directed and avoid direct eye exposure
- Low-Power OLED Display & USB-C Rechargeable Convenience: The low-power OLED display provides clear indoor readings while helping reduce battery drain. With USB-C rechargeable design, auto shut-off, and up to 1000 measurements per charge, this digital laser measure is built for repeated daily use without frequent battery replacement. Compact enough to keep in a drawer, toolbox, bag, or pocket
- Useful for Home, Work & Everyday Projects: From home renovation and furniture measuring to room planning, real estate checks, interior design, and light construction projects, this pocket-size laser distance measure is made for practical everyday use. Compact enough to keep in a drawer, toolbox, bag, or pocket, it is a stylish measuring tool that feels just as giftable as it is useful
- Fixing 404 “Not found” Errors:
- Diagnosis: GSC reports 404, URL Inspection confirms it. The URL doesn’t load in a browser. Find out where the bad link is coming from (GSC might indicate “Referring page”).
- Solution A (Broken Link): If the page exists but the link is wrong, or if the page is gone but other pages link to it, the fix is to find the links pointing to the incorrect or missing page and update them to the correct URL or remove them. Check internal links on your own site first. If the link is external, contact the other website owner if possible.
- Solution B (Page Moved or Deleted with Replacement): If the content has been moved to a new URL or replaced by a similar, relevant page, implement a 301 Redirect (permanent redirect) from the old 404 URL to the new relevant URL. This tells Google and users where the page moved.
- Solution C (Page Permanently Gone with No Replacement): If the page is removed and there is no relevant replacement page, simply let the 404 error stand. Return a proper 404 status code (which web servers do by default if a file isn’t found). Google will eventually stop trying to crawl it. Make sure you fix any internal links on your site that point to this old 404 page (Solution A). Do NOT redirect all 404 pages to your website’s homepage, as this is bad for user experience and can be seen as a soft 404 by Google if the homepage isn’t truly a replacement for the missing content.
- Fixing Server Error (5xx) Errors:
- Diagnosis: GSC reports 5xx, URL Inspection confirms a server error. Your website is inaccessible due to a problem on the server side.
- Solution: This requires investigating your web hosting environment. Check your server logs for error messages. See if your hosting plan has resource limits you might be hitting (like CPU or memory). If you’re unsure, contact your web hosting provider immediately. They can help diagnose and fix server-side issues.
- Fixing Soft 404 Errors:
- Diagnosis: GSC reports Soft 404, URL Inspection confirms the page returns a 200 status code but looks empty or like an error page.
- Solution: If the page should have content, add unique, substantial content to it. If the page should not exist or is intentionally empty (like an old product page that’s gone), ensure it returns a proper 404 Not Found or 410 Gone status code instead of 200 OK, or implement a 301 Redirect to a relevant category or product page. The goal is to accurately signal to Google that the original content isn’t there.
- Fixing “Blocked by robots.txt” Errors:
- Diagnosis: GSC reports this, URL Inspection confirms robots.txt is disallowing the page. Your robots.txt file is telling Googlebot not to crawl this URL.
- Solution: Access your website’s robots.txt file (usually at yourwebsite.com/robots.txt). Find the Disallow rule that is blocking the specific URL or directory. Edit or remove that specific Disallow rule to allow Googlebot access. Be very careful when editing robots.txt, as mistakes can accidentally block your entire site. Note: robots.txt only prevents crawling. If the page is linked from elsewhere, Google might still index the URL without crawling the content. To prevent both crawling AND indexing, use a noindex meta tag on the page itself.
- Fixing “Blocked by authorization request (401)” Errors:
- Diagnosis: GSC reports this, the page requires a login.
- Solution: If the page should be publicly accessible, remove the login requirement for that specific URL. If the page is intentionally behind a login (e.g., a user profile page), and this URL is not linked from public pages in a way that suggests it should be crawlable, then this is often an expected error for Googlebot and might not require action unless it’s hindering access to public content linked from behind the login.
- Fixing Redirect Errors:
- Diagnosis: GSC reports redirect error, URL Inspection shows problems following redirects (loop, chain too long, etc.).
- Solution: Trace the path of the redirect chain for the problematic URL. Use online redirect checker tools if needed. Identify where the chain is broken or causing a loop (e.g., URL A redirects to URL B, which redirects back to URL A; or URL A redirects to URL B, which redirects to URL C, which is a 404). Correct the redirects to eliminate loops, shorten long chains (ideally one 301 redirect directly to the final destination), and ensure the final destination URL is valid and loads correctly.
Step 5: Validate Your Fix
Once you’ve implemented the fixes for an error type, tell Google Search Console to re-check the affected URLs.
- Go back to the Pages report in Google Search Console.
- Click on the specific error type for which you’ve made fixes (e.g., “Not found (404)”).
- Click the Validate Fix button.
- Search Console will initiate a validation process. It will test a sample of the affected URLs to see if the error is resolved. This process can take anywhere from a few days to several weeks, depending on the number of URLs and Google’s crawling speed.
- Monitor the Report: Check back on the Pages report periodically. The number of URLs affected by the error should decrease as Google re-crawls and finds the issues are resolved. Search Console will notify you if the validation is successful or if some URLs still show the error.
Proactive Measures: Preventing Future Errors
- Regularly Check Internal Links: Use a website crawler tool to scan your site for broken internal links.
- Maintain Your Sitemap: Ensure your XML sitemap only includes valid, existing URLs you want indexed. Update it regularly.
- Test Changes: When moving or deleting pages, always implement proper 301 redirects. Test them after deployment.
- Monitor Search Console: Check your Pages report regularly (at least weekly) to catch new errors early.
Fixing crawl errors is ongoing website maintenance. It’s a bit like being a detective for your website, identifying problems and implementing solutions. By using Google Search Console and following these steps, you can keep your site accessible to Googlebot, improve its SEO health, and ensure a better experience for your visitors. Seeing that error count go down after your fixes is genuinely satisfying!
More shopping ads
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Special offer. See more information about Outbyte and uninstall instructions. Please review EULA and Privacy policy.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.
