What is a web crawler? - Calibre9

Web Crawler


What is a web crawler?

A web crawler (also known as a spider) are bots that are used to discover and read web pages. Google uses multiple versions of their crawler “GoogleBot” to find and index pages. If your content has not been discovered by GoogleBot, it will not appear in search results. Websites that are easy to crawl will consistently rank better than sites that are more difficult to discover.



↓ Scroll to See Frequently Asked Questions ↓

Can web crawlers access all types of web pages?

Google’s web crawler can read most types of pages. It does however, render and process JavaScript separately to HTML, and usually at a much slower rate. For this reason, it is recommended that you do not use JavaScript to load important internal links on your website. Excessive JavaScript can also slow down your site, causing further drops in rankings.

How can I optimise my website for web crawlers?

Crawlers rely on links to make their way around your website. You can optimise your site for crawlers in much the same way that you do for humans. Make sure that all pages are linked into the site structure and that the most important pages are the most linked (usually via the header and/or footer). It is also a good idea to make sure your internal linking is rendered in the page’s HTML and not loaded later via JavaScript.

How can I know if a web crawler has visited my website?

The easiest way to know if Google’s crawler has visited your site is via Google Search Console. In the Search Console, you can see all of the pages that Google’s crawler has discovered and whether it has decided to index them or not. For non-indexed pages you can see what issues are currently preventing the page from being indexed.

Can I ask Google to crawl my site?

Yes. In Google Search Console you can submit individual pages to be crawled. This is a valuable tool for times that GoogleBot has failed to discover important pages on your site.



idea-white Let’s Get Started

Why Calibre9?

We help you focus on what truly matters
Heart in Hand Icon
Easy to Access Support

Based in Collingwood, our expert team is happy to chat and answer any questions you might have.

A Human-Centred Approach

We value the working relationships we have with our clients above all else.

Shaking Hands Icon
An SEO Agency You Can Trust

We have extensive experience and innovative strategies, but the bottom line is: we deliver.

Graph Icon
Data Driven, Results Focused

Everything we do is founded on data and we work relentlessly to get the best ROI possible.

Make an Enquiry

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.