What is a robots.txt file? - Calibre9


Robots.txt file

What is a robots.txt file?

A robots.txt file is a set of instructions that search engine crawlers must follow when navigating your website. They are usually used to tell the crawler where the sitemap is located and to prevent them from crawling or indexing sensitive pages like login URLs.



↓ SCROLL TO See Frequently Asked Questions ↓

How do I add a robots.txt file to my website?

Most CMSs will create a robots.txt file by default and will allow you to edit it within their settings. For sites built in WordPress, you may need to install a plug-in such as Yoast which will allow you to easily create and edit your robots.txt file.

What should I have in my robot.txt file?

At a minimum, your robots.txt file should contain a link to your sitemap. This will ensure that all of your pages are discovered for indexing by search engines. You should also consider disallowing crawlers from accessing any login pages on your site.



idea-white Let’s Get Started

Why Calibre9?

We help you focus on what truly matters
Heart in Hand Icon
Easy to Access Support

Based in Collingwood, our expert team is happy to chat and answer any questions you might have.

A Human-Centred Approach

We value the working relationships we have with our clients above all else.

Shaking Hands Icon
An SEO Agency You Can Trust

We have extensive experience and innovative strategies, but the bottom line is: we deliver.

Graph Icon
Data Driven, Results Focused

Everything we do is founded on data and we work relentlessly to get the best ROI possible.

Make an Enquiry

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Google Rating
Based on 125 reviews