Robots.txt, also known as ‘robots exclusion protocol’, is a text file that contains instructions for crawling bots.

Robots.txt file important to let search engines like Google to know which pages to crawl (visit) and which pages not to crawl.

Search engine bot at first looks for a robots.txt file on the server and then it crawls other pages of the website. You can put your special instructions for search engines in this Robots.txt file.

Avatar for Muhammad Tamzid
Author

Muhammad Tamzid, the founder of WPTechnic, is a WordPress Enthusiast with a passion to help beginners learning WordPress. Also managing WPrevival, a 24/7 WordPress Website Development, Maintenance & Security Service company.

Write A Comment