Login to continue
Don't have an account? Register here.
Robots.txt file is a text file for restricting bots (robots and search engine crawlers) from a website or certain pages on the website.
Did you know that text files are very powerful and a single text file could destroy your entire website? That error is damaging and can happen pretty easily.
You can specify in your Robots.txt file that you don't want the search engines to be able to access your thank you page, payment gateway page, add to cart, etc.
You can add your sitemap in Robots.txt file. This tells search engines to crawl your sitemap and index the URLs in the sitemap itself.
This is the default format for Robots.txt with SitemapUser-agent: *
This is the default format for Robots.txt with multiple SitemapsUser-agent: *
Don't worry. We won't let you or your website suffer. The best thing to do to avoid this issue is to use our FREE robots.txt generator. Our tool was completely designed to generate the proper robots.txt file for your website.
Instruct search engines robots on how to crawl. LEARN MORE
Before we explain in details about Robots.txt generator tool, letâs brief in few lines about Robots.txt file.
Robots.txt is the first file that is being crawled by Search Engine spiders as it looks for instructions wrt pages that are blocked. Robots.txt can be used to clock a particular Search engine Spider from crawling a particular page or it can also be used to block the entire page from getting crawled by all spiders.
Robots.txt can block a particular page, folder or a bunch of pages.
LXRâs Robots.txt Generator tool makes the task a notch easier.
Provides effective feature to instruct whether to crawl or not to crawl a particular page.
Allows you to select the Robots/spider/bots that you would like to block from crawling the page.
Allows you to also provide sitemap link information which is also a key component to be added in robots.txt page.
Most importantly, it helps you generator error free Robots.txt file that you can update in website root.
3 Independence Way, Suite #203,
Princeton, NJ 08540