/resources/images/tool-icons/RGT.png

ROBOTS.TXT GENERATOR

Control how search engines crawl your website.

ADD DIRECTIVE

GENERATED ROBOTS.txt

Instruct search engines robots on how to crawl. LEARN MORE

Get generated code:

TOOL REVIEW


3.6

 

9


ABOUT Robots.txt Generator

Before we explain in details about Robots.txt generator tool, let’s brief in few lines about Robots.txt file.

Robots.txt is the first file that is being crawled by Search Engine spiders as it looks for instructions wrt pages that are blocked. Robots.txt can be used to clock a particular Search engine Spider from crawling a particular page or it can also be used to block the entire page from getting crawled by all spiders.

Robots.txt can block a particular page, folder or a bunch of pages.

LXR’s Robots.txt Generator tool makes the task a notch easier.

KEY FEATURES


  • Provides effective feature to instruct whether to crawl or not to crawl a particular page.

  • Allows you to select the Robots/spider/bots that you would like to block from crawling the page.

  • Allows you to also provide sitemap link information which is also a key component to be added in robots.txt page.

  • Most importantly, it helps you generator error free Robots.txt file that you can update in website root.


01 Enter the details
How to use tool image
Enter the URL of the website or page you want to get data for.

Get a comprehensive analysis for the URL you entered, plus much more!
02 Steps to follow
  • The usage of Robots.txt Generator Tool is simple enough. Here are the steps to be followed to create an effective Robots.txt file.

  • Provide the URL of your website.

  • Select whether you would like to allow or disallow certain sets of URLs.

  • Select the Search Engine bots you would like to allow or disallow.

  • Provide the URL of the specific page/folder that you would like to block.

  • You can use the (+) sign and add more number of entries in a similar way.

  • Add the URL destination of the sitemap.

  • Click on the button ‘Generate Robots.txt’.

  • *Please note that while you mention the URL you would like to block, provide only the suffix part of the URL after the domain name and the (/) sign.


Once the Robots.txt file is generated, submit it at the root directory of your website.
As iterated, Robots.txt file is the first file visited by Search Engine Robots before it crawls the entire website. Hence, it is important to provide the URL destination of the sitemap so that Search Engine spiders can access it quickly.
Ask The Expert

Our Google certified experts are ready to answer any questions you might have about your website or digital marketing campaign. How can we help you?