TOOL REVIEW

3.6

 

9

ABOUT Robots.txt Generator

Before we explain in details about Robots.txt generator tool, let’s brief in few lines about Robots.txt file.

Robots.txt is the first file that is being crawled by Search Engine spiders as it looks for instructions wrt pages that are blocked. Robots.txt can be used to clock a particular Search engine Spider from crawling a particular page or it can also be used to block the entire page from getting crawled by all spiders.

Robots.txt can block a particular page, folder or a bunch of pages.

LXR’s Robots.txt Generator tool makes the task a notch easier.

KEY FEATURES

• Provides effective feature to instruct whether to crawl or not to crawl a particular page.

• Allows you to select the Robots/spider/bots that you would like to block from crawling the page.

• Allows you to also provide sitemap link information which is also a key component to be added in robots.txt page.

• Most importantly, it helps you generator error free Robots.txt file that you can update in website root.

HOW TO USE THE TOOL

---

01 Enter the details

Enter the URL of the website or page you want to get data for.

Get a comprehensive analysis for the URL you entered, plus much more!

02 Steps to follow

• The usage of Robots.txt Generator Tool is simple enough. Here are the steps to be followed to create an effective Robots.txt file.

• Provide the URL of your website.

• Select whether you would like to allow or disallow certain sets of URLs.

• Select the Search Engine bots you would like to allow or disallow.

• Provide the URL of the specific page/folder that you would like to block.

• You can use the (+) sign and add more number of entries in a similar way.

• Add the URL destination of the sitemap.

• Click on the button ‘Generate Robots.txt’.

• *Please note that while you mention the URL you would like to block, provide only the suffix part of the URL after the domain name and the (/) sign.