Login to continue
Don't have an account? Register here.
Supporting Small Businesses During COVID-19 Learn more
Facebook
ADD DIRECTIVE
May be your friends will also enjoy this?
Instruct search engines robots on how to crawl. LEARN MORE
How has your experience been with ROBOTS.TXT GENERATOR so far?
Any specifics on how you find the services and room for improvement?
Title:
Comment:
TOOL REVIEW
3.6
9
ABOUT Robots.txt Generator
Before we explain in details about Robots.txt generator tool, let’s brief in few lines about Robots.txt file.
Robots.txt is the first file that is being crawled by Search Engine spiders as it looks for instructions wrt pages that are blocked. Robots.txt can be used to clock a particular Search engine Spider from crawling a particular page or it can also be used to block the entire page from getting crawled by all spiders.
Robots.txt can block a particular page, folder or a bunch of pages.
LXR’s Robots.txt Generator tool makes the task a notch easier.
KEY FEATURES
Provides effective feature to instruct whether to crawl or not to crawl a particular page.
Allows you to select the Robots/spider/bots that you would like to block from crawling the page.
Allows you to also provide sitemap link information which is also a key component to be added in robots.txt page.
Most importantly, it helps you generator error free Robots.txt file that you can update in website root.
The usage of Robots.txt Generator Tool is simple enough. Here are the steps to be followed to create an effective Robots.txt file.
Provide the URL of your website.
Select whether you would like to allow or disallow certain sets of URLs.
Select the Search Engine bots you would like to allow or disallow.
Provide the URL of the specific page/folder that you would like to block.
You can use the (+) sign and add more number of entries in a similar way.
Add the URL destination of the sitemap.
Click on the button ‘Generate Robots.txt’.
*Please note that while you mention the URL you would like to block, provide only the suffix part of the URL after the domain name and the (/) sign.
Our Google certified experts are ready to answer any questions you might have about your website or digital marketing campaign. How can we help you?
RECOMMENDED Tools for Businesses
(OR)