Login to continue
Don't have an account? Register here.
Continue with Google
Continue with LinkedIn
Continue With Email
By signing in, I agree to the
Privacy Policy and
Terms Of Service.
Change Password
Robots.txt Validator
4.4
1.
Enter the details
In order to fetch and assess the effectiveness of your site’s Robots.txt tool, all you need to do is provide your site’s homepage URL. The tool would instantaneously fetch the existing Robots.txt file and share the findings.
Alternatively, you may simply access the Robots.txt file of your website; you can simply copy and paste the full details in the input field.
In either case, just press the ’Validate’ button. The tool would highlight all syntax and logical errors that you may like to fix.
In the same input field you can rectify the possible errors. Furthermore, if in case you would like to omit any URL from getting crawled and/or would like to allow certain file to be crawled from a blocked directory, you can do so with the help of the tool and in the process prepare the final file that you would like to update in the root directory of your site.
2.
Result
As mentioned above, the tool allows you to make all necessary amendments in the file -
you may copy the final changes and paste it to your original Robots file before updating the same at the root directory.
Tool FAQs
What kind of errors get spotted in Robots.txt validator file?
LXR's Robots.txt file validator will show all errors and warnings in the file and also suggest amendments thereof. This includes syntax, misspellings as well as logical errors
Login to continue
Don't have an account? Register here.
LOCATIONS
Global Headquarters
3 Independence Way, Suite #203,
Princeton, NJ 08540
Call: 609.356.5112
Asia Pacific Office
601, Jain Sadhguru Capital Park,
Image Garden Road,
Madhapur, Hyderabad-500081,
Telangana, India
Ask The Expert