I would like to know that which things of website, an expert SEO must put into robots.txt file for disallow? Which pages is better to not show to Search Engines?
Robots.txt file
- Thread starter jesicawillss
- Start date
This is like asking what color of shirt your wear when going on a long drive. The answer will be different from everyone. Whatever you think should not be shown in search results are disallowed in robots. If someone do not want to show his profile page it should be disallowed. A general thumb rule is that Block the pages that you do not want to be crawled by search engine spiders. It may be a signup page, profile page, confidential doc files, excel files, pdf files, ppt files, or any other confidential html, php, aspex or any extension file. I personally prefer disallowing image folder and nothing else, but shutterstock will not do that. Everyone has different restrictions for the same or different files or folders. Just disallow whatever you think should be hidden from the world but is required for some specific group or individual.
Every page which doesn't seems to add value for visitors should be forbidden for crawling i.e disallow them in robots.txt. For example: admin pages in any cms, image directory, cgi-bin etc.
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do.
Robots.txt is a text file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means mandatory for search engines but generally search engines obey what they are asked not to do. It is important to clarify that robots.txt is not a way from preventing search engines from crawling your site
Similar threads
