I know a thing or two about robots.txt , there is a copy of robots.txt I have been using for most of my sites, can any one shed more light on robots.txt files?
----------example robots.txt file-----------------
# robots.txt for (site name should go here)
User-agent: *
Disallow:
-----------example ends---------------------
----------example robots.txt file-----------------
# robots.txt for (site name should go here)
User-agent: *
Disallow:
-----------example ends---------------------
