If you want to find out (or know more) about the mysterious robots.txt and web robots then The Web Robots Pages will definitely be of help. The author Martijn Koster has generously written a huge F…