# /robots.txt file User-agent: * Disallow: /images/* Disallow: /*.PDF$ Crawl-Delay: 360 User-agent: webcrawler Disallow: / User-agent: egothor Disallow: / User-agent: ShopWiki Disallow: / User-agent: lycra Disallow: / User-Agent: Googlebot-Image Disallow: / User-agent: Googlebot-Mobile Disallow: / User-agent: Googlebot Crawl-Delay: 360 User-agent: Slurp Crawl-Delay: 360 User-agent: msnbot Crawl-Delay: 360 User-agent: Speedy Crawl-Delay: 360