# Robots.txt file # # All robots will spider the domain User-agent: * # use Disallow: /xxx/ for each directory Disallow: /admin/ Disallow: /css/ Disallow: /farmnewsletter/ Disallow: /images/ Disallow: /lib/ Disallow: /pdf/ Disallow: /staff/ Disallow: /Templates/ Disallow: /test/ Disallow: /testsite/