# WWW Robots (also called wanderers or spiders) are programs that traverse # many pages in the World Wide Web by recursively retrieving linked pages. # This file prevents them from accessing your data. User-agent: * Disallow: /cs/ Disallow: /bt/ Disallow: /hd/