Bandwidth is relative to the amount of visitors (traffic) you have to your site. 99% of web sites currently on the web use no more than 5 GB of bandwidth a month. It’s only sites that are expected to produce a large amount of traffic that will require a larger rate of monthly bandwidth, like adult sites, […]
The sad reality is that most webmasters have no idea what a robots.txt file is. A robot in this sense is a “spider”. It’s what search engines use to crawl and index sites on the internet.
Basically a spider will crawl a site and index all the pages (that are allowed to) of that site. Once that’s complete the robot will then move on to external links and continue it’s indexing. This is how search engines find other sites and build such a large index of sites. They depend on other sites to link to relevant websites, which will link to others and so on. Read more