How does Yahoo/Google find unknown domains?
Stephane Bortzmeyer
bortzmeyer at nic.fr
Wed Nov 10 01:14:36 UTC 2010
On Wed, Nov 10, 2010 at 01:47:44AM +0100,
Michelle Konzack <linux4michelle at tamay-dogan.net> wrote
a message of 115 lines which said:
> Even my simple squirrelmal login page from <webmail.tamay-dogan.net>
> is spidered daily with more then 800 hits and I have already counted
> more then 80 different searchbots.
HTTP spidering issues are clearly off-topic for this list, which deals
with the BIND DNS server. I suggest to ask on ServerFault or a similar
forum.
> The searchbots are creating 17,3 TByte traffic per month which my
> customers have to pay to!
There is no other solution than robots.txt.
More information about the bind-users
mailing list