How does Yahoo/Google find unknown domains?

[Sten Carlsen]

Did you consider robots.txt? Well behaved spiders should respect that,
although it does not prevent anything.

I would expect google and yahoo to respect it, I have no clue about he.net.

On 08/11/10 2:02, Michelle Konzack wrote:
> Hello experts and *,
>
> I have (since several years) collected some domain names  which  do  not
> exist (since years) and registered it  in  the  last  4  month  for  the
> internal use of my Internet Service.
>
> Now I see Googlebot, Yahoo and  <he.net>  quering  my  DNS  Servers  for
> exactly those domains.
>
> If I read the conditions of Networksolutions and Co, spidering of  WHOIS
> records is prohibited also the commercial use of the data.
>
> Does someone have an experience with his crap?
>
> Unfortunately I can not deny access to the 180 servers and Google, Yahoo
> and He is bombing my network with  to  much  useless  requests.  I  have
> written a mail to Google not to attack  my  network  of  VOIP  and  IPTV
> servers, but they continue...
>
> The webservers have only an SHTTP administrativ VHost, but not <exp.com>
> or <www.exp.com> but the webserver get  any  requests  from  <*.exp.com>
> because it is an administrative VServer and the error logfile is per day
> VERY long.
>
> An htaccess does not work, because I have more then 800 VHosts  on  each
> server.
>
> Thanks, Greetings and nice Day/Evening
>     Michelle Konzack
>
>
>
> _______________________________________________
> bind-users mailing list
> bind-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users

-- 
Best regards

Sten Carlsen

No improvements come from shouting:

       "MALE BOVINE MANURE!!!" 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/bind-users/attachments/20101108/0e1dd0c5/attachment.html>