How does Yahoo/Google find unknown domains?
Sten Carlsen
stenc at s-carlsen.dk
Mon Nov 8 01:32:14 UTC 2010
Did you consider robots.txt? Well behaved spiders should respect that,
although it does not prevent anything.
I would expect google and yahoo to respect it, I have no clue about he.net.
On 08/11/10 2:02, Michelle Konzack wrote:
> Hello experts and *,
>
> I have (since several years) collected some domain names which do not
> exist (since years) and registered it in the last 4 month for the
> internal use of my Internet Service.
>
> Now I see Googlebot, Yahoo and <he.net> quering my DNS Servers for
> exactly those domains.
>
> If I read the conditions of Networksolutions and Co, spidering of WHOIS
> records is prohibited also the commercial use of the data.
>
> Does someone have an experience with his crap?
>
> Unfortunately I can not deny access to the 180 servers and Google, Yahoo
> and He is bombing my network with to much useless requests. I have
> written a mail to Google not to attack my network of VOIP and IPTV
> servers, but they continue...
>
> The webservers have only an SHTTP administrativ VHost, but not <exp.com>
> or <www.exp.com> but the webserver get any requests from <*.exp.com>
> because it is an administrative VServer and the error logfile is per day
> VERY long.
>
> An htaccess does not work, because I have more then 800 VHosts on each
> server.
>
> Thanks, Greetings and nice Day/Evening
> Michelle Konzack
>
>
>
> _______________________________________________
> bind-users mailing list
> bind-users at lists.isc.org
> https://lists.isc.org/mailman/listinfo/bind-users
--
Best regards
Sten Carlsen
No improvements come from shouting:
"MALE BOVINE MANURE!!!"
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/bind-users/attachments/20101108/0e1dd0c5/attachment.html>
More information about the bind-users
mailing list