How does Yahoo/Google find unknown domains?

[Lyle Giese]

Michelle Konzack wrote:
> Hello Robert Spangler,
>
> Am 2010-11-09 10:34:52, hacktest Du folgendes herunter:
>   
>> If these domains are for internal use only, why did you list the DNS servers 
>> for them?  You are aware that you can register a domain without listing a DNS 
>> Server?
>>     
>
> Because my own customers (exclusively) must access it.  They are my VOIP
> and IPTV servers and there is no public HTTP content  but  I  am  bombed
> with PHP/CGI requests and I do not know, where Google and Co have gotten
> those links.
>   
Are you sure it's all search bots and not script kiddies playing? 
Having a valid index.html in place that redirects to your business home
page may help.  But the search bots don't normally search around for
php/cgi scripts.  That however is typical of the script kiddies looking
for a server to hack.
> Some of the searchbots are hiting my servers  3  times  in  series  from
> three different IPs and in summary, I have more then  10.000  searchbot-
> entries per day in my Logs.
>
> My server always return an Error-Page from the VServer that there is  no
> configured HTTP host on the machine but it is ignored.  One  VHost  must
> be configured for the web administration and it is hit too and too much!
>
> Even my simple squirrelmal login page from <webmail.tamay-dogan.net>  is
> spidered daily with more then 800 hits and I have already  counted  more
> then 80 different searchbots.
>
> How braindamaged are Searchbot-Programmers?
>
> All of my webservers together have arround 86 TByte of content including
> a VERY huge debian archive (all releases and versions from 0.96 to  now)
> and my traffic per month is arround 27 TByte.
>
> The searchbots  are  creating 17,3 TByte  traffic  per  month  which  my
> customers have to pay to!
>
> Maybe I call my lawer to write letters to the serachbot owners  to  stop
> spidering my 36 domains.
>
> Oh, at Level3 in Frankfurt I pay 12 Euro/Mbit traffic  per  month  which
> mean 12 Euro per 320 GByte traffic. Not counting the price for the 700km
> FiberOptic line which is another provider (0,40 euro/m/year).
>
> I had a 1 GE line from Frankfurt but du to the excessiv serchbot traffic
> it broke several times per day.  Now I have in total twelf 1GE  (Level3,
> Verizon, DTag and  Orange).  Maximal  I  can  have  64 x 10 GE  with  my
> Transmode TS System but then I can install my own BPOP.
>
> Thanks, Greetings and nice Day/Evening
>     Michelle Konzack
>
>   
Despite how I feel about Yahoo's SLURP engine, it still honors
robots.txt.  Script kiddies don't.

Lyle Giese
LCR Computer Services, Inc.


P.S.  My last post on this.  This is not DNS related.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.isc.org/pipermail/bind-users/attachments/20101109/91c94d68/attachment.html>