How does Yahoo/Google find unknown domains?

[Stephane Bortzmeyer]

On Mon, Nov 08, 2010 at 02:02:52AM +0100,
 Michelle Konzack <linux4michelle at tamay-dogan.net> wrote 
 a message of 94 lines which said:

> If I read the conditions of Networksolutions and Co, spidering of
> WHOIS records is prohibited also the commercial use of the data.

The list of domains in .COM (and other ICANN TLDs) can be obtained
legally:

http://www.verisign.com/domain-name-services/domain-information-center/tld-zone-access/index.html

> Now I see Googlebot, Yahoo and <he.net> quering my DNS Servers for
> exactly those domains.

Hurricane Electric uses these zone files for very interesting and
useful statistics:

http://bgp.he.net/ipv6-progress-report.cgi