"reasonable" bulk resolver behavior

[Paul Vixie]

Steve Friedl <steve at unixwiz.net> writes:

> I've written a bulk IP rDNS resolver program that runs through Apache
> logs and populates a DB file with IP->name lookups that are later used
> by webalizer for webserver logfile analysis. It uses the ADNS library to
> make async calls (in a single thread) to burn through reverse lookups as
> fast as the network can take it: there could be 200 outstanding queries
> at a time, for instance.
> 
> It strikes me that maxing out the nameserver is not terribly polite
> behavior, so there is a throttle mechanism: limit to <N> outstanding
> queries at a time, and pause a bit when we reach that number.
> 
> How does one select a proper throttle point for a nameserver? I have
> no idea when the query rate becomes "rude": can anybody offer some
> suggestions for what to look for when picking this number?

when i wrote res_nsend() and the "manyhosts" program to do this same job
for altavista back in their early days (see $BIND8/contrib/manyhosts/),
i found several limitations.  one was that the resolver's selection
criteria was "use the first recursive nameserver that works" and what i
wanted was "use them all".  thus the RES_OPTIONS "rotate" option was added.

altavista routinely ran "manyhosts" with 1000 concurrent threads and five
recursive nameservers (all of which were just virtual BIND8's on the same
host, at 127.0.0.2, 127.0.0.3, and so on), and never received complaints.

for the record, i still think that using a thread-safe resolver like IRS
(which is contained in bind8 and bind9's "libbind") and pthreads is the
right way to go, compared to ADNS.  though i'd still like to do something
that ran natively on top of bind8's "eventlib" someday.
-- 
Paul Vixie