What kind of hardware?

[Kevin Darcy]

Brad Knowles wrote:

> At 2:24 PM +0100 3/8/01, Eivind Olsen wrote:
>
>         Now, one thing I do on all mail servers I configure, is to have a
> set of high-speed centralized nameservers, but I also run a
> forwarding caching nameserver on each mail server.  Each machine will
> look first to itself and see if it has the necessary information
> already cached, and if so then you need not go any further.  If the
> information is not cached locally, the query will be forwarded to the
> central set of nameservers, which are more likely to have that
> information.
>
>         This also helps ensure that all local caching nameservers have
> the same "picture" of the DNS as everyone else -- either through the
> information cached locally, or through the central nameservers to
> which the queries may be forwarded.
>
>         Under no circumstances should you run caching-only nameservers on
> each mail server *without* a centralized set of caching nameservers
> to which unknown queries are forwarded, because the one thing users
> hate above all else is inconsistency -- if they just successfully
> sent mail to a particular address five minutes ago, they want to be
> able to successfully send mail to that address again.  Having mail
> work or not, depending on which server the mail may be routed
> through, is a very sure way to go out of business very quickly.

As some of you may know, I have a low opinion of forwarding. In my view, it's a
necessary evil in some situations to get around connectivity problems, e.g.
firewall boundaries, and in *some* network/DNS architectures conveys performance
benefits *some* of the time, but an evil nonetheless and something generally to be
avoided. So I'm naturally biased against your "preferred" setup to begin with.

Having disclosed that, I have to wonder, why would a non-forwarding caching server
get *inconsistent* results? You mean, because the administrators of the target
domain screw something up, so that some authoritative servers for their domain
give out different data than the others? Frankly, I don't see this as my problem
to fix (or work around); it's *their* problem to fix. And, besides, your
"solution" -- arranging for a central cache to be used so that the lookup results
are "consistent", at least for the lifetime of the cache entries -- might amplify
the negative effects of such an inconsistency. I mean, what if somebody's slave
DNS server is temporarily hacked, with the hacker attempting to intercept the
domain's mail? If you happen to cache the malicious MX record, now suddenly
*all* of the mail destined from your servers to that domain is going into the bad
guy's mailbox, perhaps long after the intrusion is detected and the genuine
MX record restored (obviously if the bad guy is any good, he'll set a high
TTL value on the malicious record to maximize the effect of the hijacking). And
it's not just a security issue -- what if an MX is changed and zone transfers are
broken to just *one* of the slaves? If your central caches happen to get the stale
record, your mail servers could all be trying to deliver mail "consistently" to a
dead address, until the MX record expires from your central cache and maybe, if
you're lucky, gets replaced with a good one. The bottom line is, if the
authoritative servers for a domain disagree on domain data, who knows which data
set is correct? Any attempt at *forcing* consistency of that data, could result in
your view of the domain data being consistently *wrong*, for some extended period
of time. You're betting the farm on a particular roll of the dice. I'd rather
limit the damage by allowing at least *some* of my mail servers to deliver the
mail to the right destination(s), even in the face of such data inconsistencies.

Now, I understand firsthand the organizational/psychological/political value of
consistent results, especially with respect to email delivery. We recently had a
case where a networking problem blocked SMTP delivery from just *one* of our
outbound firewalls to the mail servers for a particular domain, and there was
substantial, ongoing confusion over whether there was a problem or not, whether it
was fixed or still broken, etc., because *some* of the mail to that domain would
get through, and the rest would not, depending on which firewall happened to get
the message (we load-balance of course). But, due to the ephemeral nature of DNS
cache entries, data inconsistencies between other people's authoritative
nameservers are going to be temporary *anyway*, and any behavior based on that
data, e.g. mail-delivery behavior, commensurately inconsistent. As far as I can
tell, the only thing your scheme affects is the *granularity* of that
inconsistency, i.e. the data stays consistent for the lifetime of the cache
entries in the central cache, but then may change afterwards. I certainly don't
think that the benefits of a temporary, *artificial* consistency, which could
easily make "consistent" stale and/or malicious data, warrants invoking the evils
of DNS forwarding...


- Kevin