Centralized DNS Caching for SMTP Servers? (was Re: What kind of hardware?)

[Kevin Darcy]

Brad Knowles wrote:

> At 6:44 PM -0500 3/8/01, Kevin Darcy wrote:
>
> >  I think you misunderstood. I'm not denying that mail servers should run local
> >  caching nameservers. Obviously they should. I was taking issue only with the
> >  *forwarding* part. If every mail server runs an *autonomous* caching server,
> >  this would actually alleviate the load on those central caching servers,
> >  would it not?
>
>         At the expense of providing consistent answers, and generating
> unnecessary network traffic, and unnecessarily increasing the load on
> the remote nameservers, and unnecessarily delaying mail while you
> wait for serverN+1 to resolve a domain that has already been resolved
> by serverN and is already cached locally on the central caching
> nameservers, yes.

Let's remember, however, that these WAN transactions only occur for
*uncached* names, so cache hit ratio figures very prominently in this equation.
Some names have high cache hit ratios, either because they are "popular", i.e.
frequently queried, or simply because they have large TTL values. These will tend
to stick around in any given cache *regardless* of whether the cache is centralized
or not. At the other end of the spectrum, names with low cache hit ratios (because
they are "unpopular" or have short TTL values), will tend to be *absent* from
caches, centralized or de-centralized, so centralization doesn't make much
difference here either. That leaves just the *middle* stratum -- names with
*moderate* cache hit ratios.

But the delays to go fetch that data tend to be measured in milliseconds, which
isn't noticeable in the context of mail delivery. I'd rather spend that few
milliseconds to increase my chances of a successful delivery.


> >  Okay, propagation delay, I understand that perfectly. But how many folks run
> >  non-NOTIFY-capable nameservers these days, really? These propagation problems
> >  should eventually go away, or at least subside to "noise" level.
>
>         How the heck does NOTIFY figure in at all when you're talking
> about caching nameservers and caching forwarding nameservers?!?

Get a grip. I was addressing your previous point about master/slave propagation
delays -- "if you have a case where some authoritative servers haven't picked up
the latest changes". Obviously, NOTIFY figures prominently in that equation.

> >  Hmmm... I thought "the" lesson behind all cache poisoning attacks was "don't
> >  implicitly all of the RR's one sees in a DNS response packet". Of course,
> >  there could be multiple lessons, but I'm not sure that "consistency over
> >  occasional correctness" rates very high on the list, if it's on there at
> >  all...
>
>         Indeed, there are multiple lessons that can be learned from the
> cache poisoning attacks, and the need for consistency above
> occasional correctness is certainly a very important one.
>
> >  With all due respect, if one mailbox is getting thousands of complaints a
> >  day from a userbase of 20 million people, then a) that's actually not a
> >  very high percentage, and b) sounds like an organizational problem --
> >  isn't that what low-level tech-support helpdesks largely exist for, i.e.
> >  to filter out "routine" (non-)problems so that people with clues can get
> >  their work done?
>
>         That was after having complaints being filtered by at least three
> levels below me.  And I was still getting hundreds of complaints a
> day.  Just try to imagine how many thousands or hundreds of thousands
> of complaints were actually being registered each day, in order to
> result in hundreds of these complaints per day filtering down to my
> level.

I don't want to tell you how to run your business, but it seems to me that, no
matter how you slice it, if the complaints are bogus, i.e. some messages went
through, but not others, or messages were received out of order, solely due to
problems with *other*people's* DNS, then the bogosity of those complaints should
have been recognized and disposed of by 3 levels of Help Desk.

Then again, good help is so hard to find these days... :-)

I'd hate to think that such a fundamental feature of your DNS architecture was
prompted by something as mundane as lack of training/education of support staff or
simple intra-organizational miscommunication.

By the way, don't you give your Help Desks *tools* so that they can easily diagnose
common problems on their own?

> >  Maybe it's just a difference in userbase and/or requirements. I assume this
> >  is some ISP or ISP-like entity serving these 20 million people, where email
> >  *is* one of the main reasons, if not *the* reason, for the business
> >  relationship between the provider and the end-user in the first place.
>
>         E-mail is the *only* mission-critical application.  If the user
> can't get to the web, or irc, or ftp, or anything else, they'll
> complain but they'll come back.  If they can't get to their e-mail,
> then they are permanently lost.
>
> >  When your main business is something *other* than providing Internet
> >  services, however, e.g. building cars, then often it's far more important
> >  that the mail *gets*through*, even if it has to be re-sent over and over,
> >  than just the *appearance* that everything is working smoothly.
>
>         So long as the user doesn't have to repeatedly re-transmit that
> message, you are correct.  The moment the user has to get involved
> because your system is "screwed up" and doesn't have a consistent
> picture of the world across all machines, you lose.

In your world, perhaps this is a "loss". In my world, it is only a temporary
inconvenience. We have a formal policy stating that Internet email is inherently
unreliable and that for time-sensitive and mission-critical information,
*multiple* communication methods should be used, not just email. Yes, I know that
sounds like just a big CYA, but in this particular case, it happens to be *true* --
Internet email *is* inherently unreliable. We say that up front to our users, and
most of them understand and accept it. If they don't, then, frankly, too bad for
them. Now, if one is in the business of *selling* Internet services, from a
marketing standpoint, I can understand why one wouldn't want to admit the inherent
unreliability of Internet email delivery. As I said, different userbases, different
objectives, different requirements. For us, the Internet is just one tool among
many that we use in our business. It's not the be-all, end-all *of* our business.
Old economy, new economy, all that stuff...

(This is not to deny that we are becoming increasingly dependent on email for doing
business. But we are unlikely to change our policy in the foreseeable future with
respect to Internet email. If we _were_ to amend the policy, it would probably only
be with respect to closed, better-controlled business-exchange networks like ANX,
the whole _raison_d'etre_ of which is to provide greater reliablity for
mission-critical functions).

Moreover, if an expensive fiasco occurs because of email failure, not only can we
point our users to the aforementioned policy, but when and if that does not
suffice, it becomes *very* important whether we are to blame or the external entity
is at fault. If the failure occurred because of inconsistency in *their* DNS data,
and we exercised due diligence in the face of that inconsistency, then perhaps
that's *all* that matters, in the eyes of upper management. The consistency or
inconsistency of our mail servers' delivery behavior, in response to that
DNS inconsistency, matters not a whit, only that we tried our best and that the
other guy blew it. That's the brutal *political* reality under which we operate.

> >                                                                   If an
> >  email miscommunication with one of our parts suppliers results in
> >  the shipping of the wrong part and ultimately the idling of a production
> >  line (at millions of dollars a minute), then it will be little consolation
> >  to hear that, despite the fact that a good MX was available in DNS, at
> >  least the email servers ignored that MX and failed *consistently*.
>
>         If the orders for parts A & C go through and each cost the
> customer millions of dollars, but the order for part B doesn't go
> through, and the result is that all the money spent on parts A & C is
> wasted, and the entire company goes out of business, then that is
> also small consolation.

Sorry, you lost me there. Are parts A & C somehow *dependent* on part B??? It's not
clear to me how the delivery or non-delivery of one message affects the
consequences of any of the other messages...

It occurs to me, however, that where several email messages have dependencies
between each other, long-standing business conventions (which predate email), like
the "X of Y" convention, e.g. "part 3 of 7" can help in the detection of missing or
delayed messages. If someone receives the orders for parts A & C (to use your
example), and it is *obvious* to them that there is a missing order for part B,
then presumably they'll call the sender and alert them to that fact. Under your
(centralized cache) scenario, however, maybe the recipient doesn't get *any* of the
orders. Perhaps they might draw the conclusion, incorrectly, that they lost the bid
for the entire project. So, sometimes partial results are *better* than the
black-or-white alternatives of "everything succeeds" or "everything fails". It all
depends on context.

>         It would be much better for the customer to find out ASAP that
> the order for none of the parts went through, and they need to make
> alternative arrangements in order to get their production line back
> into operation.

But in most cases, given the nature of email, these kinds of data inconsistencies
cause *delays*, rather than immediate failures, don't they? E.g. MX points to a
dead address, MX points to a machine that's not running SMTP, etc. In the majority
of cases, I assume, the mail *does* eventually get through, the only question is
whether it gets through in a timely fashion. In our business, anything that's
important and time-sensitive gets a phone confirmation. If not confirmed in a
timely fashion, then, yes, the sender will send the message again. At worst case,
our mail software sends a "4-hour warning" message; *then* they'll know that
something went wrong, and re-send the message.

So, basically I think *notification* is a red herring here: users get notification
for failed delivery attempts *regardless* of whether the cache is centralized or
not. The real bone of contention is whether to take an "all or
nothing" (centralized cache) or a "do as much as you can" (de-centralized
cache) approach.

> >                                                                      We
> >  operate on a "get the mail there no matter what" basis, rather than on a
> >  "hide the inherent inconsistencies of the Internet from the users so it
> >  won't confuse and/or annoy them" basis. I'm not saying one is inherently
> >  more "valid" or "legitimate" than the other; just that they are driven by
> >  different needs.
>
>         Note that returning inconsistent answers makes it virtually
> impossible to do any successful debugging of the problem.  This
> increases the costs on customer service, on the network operations
> center, on the third-level support personnel, and on the systems
> operations personnel.

Please clarify what you mean by "returning inconsistent answers". With a
centralized cache, either all of the mail goes through normally, or all of the mail
fails or gets delayed. With de-centralized caches, some of the mail may go through,
and other mail may fail or get delayed. Are you equating "returning inconsistent
answers" with "some mail goes through right away and some mail doesn't"? Or are you
talking about the *DNS* answers themselves being inconsistent? It's the
authoritative nameservers for the domain that are giving the inconsistent
*DNS* answers. These are controlled, by and large, by external entities, and
therefore difficult or impossible to control. Actually, in our situation, since we
*are*, by and large, the *customer* of those other entities, we wield great power
in getting them to fix whatever is wrong with their DNS. In your case, I can
understand that, as a "third party" to the email transaction, i.e. neither the
sender nor the recipient, just a conduit, maybe you're somewhat less empowered, and
therefore have to accommodate other people's bogosities, instead of being able to
force them to fix their problems. In the course of my career in DNS and
SMTP infrastructure, I've probably precipitated more major corrections to
*other*people's* DNS setups than I have to that of my own organization.

>         Indeed, consistency is one of the six basic types of security, as
> listed on page 11 of _Practical Unix Security_ by Simson Garfinkel &
> Gene Spafford.

It's been a while since I've read that -- I don't remember what G&S had to say,
specifically, about inconsistency. I'll review it. But since DNS cache data is
inherently ephemeral anyway, and, as I pointed out earlier, all you're really doing
is affecting the *granularity* of the inconsistency, it's difficult to imagine how
what we're discussing even makes it onto the radar screen of "Unix Security"...

> >  To give you another example of this _modus_operandi_: if the quality of
> >  Internet DNS administration keeps dropping like it has been, I may have
> >  to even start *restarting* the nameservers periodically on my mail
> >  gateways, in order to flush out all the botched NS RRsets that I keep
> >  seeing in the cache.
>
>         There's enough garbage out there (fully 25% of all ccTLD zones
> that are carefully checked for consistency still have lame
> delegations, and more than 50% of all zones in .com have lame
> delegations), so I don't think that the use of a centralized set of
> caching nameservers to which all unknown queries are forwarded is
> going to materially impact this issue.

Lame servers aren't the main problem. As long as there is at least one reachable
non-lame server for the domain, resolution still works. More pernicious are
glue-record problems. These kinds of problems crop up, most frequently, because the
domain owners move control of the domain from a hosting provider to their in-house
servers, where the hosting provider's nameservers stay as slaves, but they forget
to update the domain delegations, and where they list only their in-house servers
in their in-zone NS records. Since the in-house servers are likely to have names
*in* the domain, glue-record problems -- or, more accurately,
absence-of-glue-record problems -- then ensue. Periodic restarts work around such
glue-record problems by effectively, albeit temporarily, replacing the faulty
in-zone NS records with the delegation NS records.

>         IMO, all caching nameservers on a busy network should probably
> get reloaded at least once a day, just to clean out the cruft.  It's
> sad, but this appears to be necessary.
>
>         Unfortunately, this problem isn't going to be solved until the
> registries start getting a lot tougher about validating the provided
> information, and periodically re-validating that information and
> cutting off the zones that are no longer compliant.

Agreed.

>         Indeed, since they could turn this into a money-making
> proposition (e.g., "fix your problem and pay us $$$, or we cut you
> off"), it would seem that this problem should resolve itself quite
> quickly, as soon as the registries wake up and smell the roses.

No offense to Cricket, _et_al_, but the registries still seem rather clueless to
me. This is merely *one* example of how that cluelessness manifests itself.

> >                        That, of course, would be a *terrible* waste of
> >  resources -- not only my resources, but those of the TLD servers and of
> >  all of the Internet nameservers for the domains with which we communicate.
> >  But my mandate is "get the mail through", and to accomplish that, I'll
> >  resort to draconian measures like periodic restarts, if necessary...
>
>         You're not using forwarding caching nameservers on your
> individual hosts, so what do you care about the load placed on the
> TLD nameservers?

I care from a civic/community perspective.

>         If you really cared about the unnecessary load placed on servers
> outside your network, or the unnecessary amounts of traffic generated
> on your own local network, or the unnecessary delays engendered by
> having each and every one of your servers have to go out to the
> outside world to look up all information, then you'd do exactly as I
> recommend.
>
>         Otherwise, I don't think you really have a leg to stand on when
> you talk about generating unnecessary load on the TLD nameservers.

I think you inverted my argument somewhere along the line. I thought I had made
clear that I really didn't *want* to have to do periodic restarts, expressly
*because*, _inter_alia_, it would put a load on the TLD nameservers. I was just
giving an extreme example of the lengths to which I would go, in order to fulfill
my mandate of "the mail must go through". Centralizing the cache, as you suggest,
however, would IMO be a step backwards in attempting to fulfill that mandate, for
the reasons I've presented.


- Kevin