stub versus forward

[Von Alt, William]

Okay all... here is a situation that has been most perplexing today...

Here at DOE HQ, we have the "standard" split DNS config with two private
nameservers (master and slave) and two public nameservers (master and
slave).  I have a remote field site that also has a split DNS setup.

We want our public nameservers left completely out of this picture... the
goal is for my internal nameserver (authoritative for doe.gov) domain to
delegate the em.doe.gov domain to his internal nameservers. So on my
internal primary server, I setup a stub zone for em.doe.gov and list the two
remote nameservers as masters.  Here is the relevant excerpt from
named.config:

options {
        directory "/etc/named";
        pid-file "/etc/named.pid";
        check-names master warn;
        auth-nxdomain no;
        query-source address 146.138.1.215 port 53;
        transfer-format many-answers;
        forwarders {
                205.254.144.110;
                205.254.143.110;
        };
        also-notify {
                146.138.198.215;
        };

};

zone "." {
        type hint;
        file "cache.named";
};

zone "0.0.127.in-addr.arpa" {
        type master;
        file "127.0.0.db";
};

zone "em.doe.gov" {
        type stub;
        file "db.stub.em.doe.gov";
        masters {
                132.172.137.102;
                132.172.137.146;
        };
};

I also have the appropriate delegation and glue information in my doe.gov
zone as follows:

$TTL  3600
@          IN SOA SUKHOI.DOE.GOV. root at sukhoi.doe.gov. (
           19990550       ; serial
           7200           ; refresh in seconds
           3600           ; retry in seconds
           604800         ; expire in seconds
           43200 )        ; minimum in seconds

;NAMESERVERS
                IN      NS      sukhoi.doe.gov.
                IN      NS      fishbed.doe.gov.
em              IN      NS      ns3.em.doe.gov.
                IN      NS      ns7.em.doe.gov.
ns3.em.doe.gov. IN      A       132.172.137.146
ns7.em.doe.gov. IN      A       132.172.137.102
sukhoi          IN      A       146.138.1.215
fishbed         IN      A       146.138.198.215


After restarting the nameserver on my server, sukhoi, the file
db.stub.em.doe.gov is created and contains the following:

$ORIGIN .
$TTL 86400      ; 1 day
em.doe.gov              IN SOA  emsun3.em.doe.gov.
David\\\.Carts.em.doe.gov. (
                                153        ; serial
                                10800      ; refresh (3 hours)
                                3600       ; retry (1 hour)
                                604800     ; expire (1 week)
                                86400      ; minimum (1 day)
                                )
                        NS      ns3.em.doe.gov.
                        NS      ns7.em.doe.gov.
                        NS      emsun3.em.doe.gov.
$ORIGIN em.doe.gov.
emsun3                  A       132.172.137.155
ns3                     A       132.172.137.146
ns7                     A       132.172.137.102

So you can see I clearly got the appropriate stub information (SOA and NS)
about EM's internal nameservers (ns3 and ns7) and stored it in my db file.
Now with my named.conf ready to go, my new stub information, and the
delegation records contained in the doe.gov zone, I good to go, correct?
Well... here is the output from a sample nslookup:

# nslookup
Default Server:  sukhoi.doe.gov
Address:  146.138.1.215

> set type=SOA
> em.doe.gov
Server:  sukhoi.doe.gov
Address:  146.138.1.215

Non-authoritative answer:
em.doe.gov
        origin = ns1.em.doe.gov
        mail addr = David.Carts.em.doe.gov
        serial = 119
        refresh = 10800 (3H)
        retry   = 3600 (1H)
        expire  = 604800 (1W)
        minimum ttl = 86400 (1D)

Authoritative answers can be found from:
em.doe.gov      nameserver = ns1.em.doe.gov
ns1.em.doe.gov  internet address = 205.254.144.179
> set type=NS
> em.doe.gov
Server:  sukhoi.doe.gov
Address:  146.138.1.215

Non-authoritative answer:
em.doe.gov      nameserver = ns1.em.doe.gov

Authoritative answers can be found from:
ns1.em.doe.gov  internet address = 205.254.144.179

It's as if the nameserver has completely ignored all of my configurations
and delegations, and worked its way down from the root servers looking for
information on EM, such that it found there external public nameserver, ns1!
What would cause this behavior??  If I remove the em.doe.gov zone from the
named.conf file completely, leaving only my delegation and glue statements
in the doe.gov zone, it shows the same behavior!  The only way I have been
able to get the correct information (queries routed to the correct, private
name servers) is to make em.doe.gov a forward zone in named.conf, but I'd
rather not do this... I'd rather just delegate to them and have that be
that.  Any reason why even with a stub zone that contains the correct
information about private name servers, I end up returning information about
their public nameserver that is not mentioned anywhere in my private
nameserver's zone files?

As always, any help and/or advice is appreciated!

-William Von Alt
 Verizon/US Department of Energy
 301.903.2710