How to Trace "TCP Receive Error"

[Mark Andrews]

> I am seeing lots of messages like this one from BIND-9.4.1-P1:
> 
>      [ID 873579 daemon.info] dispatch b090ef8:
>        shutting down due to TCP receive error: 69.59.189.68#53:
>        connection reset
> 
> I tried a Solaris snoop trace of all traffic between the DNS server
> (which has three IP addresses) to the IP address in the message:
> 
>       snoop -v -s3000 -o /tmp/snoop.trace 69.59.189.68
> 
> but I did not get any packets captured.  I ran the trace for one hour,
> and after not capturing anything, I looked in /var/adm/messages.
> There were about 300 such messages logged.  What snoop trace parameters
> do I have to specify to trace this activity?  I am assuming (maybe
> incorrectly) that snoop is tracing activity on all three IP addresses.
> I have BIND query logging on, and I do not see this address in the
> query.log file.  Thanks.
> ----------------------------------------------------------------------
> Barry S. Finkel
> Computing and Information Systems Division
> Argonne National Laboratory          Phone:    +1 (630) 252-7277
9700 South Cass Avenue               Facsimile:+1 (630) 252-4601
> Building 222, Room D209              Internet: BSFinkel at anl.gov
> Argonne, IL   60439-4828             IBMMAIL:  I1004994

	I suspect the nameserver has some sort of filtering box in
	front of it that is attempting to determine if the client
	is real or spoofed.  A "real" client will try TCP on seeing
	"tc" even if this is not strictly true for UDP only
	client/stacks.  This then turns just about all the UDP
	queries into TCP queries.  If the nameserver behind gets
	overwhelmed with TCP connections it will start sending out
	RST.  Self inflicted TCP SYN DoS.  There is a reason DNS
	uses UDP in the first place.

	This sort of "solution" does not scale.

	From the trace below the filtering box is keeping state
	because subsequent UDP queries get through.  This doesn't
	help much as many clients only ask a single question of a
	nameserver as A and AAAA queries often go to different
	nameservers.  If the filtering boxes were to share state
	there would be less problems.

	Mark
 
farside.isc.org:marka {1} % dig any . +norec @69.59.189.68 +ignore

; <<>> DiG 9.3.3 <<>> any . +norec @69.59.189.68 +ignore
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 40882
;; flags: qr aa tc; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;.                              IN      ANY

;; Query time: 4 msec
;; SERVER: 69.59.189.68#53(69.59.189.68)
;; WHEN: Sun Jan  6 22:57:56 2008
;; MSG SIZE  rcvd: 17

farside.isc.org:marka {2} % dig any . +norec @69.59.189.68 +ignore

; <<>> DiG 9.3.3 <<>> any . +norec @69.59.189.68 +ignore
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58042
;; flags: qr aa tc; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;.                              IN      ANY

;; Query time: 3 msec
;; SERVER: 69.59.189.68#53(69.59.189.68)
;; WHEN: Sun Jan  6 22:58:09 2008
;; MSG SIZE  rcvd: 17

farside.isc.org:marka {3} % dig any . +norec @69.59.189.68 +ignore +vc

; <<>> DiG 9.3.3 <<>> any . +norec @69.59.189.68 +ignore +vc
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 40817
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;.                              IN      ANY

;; Query time: 3 msec
;; SERVER: 69.59.189.68#53(69.59.189.68)
;; WHEN: Sun Jan  6 22:58:22 2008
;; MSG SIZE  rcvd: 17

farside.isc.org:marka {4} % dig any . +norec @69.59.189.68 +ignore 

; <<>> DiG 9.3.3 <<>> any . +norec @69.59.189.68 +ignore
; (1 server found)
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: REFUSED, id: 10118
;; flags: qr; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;.                              IN      ANY

;; Query time: 5 msec
;; SERVER: 69.59.189.68#53(69.59.189.68)
;; WHEN: Sun Jan  6 22:58:25 2008
;; MSG SIZE  rcvd: 17

farside.isc.org:marka {5} % 
-- 
Mark Andrews, ISC
1 Seymour St., Dundas Valley, NSW 2117, Australia
PHONE: +61 2 9871 4742                 INTERNET: Mark_Andrews at isc.org