13.6.1. Some Systems Work, Others Don't

% nslookup 
Default Server:  crab.wrotethebook.com 
Address:  172.16.12.1 
 
> set type=NS 
> foo.edu. 
Server:  crab.wrotethebook.com 
Address:  172.16.12.1 
 
foo.edu        nameserver = gerbil.foo.edu 
foo.edu        nameserver = red.big.com 
foo.edu        nameserver = shrew.foo.edu 
gerbil.foo.edu   inet address = 198.97.99.2 
red.big.com   inet address = 184.6.16.2 
shrew.foo.edu    inet address = 198.97.99.1 
> set type=ANY 
> server gerbil.foo.edu 
Default Server:  gerbil.foo.edu 
Address:  198.97.99.2 
 
> hamster.foo.edu 
Server:  gerbil.foo.edu 
Address:  198.97.99.2 
 
hamster.foo.edu        inet address = 198.97.99.8 
> server red.big.com 
Default Server:  red.big.com 
Address:  184.6.16.2 
> hamster.foo.edu 
Server:  red.big.com 
Address:  184.6.16.2 
 *** red.big.com can't find hamster.foo.edu: Non-existent domain

This sample nslookup session contains several steps. The first step is to locate the authoritative servers for the hostname in question (hamster.foo.edu). We set the query type to NS to get the name server records, and query for the domain (foo.edu) in which the hostname is found. This returns three names of authoritative servers: gerbil.foo.edu, red.big.com, and shrew.foo.edu.

Next, we set the query type to ANY to look for any records related to the hostname in question. Then we set the server to the first server in the list, gerbil.foo.edu, and query for hamster.foo.edu. This returns an address record. So server gerbil.foo.edu works fine. We repeat the test using red.big.com as the server, and it fails. No records are returned.

The next step is to get SOA records from each server and see if they are the same:

> set type=SOA 
> foo.edu. 
Server:  red.big.com 
Address:  184.6.16.2 
 
foo.edu        origin = gerbil.foo.edu 
   mail addr = amanda.gerbil.foo.edu 
   serial=10164, refresh=43200, retry=3600, expire=3600000, 
   min=2592000 
> server gerbil.foo.edu 
Default Server:  gerbil.foo.edu 
Address:  198.97.99.2 
 
> foo.edu. 
Server:  gerbil.foo.edu 
Address:  198.97.99.2 
 
foo.edu        origin = gerbil.foo.edu 
   mail addr = amanda.gerbil.foo.edu 
   serial=10164, refresh=43200, retry=3600, expire=3600000, 
   min=2592000 
 > exit

If the SOA records have different serial numbers, perhaps the zone file, and therefore the hostname, has not yet been downloaded to the slave server. If the serial numbers are the same and the data is different, as in this case, there is a definite problem. Contact the remote domain administrator and notify her of the problem. The administrator's mailing address is shown in the "mail addr" field of the SOA record. In our example, we would send mail to [email protected] reporting the problem.

13.6.2. The Data Is Here and the Server Can't Find It!

The problem was replicated on several other slave servers. The master server would resolve the name; the slave servers wouldn't. All servers had the same SOA serial number, and a dump of the cache on each server showed that they all had the correct address records for the hostname in question. So why wouldn't they resolve the hostname to an address?

% cat /usr/etc/events.wrotethebook.com.hosts 
PCpma      IN   A       172.16.64.159 
           IN   HINFO   "pc" "n3/800eventsnutscom" 
PCrkc      IN   A       172.16.64.155 
           IN   HINFO   "pc" "n3/800eventsnutscom" 
PCafc      IN   A       172.16.64.189 
           IN   HINFO   "pc" "n3/800eventsnutscom" 
accu       IN   A       172.16.65.27 
cmgds1     IN   A       172.16.130.40 
cmg        IN   A       172.16.130.30 
PCgns      IN   A       172.16.64.167 
           IN   HINFO   "pc" "(3/800eventsnutscom" 
gw         IN   A       172.16.65.254 
zephyr     IN   A       172.16.64.188 
           IN   HINFO   "Sun" "sparcstation" 
ejw        IN   A       172.16.65.17 
PCecp      IN   A       172.16.64.193
           IN   HINFO   "pc" "n Lsparcstationstcom"

13.6.3. Cache Corruption

For example, a user reported intermittent name server failures. She had no trouble with any hostnames within the local domain or with some names outside the local domain, but names in several different remote domains would not resolve. nslookup tests produced no solid clues, so the name server cache was dumped and examined for problems. The root server entries were corrupted, so named was reloaded to clear the cache and reread the named.ca file. Here's how it was done.

# kill -INT 'cat /etc/named.pid'

Once named writes its cache to the file, we can examine the file to see if the names and addresses servers are correct. The named_dump.db file is composed of three sections: the zone table section, the Cache & Data section, and the Hints section.

13.6.4. dig: An Alternative to nslookup

As an example, we'll use dig to ask the root server b.root-servers.net for the NS records for the mit.edu domain. To do this, enter the following command:

% dig @b.root-servers.net mit.edu ns

In this example, @b.root-servers.net is the server that is being queried. The server can be identified by name or IP address. If you're troubleshooting a problem in a remote domain, specify an authoritative server for that domain. In this example we're asking for the names of servers for a top-level domain (mit.edu), so we ask a root server.

Table 13-1. dig query types

Notice that the function of nslookup's ls command is performed by the dig query type axfr.

% dig -x 18.72.0.3 
 
; <<>> DiG 2.2 <<>> -x  
;; res options: init recurs defnam dnsrch 
;; got answer: 
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3
;; QUERY SECTION:
;;      3.0.72.18.in-addr.arpa, type = ANY, class = IN

;; ANSWER SECTION:
3.0.72.18.in-addr.arpa.  6H IN PTR  BITSY.MIT.EDU.

;; AUTHORITY SECTION:
18.in-addr.arpa.        6H IN NS        W20NS.MIT.EDU.
18.in-addr.arpa.        6H IN NS        BITSY.MIT.EDU.
18.in-addr.arpa.        6H IN NS        STRAWB.MIT.EDU.

;; ADDITIONAL SECTION:
W20NS.MIT.EDU.          6H IN A         18.70.0.160
BITSY.MIT.EDU.          6H IN A         18.72.0.3
STRAWB.MIT.EDU.         6H IN A         18.71.0.151

;; Total query time: 367 msec
;; FROM: wren.foobirds.org to SERVER: default -- 0.0.0.0
;; WHEN: Thu Jul 19 16:00:39 2001
;; MSG SIZE  sent: 40  rcvd: 170 

The core of the output of the dig command is found in the various sections from the DNS response packet. The header data from the example above is:

;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 4
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3 

dig does not display the header data in the order in which it occurs in the header section, but it is easy to map the dig display to the header described in RFC 1035. The various values displayed in the example and their meanings are listed here:

opcode: QUERY

Indicates that this is a standard query.

status: NOERROR

Indicates that no error code was found in the RCODE field of the header, which means that the RCODE field contains a 0.

id: 6

Indicates that the identifier used for this message was the number 6.

flags: qr aa rd ra

flags groups together all of the one-bit fields from the header. In this case it covers four different fields in the header section and gives us information about three others. This flag group means that QR is set to 1, indicating this is a response. AA is set to 1 because this answer came from an authoritative server. RD is set to 1 to indicate that recursion was requested by the query. RA is set to 1, indicating that recursion is available on the server. TC is not listed, meaning it is set to 0 and that the response was not truncated. AD and CD are also set to 0 because DNSSEC is not in use.

QUERY: 1, ANSWER: 1, AUTHORITY: 3, ADDITIONAL: 3

These represent the header fields QDCOUNT, ANCOUNT, NSCOUNT, and ARCOUNT, which indicate the number of resource records in the remaining sections of the response. This display says that there is one entry in the question section, one resource record in the answer section, three records in the authority section, and three records in the additional section.

The sample dig command displays the following query data:

;; QUERY SECTION:
;;      3.0.72.18.in-addr.arpa, type = ANY, class = IN

The three fields of this query are clearly shown. The class field is IN because this is a query for Internet records. The query is asking for any record (type = any) associated with 3.0.72.18.in-addr.arpa. Notice how dig reversed the address and created the proper reverse domain name for this query.

Next, the dig command displays the answer, authority, and additional sections:

;; ANSWER SECTION:
3.0.72.18.in-addr.arpa.  6H IN PTR  BITSY.MIT.EDU.

;; AUTHORITY SECTION:
18.in-addr.arpa.        6H IN NS        W20NS.MIT.EDU.
18.in-addr.arpa.        6H IN NS        BITSY.MIT.EDU.
18.in-addr.arpa.        6H IN NS        STRAWB.MIT.EDU.

;; ADDITIONAL SECTION:
W20NS.MIT.EDU.          6H IN A         18.70.0.160
BITSY.MIT.EDU.          6H IN A         18.72.0.3
STRAWB.MIT.EDU.         6H IN A         18.71.0.151

The answer is just what you would expect: the PTR record for 3.0.72.18.in-addr.arpa. The record tells us that the hostname for the address 18.72.0.3 is bitsy.mit.edu.

The authority section lists the servers that are authoritative for the 18.in-addr.arpa domain. There are three NS records, each providing the name of an authoritative server. From this, we know that w20ns.mit.edu, bitsy.mit.edu, and strawb.mit.edu are authoritative for the reverse domain 18.in-addr.arpa.

The additional section completes the message by providing the address of each of the authoritative servers. The addresses are important because if the local server wants to send future queries directly to these authoritative servers, it needs to know the servers' addresses. In this case, the addresses are 18.70.0.160, 18.72.0.3, and 18.71.0.151.

In addition to the DNS response, dig provides status information in the first three lines and the last four lines of the display. The first line echoes the dig command-line options (-x in the example). The second line displays the resolver library settings, and the third line states whether or not an answer was found for the query. The last four lines show the query response time, the name and address of the server that answered the query, when the query was received, and the size of the query and response packets. All of this can be helpful information when debugging a DNS problem.