Topic: dns request storm

[rctcfamily]

Several weeks ago we installed SME 6.5 and we have been testing it on an isolated network to make sure we had all the settings working properly. Over the weekend we deployed the server as our domain controller and file server on the live network. All went fine over the weekend.

Monday morning when our employees began logging in the first few received their IP address and everything was going smoothly. Then some of the workstations did not receive an IP from the SME DHCP and defaulted to the 169.254. address. Several of the machines then sent a NBNS broadcast. After this the SME server began sending out a storm of DNS requests to various internet addresses and receiving responses. This traffic brought our 256k vsat connection to its knees.

If I disconnect the server from our network the requests stop and I can reconnect it with no further problems as long as those 169.254. addresses are not present.

Can someone tell me what is causing this and how to stop it?

thanks for your help
Ron

[NickR]

A couple of thoughts occur:

a) have you checked that the DHCP leases are being reaped (expired) correctly?

b) what is the size of the DHCP address pool?  Is it sufficient for the number of clients?

Looking at /var/log/messages will tell you if dhcpd is behaving as expected.

[CharlieBrady]

After this the SME server began sending out a storm of DNS requests to various internet addresses and receiving responses.

What were the DNS requests? And where were they sent? That might help us to guess what was generating them.

[rctcfamily]

Okay, I found the source of the bad IP addresses - some users connected to our wireless ap before I had gotten the IP address changed on it. They were not able to receive the DHCP assignment from SME.

It appears that the problems were caused by the bad IP addresses. Just after a wireless user tried to connect to the internet the SME server began sending thousands of UDP packets  to port 53 at various addresses. Each of these requests were then receiving a response and the traffic flooded our 256k vsat connection.

Below is a sample of the packets captured. There was not much going on before this started. The SME server is 192.168.1.2  After 169.254.33.215 made its NBNS request the server began sending out these DNS query packets. The rest of the log was full of the server requests interspersed with the NBNS requests. As soon as I unplugged the wireless ap the server activity stopped.

After changing the IP address of the wireless ap this same problem happened several more times. I was not able to capture the packets when it first started so I don't know if there was an IP problem before the flood. The pattern seems to start big and then taper off after about 10 or 15 minutes.

Could this be a compromised server or workstation? We didn't see problems like this with our Windows 2000 server or with the SME server on the test network.

Thanks for looking at this.
Ron

   8669 294.670967  169.254.33.215        169.254.255.255       NBNS     Name query[Short Frame]
   8688 295.354022  192.168.1.33          Broadcast             ARP      Who has 192.168.1.2?  Tell 192.168.1.33
   8689 295.354769  192.168.1.2           128.63.2.53           DNS      Standard query[Short Frame]
   8690 295.354882  192.168.1.2           198.41.0.4            DNS      Standard query[Short Frame]
   8691 295.354991  192.168.1.2           128.9.0.107           DNS      Standard query[Short Frame]
   8694 295.357460  192.168.1.2           192.5.5.241           DNS      Standard query[Short Frame]
   8695 295.366930  192.168.1.2           192.112.36.4          DNS      Standard query[Short Frame]
   8696 295.367079  192.168.1.2           192.5.5.241           DNS      Standard query[Short Frame]
   8697 295.367193  192.168.1.2           192.112.36.4          DNS      Standard query[Short Frame]
   8698 295.367322  192.168.1.2           192.203.230.10        DNS      Standard query[Short Frame]
   8699 295.368201  192.168.1.2           193.0.14.129          DNS      Standard query[Short Frame]
   8700 295.413938  169.254.33.215        169.254.255.255       NBNS     Name query[Short Frame]
   8719 296.036113  192.5.5.241           192.168.1.2           DNS      Standard query response[Short Frame]
   8720 296.037147  192.168.1.2           192.58.128.30         DNS      Standard query[Short Frame]
   8725 296.104919  192.5.5.241           192.168.1.2           DNS      Standard query response[Short Frame]
   8726 296.106752  192.168.1.2           192.36.148.17         DNS      Standard query[Short Frame]

[rctcfamily]

Just an update:

After sifting through way too many packets in a tcpdump log I've come to the conclusion that there is one workstation on the network that is making multiple DNS requests and then the server begins saturating our Internet connection with DNS requests. The initial DNS requests go to pa2.zonelabs.com which in turn causes the server to send its requests to a1981g.akamai.net using various IP addresses. I tried doing a ping to pa2.zonelabs.com and it tells me it is pinging a1981g.akamai.net  

For each dns request from the workstation the sme server sends multiple requests out, receives responses from all the servers, but never responds to the workstation which causes a new dns request from the workstation.

The workstation in question was running an illegal copy of ZA Pro loaded by a local computer shop. I removed the program, installed a/v and antispyware software, scanned the system, and did not find any obvious malware.

It appears that ZoneAlarm is trying to download an update file. What causes the server to launch this barrage of DNS packets? Malware on the workstation? A DNS problem on the server?

Needless to say, when I found the offending workstation I removed it from our network.

Any answers?
Thanks,
Ron