Topic: Extreme high load on internet connection with DNS requests?

[Marco Hess]

For quite a while I think I have an issue with my SME8 server in that quite regularly it seems to do (what I think) bulk DNS cache refreshs request.

The symptoms is that at various times during the day the number of sessions from the server to the internet peaks at many thousands (e.g. almost 7000 today) and there is also a large amount of outgoing traffic at that same time. That amount of traffic consumes the whole upload bandwidth and other  normal internet activities slow to a crawl. Things like gaming suddenly have extreme extreme lags and users get kicked out of their sessions.

It typically lasts a few minutes and maybe up to 15 to 20 minutes and then reduces to normal again.

On the ADSL gateway (Vigor2830) I can see all this on the diagnostics graphs. Also in the NAT sessions table it is clear that the majority of sessions is all outgoing from the server to some IP address and port 53.

This is also not something that started recently. I think I had this problem for years and I even suspect that problems I had with previous ADSL routers was because they crashed under the session load (e.g. NAT table exhausted).

On the Vigor there is an option to restrict the number of sessions allowed for particular IP address. I tried that but it seems that LAN browser responses are getting slow because new DNS request via the server/proxy are being blocked when all available sessions are taken up by the DNS refresh.

So what can be the problem here?

If it is the DNS cache, is there a way to configure it that it does not peak like it does?

Is there a way to clear the cache so there are not 1000's of entries in there?

I welcome any suggestions that help to resolve this.

Regards,

Marco

[Daniel B.]

You should check:

- If those DNS queries are from the SME itself (and not from clients behind SME, requests being simply forwarded by SME if it's a server&gateway)
- If those are from SME, which clients ask for them ?
- Are those real DNS queries or just using port 53 ?

The problem most likely comes from a workstation on your LAN. If you're comfortable with it, you can try to diagnose it with iftop and tcpdump

[mmccarn]

Here are the culprits I can think of for excessive DNS requests:

* qpsmtpd's DNSBL or RHSBL doing what they're supposed to do (looking up incoming email server info in order to block potential spam).  You would see a corresponding jump in incoming connections on 25 or 465 at the same time.

* A local infection (server or workstation) of some sort sending spam out.  You would see a corresponding jump in outbound connections on 25.

* A service or web app set to log hostnames instead of IPs (requiring DNS lookups on the IP of every host that connects to the app).  You would see a corresponding jump in in-bound access to the web app (probably port 80 or 443)

The DNS cache does not do any sort of automatic refresh -- if a cached entry expires it is discarded, and only checked again online if the server or a local workstation re-tries access to that host.

Some diagnostic suggestions:

iptraf
I don't know if  iptraf still comes pre-installed on SME, but if you have it on your server you can fire it up and capture the actual traffic going from your server to port 53 to see what hostnames are being looked up.

dnscache & tinydns logs
These commands will show:
a) the number of DNS queries transmitted to a DNS server by DNSCache for 5/11/2015
b) the number of DNS requests served locally by tinydns on your SME for 5/11/2015

If you subtract b from a you should get something close to the number of lookups that actually went offsite on 5/11/2015 (compare the result to what you're seeing in your vigor)

Counts of dns lookups:

Code: [Select]

cat /var/log/dnscache/* |tai64nlocal |grep "^2015-05-11.*tx" |wc -l
cat /var/log/tinydns/* |tai64nlocal |grep "^2015-05-11" |wc -l

On my home server (two users, low email volume, owncloud server setup with 3 - 4 sync clients configured) I get these results, indicating around 8000 dns lookups:

[root@office ~]# cat /var/log/dnscache/* |tai64nlocal |grep "^2015-05-11.*tx" |wc -l
8316
[root@office ~]# cat /var/log/tinydns/* |tai64nlocal |grep "^2015-05-11" |wc -l
333
[root@office ~]#

[CharlieBrady]

Is there a way to clear the cache so there are not 1000's of entries in there?

If you clear the cache you will only make the problem worse.

The dnscache logs are a bit cryptic. There is a script supplied which makes them more readable. You can use these commands to look at your dnscache logs and determine what dns lookups are taking place.

Code: [Select]

perl /service/dnscache/dnscache-log.pl < /var/log/dnscache/current | tai64nlocal | less
perl /service/dnscache/dnscache-log.pl < /var/log/dnscache.forwarder/current | tai64nlocal | less


[Marco Hess]

Thanks for the suggestions.

I have done some additional logging and monitoring on the DNS and tcpdump and it appears that the problem is indeed coming from a rogue workstation going through the server proxy.

Now we know what it is and which PC it was, it was shown that the problem does not occur when that PC remained turned off.

The AVG virus scan did not pick anything up but ComboFix reported to have removed a bunch of stuff.

We will see if this has fixed the problem. If not, we a wipe and rebuild of that PC is in order.

Thanks again for the help.

Regards,

Marco