Koozali.org: home of the SME Server
Obsolete Releases => SME Server 9.x => Topic started by: Smitro on December 07, 2018, 12:10:19 PM
-
Hi,
I need help!
Recently my internet usage went crazy. I don't use a lot of traffic, about 100 - 300mb on an average day. Then one day 17gb up and 18 gb down, same the next, then the next until all my data allowance was used up.
I went through my network trying to work out what had happened. Even my router that shows the amount of traffic being used, seemed to miss this traffic.
I disconnected everything until I found that it was my SME server that caused this. The SME server is only working as a backup, using Affa. It downloads backups from another SME Server across the internet, and the router does not have any port forwards to it.
I've tried troubleshooting this, but I'm lost. I've decided to start over...
How can I detect what is flooding in and out of my server?
The amount of traffic that is going in and out seems almost symmetrical in size. I'm not sure if it's been compromised or if something has gone rogue?
-
iptraf should be available on your sme, and lets you monitor your network traffic in ip address pairs in real time. Here's a screenshot after using all the default selections:
- IP traffic monitor
- All Interfaces
IPTraf
┌ TCP Connections (Source Host:Port) ────────────────────────────────────────── Packets ─────────── Bytes ─── Flags ───── Iface ──────┐
│┌192.168.200.2:2202 > 1194 293512 -PA- eth0 │
│└192.168.200.110:50462 > 1189 62360 --A- eth0 │
│┌192.168.200.167:56634 = 9 1572 --A- eth0 │
│└192.168.200.2:9001 = 8 3464 CLOSED eth0 │
│┌192.168.200.2:9001 > 6 1176 --A- eth0 │
│└192.168.200.167:56508 > 3 2262 -PA- eth0 │
│┌192.168.200.17:64386 > 1 46 --A- eth0 │
│└192.168.200.2:3128 = 0 0 ---- eth0 │
│┌192.168.200.2:47328 > 1 52 --A- eth0 │
│└40.69.221.239:443 = 0 0 ---- eth0 │
│┌192.168.200.167:56636 = 9 1572 --A- eth0 │
│└192.168.200.2:9001 = 8 3464 CLOSED eth0 │
│┌192.168.200.167:56638 = 8 1520 --A- eth0 │
│└192.168.200.2:9001 = 8 3464 CLOSED eth0 │
│ │
│ │
│ │
│ │
│ │
└ TCP: 7 entries ────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ UDP (100 bytes) from 127.0.0.2:45238 to 127.0.0.2:53 on lo │
│ UDP (100 bytes) from 127.0.0.2:45238 to 127.0.0.2:53 on lo │
│ UDP (100 bytes) from 192.168.200.2:42743 to 205.251.199.54:53 on eth0 │
│ UDP (72 bytes) from 192.168.200.18:42262 to 192.168.200.2:53 on eth0 │
│ UDP (269 bytes) from 205.251.199.54:53 to 192.168.200.2:42743 on eth0 │
│ UDP (132 bytes) from 127.0.0.2:53 to 127.0.0.2:45238 on lo │
│ UDP (132 bytes) from 127.0.0.2:53 to 127.0.0.2:45238 on lo │
│ UDP (169 bytes) from 192.168.200.2:53 to 192.168.200.18:42262 on eth0 │
│ UDP (169 bytes) from 192.168.200.2:53 to 192.168.200.18:42262 on eth0 │
│ ICMP dest unrch (port) (197 bytes) from 192.168.200.18 to 192.168.200.2 on eth0 │
Pkts captured (all interfaces): 2887 │ TCP flow rate: 32.40 kbits/s
Up/Dn/PgUp/PgDn-scroll Lft/Rt-vtcl scrl W-chg actv win S-sort TCP X-exit
-
Check it on both ends...
-
It seems like nothing obvious, but the bottom part seems to keep scrolling through with traffic to port 53, which should be DNS. The top 2 addresses are the server and the workstation connected via SSH.
IPTraf
l TCP Connections (Source Host:Port) qqqqqqqqqq Packets qqq Bytes Flags Iface k
xl192.168.1.253:22 > 3556 1034320 -PA- eth0 x
xm192.168.1.151:52813 > 3455 170918 --A- eth0 x
x x
x x
x x
x x
x x
x x
x x
x x
x x
x x
m TCP: 1 entries qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq Active qj
lqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqk
x UDP (65 bytes) from 192.168.1.253:65227 to 192.5.5.241:53 on eth0 x
x UDP (65 bytes) from 192.168.1.253:19548 to 192.36.148.17:53 on eth0 x
x UDP (68 bytes) from 192.168.1.253:33203 to 193.0.14.129:53 on eth0 x
x UDP (68 bytes) from 192.168.1.253:53474 to 192.33.4.12:53 on eth0 x
x UDP (78 bytes) from 192.168.1.151:137 to 192.168.1.255:137 on eth0 x
m Bottom qqqqqq Elapsed time: 0:03 qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqj
Pkts captured (all interfaces): 12681 x TCP flow rate: 38.60 kbits/s
Up/Dn/PgUp/PgDn-scroll M-more TCP info W-chg actv win S-sort TCP X-exit
-
By default the top window is showing only TCP traffic by IP pair.
You can see traffic by TCP/UDP and port by starting iptraf using "-s <network device>" -- but then you lose the IP pair information:
iptraf -s eth0
The UDP traffic in your output (at least the top two entries) are DNS root servers (http://ipinfo.io/192.5.5.241 http://ipinfo.io/192.36.148.17 ) - I don't think this traffic is the problem.
I missed the detail in your first post about "Even my router that shows the amount of traffic being used, seemed to miss this traffic." -- can you provide more details on this? Is this router between the SME server and the internet, beside it, or behind it? If the router is the last device in your network before the ISP you'd have to have a device on the WAN side of the router or the ISPs network has been compromised somewhere else.
Can the ISP give you the IP and mac address of the device that is using all the bandwidth?
If not, perhaps you can dispute their quota system.
What kind of router is it? Is it up-to-date and does it have any unpatched security problems? Does it include any VPN or proxy services that could have been compromised by a remote attacker? Does it provide wifi, and is the wifi traffic included in the stats you're looking at?
Does the router reset its stats when rebooted, and has it been rebooted lately?
If so, perhaps someone is stealing your bandwidth and then rebooting the router to reset the network stats.
How big are the affa backups (how much total data)? Could there be an affa problem that is causing it to execute a full backup on every run instead of a hardlinked differential backup?
I had a problem like this when first testing Affa and using an SMB share as the backup destination...
If the router questions are a dead end -- what is your internet bandwidth supposed to be? Assuming 10mbit, it would only take about 5 hours to use 17GB of traffic. You would need to be running iptraf when the rogue process is running to see the traffic.
Try running iptraf for an extended period, checking the progress from time to time.
If you have a physical monitor attached, you could use that.
If not, you can do this using "screen" to make sure it keeps running if your ssh session is disconnected.
To start 'screen':
screen
Start 'iptraf' and set it to monitor traffic, set it to sort its display by bytes transferred (press "s" then "b") then exit from the screen session using <Ctrl>-A D (that is, press <Ctrl>-A, then press the "d" key)
Re-attach to screen to see how it's going using "screen -r"
screen -r
Don't forget to come back and stop iptraf at some point...