Topic: Traffic Analysis

[besterl]

I would like to know if there is a contrib (maybe I am searching in the wrong place) that can give me a more detailed traffic analysis.

I will explain my need.

Every now and again I get an abuser on site that either downloads a lot of junk or set up Peer2Peer software on their PC.

I usually get to culprit in the following way:

Fist I check good old System Monitor - and see that the send / rcv is maxed.

The next step is where it becomes a bit more laborious.

I open iptraf and and do an analysis of the traffic on all ports and sort by bytes every few seconds.

The problem with iptraf though is that it gives you excellent stats on the TCP usage and a very fast scrolling Window at the bottom with all the UDP traffic (and this is the area where the offense usually happens)

Most of the time - the window is too fast and I have to export the logs, import into Excel to find the culprit and offending port.  I have port blocking installed on the firewall and usually block the offending port..

The problem is however, that I cannot block all ports except for the ones I need as I get a lot of visitors to the site and cannot just block all the ports as a lot of these guys have software like Skype and Google talk that uses different ports all the time.

I though of blocking all ports except for a certain range and to distribute the range to the site and tell users to pick from that range of ports.  The abusers however were very quick to change their peer2peer software to those ports and abuse the bandwidth again.

Ideally I would like a tool that (like System Monitor) shows me a graph. I want to click on that graph either daily or hourly or 5 minute average and get a breakdown of bandwith used per IP (and even if possible per protocol / port).

Almost like an isoqlog for TCP/IP (nudge nudge wink).

In other words - I am looking for a contrib that can give me real-time (while it is happening) analysis of the traffic in a graphical format to make life a lot easier.

Like I said before - there is probably already a contrib for this, but I have been a Larry Laffer and has been looking in all the wrong places.

Your kind and generous responses will be valued as usual.

And now - like Larry - I posted in the wrong thread.

Sysadmin - please kick to the right thread please

[byte]

Moving this topic to the SME Server 7.x Contribs forum, it is more appropriate there. Thanks

PS.Have a look at http://wiki.contribs.org/Sarg

[cactus]

Like I said before - there is probably already a contrib for this, but I have been a Larry Laffer and has been looking in all the wrong places.

Your kind and generous responses will be valued as usual.

And now - like Larry - I posted in the wrong thread.

Sysadmin - please kick to the right thread please

Aaaah, good old Larry Laffer. Those were the days

[janet]

besterl

Every now and again I get an abuser on site that either downloads a lot of junk or set up Peer2Peer software on their PC.

Why bother analyzing and manually blocking ?

Try auto blocking with
http://wiki.contribs.org/P2P_blocking

Also install and use Dansguardian to prevent downloading of certain file types either as a global block on or on a per user block (by mac/IP address or username, depending how you configure the system), see the Filtering section of the wiki article
http://wiki.contribs.org/Dansguardian
also see
http://wiki.contribs.org/Dansguardian/ConfigFiles
and
this stats add on
http://wiki.contribs.org/Dansguardian-stats

I have port blocking installed on the firewall and usually block the offending port..

That is problematic as you are aware, the ports being used can change. You are really chasing your own tail trying to block traffic by blocking ports. Other options will be more effective.

I think you are wasting your time analysing usage traffic etc, just install the right blocking tools.

Like I said before - there is probably already a contrib for this, but I have been a Larry Laffer and has been looking in all the wrong places.

Why make life hard for yourself. Look in the Contribs and Howtos sections of this web site. The links are at the top of this forum. Do a bit of reading and make yourself familiar with all the add ons that are available. There are other add ons at other sites too, search with Google or search the forums. Also see dungog.net for free and commercial add ons.

[besterl]

Thanks for the feedback so far - I have used Dansguardian and even have my won method of bloacking certain sites.

That is not the problem for me.

Certain of the users got clever and start torrent downloads at home and leave them running at the office (effectively bypassing Dansguardian and squid).

What I am really after is just a simple method of seeing who is accessing what percentage of the bandwidth.

Thanks for the positive discussions.

[mmccarn]

I've never tried it, but ntop may do what you want.

[besterl]

This looks promising

Maybe I can assist to get it working properly as well

I will give feedback as I progress

Thanks a mil

[eastend99]

My method is to have tcpdump create a file in an ibay or personal folder:

Code: [Select]

tcpdump -i eth1 -s 1500 -n -C 100 -w /myfilelocation/eth1dump -W 10 -Z root..and use wireshark to analyse the traffic see http://www.wireshark.org/download.html)

The tcpdump command creates 10 ordered files of 100MB max. with all traffic for eth1 (external inet connection in server/gateway mode). It will automatically cycle the log files if the maximum # (10) is reached. If you don't need the data in the ethernet packets you may lower the -s value.

Additionally, you can easily setup tcpdump to filter the traffic you are looking for.

[besterl]

Great feedback so far.

The tcpdump method looks promising - however - I have one small problem with it.

Usually when the traffic is maxed out I do not want to add strain by downloading such large files - but it does looks promising - I will test it.

Here is my feedback on the ntop response

I tried installing ntop using the method described in the contribs section, but did not get a display on the webpage at http://server.ip:3000

I decided to scratch around a bit more and found the following info

wget http://heanet.dl.sourceforge.net/sourceforge/ntop/ntop-3.2-0.centos4.i386.rpm


yum localinstall ntop-3.2-0.centos4.i386.rpm --enablerepo=*

I ran this and followed the instructions

cp /etc/ntop.conf.sample /etc/ntop.conf

vi /etc/ntop.conf

I changed --interface eth0 to --interface eth0,eth1
saved the file and started with
/etc/init.d/ntop init
and then
/etc/init.d/ntop start

I accessed it with http://server.ip:3000 and had the interface I wanted

The nice thing is I do not have to run this unless I do not want to and it gives me a nice breakdown of usage and protocols.

I think this will give me what I want - but I want to test the tcpdump method as well to see if it gives info that is a bit easier to read.

Will keep you posted.

[cactus]

Usually when the traffic is maxed out I do not want to add strain by downloading such large files - but it does looks promising - I will test it.

You are not downloading anything, you are capturing and saving all packets to and from your ethernet card (matching a filter if configured), so you can analyze them.

I think this will give me what I want - but I want to test the tcpdump method as well to see if it gives info that is a bit easier to read.

I doubt it will be easier to read, but it will give you much more detailed information if you ever need it as it saves all network traffic for you to filter and analyze.

[besterl]

What I probably failed to mention is that I am not based on site - as for the tcpdump method - thus I have to download the tcpdump files to my location for analysis (over a maxed out 384K SAT link).

I tested it however - and I am impressed.
I ran tcpdump -i eth1 -s 1500 -n -C 100 -w /home/e-smith/files/ibays/Primary/html/eth1dump -W 10 -Z root for about 3 minutes and downloaded the file created to my location.

I am impressed - The amount of info provided by Wireshark is phenomenal.

I have to say - both these methods are viable alternatives for doing what I want.

I suppose the next question to ask is quite simple

I now identified the offending internal IP address abusing traffic.

According to the FAQ I can block them using the following

mkdir -p /etc/e-smith/templates-custom/etc/rc.d/init.d/masq/
pico -w /etc/e-smith/templates-custom/etc/rc.d/init.d/masq/40DenyRiffRaff
/sbin/iptables -A INPUT -s internal.ip.address -j DROP
/sbin/iptables -A INPUT -s another.internal.ip -j DROP

Then

/sbin/e-smith/expand-template /etc/rc.d/init.d/masq
/etc/init.d/masq restart

However - this method is quite permanent - and I end up blocking that IP permanently (or until I remove it) from accessing the internet.

If I just run the following command
/sbin/iptables -A INPUT -s internal.ip.address -j DROP

It does not seem to kill the traffic from that IP immediately.  The idea is just to kill that IP until the next restart, or until I could speak to the offending user.

I am not too fresh on iptables - so a bit of help would go a long way.

Thanks again

[Bud]

besterl

how do i get ntop to start automatically on startup of a server reboot ( /etc/init.d/ntop start ) ?

how can i get to the ntop weblink externally eg: " http://mydyndnslinktosmeserver:3000 " ?

any help greatly appreciated 

[janet]

Bud

how do i get ntop to start automatically on startup of a server reboot ( /etc/init.d/ntop start ) ?
how can i get to the ntop weblink externally eg: " http://mydyndnslinktosmeserver:3000 " ?

See this for clues
http://wiki.contribs.org/Ntop

[janet]

besterl

mkdir -p /etc/e-smith/templates-custom/etc/rc.d/init.d/masq/
pico -w /etc/e-smith/templates-custom/etc/rc.d/init.d/masq/40DenyRiffRaff
/sbin/iptables -A INPUT -s internal.ip.address -j DROP
/sbin/iptables -A INPUT -s another.internal.ip -j DROP

......I end up blocking that IP permanently (or until I remove it) from accessing the internet.

Those commands are for blocking incoming traffic to your server, coming from specified external IPs.

I thought you want to block internal IPs from accessing the Internet ie outgoing traffic ?
The FAQ says
"Block incoming IP address
I want to block All traffic from some ip-addresses to my server."

If you take a look at this thread (see below), you will see the way to do what you want.
The position of the template fragment is important so the iptables rule is not negated by earlier rules.
You may still need to formulate a suitable iptables rule though, so use
man iptables
and see
http://forums.contribs.org/index.php/topic,46036.0/all.html

[besterl]

Hi - I was out of the office for a couple of days - did not get to the forum

Bud - in reply to your question

You can start it automatically - but is starts creating strain on your server if running a long time - but in case you want to start automatically - just edit one of the startup scripts eg. /etc/init.d/rc.local

Also - to access from external I just did port forwarding (port 3000 to localhost port 3000)