Topic: sysmon show lots of outgoing traffic afterhours

[Jáder]

I' a bit scarred because my server is showing lots of outgoing traffic after hours.
Here you can see one example: http://www.abandonemicrosoft.net/publico/Imagens/ETH1-Traffic.tiff

I've tryed to see what is generating this amount of traffic (all upload bandwidth is being used!) using tcptrack on eth1, but it shows nothing. (or I do not know how to use it to show what I' looking for).
Same thing for eth0.

I used netstat -an to show connections too... and as with tcptrack see nothing using a lot of bandwidth.
I saw lot of connections to 207.46.x.y ... and discovered it was owned by M$.
SO... or my server being used to DDoS to M$ or someone on my network is sending a lot of info to M$.
As I said before: I' SCARED!
see /var/log/message :

Code: [Select]

Aug 15 19:17:20 pantera sshd(pam_unix)[24245]: session closed for user root
Aug 15 19:35:36 pantera dhcpd: Wrote 27 leases to leases file.
Aug 15 21:58:37 pantera dhcpd: Wrote 27 leases to leases file.
Aug 15 21:58:37 pantera dhcpd: DHCPREQUEST for 192.168.124.174 from 00:11:5b:c2:ee:c4 (aux4) via eth0
Aug 15 21:58:37 pantera dhcpd: DHCPACK on 192.168.124.174 to 00:11:5b:c2:ee:c4 (aux4) via eth0
Aug 15 23:00:05 pantera sshd(pam_unix)[5456]: session opened for user root by (uid=0)
Aug 15 23:00:05 pantera sshd(pam_unix)[5456]: session closed for user root
Aug 15 23:00:09 pantera sshd(pam_unix)[5517]: session opened for user root by (uid=0)
Aug 15 23:00:09 pantera sshd(pam_unix)[5517]: session closed for user root
Aug 15 23:00:10 pantera sshd(pam_unix)[5538]: session opened for user root by (uid=0)
Aug 15 23:00:11 pantera sshd(pam_unix)[5538]: session closed for user root
Aug 15 23:00:13 pantera sshd(pam_unix)[5568]: session opened for user root by (uid=0)
Aug 15 23:00:14 pantera esmith::event[5570]: Processing event: pre-backup desktop
Aug 15 23:00:14 pantera esmith::event[5570]: Running event handler: /etc/e-smith/events/actions/generic_template_expand
Aug 15 23:00:14 pantera esmith::event[5570]: expanding /etc/dar/DailyBackup.dcf 
Aug 15 23:00:14 pantera esmith::event[5570]: generic_template_expand=action|Event|pre-backup|Action|generic_template_expand|S
tart|1250388014 66342|End|1250388014 342866|Elapsed|0.276524
Aug 15 23:00:14 pantera esmith::event[5570]: Running event handler: /etc/e-smith/events/pre-backup/S10mysql-delete-dumped-tab
les
Aug 15 23:00:15 pantera esmith::event[5570]: S10mysql-delete-dumped-tables=action|Event|pre-backup|Action|S10mysql-delete-dum
ped-tables|Start|1250388014 343155|End|1250388015 132715|Elapsed|0.78956
Aug 15 23:00:15 pantera esmith::event[5570]: Running event handler: /etc/e-smith/events/pre-backup/S20mysql-dump-tables
Aug 15 23:01:28 pantera esmith::event[5570]: S20mysql-dump-tables=action|Event|pre-backup|Action|S20mysql-dump-tables|Start|1
250388015 133065|End|1250388088 163110|Elapsed|73.030045
Aug 15 23:01:28 pantera esmith::event[5570]: Running event handler: /etc/e-smith/events/pre-backup/S50rewind-tape
Aug 15 23:01:28 pantera esmith::event[5570]: S50rewind-tape=action|Event|pre-backup|Action|S50rewind-tape|Start|1250388088 16
3513|End|1250388088 297919|Elapsed|0.134406
Aug 15 23:01:28 pantera sshd(pam_unix)[5568]: session closed for user root
Aug 15 23:01:29 pantera sshd(pam_unix)[5671]: session opened for user root by (uid=0)
Aug 16 00:19:22 pantera dhcpd: Wrote 27 leases to leases file.
Aug 16 00:19:22 pantera dhcpd: DHCPREQUEST for 192.168.124.177 from 00:18:8b:df:53:c8 (marcia) via eth0
Aug 16 00:19:22 pantera dhcpd: DHCPACK on 192.168.124.177 to 00:18:8b:df:53:c8 (marcia) via eth0
Aug 16 03:11:02 pantera squid[5480]: storeDirWriteCleanLogs: Starting...


[Stefano]

just a question.. are you sure you have not any pc doing p2p?

Stefano

[piran]

Might have a new Torrent... the hours look fishy.
I have a bunch of M$ IPs in that area blocked
for LWP::Simple and stuff.

[piran]

...
I saw lot of connections to 207.46.x.y ... and discovered it was owned by M$.

Some routers have a quality of service functionality.
Try setting QOS so that IP only gets to use the bare
minimum o/g bandwidth available until you sort it out.

[piran]

A user might have volunteered your server for TOR
(anonymising proxy stuff) or has set up an on-line
backup to an M$ cloud by way of experimentation.
Just guesses... Maybe running HTOP would show
you something working away in the background.

[mmccarn]

iptraf or ntop might help you figure out what's up.

[Jáder]

just a question.. are you sure you have not any pc doing p2p?

Stefano

AFAIK there are none.
But do you notice the traffic starts at 11pm (was yesterday, and I was there working till 2pm) As you can see there are low traffic from 10am (when I arrived and shutdown router) to 11pm . I post messages log because of start of traffic being 11pm... thought about some cron job. But cannot find anything.
I also cannot see the traffic using any tools I know... iptraf/tcptrac just show small traffic.

[Jáder]

iptraf or ntop might help you figure out what's up.

That is part of problem, I tried to discover what is generating that traffic using tcptrac and iptraf and see nothing!

[Jáder]

A user might have volunteered your server for TOR
(anonymising proxy stuff) or has set up an on-line
backup to an M$ cloud by way of experimentation.
Just guesses... Maybe running HTOP would show
you something working away in the background.

Hum... that could be possible... but shouldn' I be able to watch this traffic and trace it back to one computer using IPTRAF or TCPTRACK ?

Right now just 2 computers (other than server) are up and running. Just like when I left them yesterday 2pm. So why traffic started 11pm ...
I'm thinking the SERVER is generating this traffic... just I do not know how to see/watch/track source!!

Any ideas?

[piran]

Don't know, I'm not up with that sort stuff, it was
just a suggestion to help you sleuth out your difficulty.
Staying with the router, it should be able to directly
tell exactly where a particular masquerading/NAT
session is being routed... so that'll take it down
to the misbehaving or otherwise workstation.

[Jáder]

Piran

Thanks by your tips.
I have some problems about to use router to track it.
My setup is WEB -> ADSL modem (router mode)--> SME server (2 NICs!)--> 24p managed switch --> LAN

I have attempted find the source of traffic on every point... none shows nothing!
And now, I try to restart squid on server... and traffic dropped!
See it now at: http://www.abandonemicrosoft.net/publico/Imagens/ETH1-traffic-post-squid-restart.png

So... it APPEARS to be a SME server problem. Maybe it's time to report a bug.

[piran]

Another suggestion... someone might have volunteered your
box into a distributed processing net like the ones doing the
search for intelligent life, the weather model thing or the
one running assessments on interactions of pills and potions.
In short something deliberately set up to use out-of-hours
CPU cycles on a benign basis.

[piran]

Racking my memory now... I've seen something like this here
before now. The implementation of the nightly backup went
wrong but it was the CPU utilisation that sky'd and not the
o/g bandwidth. Have you defined (in error) an external
target for some backup or something?

[Jáder]

Racking my memory now... I've seen something like this here
before now. The implementation of the nightly backup went
wrong but it was the CPU utilisation that sky'd and not the
o/g bandwidth. Have you defined (in error) an external
target for some backup or something?

Hum... that looks promising!
I have disabled night backups, but I'm not sure I remove ALL configurations, I think  I just change it to DISABLED and removed external USB drive. This is a piece of my crontab:

Code: [Select]

# run-parts

01 * * * * root run-parts /etc/cron.hourly
02 4 * * * root run-parts /etc/cron.daily
22 4 * * 0 root run-parts /etc/cron.weekly
42 4 1 * * root run-parts /etc/cron.monthly

# logrotate
12 1 */7 * * root       /sbin/e-smith/signal-event logrotate


# Backup.Access.Roan97.mdb
07 9-19 * * mon-fri  root /home/e-smith/files/Backup.Access.Roan97.mdb.sh

# Backup task is disabled
# Workstation Backup task is disabled

02 4 * * * root /sbin/e-smith/check4updates -m

# smeserver-clamscan filesystem scan
12      0       *       *       *       root    /sbin/e-smith/smeserver-clamscan

# dar2
05 12 * * FRI root /etc/e-smith/events/actions/dar2-backup Backup.Site
50 12 * * FRI root /etc/e-smith/events/actions/dar2-backup ISO9000
30 12 * * FRI root /etc/e-smith/events/actions/dar2-backup Roberto
20 12 * * FRI root /etc/e-smith/events/actions/dar2-backup sisROAN

# rsync disabled
Note comments about backup being disabled.
Do you think could be something like that ?
Thanks
Jáder

[piran]

Sympathetic shoulder and ideas man that's me;~)
Fixing issues that's bugzilla.
Personally I wouldn't mess with the crontab directly
as it is template driven (usual comments yah dah).
Just put the dar2 stuff into manual with the control
panel and that does not bypass templates. You'll
get the proper restart actions done too. While you
are in the system manager's control panel double
check the settings for all the backups ~ particularly
the target or destination for each backup.

[Jáder]

Sympathetic shoulder and ideas man that's me;~)
Fixing issues that's bugzilla.

Great

Personally I wouldn't mess with the crontab directly
as it is template driven (usual comments yah dah).
Just put the dar2 stuff into manual with the control
panel and that does not bypass templates. You'll
get the proper restart actions done too. While you
are in the system manager's control panel double
check the settings for all the backups ~ particularly
the target or destination for each backup.

I do not mess with crontab file as far I can. I just copy it to show it´s disabled.
I have dar2 + rsync installed... but both disabled.
Also are nightly backups disabled.

I just discovered my own server is doing a affa backup to target server starting 11pm (had forgot about it... I love you AFFA).  So I think this is the problem. I just couldn´t see it because I was thinking the traffic I was seeing was ssh (naive!).
I´ll stop affa job for tonight and see if do not have graphic shaped 11pm.

The affa job just backup ONE 155MB file (.mbd)... so I don´t think it should take 6 hours... but that´s another problem. For this matter... I think case is closed. Sorry to be borring you all this time! :$


Jáder

[piran]

FWIW the CPU max issue I made for myself was
using backuppc but it was my fault (backuppc is
a good solid product I just got initially confused).

I use dar2 permanently for the server stuff and
swear by (in a good way) the amazing AFFA for
my data all over the intranet (thanks Michael).
AFFA is so good it's almost seamless in operation
which is probably why you forgot about it!
Perhaps you should set its admin email for each
day to remind you, the data it provides is useful.

You haven't been boring, this is how things go;~)
We should all hope that our 'scares' are benign.
As for the other stuff... AFFA has a kill job command
which can be used if you find it necessary, I find
that using HTOP will show me which and what is
running that apparently needs stopping. That one
155MB file should not take 6hrs! 155GB maybe but
not MB. Maybe it is an 'open' file or perhaps some
M$ bug/feature sent to mess us up (normal).

[piran]

A thought: .mbd - windows - are you using the
cygwin thing... If you are then that is probably
at the root of the stuck backup. That cygwin
is practically poison to my w2k workstations.
I use AFFA in a mapped drive mode to back
up the M$ boxes on to SME. Utterly reliable.

[Stefano]

[OT]
Piran, can I ask you how do you format your posts?
[/OT]

[CharlieBrady]

see /var/log/message :

What exactly in /var/log/messages were you concerned about? I don't see any references to 207.46.x.y.

[Jáder]

Hi Charlie

What exactly in /var/log/messages were you concerned about? I don't see any references to 207.46.x.y.

I was scared because my server was showing root task running 11pm , I could track a lot of connections to 207.46.x.y.  AND had a outgoing traffic shapped at total bandwidth for all night!

I discovered they have all different explanations and have learned a lot in process:

I had a lot of outgoing traffic starting at 11pm because my server was attempting to do a backup (using affa for this server). Disabled affa to be 100% sure. Later fixed issues (open files) to allow a quick backup.

I had a lot of connections to M$ because computers were trying to download updates to Windows.
The connections were dying because I had a infected M$ computer on LAN opening thousands of connections to Russia. I disconnected infected computer from LAN!

The dying connections were staying for too long time up/connected because WRT54g had default config (512 connections and timeout = 3600) and getting too busy (95/97% of possible connections open). Change for 1024 connections with timeout=120 (2 min) helped a lot.

When WRT54g was busy and w/high number of connections it started to drop connections... and internet access got unreliable.

oh God... so much thing happened in a few days... and a lot of things happened simultaneously...but it´s all fixed now.