Topic: Need urgent help: All emails are off

[juan.pescatori]

My email service has gone down.

The server is out of space for some reason.

Code: [Select]

[root@mail ~]# df -la
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/main-root
                     824587880 824183124         0 100% /
proc                         0         0         0   -  /proc
sysfs                        0         0         0   -  /sys
devpts                       0         0         0   -  /dev/pts
/dev/md1                101018     16971     78831  18% /boot
tmpfs                  7786436         0   7786436   0% /dev/shm
none                         0         0         0   -  /proc/sys/fs/binfmt_misc

Tried to check the logs and I find these messages:

Feb 24 14:44:05 mail esmith::event[7530]: Processing event: pre-backup 
Feb 24 14:44:05 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/actions/generic_template_expand
Feb 24 14:44:05 mail esmith::event[7530]: expanding /etc/dar/DailyBackup.dcf 
Feb 24 14:44:05 mail esmith::event[7530]: generic_template_expand=action|Event|pre-backup|Action|generic_template_expand|Start|1456314245 631137|End|1456314245 809818|Elapsed|0.178681
Feb 24 14:44:05 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/pre-backup/S10mysql-delete-dumped-tables
Feb 24 14:44:05 mail esmith::event[7530]: S10mysql-delete-dumped-tables=action|Event|pre-backup|Action|S10mysql-delete-dumped-tables|Start|1456314245 810076|End|1456314245 827713|Elapsed|0.017637
Feb 24 14:44:05 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/pre-backup/S20mysql-dump-tables
Feb 24 14:44:06 mail esmith::event[7530]: S20mysql-dump-tables=action|Event|pre-backup|Action|S20mysql-dump-tables|Start|1456314245 827915|End|1456314246 528666|Elapsed|0.700751
Feb 24 14:44:06 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/pre-backup/S30ldap-dump
Feb 24 14:44:06 mail esmith::event[7530]: S30ldap-dump=action|Event|pre-backup|Action|S30ldap-dump|Start|1456314246 528978|End|1456314246 695443|Elapsed|0.166465
Feb 24 14:44:06 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/pre-backup/S50rewind-tape
Feb 24 14:44:06 mail esmith::event[7530]: S50rewind-tape=action|Event|pre-backup|Action|S50rewind-tape|Start|1456314246 695735|End|1456314246 751418|Elapsed|0.055683

Followed by:

httpd: PHP Warning:  Unknown: write failed: No space left on device (28) in Unknown on line 0
httpd: PHP Warning:  Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/tmp) in Unknown on line 0

I suspect the issue has something to do with a backup job, gone wrong. And as a consequence all the free space was used up. I'd like to stop whatever is causing this and recover hdd space. Otherwise my 100 or so users are off email.

I'm relatively new to SME, any help is highly appreciated. Thanks a million in advance.

[Stefano]

hi..

post the result of

Code: [Select]

mount

thank you

[juan.pescatori]

Thanks Stefano for the quick reply.

This is what I get

Code: [Select]

[root@mail ~]# mount
/dev/mapper/main-root on / type ext3 (rw,usrquota,grpquota)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/md1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

[Stefano]

ok..

now the result of:

Code: [Select]

du -s -m * /mnt/smb


[mmccarn]

[guessing]
My guess is that your server normally backs up to a removable drive, but has now run a backup into a folder on the root filesystem...
[/guessing]

[Stefano]

mine too..

waiting for OP feedback

[juan.pescatori]

That gives me

Code: [Select]

[root@mail ~]# du -s -m * /mnt/smb
1       anaconda-ks.cfg
1       cat
1       groupoffice.sql
1       install.log
1       install.log.syslog
1       smeserver-affa-2.0.0-rc4.noarch.rpm
1       smeserver-geoip-1.0.5-1.noarch.rpm
1       smeserver-qmHandle-1.4-3.el5.sme.noarch.rpm
1       smeserver-saco-qmHandle-1.3.4-3.noarch.rpm
1       smeserver-vacation-1.0-36.el4.sme.noarch.rpm
1       smeserver-wbl-0.1.0-11.el4.sme.noarch.rpm
0       upgrade.log
1       upgrade.log.20121013124445
0       upgrade.log.syslog
1       upgrade.log.syslog.20121013124445
1       /mnt/smb

[Stefano]

ok..

we have to find wich dir is full

Code: [Select]

cd /
du -s -m * | sort -nr | head


[juan.pescatori]

It took a while, but here's the result.

Code: [Select]

[root@mail /]# du -s -m * | sort -nr | head
585809  home
152170  var
64432   mnt
1010    usr
138     lib
43      etc
36      sbin
12      boot
10      tmp
7       bin


[Stefano]

you have a /var partition with 150 Gb of data..

can you give us some details about what your server is supposed to do?

Code: [Select]

cd /var
du -s -m * | sort -nr | head


[juan.pescatori]

The server only does emails. I have approx. 590GB of email data on the server.

[juan.pescatori]

All was fine yesterday prior to doing a backup to removable media via the server console. I've connected the drive to another computer and haven't found any data on it. So I suspect something went wrong during the backup job.

Here is the result:

Code: [Select]

[root@mail var]# du -s -m * | sort -nr | head
607287  affa
531     log
377     lib
280     cache
207     clamav
70      spool
3       service
3       qmail
2       www
1       yp

[byte]

Can you also please post the results of:

config show backupwk

and

config show backup

?

[Stefano]

are you using affa?

[byte]

Here is the result:

Code: [Select]

[root@mail var]# du -s -m * | sort -nr | head
607287  affa


There is the problem...you have 600GB+ of data in the folder.