Koozali.org: home of the SME Server

Obsolete Releases => SME Server 8.x => Topic started by: juan.pescatori on February 25, 2016, 11:59:11 AM

Title: Need urgent help: All emails are off
Post by: juan.pescatori on February 25, 2016, 11:59:11 AM: My email service has gone down.

The server is out of space for some reason.

Code: [Select]
[root@mail ~]# df -la Filesystem 1K-blocks Used Available Use% Mounted on /dev/mapper/main-root 824587880 824183124 0 100% / proc 0 0 0 - /proc sysfs 0 0 0 - /sys devpts 0 0 0 - /dev/pts /dev/md1 101018 16971 78831 18% /boot tmpfs 7786436 0 7786436 0% /dev/shm none 0 0 0 - /proc/sys/fs/binfmt_misc
Tried to check the logs and I find these messages:

Quote
Feb 24 14:44:05 mail esmith::event[7530]: Processing event: pre-backup
Feb 24 14:44:05 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/actions/generic_template_expand
Feb 24 14:44:05 mail esmith::event[7530]: expanding /etc/dar/DailyBackup.dcf
Feb 24 14:44:05 mail esmith::event[7530]: generic_template_expand=action|Event|pre-backup|Action|generic_template_expand|Start|1456314245 631137|End|1456314245 809818|Elapsed|0.178681
Feb 24 14:44:05 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/pre-backup/S10mysql-delete-dumped-tables
Feb 24 14:44:05 mail esmith::event[7530]: S10mysql-delete-dumped-tables=action|Event|pre-backup|Action|S10mysql-delete-dumped-tables|Start|1456314245 810076|End|1456314245 827713|Elapsed|0.017637
Feb 24 14:44:05 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/pre-backup/S20mysql-dump-tables
Feb 24 14:44:06 mail esmith::event[7530]: S20mysql-dump-tables=action|Event|pre-backup|Action|S20mysql-dump-tables|Start|1456314245 827915|End|1456314246 528666|Elapsed|0.700751
Feb 24 14:44:06 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/pre-backup/S30ldap-dump
Feb 24 14:44:06 mail esmith::event[7530]: S30ldap-dump=action|Event|pre-backup|Action|S30ldap-dump|Start|1456314246 528978|End|1456314246 695443|Elapsed|0.166465
Feb 24 14:44:06 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/pre-backup/S50rewind-tape
Feb 24 14:44:06 mail esmith::event[7530]: S50rewind-tape=action|Event|pre-backup|Action|S50rewind-tape|Start|1456314246 695735|End|1456314246 751418|Elapsed|0.055683

Followed by:

Quote
httpd: PHP Warning: Unknown: write failed: No space left on device (28) in Unknown on line 0
httpd: PHP Warning: Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/tmp) in Unknown on line 0

I suspect the issue has something to do with a backup job, gone wrong. And as a consequence all the free space was used up. I'd like to stop whatever is causing this and recover hdd space. Otherwise my 100 or so users are off email.

I'm relatively new to SME, any help is highly appreciated. Thanks a million in advance.
Title: Re: Need urgent help: All emails are off
Post by: Stefano on February 25, 2016, 12:05:59 PM: hi..

post the result of
Code: [Select]
mount
thank you
Title: Re: Need urgent help: All emails are off
Post by: juan.pescatori on February 25, 2016, 12:08:54 PM: Thanks Stefano for the quick reply.

This is what I get

Code: [Select]
[root@mail ~]# mount /dev/mapper/main-root on / type ext3 (rw,usrquota,grpquota) proc on /proc type proc (rw) sysfs on /sys type sysfs (rw) devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/md1 on /boot type ext3 (rw) tmpfs on /dev/shm type tmpfs (rw) none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
Title: Re: Need urgent help: All emails are off
Post by: Stefano on February 25, 2016, 12:12:06 PM: ok..

now the result of:
Code: [Select]
du -s -m * /mnt/smb
Title: Re: Need urgent help: All emails are off
Post by: mmccarn on February 25, 2016, 12:13:45 PM: [guessing]
My guess is that your server normally backs up to a removable drive, but has now run a backup into a folder on the root filesystem...
[/guessing]
Title: Re: Need urgent help: All emails are off
Post by: Stefano on February 25, 2016, 12:14:30 PM: mine too..

waiting for OP feedback
Title: Re: Need urgent help: All emails are off
Post by: juan.pescatori on February 25, 2016, 12:16:02 PM: That gives me

Code: [Select]
[root@mail ~]# du -s -m * /mnt/smb 1 anaconda-ks.cfg 1 cat 1 groupoffice.sql 1 install.log 1 install.log.syslog 1 smeserver-affa-2.0.0-rc4.noarch.rpm 1 smeserver-geoip-1.0.5-1.noarch.rpm 1 smeserver-qmHandle-1.4-3.el5.sme.noarch.rpm 1 smeserver-saco-qmHandle-1.3.4-3.noarch.rpm 1 smeserver-vacation-1.0-36.el4.sme.noarch.rpm 1 smeserver-wbl-0.1.0-11.el4.sme.noarch.rpm 0 upgrade.log 1 upgrade.log.20121013124445 0 upgrade.log.syslog 1 upgrade.log.syslog.20121013124445 1 /mnt/smb
Title: Re: Need urgent help: All emails are off
Post by: Stefano on February 25, 2016, 12:17:35 PM: ok..

we have to find wich dir is full

Code: [Select]
cd / du -s -m * | sort -nr | head
Title: Re: Need urgent help: All emails are off
Post by: juan.pescatori on February 25, 2016, 12:31:34 PM: It took a while, but here's the result.

Code: [Select]
[root@mail /]# du -s -m * | sort -nr | head 585809 home 152170 var 64432 mnt 1010 usr 138 lib 43 etc 36 sbin 12 boot 10 tmp 7 bin
Title: Re: Need urgent help: All emails are off
Post by: Stefano on February 25, 2016, 12:36:17 PM: you have a /var partition with 150 Gb of data..

can you give us some details about what your server is supposed to do?

Code: [Select]
cd /var du -s -m * | sort -nr | head
Title: Re: Need urgent help: All emails are off
Post by: juan.pescatori on February 25, 2016, 12:44:06 PM: The server only does emails. I have approx. 590GB of email data on the server.
Title: Re: Need urgent help: All emails are off
Post by: juan.pescatori on February 25, 2016, 12:55:03 PM: All was fine yesterday prior to doing a backup to removable media via the server console. I've connected the drive to another computer and haven't found any data on it. So I suspect something went wrong during the backup job.

Here is the result:

Code: [Select]
[root@mail var]# du -s -m * | sort -nr | head 607287 affa 531 log 377 lib 280 cache 207 clamav 70 spool 3 service 3 qmail 2 www 1 yp
Title: Re: Need urgent help: All emails are off
Post by: byte on February 25, 2016, 12:56:50 PM: Can you also please post the results of:

config show backupwk

and

config show backup

?
Title: Re: Need urgent help: All emails are off
Post by: Stefano on February 25, 2016, 12:58:27 PM: are you using affa?
Title: Re: Need urgent help: All emails are off
Post by: byte on February 25, 2016, 01:00:03 PM: Quote from: juan.pescatori on February 25, 2016, 12:55:03 PM
Here is the result:

Code: [Select]
[root@mail var]# du -s -m * | sort -nr | head 607287 affa

There is the problem...you have 600GB+ of data in the folder.
Title: Re: Need urgent help: All emails are off
Post by: byte on February 25, 2016, 01:05:58 PM: Quote from: Stefano on February 25, 2016, 12:58:27 PM
are you using affa?

affa –-list-archives

Ideally, if using affa you should be backing up to a second smeserver box instead of your production box.
Title: Re: Need urgent help: All emails are off
Post by: Stefano on February 25, 2016, 01:07:30 PM: well.. OP must give us more details..
Title: Re: Need urgent help: All emails are off
Post by: juan.pescatori on February 25, 2016, 01:16:57 PM: Byte, I used affa a while back until the box I was using broke down.

"affa --list-archives" brings no result.

Code: [Select]
backupwk=service BackupTime=17:00 Compression=6 CompressionProg=gzip DaysInSet=1 FullDay=7 IncOnlyTimeout=no Login=VMHOST\KNOX Password=***** Program=dar SetsMax=1 SmbHost=192.168.2.50 SmbShare=Mail Timeout=8 VFSType=cifs status=disabled
Code: [Select]
[root@mail var]# config show backup backup=service BackupType=tape Program=flexbackup backupTime=2:00 reminderTime=14:00 status=disabled
Title: Re: Need urgent help: All emails are off
Post by: Stefano on February 25, 2016, 01:20:39 PM: so now you aren't using affa anymore?

I'd say you can safely delete affa directory...
Title: Re: Need urgent help: All emails are off
Post by: byte on February 25, 2016, 01:53:46 PM: Quote from: Stefano on February 25, 2016, 01:20:39 PM
so now you aren't using affa anymore?

I'd say you can safely delete affa directory...

I'd also make sure that any affa jobs that are scheduled to be disabled / turned off, even though the OP hasn't been using affa since the box broke it does appear that affa has still been running backups, hence no disk space left today.
Title: Re: Need urgent help: All emails are off
Post by: juan.pescatori on February 25, 2016, 03:03:43 PM: Thanks a lot Stefano and Byte. I'm most grateful for your help in getting this fixed. Removing the affa backup files has sorted out the issue. The emails are now flowing and the users are more than glad to be back online.

I'll uninstall the affa for now; even though there are currently no active jobs scheduled. I'll also keep an eye out for the trend in the usage of the remaining storage, just to be sure that there's nothing else running out the disk space.

Will definitely endeavor to improve my SME skillset. Who knows, I can be of help to someone else tomorrow!
Title: Re: Need urgent help: All emails are off
Post by: byte on February 25, 2016, 03:41:41 PM: Quote from: juan.pescatori on February 25, 2016, 03:03:43 PM
Thanks a lot Stefano and Byte. I'm most grateful for your help in getting this fixed. Removing the affa backup files has sorted out the issue. The emails are now flowing and the users are more than glad to be back online.

Good news :) now relax 8-)