Koozali.org: home of the SME Server

Need urgent help: All emails are off

Need urgent help: All emails are off
« on: February 25, 2016, 11:59:11 AM »
My email service has gone down.

The server is out of space for some reason.

Code: [Select]
[root@mail ~]# df -la
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/main-root
                     824587880 824183124         0 100% /
proc                         0         0         0   -  /proc
sysfs                        0         0         0   -  /sys
devpts                       0         0         0   -  /dev/pts
/dev/md1                101018     16971     78831  18% /boot
tmpfs                  7786436         0   7786436   0% /dev/shm
none                         0         0         0   -  /proc/sys/fs/binfmt_misc

Tried to check the logs and I find these messages:

Quote
Feb 24 14:44:05 mail esmith::event[7530]: Processing event: pre-backup 
Feb 24 14:44:05 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/actions/generic_template_expand
Feb 24 14:44:05 mail esmith::event[7530]: expanding /etc/dar/DailyBackup.dcf 
Feb 24 14:44:05 mail esmith::event[7530]: generic_template_expand=action|Event|pre-backup|Action|generic_template_expand|Start|1456314245 631137|End|1456314245 809818|Elapsed|0.178681
Feb 24 14:44:05 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/pre-backup/S10mysql-delete-dumped-tables
Feb 24 14:44:05 mail esmith::event[7530]: S10mysql-delete-dumped-tables=action|Event|pre-backup|Action|S10mysql-delete-dumped-tables|Start|1456314245 810076|End|1456314245 827713|Elapsed|0.017637
Feb 24 14:44:05 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/pre-backup/S20mysql-dump-tables
Feb 24 14:44:06 mail esmith::event[7530]: S20mysql-dump-tables=action|Event|pre-backup|Action|S20mysql-dump-tables|Start|1456314245 827915|End|1456314246 528666|Elapsed|0.700751
Feb 24 14:44:06 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/pre-backup/S30ldap-dump
Feb 24 14:44:06 mail esmith::event[7530]: S30ldap-dump=action|Event|pre-backup|Action|S30ldap-dump|Start|1456314246 528978|End|1456314246 695443|Elapsed|0.166465
Feb 24 14:44:06 mail esmith::event[7530]: Running event handler: /etc/e-smith/events/pre-backup/S50rewind-tape
Feb 24 14:44:06 mail esmith::event[7530]: S50rewind-tape=action|Event|pre-backup|Action|S50rewind-tape|Start|1456314246 695735|End|1456314246 751418|Elapsed|0.055683

Followed by:

Quote
httpd: PHP Warning:  Unknown: write failed: No space left on device (28) in Unknown on line 0
httpd: PHP Warning:  Unknown: Failed to write session data (files). Please verify that the current setting of session.save_path is correct (/tmp) in Unknown on line 0

I suspect the issue has something to do with a backup job, gone wrong. And as a consequence all the free space was used up. I'd like to stop whatever is causing this and recover hdd space. Otherwise my 100 or so users are off email.

I'm relatively new to SME, any help is highly appreciated. Thanks a million in advance.

Offline Stefano

  • *
  • 10,879
  • +3/-0
Re: Need urgent help: All emails are off
« Reply #1 on: February 25, 2016, 12:05:59 PM »
hi..

post the result of
Code: [Select]
mount

thank you

Re: Need urgent help: All emails are off
« Reply #2 on: February 25, 2016, 12:08:54 PM »
Thanks Stefano for the quick reply.

This is what I get

Code: [Select]
[root@mail ~]# mount
/dev/mapper/main-root on / type ext3 (rw,usrquota,grpquota)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/md1 on /boot type ext3 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

Offline Stefano

  • *
  • 10,879
  • +3/-0
Re: Need urgent help: All emails are off
« Reply #3 on: February 25, 2016, 12:12:06 PM »
ok..

now the result of:
Code: [Select]
du -s -m * /mnt/smb

Offline mmccarn

  • *
  • 2,647
  • +10/-0
Re: Need urgent help: All emails are off
« Reply #4 on: February 25, 2016, 12:13:45 PM »
[guessing]
My guess is that your server normally backs up to a removable drive, but has now run a backup into a folder on the root filesystem...
[/guessing]

Offline Stefano

  • *
  • 10,879
  • +3/-0
Re: Need urgent help: All emails are off
« Reply #5 on: February 25, 2016, 12:14:30 PM »
mine too..

waiting for OP feedback

Re: Need urgent help: All emails are off
« Reply #6 on: February 25, 2016, 12:16:02 PM »
That gives me

Code: [Select]
[root@mail ~]# du -s -m * /mnt/smb
1       anaconda-ks.cfg
1       cat
1       groupoffice.sql
1       install.log
1       install.log.syslog
1       smeserver-affa-2.0.0-rc4.noarch.rpm
1       smeserver-geoip-1.0.5-1.noarch.rpm
1       smeserver-qmHandle-1.4-3.el5.sme.noarch.rpm
1       smeserver-saco-qmHandle-1.3.4-3.noarch.rpm
1       smeserver-vacation-1.0-36.el4.sme.noarch.rpm
1       smeserver-wbl-0.1.0-11.el4.sme.noarch.rpm
0       upgrade.log
1       upgrade.log.20121013124445
0       upgrade.log.syslog
1       upgrade.log.syslog.20121013124445
1       /mnt/smb

Offline Stefano

  • *
  • 10,879
  • +3/-0
Re: Need urgent help: All emails are off
« Reply #7 on: February 25, 2016, 12:17:35 PM »
ok..

we have to find wich dir is full

Code: [Select]
cd /
du -s -m * | sort -nr | head

Re: Need urgent help: All emails are off
« Reply #8 on: February 25, 2016, 12:31:34 PM »
It took a while, but here's the result.

Code: [Select]
[root@mail /]# du -s -m * | sort -nr | head
585809  home
152170  var
64432   mnt
1010    usr
138     lib
43      etc
36      sbin
12      boot
10      tmp
7       bin

Offline Stefano

  • *
  • 10,879
  • +3/-0
Re: Need urgent help: All emails are off
« Reply #9 on: February 25, 2016, 12:36:17 PM »
you have a /var partition with 150 Gb of data..

can you give us some details about what your server is supposed to do?

Code: [Select]
cd /var
du -s -m * | sort -nr | head


Re: Need urgent help: All emails are off
« Reply #10 on: February 25, 2016, 12:44:06 PM »
The server only does emails. I have approx. 590GB of email data on the server.

Re: Need urgent help: All emails are off
« Reply #11 on: February 25, 2016, 12:55:03 PM »
All was fine yesterday prior to doing a backup to removable media via the server console. I've connected the drive to another computer and haven't found any data on it. So I suspect something went wrong during the backup job.

Here is the result:

Code: [Select]
[root@mail var]# du -s -m * | sort -nr | head
607287  affa
531     log
377     lib
280     cache
207     clamav
70      spool
3       service
3       qmail
2       www
1       yp

Offline byte

  • *
  • 2,183
  • +2/-0
Re: Need urgent help: All emails are off
« Reply #12 on: February 25, 2016, 12:56:50 PM »
Can you also please post the results of:

config show backupwk

and

config show backup

?
--[byte]--

Have you filled in a Bug Report over @ http://bugs.contribs.org ? Please don't wait to be told this way you help us to help you/others - Thanks!

Offline Stefano

  • *
  • 10,879
  • +3/-0
Re: Need urgent help: All emails are off
« Reply #13 on: February 25, 2016, 12:58:27 PM »
are you using affa?

Offline byte

  • *
  • 2,183
  • +2/-0
Re: Need urgent help: All emails are off
« Reply #14 on: February 25, 2016, 01:00:03 PM »
Here is the result:

Code: [Select]
[root@mail var]# du -s -m * | sort -nr | head
607287  affa

There is the problem...you have 600GB+ of data in the folder.
--[byte]--

Have you filled in a Bug Report over @ http://bugs.contribs.org ? Please don't wait to be told this way you help us to help you/others - Thanks!

Offline byte

  • *
  • 2,183
  • +2/-0
Re: Need urgent help: All emails are off
« Reply #15 on: February 25, 2016, 01:05:58 PM »
are you using affa?

affa –-list-archives

Ideally, if using affa you should be backing up to a second smeserver box instead of your production box.
--[byte]--

Have you filled in a Bug Report over @ http://bugs.contribs.org ? Please don't wait to be told this way you help us to help you/others - Thanks!

Offline Stefano

  • *
  • 10,879
  • +3/-0
Re: Need urgent help: All emails are off
« Reply #16 on: February 25, 2016, 01:07:30 PM »
well.. OP must give us more details..

Re: Need urgent help: All emails are off
« Reply #17 on: February 25, 2016, 01:16:57 PM »
Byte, I used affa a while back until the box I was using broke down.

"affa --list-archives" brings no result.


Code: [Select]
backupwk=service
    BackupTime=17:00
    Compression=6
    CompressionProg=gzip
    DaysInSet=1
    FullDay=7
    IncOnlyTimeout=no
    Login=VMHOST\KNOX
    Password=*****
    Program=dar
    SetsMax=1
    SmbHost=192.168.2.50
    SmbShare=Mail
    Timeout=8
    VFSType=cifs
    status=disabled

Code: [Select]
[root@mail var]# config show backup
backup=service
    BackupType=tape
    Program=flexbackup
    backupTime=2:00
    reminderTime=14:00
    status=disabled

Offline Stefano

  • *
  • 10,879
  • +3/-0
Re: Need urgent help: All emails are off
« Reply #18 on: February 25, 2016, 01:20:39 PM »
so now you aren't using affa anymore?

I'd say you can safely delete affa directory...

Offline byte

  • *
  • 2,183
  • +2/-0
Re: Need urgent help: All emails are off
« Reply #19 on: February 25, 2016, 01:53:46 PM »
so now you aren't using affa anymore?

I'd say you can safely delete affa directory...

I'd also make sure that any affa jobs that are scheduled to be disabled / turned off, even though the OP hasn't been using affa since the box broke it does appear that affa has still been running backups, hence no disk space left today.
--[byte]--

Have you filled in a Bug Report over @ http://bugs.contribs.org ? Please don't wait to be told this way you help us to help you/others - Thanks!

Re: Need urgent help: All emails are off
« Reply #20 on: February 25, 2016, 03:03:43 PM »
Thanks a lot Stefano and Byte. I'm most grateful for your help in getting this fixed. Removing the affa backup files has sorted out the issue. The emails are now flowing and the users are more than glad to be back online.

I'll uninstall the affa for now; even though there are currently no active jobs scheduled. I'll also keep an eye out for the trend in the usage of the remaining storage, just to be sure that there's nothing else running out the disk space.

Will definitely endeavor to improve my SME skillset. Who knows, I can be of help to someone else tomorrow!

Offline byte

  • *
  • 2,183
  • +2/-0
Re: Need urgent help: All emails are off
« Reply #21 on: February 25, 2016, 03:41:41 PM »
Thanks a lot Stefano and Byte. I'm most grateful for your help in getting this fixed. Removing the affa backup files has sorted out the issue. The emails are now flowing and the users are more than glad to be back online.

Good news :) now relax  8-)
--[byte]--

Have you filled in a Bug Report over @ http://bugs.contribs.org ? Please don't wait to be told this way you help us to help you/others - Thanks!