Topic: disk used 100% 418GB

[davidS]

hello
my disk exceeded to 418gb 100% used
usualy  my server is almost empty 10% top.
disk used report
/dev/mapper/main-root                      442G  419G  5.4M 100% /
pls help me to find out how this can nhappened .

[mmccarn]

This command will take a while to run, but will show you folder sizes by directory in the root:

Code: [Select]

du -sh /*
Once you know which directory is using all your space (probably /home, but possibly /var), you can repeat the command in that folder:

Code: [Select]

du -sh /home/*
A few scenarios that have filled my disk in the past:
- Backups configured to use a removable disk which did not mount correctly (resulting in repeated backups back into the root folder)
--> /mnt will be larger than expected

- Compromized Web apps exploited by attackers
--> /home/e-smith/files/ibays (or /opt, if you have web apps installed in /opt) will be larger than expected

- Log files run amuck
--> /var/log will be larger than expected

- Mail loops
--> /var/qmail and/or /home/e-smith/files/users will be larger than expected

- Mail server compromised (and being used to send spam)
--> /var/qmail will be larger than expected

[davidS]

thank you very much
it is the /var/log folder 405GB
what should i do ? how do i delete it ?

[mmccarn]

Keep going -

Code: [Select]

du -sh /var/log/*
- which set of log files is it?

[davidS]

it's the mysqld.log

[mmccarn]

Hmmm...

/var/log/mysqld.log on my system is only 76232 bytes...

If you google 'mysqld.log huge' you'll get a few results showing how to truncate it -- BUT if you truncate it you may not be able to find out what happened.

You may want to stop MySQL (sv d mysqld) and look at that log file to see if you can figure out what happened.

If you decide you need to save a copy for future investigation, you'll need to delete something else to free up at least a small amount of space, then attach a USB drive and copy off the logfile.

[CharlieBrady]

it's the mysqld.log

So there are two problems here.

One, you have something wrong on your system which means there is a lot of mysql logging being produced. You need to look at those logs, determine what is happening to produce those logs, and fix it.

Two, there is an issue with SME server which isn't limiting the size of the logs accumulated by mysql. Most services on SME server keep only a limited amount of logs for the service (using multilog or svlogd).

[mmccarn]

With Charlie's hint, after you've figure out why mysql is throwing enough errors to fill your disk, you can move future mysql errors from /var/log/mysqld.log into the multilog-managed logs (/var/log/mysqld/*) by commenting out the "log-error" setting in /etc/my.cnf:

Code: [Select]

mkdir -p /etc/e-smith/templates-custom/etc/my.cnf
cd /etc/e-smith/templates-custom/etc/my.cnf
echo '# log-error disabled by custom template
# log-error=/var/log/mysqld.log' > 010mysqld-log-error
expand-template /etc/my.cnf
sv t mysqld

There is also a note in the mysql docs that the default settings include warnings in the error log.  If you look at your log and see that it's full of warnings that don't indicate problems you need to address, you may be able to reduce your logging with a custom template fragment that sets 'log-warnings=0'.

Reference:
http://dev.mysql.com/doc/refman/5.0/en/error-log.html

yum whatprovides "*/010mysqld-log-error"
reports that /etc/e-smith/templates/etc/my.cnf/010mysqld-log-error is supplied by:
e-smith-mysql-2.2.0-6.el5.sme.noarch


[edit]
Reported in Bug 8356

[davidS]

thank you guys
i could not work with the servre so i deleted the file first.
after chacking the mysql databases i found out that some of db that working with InnoDB are corrupted, after fixing some tables and restoring from old backup, the server is up and
everything seems to be normal.
i also removed the log file from my.cnf like you suggested .
and i will keep follow the log and the disk space
thank you