Topic: Log directory troubles & crashes

[pietdejong]

Hi All,

I have a server with SME, which had a tremendous amount of log-files in the /var/log directory.
These files where related to mgetty for dialin.

I was able to remove all the files, using the following command:  find /var/log/ -name 'mgetty*' -exec rm {} \;
because rm byitself would crash the shell, presumably because of memory problems.

The log files are all gone, but the directory size is remaining the same, see below,
1. ls -als from /var
2. ls -als from /var/log

Also strange is that I noticed that logrotate has now been running for over 224 minutes @ 99% CPU time..

Question:
1. Why is /var/log still showing 46022656 for size ?
2. Prior to me deleting the log-files thew server crashed twice, could this be caused by the large log files ?
3. Whats happening with logrotate, surely it shouldnt take this long to rotate the current log-files ?


1.============= /var ===================
    4 drwxr-xr-x    4 root     root         4096 Oct 23  2003 ftp
    4 drwxr-xr-x   14 root     root         4096 Oct 23  2003 lib
    4 drwxr-xr-x    2 root     root         4096 Feb  6  1996 local
    4 drwxrwxr-x    4 root     lock         4096 Jun 22 07:49 lock
44992 drwxr-xr-x   19 root     root     46022656 Jun 22 01:12 log
    0 lrwxrwxrwx    1 root     root           10 Oct 23  2003 mail -> spool/mail



2. ============ /var/log ===============





44992 drwxr-xr-x   19 root     root     46022656 Jun 22 01:12 .
    4 drwxr-xr-x   22 root     root         4096 May 17 12:49 ..
    0 lrwxrwxrwx    1 root     root           23 Jun 22 01:12 boot.log -> boot.log.20040622011202
   20 -rw-------    1 root     root        18844 May 21 22:32 boot.log.20040515011201
   12 -rw-------    1 root     root         9772 May 28 18:10 boot.log.20040522011202
    4 -rw-------    1 root     root         2943 May 31 21:55 boot.log.20040529011201
   12 -rw-------    1 root     root        11568 Jun  8 00:55 boot.log.20040601011201
   16 -rw-------    1 root     root        14058 Jun 15 00:18 boot.log.20040608011202
   24 -rw-------    1 root     root        22490 Jun 21 23:35 boot.log.20040615011201
    4 -rw-------    1 root     root          752 Jun 22 07:50 boot.log.20040622011202
    0 lrwxrwxrwx    1 root     root           19 Jun 22 01:12 cron -> cron.20040622011202
  212 -rw-------    1 root     root       211029 May 22 01:12 cron.20040515011201
  260 -rw-------    1 root     root       258873 May 29 01:12 cron.20040522011202
  116 -rw-------    1 root     root       111085 Jun  1 01:12 cron.20040529011201
  260 -rw-------    1 root     root       259035 Jun  8 01:12 cron.20040601011201
  260 -rw-------    1 root     root       258888 Jun 15 01:12 cron.20040608011202
  260 -rw-------    1 root     root       259370 Jun 22 01:12 cron.20040615011201
   12 -rw-------    1 root     root        10330 Jun 22 07:50 cron.20040622011202
    4 drwxr-x---    2 cvmlog   cvmlog       4096 Jun 21 08:47 cvm
    4 drwxr-s---    2 smelog   smelog       4096 Jun 21 08:47 dhcpcd
    4 drwxr-s---    2 smelog   smelog       4096 Jun 21 08:47 dhcpd
    4 drwxr-xr-x    2 root     root         4096 Nov 16  2003 diald
   16 -rw-r--r--    1 root     root        13268 Jun 21 08:47 dmesg
    4 drwxr-sr-x    2 dnslog   dnslog       4096 Jun 21 08:47 dnscache
    4 drwxr-xr-x    2 root     root         4096 Feb  4 11:28 flexbackup
    4 drwxr-xr-x    2 root     root         4096 Jun 22 01:12 httpd
    4 drwxr-x---    2 imaplog  imaplog      4096 Jun 21 08:47 imap
   72 -rw-r--r--    1 root     root        67133 Jun 21 08:47 ksyms.0
   72 -rw-r--r--    1 root     root        67133 May 20 16:11 ksyms.1
   72 -rw-r--r--    1 root     root        67133 May 14 09:26 ksyms.2
   72 -rw-r--r--    1 root     root        67133 Mar 31 12:23 ksyms.3
   72 -rw-r--r--    1 root     root        67133 Feb 12 11:10 ksyms.4
   72 -rw-r--r--    1 root     root        67133 Feb  9 12:55 ksyms.5
    0 lrwxrwxrwx    1 root     root           22 Jun 22 01:12 maillog -> maillog.20040622011202
    4 -rw-------    1 root     root         1340 May 21 07:01 maillog.20040515011201
    4 -rw-------    1 root     root          630 May 28 08:15 maillog.20040522011202
    4 -rw-------    1 root     root          136 May 30 01:48 maillog.20040529011201
    4 -rw-------    1 root     root           68 Jun  5 08:30 maillog.20040601011201
    4 -rw-------    1 root     root         1386 Jun 15 00:30 maillog.20040608011202
    4 -rw-------    1 root     root          135 Jun 20 02:00 maillog.20040615011201
    0 -rw-------    1 root     root            0 Jun 22 01:12 maillog.20040622011202
    0 lrwxrwxrwx    1 root     root           23 Jun 22 01:12 messages -> messages.20040622011202
  884 -rw-------    1 root     root       899320 May 22 01:12 messages.20040515011201
  708 -rw-------    1 root     root       718911 May 29 01:12 messages.20040522011202
  232 -rw-------    1 root     root       232344 Jun  1 01:12 messages.20040529011201
  984 -rw-------    1 root     root       999811 Jun  8 01:12 messages.20040601011201
  868 -rw-------    1 root     root       882556 Jun 15 01:12 messages.20040608011202
 1012 -rw-------    1 root     root      1030988 Jun 22 01:12 messages.20040615011201
   40 -rw-------    1 root     root        39710 Jun 22 07:50 messages.20040622011202
   16 -rw-r--r--    1 root     root        14402 Jun 22 07:50 mgetty.log.ttyS0
    4 -rw-r-----    1 mysql    mysql         797 Jun 21 08:48 mysqld.log
    0 -rw-r-----    1 mysql    mysql           0 Apr 25 09:25 mysqld.log.1
    0 -rw-r-----    1 mysql    mysql           0 Apr 18 05:27 mysqld.log.2
    0 -rw-r-----    1 mysql    mysql           0 Apr 11 04:24 mysqld.log.3
    0 -rw-r-----    1 mysql    mysql           0 Apr  4 04:07 mysqld.log.4
    4 drwxr-x---    2 smelog   smelog       4096 Jun 21 08:47 popd
    4 drwxr-s---    2 qmaill   nofiles      4096 Jun 21 08:47 pppoe
    4 drwxr-x---    2 root     root         4096 Jun 24  2003 proftpd
    4 drwxr-s---    2 qmaill   nofiles      4096 Jun 21 08:47 qmail
   12 -rw-r--r--    1 root     root        11317 Jun 21 04:02 rpmpkgs
   12 -rw-r--r--    1 root     root        11201 May  1 11:24 rpmpkgs.1
   12 -rw-r--r--    1 root     root        11201 Apr 24 06:01 rpmpkgs.2
   12 -rw-r--r--    1 root     root        11201 Apr 17 04:34 rpmpkgs.3
   12 -rw-r--r--    1 root     root        11201 Apr 10 04:10 rpmpkgs.4
    4 drwx------    2 root     root         4096 Jun 20 10:18 samba
    0 lrwxrwxrwx    1 root     root           21 Jun 22 01:12 secure -> secure.20040622011202
    4 -rw-------    1 root     root         1379 May 18 08:45 secure.20040515011201
    4 -rw-------    1 root     root          414 May 25 09:48 secure.20040522011202
    0 -rw-------    1 root     root            0 May 29 01:12 secure.20040529011201
    0 -rw-------    1 root     root            0 Jun  1 01:12 secure.20040601011201
    0 -rw-------    1 root     root            0 Jun  8 01:12 secure.20040608011202
    4 -rw-------    1 root     root         1788 Jun 20 12:09 secure.20040615011201
    0 -rw-------    1 root     root            0 Jun 22 01:12 secure.20040622011202
    4 drwxr-s---    2 qmaill   nofiles      4096 Jun 21 08:47 smtpfront-qmail
    0 lrwxrwxrwx    1 root     root           22 Jun 22 01:12 spooler -> spooler.20040622011202
    0 -rw-------    1 root     root            0 May 22 01:12 spooler.20040522011202
    0 -rw-------    1 root     root            0 May 29 01:12 spooler.20040529011201
    0 -rw-------    1 root     root            0 Jun  1 01:12 spooler.20040601011201
    0 -rw-------    1 root     root            0 Jun  8 01:12 spooler.20040608011202
    0 -rw-------    1 root     root            0 Jun 15 01:12 spooler.20040615011201
    0 -rw-------    1 root     root            0 Jun 22 01:12 spooler.20040622011202
    4 drwxr-xr-x    2 squid    squid        4096 Jun 10 16:17 squid
    4 drwxr-sr-x    2 dnslog   dnslog       4096 Jun 21 08:47 tinydns
    4 drwxr-xr-x    2 root     root         4096 Apr  8  2002 vbox
 1168 -rw-rw-r--    1 root     utmp      1189632 Jun 22 07:50 wtmp
  496 -rw-rw-r--    1 root     utmp       501888 May  1 06:19 wtmp.1

[pietdejong]

Output from top regarding logrotate:

19903 root      19   0 77488  75M   520 R    87.6 15.0 244:20 logrotate
    6 root       9   0     0    0     0 SW    0.2  0.0   2:04 kscand
 2566 squid      9   0 13476  10M  1012 R     0.2  1.9   0:07 squid
    1 root       8   0   476  448   412 S     0.0  0.0   0:06 init
    2 root       9   0     0    0     0 SW    0.0  0.0   0:00 keventd

[pietdejong]

Question related to this whole matter:

Does anyone know how to force an fsck on reboot ?

I would like to run a disk check and ensure everything is ok.

Thx
Piet

[schirrms]

hi,

Dunno if it can help, but :

a directory is no more than a file storing filename associated with the node number (the filesystem only use number to store file and the name are only here for our poor humans )

So if you have had a monstruous number of files in the directory /var/log, the size of the /var/log file 'yes, I mean file' ! ) becomes great also.
And the fact that you have suppressed the file don't reduce the size of the 'directory file'. Instead, the inode number is set to 0 (that's the general approach, i didn't works a lot with the ext3fs)
In old Unix system, there was only one way to change that :
mkdir /var/log2
mv /var/log/* /var/log2
rmdir (or rm -r) /var/log
mv /var/log2 /var/log
unfortunately, you cannot do that in runlevel 7 (to much open files)
But Linux comes with so many wonderful tools, maybe is somewhere one to 'pack' your directory ?

HTH,