Koozali.org: home of the SME Server
Obsolete Releases => SME Server 7.x => Topic started by: cactus on December 19, 2006, 10:34:51 PM
-
Can someone help me out? Since a few days my server seems to swap and swap the HD is writing non-stop and today all went wrong and my server stopped functioning until a reboot kicked it back up. (SWAP usage goes up radically late in the evening of Tuesday and has been low ever since the last reinstall 2,5 weeks before.
Using the sysmon graphs on my server I found the following graphics
(http://www.snetram.nl/wp-content/uploads/2006/12/sysmon.png)
(http://www.snetram.nl/wp-content/uploads/2006/12/sysmon2.png)
And digging through the /var/log/messages files the following, probably explaining the gap until around 17h:
Dec 19 11:48:24 homer kernel: [<d48a514e>] ext3_write_inode+0x22/0x3f [ext3]
Dec 19 11:48:25 homer kernel: [write_inode+48/55] write_inode+0x30/0x37
Dec 19 11:48:25 homer kernel: [<c018e643>] write_inode+0x30/0x37
Dec 19 11:48:25 homer kernel: [__sync_single_inode+203/657] __sync_single_inode+0xcb/0x291
Dec 19 11:48:25 homer kernel: [<c018e715>] __sync_single_inode+0xcb/0x291
Dec 19 11:48:25 homer kernel: [sync_sb_inodes+423/841] sync_sb_inodes+0x1a7/0x349
Dec 19 11:48:25 homer kernel: [<c018ebed>] sync_sb_inodes+0x1a7/0x349
Dec 19 11:48:25 homer kernel: [writeback_inodes+450/820] writeback_inodes+0x1c2/0x334
Dec 19 11:48:25 homer kernel: [<c018ef51>] writeback_inodes+0x1c2/0x334
Dec 19 11:48:26 homer kernel: [balance_dirty_pages+124/284] balance_dirty_pages+0x7c/0x11c
Dec 19 11:48:26 homer kernel: [<c014c849>] balance_dirty_pages+0x7c/0x11c
Dec 19 11:48:26 homer kernel: [<d48a3469>] ext3_ordered_commit_write+0xb6/0xc5 [ext3]
Dec 19 11:48:26 homer kernel: [generic_file_buffered_write+884/1110] generic_file_buffered_write+0x374/0x456
Dec 19 11:49:33 homer kernel: [<c01492f3>] generic_file_buffered_write+0x374/0x456
Dec 19 11:49:35 homer kernel: [<d484385e>] dm_request+0x75/0x88 [dm_mod]
Dec 19 11:49:36 homer kernel: [current_fs_time+68/76] current_fs_time+0x44/0x4c
Dec 19 11:49:37 homer kernel: [<c01266ac>] current_fs_time+0x44/0x4c
Dec 19 11:49:38 homer kernel: [__generic_file_aio_write_nolock+829/875] __generic_file_aio_write_nolock+0x33d/0x36b
Dec 19 11:49:39 homer kernel: [<c0149712>] __generic_file_aio_write_nolock+0x33d/0x36b
Dec 19 11:49:40 homer kernel: [generic_file_aio_write_nolock+57/127] generic_file_aio_write_nolock+0x39/0x7f
Dec 19 11:50:21 homer httpd: PHP Fatal error: Cannot redeclare unregister_globals() (previously declared in /home/e-smith/files/ibays/Primary/html/wp-settings.p^A:3) in /home/e-smith/files/ibays/Primary/html/wp-settings.php on line 3
Dec 19 11:51:42 homer kernel: [<c0149779>] generic_file_aio_write_nolock+0x39/0x7f
Dec 19 11:51:46 homer kernel: [generic_file_aio_write+119/205] generic_file_aio_write+0x77/0xcd
Dec 19 11:51:47 homer kernel: [<c0149968>] generic_file_aio_write+0x77/0xcd
Dec 19 11:51:48 homer kernel: [<d48a10af>] ext3_file_write+0x19/0x8a [ext3]
Dec 19 11:51:49 homer kernel: [do_sync_write+151/201] do_sync_write+0x97/0xc9
Dec 19 11:51:50 homer kernel: [<c01680fd>] do_sync_write+0x97/0xc9
Dec 19 11:51:51 homer kernel: [autoremove_wake_function+0/45] autoremove_wake_function+0x0/0x2d
Dec 19 11:51:52 homer kernel: [<c011e57b>] autoremove_wake_function+0x0/0x2d
Dec 19 11:51:53 homer kernel: [<d486a50f>] do_get_write_access+0x5c0/0x5dc [jbd]
Dec 19 11:51:54 homer kernel: [v2_write_dquot+454/518] v2_write_dquot+0x1c6/0x206
Dec 19 11:51:55 homer kernel: [<c01a119e>] v2_write_dquot+0x1c6/0x206
Dec 19 11:51:55 homer kernel: [dquot_commit+459/546] dquot_commit+0x1cb/0x222
Dec 19 11:51:55 homer kernel: [<c019cd9b>] dquot_commit+0x1cb/0x222
Dec 19 11:51:55 homer kernel: [<d48ab204>] ext3_write_dquot+0x3c/0x52 [ext3]
Dec 19 11:51:56 homer kernel: [dqput+568/866] dqput+0x238/0x362
Dec 19 11:51:57 homer kernel: [<c019d842>] dqput+0x238/0x362
Dec 19 11:51:58 homer kernel: [dquot_drop+75/126] dquot_drop+0x4b/0x7e
Dec 19 11:51:58 homer kernel: [<c019e740>] dquot_drop+0x4b/0x7e
Dec 19 11:52:01 homer kernel: [<d48ab1b2>] ext3_dquot_drop+0x25/0x3b [ext3]
Dec 19 11:52:02 homer kernel: [<d48a130c>] ext3_free_inode+0xed/0x3e1 [ext3]
Dec 19 11:52:02 homer kernel: [<d48a53b7>] ext3_mark_iloc_dirty+0xf/0x16 [ext3]
Dec 19 11:52:03 homer kernel: [<d48a5479>] ext3_mark_inode_dirty+0x3a/0x41 [ext3]
Dec 19 11:52:04 homer kernel: [<d48a24c8>] ext3_delete_inode+0x93/0xaa [ext3]
Dec 19 11:52:04 homer kernel: [<d48a2435>] ext3_delete_inode+0x0/0xaa [ext3]
Dec 19 11:52:06 homer kernel: [generic_delete_inode+239/553] generic_delete_inode+0xef/0x229
Dec 19 11:52:06 homer kernel: [<c01867e7>] generic_delete_inode+0xef/0x229
Dec 19 11:52:06 homer kernel: [iput+95/97] iput+0x5f/0x61
Dec 19 11:52:07 homer kernel: [<c0186c0e>] iput+0x5f/0x61
Dec 19 11:52:08 homer kernel: [sys_unlink+224/315] sys_unlink+0xe0/0x13b
Dec 19 11:52:08 homer kernel: [<c0179e33>] sys_unlink+0xe0/0x13b
Dec 19 11:52:20 homer kernel: [do_IRQ+691/703] do_IRQ+0x2b3/0x2bf
Dec 19 11:52:26 homer kernel: [<c01088fc>] do_IRQ+0x2b3/0x2bf
Dec 19 11:52:27 homer kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Dec 19 11:52:28 homer kernel: [<c030f8cb>] syscall_call+0x7/0xb
Can some one tell me what this all means? Is my SWAP space full?
As I installed some RPM's since the problem began I removed the installed RPM's
Dec 12 08:40:53 Updated: smeserver-yum.noarch 1.2.0-25.el4.sme
Dec 12 09:03:01 Updated: pcre.i386 6.7-1
Dec 12 22:53:25 Installed: fontconfig.i386 2.2.3-7.centos4
Dec 12 22:53:35 Installed: xorg-x11-libs.i386 6.8.2-1.EL.13.37
Dec 12 22:53:42 Installed: ImageMagick.i386 6.0.7.1-14
Dec 12 22:53:43 Installed: xorg-x11-Mesa-libGL.i386 6.8.2-1.EL.13.37
Dec 13 22:41:43 Updated: e-smith-backup.noarch 1.14.0-05jplb04
Dec 15 23:38:44 Installed: smeserver-dar2.noarch 0.0.1-0dmay
Dec 19 17:07:39 Installed: pcre.i386 4.5-3.2.RHEL4
Dec 19 17:39:13 Erased: xorg-x11-Mesa-libGL
Dec 19 17:39:20 Erased: ImageMagick
Dec 19 17:39:24 Erased: fontconfig
Dec 19 17:39:25 Erased: xorg-x11-libs
First of all I removed and reinstalled the original pcre followed by post-upgrade and reboot (I had to erase the dungog pcre RPM using rpm -e --nodeps because of over 200 dependencies). No changes after a while as you can see in the second graph.
Then I removed ImageMagick and still no improvement (second graph as well).
Does anybody know how to find what is causing my swap space to do such strange things?
-
And digging through the /var/log/messages files the following, probably explaining the gap until around 17h:
Dec 19 11:48:24 homer kernel: [<d48a514e>] ext3_write_inode+0x22/0x3f [ext3]
Dec 19 11:48:25 homer kernel: [write_inode+48/55] write_inode+0x30/0x37
Possible hardware? maybe a bad sector on drive? or bad cable?
-
I had a server acting like this (see this post: http://forums.contribs.org/index.php?topic=34420.0)
You may need more than 300MB ram if you're running spamassassin & rhsbl.