Server crashing: kernel BUG at mm/rmap.c:42!

kevinb

237
+0/-0

Server crashing: kernel BUG at mm/rmap.c:42!

« on: February 11, 2008, 05:54:41 PM »

Hello Everyone,

I am not sure if this belongs here or in the bug tracker (http://bugs.contribs.org/show_bug.cgi?id=3901) so I am putting it in both places:

Our server has recently decided to crash every Saturday (the last two Saturdays anyway).

I am thinking this maybe a hardware problem.

Here is part of the log:

Quote

Feb 9 12:38:58 isaz-server kernel: ------------[ cut here ]------------
Feb 9 12:38:58 isaz-server kernel: kernel BUG at arch/i386/mm/highmem.c:42!
Feb 9 12:38:58 isaz-server kernel: invalid operand: 0000 [#1]
Feb 9 12:38:58 isaz-server kernel: SMP
Feb 9 12:38:58 isaz-server kernel: Modules linked in: ppp_mppe(U) ppp_async crc_ccitt ppp_generic(U) slhc nls_utf8 cifs appletalk(U) via_rhine e100 ipt_ULOG ipt_REJECT ipt_MASQUERADE ipt_state ipt_TOS ip_nat_ftp ip_conntrack_ftp iptable_mangle iptable_nat ip_conntrack iptable_filter ip_tables button battery ac uhci_hcd snd_intel8x0 snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore mii bonding(U) floppy dm_snapshot dm_zero dm_mirror ext3 jbd raid5 xor raid1 dm_mod ata_piix libata aic7xxx sd_mod scsi_mod
Feb 9 12:38:58 isaz-server kernel: CPU: 1
Feb 9 12:38:58 isaz-server kernel: EIP: 0060:[kmap_atomic+115/376] Tainted: G B VLI
Feb 9 12:38:58 isaz-server kernel: EIP: 0060:[<c011c8f2>] Tainted: G B VLI
Feb 9 12:38:58 isaz-server kernel: EFLAGS: 00010286 (2.6.9-67.0.1.ELsmp)
Feb 9 12:38:58 isaz-server kernel: EIP is at kmap_atomic+0x73/0x178
Feb 9 12:38:58 isaz-server kernel: eax: c000ad68 ebx: 00000000 ecx: dbfeb163 edx: 0000003c
Feb 9 12:38:58 isaz-server kernel: esi: 00000000 edi: ef556028 ebp: c000af48 esp: ee4bbda0
Feb 9 12:38:58 isaz-server kernel: ds: 007b es: 007b ss: 0068
Feb 9 12:38:58 isaz-server kernel: Process qmail-lspawn (pid: 13332, threadinfo=ee4bb000 task=f6f02830)
Feb 9 12:38:58 isaz-server kernel: Stack: 00000000 00000080 ee4bbdfc c014150c ee4bbe58 ee4bbdd4 c01412c2 00000000
Feb 9 12:38:58 isaz-server kernel: 00000000 fffad000 c1fdf240 0007ef92 00000000 ef556028 c2d3a1c0 c014d1fa
Feb 9 12:38:58 isaz-server kernel: 00000000 00000000 00000000 00000000 00a64000 c2d3a1c0 ee4bbe58 c2f0c800
Feb 9 12:38:58 isaz-server kernel: Call Trace:
Feb 9 12:38:58 isaz-server kernel: [__generic_file_aio_read+385/434] __generic_file_aio_read+0x181/0x1b2
Feb 9 12:38:58 isaz-server kernel: [<c014150c>] __generic_file_aio_read+0x181/0x1b2
Feb 9 12:38:58 isaz-server kernel: [file_read_actor+0/201] file_read_actor+0x0/0xc9
Feb 9 12:38:58 isaz-server kernel: [<c01412c2>] file_read_actor+0x0/0xc9
Feb 9 12:38:58 isaz-server kernel: [zap_pte_range+152/841] zap_pte_range+0x98/0x349
Feb 9 12:38:58 isaz-server kernel: [<c014d1fa>] zap_pte_range+0x98/0x349
Feb 9 12:38:58 isaz-server kernel: [zap_pmd_range+89/124] zap_pmd_range+0x59/0x7c
Feb 9 12:38:58 isaz-server kernel: [<c014d504>] zap_pmd_range+0x59/0x7c
Feb 9 12:38:58 isaz-server kernel: [unmap_page_range+60/95] unmap_page_range+0x3c/0x5f
Feb 9 12:38:58 isaz-server kernel: [<c014d563>] unmap_page_range+0x3c/0x5f
Feb 9 12:38:58 isaz-server kernel: [unmap_vmas+241/517] unmap_vmas+0xf1/0x205
Feb 9 12:38:58 isaz-server kernel: [<c014d677>] unmap_vmas+0xf1/0x205
Feb 9 12:38:58 isaz-server kernel: [exit_mmap+121/328] exit_mmap+0x79/0x148
Feb 9 12:38:58 isaz-server kernel: [<c0151994>] exit_mmap+0x79/0x148
Feb 9 12:38:58 isaz-server kernel: [mmput+78/114] mmput+0x4e/0x72
Feb 9 12:38:58 isaz-server kernel: [<c01207c4>] mmput+0x4e/0x72
Feb 9 12:38:58 isaz-server kernel: [exec_mmap+402/429] exec_mmap+0x192/0x1ad
Feb 9 12:38:58 isaz-server kernel: [<c0164c3a>] exec_mmap+0x192/0x1ad
Feb 9 12:38:58 isaz-server kernel: [flush_old_exec+67/588] flush_old_exec+0x43/0x24c
Feb 9 12:38:58 isaz-server kernel: [<c0164cfc>] flush_old_exec+0x43/0x24c
Feb 9 12:38:58 isaz-server kernel: [kernel_read+49/59] kernel_read+0x31/0x3b
Feb 9 12:38:58 isaz-server kernel: [<c0164a9e>] kernel_read+0x31/0x3b
Feb 9 12:38:58 isaz-server kernel: [load_elf_binary+1391/3163] load_elf_binary+0x56f/0xc5b
Feb 9 12:38:58 isaz-server kernel: [<c0180d06>] load_elf_binary+0x56f/0xc5b
Feb 9 12:38:58 isaz-server kernel: [kunmap_high+99/128] kunmap_high+0x63/0x80
Feb 9 12:38:58 isaz-server kernel: [<c014c0cb>] kunmap_high+0x63/0x80
Feb 9 12:38:58 isaz-server kernel: [copy_strings+555/565] copy_strings+0x22b/0x235
Feb 9 12:38:58 isaz-server kernel: [<c016466d>] copy_strings+0x22b/0x235
Feb 9 12:38:58 isaz-server kernel: [load_elf_binary+0/3163] load_elf_binary+0x0/0xc5b
Feb 9 12:38:58 isaz-server kernel: [<c0180797>] load_elf_binary+0x0/0xc5b
Feb 9 12:38:58 isaz-server kernel: [search_binary_handler+183/554] search_binary_handler+0xb7/0x22a
Feb 9 12:38:58 isaz-server kernel: [<c01657cf>] search_binary_handler+0xb7/0x22a
Feb 9 12:38:58 isaz-server kernel: [do_execve+365/509] do_execve+0x16d/0x1fd
Feb 9 12:38:58 isaz-server kernel: [<c0165aaf>] do_execve+0x16d/0x1fd
Feb 9 12:38:58 isaz-server kernel: [sys_execve+43/138] sys_execve+0x2b/0x8a
Feb 9 12:38:58 isaz-server kernel: [<c01049d5>] sys_execve+0x2b/0x8a
Feb 9 12:38:58 isaz-server kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Feb 9 12:38:58 isaz-server kernel: [<c02d8613>] syscall_call+0x7/0xb
Feb 9 12:38:58 isaz-server kernel: Code: d8 41 c0 01 c2 8d 42 16 c1 e0 0c 29 c1 89 4c 24 24 8d 04 d5 00 00 00 00 89 e9 29 c1 89 c8 8b 09 8b 58 04 85 c9 75 04 85 db 74 08 <0f> 0b 2a 00 28 9c 2e c0 8b 5c 24 28 8b 0d f8 b9 32 c0 8b 03 89
Feb 9 12:38:58 isaz-server kernel: <0>Fatal exception: panic in 5 seconds
Feb 9 12:38:58 isaz-server kernel: bad: scheduling while atomic!
Feb 9 12:38:58 isaz-server kernel: [schedule+45/2291] schedule+0x2d/0x8f3
Feb 9 12:38:58 isaz-server kernel: [<c02d5c31>] schedule+0x2d/0x8f3
Feb 9 12:38:58 isaz-server kernel: [__mod_timer+257/267] __mod_timer+0x101/0x10b
Feb 9 12:38:58 isaz-server kernel: [<c0129ecd>] __mod_timer+0x101/0x10b
Feb 9 12:38:58 isaz-server kernel: [poke_blanked_console+143/154] poke_blanked_console+0x8f/0x9a
Feb 9 12:38:58 isaz-server kernel: [<c020f6bd>] poke_blanked_console+0x8f/0x9a
Feb 9 12:38:58 isaz-server kernel: [vt_console_print+660/677] vt_console_print+0x294/0x2a5
Feb 9 12:38:58 isaz-server kernel: [<c020ea51>] vt_console_print+0x294/0x2a5
Feb 9 12:38:58 isaz-server kernel: [__mod_timer+257/267] __mod_timer+0x101/0x10b
Feb 9 12:38:58 isaz-server kernel: [<c0129ecd>] __mod_timer+0x101/0x10b
Feb 9 12:38:58 isaz-server kernel: [schedule_timeout+309/345] schedule_timeout+0x135/0x159
Feb 9 12:38:58 isaz-server kernel: [<c02d6d61>] schedule_timeout+0x135/0x159
Feb 9 12:38:58 isaz-server kernel: [process_timeout+0/19] process_timeout+0x0/0x13
Feb 9 12:38:58 isaz-server kernel: [<c012a772>] process_timeout+0x0/0x13
Feb 9 12:38:58 isaz-server kernel: [printk+14/17] printk+0xe/0x11
Feb 9 12:38:58 isaz-server kernel: [<c01228f0>] printk+0xe/0x11
Feb 9 12:38:58 isaz-server kernel: [die+346/363] die+0x15a/0x16b
Feb 9 12:38:58 isaz-server kernel: [<c01060c2>] die+0x15a/0x16b
Feb 9 12:38:58 isaz-server kernel: [do_invalid_op+207/242] do_invalid_op+0xcf/0xf2
Feb 9 12:38:58 isaz-server kernel: [<c0106425>] do_invalid_op+0xcf/0xf2
Feb 9 12:38:58 isaz-server kernel: [<f888cdc7>] dm_any_congested+0x2f/0x35 [dm_mod]
Feb 9 12:38:58 isaz-server kernel: [kmap_atomic+115/376] kmap_atomic+0x73/0x178
Feb 9 12:38:58 isaz-server kernel: [<c011c8f2>] kmap_atomic+0x73/0x178
Feb 9 12:38:58 isaz-server kernel: [current_fs_time+68/76] current_fs_time+0x44/0x4c
Feb 9 12:38:58 isaz-server kernel: [<c0126964>] current_fs_time+0x44/0x4c
Feb 9 12:38:58 isaz-server kernel: [update_atime+63/147] update_atime+0x3f/0x93
Feb 9 12:38:58 isaz-server kernel: [<c017346a>] update_atime+0x3f/0x93
Feb 9 12:38:58 isaz-server kernel: [do_generic_mapping_read+1085/1093] do_generic_mapping_read+0x43d/0x445
Feb 9 12:38:58 isaz-server kernel: [<c01412ba>] do_generic_mapping_read+0x43d/0x445
Feb 9 12:38:58 isaz-server kernel: [do_invalid_op+0/242] do_invalid_op+0x0/0xf2
Feb 9 12:38:58 isaz-server kernel: [<c0106356>] do_invalid_op+0x0/0xf2
Feb 9 12:38:58 isaz-server kernel: [error_code+47/56] error_code+0x2f/0x38
Feb 9 12:38:58 isaz-server kernel: [<c02d910b>] error_code+0x2f/0x38
Feb 9 12:38:58 isaz-server kernel: [kmap_atomic+115/376] kmap_atomic+0x73/0x178
Feb 9 12:38:58 isaz-server kernel: [<c011c8f2>] kmap_atomic+0x73/0x178
Feb 9 12:38:58 isaz-server kernel: [__generic_file_aio_read+385/434] __generic_file_aio_read+0x181/0x1b2
Feb 9 12:38:58 isaz-server kernel: [<c014150c>] __generic_file_aio_read+0x181/0x1b2
Feb 9 12:38:58 isaz-server kernel: [file_read_actor+0/201] file_read_actor+0x0/0xc9
Feb 9 12:38:58 isaz-server kernel: [<c01412c2>] file_read_actor+0x0/0xc9
Feb 9 12:38:58 isaz-server kernel: [zap_pte_range+152/841] zap_pte_range+0x98/0x349
Feb 9 12:38:58 isaz-server kernel: [<c014d1fa>] zap_pte_range+0x98/0x349
Feb 9 12:38:58 isaz-server kernel: [zap_pmd_range+89/124] zap_pmd_range+0x59/0x7c
Feb 9 12:38:58 isaz-server kernel: [<c014d504>] zap_pmd_range+0x59/0x7c
Feb 9 12:38:58 isaz-server kernel: [unmap_page_range+60/95] unmap_page_range+0x3c/0x5f
Feb 9 12:38:58 isaz-server kernel: [<c014d563>] unmap_page_range+0x3c/0x5f
Feb 9 12:38:58 isaz-server kernel: [unmap_vmas+241/517] unmap_vmas+0xf1/0x205
Feb 9 12:38:58 isaz-server kernel: [<c014d677>] unmap_vmas+0xf1/0x205
Feb 9 12:38:58 isaz-server kernel: [exit_mmap+121/328] exit_mmap+0x79/0x148
Feb 9 12:38:58 isaz-server kernel: [<c0151994>] exit_mmap+0x79/0x148
Feb 9 12:38:58 isaz-server kernel: [mmput+78/114] mmput+0x4e/0x72
Feb 9 12:38:58 isaz-server kernel: [<c01207c4>] mmput+0x4e/0x72
Feb 9 12:38:58 isaz-server kernel: [exec_mmap+402/429] exec_mmap+0x192/0x1ad
Feb 9 12:38:58 isaz-server kernel: [<c0164c3a>] exec_mmap+0x192/0x1ad
Feb 9 12:38:58 isaz-server kernel: [flush_old_exec+67/588] flush_old_exec+0x43/0x24c
Feb 9 12:38:58 isaz-server kernel: [<c0164cfc>] flush_old_exec+0x43/0x24c
Feb 9 12:38:58 isaz-server kernel: [kernel_read+49/59] kernel_read+0x31/0x3b
Feb 9 12:38:58 isaz-server kernel: [<c0164a9e>] kernel_read+0x31/0x3b
Feb 9 12:38:58 isaz-server kernel: [load_elf_binary+1391/3163] load_elf_binary+0x56f/0xc5b
Feb 9 12:38:58 isaz-server kernel: [<c0180d06>] load_elf_binary+0x56f/0xc5b
Feb 9 12:38:58 isaz-server kernel: [kunmap_high+99/128] kunmap_high+0x63/0x80
Feb 9 12:38:58 isaz-server kernel: [<c014c0cb>] kunmap_high+0x63/0x80
Feb 9 12:38:58 isaz-server kernel: [copy_strings+555/565] copy_strings+0x22b/0x235
Feb 9 12:38:58 isaz-server kernel: [<c016466d>] copy_strings+0x22b/0x235
Feb 9 12:38:58 isaz-server kernel: [load_elf_binary+0/3163] load_elf_binary+0x0/0xc5b
Feb 9 12:38:58 isaz-server kernel: [<c0180797>] load_elf_binary+0x0/0xc5b
Feb 9 12:38:58 isaz-server kernel: [search_binary_handler+183/554] search_binary_handler+0xb7/0x22a
Feb 9 12:38:58 isaz-server kernel: [<c01657cf>] search_binary_handler+0xb7/0x22a
Feb 9 12:38:58 isaz-server kernel: [do_execve+365/509] do_execve+0x16d/0x1fd
Feb 9 12:38:58 isaz-server kernel: [<c0165aaf>] do_execve+0x16d/0x1fd
Feb 9 12:38:58 isaz-server kernel: [sys_execve+43/138] sys_execve+0x2b/0x8a
Feb 9 12:38:58 isaz-server kernel: [<c01049d5>] sys_execve+0x2b/0x8a
Feb 9 12:38:58 isaz-server kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Feb 9 12:38:59 isaz-server kernel: [<c02d8613>] syscall_call+0x7/0xb
Feb 9 12:38:59 isaz-server kernel: ------------[ cut here ]------------

This happens four times and then a lockup.

The computer is an IBM Intellistation with dual 2.4Ghz Xeons and 4Gb RAM with four 36GB U320 SCSI drives and software RAID running SME 7.3 with all the latest updates. Been rock solid for over nine months.

I can supply the whole log if it would help.

Thank you for your help!

Kevin

Logged

cactus

4,880
+3/-0

Re: Server crashing: kernel BUG at mm/rmap.c:42!

« Reply #1 on: February 11, 2008, 09:56:20 PM »

Quote from: kevinb on February 11, 2008, 05:54:41 PM

Our server has recently decided to crash every Saturday (the last two Saturdays anyway).

I am thinking this maybe a hardware problem.

Based on the reproducibility I would start looking at jobs that are scheduled to be run on Saturday in your cron files, perhaps in combination with faulty hardware.

To all other readers please post to the bug report instead of posting here to keep all information in one place. I suggest locking this thread in order to 'force' all useful information to end up in the bug tracker.

« Last Edit: February 11, 2008, 09:58:48 PM by cactus »

Logged

Be careful whose advice you buy, but be patient with those who supply it. Advice is a form of nostalgia, dispensing it is a way of fishing the past from the disposal, wiping it off, painting over the ugly parts and recycling it for more than its worth ~ Baz Luhrmann - Everybody's Free (To Wear Sunscreen)

chris burnat

1,135
+2/-0

Re: Server crashing: kernel BUG at mm/rmap.c:42!

« Reply #2 on: February 12, 2008, 08:47:41 AM »

Quote from: cactus on February 11, 2008, 09:56:20 PM

I suggest locking this thread in order to 'force' all useful information to end up in the bug tracker.

Done.
Please follow this issue at http://bugs.contribs.org/show_bug.cgi?id=3901
Thanks.

« Last Edit: February 12, 2008, 08:52:12 AM by chris burnat »

Logged

- chris
If it does not work out of the box, please fill in a Bug Report @ Bugzilla (http://bugs.contribs.org) - check: http://wiki.contribs.org/Bugzilla_Help . Thanks.