Koozali.org: home of the SME Server

Obsolete Releases => SME Server 9.x => Topic started by: Jáder on February 14, 2019, 12:35:25 PM

Title: Kernel Panic on new kernels
Post by: Jáder on February 14, 2019, 12:35:25 PM: Hi,

I´m with a kernel panic error in a very stable server (HP ML110).

edit: add screens to ilustrate the problem: (I´ll remove them later)

Kernel panic with 2.6.32-754-9.1
https://radio.antinsect.com.br/SME92-kernelPanic-2.6.32-754-9.1.el6.x86_64.jpg (https://radio.antinsect.com.br/SME92-kernelPanic-2.6.32-754-9.1.el6.x86_64.jpg)

Kernel panic with 2.6.32-754-10.1
https://radio.antinsect.com.br/SME92-kernelPanic-2.6.32-754-10.1.el6.x86_64.jpg (https://radio.antinsect.com.br/SME92-kernelPanic-2.6.32-754-10.1.el6.x86_64.jpg)

Boot Menu
https://radio.antinsect.com.br/SME92-kernelPanic-BootMenu.jpg (https://radio.antinsect.com.br/SME92-kernelPanic-BootMenu.jpg)

This is my boot menu:
Code: [Select]
[root@andorinha ~]# cat /boot/grub/menu.lst.ajm # grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro rd_NO_PLYMOUTH root=/dev/mapper/main-root # initrd /initrd-[generic-]version.img #boot=/dev/sda default=0 timeout=5 splashimage=(hd0,0)/grub/smeserver.xpm.gz foreground 000000 background 4E95D3 title SME Server 9.2 (2.6.32-754.10.1.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-754.10.1.el6.x86_64 ro rd_NO_PLYMOUTH root=/dev/mapper/main-root rd_NO_LUKS LANG=en_US.UTF-8 rd_MD_UUID=b3034d6a:23ab9f17:3fa8e16b:5ef3a368 rd_LVM_LV=main/root nodmraid rd_LVM_LV=main/swap SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto initrd /initramfs-2.6.32-754.10.1.el6.x86_64.img title SME Server 9.2 (2.6.32-754.9.1.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-754.9.1.el6.x86_64 ro rd_NO_PLYMOUTH root=/dev/mapper/main-root rd_NO_LUKS LANG=en_US.UTF-8 rd_MD_UUID=b3034d6a:23ab9f17:3fa8e16b:5ef3a368 rd_LVM_LV=main/root nodmraid rd_LVM_LV=main/swap SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto initrd /initramfs-2.6.32-754.9.1.el6.x86_64.img title SME Server 9.2 (2.6.32-754.6.3.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-754.6.3.el6.x86_64 ro rd_NO_PLYMOUTH root=/dev/mapper/main-root rd_NO_LUKS LANG=en_US.UTF-8 rd_MD_UUID=b3034d6a:23ab9f17:3fa8e16b:5ef3a368 rd_LVM_LV=main/root nodmraid rd_LVM_LV=main/swap SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto initrd /initramfs-2.6.32-754.6.3.el6.x86_64.img
I´ll had to remove the first 2 entries to avoid problems (last reboots , I´ve to choose manually the last option) and also will remove kernel package because all RPM keep only 3 versions ... so I´m about to loose last option and it ´s the only working. :o

I have these packages installed:
Code: [Select]
[root@andorinha ~]# rpm -qa|grep kernel kernel-2.6.32-754.9.1.el6.x86_64 kernel-devel-2.6.32-754.6.3.el6.x86_64 kernel-2.6.32-754.10.1.el6.x86_64 dracut-kernel-004-411.el6.noarch kernel-2.6.32-754.6.3.el6.x86_64 kernel-firmware-2.6.32-754.10.1.el6.noarch kernel-headers-2.6.32-754.10.1.el6.x86_64 kernel-devel-2.6.32-754.9.1.el6.x86_64 kernel-devel-2.6.32-754.10.1.el6.x86_64and now removindo olde kernels -9 (to keep on safe side and avoid any other RPM package remove my working kernel package)
Code: [Select]
[root@andorinha ~]# rpm -e kernel-devel-2.6.32-754.9.1.el6.x86_64 kernel-2.6.32-754.9.1.el6.x86_64 warning: erase unlink of /lib/modules/2.6.32-754.9.1.el6.x86_64/updates failed: No such file or directory warning: erase unlink of /lib/modules/2.6.32-754.9.1.el6.x86_64/modules.order failed: No such file or directory warning: erase unlink of /lib/modules/2.6.32-754.9.1.el6.x86_64/modules.networking failed: No such file or directory warning: erase unlink of /lib/modules/2.6.32-754.9.1.el6.x86_64/modules.modesetting failed: No such file or directory warning: erase unlink of /lib/modules/2.6.32-754.9.1.el6.x86_64/modules.drm failed: No such file or directory warning: erase unlink of /lib/modules/2.6.32-754.9.1.el6.x86_64/modules.block failed: No such file or directoryand verifying the remaining kernel packages:
Code: [Select]
[root@andorinha ~]# rpm -qa|grep kernel kernel-devel-2.6.32-754.6.3.el6.x86_64 kernel-2.6.32-754.10.1.el6.x86_64 dracut-kernel-004-411.el6.noarch kernel-2.6.32-754.6.3.el6.x86_64 kernel-firmware-2.6.32-754.10.1.el6.noarch kernel-headers-2.6.32-754.10.1.el6.x86_64 kernel-devel-2.6.32-754.10.1.el6.x86_64and now updating menu.lst
Code: [Select]
# grub.conf generated by anaconda # # Note that you do not have to rerun grub after making changes to this file # NOTICE: You have a /boot partition. This means that # all kernel and initrd paths are relative to /boot/, eg. # root (hd0,0) # kernel /vmlinuz-version ro rd_NO_PLYMOUTH root=/dev/mapper/main-root # initrd /initrd-[generic-]version.img #boot=/dev/sda default=0 timeout=10 splashimage=(hd0,0)/grub/smeserver.xpm.gz foreground 000000 background 4E95D3 title SME Server 9.2 (2.6.32-754.6.3.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-754.6.3.el6.x86_64 ro rd_NO_PLYMOUTH root=/dev/mapper/main-root rd_NO_LUKS LANG=en_US.UTF-8 rd_MD_UUID=b3034d6a:23ab9f17:3fa8e16b:5ef3a368 rd_LVM_LV=main/root nodmraid rd_LVM_LV=main/swap SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto initrd /initramfs-2.6.32-754.6.3.el6.x86_64.img title Jm diz RUIM - SME Server 9.2 (2.6.32-754.10.1.el6.x86_64) root (hd0,0) kernel /vmlinuz-2.6.32-754.10.1.el6.x86_64 ro rd_NO_PLYMOUTH root=/dev/mapper/main-root rd_NO_LUKS LANG=en_US.UTF-8 rd_MD_UUID=b3034d6a:23ab9f17:3fa8e16b:5ef3a368 rd_LVM_LV=main/root nodmraid rd_LVM_LV=main/swap SYSFONT=latarcyrheb-sun16 KEYBOARDTYPE=pc KEYTABLE=us rd_NO_DM rhgb quiet crashkernel=auto initrd /initramfs-2.6.32-754.10.1.el6.x86_64.img
Note I´ve increased timeout to allow user to see menu and choose right one (the oldest) if necessary and removed the 754-9 option, because I removed it as package using RPM.

Anyone had same problem ?
How can I diagnose the kernel panic error ?

Thanks,

Jáder
Title: Re: Kernel Panic on new kernels
Post by: ReetP on February 14, 2019, 04:21:25 PM: First question I can think of is why have you got kernel-devel packages installed??

Have you been rolling your own stuff?

Next, have you had a read of some links on the interwbs?

Have a search for say:

kernel 2.6.32-754 panic RAMDISK incomplete write

This is a pain.... behind their firewall as usual.

https://access.redhat.com/solutions/715833

Insufficent /boot space?

https://www.centos.org/forums/viewtopic.php?t=3864

Maybe causing corruption of the intramfs image?

http://www.thesmartadmin.com/ramdisk-corruption-causes-new-rhel-or-centos-to-fail
Title: Re: Kernel Panic on new kernels
Post by: TerryF on February 14, 2019, 06:45:39 PM: Quote from: Jáder on February 14, 2019, 12:35:25 PM
Anyone had same problem ?
How can I diagnose the kernel panic error ?

No issues, HP N54L
[root@xxxxxx ~]# uname -r
2.6.32-754.10.1.el6.x86_64

[root@xxxxxx ~]# rpm -qa|grep kernel
dracut-kernel-004-411.el6.noarch
kernel-2.6.32-754.6.3.el6.x86_64
kernel-firmware-2.6.32-754.10.1.el6.noarch
kernel-2.6.32-754.9.1.el6.x86_64
kernel-2.6.32-754.10.1.el6.x86_64
kernel-headers-2.6.32-754.10.1.el6.x86_64

What ReetP said...
Title: Re: Kernel Panic on new kernels
Post by: Jáder on February 16, 2019, 12:11:08 PM: Quote from: ReetP on February 14, 2019, 04:21:25 PM
First question I can think of is why have you got kernel-devel packages installed??

Have you been rolling your own stuff?
I'm really not sure. I'm not developing nothing!
EDIT: I think I've found out: I'm using VirtualBox and most of time it need to be compiled to load when new kernel is installed. Anyway that would not represent a threat or failure point se early on boot process.

Quote
Next, have you had a read of some links on the interwbs?
Have a search for say:

kernel 2.6.32-754 panic RAMDISK incomplete write

This is a pain.... behind their firewall as usual.

https://access.redhat.com/solutions/715833
I'll start reading now

Quote
Insufficent /boot space?
https://www.centos.org/forums/viewtopic.php?t=3864
Maybe causing corruption of the intramfs image?
http://www.thesmartadmin.com/ramdisk-corruption-causes-new-rhel-or-centos-to-fail

No
There are lot's of boot space , at least now I removed the other kernel.
Code: [Select]
[root@andorinha ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/main-root 1.8T 746G 988G 44% / tmpfs 3.9G 666M 3.2G 18% /dev/shm /dev/md0 239M 53M 174M 24% /boot
Title: Re: Kernel Panic on new kernels
Post by: Jáder on February 16, 2019, 12:21:34 PM: Quote from: TerryF on February 14, 2019, 06:45:39 PM
No issues, HP N54L
[root@xxxxxx ~]# uname -r
2.6.32-754.10.1.el6.x86_64

[root@xxxxxx ~]# rpm -qa|grep kernel
dracut-kernel-004-411.el6.noarch
kernel-2.6.32-754.6.3.el6.x86_64
kernel-firmware-2.6.32-754.10.1.el6.noarch
kernel-2.6.32-754.9.1.el6.x86_64
kernel-2.6.32-754.10.1.el6.x86_64
kernel-headers-2.6.32-754.10.1.el6.x86_64

What ReetP said...

I have several N40L not showing problem at all. Regards...
Title: Re: Kernel Panic on new kernels
Post by: Jean-Philippe Pialasse on February 17, 2019, 09:29:36 AM: I remember having seen something similar here or on CentOs forums. Was a failure compiling the initrd on the update before reboot.
Strange it has happen on two kernel in a row.
as far as i remember the quick fix was to rebuild the initrd manually
Title: Re: Kernel Panic on new kernels
Post by: Jean-Philippe Pialasse on February 17, 2019, 09:37:36 AM: Quote from: Jáder on February 16, 2019, 12:11:08 PM
No
There are lot's of boot space , at least now I removed the other kernel.
Code: [Select]
[root@andorinha ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/mapper/main-root 1.8T 746G 988G 44% / tmpfs 3.9G 666M 3.2G 18% /dev/shm /dev/md0 239M 53M 174M 24% /boot

I would head that direction
53M is not a lot, and even less sufficient if you had you just removed things to get that.
each new kernel will need at least 25Mo
SO this is probable the last two kernels had an issue for space.
Title: Re: Kernel Panic on new kernels
Post by: Jáder on February 18, 2019, 12:29:57 AM: Quote from: Jean-Philippe Pialasse on February 17, 2019, 09:37:36 AM
I would head that direction
53M is not a lot, and even less sufficient if you had you just removed things to get that.
each new kernel will need at least 25Mo
SO this is probable the last two kernels had an issue for space.
I have 174MB free...anyways I'd like to try the rebuild of initrd ... can you help me about this or just following standard linux procedure would be fine ?

EDIT: moved forward : Moved old bad copy (754-10) as .bad and run:
Code: [Select]
dracut -f /boot/initramfs-2.6.32-754.10.1.el6.x86_64.img 2.6.32-754.10.1.el6.x86_64Now I have:
Code: [Select]
[root@andorinha ~]# ls /boot/initramfs-2.6.32-754.10* -lart -rw------- 1 root root 18727102 Jan 19 16:11 /boot/initramfs-2.6.32-754.10.1.el6.x86_64.img.bad -rw------- 1 root root 18726924 Feb 17 20:34 /boot/initramfs-2.6.32-754.10.1.el6.x86_64.img [root@andorinha ~]#
So I may be victim of a lack of free space, and that would be verified by installer. :)
I'll try if this new initram works and open a NFR to verify this lack of free space.
Title: Re: Kernel Panic on new kernels
Post by: ReetP on February 19, 2019, 04:38:24 PM: It is probably enough to allow for a couple of kernels with older ones being auto removed, but not enough when you start adding your own kernels.

It isn't a bug per se with a stock install but may need notes adding to the virtualbox wiki page.