Topic: Waking up of SME 9.1 using WoL (SME server used for Affa)

[TerryF]

OK definitely SME related, confirm behaviour on a HP Microserver..very very similar to old SME 8 bug as quoted above.

I will also look at using an older kernel and see if its an update to a module as with the olds SME 8 bug

If it is then a bug needs to be raised, in CentOS and RH

[TerryF]

Nope..going back three updates still evident..bug please

[TerryF]

See comment 45 in the sme8 bug https://bugs.contribs.org/show_bug.cgi?id=7928

WOL was working in SME9b2 note driver versions then
[root@sme9b2 ~]# modinfo tg3 | grep -i version
version:        3.124
srcversion:     849BDFC1D60C5393F1EC438
vermagic:       2.6.32-358.23.2.el6.x86_64 SMP mod_unload modversions

SME 9.1 driver version
[root@fagehome ~]# modinfo tg3 | grep -i version
version:        3.137
srcversion:     1E40B98DF422DE93879F7FF
vermagic:       2.6.32-642.11.1.el6.x86_64 SMP mod_unload modversions

[TerryF]

My observations, On a HP Microserver, server/gateway, using a Intel® Gigabit CT Desktop Adapter (Intel® 82574L Gigabit Ethernet Controller) issuing a magic packet from a windows 10 pc wakes the server up from a power down after using console to shutdown. So appears to be a driver issue.

Are you setup as server only or server/gateway.

If server/gateway are you issuing the wol directive to the right nic?

Swapping of nics has been observed when installing sme9 in the past

Did you try issuing # ethtool -s eth0 wol g and then shutdown and try the wol directive again?

[turandot]

My server hardware has just a single NIC, so confusion between various NICs is not possible. Consequently I am running the server in the "server only" mode.

Applying the command...

Code: [Select]

ethtool -s eth0 wol g...just before shutdown does not make any difference: WoL is not working with SME 9.1.

I also experimented with nested commands

Code: [Select]

ifconfig eth0 down; signal-event haltno difference: WoL is not working with SME 9.1

I also checked /var/log/messages for errors during shutdown, and I did not find any error message at all. Same on the local console during shutdown: no error message is displayed.

This is the output of /proc/acpi/wakeup under SME 9.1:

Code: [Select]

[root@smebackup ~]# cat /proc/acpi/wakeup
Device  S-state   Status   Sysfs node
P0P2      S4     disabled
P0P1      S4     disabled  pci:0000:00:1e.0
EUSB      S4     disabled  pci:0000:00:1d.7
MC97      S4     disabled
P0P4      S4     disabled  pci:0000:00:1c.0
P0P5      S4     disabled  pci:0000:00:1c.1
P0P6      S4     disabled  pci:0000:00:1c.2
P0P7      S4     disabled  pci:0000:00:1c.3
P0P8      S4     disabled
P0P9      S4     disabled
USB0      S3     disabled  pci:0000:00:1d.0
USB1      S3     disabled  pci:0000:00:1d.1
USB2      S3     disabled  pci:0000:00:1d.2
USB3      S3     disabled  pci:0000:00:1d.3
[root@smebackup ~]#
Situation under SME 8.2:

Code: [Select]

[root@smebackup ~]# cat /proc/acpi/wakeup
Device  Sleep state     Status
P0P2       4            disabled
P0P1       4            disabled
EUSB       4            disabled
MC97       4            disabled
P0P4       4            disabled
P0P5       4            disabled
P0P6       4            disabled
P0P7       4            disabled
P0P8       4            disabled
P0P9       4            disabled
USB0       3            disabled
USB1       3            disabled
USB2       3            disabled
USB3       3            disabled
[root@smebackup ~]#
It is interesting to note that the LAN device is neither listed under SME 8.2 nor under 9.1. Despite of that, WoL is working under SME 8.2.

[TerryF]

So it looks to be a specific driver issue..

Just to confirm you have a Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet card?

Althought not SME/CentOS/RH, still worth a read
https://bbs.archlinux.org/viewtopic.php?id=192514
https://wiki.archlinux.org/index.php/Wake-on-LAN#Realtek
Plenty of other refs from Mr Google to do with issues

[TerryF]

SME9.1 fully updated shows following

[root@fagehome ~]# modinfo r8168 | grep -i version
ERROR: modinfo: could not find module r8168
[root@fagehome ~]# modinfo r8169 | grep -i version
version:        2.3LK-NAPI
srcversion:     3587FAC210AABB4E2A8CB0F
vermagic:       2.6.32-642.11.1.el6.x86_64 SMP mod_unload modversions

Perhaps try an updated driver.. http://rpm.pbone.net/index.php3/stat/4/idpl/30514405/dir/redhat_el_6/com/#provides

Test box only, a bit dodgy on a prod box...good luck

[turandot]

On SME 8.2:

Code: [Select]

[root@smebackup ~]# lspci -nn | grep Realtek
01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 02)
[root@smebackup ~]#

On SME 9.1:

Code: [Select]

[root@smebackup ~]# lspci -nn | grep Realtek
01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 02)
[root@smebackup ~]#

On SME 8.2:

Code: [Select]

[root@smebackup ~]# modinfo r8168 | grep -i version
modinfo: could not find module r8168
[root@smebackup ~]# modinfo r8169 | grep -i version
version:        2.3LK-1-NAPI
srcversion:     2F438BE7BC0FF431B08F6F4
[root@smebackup ~]#

On SME 9.1:

Code: [Select]

[root@smebackup ~]# modinfo r8168 | grep -i version
ERROR: modinfo: could not find module r8168
[root@smebackup ~]# modinfo r8169 | grep -i version
version:        2.3LK-NAPI
srcversion:     3587FAC210AABB4E2A8CB0F
vermagic:       2.6.32-642.11.1.el6.x86_64 SMP mod_unload modversions
[root@smebackup ~]#

The NIC is integrated on the motherboard of the system MSI Wind 2713 NetTop, which does not come with any expansion slot. So I don't have any chance to test whatsoever other NIC. I checked the Bios version: it is the latest version available in the Internet. The link LED stays on, when the system is powered down. I am a bit reluctant to make any conclusion, but my gut feeling is pointing to a driver issue as well.

a bit dodgy on a prod box...

This is the point And I don't have too much experience to fumble with kernel modules...

To test this, I downloaded the kernel module as given here: http://rpm.pbone.net/index.php3/stat/4/idpl/30514405/dir/redhat_el_6/com/#provides and pushed it to SME 9.1 to the directory /root . Then I installed it using yum install /root/kmod-r8168-8.040.00-1.el6.elrepo.x86_64.rpm I was surprised, because this took quite a while, but it worked successfully.

Following that, I applied modprobe -r r8169 and modprobe r8168 . This worked as well. Please remember that I don't have too much experience, I tried to step back applying modprobe -r r8168 , which works. However when then applying modprobe r8169 , a kernel panic results, and the system crashes. I can reproduce this: when I reboot and repeat modprobe -r r8168 and modprobe r8169 , I will get another kernel panic.

Booting up again (without applying the manual modprobe commands), everything seems to work fine again. However I am struggling to understand, which kernel module is currently operational:

Code: [Select]

[root@smebackup ~]# modprobe -l r8168
weak-updates/r8168/r8168.ko
[root@smebackup ~]# modprobe -l r8169
kernel/drivers/net/r8169.ko
[root@smebackup ~]#
[root@smebackup ~]# modinfo r8168 | grep -i version
version:        8.040.00-NAPI
srcversion:     BC1CAF8ECC36006913B85E5
vermagic:       2.6.32-279.el6.x86_64 SMP mod_unload modversions
[root@smebackup ~]# modinfo r8169 | grep -i version
version:        2.3LK-NAPI
srcversion:     3587FAC210AABB4E2A8CB0F
vermagic:       2.6.32-642.11.1.el6.x86_64 SMP mod_unload modversions
[root@smebackup ~]#

Interestingly: WoL now works in this configuration!!! 

Do you have any recommendation what I should do? Most important questions I have:

• Which kernel module is currently working: r8168, or r8169?
• Should I operate a productive system with this manually added kernel module?

[TerryF]

Cheers all round good result, confirms its the driver module, well to me it does.

What next? file a bug with the results of your testing and the outcome with the newer driver module, this also needs to be done at CentOS and Red Hat, all good fun

From my understanding of discussions in other forums etc the nic is now using the r8168 driver.

As I understand it, a kernel upgrade will revert to whatever is in that kernel, "I think " I am sure someone will correct me if that is otherwise.

There is also a newer driver than the one you used:  http://syd.mirror.rackspace.com/elrepo/elrepo/el6/x86_64/RPMS/ I also presume if you enable the elrepo repo you will benefit from updates using yum, again I stand to be corrected..

Fun isn't it

[turandot]

Fun isn't it

For sure: you always stumble across stuff you don't know up to now, and an interesting journey what you see and learn

Sorry to be a bit niggling: would you recommend to run this setup on a productive machine?

And you are really recommending to open a bug on https://bugzilla.redhat.com/ ? Against RHEL 6 and kmod r8169? Seems (at least to me) honorable...

[ReetP]

However I am struggling to understand, which kernel module is currently operational:

Try this to see what is loaded :

Code: [Select]

lsmod |grep r816
You could try blacklisting either module for testing

Have a look in /etc/modprobe.d/blacklist.conf

blacklist nameOfYourModule

Or prevent it loading in /etc/modprobe.d/dist.conf with

alias nameOfYourModule off

Oddly you seem to have a 8168 based card but v8.2 was using the 8189 driver (check it with lsmod and look at your output above "modinfo: could not find module r8168")

Perhaps they updated the 8169 driver in CentOS 6 and made it more specific so you really need the 8168 driver.

Suggest checking which module gets loaded in v8 first and work from there.

B. Rgds
John

[turandot]

Output from SME 8.2 (out of the box):

Code: [Select]

[root@smebackup ~]# lsmod | grep r816
r8169                  43077  0
mii                     9409  1 r8169
[root@smebackup ~]#

And from SME 9.1 (patched with kmod r8168):

Code: [Select]

[root@smebackup ~]# lsmod | grep r816
r8168                 324411  0
[root@smebackup ~]#

Why the heck does SME 9.1 crash with a kernel panic, if I now try to load r8169!? I see only a single explanation: the installation of r8168 changed something, which is not compliant with r8169. Isn't this one more indication, that something is wrong with r8169!?

[TerryF]

Oddly you seem to have a 8168 based card but v8.2 was using the 8189 driver (check it with lsmod and look at your output above "modinfo: could not find module r8168")

Perhaps they updated the 8169 driver in CentOS 6 and made it more specific so you really need the 8168 driver.
B. Rgds
John

That's what the discussion revolves around in the CentOS, arch and Ubuntu forums re r8169 is being hence issues arising other than just wol. Resolved by changing to the r8168 driver.  In our case it seems as thought the version of the r8169 driver ion 8.2 was still working regarding wol but the later version in 9.1 is not.

If wol is required on the production box then yes, elrepo is reputable and a trusted source. Also if no bug is lodged with RH etc then when a kernel update is released you will have to update the kmod elrepo module when they produce an updated rpm.

re opening a bug in RH and CentOS, yes, its how the 8.2 wol bug in the tg3 driver was resolved by RH updating the kernel with the updated  one, just took a couple of months just register on both bugzillas and away you go. There is a Red Hat bug for the 8.2 wol bug, unfortunately kernel bugs are private so I cannot show it to you.

[turandot]

I opened bug report 1410994 "WoL does not work with r8169 on RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 02)" on bugzilla.redhat.com . I have dropped in information, which summarizes the testing as follows:



Description of problem: Wake on LAN (WoL) does not work


Version-Release number of selected component (if applicable):

On RHEL 6.7:

modinfo r8169 | grep -i version
version:        2.3LK-NAPI
srcversion:     3587FAC210AABB4E2A8CB0F
vermagic:       2.6.32-642.11.1.el6.x86_64 SMP mod_unload modversions


How reproducible: 100%


Steps to Reproduce:

1. Install RHEL 5 on a system using HDD1 --> WoL works

On RHEL 5:

modinfo r8169 | grep -i version
version:        2.3LK-1-NAPI
srcversion:     2F438BE7BC0FF431B08F6F4


2. Take out HDD1, swap in HDD2, do fresh install of RHEL 6.7

Actual results: WoL does not work


Expected results: WoL should work


Additional info:

WoL works again with RHEL 6.7, if the main power line was cycled, and the NIC did a cold start. So the NIC "only" refuses the WoL magic packet, if the main power was kept switched on all the time. (Together with the RHEL 5 results, hardware issues can consequently be excluded.)


The r8168 driver for RHEL as supplied here http://rpm.pbone.net/index.php3/stat/4/idpl/30514405/dir/redhat_el_6/com/#provides works i.e. supports WoL.

Steps to achieve that:
a) download driver package kmod-r8168-8.040.00-1.el6.elrepo.x86_64.rpm
b) drop to e.g. /root/kmod-r8168-8.040.00-1.el6.elrepo.x86_64.rpm
c) install by yum install /root/kmod-r8168-8.040.00-1.el6.elrepo.x86_64.rpm
d) modprobe -r r8169
e) modprobe r8168

modinfo r8168 | grep -i version
version:        8.040.00-NAPI
srcversion:     BC1CAF8ECC36006913B85E5
vermagic:       2.6.32-279.el6.x86_64 SMP mod_unload modversions

Having that WoL supporting configuration, I cannot revert to r8169 anymore. Step a) of the following procedure works, but step b)...

a) modprobe -r r8168
b) modprobe r8169

...causes a kernel panic and hence a system crash. After reboot (loading r8168) everything is fine again.



Happy waiting

[TerryF]

Nice If you want to tie it all up neatly you could also raise a bug here and add a link to the RH bug there, at worst the forum posts here and the bug will tell any SME user whats what and how to  get a solution that works while RH get on it.

Great bit of sleuthing, if the last time this occurred is any indication it will be a couple of months before a resolution and a updated kernel appears. RH will list in the kernels changelog the fix when they do it.

Hope you don't mind me to have kept after you to do the triage its just once you go through the process you learn something, get a bit of a kick and show others it is relatively straightforward..

Good Job, nothing better than a bit of global collaboration