Topic: r8169 driver problem, want to test optional kernel image

[reidar]

I am having extremly poor performance on my local eth0 Gigabit network card:

01:0a.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL-8169 Gigabit Et
hernet (rev 10)

The card is using the r8169 driver. I have seen on the net that there have been several issues with this driver. I am currently running the 2.6.18-194.11.1 kernel, but it looks like the SME 8 also is shipped with an optional 2.6.18-238.9.1 kernel. I would like to test this kernel to see if the r8169 driver performs better. My grub boot up menu only have one entry being the 2.6.18-194.11.1 image. My /boot/grub/grub.conf file looks like this:

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,0)
#          kernel /vmlinuz-version ro root=/dev/main/root
#          initrd /initrd-version.img
#boot=/dev/md1
default=1
timeout=5
splashimage=(hd1,0)/grub/smeserver.xpm.gz
foreground 000000
background 4E95D3
title SME Server (2.6.18-238.9.1.el5PAE)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-238.9.1.el5PAE ro root=/dev/main/root
        initrd /initrd-2.6.18-238.9.1.el5PAE.img
title SME Server (2.6.18-194.11.1.el5PAE)
        root (hd0,0)
        kernel /vmlinuz-2.6.18-194.11.1.el5PAE ro root=/dev/main/root
        initrd /initrd-2.6.18-194.11.1.el5PAE.img

I have the following questions:

1) Why do I not see the 2.6.18-238.9.1 in the boot menu. Do I have to run som grub command to activate the conf file?

2) Are there any known issues preventing me from using the 2.6.18-238.9.1 kernel?

3) If the above does not work are there any ways to configure the NIC through using ethtool that could fix this driver performance bug?

[CharlieBrady]

Use the up arrow key to select 2.6.18-238.9.1.el5PAE at boot time.

[reidar]

I understand that I can use the up/down arrows to navigate the boot menu. The poblem is that there is only one entry at boot time being the 2.6.18-194.11.1 kernel. My question was, why do not the startup boot menu reflect the grub.conf that have two entries. The grub.conf located in the /boot/grub and the /etc are equal. Is there a third file that the grub loader might be picking up?

[Jáder]

[...] The grub.conf located in the /boot/grub and the /etc are equal. Is there a third file that the grub loader might be picking up?

Their are same file.
Are you sure you have the new kernel update ? If so, try to expand templates to regenerate grub.conf using

Code: [Select]

expand-template /boot/grub/grub.conf


[CharlieBrady]

I understand that I can use the up/down arrows to navigate the boot menu. The poblem is that there is only one entry at boot time being the 2.6.18-194.11.1 kernel. My question was, why do not the startup boot menu reflect the grub.conf that have two entries.

Show:

cat /proc/mdstat

My guess is that you have a degraded RAID1 array. The grub menu you see at boot time is your first hard drive, but that drive is not being used at runtime. Therefore it is not being updated with changes.

[reidar]

Thanks a lot for your support on this issue!

Here is the output of the mdstat command.

-bash-3.2$ cat /proc/mdstat
Personalities : [raid1]
md1 : active raid1 sda1[0]
      104320 blocks [2/1] [U_]

md2 : active raid1 sda2[0]
      976655488 blocks [2/1] [U_]

unused devices: <none>

[reidar]

I guess the output of the mdstat states that one of the raid disks are not working?

Could this be caused by having a usb disk plugged in at installation time?

I have two 1TB sata disks and a 300GB usb disk used for backup, see the system info below. The idea is of course to use the two 1TB disks for the RAID. I am currently using the affa contrib for backup to the usb disk and it works fine.

What is the cure for the RAID problem, I guess I have to run some mdadm commands. How do I identify which one of the disks are ok to be mirrored to the not ok disk and what commands should I use?

Any help is greatly appreciated!

== BEGIN fdisk -l ==
Disk /dev/md2 doesn't contain a valid partition table
Disk /dev/md1 doesn't contain a valid partition table

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          13      104391   fd  Linux raid autodetect
/dev/sda2              14      121601   976655610   fd  Linux raid autodetect

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          13      104391   fd  Linux raid autodetect
/dev/sdb2              14      121601   976655610   fd  Linux raid autodetect

Disk /dev/md2: 1000.0 GB, 1000095219712 bytes
2 heads, 4 sectors/track, 244163872 cylinders
Units = cylinders of 8 * 512 = 4096 bytes


Disk /dev/md1: 106 MB, 106823680 bytes
2 heads, 4 sectors/track, 26080 cylinders
Units = cylinders of 8 * 512 = 4096 bytes


Disk /dev/sdc: 300.0 GB, 300090728448 bytes
255 heads, 63 sectors/track, 36483 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1               1       36483   293049666   83  Linux
== END   fdisk -l ==

[reidar]

From the mdstat information I take it is the sdb disk that is the problem. I ran the command "smartctl  -t long /dev/sdb" and verified that there is no problem with the disk.

The command "grep sdb /var/log/messages" gives the following output:
[root@e-smith ~]# grep sdb /var/log/messages
May 29 08:58:55 e-smith kernel: SCSI device sdb: 1953525168 512-byte hdwr sectors (1000205 MB)
May 29 08:58:55 e-smith kernel: sdb: Write Protect is off
May 29 08:58:55 e-smith kernel: sdb: Mode Sense: 00 3a 00 00
May 29 08:58:55 e-smith kernel: SCSI device sdb: drive cache: write back
May 29 08:58:55 e-smith kernel: SCSI device sdb: 1953525168 512-byte hdwr sectors (1000205 MB)
May 29 08:58:55 e-smith kernel: sdb: Write Protect is off
May 29 08:58:55 e-smith kernel: sdb: Mode Sense: 00 3a 00 00
May 29 08:58:55 e-smith kernel: SCSI device sdb: drive cache: write back
May 29 08:58:55 e-smith kernel:  sdb: sdb1 sdb2
May 29 08:58:55 e-smith kernel: sd 3:0:0:0: Attached scsi disk sdb
May 29 08:58:55 e-smith kernel: md: considering sdb2 ...
May 29 08:58:55 e-smith kernel: md:  adding sdb2 ...
May 29 08:58:55 e-smith kernel: md: sdb1 has different UUID to sdb2
May 29 08:58:55 e-smith kernel: md: sda1 has different UUID to sdb2
May 29 08:58:55 e-smith kernel: md: bind<sdb2>
May 29 08:58:55 e-smith kernel: md: running: <sdb2><sda2>
May 29 08:58:55 e-smith kernel: md: kicking non-fresh sdb2 from array!
May 29 08:58:55 e-smith kernel: md: unbind<sdb2>
May 29 08:58:55 e-smith kernel: md: export_rdev(sdb2)
May 29 08:58:55 e-smith kernel: md: considering sdb1 ...
May 29 08:58:55 e-smith kernel: md:  adding sdb1 ...
May 29 08:58:55 e-smith kernel: md: bind<sdb1>
May 29 08:58:55 e-smith kernel: md: running: <sdb1><sda1>
May 29 08:58:55 e-smith kernel: md: kicking non-fresh sdb1 from array!
May 29 08:58:55 e-smith kernel: md: unbind<sdb1>
May 29 08:58:55 e-smith kernel: md: export_rdev(sdb1)
May 29 08:58:56 e-smith kernel: md: considering sdb1 ...
May 29 08:58:56 e-smith kernel: md:  adding sdb1 ...
May 29 08:58:56 e-smith kernel: md: sdb2 has different UUID to sdb1
May 29 08:58:56 e-smith kernel: md: md1 already running, cannot run sdb1
May 29 08:58:56 e-smith kernel: md: export_rdev(sdb1)
May 29 08:58:56 e-smith kernel: md: considering sdb2 ...
May 29 08:58:56 e-smith kernel: md:  adding sdb2 ...
May 29 08:58:56 e-smith kernel: md: md2 already running, cannot run sdb2
May 29 08:58:56 e-smith kernel: md: export_rdev(sdb2)

Can anyone tell me why the sdb partitions can not be added to the md? How should I troubleshoot this further? How can I manually fix the problem?

[janet]

reidar

You need to "clean" the sdb drive before trying to make it part of the array again.
Use the dd command that is referred to in the wiki article  at http://wiki.contribs.org/Raid or download delpart.exe from net, put it on a boot CD or floppy and then delete the partitions of sdb. There are other suitable software tools around too.

Then use the admin console in SME to add the drive to the array & it will resync

[CharlieBrady]

From the mdstat information I take it is the sdb disk that is the problem. I ran the command "smartctl  -t long /dev/sdb" and verified that there is no problem with the disk.
....
Can anyone tell me why the sdb partitions can not be added to the md?

The log file is already telling you. sdb1 and sdb2 are marked as "non-fresh".  You haven't told us your system history, so we don't know when that might have occurred. Was this a fresh install with both drives in place, and you have never booted the system with only one drive in place?

The evidence suggests that you have two different problems. One is that sdb1 and sdb2 is not being included in the RAID1 arrays. The other is that your system seems to be booting from sdb (it's booting from a grub configuration which is different than the one you can see on the /boot partition).

How can I manually fix the problem?

You can add sdb1 and sdb2 into md1 and md2 using:

mdadm --add /dev/md1 /dev/sdb1
mdadm --add /dev/md2 /dev/sdb2

You also need to investigate why grub seems not to be installed or not working on sda.

[reidar]

Once again, thanks a lot for your help. This has been a good learning experience!

I managed to bring the sdb disk back into the RAID array by utilizing the mdadm add command. So, the first part of the problem is fixed.

The other is that your system seems to be booting from sdb (it's booting from a grub configuration which is different than the one you can see on the /boot partition).

The boot sequence defined in the bios is probably set to boot the logical mapped sdb disk before the sda disk. But, when the disks are properly mirrored, it should not matter which one is first. Should I change the bios boot sequence to boot sda before sdb?

Anyway, after synchronizing the disks, the correct boot menu appeared in the next boot and I managed to boot from the newest kernel. Unfortunately this did not fix my network problem. Every now and then the card does not come up on boot and I have the performance issue. Copying a 0.2 GB file over the local Gigabit network from a client to a server disk takes approximately 1 hour. The weird thing is that this used to be OK, something has happened. I suspect it is connected to getting more files on the disk and a connection to samba as described here: http://bugs.centos.org/view.php?id=3218. It is different from this bug since I also have write problems. My question is then, are there any plans for a kernel update on the SME8 in the near future that might fix this problem?

You haven't told us your system history, so we don't know when that might have occurred.

The system is a fresh install from a CD with both disks installed. I have never intentionally booted the system with only one disk. The following is what I suspect has happened:

1. Fresh install on new hardware, 2 disks no USB and no internet connection plugged in.

2. Boot with USB plugged in and an external net connection. The boot took forever probably because it was trying to boot from my USB, but it finally came up. I suspect that it only came up with the sda disk. I did a yum update causing the new kernel to be installed on the sda disk, I also installed the affa contrib and did a restore from the USB.

3. In the next boot , I most probably changed the bios boot sequence due to the problem above. The boot finished in a normal time and I came up with both disks, but a degraded array. Since the bios boot sequence is booting sdb before sda, the boot menu was reflecting the grub.conf on the sdb disk which originated from the CD install and not the grub.conf on the sda originating from the yum update.

[reidar]

Just to conclude on the issue in case anyone was searching the forum for information on the NIC.

It turned out that the low performance of the LAN was caused by a bug in the firmware of the router on the other side of the cable. After fixing the router, the LAN perfoms OK, issue closed.