Topic: Disk IO issues with SME 8

[mhr]

Hi,

I've recently built a new server based on an Asus AT5NM10T-I (Atom D525), running SME8 latest beta. I have tried two system disks: a 2.5" Hitachi 750GB and a 3.5" WD 2TB disk, both connected to the first on-board SATA connector in AHCPI mode. This system is to replace an 8yrs old Via C3/1GHz system running SME 7.5.1.

While the installation went smoothly, and hdparm is reporting reasonable numbers, this new system sometimes would stall for a few seconds on simple things like a "ls" in a folder with one single file. Or having Logitech Media Server scan my music collection would be slower than on the old system. Profiling showed that database commits were several(!) times slower than on the old box (seconds vs. a few tens of milliseconds). The process would kind of idle during the commits, CPU wouldn't show any significant load then.

Does anybody have an idea what might be causing such problems with these disks or the controller?

Michael

[Jáder]

1) bad disk (if it's only in some parts of filesystem)

2) if HDD is slower all times: may be a bad configuration of BIOS (SATA in compatiblity mode and not native mode).
BE AWARE that change the BIOS mode can compromise your data ... YOU CAN LOST EVERYTHING.
I'm not sure... just warning you... to be the safe side!

3) I'm sure is not SME8 neither Atom CPU... remember you're using UltraDMA (right?) so CPU is not a factor here.

[Michail Pappas]

Not an expert here, just trying to help

Are the disks seen as /dev/hdX or as /dev/sdX? If it is the former, there's a BIOS setting that will give you a 10x improvement in performance. IIRC, it's about switching the IDE controller from "legacy" operation to "enhanced".

If it's not that, check your dmesg output for hints of problems. Does the issue appear on either disk? Which is the output of:

Code: [Select]

smartctl -a /dev/sda
smartctl -a /dev/sdb
smartctl -a /dev/had
smartctl -a /dev/hdb


[mhr]

Thanks for all your input. Disks are recognized as sdX. There's nothing obvious in dmesg. Anything particular I should be looking for?

Code: [Select]

smartctl version 5.38 [i686-redhat-linux-gnu] Copyright (C) 2002-8 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

=== START OF INFORMATION SECTION ===
Device Model:     Hitachi HTS547575A9E384
Serial Number:    J2140054G43AYA
Firmware Version: JE4OA40J
User Capacity:    750,156,374,016 bytes
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   8
ATA Standard is:  Not recognized. Minor revision code: 0x28
Local Time is:    Thu Feb 23 22:15:58 2012 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (  45) seconds.
Offline data collection
capabilities: (0x5b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: (   2) minutes.
Extended self-test routine
recommended polling time: ( 201) minutes.
SCT capabilities:        (0x003d) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000b   100   100   062    Pre-fail  Always       -       0
  2 Throughput_Performance  0x0005   100   100   040    Pre-fail  Offline      -       0
  3 Spin_Up_Time            0x0007   166   166   033    Pre-fail  Always       -       1
  4 Start_Stop_Count        0x0012   100   100   000    Old_age   Always       -       57
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000b   100   100   067    Pre-fail  Always       -       0
  8 Seek_Time_Performance   0x0005   100   100   040    Pre-fail  Offline      -       0
  9 Power_On_Hours          0x0012   100   100   000    Old_age   Always       -       369
 10 Spin_Retry_Count        0x0013   100   100   060    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       57
191 G-Sense_Error_Rate      0x000a   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       9
193 Load_Cycle_Count        0x0012   096   096   000    Old_age   Always       -       40102
194 Temperature_Celsius     0x0002   253   253   000    Old_age   Always       -       22 (Lifetime Min/Max 16/32)
196 Reallocated_Event_Count 0x0032   100   100   000    Old_age   Always       -       1
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0
223 Load_Retry_Count        0x000a   100   100   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]


SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Michael

[Michail Pappas]

If this happens, regardless of disk used, then it might point towards a motherboard/hdd controller issue.

In BIOS there are two settings related to controller operation: "Configure SATA as [IDE]" and SATA "Run Mode Configuration". Ideally they should be set to AHCI and Enhanced, respectively. If they are not set like this, set them and retry to see what happens. If they are already set, change the former to "IDE", but leave the latter to "Enhanced". Any change?

[mhr]

Changing from ahcpi to ide would require to re-install everything, right? Not something I'd want to do... Thanks anyway!

[Jáder]

Even if you do not want to do... may be the only way.
I suggest you to get another HDD and try it on.
But I saw in your initial post you're using AHCPI.

Could you post your "hdparm -tT /dev/sdX" results so we can see how fast are your disks now ?
How much memory do have this config ? 2GB ? 4GB ?  What database do you use ?
Are your server using swap ?
Do you have sysinfo installed  to monitor what's going on in your server ?
I have a personal install of Zabbix to monitor servers, can be use to monitor lot's of items... I could get data from you... you just should install agent and report data to my server. Are you interested ?

Jáder

[mhr]

/dev/sda:
 Timing cached reads:   1782 MB in  2.00 seconds = 891.04 MB/sec
 Timing buffered disk reads: 216 MB in  3.02 seconds =  71.55 MB/sec

4GB RAM, lots free:
             total       used       free     shared    buffers     cached
Mem:       4137968    1099724    3038244          0     271280     201040
Swap:      5144568         88    5144480

DB is SQLite

[Michail Pappas]

Changing from ahcpi to ide would require to re-install everything, right?

Are you sure about that? AFAIK changing to IDE never harms anything. Plus changing to AHCI is an issue (again AFAIK) in operating systems where there is no AHCI driver included in the default installation (like XP). Not an issue in SME 8...

[Jáder]

/dev/sda:
 Timing cached reads:   1782 MB in  2.00 seconds = 891.04 MB/sec
 Timing buffered disk reads: 216 MB in  3.02 seconds =  71.55 MB/sec

not THAT BAD... These are mine:

Code: [Select]

[root@guepardo ~]# hdparm -tT /dev/sda

/dev/sda:
 Timing cached reads:   2992 MB in  2.00 seconds = 1496.70 MB/sec
 Timing buffered disk reads:  236 MB in  3.01 seconds =  78.39 MB/sec
Note my HDD is almost twice as fast as yours.

And about my HW:

Code: [Select]

[root@guepardo ~]# inxi -CFfpluorsic10
System:    Host: guepardo Kernel: 2.6.18-274.18.1.el5 i686 (32 bit) Console tty 1 Distro: SME Server release 8.0beta7
Machine:   No machine data found. Your kernel does not support this feature.
CPU:       Single core Intel Pentium 4 CPU (-HT-) cache: 1024 KB
           Clock Speeds: 1: 3059.499 MHz 2: 3059.499 MHz
           CPU Flags: fpu vme de pse tsc msr pae mce cx8 apic mtrr pge mca cmov pat pse36 clflush dts acpi
           mmx fxsr sse sse2 ss ht tm pbe nx lm constant_tsc pni monitor ds_cpl tm2 cid cx16 xtpr
Graphics:  Card: Intel 82915G/GV/910GL Integrated Graphics Controller
           X-Vendor: N/A driver: tty size: 100x24 Advanced Data: N/A for root out of X
Audio:     Card: Intel 82801FB/FBM/FR/FW/FRW (ICH6 Family) AC'97 Audio Controller driver: Intel ICH
           Sound: Advanced Linux Sound Architecture ver: 1.0.14rc3
Network:   Card-1: Broadcom NetXtreme BCM5751 Gigabit Ethernet PCI Express driver: tg3
           IF: eth0 state: up speed: N/A duplex: N/A mac: 00:11:43:d0:0a:50
           Card-2: Realtek RTL-8029(AS) driver: ne2k-pci
           IF: eth1 state: unknown speed: N/A duplex: N/A mac: 00:80:ad:78:46:2c
           WAN IP: 177.18.60.212 IF: bond0 ip: N/A IF: sit0 ip: N/A IF: eth0 ip: 192.168.13.1
           IF: eth1 ip: N/A IF: ppp0 ip: 177.18.60.212
Drives:    HDD Total Size: 1000.2GB (66.4% used) 1: /dev/sda ST31000333AS 1000.2GB
Partition: ID: / size: 899G used: 619G (73%) fs: ext3 dev: /dev/mapper/main-root label: N/A uuid: N/A
           ID: /boot size: 99M used: 25M (27%) fs: ext3 dev: /dev/md1
           label: N/A uuid: 60b4f274-ed62-44cf-bc07-ec2761cdaab1
           ID: swap-1 size: 4.23GB used: 0.00GB (0%) fs: swap dev: /dev/mapper/main-swap label: N/A uuid: N/A
Unmounted: ID: /dev/sda1 size: 0.11G label: N/A uuid: 1343e6bf-2087-da80-c360-062e97c4f263 fs: ext3
           ID: /dev/sda2 size: 1000.10G label: N/A uuid: 2ac61081-7ef1-151e-2f1c-7351ad1f8e4b
           ID: /dev/md2 size: 1000.10G label: N/A uuid: N/A
           ID: /dev/dm-0 size: 995.86G label: N/A uuid: N/A
           ID: /dev/dm-1 size: 4.23G label: N/A uuid: N/A
Sensors:   Error: You do not have the sensors app installed.
Repos:     Active apt sources in file: /etc/apt/sources.list.d/rpmforge.list
           repomd http://apt.sw.be redhat/el$(VERSION)/en/$(ARCH)/dag
Info:      Processes: 253 Uptime: 1 day Memory: 671.8/2024.0MB Runlevel: 7 Client: Shell inxi: 1.7.23

and my HDD:

Code: [Select]

[root@guepardo ~]# hdparm -i /dev/sda

/dev/sda:

 Model=ST31000333AS                            , FwRev=CC1H    , SerialNo=            9TE1WCN1
 Config={ HardSect NotMFM HdSw>15uSec Fixed DTR>10Mbs RotSpdTol>.5% }
 RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=4
 BuffType=unknown, BuffSize=0kB, MaxMultSect=16, MultSect=?8?
 CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=268435455
 IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
 PIO modes:  pio0 pio1 pio2 pio3 pio4
 DMA modes:  mdma0 mdma1 mdma2
 UDMA modes: udma0 udma1 udma2
 AdvancedPM=no WriteCache=enabled
 Drive conforms to: unknown:  ATA/ATAPI-4 ATA/ATAPI-5 ATA/ATAPI-6 ATA/ATAPI-7

 * signifies the current active mode
now tell me WHEN do you feel the slowing of your server... all times ?
Can I replicate this on my own server ? How ?

Regards

Jáder

[mhr]

Simplest to reproduce is probably running Logitech Media Server's built in scanner. It's slower on this new box than on my 7+ years old Via C3 1GHz...

[Jáder]

Simplest to reproduce is probably running Logitech Media Server's built in scanner. It's slower on this new box than on my 7+ years old Via C3 1GHz...

Could you get another Sata HDD to install on this new motherboard/computer changing BIOS to test HDD speed ?
Your HDD is half speed than mine!

I presume your old HDD (from Via C3) it's an IDE HDD so their hdparm numbers should be much lower than SATA numbers.
If you have slow response from a simple ls ... I'm sure something is wrong.

I think it's strange how many RAM is not used as cache on your server. See mine:

Code: [Select]

free [root@guepardo ~]# free -m
             total       used       free     shared    buffers     cached
Mem:          2024       1968         55          0        492        766
-/+ buffers/cache:        709       1314
Swap:         4031          0       4031
[root@guepardo ~]#
I  have only 55MB free from 2GB!
And your appear to use just 1GB of 4GB ? (please repeat command as "free -m")
(and format the info using the "#"(for code) tool ... it's so easy to do, easier to read).

Jáder

[alexeym]

hi Mhr,

have you solved your issue? Looks like I have the same shitty IO performance...

[mhr]

No, unfortunately I haven't resolved the issue, I'm just living with it :-/

[alexeym]

damn...

tell me, your 3.5" WD 2TB is Green or not?