sata_via - too slow
Hello,
I am having performance problems on dual opteron server with two attached SATA disks.
Unfortunately, my hacking skills are probably not good enough to provide good bug report. I will at least try to describe what is going on...
Simply put, any disk access blocks almost everything else and the system becomes very unresponsitive to any user actions. Even bash console is very slow...
From what I can see in top or vmstat, both CPUs spend all of their time in 'wait' state.
typical top output...
Cpu0 : 7.5% us, 2.3% sy, 0.0% ni, 11.1% id, 77.8% wa, 0.5% hi, 0.9% si Cpu1 : 9.7% us, 1.1% sy, 0.0% ni, 28.7% id, 60.4% wa, 0.0% hi, 0.0% si
and it is being worse, I can see frequently, that the 'wait' gets frequently over 90%.
from 'top', I can see that almost every program that generates disk access waits most frequently in 'wait_on_buffer', this was my first clue that the delays actually come from the 'disk access' layer, rather that from the filesystem itself. I have also created an ext2 partition, but the 'waits' happened there as well, so I'd naively rule out the reiserfs I am using.
otherwise, I am using kernel 2.6.8-rc2-bk7 right now, but this happens since the time I have the sata disks. it runs on dual opteron system / debian, but not in native mode, all code is i386.
I have tried the kernel both in SMP or UP, with noacpi, iommu=off, preemptible kernel or not without difference....
vmstat output from UP kernel when zipping bigger archive.
0 3 0 532912 68860 367552 0 0 4700 0 2575 448 1 4 0 95 0 3 0 524672 68864 375568 0 0 5296 17044 2406 547 1 4 0 95 0 3 0 516736 68872 383588 0 0 5376 1228 2561 514 1 8 0 91 0 3 0 508160 68884 392248 0 0 5844 4 2652 592 1 5 0 94 0 4 0 506624 68888 393728 0 0 1236 1188 1630 205 1 3 0 96 1 1 0 501200 68892 399224 0 0 3872 0 2327 346 0 5 0 95 0 1 0 488592 68900 411648 0 0 7496 0 2496 607 2 7 0 91
lspci output:
0000:00:01.0 PCI bridge: VIA Technologies, Inc. VT8237 PCI bridge [K8T800 South] (prog-if 00 [Normal decode])
Flags: bus master, 66MHz, medium devsel, latency 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
Memory behind bridge: f8000000-f9ffffff
Prefetchable memory behind bridge: f0000000-f7ffffff
0000:00:0f.0 RAID bus controller: VIA Technologies, Inc. VIA VT6420 SATA RAID Controller (rev 80)
Subsystem: Micro-Star International Co., Ltd.: Unknown device 1300
Flags: bus master, medium devsel, latency 32, IRQ 11
I am _not_ using RAID btw....
hopefully relevant dmesg output:
libata version 1.02 loaded.sata_via version 0.20sata_via(0000:00:0f.0): routed to hard irq line 11
ata1: SATA max UDMA/133 cmd 0xC000 ctl 0xC402 bmdma 0xD000 irq 11
ata2: SATA max UDMA/133 cmd 0xC800 ctl 0xCC02 bmdma 0xD008 irq 11
ata1: dev 0 cfg 49:2f00 82:346b 83:7f21 84:4003 85:3469 86:3c01 87:4003 88:203f
ata1: dev 0 ATA, max UDMA/100, 488397168 sectors: lba48
ata1: dev 0 configured for UDMA/100
scsi0 : sata_via
ata2: dev 0 cfg 49:2f00 82:346b 83:7f21 84:4003 85:3469 86:3c01 87:4003 88:407f
ata2: dev 0 ATA, max UDMA/133, 72303840 sectors: lba48
ata2: dev 0 configured for UDMA/133
scsi1 : sata_via
Using anticipatory io scheduler
Vendor: ATA Model: WDC WD2500JD-00F Rev: 02.0 Type: Direct-Access ANSI SCSI revision: 05 Vendor: ATA Model: WDC WD360GD-00FN Rev: 35.0 Type: Direct-Access ANSI SCSI revision: 05
SCSI device sda: 488397168 512-byte hdwr sectors (250059 MB) SCSI device sda: drive cache: write back
sda: sda1 sda2
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0 SCSI device sdb: 72303840 512-byte hdwr sectors (37020 MB) SCSI device sdb: drive cache: write back
sdb: sdb1
Attached scsi disk sdb at scsi1, channel 0, id 0, lun 0
From what I have understood, these drives are SATA with PATA bridge on them. The problems described happen on both. No difference.
Interesting is, that
root@opteron:$ hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 144 MB in 3.03 seconds = 47.45 MB/sec
So it is not _that_ slow...
If someone has a clue what might be wrong, please let me know (Please CC me), I will gladly provide any information that might be useful.
I understand that the problem might not be even sata related even if I naively think so. I simply do not have a clue, where to look or try next.
Thanks!
Petr Sebor
Signature · 1 line
Petr Sebor / SCS Software [ http://www.scssoft.com ]
Quoted 3 lines · Petr Sebor
and it is being worse, I can see frequently, that the 'wait' gets frequently over 90%. I have tried the kernel both in SMP or UP, with noacpi, iommu=off, preemptible kernel or not without difference....
Which 86_64.org patches are you using?
Quoted 11 lines
From what I have understood, these drives are SATA with PATA bridge on them. The problems described happen on both. No difference.
Interesting is, that
root@opteron:$ hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 144 MB in 3.03 seconds = 47.45 MB/secSo it is not _that_ slow...
I highly doubt it's related to libata or sata_via at all. A high wait indicates your processors handle data a lot faster than your harddrives can deliver and processes are blocked waiting on I/O, it would otherwise be counted as idle time in the 2.4 line; this is normal AFAIK. Your performance is nominal, the problem is most likely elsewhere.
What motherboard is this? The MSI FAR2 whatever?
-ryan
Quoted 8 lines · J. Ryan Earl
Quoted 3 lines · Petr Sebor
and it is being worse, I can see frequently, that the 'wait' gets frequently over 90%. I have tried the kernel both in SMP or UP, with noacpi, iommu=off, preemptible kernel or not without difference....
Which 86_64.org patches are you using?
None. It is vanilla 2.6.7 kernel with 2.6.8-rc2-bkX patch. I'd try to patch the kernel with x86_64.org's stuff, but I thought this applies only if I run the kernel in native - 64bit mode. I haven't gotten that far... The kernel is compiled with K7 Athlon CPU optimizations, nothing more.
Quoted 8 lines
Quoted 8 lines
From what I have understood, these drives are SATA with PATA bridge on them. The problems described happen on both. No difference.
Interesting is, that
root@opteron:$ hdparm -t /dev/sda
Quoted 7 lines
Quoted 4 lines
/dev/sda:
Timing buffered disk reads: 144 MB in 3.03 seconds = 47.45 MB/secSo it is not _that_ slow...
I highly doubt it's related to libata or
sata_viaat all. A high wait indicates your processors handle data a lot faster than your harddrives can deliver and processes are blocked waiting on I/O, it would otherwise be counted as idle time in the 2.4 line; this is normal AFAIK. Your performance is nominal, the problem is most likely elsewhere.
The more I naively think about it the more I am getting the similar feeling. I fear I have incorrectly blamed the sata layer, but I don't remember having similar problems with ide drives.
Quoted 1 line
What motherboard is this? The MSI FAR2 whatever?
It is MSI Master2-FAR.. sorry for not mentioning that.
Petr
Quoted 4 lines · Petr Sebor
Quoted 1 line
What motherboard is this? The MSI FAR2 whatever?
It is MSI Master2-FAR.. sorry for not mentioning that.
Really? Do you have APIC working? I haven't gotten onboard IDE to work without DMA timeouts when APIC is enabled on the KT800 MB I have, but it's UP so I can live without it. The current buglist said it's fixed for -some- motherboards. I would suggest you ask about this on the x86_64.org list, you're more likely to find an answer there.
Sorry for the slow reply, had to beat Doom3 before I could move on with my life.
-ryan