SATA Performance with Intel ICH6
Dear all,
I found out, that writing to a SATA harddisk costs around 20% of the computers cpu time. I write blocks of 1MB size to a file. Write performance is around 51MB/s what I think is really good. My computer has an Intel ICH6 chipset and a 3.2GHz Pentium 4 processor.
If I understand the design of this chipset correctly, then I would have expected, that the CPU needs to do only few work, instead I found out, that writing to disk seems to be really hard work for the CPU.
Can I do anything to optimize writing from memory to disk?
My final aim is to get around 140MB/s of data from 3 different Gigabit Ethernet cards and store it on 3 harddisk drives that perform 50MB/s.
Quoted 1 line
From the SATA bus side there should be no problem. Each of the 4 SATAs on
this ICH6 chipset are capable of 150MB/s.
So what makes my CPU that slow? Is it a hardware problem or a problem of SATA driver of my operating system?
time dd if=/dev/zero of=test.zero bs=1M count=1000 results in
real 0m52.561s
user 0m0.003s
sys 0m7.407s
and strace dd... gives among other information
6.84s 1004calls syscall: write
So I spend 45s of 52s within the kernel. Why so long?
By the way: I'm working with SuSE Linux 9.2 on a Dell Desktop PC, 1GB RAM
Thank you for answers,
Martin
Quoted 3 lines · On Fri, 24 Nov 2006 14:07:01 +0100 "Martin A. Fink" <[email protected]> wrote:
If I understand the design of this chipset correctly, then I would have expected, that the CPU needs to do only few work, instead I found out, that writing to disk seems to be really hard work for the CPU.
It has some work to do - the amount in question depends upon the file system and device drivers in use. For very high throughput read up on the O_DIRECT feature.
Quoted 4 lines
My final aim is to get around 140MB/s of data from 3 different Gigabit Ethernet cards and store it on 3 harddisk drives that perform 50MB/s.
Quoted 1 line
From the SATA bus side there should be no problem. Each of the 4 SATAs on
this ICH6 chipset are capable of 150MB/s.
I doubt an ICH6 has the total memory bandwidth to achieve that to be honest, but with PCI-E maybe you can.
Quoted 2 lines
So what makes my CPU that slow? Is it a hardware problem or a problem of SATA driver of my operating system?
You don't give anything like enough information to even guess this. What controller, what disks, what driver, what kernel version ?
Quoted 1 line
By the way: I'm working with SuSE Linux 9.2 on a Dell Desktop PC, 1GB RAM
For vendor kernels, especially older ones it is probably best to ask the vendor first.
Alan
Am Freitag, 24. November 2006 14:33 schrieben Sie:
Quoted 10 lines
On Fri, 24 Nov 2006 14:07:01 +0100
Quoted 3 lines · Martin A. Fink
If I understand the design of this chipset correctly, then I would have expected, that the CPU needs to do only few work, instead I found out, that writing to disk seems to be really hard work for the CPU.
It has some work to do - the amount in question depends upon the file system and device drivers in use. For very high throughput read up on the
O_DIRECTfeature.
Well this seems to be independend from the file system. I tried to write directly to the raw device, but nevertheless the cpu time was 20% (sys time).
Quoted 11 lines
Quoted 8 lines
My final aim is to get around 140MB/s of data from 3 different Gigabit Ethernet cards and store it on 3 harddisk drives that perform 50MB/s.
Quoted 2 lines
From the SATA bus side there should be no problem. Each of the 4 SATAs on
this ICH6 chipset are capable of 150MB/s.
I doubt an ICH6 has the total memory bandwidth to achieve that to be honest, but with PCI-E maybe you can.
This is an interessting point. The specification say that I can handle around 120 to 150 MB/s each of the 4 S-ATA ports. With ICH6 the S-ATA ports seem to be directly connected to the Southbridge, and the Southbridge is directly connected to Northbridge via PCI Express. So it should be possible to get 150 MB/s from north to south and from there in packages of 55 MB/s to the disks ?!
Quoted 6 lines
Quoted 3 lines
So what makes my CPU that slow? Is it a hardware problem or a problem of SATA driver of my operating system?
You don't give anything like enough information to even guess this. What controller, what disks, what driver, what kernel version ?
I use a Pentium 4 processor (Xeon), Intel ICH6 chipset, to write on a 250 GB S-ATA disk directly connected to one of the 4 onboard S-ATA connections. kernel version is Linux version 2.6.8-24.25-smp. As disks I used ones from Winchester as well as Maxtor. Both behave the same.
Here is some information from /var/log/boot.msg
<6>ACPI: RSDP (v000 DELL ) @ 0x000feb90
<6>ACPI: RSDT (v001 DELL WS 360 0x00000007 ASL 0x00000061) @ 0x000fd165 <6>ACPI: FADT (v001 DELL WS 360 0x00000007 ASL 0x00000061) @ 0x000fd19d <6>ACPI: SSDT (v001 DELL st_ex 0x00001000 MSFT 0x0100000d) @ 0xfffc8f10 <6>ACPI: MADT (v001 DELL WS 360 0x00000007 ASL 0x00000061) @ 0x000fd211 <6>ACPI: BOOT (v001 DELL WS 360 0x00000007 ASL 0x00000061) @ 0x000fd27d <6>ACPI: ASF! (v016 DELL WS 360 0x00000007 ASL 0x00000061) @ 0x000fd2a5
<6>ACPI: DSDT (v001 DELL dt_ex 0x00001000 MSFT 0x0100000d) @ 0x00000000
<6>ACPI: PM-Timer IO Port: 0x808
<7>ACPI: Local APIC address 0xfee00000
<6>ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) <4>Processor #0 15:2 APIC version 20
<6>ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] disabled) <6>ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] disabled) <6>ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] disabled) <6>ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) <4>IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23 <6>ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) <6>ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) <7>ACPI: IRQ0 used by override.
<7>ACPI: IRQ2 used by override.
<7>ACPI: IRQ9 used by override.
<4>Enabling APIC mode: Flat. Using 1 I/O APICs <6>Using ACPI (MADT) for SMP configuration information
<6>CPU: L2 cache: 512K
<6>CPU: Hyper-Threading is disabled
<7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] <7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT] <4>ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 15) <4>ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 *10 11 12 15) <4>ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *9 10 11 12 15) <4>ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 *10 11 12 15) <4>ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 15) *0, disabled.
<4>ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 *5 6 7 9 10 11 12 15) <4>ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 *5 6 7 9 10 11 12 15) <4>ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 *5 6 7 9 10 11 12 15) <6>Linux Plug and Play Support v0.97 (c) Adam Belay
<6>PCI: Using ACPI for IRQ routing
<6>ACPI: PCI interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 169 <6>ACPI: PCI interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 177 <6>ACPI: PCI interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 185 <6>ACPI: PCI interrupt 0000:00:1d.3[A] -> GSI 16 (level, low) -> IRQ 169 <6>ACPI: PCI interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 193 <6>ACPI: PCI interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 185 <6>ACPI: PCI interrupt 0000:00:1f.2[A] -> GSI 18 (level, low) -> IRQ 185 <6>ACPI: PCI interrupt 0000:00:1f.3[B] -> GSI 17 (level, low) -> IRQ 201 <6>ACPI: PCI interrupt 0000:00:1f.5[B] -> GSI 17 (level, low) -> IRQ 201 <6>ACPI: PCI interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 169 <6>ACPI: PCI interrupt 0000:02:00.0[A] -> GSI 21 (level, low) -> IRQ 209 <6>ACPI: PCI interrupt 0000:02:01.0[A] -> GSI 22 (level, low) -> IRQ 217 <6>ACPI: PCI interrupt 0000:02:02.0[A] -> GSI 17 (level, low) -> IRQ 201 <6>ACPI: PCI interrupt 0000:02:0c.0[A] -> GSI 18 (level, low) -> IRQ 185
<6>scsi1 : ata_piix
<5> Vendor: ATA Model: WDC WD740GD-75FL Rev: 21.0 <5> Type: Direct-Access ANSI SCSI revision: 05
<5>SCSI device sda: 144531250 512-byte hdwr sectors (74000 MB) <5>SCSI device sda: drive cache: write back
<5>SCSI device sda: 144531250 512-byte hdwr sectors (74000 MB) <5>SCSI device sda: drive cache: write back
<6> sda: sda1 sda2 < sda5 sda6 sda7 sda8 > <5>Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Quoted 7 lines
Quoted 2 lines
By the way: I'm working with SuSE Linux 9.2 on a Dell Desktop PC, 1GB RAM
For vendor kernels, especially older ones it is probably best to ask the vendor first.
Alan
Here some measurement results:
time dd if=/dev/zero of=test.zero bs=1M count=1000 results in
real 0m52.561s
user 0m0.003s
sys 0m7.407s
and strace dd... gives among other information
6.84s 1004calls syscall: write
So I spend 45s of 52s within the kernel. Why so long?
Quoted 2 lines · On Fri, 24 Nov 2006 15:10:11 +0100 "Martin A. Fink" <[email protected]> wrote:
Well this seems to be independend from the file system. I tried to write directly to the raw device, but nevertheless the cpu time was 20% (sys time).
sys time is not neccessarily CPU time.
Quoted 2 lines
This is an interessting point. The specification say that I can handle around 120 to 150 MB/s each of the 4 S-ATA ports. With ICH6 the S-ATA ports seem to
At once ?
Quoted 4 lines
be directly connected to the Southbridge, and the Southbridge is directly connected to Northbridge via PCI Express. So it should be possible to get 150 MB/s from north to south and from there in packages of 55 MB/s to the disks ?!
Ask the vendor.
Quoted 3 lines
<6>scsi1 :
ata_piix<5> Vendor: ATA Model: WDC WD740GD-75FL Rev: 21.0 <5> Type: Direct-Access ANSI SCSI revision: 05
The PIIX interface needs CPU intervention each command, so in practice about every 64K or so, and the CPU gets stalled waiting for the disk during the setup of each I/O. The newer kernels support AHCI which does not have this overhead, but it is only present on the newest intel controllers.
Quoted 4 lines
and strace dd... gives among other information
6.84s 1004calls syscall: writeSo I spend 45s of 52s within the kernel. Why so long?
Waiting for the disk I would imagine.
Alan
Am Freitag, 24. November 2006 15:30 schrieben Sie:
Quoted 5 lines
Quoted 2 lines · On Fri, 24 Nov 2006 15:10:11 +0100 "Martin A. Fink" <[email protected]> wrote:
Well this seems to be independend from the file system. I tried to write directly to the raw device, but nevertheless the cpu time was 20% (sys
time).
Quoted 4 lines
sys time is not neccessarily CPU time.
Quoted 1 line
This is an interessting point. The specification say that I can handle
around
Quoted 1 line
Quoted 1 line
120 to 150 MB/s each of the 4 S-ATA ports. With ICH6 the S-ATA ports seem
to
Quoted 5 lines
At once ?
Quoted 2 lines
be directly connected to the Southbridge, and the Southbridge is directly connected to Northbridge via PCI Express. So it should be possible to get
150
Quoted 9 lines
Quoted 2 lines
MB/s from north to south and from there in packages of 55 MB/s to the disks ?!
Ask the vendor.
Quoted 3 lines
<6>scsi1 :
ata_piix<5> Vendor: ATA Model: WDC WD740GD-75FL Rev: 21.0 <5> Type: Direct-Access ANSI SCSI revision: 05
Quoted 5 lines
The PIIX interface needs CPU intervention each command, so in practice about every 64K or so, and the CPU gets stalled waiting for the disk during the setup of each I/O. The newer kernels support AHCI which does not have this overhead, but it is only present on the newest intel controllers.
Can you tell me the name of this controllers? Is it ICH7 or 8 ? What kernel versions?
Thank you very much
Quoted 16 lines
Quoted 5 lines
and strace dd... gives among other information
6.84s 1004calls syscall: writeSo I spend 45s of 52s within the kernel. Why so long?
Waiting for the disk I would imagine.
Alan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 8 lines
Dipl. Physiker Martin Anton Fink Max Planck Institute for extraterrestrial Physics Giessenbachstrasse 85741 Garching Germany Tel. +49-(0)89-30000-3645 Fax. +49-(0)89-30000-3569
Quoted 5 lines · On Fri, 24 Nov 2006 15:10:11 +0100 "Martin A. Fink" <[email protected]> wrote:
Here some measurement results:
time dd if=/dev/zero of=test.zero bs=1M count=1000 results in
Quoted 8 lines
real 0m52.561s
user 0m0.003s
sys 0m7.407sand strace dd... gives among other information
6.84s 1004calls syscall: writeSo I spend 45s of 52s within the kernel. Why so long?
Profiling the kernel will tell. Preferably with oprofile, but if for some reason that doesn't work, with the old profiler.
I use this:
#!/bin/sh
sudo opcontrol --stop
sudo opcontrol --shutdown
sudo rm -rf /var/lib/oprofile
sudo opcontrol --vmlinux=/boot/vmlinux-$(uname -r) sudo opcontrol --start-daemon
sudo opcontrol --start
time $@
sudo opcontrol --stop
sudo opcontrol --shutdown
sudo opreport -l /boot/vmlinux-$(uname -r) | head -50
I have seen PIO and MMIO operations on PCI busses take tremendous amounts of time when that bus is saturated with DMA transfers. But that was back in the old days.
Quoted 23 lines · Martin A. Fink
Dear all,
I found out, that writing to a SATA harddisk costs around 20% of the computers cpu time. I write blocks of 1MB size to a file. Write performance is around 51MB/s what I think is really good. My computer has an Intel ICH6 chipset and a 3.2GHz Pentium 4 processor.
If I understand the design of this chipset correctly, then I would have expected, that the CPU needs to do only few work, instead I found out, that writing to disk seems to be really hard work for the CPU.Can I do anything to optimize writing from memory to disk?
My final aim is to get around 140MB/s of data from 3 different Gigabit Ethernet cards and store it on 3 harddisk drives that perform 50MB/s.
Quoted 1 line
From the SATA bus side there should be no problem. Each of the 4 SATAs on
this ICH6 chipset are capable of 150MB/s.
So what makes my CPU that slow? Is it a hardware problem or a problem of SATA driver of my operating system?
time dd if=/dev/zero of=test.zero bs=1M count=1000 results in
Quoted 9 lines
real 0m52.561s
user 0m0.003s
sys 0m7.407sand strace dd... gives among other information
6.84s 1004calls syscall: writeSo I spend 45s of 52s within the kernel. Why so long?
forgetting the fact that you're only writing 0's and that makes me wonder if the filesystems are cheating because the data it's writing is all the same, you aren't syncing that data. You're not benchmarking the actual write speed, you have buffer being used still with uncommitted data.
try using iozone or something a little more stressful on the system.
If you rerun the "benchmark" and your scores are much better (like with your example the second run gets 260MB/sec for me) then it's not a good benchmark. It should be consistantly the same no matter how many times you run it.
Quoted 6 lines
By the way: I'm working with SuSE Linux 9.2 on a Dell Desktop PC, 1GB RAM
Thank you for answers,
Martin
Quoted 10 lines · Martin A. Fink
Dear all,
I found out, that writing to a SATA harddisk costs around 20% of the computers cpu time. I write blocks of 1MB size to a file. Write performance is around 51MB/s what I think is really good. My computer has an Intel ICH6 chipset and a 3.2GHz Pentium 4 processor.
If I understand the design of this chipset correctly, then I would have expected, that the CPU needs to do only few work, instead I found out, that writing to disk seems to be really hard work for the CPU.
Quoted 7 lines
Can I do anything to optimize writing from memory to disk?
My final aim is to get around 140MB/s of data from 3 different Gigabit Ethernet cards and store it on 3 harddisk drives that perform 50MB/s. From the SATA bus side there should be no problem. Each of the 4 SATAs on this ICH6 chipset are capable of 150MB/s.
Theoretically for when CPUs and disks are capable of giving those speeds...
Quoted 10 lines
So what makes my CPU that slow? Is it a hardware problem or a problem of SATA driver of my operating system?
time dd if=/dev/zero of=test.zero bs=1M count=1000 results in
real 0m52.561s
user 0m0.003s
sys 0m7.407s
Guess first what are your busses limit. Get a try with hdparm. I have this:
00:1f.2 IDE interface: Intel Corporation 82801EB (ICH5) SATA Controller (rev 02)
scsi 2:0:0:0: Direct-Access ATA ST3200822AS 3.01 PQ: 0 ANSI: 5
/dev/sdc:
Timing cached reads: 1432 MB in 2.00 seconds = 715.90 MB/sec
Timing buffered disk reads: 162 MB in 3.02 seconds = 53.70 MB/sec
That gives you the max speed limit of the kernel transfers+pci speed+disk cache speed combo. Then you should add the real disk speed, but that's for a real benchmark for iozone.
Also try to run several in parallel, one on each disk.
And something is really strange in your box:
werewolf:~> time -p dd if=/dev/zero of=test.zero bs=1M count=1000 1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 8.64253 seconds, 121 MB/s real 8.64
user 0.00
sys 4.70
(dual xeon 2.4GHz)
CPU usage was 100% at the start but dropped to about 2-3-4%. Be sure to remove manually test.zero before running each test, if not you will account the time the system spends cleaning the page cache when dd unlinks or truncates test.zero:
werewolf:~> rm -f test.zero
werewolf:~> time -p dd if=/dev/zero of=test.zero bs=1M count=1000 1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 8.29974 seconds, 126 MB/s
real 8.30
user 0.00
sys 4.53
werewolf:~> time -p dd if=/dev/zero of=test.zero bs=1M count=1000 1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 8.30443 seconds, 126 MB/s
real 31.86
user 0.00
sys 5.52
werewolf:~> rm -f test.zero; time -p dd if=/dev/zero of=test.zero bs=1M count=1000 1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 7.98103 seconds, 131 MB/s
real 8.07
user 0.00
sys 4.61
Quoted 5 lines
and strace dd... gives among other information
6.84s 1004calls syscall: writeSo I spend 45s of 52s within the kernel. Why so long?
--
J.A. Magallon <jamagallon()ono!com> \ Software is like sex:
\ It's better when it's free
Mandriva Linux release 2007.1 (Cooker) for i586
Linux 2.6.18-jam13 (gcc 4.1.2 20061110 (prerelease) (4.1.2-0.20061110.1mdv2007.1)) #1 SMP PREEMPT