Skip to contentSkip to conversation

SATA Performance with Intel ICH6

DISCUSSION Martin A. Fink 5 participants 8 messages Nov 24, 2006 – Nov 25, 2006

lore
Share & tools
lore
#0 lore raw

Dear all,

I found out, that writing to a SATA harddisk costs around 20% of the computers cpu time. I write blocks of 1MB size to a file. Write performance is around 51MB/s what I think is really good. My computer has an Intel ICH6 chipset and a 3.2GHz Pentium 4 processor.
If I understand the design of this chipset correctly, then I would have expected, that the CPU needs to do only few work, instead I found out, that writing to disk seems to be really hard work for the CPU.

Can I do anything to optimize writing from memory to disk?

My final aim is to get around 140MB/s of data from 3 different Gigabit Ethernet cards and store it on 3 harddisk drives that perform 50MB/s.

Quoted 1 line

From the SATA bus side there should be no problem. Each of the 4 SATAs on

this ICH6 chipset are capable of 150MB/s.

So what makes my CPU that slow? Is it a hardware problem or a problem of SATA driver of my operating system?

time dd if=/dev/zero of=test.zero bs=1M count=1000 results in

real 0m52.561s
user 0m0.003s
sys 0m7.407s

and strace dd... gives among other information
6.84s 1004calls syscall: write

So I spend 45s of 52s within the kernel. Why so long?

By the way: I'm working with SuSE Linux 9.2 on a Dell Desktop PC, 1GB RAM

Thank you for answers,

Martin

#1 lore raw
Quoted 3 lines · On Fri, 24 Nov 2006 14:07:01 +0100 "Martin A. Fink" <[email protected]> wrote:

If I understand the design of this chipset correctly, then I would have expected, that the CPU needs to do only few work, instead I found out, that writing to disk seems to be really hard work for the CPU.

It has some work to do - the amount in question depends upon the file system and device drivers in use. For very high throughput read up on the O_DIRECT feature.

Quoted 4 lines

My final aim is to get around 140MB/s of data from 3 different Gigabit Ethernet cards and store it on 3 harddisk drives that perform 50MB/s.

Quoted 1 line

From the SATA bus side there should be no problem. Each of the 4 SATAs on

this ICH6 chipset are capable of 150MB/s.

I doubt an ICH6 has the total memory bandwidth to achieve that to be honest, but with PCI-E maybe you can.

Quoted 2 lines

So what makes my CPU that slow? Is it a hardware problem or a problem of SATA driver of my operating system?

You don't give anything like enough information to even guess this. What controller, what disks, what driver, what kernel version ?

Quoted 1 line

By the way: I'm working with SuSE Linux 9.2 on a Dell Desktop PC, 1GB RAM

For vendor kernels, especially older ones it is probably best to ask the vendor first.

Alan

#2 lore raw

Am Freitag, 24. November 2006 14:33 schrieben Sie:

Quoted 10 lines

On Fri, 24 Nov 2006 14:07:01 +0100

Quoted 3 lines · Martin A. Fink

If I understand the design of this chipset correctly, then I would have expected, that the CPU needs to do only few work, instead I found out, that writing to disk seems to be really hard work for the CPU.

It has some work to do - the amount in question depends upon the file system and device drivers in use. For very high throughput read up on the O_DIRECT feature.

Well this seems to be independend from the file system. I tried to write directly to the raw device, but nevertheless the cpu time was 20% (sys time).

Quoted 11 lines
Quoted 8 lines

My final aim is to get around 140MB/s of data from 3 different Gigabit Ethernet cards and store it on 3 harddisk drives that perform 50MB/s.

Quoted 2 lines

From the SATA bus side there should be no problem. Each of the 4 SATAs on

this ICH6 chipset are capable of 150MB/s.

I doubt an ICH6 has the total memory bandwidth to achieve that to be honest, but with PCI-E maybe you can.

This is an interessting point. The specification say that I can handle around 120 to 150 MB/s each of the 4 S-ATA ports. With ICH6 the S-ATA ports seem to be directly connected to the Southbridge, and the Southbridge is directly connected to Northbridge via PCI Express. So it should be possible to get 150 MB/s from north to south and from there in packages of 55 MB/s to the disks ?!

Quoted 6 lines
Quoted 3 lines

So what makes my CPU that slow? Is it a hardware problem or a problem of SATA driver of my operating system?

You don't give anything like enough information to even guess this. What controller, what disks, what driver, what kernel version ?

I use a Pentium 4 processor (Xeon), Intel ICH6 chipset, to write on a 250 GB S-ATA disk directly connected to one of the 4 onboard S-ATA connections. kernel version is Linux version 2.6.8-24.25-smp. As disks I used ones from Winchester as well as Maxtor. Both behave the same.
Here is some information from /var/log/boot.msg

<6>ACPI: RSDP (v000 DELL ) @ 0x000feb90

<6>ACPI: RSDT (v001 DELL    WS 360  0x00000007 ASL  0x00000061) @ 0x000fd165
<6>ACPI: FADT (v001 DELL    WS 360  0x00000007 ASL  0x00000061) @ 0x000fd19d
<6>ACPI: SSDT (v001   DELL    st_ex 0x00001000 MSFT 0x0100000d) @ 0xfffc8f10
<6>ACPI: MADT (v001 DELL    WS 360  0x00000007 ASL  0x00000061) @ 0x000fd211
<6>ACPI: BOOT (v001 DELL    WS 360  0x00000007 ASL  0x00000061) @ 0x000fd27d
<6>ACPI: ASF! (v016 DELL    WS 360  0x00000007 ASL  0x00000061) @ 0x000fd2a5

<6>ACPI: DSDT (v001 DELL dt_ex 0x00001000 MSFT 0x0100000d) @ 0x00000000
<6>ACPI: PM-Timer IO Port: 0x808
<7>ACPI: Local APIC address 0xfee00000
<6>ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled) <4>Processor #0 15:2 APIC version 20
<6>ACPI: LAPIC (acpi_id[0x02] lapic_id[0x01] disabled) <6>ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] disabled) <6>ACPI: LAPIC (acpi_id[0x04] lapic_id[0x03] disabled) <6>ACPI: IOAPIC (id[0x01] address[0xfec00000] gsi_base[0]) <4>IOAPIC[0]: apic_id 1, version 32, address 0xfec00000, GSI 0-23 <6>ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) <6>ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) <7>ACPI: IRQ0 used by override.
<7>ACPI: IRQ2 used by override.
<7>ACPI: IRQ9 used by override.
<4>Enabling APIC mode: Flat. Using 1 I/O APICs <6>Using ACPI (MADT) for SMP configuration information

<6>CPU: L2 cache: 512K
<6>CPU: Hyper-Threading is disabled

<7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] <7>ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT] <4>ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11 12 15) <4>ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 *10 11 12 15) <4>ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 *9 10 11 12 15) <4>ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 *10 11 12 15) <4>ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11 12 15) *0, disabled.
<4>ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 *5 6 7 9 10 11 12 15) <4>ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 *5 6 7 9 10 11 12 15) <4>ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 *5 6 7 9 10 11 12 15) <6>Linux Plug and Play Support v0.97 (c) Adam Belay
<6>PCI: Using ACPI for IRQ routing
<6>ACPI: PCI interrupt 0000:00:1d.0[A] -> GSI 16 (level, low) -> IRQ 169 <6>ACPI: PCI interrupt 0000:00:1d.1[B] -> GSI 19 (level, low) -> IRQ 177 <6>ACPI: PCI interrupt 0000:00:1d.2[C] -> GSI 18 (level, low) -> IRQ 185 <6>ACPI: PCI interrupt 0000:00:1d.3[A] -> GSI 16 (level, low) -> IRQ 169 <6>ACPI: PCI interrupt 0000:00:1d.7[D] -> GSI 23 (level, low) -> IRQ 193 <6>ACPI: PCI interrupt 0000:00:1f.1[A] -> GSI 18 (level, low) -> IRQ 185 <6>ACPI: PCI interrupt 0000:00:1f.2[A] -> GSI 18 (level, low) -> IRQ 185 <6>ACPI: PCI interrupt 0000:00:1f.3[B] -> GSI 17 (level, low) -> IRQ 201 <6>ACPI: PCI interrupt 0000:00:1f.5[B] -> GSI 17 (level, low) -> IRQ 201 <6>ACPI: PCI interrupt 0000:01:00.0[A] -> GSI 16 (level, low) -> IRQ 169 <6>ACPI: PCI interrupt 0000:02:00.0[A] -> GSI 21 (level, low) -> IRQ 209 <6>ACPI: PCI interrupt 0000:02:01.0[A] -> GSI 22 (level, low) -> IRQ 217 <6>ACPI: PCI interrupt 0000:02:02.0[A] -> GSI 17 (level, low) -> IRQ 201 <6>ACPI: PCI interrupt 0000:02:0c.0[A] -> GSI 18 (level, low) -> IRQ 185

<6>scsi1 : ata_piix

<5>  Vendor: ATA       Model: WDC WD740GD-75FL  Rev: 21.0
<5>  Type:   Direct-Access                      ANSI SCSI revision: 05

<5>SCSI device sda: 144531250 512-byte hdwr sectors (74000 MB) <5>SCSI device sda: drive cache: write back
<5>SCSI device sda: 144531250 512-byte hdwr sectors (74000 MB) <5>SCSI device sda: drive cache: write back
<6> sda: sda1 sda2 < sda5 sda6 sda7 sda8 > <5>Attached scsi disk sda at scsi0, channel 0, id 0, lun 0

Quoted 7 lines
Quoted 2 lines

By the way: I'm working with SuSE Linux 9.2 on a Dell Desktop PC, 1GB RAM

For vendor kernels, especially older ones it is probably best to ask the vendor first.

Alan

Here some measurement results:

time dd if=/dev/zero of=test.zero bs=1M count=1000 results in

real 0m52.561s
user 0m0.003s
sys 0m7.407s

and strace dd... gives among other information
6.84s 1004calls syscall: write

So I spend 45s of 52s within the kernel. Why so long?

#3 lore raw
Quoted 2 lines · On Fri, 24 Nov 2006 15:10:11 +0100 "Martin A. Fink" <[email protected]> wrote:

Well this seems to be independend from the file system. I tried to write directly to the raw device, but nevertheless the cpu time was 20% (sys time).

sys time is not neccessarily CPU time.

Quoted 2 lines

This is an interessting point. The specification say that I can handle around 120 to 150 MB/s each of the 4 S-ATA ports. With ICH6 the S-ATA ports seem to

At once ?

Quoted 4 lines

be directly connected to the Southbridge, and the Southbridge is directly connected to Northbridge via PCI Express. So it should be possible to get 150 MB/s from north to south and from there in packages of 55 MB/s to the disks ?!

Ask the vendor.

Quoted 3 lines

<6>scsi1 : ata_piix

<5>  Vendor: ATA       Model: WDC WD740GD-75FL  Rev: 21.0
<5>  Type:   Direct-Access                      ANSI SCSI revision: 05

The PIIX interface needs CPU intervention each command, so in practice about every 64K or so, and the CPU gets stalled waiting for the disk during the setup of each I/O. The newer kernels support AHCI which does not have this overhead, but it is only present on the newest intel controllers.

Quoted 4 lines

and strace dd... gives among other information
6.84s 1004calls syscall: write

So I spend 45s of 52s within the kernel. Why so long?

Waiting for the disk I would imagine.

Alan

#4 lore raw

Am Freitag, 24. November 2006 15:30 schrieben Sie:

Quoted 5 lines
Quoted 2 lines · On Fri, 24 Nov 2006 15:10:11 +0100 "Martin A. Fink" <[email protected]> wrote:

Well this seems to be independend from the file system. I tried to write directly to the raw device, but nevertheless the cpu time was 20% (sys

time).

Quoted 4 lines

sys time is not neccessarily CPU time.

Quoted 1 line

This is an interessting point. The specification say that I can handle

around

Quoted 1 line
Quoted 1 line

120 to 150 MB/s each of the 4 S-ATA ports. With ICH6 the S-ATA ports seem

to

Quoted 5 lines

At once ?

Quoted 2 lines

be directly connected to the Southbridge, and the Southbridge is directly connected to Northbridge via PCI Express. So it should be possible to get

150

Quoted 9 lines
Quoted 2 lines

MB/s from north to south and from there in packages of 55 MB/s to the disks ?!

Ask the vendor.

Quoted 3 lines

<6>scsi1 : ata_piix

<5>  Vendor: ATA       Model: WDC WD740GD-75FL  Rev: 21.0
<5>  Type:   Direct-Access                      ANSI SCSI revision: 05
Quoted 5 lines

The PIIX interface needs CPU intervention each command, so in practice about every 64K or so, and the CPU gets stalled waiting for the disk during the setup of each I/O. The newer kernels support AHCI which does not have this overhead, but it is only present on the newest intel controllers.

Can you tell me the name of this controllers? Is it ICH7 or 8 ? What kernel versions?

Thank you very much

Quoted 16 lines
Quoted 5 lines

and strace dd... gives among other information
6.84s 1004calls syscall: write

So I spend 45s of 52s within the kernel. Why so long?

Waiting for the disk I would imagine.

Alan

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 8 lines
Dipl. Physiker
Martin Anton Fink
Max Planck Institute for extraterrestrial Physics
Giessenbachstrasse
85741 Garching
Germany
Tel. +49-(0)89-30000-3645
Fax. +49-(0)89-30000-3569
#5 lore raw
In reply to Martin A. Fink
Quoted 5 lines · On Fri, 24 Nov 2006 15:10:11 +0100 "Martin A. Fink" <[email protected]> wrote:

Here some measurement results:

time dd if=/dev/zero of=test.zero bs=1M count=1000 results in

Quoted 8 lines

real 0m52.561s
user 0m0.003s
sys 0m7.407s

and strace dd... gives among other information
6.84s 1004calls syscall: write

So I spend 45s of 52s within the kernel. Why so long?

Profiling the kernel will tell. Preferably with oprofile, but if for some reason that doesn't work, with the old profiler.

I use this:

#!/bin/sh
sudo opcontrol --stop
sudo opcontrol --shutdown
sudo rm -rf /var/lib/oprofile
sudo opcontrol --vmlinux=/boot/vmlinux-$(uname -r) sudo opcontrol --start-daemon
sudo opcontrol --start
time $@
sudo opcontrol --stop
sudo opcontrol --shutdown
sudo opreport -l /boot/vmlinux-$(uname -r) | head -50

I have seen PIO and MMIO operations on PCI busses take tremendous amounts of time when that bus is saturated with DMA transfers. But that was back in the old days.

#6 lore raw
In reply to Martin A. Fink
Quoted 23 lines · Martin A. Fink

Dear all,

I found out, that writing to a SATA harddisk costs around 20% of the computers cpu time. I write blocks of 1MB size to a file. Write performance is around 51MB/s what I think is really good. My computer has an Intel ICH6 chipset and a 3.2GHz Pentium 4 processor.
If I understand the design of this chipset correctly, then I would have expected, that the CPU needs to do only few work, instead I found out, that writing to disk seems to be really hard work for the CPU.

Can I do anything to optimize writing from memory to disk?

My final aim is to get around 140MB/s of data from 3 different Gigabit Ethernet cards and store it on 3 harddisk drives that perform 50MB/s.

Quoted 1 line

From the SATA bus side there should be no problem. Each of the 4 SATAs on

this ICH6 chipset are capable of 150MB/s.

So what makes my CPU that slow? Is it a hardware problem or a problem of SATA driver of my operating system?

time dd if=/dev/zero of=test.zero bs=1M count=1000 results in

Quoted 9 lines

real 0m52.561s
user 0m0.003s
sys 0m7.407s

and strace dd... gives among other information
6.84s 1004calls syscall: write

So I spend 45s of 52s within the kernel. Why so long?

forgetting the fact that you're only writing 0's and that makes me wonder if the filesystems are cheating because the data it's writing is all the same, you aren't syncing that data. You're not benchmarking the actual write speed, you have buffer being used still with uncommitted data.

try using iozone or something a little more stressful on the system.

If you rerun the "benchmark" and your scores are much better (like with your example the second run gets 260MB/sec for me) then it's not a good benchmark. It should be consistantly the same no matter how many times you run it.

Quoted 6 lines

By the way: I'm working with SuSE Linux 9.2 on a Dell Desktop PC, 1GB RAM

Thank you for answers,

Martin

#7 lore raw
In reply to Martin A. Fink
Quoted 10 lines · Martin A. Fink

Dear all,

I found out, that writing to a SATA harddisk costs around 20% of the computers cpu time. I write blocks of 1MB size to a file. Write performance is around 51MB/s what I think is really good. My computer has an Intel ICH6 chipset and a 3.2GHz Pentium 4 processor.
If I understand the design of this chipset correctly, then I would have expected, that the CPU needs to do only few work, instead I found out, that writing to disk seems to be really hard work for the CPU.

Quoted 7 lines

Can I do anything to optimize writing from memory to disk?

My final aim is to get around 140MB/s of data from 3 different Gigabit Ethernet cards and store it on 3 harddisk drives that perform 50MB/s. From the SATA bus side there should be no problem. Each of the 4 SATAs on this ICH6 chipset are capable of 150MB/s.

Theoretically for when CPUs and disks are capable of giving those speeds...

Quoted 10 lines

So what makes my CPU that slow? Is it a hardware problem or a problem of SATA driver of my operating system?

time dd if=/dev/zero of=test.zero bs=1M count=1000 results in

real 0m52.561s
user 0m0.003s
sys 0m7.407s

Guess first what are your busses limit. Get a try with hdparm. I have this:

00:1f.2 IDE interface: Intel Corporation 82801EB (ICH5) SATA Controller (rev 02)
scsi 2:0:0:0: Direct-Access ATA ST3200822AS 3.01 PQ: 0 ANSI: 5

/dev/sdc:
Timing cached reads: 1432 MB in 2.00 seconds = 715.90 MB/sec
Timing buffered disk reads: 162 MB in 3.02 seconds = 53.70 MB/sec

That gives you the max speed limit of the kernel transfers+pci speed+disk cache speed combo. Then you should add the real disk speed, but that's for a real benchmark for iozone.

Also try to run several in parallel, one on each disk.

And something is really strange in your box:

werewolf:~> time -p dd if=/dev/zero of=test.zero bs=1M count=1000 1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 8.64253 seconds, 121 MB/s real 8.64
user 0.00
sys 4.70

(dual xeon 2.4GHz)

CPU usage was 100% at the start but dropped to about 2-3-4%. Be sure to remove manually test.zero before running each test, if not you will account the time the system spends cleaning the page cache when dd unlinks or truncates test.zero:

werewolf:~> rm -f test.zero
werewolf:~> time -p dd if=/dev/zero of=test.zero bs=1M count=1000 1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 8.29974 seconds, 126 MB/s
real 8.30
user 0.00
sys 4.53
werewolf:~> time -p dd if=/dev/zero of=test.zero bs=1M count=1000 1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 8.30443 seconds, 126 MB/s
real 31.86
user 0.00
sys 5.52
werewolf:~> rm -f test.zero; time -p dd if=/dev/zero of=test.zero bs=1M count=1000 1000+0 records in
1000+0 records out
1048576000 bytes (1.0 GB) copied, 7.98103 seconds, 131 MB/s
real 8.07
user 0.00
sys 4.61

Quoted 5 lines

and strace dd... gives among other information
6.84s 1004calls syscall: write

So I spend 45s of 52s within the kernel. Why so long?

--
J.A. Magallon <jamagallon()ono!com> \ Software is like sex:
\ It's better when it's free
Mandriva Linux release 2007.1 (Cooker) for i586
Linux 2.6.18-jam13 (gcc 4.1.2 20061110 (prerelease) (4.1.2-0.20061110.1mdv2007.1)) #1 SMP PREEMPT