Raid6 write performance

Hi,

I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I am having is that no matter what chunk size I use, the write benchmark always comes out at single drive speed, although I should be seeing double drive speed (read speed is at near 4x as expected). Is there some hidden setting that I am overlooking, or is this a current known limitation of raid6? In contrast if I make a raid5 on these 4 drives, I get the expected 3xdrive write speed, and occasionally 4xdrive linear read speed.

When the write test is running, I get about 14% of system cpu a sporadic 40% of iowait and the rest idle at all times (machine is in runlevel 1 so not to screw with results). Anyone has any ideas?

Thank you.

I tested RAID 5 and RAID 6 with 12 x 15K SAS drives on Ubuntu 8.04 64-bit and found their performance to be about the same. I used 256 K chunk size, v1.0 superblocks, stripecachesize of 16384, and readahead of 65536.

RAID 5 reads: 774 MB/sec
RAID 5 writes: 585 MB/sec

RAID 6 reads: 742 MB/sec
RAID 6 writes: 559 MB/sec

My CPU utilization remains under 10% though during writes, and I'm wondering what can be done to get write performance closer to read performance. I have dual quad-core CPUs so there's plenty of CPU to go around. Any ideas on that front?

-Thomas

-----Original Message-----
From: Peter Rabbitson <[email protected]>
To: [email protected]
Sent: Sun, 18 Jan 2009 11:40 pm
Subject: Raid6 write performance

Hi,

I am experimenting with raid6 on 4 drives on 2.6.27.11. The problem I am having is that no matter what chunk size I use, the write benchmark always comes out at single drive speed, although I should be seeing double drive speed (read speed is at near 4x as expected). Is there some hidden setting that I am overlooking, or is this a current known limitation of raid6? In contrast if I make a raid5 on these 4 drives, I get the expected 3xdrive write speed, and occasionally 4xdrive linear read speed.

When the write test is running, I get about 14% of system cpu a sporadic 40% of iowait and the rest idle at all times (machine is in runlevel 1 so not to screw with results). Anyone has any ideas?

Thank you.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Read bandwidth will always be quicker than writes with parity. Data and parity both need to be written in an atomic way so the completion will be gated by the last write to be "posted as done" back to the system.

It can pay to locate the journal on a physically different, smaller and faster resource.
One informative experiment might be to mount the file system as ext2 and compare and contrast with the same FS mounted as ext3. I am not recommending ext2 over ext3 other than as an experiment to see what the impact of the journal activity is...

        NiftyFedora
        T o m   M i t c h e l l

Please disregard this, I appear to be having some sort of weird hardware congestion. Can someone explain what could be going on?

Standalone read speed:
--------------------------------

root@Arzamas:~# for i in b c d e ; do dd if=/dev/sd${i}2 of=/dev/null
bs=32M count=32 ; done
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.4133 s, 103 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.4781 s, 102 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.4317 s, 103 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.1242 s, 106 MB/s
root@Arzamas:~#

Combined read speed - limited by the width of a 32bit 66mhz PCI (dev 02:04.0 below)
--------------------------------

root@Arzamas:~# for i in b c d e ; do dd if=/dev/sd${i}2 of=/dev/null
bs=32M count=32 & done
[1] 19491
[2] 19492
[3] 19493
[4] 19494
root@Arzamas:~# 32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 17.8422 s, 60.2 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 18.2496 s, 58.8 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 18.5542 s, 57.9 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 19.0774 s, 56.3 MB/s

Standalone write:
-------------------------------

root@Arzamas:~# for i in b c d e ; do dd if=/dev/zero of=/dev/sd${i}2
bs=32M count=32 ; done
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.7629 s, 99.8 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.753 s, 99.9 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 12.1836 s, 88.1 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 10.2433 s, 105 MB/s
root@Arzamas:~#

Combined write - half of the read speed???:
-------------------------------

root@Arzamas:~# for i in b c d e ; do dd if=/dev/zero of=/dev/sd${i}2
bs=32M count=32 & done
[1] 19506
[2] 19507
[3] 19508
[4] 19509
root@Arzamas:~# 32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 40.1631 s, 26.7 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 40.2603 s, 26.7 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 40.2862 s, 26.7 MB/s
32+0 records in
32+0 records out
1073741824 bytes (1.1 GB) copied, 40.3178 s, 26.6 MB/s

Hardware:
---------------------------

root@Arzamas:~# lspci
00:00.0 Host bridge: Intel Corporation 82875P/E7210 Memory Controller
Hub (rev 02)
00:03.0 PCI bridge: Intel Corporation 82875P/E7210 Processor to PCI to
CSA Bridge (rev 02)
00:06.0 System peripheral: Intel Corporation 82875P/E7210 Processor to
I/O Memory Interface (rev 02)
00:1c.0 PCI bridge: Intel Corporation 6300ESB 64-bit PCI-X Bridge (rev 02)
00:1d.0 USB Controller: Intel Corporation 6300ESB USB Universal Host
Controller (rev 02)
00:1d.1 USB Controller: Intel Corporation 6300ESB USB Universal Host
Controller (rev 02)
00:1d.4 System peripheral: Intel Corporation 6300ESB Watchdog Timer (rev 02)
00:1d.5 PIC: Intel Corporation 6300ESB I/O Advanced Programmable
Interrupt Controller (rev 02)
00:1d.7 USB Controller: Intel Corporation 6300ESB USB2 Enhanced Host
Controller (rev 02)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 0a)
00:1f.0 ISA bridge: Intel Corporation 6300ESB LPC Interface Controller
(rev 02)
00:1f.1 IDE interface: Intel Corporation 6300ESB PATA Storage Controller
(rev 02)
00:1f.3 SMBus: Intel Corporation 6300ESB SMBus Controller (rev 02)
01:01.0 Ethernet controller: Intel Corporation 82547GI Gigabit Ethernet
Controller
02:01.0 SCSI storage controller: Adaptec AIC-7901 U320 (rev 10)
02:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701
Gigabit Ethernet (rev 15)
02:03.0 Mass storage controller: Silicon Image, Inc. SiI 3124 PCI-X
Serial ATA Controller (rev 02)
02:04.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet
Controller (rev 05)
03:00.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27)
03:01.0 Ethernet controller: Intel Corporation 82557/8/9/0/1 Ethernet
Pro 100 (rev 10)
03:02.0 Ethernet controller: 3Com Corporation 3c905C-TX/TX-M [Tornado]
(rev 78)
root@Arzamas:~#

root@Arzamas:~# lspci -v -s 02:
02:01.0 SCSI storage controller: Adaptec AIC-7901 U320 (rev 10)
	Subsystem: Intel Corporation Device 342f
	Flags: bus master, 66MHz, slow devsel, latency 64, IRQ 27
	I/O ports at b800 [disabled] [size=256]
	Memory at fc5fe000 (64-bit, non-prefetchable) [size=8K]
	I/O ports at b400 [disabled] [size=256]
	Expansion ROM at fc500000 [disabled] [size=512K]
	Capabilities: [dc] Power Management version 2
	Capabilities: [a0] Message Signalled Interrupts: Mask- 64bit+ Queue=0/1
Enable-
	Capabilities: [94] PCI-X non-bridge device
	Kernel driver in use: aic79xx

02:02.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5701 Gigabit Ethernet (rev 15)
Subsystem: Compaq Computer Corporation NC7770 Gigabit Server Adapter
(PCI-X, 10/100/1000-T)
Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 24 Memory at fc5e0000 (64-bit, non-prefetchable) [size=64K] [virtual] Expansion ROM at fc5d0000 [disabled] [size=64K]
Capabilities: [40] PCI-X non-bridge device
Capabilities: [48] Power Management version 2
Capabilities: [50] Vital Product Data <?>
Capabilities: [58] Message Signalled Interrupts: Mask- 64bit+ Queue=0/3
Enable-
Kernel driver in use: tg3

02:03.0 Mass storage controller: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller (rev 02) <----- where the 4 drives are attached to
Subsystem: Silicon Image, Inc. SiI 3124 PCI-X Serial ATA Controller
Flags: bus master, stepping, 66MHz, medium devsel, latency 64, IRQ 25 Memory at fc5fd800 (64-bit, non-prefetchable) [size=128] Memory at fc5f0000 (64-bit, non-prefetchable) [size=32K] I/O ports at bc00 [size=16]
Expansion ROM at fc480000 [disabled] [size=512K]
Capabilities: [64] Power Management version 2
Capabilities: [40] PCI-X non-bridge device
Capabilities: [54] Message Signalled Interrupts: Mask- 64bit+ Queue=0/0
Enable-
Kernel driver in use: sata_sil24

02:04.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
Subsystem: Intel Corporation PRO/1000 GT Desktop Adapter
Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 26 Memory at fc5a0000 (32-bit, non-prefetchable) [size=128K] Memory at fc580000 (32-bit, non-prefetchable) [size=128K] I/O ports at b000 [size=64]
Expansion ROM at fc460000 [disabled] [size=128K]
Capabilities: [dc] Power Management version 2
Capabilities: [e4] PCI-X non-bridge device
Kernel driver in use: e1000

root@Arzamas:~#

root@Arzamas:~# cat /proc/interrupts
           CPU0       CPU1
  0:        127          0   IO-APIC-edge      timer
  1:          8          0   IO-APIC-edge      i8042
  6:          3          0   IO-APIC-edge      floppy
  8:          2          0   IO-APIC-edge      rtc
 10:          0          0   IO-APIC-fasteoi   acpi
 12:        105          0   IO-APIC-edge      i8042
 14:        100          0   IO-APIC-edge      ata_piix
 15:          0          0   IO-APIC-edge      ata_piix
 16:     245660          0   IO-APIC-fasteoi   uhci_hcd:usb2, wan_a
 17:       5967          0   IO-APIC-fasteoi   lan_untr
 18:      92684          0   IO-APIC-fasteoi   lan_main
 19:          0          0   IO-APIC-fasteoi   uhci_hcd:usb3
 23:          2          0   IO-APIC-fasteoi   ehci_hcd:usb1
 24:      73850          0   IO-APIC-fasteoi   wan_b
 25:    1032325          0   IO-APIC-fasteoi   sata_sil24
 26:       5494          0   IO-APIC-fasteoi   lan_srv
 27:         22          0   IO-APIC-fasteoi   aic79xx
NMI:          0          0   Non-maskable interrupts
LOC:    1383148    1381673   Local timer interrupts
RES:      52448      46803   Rescheduling interrupts
CAL:       2552       1883   function call interrupts
TLB:       8035      16978   TLB shootdowns
TRM:          0          0   Thermal event interrupts
SPU:          0          0   Spurious interrupts
ERR:          0
MIS:          0
root@Arzamas:~#

Hello Peter,

I don't know what it is going on, but:

No! You you do have PCI-X, so 64bit with 66MHz. Maybe you are even luckily and it is as 133MHz, depends on how many other devices are connected to this bus. lspci will always say it is running at 66MHz, though, it also might run at 100 or 133 MHz,

Cheers,
Bernd

Bernd Schubert
Q-Leap Networks GmbH

And on the same bus (the last lspci) I have:

02:04.0 Ethernet controller: Intel Corporation 82541PI Gigabit Ethernet Controller (rev 05)
Subsystem: Intel Corporation PRO/1000 GT Desktop Adapter
Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 26 Memory at fc5a0000 (32-bit, non-prefetchable) [size=128K] Memory at fc580000 (32-bit, non-prefetchable) [size=128K] I/O ports at b000 [size=64]
Expansion ROM at fc460000 [disabled] [size=128K]
Capabilities: [dc] Power Management version 2
Capabilities: [e4] PCI-X non-bridge device
Kernel driver in use: e1000

which afaik brings everything to 32bits no?

Hmm, I don't know what happens if a 32-bit card is in a 64bit PCI-X bus. Wouldn't it only use half of the available lines for this card and all lines for the 64bit cards? Hmm, on the other hand your maximum combined rate seems to be at around 240 MB/s, which is close to 32bit x 66 MHz.Any chance you can remove this NIC and retest?

Cheers,
Bernd

Bernd Schubert
Q-Leap Networks GmbH

Seems like it is the same equipment that you also did the raid10,f2 tests for. I think it would be interesting to have a consolidated list of performance comparisons, when you have completed your tweakings. Is that part of your plan?

best regards
keld

Not on this machine, but yes, I am planning to do something like this on my workstation. I need to get my drives/arrays in order first (got bitten by the 7200.11 fiasco).
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Also see noatime, relatime, etc.

Bill Davidsen <[email protected]>
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark 

I have no idea why you "should" be seeing double drive speed. All drives have to be written, so you'd logically see single drive speed.

-hpa

H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

Hello,

2009/2/28 H. Peter Anvin <[email protected]>:

I'm afraid that might be incorrect.

Let's assume we want to write 100MB of data onto a 4 drive raid6. Let's divide 100MB of data into two parts, say A and B, each 50MB big. Writing the data on the raid, would mean writing:
* A on disk1
* B on disk2
* XOR(A,B) on disk3
* Q(A,B) on disk4
That is actually assuming 50MB chunk, and whole chunk writes, etc. Each of written portions would have been 50MB in size. That sounds reasonable to me, as with 2 data disks, only half of data has to be written on each. The fact that disks are really striped with data, XOR and Q doesn't change the image in terms of amount written.

I do hope I had understood the situation correctly, but I'll be ever happy to be proved wrong.

Kind regards,
Mike

Because with properly adjusted elevators and chunk sizes it is reasonable to expect N * S write speed from _any_ raid, where N is the number of different data bearing disks in a stripe, and S is the speed of a hard drive (assuming the drive speeds are equal). So for raid5 we have N = numdisks-1, for raid6 numdisks-2, for raid10 -n4 -pf3 we get 4-(3-1) and so on. I have personally verified the write behavior for raid10 and raid5, don't see why it should/would be different for raid6.

In any case the witnessed problem was due to a hardware misconfiguration, which has not been resolved to this day. Thus thread is dead :)

Ah, sorry, yes you're of course right. I was thinking about latency, not throughput, for some idiotic reason.

-hpa

H. Peter Anvin, Intel Open Source Technology Center
I work for Intel.  I don't speak on their behalf.

--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

That's a lovely theory, but in practice I have to say I have never measured any such thing, using benchmarks intended to match real world, or even heavy disk writes of a dumb nature like dd. I have tested through the raw device, and through filesystems, tuned stripe-cache-size and buffers, tried setting "stride" in ext3, all to conclude that with raid5 I see essentially write speed of 1x a single drive and read speed of (N-1)x as you suggest. Actually, looking at results for arrays with more drives I can see a trend to write at (N/3)x speed, being a seek-write for full chunk data and seef-read-write for XOR. But even on six drive arrays I don't get near (N-1)x in measurable.

Bill Davidsen <[email protected]>
 "Woe unto the statesman who makes war without a reason that will still
 be valid when the war is over..." Otto von Bismark