md RAID over SATA performance
Hi all...
I am buildin an array with a linux box, and I run 2.6.8.1 (not stock, but the mandrake cooker version).
Disks are 6 SATA drives, plugged to a couple of Promise FastTrak S150 TX4 cards.
Problem is that i get really poor performance. A single drive gives about 57 Mb/s, and a raid1 with two just gives about 64 Mb/s (measured with hdparm -tT). With the six drives:
nada:~/soft/kernel# hdparm -tT /dev/sda
/dev/sda:
Timing buffer-cache reads: 844 MB in 2.01 seconds = 419.96 MB/sec Timing buffered disk reads: 174 MB in 3.01 seconds = 57.84 MB/sec nada:~/soft/kernel# hdparm -tT /dev/md0
/dev/md0:
Timing buffer-cache reads: 884 MB in 2.00 seconds = 441.63 MB/sec Timing buffered disk reads: 310 MB in 3.01 seconds = 103.07 MB/sec
The real goal is to build a raid5 system, but I just tested with raid1.
Any ideas/suggestions ?
TIA
More info:
nada:~/soft/kernel# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.01
Creation Time : Fri Sep 3 01:52:22 2004
Raid Level : raid0
Array Size : 1470669312 (1402.54 GiB 1505.97 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent Update Time : Fri Sep 3 01:52:22 2004
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Chunk Size : 128K
UUID : c7373f10:3d04436b:a7d2bbce:8317b56f
Events : 0.14Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
2 8 33 2 active sync /dev/sdc1
3 8 49 3 active sync /dev/sdd1
4 8 65 4 active sync /dev/sde1
5 8 81 5 active sync /dev/sdf1nada:~/soft/kernel# lspci 00:00.0 Host bridge: VIA Technologies, Inc. VT8633 [Apollo Pro266] (rev 01) 00:01.0 PCI bridge: VIA Technologies, Inc. VT8633 [Apollo Pro266 AGP] 00:07.0 RAID bus controller: Promise Technology, Inc. PDC20319 (FastTrak S150 TX4) (rev 02) 00:08.0 RAID bus controller: Promise Technology, Inc. PDC20319 (FastTrak S150 TX4) (rev 02) 00:0b.0 Multimedia audio controller: Ensoniq 5880 AudioPCI (rev 02) 00:0c.0 RAID bus controller: Promise Technology, Inc. PDC20267 (FastTrak100/Ultra100) (rev 02) 00:0d.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 08) 00:11.0 ISA bridge: VIA Technologies, Inc. VT8233 PCI to ISA Bridge 00:11.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06) 00:11.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 1b) 00:11.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 1b) 00:11.4 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 1b) 00:12.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 62) 01:00.0 VGA compatible controller: nVidia Corporation NV11 [GeForce2 MX/MX 400] (rev b2)
--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandrakelinux release 10.1 (Beta 1) for i586
Linux 2.6.8.1-mm4 (gcc 3.4.1 (Mandrakelinux (Alpha 3.4.1-3mdk)) #8
Quoted 10 lines · J.A. Magallon
Hi all...
I am buildin an array with a linux box, and I run 2.6.8.1 (not stock, but the mandrake cooker version).
Disks are 6 SATA drives, plugged to a couple of Promise FastTrak S150 TX4 cards.
Problem is that i get really poor performance. A single drive gives about 57 Mb/s, and a raid1 with two just gives about 64 Mb/s (measured with
^^^^^
OOPS, I meant raid0...
Quoted 52 lines
hdparm -tT). With the six drives:
nada:~/soft/kernel# hdparm -tT /dev/sda
/dev/sda:Timing buffer-cache reads: 844 MB in 2.01 seconds = 419.96 MB/sec Timing buffered disk reads: 174 MB in 3.01 seconds = 57.84 MB/sec nada:~/soft/kernel# hdparm -tT /dev/md0
/dev/md0:Timing buffer-cache reads: 884 MB in 2.00 seconds = 441.63 MB/sec Timing buffered disk reads: 310 MB in 3.01 seconds = 103.07 MB/secThe real goal is to build a raid5 system, but I just tested with raid1.
Any ideas/suggestions ?
TIA
More info:
nada:~/soft/kernel# mdadm --detail /dev/md0 /dev/md0: Version : 00.90.01 Creation Time : Fri Sep 3 01:52:22 2004 Raid Level : raid0 Array Size : 1470669312 (1402.54 GiB 1505.97 GB) Raid Devices : 6 Total Devices : 6 Preferred Minor : 0 Persistence : Superblock is persistentUpdate Time : Fri Sep 3 01:52:22 2004
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0Chunk Size : 128K
UUID : c7373f10:3d04436b:a7d2bbce:8317b56f
Events : 0.14Number Major Minor RaidDevice State
0 8 1 0 active sync/dev/sda1
1 8 17 1 active sync/dev/sdb1
2 8 33 2 active sync/dev/sdc1
3 8 49 3 active sync/dev/sdd1
4 8 65 4 active sync/dev/sde1
5 8 81 5 active sync/dev/sdf1
Quoted 16 lines
nada:~/soft/kernel# lspci 00:00.0 Host bridge: VIA Technologies, Inc. VT8633 [Apollo Pro266] (rev 01) 00:01.0 PCI bridge: VIA Technologies, Inc. VT8633 [Apollo Pro266 AGP] 00:07.0 RAID bus controller: Promise Technology, Inc. PDC20319 (FastTrak S150 TX4) (rev 02) 00:08.0 RAID bus controller: Promise Technology, Inc. PDC20319 (FastTrak S150 TX4) (rev 02) 00:0b.0 Multimedia audio controller: Ensoniq 5880 AudioPCI (rev 02) 00:0c.0 RAID bus controller: Promise Technology, Inc. PDC20267 (FastTrak100/Ultra100) (rev 02) 00:0d.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 08) 00:11.0 ISA bridge: VIA Technologies, Inc. VT8233 PCI to ISA Bridge 00:11.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06) 00:11.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 1b) 00:11.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 1b) 00:11.4 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 1b) 00:12.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 62) 01:00.0 VGA compatible controller: nVidia Corporation NV11 [GeForce2 MX/MX 400] (rev b2)
--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandrakelinux release 10.1 (Beta 1) for i586
Linux 2.6.8.1-mm4 (gcc 3.4.1 (Mandrakelinux (Alpha 3.4.1-3mdk)) #8
Quoted 10 lines · J.A. Magallon
Quoted 6 lines · J.A. Magallon
Hi all...
I am buildin an array with a linux box, and I run 2.6.8.1 (not stock, but the mandrake cooker version).
Problem is that i get really poor performance.
Try:
blockdev --getra /dev/sda1
blockdev --getra /dev/md0
and if needed:
blockdev --setra 4096 /dev/sda1
blockdev --setra 4096 /dev/md0
Also if you use lvm:
blockdev --setra 4096 /dev/video_vg/video_lv
as appropriate.
Tune as needed :)
David
Quoted 6 lines
blockdev --getra
/dev/sda1
blockdev --getra/dev/md0and if needed:
blockdev --setra 4096/dev/sda1
blockdev --setra 4096/dev/md0
The standard hdparm version of the above is the "-a" flag.
Cheers
Signature · 2 lines
Mark Lord (hdparm keeper & the original "Linux IDE Guy")
Quoted 3 lines · J.A. Magallon
Problem is that i get really poor performance. A single drive gives about 57 Mb/s, and a raid1 with two just gives about 64 Mb/s (measured with hdparm -tT). With the six drives:
Looks about right. Whats your total PCI bus bandwidth, whats your VIA V-Link bandwidth between the bridges ?
Quoted 19 lines · J.A. Magallon
Hi all...
I am buildin an array with a linux box, and I run 2.6.8.1 (not stock, but the mandrake cooker version).
Disks are 6 SATA drives, plugged to a couple of Promise FastTrak S150 TX4 cards.
Problem is that i get really poor performance. A single drive gives about 57 Mb/s, and a raid1 with two just gives about 64 Mb/s (measured with hdparm -tT). With the six drives:
nada:~/soft/kernel# hdparm -tT /dev/sda
/dev/sda:Timing buffer-cache reads: 844 MB in 2.01 seconds = 419.96 MB/sec Timing buffered disk reads: 174 MB in 3.01 seconds = 57.84 MB/sec nada:~/soft/kernel# hdparm -tT /dev/md0
Quoted 6 lines
/dev/md0:Timing buffer-cache reads: 884 MB in 2.00 seconds = 441.63 MB/sec Timing buffered disk reads: 310 MB in 3.01 seconds = 103.07 MB/secThe real goal is to build a raid5 system, but I just tested with raid1.
Here are some real-world figures for you (Not that they mean anything!)
srv:/home/brad# hdparm -t -T /dev/md0
/dev/md0:
Timing buffer-cache reads: 1340 MB in 2.00 seconds = 668.76 MB/sec
BLKGETSIZE failed: File too large
Timing buffered disk reads: 182 MB in 3.02 seconds = 60.31 MB/sec
srv:/home/brad# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/hda1 19G 11G 6.5G 64% /
tmpfs 252M 0 252M 0% /dev/shm
/dev/hda3 165G 21G 136G 13% /raid0
/dev/md0 2.1T 2.0T 114G 95% /raid
/dev/md2 459G 101G 358G 22% /raid2
srv:/home/brad# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.01
Creation Time : Sun May 2 18:02:14 2004
Raid Level : raid5
Array Size : 2206003968 (2103.81 GiB 2258.95 GB)
Device Size : 245111552 (233.76 GiB 250.99 GB)
Raid Devices : 10
Total Devices : 10
Preferred Minor : 0
Persistence : Superblock is persistent Update Time : Fri Sep 3 16:08:03 2004
State : clean, no-errors
Active Devices : 10
Working Devices : 10
Failed Devices : 0
Spare Devices : 0
Layout : left-asymmetric
Chunk Size : 128K
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/devfs/scsi/host0/bus0/target0/lun0/part1
1 8 17 1 active sync /dev/devfs/scsi/host1/bus0/target0/lun0/part1
2 8 33 2 active sync /dev/devfs/scsi/host2/bus0/target0/lun0/part1
3 8 49 3 active sync /dev/devfs/scsi/host3/bus0/target0/lun0/part1
4 8 65 4 active sync /dev/devfs/scsi/host4/bus0/target0/lun0/part1
5 8 81 5 active sync /dev/devfs/scsi/host5/bus0/target0/lun0/part1
6 8 97 6 active sync /dev/devfs/scsi/host6/bus0/target0/lun0/part1
7 8 113 7 active sync /dev/devfs/scsi/host7/bus0/target0/lun0/part1
8 8 129 8 active sync /dev/sdi1
9 8 145 9 active sync /dev/sdj1
UUID : 05cc3f43:de1ecfa4:83a51293:78015f1e
Events : 0.681631
Even with hardware RAID the numbers I see are not often much better than this.
Unless O_DIRECT is used (hdparm --direct), in which case they immediately jump to a level that is more limited by the PCI and memory speeds of the system.
Eg. 205Mbytes/sec for a 4-drive RAID0 over 64-bit/66Mhz PCI.
I guess the page_cache overhead is rather substantial for the simple read-a-sequential-block test.
Cheers
Signature · 2 lines
Mark Lord (hdparm keeper & the original "Linux IDE Guy")
Quoted 1 line · Mark Lord
Unless
O_DIRECTis used (hdparm --direct), in which case they
In which version of hdparm? I'm using 5.6 (I believe it's latest) which
doesn't know --direct switch.
Signature · 2 lines
Tomasz Torcz "God, root, what's the difference?" [email protected] "God is more forgiving."
hdparm-5.7 has this feature.
Cheers
Signature · 2 lines
Mark Lord (hdparm keeper & the original "Linux IDE Guy")
Quoted 8 lines · Tomasz Torcz
Quoted 1 line · Mark Lord
Unless
O_DIRECTis used (hdparm --direct), in which case theyIn which version of hdparm? I'm using 5.6 (I believe it's latest) which
doesn't know --direct switch.
Quoted 5 lines · J.A. Magallon
Hi all...
I am buildin an array with a linux box, and I run 2.6.8.1 (not stock, but the mandrake cooker version).
Quoted 8 lines
Disks are 6 SATA drives, plugged to a couple of Promise FastTrak S150 TX4 cards.
Problem is that i get really poor performance. A single drive gives about 57 Mb/s, and a raid1 with two just gives about 64 Mb/s (measured with hdparm -tT). With the six drives:
Thanks to everybody for its answers...
My confusion/problem was that I has always tried with only two drives, and with more or less old drives that gave around 20-25Mb/s each, so the raid worked very optimally at about 50 Mb/s.
But with modern drives at 55 Mb/s, I hit the PCI speed limits, I suppose. Board is a Supermicro P3TDDE, Via ApolloPro 266T chipset (aghhh), 33 MHz PCI, 2xPIII@933.
With a stock 2.6.8.1, I get this:
nada:~# hdparm -tT /dev/md0
/dev/md0:
Timing buffer-cache reads: 864 MB in 2.00 seconds = 431.63 MB/sec Timing buffered disk reads: 252 MB in 3.01 seconds = 83.85 MB/sec
with this setup:
nada:~# mdadm --detail /dev/md0
/dev/md0:
Version : 00.90.01
Creation Time : Fri Sep 3 02:17:28 2004
Raid Level : raid5
Array Size : 1225557760 (1168.78 GiB 1254.97 GB)
Device Size : 245111552 (233.76 GiB 250.99 GB)
Raid Devices : 6
Total Devices : 6
Preferred Minor : 0
Persistence : Superblock is persistent Update Time : Fri Sep 3 19:01:39 2004
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
UUID : fd6fcad0:21da140b:072a82b1:11b3db21
Events : 0.239
Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
2 8 33 2 active sync /dev/sdc1
3 8 49 3 active sync /dev/sdd1
4 8 65 4 active sync /dev/sde1
5 8 81 5 active sync /dev/sdf1--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandrakelinux release 10.1 (RC 1) for i586
Linux 2.6.8.1-mm4 (gcc 3.4.1 (Mandrakelinux (Alpha 3.4.1-3mdk)) #8