Skip to contentSkip to conversation

md RAID over SATA performance

DISCUSSION J.A. Magallon 6 participants 10 messages Sep 3, 2004 – Sep 4, 2004

lore
Share & tools
lore
#0 lore raw

Hi all...

I am buildin an array with a linux box, and I run 2.6.8.1 (not stock, but the mandrake cooker version).

Disks are 6 SATA drives, plugged to a couple of Promise FastTrak S150 TX4 cards.

Problem is that i get really poor performance. A single drive gives about 57 Mb/s, and a raid1 with two just gives about 64 Mb/s (measured with hdparm -tT). With the six drives:

nada:~/soft/kernel# hdparm -tT /dev/sda

/dev/sda:

Timing buffer-cache reads:   844 MB in  2.01 seconds = 419.96 MB/sec
Timing buffered disk reads:  174 MB in  3.01 seconds =  57.84 MB/sec
nada:~/soft/kernel# hdparm -tT /dev/md0

/dev/md0:

Timing buffer-cache reads:   884 MB in  2.00 seconds = 441.63 MB/sec
Timing buffered disk reads:  310 MB in  3.01 seconds = 103.07 MB/sec

The real goal is to build a raid5 system, but I just tested with raid1.

Any ideas/suggestions ?

TIA

More info:

nada:~/soft/kernel# mdadm --detail /dev/md0
/dev/md0:
       Version : 00.90.01
 Creation Time : Fri Sep  3 01:52:22 2004
    Raid Level : raid0
    Array Size : 1470669312 (1402.54 GiB 1505.97 GB)
  Raid Devices : 6
 Total Devices : 6
Preferred Minor : 0
   Persistence : Superblock is persistent

Update Time : Fri Sep 3 01:52:22 2004
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

    Chunk Size : 128K

          UUID : c7373f10:3d04436b:a7d2bbce:8317b56f
        Events : 0.14

Number Major Minor RaidDevice State

      0       8        1        0      active sync   /dev/sda1
      1       8       17        1      active sync   /dev/sdb1
      2       8       33        2      active sync   /dev/sdc1
      3       8       49        3      active sync   /dev/sdd1
      4       8       65        4      active sync   /dev/sde1
      5       8       81        5      active sync   /dev/sdf1
nada:~/soft/kernel# lspci
00:00.0 Host bridge: VIA Technologies, Inc. VT8633 [Apollo Pro266] (rev 01)
00:01.0 PCI bridge: VIA Technologies, Inc. VT8633 [Apollo Pro266 AGP]
00:07.0 RAID bus controller: Promise Technology, Inc. PDC20319 (FastTrak S150 TX4) (rev 02)
00:08.0 RAID bus controller: Promise Technology, Inc. PDC20319 (FastTrak S150 TX4) (rev 02)
00:0b.0 Multimedia audio controller: Ensoniq 5880 AudioPCI (rev 02)
00:0c.0 RAID bus controller: Promise Technology, Inc. PDC20267 (FastTrak100/Ultra100) (rev 02)
00:0d.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 08)
00:11.0 ISA bridge: VIA Technologies, Inc. VT8233 PCI to ISA Bridge
00:11.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
00:11.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 1b)
00:11.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 1b)
00:11.4 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 1b)
00:12.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 62)
01:00.0 VGA compatible controller: nVidia Corporation NV11 [GeForce2 MX/MX 400] (rev b2)

--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandrakelinux release 10.1 (Beta 1) for i586
Linux 2.6.8.1-mm4 (gcc 3.4.1 (Mandrakelinux (Alpha 3.4.1-3mdk)) #8

#1 lore raw
Quoted 10 lines · J.A. Magallon

Hi all...

I am buildin an array with a linux box, and I run 2.6.8.1 (not stock, but the mandrake cooker version).

Disks are 6 SATA drives, plugged to a couple of Promise FastTrak S150 TX4 cards.

Problem is that i get really poor performance. A single drive gives about 57 Mb/s, and a raid1 with two just gives about 64 Mb/s (measured with

                ^^^^^

OOPS, I meant raid0...

Quoted 52 lines

hdparm -tT). With the six drives:

nada:~/soft/kernel# hdparm -tT /dev/sda

/dev/sda:

 Timing buffer-cache reads:   844 MB in  2.01 seconds = 419.96 MB/sec
 Timing buffered disk reads:  174 MB in  3.01 seconds =  57.84 MB/sec
nada:~/soft/kernel# hdparm -tT /dev/md0

/dev/md0:

 Timing buffer-cache reads:   884 MB in  2.00 seconds = 441.63 MB/sec
 Timing buffered disk reads:  310 MB in  3.01 seconds = 103.07 MB/sec

The real goal is to build a raid5 system, but I just tested with raid1.

Any ideas/suggestions ?

TIA

More info:

nada:~/soft/kernel# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.01
  Creation Time : Fri Sep  3 01:52:22 2004
     Raid Level : raid0
     Array Size : 1470669312 (1402.54 GiB 1505.97 GB)
   Raid Devices : 6
  Total Devices : 6
Preferred Minor : 0
    Persistence : Superblock is persistent

Update Time : Fri Sep 3 01:52:22 2004
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Chunk Size : 128K

UUID : c7373f10:3d04436b:a7d2bbce:8317b56f
Events : 0.14

Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/sda1
1 8 17 1 active sync /dev/sdb1
2 8 33 2 active sync /dev/sdc1
3 8 49 3 active sync /dev/sdd1
4 8 65 4 active sync /dev/sde1
5 8 81 5 active sync /dev/sdf1

Quoted 16 lines
nada:~/soft/kernel# lspci
00:00.0 Host bridge: VIA Technologies, Inc. VT8633 [Apollo Pro266] (rev 01)
00:01.0 PCI bridge: VIA Technologies, Inc. VT8633 [Apollo Pro266 AGP]
00:07.0 RAID bus controller: Promise Technology, Inc. PDC20319 (FastTrak S150 TX4) (rev 02)
00:08.0 RAID bus controller: Promise Technology, Inc. PDC20319 (FastTrak S150 TX4) (rev 02)
00:0b.0 Multimedia audio controller: Ensoniq 5880 AudioPCI (rev 02)
00:0c.0 RAID bus controller: Promise Technology, Inc. PDC20267 (FastTrak100/Ultra100) (rev 02)
00:0d.0 Ethernet controller: Intel Corp. 82557/8/9 [Ethernet Pro 100] (rev 08)
00:11.0 ISA bridge: VIA Technologies, Inc. VT8233 PCI to ISA Bridge
00:11.1 IDE interface: VIA Technologies, Inc. VT82C586A/B/VT82C686/A/B/VT823x/A/C PIPC Bus Master IDE (rev 06)
00:11.2 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 1b)
00:11.3 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 1b)
00:11.4 USB Controller: VIA Technologies, Inc. VT82xxxxx UHCI USB 1.1 Controller (rev 1b)
00:12.0 Ethernet controller: VIA Technologies, Inc. VT6102 [Rhine-II] (rev 62)
01:00.0 VGA compatible controller: nVidia Corporation NV11 [GeForce2 MX/MX 400] (rev b2)

--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandrakelinux release 10.1 (Beta 1) for i586
Linux 2.6.8.1-mm4 (gcc 3.4.1 (Mandrakelinux (Alpha 3.4.1-3mdk)) #8

#2 lore raw
Quoted 10 lines · J.A. Magallon
Quoted 6 lines · J.A. Magallon

Hi all...

I am buildin an array with a linux box, and I run 2.6.8.1 (not stock, but the mandrake cooker version).

Problem is that i get really poor performance.

Try:
blockdev --getra /dev/sda1
blockdev --getra /dev/md0

and if needed:
blockdev --setra 4096 /dev/sda1
blockdev --setra 4096 /dev/md0

Also if you use lvm:
blockdev --setra 4096 /dev/video_vg/video_lv
as appropriate.

Tune as needed :)

David

#3 lore raw
Quoted 6 lines

blockdev --getra /dev/sda1
blockdev --getra /dev/md0

and if needed:
blockdev --setra 4096 /dev/sda1
blockdev --setra 4096 /dev/md0

The standard hdparm version of the above is the "-a" flag.

Cheers

Signature · 2 lines
Mark Lord
(hdparm keeper & the original "Linux IDE Guy")
#4 lore raw
In reply to J.A. Magallon
Quoted 3 lines · J.A. Magallon

Problem is that i get really poor performance. A single drive gives about 57 Mb/s, and a raid1 with two just gives about 64 Mb/s (measured with hdparm -tT). With the six drives:

Looks about right. Whats your total PCI bus bandwidth, whats your VIA V-Link bandwidth between the bridges ?

#5 lore raw
In reply to J.A. Magallon
Quoted 19 lines · J.A. Magallon

Hi all...

I am buildin an array with a linux box, and I run 2.6.8.1 (not stock, but the mandrake cooker version).

Disks are 6 SATA drives, plugged to a couple of Promise FastTrak S150 TX4 cards.

Problem is that i get really poor performance. A single drive gives about 57 Mb/s, and a raid1 with two just gives about 64 Mb/s (measured with hdparm -tT). With the six drives:

nada:~/soft/kernel# hdparm -tT /dev/sda

/dev/sda:

Timing buffer-cache reads:   844 MB in  2.01 seconds = 419.96 MB/sec
Timing buffered disk reads:  174 MB in  3.01 seconds =  57.84 MB/sec
nada:~/soft/kernel# hdparm -tT /dev/md0
Quoted 6 lines

/dev/md0:

Timing buffer-cache reads:   884 MB in  2.00 seconds = 441.63 MB/sec
Timing buffered disk reads:  310 MB in  3.01 seconds = 103.07 MB/sec

The real goal is to build a raid5 system, but I just tested with raid1.

Here are some real-world figures for you (Not that they mean anything!)

srv:/home/brad# hdparm -t -T /dev/md0

/dev/md0:
Timing buffer-cache reads: 1340 MB in 2.00 seconds = 668.76 MB/sec
BLKGETSIZE failed: File too large
Timing buffered disk reads: 182 MB in 3.02 seconds = 60.31 MB/sec

srv:/home/brad# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1              19G   11G  6.5G  64% /
tmpfs                 252M     0  252M   0% /dev/shm
/dev/hda3             165G   21G  136G  13% /raid0
/dev/md0              2.1T  2.0T  114G  95% /raid
/dev/md2              459G  101G  358G  22% /raid2
srv:/home/brad# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.01
  Creation Time : Sun May  2 18:02:14 2004
     Raid Level : raid5
     Array Size : 2206003968 (2103.81 GiB 2258.95 GB)
    Device Size : 245111552 (233.76 GiB 250.99 GB)
   Raid Devices : 10
  Total Devices : 10
Preferred Minor : 0
    Persistence : Superblock is persistent

Update Time : Fri Sep 3 16:08:03 2004
State : clean, no-errors
Active Devices : 10
Working Devices : 10
Failed Devices : 0
Spare Devices : 0

Layout : left-asymmetric
Chunk Size : 128K

Number Major Minor RaidDevice State
0 8 1 0 active sync /dev/devfs/scsi/host0/bus0/target0/lun0/part1
1 8 17 1 active sync /dev/devfs/scsi/host1/bus0/target0/lun0/part1
2 8 33 2 active sync /dev/devfs/scsi/host2/bus0/target0/lun0/part1
3 8 49 3 active sync /dev/devfs/scsi/host3/bus0/target0/lun0/part1
4 8 65 4 active sync /dev/devfs/scsi/host4/bus0/target0/lun0/part1
5 8 81 5 active sync /dev/devfs/scsi/host5/bus0/target0/lun0/part1
6 8 97 6 active sync /dev/devfs/scsi/host6/bus0/target0/lun0/part1
7 8 113 7 active sync /dev/devfs/scsi/host7/bus0/target0/lun0/part1
8 8 129 8 active sync /dev/sdi1
9 8 145 9 active sync /dev/sdj1
UUID : 05cc3f43:de1ecfa4:83a51293:78015f1e
Events : 0.681631

#6 lore raw
In reply to J.A. Magallon

Even with hardware RAID the numbers I see are not often much better than this.

Unless O_DIRECT is used (hdparm --direct), in which case they immediately jump to a level that is more limited by the PCI and memory speeds of the system.

Eg. 205Mbytes/sec for a 4-drive RAID0 over 64-bit/66Mhz PCI.

I guess the page_cache overhead is rather substantial for the simple read-a-sequential-block test.

Cheers

Signature · 2 lines
Mark Lord
(hdparm keeper & the original "Linux IDE Guy")
#7 lore raw
Quoted 1 line · Mark Lord

Unless O_DIRECT is used (hdparm --direct), in which case they

In which version of hdparm? I'm using 5.6 (I believe it's latest) which
doesn't know --direct switch.

Signature · 2 lines
Tomasz Torcz                 "God, root, what's the difference?" 
[email protected]         "God is more forgiving."   
#8 lore raw

hdparm-5.7 has this feature.

Cheers

Signature · 2 lines
Mark Lord
(hdparm keeper & the original "Linux IDE Guy")
Quoted 8 lines · Tomasz Torcz
Quoted 1 line · Mark Lord

Unless O_DIRECT is used (hdparm --direct), in which case they

In which version of hdparm? I'm using 5.6 (I believe it's latest) which
doesn't know --direct switch.

#9 lore raw
In reply to J.A. Magallon
Quoted 5 lines · J.A. Magallon

Hi all...

I am buildin an array with a linux box, and I run 2.6.8.1 (not stock, but the mandrake cooker version).

Quoted 8 lines

Disks are 6 SATA drives, plugged to a couple of Promise FastTrak S150 TX4 cards.

Problem is that i get really poor performance. A single drive gives about 57 Mb/s, and a raid1 with two just gives about 64 Mb/s (measured with hdparm -tT). With the six drives:

Thanks to everybody for its answers...

My confusion/problem was that I has always tried with only two drives, and with more or less old drives that gave around 20-25Mb/s each, so the raid worked very optimally at about 50 Mb/s.
But with modern drives at 55 Mb/s, I hit the PCI speed limits, I suppose. Board is a Supermicro P3TDDE, Via ApolloPro 266T chipset (aghhh), 33 MHz PCI, 2xPIII@933.

With a stock 2.6.8.1, I get this:

nada:~# hdparm -tT /dev/md0

/dev/md0:

Timing buffer-cache reads:   864 MB in  2.00 seconds = 431.63 MB/sec
Timing buffered disk reads:  252 MB in  3.01 seconds =  83.85 MB/sec

with this setup:

nada:~# mdadm --detail /dev/md0
/dev/md0:
       Version : 00.90.01
 Creation Time : Fri Sep  3 02:17:28 2004
    Raid Level : raid5
    Array Size : 1225557760 (1168.78 GiB 1254.97 GB)
   Device Size : 245111552 (233.76 GiB 250.99 GB)
  Raid Devices : 6
 Total Devices : 6
Preferred Minor : 0
   Persistence : Superblock is persistent

Update Time : Fri Sep 3 19:01:39 2004
State : clean
Active Devices : 6
Working Devices : 6
Failed Devices : 0
Spare Devices : 0

Layout : left-symmetric
Chunk Size : 256K

UUID : fd6fcad0:21da140b:072a82b1:11b3db21
Events : 0.239

Number Major Minor RaidDevice State

      0       8        1        0      active sync   /dev/sda1
      1       8       17        1      active sync   /dev/sdb1
      2       8       33        2      active sync   /dev/sdc1
      3       8       49        3      active sync   /dev/sdd1
      4       8       65        4      active sync   /dev/sde1
      5       8       81        5      active sync   /dev/sdf1

--
J.A. Magallon <jamagallon()able!es> \ Software is like sex:
werewolf!able!es \ It's better when it's free
Mandrakelinux release 10.1 (RC 1) for i586
Linux 2.6.8.1-mm4 (gcc 3.4.1 (Mandrakelinux (Alpha 3.4.1-3mdk)) #8