RAID 0 over HW RAID
Hello,
We want to combine two HW RAID controllers with RAID 0 using MD and see some performance issues.
Setup:
- Linux 2.6.16-64 bit, Dual XEON, 1 GB RAM
- 2 RAID controllers: ARECA with 7 SATA disks each (RAID5)
- SW RAID 0 over the two HW RAID controllers
- stripe size is always 64k
Measured with IOMETER (MB/s, 64 kb block size with sequential I/O).
one HW RAID controller: - R: 360 W: 240 two HW RAID controllers: - R: 619 W: 480 (one IOMETER worker per device)
MD0 over two HW RAID controllers:
- R 367 W: 433 (one IOMETER worker over md device)
Read throughput is similar to a single controller. Any hint how to improve that? Using a larger block size does not help.
We are considering using MD to combine HW RAID controllers with battery backup support for better data protection. In this scenario md should do no write caching. Is it possible to use something like O_DIRECT with md?
Regards,
Mirko
Quoted 1 line
- 2 RAID controllers: ARECA with 7 SATA disks each (RAID5)
what are the /sys/block settings for the blockdevs these export? I'm thinking about max*sectors_kb.
Quoted 3 lines
- stripe size is always 64k
Measured with IOMETER (MB/s, 64 kb block size with sequential I/O).
I don't see how that could be expected to work well. you're doing sequential 64K IO from user-space (that is, inherently one at a time), and those map onto a single chunk via md raid0. (well, if the IOs are aligned - but in any case you won't be generating 128K IOs which would be the min expected to really make the raid0 shine.)
Quoted 10 lines
one HW RAID controller:
- R: 360 W: 240
two HW RAID controllers:
- R: 619 W: 480 (one IOMETER worker per device) MD0 over two HW RAID controllers:
- R 367 W: 433 (one IOMETER worker over md device)Read throughput is similar to a single controller. Any hint how to improve that?
Using a larger block size does not help.
which blocksize are you talking about? larger blocksize at the app level should help. _smaller_ block/chunk size at the md level. and of course those both interact with the block size prefered by the areca.
Quoted 2 lines
We are considering using MD to combine HW RAID controllers with battery backup support for better data protection.
maybe. all this does is permit the HW controller to reorder transactions, which is not going to matter much if your loads are, in fact, sequential.
Quoted 2 lines
In this scenario md should do
no write caching.
in my humble understanding, MD doesn't do WC.
Quoted 1 line
Is it possible to use something like
O_DIRECTwith md?
certainly (exactly O_DIRECT). this is mainly instruction to the pagecache, not MD. I presume O_DIRECT mainly just follows a write by a barrier, which MD can respect and pass to the areca driver (which presumably also respects it, though the point of battery-backed cache would be to let the barrier complete before the IO...)
Hello,
/sys/block/sdc/queue/max_sectors_kb is 256 for both HW RAID devices.
We have tested with larger block sizes (256K, 1MB) with actually provides a bit lower performance. Access is sequentiell.
We made some more tests with dd for measuring performance. With two strange issues where I have no explanation for.
1)
test:~# dd if=/dev/sdc of=/dev/null bs=128k count=30000 30000+0 records in 30000+0 records out 3932160000 bytes transferred in 11.311464 seconds (347626088 bytes/sec)
test:~# dd if=/dev/sdc1 of=/dev/null bs=128k count=30000 30000+0 records in 30000+0 records out 3932160000 bytes transferred in 21.004938 seconds (187201694 bytes/sec)
Read performance from the same HW RAID is different for entire device (sdc) compared with partition (sdc1).
2)
test:~# dd if=/dev/md0 of=/dev/null bs=128k count=30000 30000+0 records in 30000+0 records out 3932160000 bytes transferred in 9.950705 seconds (395163959 bytes/sec)
test:~# dd if=/dev/md0 of=/dev/null bs=128k count=30000 skip=1000 30000+0 records in 30000+0 records out 3932160000 bytes transferred in 6.398646 seconds (614530000 bytes/sec)
When skipping some MBytes performance improves significantly and is almost the sum of the two HW RAID controllers.
Regards,
Mirko
Quoted 57 lines · Mark Hahn
Quoted 2 lines
- 2 RAID controllers: ARECA with 7 SATA disks each (RAID5)
what are the
/sys/blocksettings for the blockdevs these export? I'm thinking about max*sectors_kb.Quoted 4 lines
- stripe size is always 64k
Measured with IOMETER (MB/s, 64 kb block size with sequential I/O).
I don't see how that could be expected to work well. you're doing sequential 64K IO from user-space (that is, inherently one at a time), and those map onto a single chunk via md raid0. (well, if the IOs
are aligned - but in any case you won't be generating 128K IOs which
would be the min expected to really make the raid0 shine.)Quoted 10 lines
one HW RAID controller:
- R: 360 W: 240
two HW RAID controllers:
- R: 619 W: 480 (one IOMETER worker per device) MD0 over two HW RAID controllers:
- R 367 W: 433 (one IOMETER worker over md device)Read throughput is similar to a single controller. Any hint how to improve that? Using a larger block size does not help.
which blocksize are you talking about? larger blocksize at the app level should help. _smaller_ block/chunk size at the md level. and of course those both interact with the block size prefered by the areca.
Quoted 2 lines
We are considering using MD to combine HW RAID controllers with battery backup support for better data protection.
maybe. all this does is permit the HW controller to reorder transactions, which is not going to matter much if your loads are, in fact, sequential.
Quoted 2 lines
In this scenario md should do no write caching.
in my humble understanding, MD doesn't do WC.
Quoted 2 lines
Is it possible to use something like
O_DIRECTwith md?certainly (exactly
O_DIRECT). this is mainly instruction to the pagecache, not MD. I presumeO_DIRECTmainly just follows a write by a barrier, which MD can respect and pass to the areca driver
(which presumably also respects it, though the point of battery-backed
cache would be to let the barrier complete before the IO...)