RAID-5 streaming read performance
I was wondering what I should expect in terms of streaming read performance when using (software) RAID-5 with four SATA drives. I thought I would get a noticeable improvement compared to reads from a single device, but that's not the case. I tested this by using dd to read 300MB directly from disk partitions /dev/sda7, etc, and also using dd to read 300MB directly from the raid device (/dev/md2 in this case). I get around 57MB/s from each of the disk partitions that make up the raid device, and about 58MB/s from the raid device. On the other hand, if I run parallel reads from the component partitions, I get 25 to 30MB/s each, so the bus can clearly achieve more than 100MB/s.
Before each read, I try to clear the kernel's cache by reading 900MB from an unrelated partition on the disk. (Is this guaranteed to work? Is there a better and/or faster way to clear cache?)
I have AAM quiet mode/low performance enabled on the drives, but (a) this shouldn't matter too much for streaming reads, and (b) it's the relative performance of the reads from the partitions and the RAID device that I'm curious about.
I also get poor write performance, but that's harder to isolate because I have to go through the lvm and filesystem layers too.
I also get poor performance from my RAID-1 array and my other RAID-5 arrays.
Details of my tests and set-up below.
Thanks for any suggestions,
Dan
System:
- Athlon 2500+
- kernel 2.6.12.2 (also tried 2.6.11.11)
- four SATA drives (3 160G, 1 200G); Samsung Spinpoint
- SiI3114 controller (latency_timer=32 by default; tried 128 too)
- 1G ram
- blockdev --getra /dev/sda --> 256 (didn't play with these) - blockdev --getra /dev/md2 --> 768 (didn't play with this)
- tried anticipatory, deadline and cfq schedules, with no significant difference.
- machine essentially idle during tests
Here is part of /proc/mdstat (the full output is below):
md2 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda7[0]
218612160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
Here's the test script and output:
# Clear cache:
dd if=/dev/sda8 of=/dev/null bs=1M count=900 > /dev/null 2>&1 for f in sda7 sdb5 sdc5 sdd5 ; do
echo $f
dd if=/dev/$f of=/dev/null bs=1M count=300 2>&1 | grep bytes/sec echo
done
# Clear cache:
dd if=/dev/sda8 of=/dev/null bs=1M count=900 > /dev/null 2>&1 for f in md2 ; do
echo $f
dd if=/dev/$f of=/dev/null bs=1M count=300 2>&1 | grep bytes/sec echo
done
Output:
sda7
314572800 bytes transferred in 5.401071 seconds (58242671 bytes/sec)
sdb5
314572800 bytes transferred in 5.621170 seconds (55962158 bytes/sec)
sdc5
314572800 bytes transferred in 5.635491 seconds (55819947 bytes/sec)
sdd5
314572800 bytes transferred in 5.333374 seconds (58981951 bytes/sec)
md2
314572800 bytes transferred in 5.386627 seconds (58398846 bytes/sec)
# cat /proc/mdstat
md1 : active raid5 sdd1[2] sdc1[1] sda2[0]
578048 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]md4 : active raid5 sdd2[3] sdc2[2] sdb2[1] sda6[0]
30748032 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
md2 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda7[0]
218612160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
md3 : active raid5 sdd6[3] sdc6[2] sdb6[1] sda8[0]
218636160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]
md0 : active raid1 sdb1[0] sda5[1]
289024 blocks [2/2] [UU]
# mdadm --detail /dev/md2
/dev/md2:
Version : 00.90.01
Creation Time : Mon Jul 4 23:54:34 2005
Raid Level : raid5
Array Size : 218612160 (208.48 GiB 223.86 GB)
Device Size : 72870720 (69.49 GiB 74.62 GB)
Raid Devices : 4
Total Devices : 4
Preferred Minor : 2
Persistence : Superblock is persistent Update Time : Thu Jul 7 21:52:50 2005
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
UUID : c4056d19:7b4bb550:44925b88:91d5bc8a
Events : 0.10873823
Number Major Minor RaidDevice State
0 8 7 0 active sync /dev/sda7
1 8 21 1 active sync /dev/sdb5
2 8 37 2 active sync /dev/sdc5
3 8 53 3 active sync /dev/sdd5
Quoted 30 lines · Dan Christensen
I was wondering what I should expect in terms of streaming read performance when using (software) RAID-5 with four SATA drives. I thought I would get a noticeable improvement compared to reads from a single device, but that's not the case. I tested this by using dd to read 300MB directly from disk partitions
/dev/sda7, etc, and also using dd to read 300MB directly from the raid device (/dev/md2in this case). I get around 57MB/s from each of the disk partitions that make up the raid device, and about 58MB/s from the raid device. On the other hand, if I run parallel reads from the component partitions, I get 25 to 30MB/s each, so the bus can clearly achieve more than 100MB/s.Before each read, I try to clear the kernel's cache by reading 900MB from an unrelated partition on the disk. (Is this guaranteed to work? Is there a better and/or faster way to clear cache?)
I have AAM quiet mode/low performance enabled on the drives, but (a) this shouldn't matter too much for streaming reads, and (b) it's the relative performance of the reads from the partitions and the RAID device that I'm curious about.
I also get poor write performance, but that's harder to isolate because I have to go through the lvm and filesystem layers too.
I also get poor performance from my RAID-1 array and my other RAID-5 arrays.
Details of my tests and set-up below.
Thanks for any suggestions,
Quoted 8 lines
Dan
System:
- Athlon 2500+
- kernel 2.6.12.2 (also tried 2.6.11.11)
- four SATA drives (3 160G, 1 200G); Samsung Spinpoint
- SiI3114 controller (latency_timer=32 by default; tried 128 too)
only 1 card? 4 port? try some other brand card and try to use several cards at the same time. i met some poor cards before.
Quoted 98 lines
- 1G ram
- blockdev --getra /dev/sda --> 256 (didn't play with these) - blockdev --getra /dev/md2 --> 768 (didn't play with this)- tried anticipatory, deadline and cfq schedules, with no significant difference.
- machine essentially idle during testsHere is part of
/proc/mdstat(the full output is below):md2 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda7[0]
218612160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]Here's the test script and output:
# Clear cache:
dd if=/dev/sda8 of=/dev/null bs=1M count=900 >/dev/null2>&1 for f in sda7 sdb5 sdc5 sdd5 ; do
echo $f
dd if=/dev/$f of=/dev/null bs=1M count=300 2>&1 | grep bytes/sec echo
done# Clear cache:
dd if=/dev/sda8 of=/dev/null bs=1M count=900 >/dev/null2>&1 for f in md2 ; do
echo $f
dd if=/dev/$f of=/dev/null bs=1M count=300 2>&1 | grep bytes/sec echo
doneOutput:
sda7
314572800 bytes transferred in 5.401071 seconds (58242671 bytes/sec)sdb5
314572800 bytes transferred in 5.621170 seconds (55962158 bytes/sec)sdc5
314572800 bytes transferred in 5.635491 seconds (55819947 bytes/sec)sdd5
314572800 bytes transferred in 5.333374 seconds (58981951 bytes/sec)md2
314572800 bytes transferred in 5.386627 seconds (58398846 bytes/sec)# cat /proc/mdstat md1 : active raid5 sdd1[2] sdc1[1] sda2[0] 578048 blocks level 5, 64k chunk, algorithm 2 [3/3] [UUU]md4 : active raid5 sdd2[3] sdc2[2] sdb2[1] sda6[0]
30748032 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]md2 : active raid5 sdd5[3] sdc5[2] sdb5[1] sda7[0]
218612160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]md3 : active raid5 sdd6[3] sdc6[2] sdb6[1] sda8[0]
218636160 blocks level 5, 64k chunk, algorithm 2 [4/4] [UUUU]md0 : active raid1 sdb1[0] sda5[1]
289024 blocks [2/2] [UU]# mdadm --detail /dev/md2 /dev/md2: Version : 00.90.01 Creation Time : Mon Jul 4 23:54:34 2005 Raid Level : raid5 Array Size : 218612160 (208.48 GiB 223.86 GB) Device Size : 72870720 (69.49 GiB 74.62 GB) Raid Devices : 4 Total Devices : 4 Preferred Minor : 2 Persistence : Superblock is persistentUpdate Time : Thu Jul 7 21:52:50 2005
State : clean
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0Layout : left-symmetric
Chunk Size : 64KUUID : c4056d19:7b4bb550:44925b88:91d5bc8a
Events : 0.10873823Number Major Minor RaidDevice State
0 8 7 0 active sync/dev/sda7
1 8 21 1 active sync/dev/sdb5
2 8 37 2 active sync/dev/sdc5
3 8 53 3 active sync/dev/sdd5-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 14 lines · Ming Zhang
Quoted 13 lines · Dan Christensen
I was wondering what I should expect in terms of streaming read performance when using (software) RAID-5 with four SATA drives. I thought I would get a noticeable improvement compared to reads from a single device, but that's not the case. I tested this by using dd to read 300MB directly from disk partitions
/dev/sda7, etc, and also using dd to read 300MB directly from the raid device (/dev/md2in this case). I get around 57MB/s from each of the disk partitions that make up the raid device, and about 58MB/s from the raid device. On the other hand, if I run parallel reads from the component partitions, I get 25 to 30MB/s each, so the bus can clearly achieve more than 100MB/s.[...]
Quoted 8 lines
Quoted 5 lines
System:
- Athlon 2500+
- kernel 2.6.12.2 (also tried 2.6.11.11)
- four SATA drives (3 160G, 1 200G); Samsung Spinpoint
- SiI3114 controller (latency_timer=32 by default; tried 128 too)only 1 card? 4 port? try some other brand card and try to use several cards at the same time. i met some poor cards before.
Yes, one 4-port controller. It's on the motherboard.
I thought that since I get good throughput doing parallel reads from the four drives (see above) that would eliminate the controller as the bottleneck. Am I wrong?
Dan
Quoted 32 lines · Dan Christensen
Quoted 23 lines · Ming Zhang
Quoted 11 lines · Dan Christensen
I was wondering what I should expect in terms of streaming read performance when using (software) RAID-5 with four SATA drives. I thought I would get a noticeable improvement compared to reads from a single device, but that's not the case. I tested this by using dd to read 300MB directly from disk partitions
/dev/sda7, etc, and also using dd to read 300MB directly from the raid device (/dev/md2in this case). I get around 57MB/s from each of the disk partitions that make up the raid device, and about 58MB/s from the raid device. On the other hand, if I run parallel reads from the component partitions, I get 25 to 30MB/s each, so the bus can clearly achieve more than 100MB/s.Quoted 9 lines
[...]
System:
- Athlon 2500+
- kernel 2.6.12.2 (also tried 2.6.11.11)
- four SATA drives (3 160G, 1 200G); Samsung Spinpoint
- SiI3114 controller (latency_timer=32 by default; tried 128 too)Quoted 4 lines
only 1 card? 4 port? try some other brand card and try to use several cards at the same time. i met some poor cards before.
Quoted 9 lines
Yes, one 4-port controller. It's on the motherboard.
I thought that since I get good throughput doing parallel reads from the four drives (see above) that would eliminate the controller as the bottleneck. Am I wrong?
Dan
Slavery was abolished in the 1800's.
b-
Quoted 9 lines
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html.
Quoted 22 lines · Dan Christensen
Quoted 20 lines · Ming Zhang
Quoted 18 lines · Dan Christensen
I was wondering what I should expect in terms of streaming read performance when using (software) RAID-5 with four SATA drives. I thought I would get a noticeable improvement compared to reads from a single device, but that's not the case. I tested this by using dd to read 300MB directly from disk partitions
/dev/sda7, etc, and also using dd to read 300MB directly from the raid device (/dev/md2in this case). I get around 57MB/s from each of the disk partitions that make up the raid device, and about 58MB/s from the raid device. On the other hand, if I run parallel reads from the component partitions, I get 25 to 30MB/s each, so the bus can clearly achieve more than 100MB/s.[...]
System:
- Athlon 2500+
- kernel 2.6.12.2 (also tried 2.6.11.11)
- four SATA drives (3 160G, 1 200G); Samsung Spinpoint
- SiI3114 controller (latency_timer=32 by default; tried 128 too)
Quoted 9 lines
Quoted 2 lines
only 1 card? 4 port? try some other brand card and try to use several cards at the same time. i met some poor cards before.
Yes, one 4-port controller. It's on the motherboard.
I thought that since I get good throughput doing parallel reads from the four drives (see above) that would eliminate the controller as the bottleneck. Am I wrong?
have u try the parallel write?
Quoted 5 lines
Dan
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Ming Zhang
have u try the parallel write?
I haven't tested it as thoroughly, as it brings lvm and the filesystem into the mix. (The disks are in "production" use, and are fairly full, so I can't do writes directly to the disk partitions/raid device.)
My preliminary finding is that raid writes are faster than non-raid
writes: 49MB/s vs 39MB/s. Still not stellar performance, though. Question for the list: if I'm doing a long sequential write, naively
each parity block will get recalculated and rewritten several times, once for each non-parity block in the stripe. Does the write-caching that the kernel does mean that each parity block will only get written once?
Dan
Quoted 8 lines · Dan Christensen
Quoted 1 line · Ming Zhang
have u try the parallel write?
I haven't tested it as thoroughly, as it brings lvm and the filesystem into the mix. (The disks are in "production" use, and are fairly full, so I can't do writes directly to the disk partitions/raid device.)
test on a production environment is too dangerous. :P and many benchmark tool u can not perform as well.
LVM overhead is small, but file system overhead is hard to say.
Quoted 9 lines
My preliminary finding is that raid writes are faster than non-raid
writes: 49MB/s vs 39MB/s. Still not stellar performance, though. Question for the list: if I'm doing a long sequential write, naivelyeach parity block will get recalculated and rewritten several times, once for each non-parity block in the stripe. Does the write-caching that the kernel does mean that each parity block will only get written once?
if you write sequential, you might see a stripe write thus write only once.
but if you write on file system and file system has meta data write, log write, then things become complicated. you can use iostat to see r/w on your disk.
Quoted 1 line
Dan
Quoted 2 lines · Ming Zhang
test on a production environment is too dangerous. :P and many benchmark tool u can not perform as well.
Well, I put "production" in quotes because this is just a home mythtv box. :-) So there are plenty of times when it is idle and I can do benchmarks. But I can't erase the hard drives in my tests.
Quoted 1 line
LVM overhead is small, but file system overhead is hard to say.
I expected LVM overhead to be small, but in my tests it is very high. I plan to discuss this on the lvm mailing list after I've got the RAID working as well as possible, but as an example:
Streaming reads using dd to /dev/null:
component partitions, e.g. /dev/sda7: 58MB/s
raid device /dev/md2: 59MB/s lvm device /dev/main/media: 34MB/s
So something is seriously wrong with my lvm set-up (or with lvm). The lvm device is linearly mapped to the initial blocks of md2, so the last two tests should be reading the same blocks from disk.
Quoted 10 lines
Quoted 7 lines
My preliminary finding is that raid writes are faster than non-raid
writes: 49MB/s vs 39MB/s. Still not stellar performance, though. Question for the list: if I'm doing a long sequential write, naivelyeach parity block will get recalculated and rewritten several times, once for each non-parity block in the stripe. Does the write-caching that the kernel does mean that each parity block will only get written once?
if you write sequential, you might see a stripe write thus write only once.
Glad to hear it. In that case, sequential writes to a RAID-5 device with 4 physical drives should be up to 3 times faster than writes to a single device (ignoring journaling, time for calculating parity, bus bandwidth issues, etc).
Is this "stripe write" something that the md layer does to optimize things? In other words, does the md layer cache writes and write a stripe at a time when that's possible? Or is this just an automatic effect of the general purpose write-caching that the kernel does?
Quoted 2 lines
but if you write on file system and file system has meta data write, log write, then things become complicated.
Yes. For now I'm starting at the bottom and working up...
Quoted 1 line
you can use iostat to see r/w on your disk.
Thanks, I'll try that.
Dan
Quoted 17 lines · Dan Christensen
Quoted 2 lines · Ming Zhang
test on a production environment is too dangerous. :P and many benchmark tool u can not perform as well.
Well, I put "production" in quotes because this is just a home mythtv box. :-) So there are plenty of times when it is idle and I can do benchmarks. But I can't erase the hard drives in my tests.
Quoted 1 line
LVM overhead is small, but file system overhead is hard to say.
I expected LVM overhead to be small, but in my tests it is very high. I plan to discuss this on the lvm mailing list after I've got the RAID working as well as possible, but as an example:
Streaming reads using dd to
/dev/null:
Quoted 7 lines
component partitions, e.g.
/dev/sda7: 58MB/sraid device /dev/md2: 59MB/s lvm device /dev/main/media: 34MB/sSo something is seriously wrong with my lvm set-up (or with lvm). The lvm device is linearly mapped to the initial blocks of md2, so the last two tests should be reading the same blocks from disk.
this is interesting.
Quoted 9 lines
Quoted 9 lines
Quoted 8 lines
My preliminary finding is that raid writes are faster than non-raid
writes: 49MB/s vs 39MB/s. Still not stellar performance, though. Question for the list: if I'm doing a long sequential write, naivelyeach parity block will get recalculated and rewritten several times, once for each non-parity block in the stripe. Does the write-caching that the kernel does mean that each parity block will only get written once?
Quoted 7 lines
Quoted 2 lines
if you write sequential, you might see a stripe write thus write only once.
Glad to hear it. In that case, sequential writes to a RAID-5 device with 4 physical drives should be up to 3 times faster than writes to a single device (ignoring journaling, time for calculating parity, bus bandwidth issues, etc).
sounds reasonable but hard to see i feel.
Quoted 5 lines
Is this "stripe write" something that the md layer does to optimize things? In other words, does the md layer cache writes and write a stripe at a time when that's possible? Or is this just an automatic effect of the general purpose write-caching that the kernel does?
md people will give you more details. :)
Quoted 11 lines
Quoted 3 lines
but if you write on file system and file system has meta data write, log write, then things become complicated.
Yes. For now I'm starting at the bottom and working up...
Quoted 1 line
you can use iostat to see r/w on your disk.
Thanks, I'll try that.
Dan
Here's a question for people running software raid-5: do you get significantly better read speed from a raid-5 device than from it's component partitions/hard drives, using the simple dd test I did? Knowing this will help determine whether something is funny with my set-up and/or hardware, or if just had unrealistic expectations about software raid performance.
Feel free to reply directly to me if you don't want to clutter the list. My dumb script is below.
Thanks,
Dan
#!/bin/sh
dd if=/dev/sda8 of=/dev/null bs=1M count=900 > /dev/null 2>&1 for f in sda7 sdb5 sdc5 sdd5 ; do
echo $f;
dd if=/dev/$f of=/dev/null bs=1M count=300 2>&1 | grep bytes/sec echo;
done
dd if=/dev/sda8 of=/dev/null bs=1M count=900 > /dev/null 2>&1 for f in md2 ; do
echo $f;
dd if=/dev/$f of=/dev/null bs=1M count=300 2>&1 | grep bytes/sec echo;
done
Quoted 3 lines · On Wednesday July 13, [email protected] wrote:
Here's a question for people running software raid-5: do you get significantly better read speed from a raid-5 device than from it's component partitions/hard drives, using the simple dd test I did?
SCSI-160 bus, using just 4 of the 15000rpm drives:
each drive by itself delivers about 67M/s
Three drives in parallel deliver 40M/s each, total of 120M/s 4 give 30M/s each or a total of 120M/s
raid5 over 4 drives delivers 132M/s
(We've just ordered a SCSI-320 card to make better use of the drives).
So with top-quality (and price) hardware, it seems to do the right thing.
NeilBrown
Quoted 5 lines · Neil Brown
Quoted 3 lines · On Wednesday July 13, [email protected] wrote:
Here's a question for people running software raid-5: do you get significantly better read speed from a raid-5 device than from it's component partitions/hard drives, using the simple dd test I did?
Quoted 7 lines
SCSI-160 bus, using just 4 of the 15000rpm drives:
each drive by itself delivers about 67M/s
Three drives in parallel deliver 40M/s each, total of 120M/s 4 give 30M/s each or a total of 120M/sraid5 over 4 drives delivers 132M/s
why here a 132MB/s instead of 120MB/s (3 * 40MB/s) as u mentioned? any factor lead to this increase?
Quoted 11 lines
(We've just ordered a SCSI-320 card to make better use of the drives).
So with top-quality (and price) hardware, it seems to do the right thing.
NeilBrown
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · On Wednesday July 13, [email protected] wrote:
Quoted 7 lines · Neil Brown
Quoted 3 lines · On Wednesday July 13, [email protected] wrote:
Here's a question for people running software raid-5: do you get significantly better read speed from a raid-5 device than from it's component partitions/hard drives, using the simple dd test I did?
SCSI-160 bus, using just 4 of the 15000rpm drives:
Quoted 9 lines
Quoted 5 lines
each drive by itself delivers about 67M/s
Three drives in parallel deliver 40M/s each, total of 120M/s 4 give 30M/s each or a total of 120M/sraid5 over 4 drives delivers 132M/s
why here a 132MB/s instead of 120MB/s (3 * 40MB/s) as u mentioned? any factor lead to this increase?
I did another test over 10 times the amount of data, and for 34M/s for 4 concurrent individual drives, which multiplies out to 136M/s. The same amount of data of the raid5 gives 137M/s, so I think it was just experimental error.
NeilBrown
Quoted 19 lines · Neil Brown
Quoted 8 lines · On Wednesday July 13, [email protected] wrote:
Quoted 7 lines · Neil Brown
Quoted 3 lines · On Wednesday July 13, [email protected] wrote:
Here's a question for people running software raid-5: do you get significantly better read speed from a raid-5 device than from it's component partitions/hard drives, using the simple dd test I did?
SCSI-160 bus, using just 4 of the 15000rpm drives:
Quoted 9 lines
Quoted 5 lines
each drive by itself delivers about 67M/s
Three drives in parallel deliver 40M/s each, total of 120M/s 4 give 30M/s each or a total of 120M/sraid5 over 4 drives delivers 132M/s
why here a 132MB/s instead of 120MB/s (3 * 40MB/s) as u mentioned? any factor lead to this increase?
Quoted 4 lines
I did another test over 10 times the amount of data, and for 34M/s for 4 concurrent individual drives, which multiplies out to 136M/s. The same amount of data of the raid5 gives 137M/s, so I think it was just experimental error.
ic. thanks for explanation. yes, agree.
it seems that u can get a near linear performance with decent SCSI HW while what we can get from SATA is not good. :P
Ming
Quoted 2 lines
NeilBrown
Quoted 4 lines · Dan Christensen
Ming Zhang <[email protected]> writes:
Quoted 10 lines
Quoted 4 lines
test on a production environment is too dangerous. :P and many benchmark tool u can not perform as well.
Well, I put "production" in quotes because this is just a home mythtv box. :-) So there are plenty of times when it is idle and I can do benchmarks. But I can't erase the hard drives in my tests.
Me too.
Quoted 7 lines
Quoted 3 lines
LVM overhead is small, but file system overhead is hard to say.
I expected LVM overhead to be small, but in my tests it is very high. I plan to discuss this on the lvm mailing list after I've got the RAID working as well as possible, but as an example:
Quoted 7 lines
Streaming reads using dd to
/dev/null:component partitions, e.g.
/dev/sda7: 58MB/sraid device /dev/md2: 59MB/s lvm device /dev/main/media: 34MB/s
This is not my experience.
What are the readahead settings?
I found significant variation in performance by varying the readahead at raw, md and lvm device level
In my setup I get
component partitions, e.g. /dev/sda7: 39MB/s
raid device /dev/md2: 31MB/s lvm device /dev/main/media: 53MB/s
(oldish system - but note that lvm device is *much* faster)
For your entertainment you may like to try this to 'tune' your readahead
- it's OK to use so long as you're not recording:
(FYI I find that setting readahead to 0 on all devices and 4096 on the lvm device gets me the best performance - which makes sense if you think about it...)
#!/bin/bash
RAW_DEVS="/dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/hdb" MD_DEVS=/dev/md0
LV_DEVS=/dev/huge_vg/huge_lv
LV_RAS="0 128 256 1024 4096 8192"
MD_RAS="0 128 256 1024 4096 8192"
RAW_RAS="0 128 256 1024 4096 8192"
function show_ra()
{
for i in $RAW_DEVS $MD_DEVS $LV_DEVS
do echo -n "$i blockdev --getra $i :: "
done
echo
}
function set_ra()
{
RA=$1
shift
for dev in $@
do
blockdev --setra $RA $dev
done
}
function show_performance()
{
COUNT=4000000
dd if=$LV_DEVS of=/dev/null count=$COUNT 2>&1 | grep seconds
}
for RAW_RA in $RAW_RAS
do
set_ra $RAW_RA $RAW_DEVS
for MD_RA in $MD_RAS
do
set_ra $MD_RA $MD_DEVS
for LV_RA in $LV_RAS
do
set_ra $LV_RA $LV_DEVS
show_ra
show_performance
done
done
done
Quoted 37 lines · David Greaves
Quoted 4 lines · Dan Christensen
Ming Zhang <[email protected]> writes:
Quoted 10 lines
Quoted 4 lines
test on a production environment is too dangerous. :P and many benchmark tool u can not perform as well.
Well, I put "production" in quotes because this is just a home mythtv box. :-) So there are plenty of times when it is idle and I can do benchmarks. But I can't erase the hard drives in my tests.
Me too.
Quoted 7 lines
Quoted 3 lines
LVM overhead is small, but file system overhead is hard to say.
I expected LVM overhead to be small, but in my tests it is very high. I plan to discuss this on the lvm mailing list after I've got the RAID working as well as possible, but as an example:
Quoted 7 lines
Streaming reads using dd to
/dev/null:component partitions, e.g.
/dev/sda7: 58MB/sraid device /dev/md2: 59MB/s lvm device /dev/main/media: 34MB/sThis is not my experience.
What are the readahead settings?
I found significant variation in performance by varying the readahead at raw, md and lvm device level
Quoted 7 lines
In my setup I get
component partitions, e.g.
/dev/sda7: 39MB/sraid device /dev/md2: 31MB/s lvm device /dev/main/media: 53MB/s(oldish system - but note that lvm device is *much* faster)
this is so interesting to see! seems that some read ahead parameters have negative impact.
Quoted 56 lines
For your entertainment you may like to try this to 'tune' your readahead
- it's OK to use so long as you're not recording:(FYI I find that setting readahead to 0 on all devices and 4096 on the lvm device gets me the best performance - which makes sense if you think about it...)
#!/bin/bash
RAW_DEVS="/dev/sda/dev/sdb/dev/sdc/dev/sdd/dev/sde/dev/sdf/dev/hdb" MD_DEVS=/dev/md0
LV_DEVS=/dev/huge_vg/huge_lvLV_RAS="0 128 256 1024 4096 8192"
MD_RAS="0 128 256 1024 4096 8192"
RAW_RAS="0 128 256 1024 4096 8192"function
show_ra()
{
for i in $RAW_DEVS $MD_DEVS $LV_DEVS
do echo -n "$iblockdev --getra $i:: "
done
echo
}function
set_ra()
{
RA=$1
shift
for dev in $@
do
blockdev --setra $RA $dev
done
}function
show_performance()
{
COUNT=4000000
dd if=$LV_DEVS of=/dev/null count=$COUNT 2>&1 | grep seconds
}for
RAW_RAin $RAW_RAS
do
set_ra$RAW_RA $RAW_DEVS
forMD_RAin $MD_RAS
do
set_ra$MD_RA $MD_DEVS
forLV_RAin $LV_RAS
do
set_ra$LV_RA $LV_DEVS
show_ra
show_performance
done
done
done
Quoted 4 lines · Ming Zhang
Quoted 4 lines
component partitions, e.g.
/dev/sda7: 39MB/sraid device /dev/md2: 31MB/s lvm device /dev/main/media: 53MB/s
Quoted 8 lines
Quoted 3 lines
(oldish system - but note that lvm device is *much* faster)
this is so interesting to see! seems that some read ahead parameters have negative impact.
I guess each raw device does some readahead, then the md0 does some readahead and then the lvm does some readahead. Theoretically the md0 and lvm should overlap - but I guess that much of the raw device level readahead is discarded.
David
Quoted 14 lines · David Greaves
Quoted 4 lines · Ming Zhang
Quoted 4 lines
component partitions, e.g.
/dev/sda7: 39MB/sraid device /dev/md2: 31MB/s lvm device /dev/main/media: 53MB/sQuoted 8 lines
Quoted 3 lines
(oldish system - but note that lvm device is *much* faster)
this is so interesting to see! seems that some read ahead parameters have negative impact.
Quoted 7 lines
I guess each raw device does some readahead, then the md0 does some readahead and then the lvm does some readahead. Theoretically the md0 and lvm should overlap - but I guess that much of the raw device level readahead is discarded.
David
for a streaming read, what you readahead now will always be used exact once in near future. at least i think raw device readahead can be turned on at the same time with one of OS components, raid or lvm, readahead being turned on. but in your case, u get best result when turn only one on.
ming
Quoted 31 lines · Ming Zhang
Quoted 19 lines · David Greaves
Quoted 4 lines · Ming Zhang
Quoted 4 lines
component partitions, e.g.
/dev/sda7: 39MB/sraid device /dev/md2: 31MB/s lvm device /dev/main/media: 53MB/sQuoted 11 lines
Quoted 5 lines
(oldish system - but note that lvm device is *much* faster)
this is so interesting to see! seems that some read ahead parameters have negative impact.
Quoted 9 lines
I guess each raw device does some readahead, then the md0 does some readahead and then the lvm does some readahead. Theoretically the md0 and lvm should overlap - but I guess that much of the raw device level readahead is discarded.
David
Quoted 12 lines
for a streaming read, what you readahead now will always be used exact once in near future. at least i think raw device readahead can be turned on at the same time with one of OS components, raid or lvm, readahead being turned on. but in your case, u get best result when turn only one on.
ming
I doubt it's just me - what results do others get with that script?
David
Quoted 45 lines · David Greaves
Quoted 31 lines · Ming Zhang
Quoted 19 lines · David Greaves
Quoted 4 lines · Ming Zhang
Quoted 4 lines
component partitions, e.g.
/dev/sda7: 39MB/sraid device /dev/md2: 31MB/s lvm device /dev/main/media: 53MB/sQuoted 11 lines
Quoted 5 lines
(oldish system - but note that lvm device is *much* faster)
this is so interesting to see! seems that some read ahead parameters have negative impact.
Quoted 9 lines
I guess each raw device does some readahead, then the md0 does some readahead and then the lvm does some readahead. Theoretically the md0 and lvm should overlap - but I guess that much of the raw device level readahead is discarded.
David
Quoted 12 lines
for a streaming read, what you readahead now will always be used exact once in near future. at least i think raw device readahead can be turned on at the same time with one of OS components, raid or lvm, readahead being turned on. but in your case, u get best result when turn only one on.
ming
Quoted 4 lines
I doubt it's just me - what results do others get with that script?
David
my box is in use now. i might try it tomorrow to see what happen. :P
Ming
Quoted 4 lines · On Wednesday July 13, [email protected] wrote:
I guess each raw device does some readahead, then the md0 does some readahead and then the lvm does some readahead. Theoretically the md0 and lvm should overlap - but I guess that much of the raw device level readahead is discarded.
No. Devices don't to readahead (well, modern drives may well read-ahead into an on-drive buffer, but that is completely transparent and separate from any readahead that linux does).
Each device just declares how much readahead it thinks is appropriate for that devices.
The linux mm layer does read-ahead by requesting devices for blocks that haven't actually been asked for by upper layers. The amount of readahead depends on the behaviour of the app doing the reads, and the setting declared by the devices.
raid5 declares a read-ahead size of twice the stripe size. i.e. chunks * (disks-1) * 2.
Possibly it should make it bigger if the underlying devices would all be happy with that, however I haven't given the issue a lot of thought, and it is tunable from userspace.
NeilBrown
Quoted 7 lines · David Greaves
In my setup I get
component partitions, e.g.
/dev/sda7: 39MB/sraid device /dev/md2: 31MB/s lvm device /dev/main/media: 53MB/s(oldish system - but note that lvm device is *much* faster)
Did you test component device and raid device speed using the read-ahead settings tuned for lvm reads? If so, that's not a fair comparison. :-)
Quoted 2 lines
For your entertainment you may like to try this to 'tune' your readahead
- it's OK to use so long as you're not recording:
Thanks, I played around with that a lot. I tuned readahead to optimize lvm device reads, and this improved things greatly. It turns out the default lvm settings had readahead set to 0! But by tuning things, I could get my read speed up to 59MB/s. This is with raw device readahead 256, md device readahead 1024 and lvm readahead 2048. (The speed was most sensitive to the last one, but did seem to depend on the other ones a bit too.)
I separately tuned the raid device read speed. To maximize this, I needed to set the raw device readahead to 1024 and the raid device readahead to 4096. This brought my raid read speed from 59MB/s to 78MB/s. Better! (But note that now this makes the lvm read speed look bad.)
My raw device read speed is independent of the readahead setting, as long as it is at least 256. The speed is about 58MB/s.
Summary:
raw device: 58MB/s raid device: 78MB/s lvm device: 59MB/s
raid still isn't achieving the 106MB/s that I can get with parallel direct reads, but at least it's getting closer.
As a simple test, I wrote a program like dd that reads and discards 64k chunks of data from a device, but which skips 1 out of every four chunks (simulating skipping parity blocks). It's not surprising that this program can only read from a raw device at about 75% the rate of dd, since the kernel readahead is probably causing the skipped blocks to be read anyways (or maybe because the disk head has to pass over those sections of the disk anyways).
I then ran four copies of this program in parallel, reading from the raw devices that make up my raid partition. And, like md, they only achieved about 78MB/s. This is very close to 75% of 106MB/s. Again, not surprising, since I need to have raw device readahead turned on for this to be efficient at all, so 25% of the chunks that pass through the controller are ignored.
But I still don't understand why the md layer can't do better. If I turn off readahead of the raw devices, and keep it for the raid device, then parity blocks should never be requested, so they shouldn't use any bus/controller bandwidth. And even if each drive is only acting at 75% efficiency, the four drives should still be able to saturate the bus/controller. So I can't figure out what's going on here.
Is there a way for me to simulate readahead in userspace, i.e. can I do lots of sequential asynchronous reads in parallel?
Also, is there a way to disable caching of reads? Having to clear the cache by reading 900M each time slows down testing. I guess I could reboot with mem=100M, but it'd be nice to disable/enable caching on the fly. Hmm, maybe I can just run something like memtest which locks a bunch of ram...
Thanks for all of the help so far!
Dan
Quoted 9 lines
Quoted 5 lines
component partitions, e.g.
/dev/sda7: 39MB/sraid device /dev/md2: 31MB/s lvm device /dev/main/media: 53MB/s(oldish system - but note that lvm device is *much* faster)
Did you test component device and raid device speed using the read-ahead settings tuned for lvm reads? If so, that's not a fair comparison. :-)
I did an eval with a vendor who claimed that their lvm actually improved bandwidth because it somehow triggered better full-stripe operations, or readahead, or something. filtered through a marketing person, of course ;(
Quoted 2 lines
Is there a way for me to simulate readahead in userspace, i.e. can I do lots of sequential asynchronous reads in parallel?
there is async IO, but I don't think this is going to help you much.
Quoted 1 line
Also, is there a way to disable caching of reads? Having to clear
yes: O_DIRECT.
I'm attaching a little program I wrote which basically just shows you incremental bandwidth. you can use it to show the zones on a disk (just iorate -r -l 9999 /dev/hda and plot the results), or to do normal r/w bandwidth without being confused by the page-cache. you can even use it as a filter to measure tape backup performance.
it doesn't try to do anything with random seeks. it doesn't do anything multi-stream.
regards, mark hahn.
/* iorate.c - measure rates of sequential IO, showing incremental
bandwidth written by Mark Hahn ([email protected]) 2003,2004,2005 the main point of this code is to illustrate the danger of running naive bandwidth tests on files that are small relative to the memory/disk bandwidth ratio of your system. that is, on any system, the incremental bandwidth will start out huge, since IO is purely to the page cache. once you exceed that size, bandwidth will be dominated by the real disk performance. but using the average of these two modes is a mistake, even if you use very large files.
*/
#define _LARGEFILE64_SOURCE 1
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <errno.h>
#include <sys/time.h>
#include <sys/fcntl.h>
#include <sys/stat.h>
#include <stdarg.h>
#include <string.h>
#include <sys/mman.h>
#ifdef O_LARGEFILE
#define LF O_LARGEFILE
#elif defined(_O_LARGEFILE)
#define LF _O_LARGEFILE
#else
#define LF 0
#endif
#ifndef O_DIRECT
#define O_DIRECT 040000
#endif
typedef unsigned long long u64;
u64 bytes = 0, bytesLast = 0;
double timeStart = 0, timeLast = 0;
/* default reporting interval is every 2 seconds;
in 2004, an entry-level desktop disk will sustain around 50 MB/s, so the default bytes interval is 100 MB. whichever comes first. */
u64 byteInterval = 100;
double timeInterval = 2;
double gtod() {
struct timeval tv;
gettimeofday(&tv,0);
return tv.tv_sec + 1e-6 * tv.tv_usec;
}void dumpstats(int force) {
u64 db = bytes - bytesLast;
double now = gtod();
double dt;
static int first = 1;
if (timeLast == 0) timeStart = timeLast = now;
dt = now - timeLast;
if (!force && db < byteInterval && dt < timeInterval) return;
if (first) {
printf("#%7s %7s %7s %7s\n",
"secs",
"MB",
"MB/sec",
"MB/sec");
first = 0;
}
printf("%7.3f %7.3f %7.3f %7.3f\n",
now - timeStart,
1e-6 * bytes,
1e-6 * db / dt,
1e-6 * bytes / (now-timeStart));
timeLast = now;
bytesLast = bytes;
}void usage() {
fprintf(stderr,"iorate [-r/w filename] [-d] [-c chunksz][-b byteivl][-t ivl][-l szlim] [-r in] [-w out]\n");
fprintf(stderr,"-r in or -w out select which file is read or written ('-' for stdin/out)\n");
fprintf(stderr,"-c chunksz - size of chunks written (KB);\n");
fprintf(stderr,"-t timeinterval - collect rate each timeinterval seconds;\n");
fprintf(stderr,"-b byteinterval - collect rate each byteinterval MB;\n");
fprintf(stderr,"-l limit - total output size limit (MB);\n");
fprintf(stderr,"-d use O_DIRECT\n");
fprintf(stderr,"defaults are: '-c 8 -b 20 -t 10 -l 10'\n");
exit(1);
}void fatal(char *format, ...) {
va_list ap;
va_start(ap,format);
vfprintf(stderr,format,ap);
fprintf(stderr,": errno=%d (%s)\n",errno,strerror(errno));
va_end(ap);
dumpstats(1);
exit(1);
}/* allocate a buffer using mmap to ensure it's page-aligned.
O_DIRECT *could* be more strict than that, but probably isn't */
void *myalloc(unsigned size) {
unsigned s = (size + 4095) & ~4095U;
void *p = mmap(0,
s,
PROT_READ|PROT_WRITE,
MAP_ANONYMOUS|MAP_PRIVATE,
-1, 0);
if (p == MAP_FAILED)
return 0;
return p;
}int main(int argc, char *argv[]) {
unsigned size = 8;
char *buffer;
u64 limit = 10;
char *fnameI = 0;
char *fnameO = 0;
int fdI = 0;
int fdO = 1;
int doRead = 0;
int doWrite = 0;
int doDirect = 0;
int letter;
while ((letter = getopt(argc,argv,"r:w:b:c:l:t:d")) != -1) {
switch(letter) {
case 'r':
fnameI = optarg;
doRead = 1;
break;
case 'w':
fnameO = optarg;
doWrite = 1;
break;
case 'b':
byteInterval = atoi(optarg);
break;
case 'c':
size = atoi(optarg);
break;
case 'l':
limit = atoi(optarg);
break;
case 't':
timeInterval = atof(optarg);
break;
case 'd':
doDirect = 1;
break;
default:
usage();
}
}
if (argc != optind) usage();
byteInterval *= 1e6;
limit *= 1e6;
size *= 1024;
setbuf(stdout, 0);
fprintf(stderr,"chunk %dK, byteInterval %uM, timeInterval %f, limit %uM\n",
size>>10,
(unsigned)(byteInterval>>20),
timeInterval,
(unsigned)(limit>>20));
if (doRead && fnameI && strcmp(fnameI,"-")) {
fdI = open(fnameI, O_RDONLY | LF);
if (fdI == -1) fatal("open(read) failed");
}
if (doWrite && fnameO && strcmp(fnameO,"-")) {
int flags = O_RDWR | O_CREAT | LF;
if (doDirect) flags |= O_DIRECT;
fdO = open(fnameO, flags, 0600);
if (fdO == -1) fatal("open(write) failed");
}
buffer = myalloc(size);
memset(buffer,'m',size);
timeStart = timeLast = gtod();
bytes = 0;
while (bytes < limit) {
int c = size;
dumpstats(0);
if (doRead) {
c = read(fdI,buffer,c);
if (c == -1) fatal("read failed");
}
if (doWrite) {
c = write(fdO,buffer,c);
if (c == -1) fatal("write failed");
}
bytes += c;
/* short read/write means EOF. */
if (c < size) break;
}
dumpstats(1);
return 0;
}Quoted 8 lines · Mark Hahn
Quoted 2 lines
Is there a way for me to simulate readahead in userspace, i.e. can I do lots of sequential asynchronous reads in parallel?
there is async IO, but I don't think this is going to help you much.
Quoted 1 line
Also, is there a way to disable caching of reads? Having to clear
yes:
O_DIRECT.
That might disable caching of reads, but it also disables readahead, so unless I manually use aio to simulate readahead, this isn't going to solve my problem, which is having to clear the cache before each test to get relevant results.
I'm really surprised there isn't something in /proc you can use to clear or disable the cache. Would be very useful for benchmarking!
Dan
i also want a way to clear part of the whole page cache by file id. :) i also want a way to tell the cache distribution, how many for file A and B, ....
ming
Quoted 3 lines · Dan Christensen
I'm really surprised there isn't something in /proc you can use to clear or disable the cache. Would be very useful for benchmarking!
Quoted 1 line
i also want a way to clear part of the whole page cache by file id. :)
understandably, kernel developers are don't high-prioritize this sort of not-useful-for-normal-work feature.
Quoted 2 lines
i also want a way to tell the cache distribution, how many for file A and B, ....
you should probably try mmaping the file and using mincore. come to think of it, mmap+madvise might be a sensible way to flush pages corresponding to a particular file, as well.
Quoted 2 lines
Quoted 2 lines
I'm really surprised there isn't something in /proc you can use to clear or disable the cache. Would be very useful for benchmarking!
I assume you noticed "blockdev --flushbufs", no? it works for me (ie, a small, repeated streaming read of a disk device will show pagecache speed).
I think the problem is that it's difficult to dissociate readahead, writebehind and normal lru-ish caching. there was quite a flurry of activity around 2.4.10 related to this, and it left a bad taste in everyone's mouth. I think the main conclusion was that too much fanciness results in a fragile, more subtle and difficult-to-maintain system that performs better, true, but over a narrower range of workloads.
regards, mark hahn
sharcnet/mcmaster.
Quoted 4 lines · Mark Hahn
Quoted 1 line
i also want a way to clear part of the whole page cache by file id. :)
understandably, kernel developers are don't high-prioritize this sort of not-useful-for-normal-work feature.
agree.
Quoted 7 lines
Quoted 3 lines
i also want a way to tell the cache distribution, how many for file A and B, ....
you should probably try mmaping the file and using mincore. come to think of it, mmap+madvise might be a sensible way to flush pages corresponding to a particular file, as well.
i prefer a generic way. :) it will be useful to tune the system. maybe write a program to iterate the kernel structure can do this.
Quoted 7 lines
Quoted 3 lines
Quoted 3 lines
I'm really surprised there isn't something in /proc you can use to clear or disable the cache. Would be very useful for benchmarking!
I assume you noticed "blockdev --flushbufs", no? it works for me (ie, a small, repeated streaming read of a disk device will show pagecache speed).
it will do flush right? but will it flush and clean cache?
Quoted 7 lines
I think the problem is that it's difficult to dissociate readahead, writebehind and normal lru-ish caching. there was quite a flurry of activity around 2.4.10 related to this, and it left a bad taste in everyone's mouth. I think the main conclusion was that too much fanciness results in a fragile, more subtle and difficult-to-maintain system that performs better, true, but over a narrower range of workloads.
maybe this will happen again for 2.6.x? i think there are still many gray areas that can be checked. also many places can be improved. a test i did show that even you have sda and sdb to form a raid0, the page cache for sda and sdb will not be used by raid0. kind of funny.
Quoted 4 lines
regards, mark hahn
sharcnet/mcmaster.
thx!
Ming
Quoted 7 lines · Ming Zhang
Quoted 5 lines · Mark Hahn
Quoted 2 lines
i also want a way to clear part of the whole page cache by file id. :)
understandably, kernel developers are don't high-prioritize this sort of not-useful-for-normal-work feature.
agree.
Clearing just part of the page cache sounds too complicated to be worth it, but clearing it all seems reasonable; some kernel developers spend time doing benchmarks too!
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines · Dan Christensen
I'm really surprised there isn't something in /proc you can use to clear or disable the cache. Would be very useful for benchmarking!
I assume you noticed "blockdev --flushbufs", no? it works for me
I had tried this and noticed that it didn't work for files on a filesystem. But it does seem to work for block devices. That's great, thanks. I didn't realize the cache was so complicated; it can be retained for files but not for the block device underlying those files!
Quoted 3 lines
a test i did show that even you have sda and sdb to form a raid0, the page cache for sda and sdb will not be used by raid0. kind of funny.
I thought I had noticed raid devices making use of cache from underlying devices, but a test I just did agrees with your result, for both RAID-1 and RAID-5. Again, this seems odd. Shouldn't the raid layer take advantage of a block that's already in RAM? I guess this won't matter in practice, since you usually don't read from both a raid device and an underlying device.
Dan
Quoted 6 lines · Dan Christensen
Quoted 4 lines · Ming Zhang
Quoted 3 lines · Mark Hahn
Quoted 2 lines
i also want a way to clear part of the whole page cache by file id. :)
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
understandably, kernel developers are don't high-prioritize this sort of not-useful-for-normal-work feature.
agree.
Clearing just part of the page cache sounds too complicated to be worth it, but clearing it all seems reasonable; some kernel developers spend time doing benchmarks too!
maybe they do not care to run a program to clear it every time. :P
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines · Dan Christensen
I'm really surprised there isn't something in /proc you can use to clear or disable the cache. Would be very useful for benchmarking!
Quoted 7 lines
Quoted 1 line
Quoted 1 line
I assume you noticed "blockdev --flushbufs", no? it works for me
I had tried this and noticed that it didn't work for files on a filesystem. But it does seem to work for block devices. That's great, thanks. I didn't realize the cache was so complicated; it can be retained for files but not for the block device underlying those files!
yes, that is the why the command name is blockdev. :) i guess for files we just need to call fsync system call? is that call work on block device as well?
Quoted 11 lines
Quoted 4 lines
a test i did show that even you have sda and sdb to form a raid0, the page cache for sda and sdb will not be used by raid0. kind of funny.
I thought I had noticed raid devices making use of cache from underlying devices, but a test I just did agrees with your result, for both RAID-1 and RAID-5. Again, this seems odd. Shouldn't the raid layer take advantage of a block that's already in RAM? I guess this won't matter in practice, since you usually don't read from both a raid device and an underlying device.
you are right, that is weired in real world.
ming
Quoted 58 lines · Dan Christensen
Quoted 7 lines · David Greaves
In my setup I get
component partitions, e.g.
/dev/sda7: 39MB/sraid device /dev/md2: 31MB/s lvm device /dev/main/media: 53MB/s(oldish system - but note that lvm device is *much* faster)
Did you test component device and raid device speed using the read-ahead settings tuned for lvm reads? If so, that's not a fair comparison. :-)
Quoted 2 lines
For your entertainment you may like to try this to 'tune' your readahead
- it's OK to use so long as you're not recording:Thanks, I played around with that a lot. I tuned readahead to optimize lvm device reads, and this improved things greatly. It turns out the default lvm settings had readahead set to 0! But by tuning things, I could get my read speed up to 59MB/s. This is with raw device readahead 256, md device readahead 1024 and lvm readahead 2048. (The speed was most sensitive to the last one, but did seem to depend on the other ones a bit too.)
I separately tuned the raid device read speed. To maximize this, I needed to set the raw device readahead to 1024 and the raid device readahead to 4096. This brought my raid read speed from 59MB/s to 78MB/s. Better! (But note that now this makes the lvm read speed look bad.)
My raw device read speed is independent of the readahead setting, as long as it is at least 256. The speed is about 58MB/s.
Summary:
raw device: 58MB/s raid device: 78MB/s lvm device: 59MB/sraid still isn't achieving the 106MB/s that I can get with parallel direct reads, but at least it's getting closer.
As a simple test, I wrote a program like dd that reads and discards 64k chunks of data from a device, but which skips 1 out of every four chunks (simulating skipping parity blocks). It's not surprising that this program can only read from a raw device at about 75% the rate of dd, since the kernel readahead is probably causing the skipped blocks to be read anyways (or maybe because the disk head has to pass over those sections of the disk anyways).
I then ran four copies of this program in parallel, reading from the raw devices that make up my raid partition. And, like md, they only achieved about 78MB/s. This is very close to 75% of 106MB/s. Again, not surprising, since I need to have raw device readahead turned on for this to be efficient at all, so 25% of the chunks that pass through the controller are ignored.
Quoted 7 lines
But I still don't understand why the md layer can't do better. If I turn off readahead of the raw devices, and keep it for the raid device, then parity blocks should never be requested, so they shouldn't use any bus/controller bandwidth. And even if each drive is only acting at 75% efficiency, the four drives should still be able to saturate the bus/controller. So I can't figure out what's going on here.
when read, i do not think MD will read parity at all. but since parity is on all disk, there might be a seek here. so you might want to try a raid4 to see what happen as well.
Quoted 9 lines
Is there a way for me to simulate readahead in userspace, i.e. can I do lots of sequential asynchronous reads in parallel?
Also, is there a way to disable caching of reads? Having to clear the cache by reading 900M each time slows down testing. I guess I could reboot with mem=100M, but it'd be nice to disable/enable caching on the fly. Hmm, maybe I can just run something like memtest which locks a bunch of ram...
after you run your code, check the meminfo, the cached value might be much lower than u expected. my feeling is that linux page cache will discard all cache if last file handle closed.
Quoted 4 lines
Thanks for all of the help so far!
Dan
my problem here. this only apply to sdX not mdX. pls ignore this.
ming
Quoted 9 lines · Ming Zhang
Quoted 5 lines
Also, is there a way to disable caching of reads? Having to clear the cache by reading 900M each time slows down testing. I guess I could reboot with mem=100M, but it'd be nice to disable/enable caching on the fly. Hmm, maybe I can just run something like memtest which locks a bunch of ram...
after you run your code, check the meminfo, the cached value might be much lower than u expected. my feeling is that linux page cache will discard all cache if last file handle closed.
Quoted 10 lines · Ming Zhang
Quoted 7 lines · Dan Christensen
But I still don't understand why the md layer can't do better. If I turn off readahead of the raw devices, and keep it for the raid device, then parity blocks should never be requested, so they shouldn't use any bus/controller bandwidth. And even if each drive is only acting at 75% efficiency, the four drives should still be able to saturate the bus/controller. So I can't figure out what's going on here.
Quoted 2 lines
when read, i do not think MD will read parity at all. but since parity is on all disk, there might be a seek here.
Yes, there will be a seek, or internal drive readahead, so each drive will operate at around 75% efficiency. But since that shouldn't affect bus/controller traffic, I still would expect to get over 100MB/s with my hardware.
Quoted 5 lines
Quoted 1 line
Also, is there a way to disable caching of reads?
after you run your code, check the meminfo, the cached value might be much lower than u expected. my feeling is that linux page cache will discard all cache if last file handle closed.
Quoted 1 line · Ming Zhang
my problem here. this only apply to sdX not mdX. pls ignore this.
I'm not sure what you mean. For reads from sdX, mdX, files on sdX or files on mdX, the cache is retained. So it's necessary to clear this cache to get valid timing results.
Dan
Quoted 12 lines · Dan Christensen
Quoted 10 lines · Ming Zhang
Quoted 7 lines · Dan Christensen
But I still don't understand why the md layer can't do better. If I turn off readahead of the raw devices, and keep it for the raid device, then parity blocks should never be requested, so they shouldn't use any bus/controller bandwidth. And even if each drive is only acting at 75% efficiency, the four drives should still be able to saturate the bus/controller. So I can't figure out what's going on here.
Quoted 7 lines
Quoted 2 lines
when read, i do not think MD will read parity at all. but since parity is on all disk, there might be a seek here.
Yes, there will be a seek, or internal drive readahead, so each drive will operate at around 75% efficiency. But since that shouldn't affect bus/controller traffic, I still would expect to get over 100MB/s with my hardware.
agree. but what if your controller is a bottleneck? u need to have another card to find out.
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
Also, is there a way to disable caching of reads?
after you run your code, check the meminfo, the cached value might be much lower than u expected. my feeling is that linux page cache will discard all cache if last file handle closed.
Quoted 7 lines
Quoted 1 line · Ming Zhang
my problem here. this only apply to sdX not mdX. pls ignore this.
I'm not sure what you mean. For reads from sdX, mdX, files on sdX or files on mdX, the cache is retained. So it's necessary to clear this cache to get valid timing results.
yes, i was insane at that time, pls ignore these blah blah.
Quoted 7 lines
Dan
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
[Ming, could you trim quoted material down a bit more, and leave a blank line between quoted material and your new text? Thanks.]
Quoted 9 lines · Ming Zhang
Quoted 5 lines · Dan Christensen
Yes, there will be a seek, or internal drive readahead, so each drive will operate at around 75% efficiency. But since that shouldn't affect bus/controller traffic, I still would expect to get over 100MB/s with my hardware.
agree. but what if your controller is a bottleneck? u need to have another card to find out.
The controller and/or bus *is* the bottleneck, but I've already shown that I can get 106MB/s through them.
Dan
Quoted 2 lines · Dan Christensen
[Ming, could you trim quoted material down a bit more, and leave a blank line between quoted material and your new text? Thanks.]
thanks. sorry about that.
Quoted 5 lines
Quoted 2 lines · Ming Zhang
Quoted 1 line · Dan Christensen
Quoted 7 lines
Quoted 2 lines
agree. but what if your controller is a bottleneck? u need to have another card to find out.
The controller and/or bus *is* the bottleneck, but I've already shown that I can get 106MB/s through them.
Dan
then can u test RAID0 a bit? That is easier to analyze.
Ming
Quoted 1 line · Ming Zhang
then can u test RAID0 a bit? That is easier to analyze.
I can't easily test RAID-0 with my set-up, but I can test RAID-1 with two partitions. I found that the read speed from the md device was about the same as the read speed from each partition. This was with readahead set to 4096 on the md device, so I had hoped that it would do better. Based on the output of iostat, it looks like the reads were shared roughly equally between the two partitions (53%/47%). Does the RAID-1 code try to take the first stripe from disk 1, the second from disk 2, alternately? Or is it clever enough to try to take the first dozen from disk 1, the next dozen from disk 2, etc, in order to get larger, contiguous reads?
It's less clear to me that RAID-1 with two drives will be able to overcome the overhead of skipping various blocks. But it seems like RAID-5 with four drives should be able to saturate my bus/controller. For example, RAID-5 could just do sequential reads from 3 of the 4 drives, and use the parity chunks it reads to reconstruct the data chunks from the fourth drive. If I do parallel reads from 3 of my 4 disks, I can still get 106MB/s.
Dan
PS: Here's my simple test script, cleaned up a bit:
#!/bin/sh
# Devices to test for speed, and megabytes to read. MDDEV=/dev/md2
MDMB=300
RAWDEVS="/dev/sda7 /dev/sdb5 /dev/sdc5 /dev/sdd5" RAWMB=300
# Device to read to clear cache, and amount in megabytes. CACHEDEV=/dev/sda8
CACHEMB=900
clearcache () {
echo "Clearing cache..."
dd if=$CACHEDEV of=/dev/null bs=1M count=$CACHEMB > /dev/null 2>&1
}
testdev () {
echo "Read test from $1..."
dd if=$1 of=/dev/null bs=1M count=$2 2>&1 | grep bytes/sec echo
}
clearcache
for f in $RAWDEVS ; do
testdev $f $RAWMB
done
clearcache
testdev $MDDEV $MDMB
raid5 can not be that smart. :P
ming
Quoted 6 lines · Dan Christensen
bus/controller. For example, RAID-5 could just do sequential reads from 3 of the 4 drives, and use the parity chunks it reads to reconstruct the data chunks from the fourth drive. If I do parallel reads from 3 of my 4 disks, I can still get 106MB/s.
Summary so far:
RAID-5, four SATA hard drives, 2.6.12.2 kernel. Testing streaming read speed. With readahead optimized, I get:
each raw device: 58MB/s raid device: 78MB/s
3 or 4 parallel reads
from the raw devices: 106MB/s
I'm trying to figure out why the last two numbers differ.
I was afraid that for some reason the kernel was requesting the parity blocks instead of just the data blocks, but by using iostat it's pretty clear that the right number of blocks are being requested from the raw devices. If I write a dumb program that reads 3 out of every 4 64k chunks of a raw device, the kernel readahead kicks in and chunks I skip over do contribute to the iostat numbers. But the raid layer is correctly avoiding this readahead.
One other theory at this point is that my controller is trying to be clever and doing some readahead itself. Even if this is the case, I'd be surprised if this would cause a problem, since the data won't have to go over the bus. But maybe the controller is doing this and is causing itself to become overloaded? My controller is a Silicon Image 3114. Details at the end, for the record.
Second theory: for contiguous streams from the raw devices, the reads are done in really big chunks. But for md layer reads, the biggest possible chunk is 3 x 64k, if you want to skip parity blocks. Could 3 x 64k be small enough to cause overhead? Seems unlikely.
Those are my only guesses. Any others?
It seems strange that I can beat the md layer in userspace by 33%, by just reading from three of the devices and using parity to reconstruct the fourth!
Thanks again for all the help. I've learned a lot! And I haven't even started working on write speed...
Dan
0000:01:0b.0 RAID bus controller: Silicon Image, Inc. (formerly CMD Technology Inc) SiI 3114 [SATALink/SATARaid] Serial ATA Controller (rev 02)
Subsystem: Silicon Image, Inc. (formerly CMD Technology Inc): Unknown device 6114
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66MHz+ UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR-
Latency: 32, Cache Line Size: 0x08 (32 bytes)
Interrupt: pin A routed to IRQ 177
Region 0: I/O ports at 9400 [size=8]
Region 1: I/O ports at 9800 [size=4]
Region 2: I/O ports at 9c00 [size=8]
Region 3: I/O ports at a000 [size=4]
Region 4: I/O ports at a400 [size=16]
Region 5: Memory at e1001000 (32-bit, non-prefetchable) [size=1K]
Capabilities: [60] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=2 PME-Hello
Quoted 5 lines · Dan Christensen
Summary so far:
RAID-5, four SATA hard drives, 2.6.12.2 kernel. Testing streaming read speed. With readahead optimized, I get:
Quoted 7 lines
each raw device: 58MB/s raid device: 78MB/s3 or 4 parallel reads
from the raw devices: 106MB/sI'm trying to figure out why the last two numbers differ.
Have you checked what the performance with a 2.4.x kernel is? If I remember correctly there was some discussion on this list that 2.4 raid5 has better read performance.
Holger
in my previous test, using SATA, i got better result in 2.6 instead of 2.4. :P
Ming
Quoted 9 lines · Holger Kiehl
Quoted 2 lines
I'm trying to figure out why the last two numbers differ.
Have you checked what the performance with a 2.4.x kernel is? If I remember correctly there was some discussion on this list that 2.4 raid5
has better read performance.Holger
Quoted 5 lines · On Wednesday July 13, [email protected] wrote:
Question for the list: if I'm doing a long sequential write, naively each parity block will get recalculated and rewritten several times, once for each non-parity block in the stripe. Does the write-caching that the kernel does mean that each parity block will only get written once?
Raid5 does the best it can. It delays write requests as long as possible, and then when it must do the write, it writes every other block in the stripe that it has been asked to write, so only one parity update is needed for all those blocks.
My tests suggest that for long sequential writes (Without syncs) this achieves full-stripe writes most of the time.
NeilBrown