write-behind has no measurable effect?
Hi,
I experimented a bit with write-mostly and write-behind and found that write-mostly provides a very significant benefit (see below) but write-behind seems to have no effect whatsoever.
This is not what I expected and I wonder if I missed something.
I built a RAID1 array from a 64GB Corsair SSD and two 7200rpm SATA hard disks. I created xfs on the array, then benchmarked it using bonnie++, iozone and by compiling linux 2.6.37 (with allyesconfig).
Some interesting benchmark results follow. I used a 2.6.38-rc2 kernel for these measurements.
First, the stats that were identical (within a reasonable margin of error) across all measurements:
bonnie++ blockwise sequential write: ~110MB/s
bonnie++ blockwise sequential rewrite: ~60MB/s
bonnie++ blockwise sequential read: ~160-175MB/s
iozone read, 16k block size: ~135MB/s
kernel compilation time, user: ~5450s (*)
kernel compilation time, system: 570s (*)
(*) I didn't measure kernel compilation times without write-mostly; I expect they would've been worse.
Now for some of the measurements that resulted in (to me) surprising differences:
Using just the SSD (so no RAID), xfs mounted with
"noatime,noikeep,attr2,logbufs=8,logbsize=256k":
bonnie++ seeks/s: 7791
iozone random read, 16k block size: ~46MB/s
iozone random write, 16k block size: ~44MB/s
iozone random read, 512k block size: ~130MB/s
iozone random write, 512k block size: ~140MB/s
wall clock kernel compile time: 887s
RAID1 from two disks and one SSD, the disks set to write-behind:
mdadm --create /dev/md/ssdraid --force --assume-clean --level=1 \
--raid-devices=3 --bitmap=internal --bitmap-chunk=262144 \/dev/sdo2 --write-behind=16383 -W /dev/sd[nm]2
xfs mount options:
noatime,logbsize=256k,logbufs=8,noikeep,attr2,nodiratime,delaylog
bonnie++ seeks/s: 2087
iozone random read, 16k block size: ~43MB/s
iozone random write, 16k block size: ~3.7MB/s
iozone random read, 512k block size: ~126MB/s
iozone random write, 512k block size: ~69MB/s
wall clock kernel compile time: 936s
(Note the drastically reduced random write performance.)
Now the same setup, but with write-behind=0:
bonnie++ seeks/s: 1843
iozone random read, 16k block size: ~48MB/s
iozone random write, 16k block size: ~3.7MB/s
iozone random read, 512k block size: ~126MB/s
iozone random write, 512k block size: ~69MB/s
wall clock kernel compile time: 935s
So, the difference between write-behind=0 and write-behind=16383 (which seems to be the maximum) is negligible (if not imaginary).
For reference, some results with even write-mostly disabled:
bonnie++ seeks/s: 487.4
iozone random read, 16k block size: ~3.7MB/s
iozone random write, 16k block size: ~3.7MB/s
iozone random read, 512k block size: ~58MB/s
iozone random write, 512k block size: ~69MB/s
(The full result set is available from
<http://elan.rulez.org/~korn/tmp/iobench.ods>, 27k.)
It's easy to see from the results that write-mostly does as advertised:
reads are mostly served by the SSD, so that random reads are approximately as fast as when I only used the SSD.
I'd have expected write-behind to increase the apparent random write performance though, and this didn't happen (there was no measurable difference).
I thought maybe the iozone benchmark was too synthetic (too many writes in too short a time, so that the buffer effect of write-behind is lost); that's why I tried the kernel compilation, but I the raid array was as slow with write-behind as without it.
Any idea why write-behind doesn't seem to have an effect?
Thanks
Andras
Signature · 2 lines
Andras Korn <korn at elan.rulez.org>
Keep your ears open - but your legs crossed.Quoted 5 lines · On Mon, 14 Feb 2011 22:38:17 +0100 Andras Korn <[email protected]>…
Hi,
I experimented a bit with write-mostly and write-behind and found that write-mostly provides a very significant benefit (see below) but write-behind seems to have no effect whatsoever.
The use-case where write-behind can be expected to have an effect is when the throughput is low enough to be well within the capacity of all devices, but the latency of the write-behind device is higher than desired. write-behind will allow that high latency to be hidden (as long as the throughput limit is not exceeded).
I suspect your tests did not test for low latency in a low-throughput scenario.
NeilBrown
Quoted 4 lines · NeilBrown
Quoted 3 lines
I experimented a bit with write-mostly and write-behind and found that write-mostly provides a very significant benefit (see below) but write-behind seems to have no effect whatsoever.
Quoted 8 lines
The use-case where write-behind can be expected to have an effect is when the throughput is low enough to be well within the capacity of all devices, but the latency of the write-behind device is higher than desired. write-behind will allow that high latency to be hidden (as long as the throughput limit is not exceeded).
I suspect your tests did not test for low latency in a low-throughput scenario.
I thought they did. "High latency" was, in my case, caused by the high seek times (compared to the SSD) of the spinning disks. Throughput-wise, they certainly could have kept up (their sequential read/write performance even exceeds that of the SSD).
But maybe I misunderstand how write-behind works. I thought/hoped it would commit writes to the fast drive(s) and mark affected areas dirty in the intent map, then lazily sync the dirty areas over to the slow disk(s).
What does it actually do? md(4) isn't very forthcoming, and the wiki has no relevant hits either.
Thanks.
Signature · 2 lines
Andras Korn <korn at elan.rulez.org>
Baroque: (def.) When you are out of Monet.Quoted 12 lines · On Mon, 14 Feb 2011 23:57:54 +0100 Andras Korn <[email protected]>…
Quoted 10 lines · NeilBrown
Quoted 3 lines
I experimented a bit with write-mostly and write-behind and found that write-mostly provides a very significant benefit (see below) but write-behind seems to have no effect whatsoever.
The use-case where write-behind can be expected to have an effect is when the throughput is low enough to be well within the capacity of all devices, but the latency of the write-behind device is higher than desired. write-behind will allow that high latency to be hidden (as long as the throughput limit is not exceeded).
Quoted 7 lines
Quoted 2 lines
I suspect your tests did not test for low latency in a low-throughput scenario.
I thought they did. "High latency" was, in my case, caused by the high seek times (compared to the SSD) of the spinning disks. Throughput-wise, they certainly could have kept up (their sequential read/write performance even exceeds that of the SSD).
A "MB/s" number is not going to show a difference with write-behind as it is fundamentally about throughput. We cannot turn random writes into sequential writes just be doing 'write-behind' as the same locations on disk still have to be written to.
You need a number like transactions-per-second to see a different. If you write with O_SYNC, the write-behind will probably show a difference.
Quoted 7 lines
But maybe I misunderstand how write-behind works. I thought/hoped it would commit writes to the fast drive(s) and mark affected areas dirty in the intent map, then lazily sync the dirty areas over to the slow disk(s).
What does it actually do? md(4) isn't very forthcoming, and the wiki has no relevant hits either.
write-behind makes a copy of the data, submits writes to all devices in parallel, and reports success to the upper layer as soon as all the non-write-behind writes have finished.
The approach you suggest could be synthesised by:
- add a write-intent bitmap with fairly small chunks. This should be an external bitmap and should be directly on the fastest drive
- have some daemon that fails the 'slow' device, waits 30 seconds, re-adds it, waits for recovery to complete, and loops back.
Actually I just realised another reason why you don' see any improvement. You are using an internal bitmap. This requires a synch write to both devices. The use-case for which write-behind was developed involved an external bitmap.
Maybe I should disable bitmap updates to write-behind devices .....
NeilBrown
Quoted 8 lines · NeilBrown
Quoted 7 lines
Quoted 2 lines
I suspect your tests did not test for low latency in a low-throughput scenario.
I thought they did. "High latency" was, in my case, caused by the high seek times (compared to the SSD) of the spinning disks. Throughput-wise, they certainly could have kept up (their sequential read/write performance even exceeds that of the SSD).
Quoted 4 lines
A "MB/s" number is not going to show a difference with write-behind as it is fundamentally about throughput. We cannot turn random writes into sequential writes just be doing 'write-behind' as the same locations on disk still have to be written to.
Thanks, I understand now; I had hoped write-behind would in fact re-order the writes to the slow devices. In retrospect, I'm not sure what gave me that notion. (Reckless optimism, probably. :)
Quoted 6 lines
Quoted 2 lines
What does it actually do? md(4) isn't very forthcoming, and the wiki has no relevant hits either.
write-behind makes a copy of the data, submits writes to all devices in parallel, and reports success to the upper layer as soon as all the non-write-behind writes have finished.
So this really only makes a difference for synchronous writes (because otherwise success would be reported as soon as the write is buffered), right?
Quoted 6 lines
The approach you suggest could be synthesised by:
- add a write-intent bitmap with fairly small chunks. This should be an external bitmap and should be directly on the fastest drive
- have some daemon that fails the 'slow' device, waits 30 seconds, re-adds it, waits for recovery to complete, and loops back.
Ewww. :)
Quoted 3 lines
Actually I just realised another reason why you don' see any improvement. You are using an internal bitmap. This requires a synch write to both devices.
Yes, that was something I actually wanted to ask. Since it's write_behind_, it wouldn't need to be a synchronous write though - you could at least allow the write-mostly disk to reorder it, couldn't you?
Quoted 2 lines
The use-case for which write-behind was developed involved an external
bitmap.
My use case, fwiw, is that I have a single SSD and would like to exploit its close-to-zero seek time while also providing redundancy (using spinning disks) with eventual consistency. It's not for databases or anything irreplaceable, just things like logs, svn working copies, vserver system files... and an external jfs journal. (I know journal i/o is very nearly sequential, but I don't have a spinning disk to dedicate to it, and if I use the same disk for other purposes as well, seeking would definitely occur, decreasing performance.)
Quoted 1 line
Maybe I should disable bitmap updates to write-behind devices .....
Or make them asynchronous, or lazy (like, update the bitmap whenever you must seek into the vicinity anyway), or just infrequent. But yes, this sounds like a very good idea.
Another approach to take would be to mark as dirty, on the fast devices, all areas being written to, and in the background continuously synch them to the slow devices, sequentially (marking as clean synched-and-as-yet-unwritten-to areas); so that the array would be resyncing continually, but be very fast for random writes. This would of course also require the bitmap to only be synchronously updated on the fast devices.
Otoh, this is really a different mechanism from the current write-behind, aimed at a different use-case, so maybe it could be implemented orthogonally. (Patches welcome, I'm sure; it's times like these I hate not being a coder.)
Signature · 2 lines
Andras Korn <korn at elan.rulez.org>
Take my advice, I don't use it anyway.On 15/02/2011 01:00, Andras Korn wrote:
[...]
Quoted 11 lines
Another approach to take would be to mark as dirty, on the fast devices, all areas being written to, and in the background continuously synch them to the slow devices, sequentially (marking as clean synched-and-as-yet-unwritten-to areas); so that the array would be resyncing continually, but be very fast for random writes. This would of course also require the bitmap to only be synchronously updated on the fast devices.
Otoh, this is really a different mechanism from the current write-behind, aimed at a different use-case, so maybe it could be implemented orthogonally. (Patches welcome, I'm sure; it's times like these I hate not being a coder.)
I wonder whether bcache might do roughly what you want? I haven't tried it myself but it does sound interesting: "Hard drives are cheap and big, SSDs are fast but small and expensive. Wouldn't it be nice if you could transparently get the advantages of both? With Bcache, you can have your cake and eat it too." See http://bcache.evilpiepirate.org/
Cheers,
John.
Quoted 7 lines · John Robinson
Quoted 7 lines
Another approach to take would be to mark as dirty, on the fast devices, all areas being written to, and in the background continuously synch them to the slow devices, sequentially (marking as clean synched-and-as-yet-unwritten-to areas); so that the array would be resyncing continually, but be very fast for random writes. This would of course also require the bitmap to only be synchronously updated on the fast devices.
Quoted 6 lines
Quoted 4 lines
Otoh, this is really a different mechanism from the current write-behind, aimed at a different use-case, so maybe it could be implemented orthogonally. (Patches welcome, I'm sure; it's times like these I hate not being a coder.)
I wonder whether bcache might do roughly what you want? I haven't
It only does very roughly what I want (the idea there is to _cache_ a much larger spinning disk using a relatively small SSD, whereas I basically want them both to be the same size, with the disk eventually mirroring the contents of the SSD); also, development of bcache has stalled (it doesn't even compile with recent kernels and the developer has stated that he's taking a break).
I also know of flashcache, which is similar to bcache and is more actively developed, but is still lagging quite a few versions behind (the latest kernel it works with is 2.6.32, I think; it certainly doesn't compile with 2.6.38).
So, while both of these may actually be good at what they do, neither of them does what I have in mind and I also can't use either of them because I need a newer kernel than what they support.
But thanks anyway.
Signature · 2 lines
Andras Korn <korn at elan.rulez.org>
I'm not nearly as think as you confused I am.A message that is not in the archive (sent off-list or not yet received): [email protected]
andras could you make some benchmarks to raid1 with round robin read balance? at this site:
www.spadim.com.br/raid1/
it's kernel 2.6.37 based
2011/2/15 Roberto Spadim <[email protected]>
Quoted 53 lines
andras could you make some benchmarks to raid1 with round robin read balance? at this site:
www.spadim.com.br/raid1/it's kernel 2.6.37 based
2011/2/14 Andras Korn <[email protected]>
Quoted 35 lines · John Robinson
Quoted 8 lines
Quoted 8 lines
Another approach to take would be to mark as dirty, on the fast devices, all areas being written to, and in the background continuously synch them to the slow devices, sequentially (marking as clean synched-and-as-yet-unwritten-to areas); so that the array would be resyncing continually, but be very fast for random writes. This would of course also require the bitmap to only be synchronously updated on the fast devices.
Quoted 6 lines
Quoted 4 lines
Otoh, this is really a different mechanism from the current write-behind, aimed at a different use-case, so maybe it could be implemented orthogonally. (Patches welcome, I'm sure; it's times like these I hate not being a coder.)
I wonder whether bcache might do roughly what you want? I haven't
It only does very roughly what I want (the idea there is to _cache_ a much larger spinning disk using a relatively small SSD, whereas I basically want them both to be the same size, with the disk eventually mirroring the contents of the SSD); also, development of bcache has stalled (it doesn't even compile with recent kernels and the developer has stated that he's taking a break).
I also know of flashcache, which is similar to bcache and is more actively developed, but is still lagging quite a few versions behind (the latest kernel it works with is 2.6.32, I think; it certainly doesn't compile with 2.6.38).
So, while both of these may actually be good at what they do, neither of them does what I have in mind and I also can't use either of them because I need a newer kernel than what they support.
But thanks anyway.
Quoted 7 lines
--
Andras Korn <korn at elan.rulez.org>
I'm not nearly as think as you confused I am. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines
--
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
Roberto Spadim
Spadim Technology / SPAEmpresarial
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 5 lines · Roberto Spadim
andras could you make some benchmarks to raid1 with round robin read balance? at this site:
www.spadim.com.br/raid1/it's kernel 2.6.37 based
Yes, I can do that. Can you give me some hints on what specific configuration to try? I see you have some sysfs tunables. My raid1 array consists of two spinning disks and an SSD, all local.
Do you expect this patch to make a difference in my case? With the spinning disks marked as write-mostly, I'm getting close to the read performance of the SSD (except for very small random reads, for some reason).
It's random writes that are much slower than with only the SSD.
Signature · 2 lines
Andras Korn <korn at elan.rulez.org>
There is no spoon(). But there is a fork().writes will be the speed of slowest mirror (ssd or hd) read can have speed improvement with this patch
there´s some options
/sys/block/md0/md/read_balance_mode
near_head = today implementationround_robin => could be usefull if you have ssd only, since round_robin consider that access time is the same for any drive (hard disk access time is diferent for random and sequencial) stripe => i didn´t get good benchmarks, but it´s nice to have it, since we could put it over network, some sectors on one disk others on another (nbd)time_based => here you should send me some information about you disk for example:
access time of mirrors (check your drive information website, ssd
are normaly <0.1ms, hard disks near 10~20 ms)
sequencial read speed (use dd if=/dev/sda of=/dev/zero bs=4096,
change the block size to size you will use with your filesystem, for vertex2 ssd i´m using bs=4096, for disks it´s a good value too, since disks bs ~= number of heads (2,4,8) )
send me the access time and sequencial read speed and i make the values to tune your sysfs (/sys/block/md0/md/read_balance_config)
you will need to :
echo "time_based" > /sys/block/md0/md/read_balance_mode echo "disks informations" > /sys/block/md0/md/read_balance_config
for each mirror, you can´t use sysfs file to configure it, maybe a bash script is a better solution to configure it, on future version i will change it and put at /sys/block/md0/md/dev-xxxx/
2011/2/15 Andras Korn <[email protected]>:
Quoted 26 lines · Roberto Spadim
Quoted 6 lines
andras could you make some benchmarks to raid1 with round robin read balance? at this site:
www.spadim.com.br/raid1/it's kernel 2.6.37 based
Yes, I can do that. Can you give me some hints on what specific configuration to try? I see you have some sysfs tunables. My raid1 array consists of two spinning disks and an SSD, all local.
Do you expect this patch to make a difference in my case? With the spinning disks marked as write-mostly, I'm getting close to the read performance of the SSD (except for very small random reads, for some reason).
It's random writes that are much slower than with only the SSD.
--
Andras Korn <korn at elan.rulez.org> There is nospoon(). But there is afork(). --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 6 lines
Roberto Spadim Spadim Technology / SPAEmpresarial -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
at raid1.c there´s a example with ssd and hd with different speeds
2011/2/15 Roberto Spadim <[email protected]>:
Quoted 69 lines
writes will be the speed of slowest mirror (ssd or hd) read can have speed improvement with this patch
there´s some options
/sys/block/md0/md/read_balance_mode
near_head= today implementationround_robin=> could be usefull if you have ssd only, sinceround_robinconsider that access time is the same for any drive (hard disk access time is diferent for random and sequencial) stripe => i didn´t get good benchmarks, but it´s nice to have it, since we could put it over network, some sectors on one disk others on another (nbd)time_based=> here you should send me some information about you disk for example:
access time of mirrors (check your drive information website, ssd are normaly <0.1ms, hard disks near 10~20 ms)
sequencial read speed (use dd if=/dev/sda of=/dev/zero bs=4096, change the block size to size you will use with your filesystem, for vertex2 ssd i´m using bs=4096, for disks it´s a good value too, since disks bs ~= number of heads (2,4,8) )send me the access time and sequencial read speed and i make the values to tune your sysfs (
/sys/block/md0/md/read_balance_config)you will need to :
echo "
time_based" >/sys/block/md0/md/read_balance_modeecho "disks informations" >/sys/block/md0/md/read_balance_configfor each mirror, you can´t use sysfs file to configure it, maybe a bash script is a better solution to configure it, on future version i will change it and put at
/sys/block/md0/md/dev-xxxx/2011/2/15 Andras Korn <[email protected]>:
Quoted 18 lines · Roberto Spadim
Quoted 6 lines
andras could you make some benchmarks to raid1 with round robin read balance? at this site:
www.spadim.com.br/raid1/it's kernel 2.6.37 based
Yes, I can do that. Can you give me some hints on what specific configuration to try? I see you have some sysfs tunables. My raid1 array consists of two spinning disks and an SSD, all local.
Do you expect this patch to make a difference in my case? With the spinning disks marked as write-mostly, I'm getting close to the read performance of the SSD (except for very small random reads, for some reason).
It's random writes that are much slower than with only the SSD.
Quoted 8 lines
--
Andras Korn <korn at elan.rulez.org> There is nospoon(). But there is afork(). --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html--
Roberto Spadim
Spadim Technology / SPAEmpresarial
Signature · 6 lines
Roberto Spadim Spadim Technology / SPAEmpresarial -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Roberto Spadim
andras could you make some benchmarks to raid1 with round robin read balance? at this site:
www.spadim.com.br/raid1/
For the record: we did some benchmarks and while the patch shows promise and seems to cause no problems, it resulted in no measurable performance increase for a RAID1 array composed of an SSD and two 7200rpm HDDs.
Andras
Signature · 2 lines
Andras Korn <korn at elan.rulez.org>
If it ain't broken, play with it until it is.a question here...
what happen if all disks are write-mostly and just ssd is write-behind?
why?
write-behind is an async feature (md only return ok to filesystem if non write-behind disks are sync writen)
write-mostly is a read_balance optimization (ony read from that device if all non-write-mostly devices fail)
making all disks write-mostly could allow us to use write-behind on slowest(s) disk(s)
another idea...
could we change raid1 write code? how?
if a total of X write are done, return ok to filesystem, all other devices are marked as write-behind (automatic write-behind) after sync writes disks are marked as non-write-behind again
maybe a optimization is: what disk MUST be sync(only non write-behind), what disk MUST BE async (only write-behind), what disk can be async/sync (any write-behind type)
another question...
can read balance use write-mostly device in a very busy system without failed devices (all mirrors are in sync)?
2011/2/16 Andras Korn <[email protected]>:
Quoted 20 lines · Roberto Spadim
Quoted 4 lines
andras could you make some benchmarks to raid1 with round robin read balance? at this site:
www.spadim.com.br/raid1/For the record: we did some benchmarks and while the patch shows promise and seems to cause no problems, it resulted in no measurable performance increase for a RAID1 array composed of an SSD and two 7200rpm HDDs.
Andras
--
Andras Korn <korn at elan.rulez.org>
If it ain't broken, play with it until it is. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 6 lines
Roberto Spadim Spadim Technology / SPAEmpresarial -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Not to be too cute, but the man page for mdadm says that --write-behind is only attempted on drives marked --write-mostly. I did not see a --write-mostly in your array create statement.
Also, are you trying to create a three-way-mirror or mirror the one ssd to two HDDs as stripes. If you want the latter, you need to create a raid0 array and then the raid1 array.
For testing, two drives might produce fewer anomolies.
Doug Dumitru
EasyCo LLC
Quoted 102 lines · Andras Korn
Hi,
I experimented a bit with write-mostly and write-behind and found that write-mostly provides a very significant benefit (see below) but write-behind seems to have no effect whatsoever.
This is not what I expected and I wonder if I missed something.
I built a RAID1 array from a 64GB Corsair SSD and two 7200rpm SATA hard disks. I created xfs on the array, then benchmarked it using bonnie++, iozone and by compiling linux 2.6.37 (with allyesconfig).
Some interesting benchmark results follow. I used a 2.6.38-rc2 kernel for these measurements.
First, the stats that were identical (within a reasonable margin of error) across all measurements:
bonnie++ blockwise sequential write: ~110MB/s
bonnie++ blockwise sequential rewrite: ~60MB/s
bonnie++ blockwise sequential read: ~160-175MB/s
iozone read, 16k block size: ~135MB/s
kernel compilation time, user: ~5450s (*)
kernel compilation time, system: 570s (*)(*) I didn't measure kernel compilation times without write-mostly; I expect they would've been worse.
Now for some of the measurements that resulted in (to me) surprising differences:
Using just the SSD (so no RAID), xfs mounted with
"noatime,noikeep,attr2,logbufs=8,logbsize=256k":bonnie++ seeks/s: 7791
iozone random read, 16k block size: ~46MB/s
iozone random write, 16k block size: ~44MB/s
iozone random read, 512k block size: ~130MB/s
iozone random write, 512k block size: ~140MB/s
wall clock kernel compile time: 887sRAID1 from two disks and one SSD, the disks set to write-behind:
mdadm --create
/dev/md/ssdraid--force --assume-clean --level=1 \
--raid-devices=3 --bitmap=internal --bitmap-chunk=262144 \/dev/sdo2--write-behind=16383 -W /dev/sd[nm]2xfs mount options:
noatime,logbsize=256k,logbufs=8,noikeep,attr2,nodiratime,delaylogbonnie++ seeks/s: 2087
iozone random read, 16k block size: ~43MB/s
iozone random write, 16k block size: ~3.7MB/s
iozone random read, 512k block size: ~126MB/s
iozone random write, 512k block size: ~69MB/s
wall clock kernel compile time: 936s(Note the drastically reduced random write performance.)
Now the same setup, but with write-behind=0:
bonnie++ seeks/s: 1843
iozone random read, 16k block size: ~48MB/s
iozone random write, 16k block size: ~3.7MB/s
iozone random read, 512k block size: ~126MB/s
iozone random write, 512k block size: ~69MB/s
wall clock kernel compile time: 935sSo, the difference between write-behind=0 and write-behind=16383 (which seems to be the maximum) is negligible (if not imaginary).
For reference, some results with even write-mostly disabled:
bonnie++ seeks/s: 487.4
iozone random read, 16k block size: ~3.7MB/s
iozone random write, 16k block size: ~3.7MB/s
iozone random read, 512k block size: ~58MB/s
iozone random write, 512k block size: ~69MB/s(The full result set is available from
<http://elan.rulez.org/~korn/tmp/iobench.ods>, 27k.)It's easy to see from the results that write-mostly does as advertised:
reads are mostly served by the SSD, so that random reads are approximately as fast as when I only used the SSD.I'd have expected write-behind to increase the apparent random write performance though, and this didn't happen (there was no measurable difference).
I thought maybe the iozone benchmark was too synthetic (too many writes in too short a time, so that the buffer effect of write-behind is lost); that's why I tried the kernel compilation, but I the raid array was as slow with write-behind as without it.
Any idea why write-behind doesn't seem to have an effect?
Thanks
Andras
Quoted 7 lines
--
Andras Korn <korn at elan.rulez.org> Keep your ears open - but your legs crossed. --
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Doug Dumitru
EasyCo LLC
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Doug Dumitru
Not to be too cute, but the man page for mdadm says that --write-behind is only attempted on drives marked --write-mostly. I did not see a --write-mostly in your array create statement.
Yeah, I abbreviated it as -W:
Quoted 5 lines
Quoted 3 lines
mdadm --create
/dev/md/ssdraid--force --assume-clean --level=1 \
--raid-devices=3 --bitmap=internal --bitmap-chunk=262144 \/dev/sdo2--write-behind=16383 -W /dev/sd[nm]2Also, are you trying to create a three-way-mirror or mirror the one
A three-way mirror.
Signature · 6 lines
Andras Korn <korn at elan.rulez.org>
Caesar si viveret, ad remum dareris.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html