Write-intent bitmap decreases or increase performance of RAID5?
From what I read from the internet, write-intent bitmap decreases raid performance and provides more stability. But in my situation, it also increases raid performance dramatically!
My RAID5 write speed without write-intent bitmap is 90MB/s, and is increased to 200MB/s after enabling internal write-intent bitmap. I use ext4 filesystem on this raid array. The following is my RAID5 array information:
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.0
Creation Time : Mon Nov 16 02:46:50 2009
Raid Level : raid5
Array Size : 10255968576 (9780.85 GiB 10502.11 GB)
Used Dev Size : 1465138368 (1397.26 GiB 1500.30 GB)
Raid Devices : 8
Total Devices : 8
Persistence : Superblock is persistentIntent Bitmap : Internal
Update Time : Wed Jun 30 03:51:56 2010
State : active
Active Devices : 8
Working Devices : 8
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : AOpen:00
UUID : abb2242c:6a250d29:df65e52b:426f4ed9
Events : 405636
Number Major Minor RaidDevice State
4 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
2 8 49 2 active sync /dev/sdd1
3 8 65 3 active sync /dev/sde1
5 8 81 4 active sync /dev/sdf1
6 8 97 5 active sync /dev/sdg1
7 8 113 6 active sync /dev/sdh1
8 8 129 7 active sync /dev/sdi1
Signature · 3 lines
Shaochun Wang <[email protected]> Jabber: [email protected]
Quoted 7 lines · On Wed, 30 Jun 2010 04:11:54 +0800 Shaochun Wang <[email protected]> wrote:
From what I read from the internet, write-intent bitmap decreases raid performance and provides more stability. But in my situation, it also increases raid performance dramatically!
My RAID5 write speed without write-intent bitmap is 90MB/s, and is increased to 200MB/s after enabling internal write-intent bitmap. I use ext4 filesystem on this raid array.
How do you measure that speed exactly? The proper (and also quite simple) way to do it is to run the following test several times (assuming that current dir is some directory on the array):
dd if=/dev/zero of=zerofile bs=1M count=2048 conv=fdatasync,notrunc
Also, I see you run an 8-drive RAID5 on 1.5TB drives, so make sure you are aware of the issues described in:
http://hardware.slashdot.org/article.pl?sid=08/10/21/2126252
http://louwrentius.blogspot.com/2008/08/why-raid-1-and-5-will-kill-you-some-day.html
Signature · 2 lines
With respect, Roman
Quoted 1 line · Roman Mamedov
dd if=/dev/zero of=zerofile bs=1M count=2048 conv=fdatasync,notrunc
To be a bit more sure about accurate measurement, make sure the amount of data you're transferring is twice the amount of RAM in the machine, the above is only 2 gigs of data.
Signature · 1 line
Mikael Abrahamsson email: [email protected]
Quoted 7 lines · On Wed, 30 Jun 2010 07:54:55 +0200 (CEST) Mikael Abrahamsson…
Quoted 1 line · Roman Mamedov
dd if=/dev/zero of=zerofile bs=1M count=2048 conv=fdatasync,notrunc
To be a bit more sure about accurate measurement, make sure the amount of data you're transferring is twice the amount of RAM in the machine, the above is only 2 gigs of data.
This is not necessary. See the dd man page about the conv=fdatasync switch. You can even test with just 256 MB this way, and still get an extremely accurate result.
Three consecutive runs with "count=256" on a machine with 4GB of RAM:
268435456 bytes (268 MB) copied, 4.76846 s, 56.3 MB/s 268435456 bytes (268 MB) copied, 4.79682 s, 56.0 MB/s 268435456 bytes (268 MB) copied, 4.79307 s, 56.0 MB/s
Signature · 2 lines
With respect, Roman
Quoted 3 lines · Roman Mamedov
This is not necessary. See the dd man page about the conv=fdatasync switch. You can even test with just 256 MB this way, and still get an extremely accurate result.
You're right, for writing this is ok. For read tests it's important to use more than ram though, to avoid doing performance testing from block cache instead of from the actual drives.
Signature · 1 line
Mikael Abrahamsson email: [email protected]
Quoted 9 lines · On Wed, 30 Jun 2010 08:58:28 +0200 (CEST) Mikael Abrahamsson…
Quoted 3 lines · Roman Mamedov
This is not necessary. See the dd man page about the conv=fdatasync switch. You can even test with just 256 MB this way, and still get an extremely accurate result.
You're right, for writing this is ok. For read tests it's important to use more than ram though, to avoid doing performance testing from block cache instead of from the actual drives.
Yeah, though there's also an alternative way of mitigating the effect of the cache when testing read speed, see http://linux-mm.org/Drop_Caches
Signature · 2 lines
With respect, Roman
flag: direct use direct I/O for data
can be useful too
Quoted 20 lines · Roman Mamedov
Quoted 9 lines · On Wed, 30 Jun 2010 08:58:28 +0200 (CEST) Mikael Abrahamsson…
Quoted 3 lines · Roman Mamedov
This is not necessary. See the dd man page about the conv=fdatasync switch. You can even test with just 256 MB this way, and still get an extremely accurate result.
You're right, for writing this is ok. For read tests it's important to use more than ram though, to avoid doing performance testing from block cache instead of from the actual drives.
Yeah, though there's also an alternative way of mitigating the effect of the cache when testing read speed, see http://linux-mm.org/Drop_Caches
--
With respect,
Roman
Signature · 2 lines
Best regards, [COOLCOLD-RIPN]
Quoted 8 lines · Mikael Abrahamsson
Quoted 3 lines · Roman Mamedov
This is not necessary. See the dd man page about the conv=fdatasync switch. You can even test with just 256 MB this way, and still get an extremely accurate result.
You're right, for writing this is ok. For read tests it's important to use more than ram though, to avoid doing performance testing from block cache instead of from the actual drives.
See /proc/sys/vm/drop_caches.
Signature · 3 lines
Bill Davidsen <[email protected]> "We can't solve today's problems by using the same thinking we used in creating them." - Einstein
Quoted 7 lines · Mikael Abrahamsson
Quoted 1 line · Roman Mamedov
dd if=/dev/zero of=zerofile bs=1M count=2048 conv=fdatasync,notrunc
To be a bit more sure about accurate measurement, make sure the amount of data you're transferring is twice the amount of RAM in the machine, the above is only 2 gigs of data.
As suggested, I did the test again and got the following result:
-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc Password: 5000+0 records in 5000+0 records out 5242880000 bytes (5.2 GB) copied, 63.497 s, 82.6 MB/s -bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc 5000+0 records in 5000+0 records out 5242880000 bytes (5.2 GB) copied, 18.1033 s, 290 MB/s
I don't know why the second dd becomes 290MB/s and the first 82.6MB/s.
Signature · 8 lines
Shaochun Wang(王绍春) <[email protected]> State Key Laboratory of Computer Science, Institute of Software, Chinese Academy of Sciences -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
How much RAM does your machine have?
Write to the disk an amount that is larger than the amount of RAM and see if you get the same result.
Also you can use hdparm to flush the caches before running dd the 2nd time.
2010/7/1 Shaochun Wang <[email protected]>
Quoted 24 lines · Mikael Abrahamsson
Quoted 7 lines · Roman Mamedov
Quoted 2 lines
dd if=/dev/zero of=zerofile bs=1M count=2048 conv=fdatasync,notrunc
To be a bit more sure about accurate measurement, make sure the amount of data you're transferring is twice the amount of RAM in the machine, the above is only 2 gigs of data.
As suggested, I did the test again and got the following result:
-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc Password:
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 63.497 s, 82.6 MB/s-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc 5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 18.1033 s, 290 MB/sI don't know why the second dd becomes 290MB/s and the first 82.6MB/s.
Quoted 9 lines
--
Shaochun Wang(王绍春) <[email protected]>
State Key Laboratory of Computer Science,
Institute of Software,
Chinese Academy of Sciences
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Majed B.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · On Thu, 1 Jul 2010 23:30:26 +0300 "Majed B." <[email protected]> wrote:
How much RAM does your machine have?
Write to the disk an amount that is larger than the amount of RAM and see if you get the same result.
Also you can use hdparm to flush the caches before running dd the 2nd time.
Here we go again. Read up on what conv=fdatasync means, and why cache is irrelevant in this test.
Signature · 2 lines
With respect, Roman
Quoted 7 lines · Majed B.
How much RAM does your machine have?
Write to the disk an amount that is larger than the amount of RAM and see if you get the same result.
Also you can use hdparm to flush the caches before running dd the 2nd time.
See the meaning of the fdatasync option to dd and the /proc/sys/vm/drop_cache control. You are right about hdparm, but it only matters if you are using tiny data, drive caches are small.
Signature · 3 lines
Bill Davidsen <[email protected]> "We can't solve today's problems by using the same thinking we used in creating them." - Einstein
Quoted 11 lines · On Thu, 1 Jul 2010 14:40:44 +0800 Shaochun Wang <[email protected]> wrote:
-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc Password: 5000+0 records in 5000+0 records out 5242880000 bytes (5.2 GB) copied, 63.497 s, 82.6 MB/s -bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc 5000+0 records in 5000+0 records out 5242880000 bytes (5.2 GB) copied, 18.1033 s, 290 MB/s
Quoted 1 line
I don't know why the second dd becomes 290MB/s and the first 82.6MB/s.
That's because the first time the filesystem had to increase the file's size 5000 times by allocating additional 1 MB, and the second time it was just writing to an already allocated file. If you see such a big difference here, run that test 3 or more times, and discard the first run's results.
Signature · 2 lines
With respect, Roman
Quoted 14 lines · Roman Mamedov
Quoted 11 lines · On Thu, 1 Jul 2010 14:40:44 +0800 Shaochun Wang <[email protected]> wrote:
-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc Password: 5000+0 records in 5000+0 records out 5242880000 bytes (5.2 GB) copied, 63.497 s, 82.6 MB/s -bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc 5000+0 records in 5000+0 records out 5242880000 bytes (5.2 GB) copied, 18.1033 s, 290 MB/s
Quoted 6 lines
Quoted 1 line
I don't know why the second dd becomes 290MB/s and the first 82.6MB/s.
That's because the first time the filesystem had to increase the file's size 5000 times by allocating additional 1 MB, and the second time it was just writing to an already allocated file. If you see such a big difference here, run that test 3 or more times, and discard the first run's results.
Or use a proper filesystem benchmarking tool like bonnie++ and read its documentation so you know what it's telling you. dd is (in my opinion) only really any use for testing raw device streaming write/read speed.
And no, I don't understand why you get better performance with the write-intent bitmap turned on, unless you said that because you saw something like the above (as Roman says, your initial conditions were different so it's not a valid comparison). Usually, you need to tweak the write-intent bitmap's chunk size to suit your array and desired recovery speed to avoid it killing performance. I use a 16MB chunk on arrays with a few cheap drives, others go as high as 128MB on arrays with lots of high-performance quality drives.
Cheers,
John.
Quoted 15 lines · Roman Mamedov
Quoted 11 lines · On Thu, 1 Jul 2010 14:40:44 +0800 Shaochun Wang <[email protected]> wrote:
-bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc Password: 5000+0 records in 5000+0 records out 5242880000 bytes (5.2 GB) copied, 63.497 s, 82.6 MB/s -bash-4.1$ sudo dd if=/dev/zero of=test.dd bs=1M count=5000 conv=fdatasync,notrunc 5000+0 records in 5000+0 records out 5242880000 bytes (5.2 GB) copied, 18.1033 s, 290 MB/s
Quoted 9 lines
Quoted 2 lines
I don't know why the second dd becomes 290MB/s and the first 82.6MB/s.
That's because the first time the filesystem had to increase the file's size 5000 times by allocating additional 1 MB, and the second time it was just writing to an already allocated file. If you see such a big difference here, run that test 3 or more times, and discard the first run's results.
Unless the notrunc option is used, the file is truncated and allocated all over again. I'm pretty sure the inode is reused, that's down in the noise.
Signature · 3 lines
Bill Davidsen <[email protected]> "We can't solve today's problems by using the same thinking we used in creating them." - Einstein