Awful RAID5 random read performance
A friend writes:
On a recent machine set up with Raid5.
On a AMD Phenom II X4 810, and 4GB ram.
4 Seagate 7200.12 SATA 1TB drives,
I'm getting some rather impressive numbers for sequential read (300MB/s+) and write (170MB/s+) but the random read is proving to be absolutely atrocious.
iostat says its going at about 0.5MB/s,
I've seen plenty of references to people getting numbers in the double digits for random reads on a md raid5 array, and one with slower disks to boot.
I tried disabling automatic acoustic management on the drives, but it didn't seem to help at all.
I really don't care that much about the noise coming from a file server stuck in a closet ;)
Does anyone know why I'm seeing such bad random read times? There's got to be some configuration error on my part, but I just can't seem to find what it might be.
I've spent a few hours on google looking for various tweaks but almost nobody even mentions random read/write times.
All everyone seems to care about is block sequential access time. I'd be willing to sacrifice a fair portion of my rather excellent sequential times for much better random access.
Signature · 1 line
Regards, Maurice
Quoted 10 lines · Maurice Hilarius
A friend writes:
On a recent machine set up with Raid5.
On a AMD Phenom II X4 810, and 4GB ram.
4 Seagate 7200.12 SATA 1TB drives,I'm getting some rather impressive numbers for sequential read (300MB/s+) and write (170MB/s+) but the random read is proving to be absolutely atrocious.
iostat says its going at about 0.5MB/s,
The key thing about random i/o is the block size. With, say, 512bytes blocks and single thread you will see less than 0.5Mb/sec. With 64kbytes blocksize it will be much better.
To diagnose: first try the same test on bare disk without raid layer. Next try to vary block size and number of concurrent threads doing I/O. There's no tweaks needed really.
/mjt
Quoted 12 lines · Michael Tokarev
Quoted 10 lines · Maurice Hilarius
A friend writes:
On a recent machine set up with Raid5.
On a AMD Phenom II X4 810, and 4GB ram.
4 Seagate 7200.12 SATA 1TB drives,I'm getting some rather impressive numbers for sequential read (300MB/s+) and write (170MB/s+) but the random read is proving to be absolutely atrocious.
iostat says its going at about 0.5MB/s,
Quoted 7 lines
The key thing about random i/o is the block size. With, say, 512bytes blocks and single thread you will see less than 0.5Mb/sec. With 64kbytes blocksize it will be much better.
To diagnose: first try the same test on bare disk without raid layer. Next try to vary block size and number of concurrent threads doing I/O. There's no tweaks needed really.
I happen to be the friend Maurice was talking about. I let the raid layer keep its default chunk size of 64K. The smaller size (below like 2MB) tests in iozone are very very slow. I recently tried disabling readahead, Acoustic Management, and played with the io scheduler and all any of it has done is make the sequential access slower and has barely touched the smaller sized random access test results. Even with the 64K iozone test random read/write is only in the 7 and 11MB/s range.
It just seems too low to me.
Quoted 5 lines
/mjt
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 16 lines · Thomas Fjellstrom
Quoted 15 lines · Michael Tokarev
Quoted 10 lines · Maurice Hilarius
A friend writes:
On a recent machine set up with Raid5.
On a AMD Phenom II X4 810, and 4GB ram.
4 Seagate 7200.12 SATA 1TB drives,I'm getting some rather impressive numbers for sequential read (300MB/s+) and write (170MB/s+) but the random read is proving to be absolutely atrocious.
iostat says its going at about 0.5MB/s,The key thing about random i/o is the block size. With, say, 512bytes blocks and single thread you will see less than 0.5Mb/sec. With 64kbytes blocksize it will be much better.
Quoted 7 lines
Quoted 3 lines
To diagnose: first try the same test on bare disk without raid layer. Next try to vary block size and number of concurrent threads doing I/O. There's no tweaks needed really.
I happen to be the friend Maurice was talking about. I let the raid layer keep its default chunk size of 64K. The smaller size (below like 2MB) tests in iozone are very very slow. I recently tried disabling readahead, Acoustic Management, and played with the io scheduler and all any of it has done is make the sequential access slower and has barely touched the smaller sized random access test results. Even with the 64K iozone test random read/write is only in the 7 and 11MB/s range.
It just seems too low to me.
I don't think so; can you try a similar test on single drives not using md RAID-5?
The killer is seeks, which is what random I/O uses lots of; with a 10ms seek time you're only going to get ~100 seeks/second and if you're only reading 512 bytes after each seek you're only going to get ~500 kbytes/second. Bigger block sizes will show higher throughput, but you'll still only get ~100 seeks/second.
Clearly when you're doing this over 4 drives you can have ~400 seeks/second but that's still limiting you to ~400 reads/second for smallish block sizes.
Cheers,
John.
Quoted 25 lines
Quoted 1 line
I happen to be the friend Maurice was talking about. I let the raid
layer keep
Quoted 1 line
its default chunk size of 64K. The smaller size (below like 2MB) tests
in
Quoted 1 line
iozone are very very slow. I recently tried disabling readahead,
Acoustic
Quoted 1 line
Management, and played with the io scheduler and all any of it has done
is
Quoted 1 line
make the sequential access slower and has barely touched the smaller
sized
Quoted 1 line
random access test results. Even with the 64K iozone test random
read/write is
Quoted 3 lines
only in the 7 and 11MB/s range.
It just seems too low to me.
I don't think so; can you try a similar test on single drives not using md RAID-5?
The killer is seeks, which is what random I/O uses lots of; with a 10ms seek time you're only going to get ~100 seeks/second and if you're only reading 512 bytes after each seek you're only going to get ~500 kbytes/second. Bigger block sizes will show higher throughput, but you'll still only get ~100 seeks/second.
Quoted 3 lines
Clearly when you're doing this over 4 drives you can have ~400 seeks/second but that's still limiting you to ~400 reads/second for smallish block sizes.
John is perfectly correct, although of course a 10ms seek is a
fairly slow one. The point is, it is drive dependent, and there may not be much one can do about it at the software layer. That said, you might try a different scheduler, as the seek order can make a difference. Drives with larger caches may help some, although the increase in performance with larger cache sizes diminishes rapidly beyond a certain point. As one would infer from John's post, increasing the number of drives in the array will help a lot, since increasing the number of drives raises the limit on the number of seeks / second.
What file system are you using? It can make a difference, and
surely has a bigger impact than most tweaks to the RAID subsystem.
The biggest question in my mind, however, is why is random access a
big issue for you? Are you running a very large relational database with tens of thousands of tiny files? For most systems, high volume accesses consist mostly of large sequential I/O. The majority of random I/O is of rather short duration, meaning even with comparatively poor performance, it doesn't take long to get the job done. Fifty to eighty Megabits per second is nothing at which to sneeze for random access of small files. A few years ago, many drives would have been barely able to manage that on a sustained basis for sequential I/O.
Quoted 54 lines · Leslie Rhorer
Quoted 37 lines
Quoted 1 line
I happen to be the friend Maurice was talking about. I let the raid
layer keep
Quoted 1 line
its default chunk size of 64K. The smaller size (below like 2MB) tests
in
Quoted 1 line
iozone are very very slow. I recently tried disabling readahead,
Acoustic
Quoted 1 line
Management, and played with the io scheduler and all any of it has done
is
Quoted 1 line
make the sequential access slower and has barely touched the smaller
sized
Quoted 1 line
random access test results. Even with the 64K iozone test random
read/write is
Quoted 3 lines
only in the 7 and 11MB/s range.
It just seems too low to me.
I don't think so; can you try a similar test on single drives not using md RAID-5?
The killer is seeks, which is what random I/O uses lots of; with a 10ms seek time you're only going to get ~100 seeks/second and if you're only reading 512 bytes after each seek you're only going to get ~500 kbytes/second. Bigger block sizes will show higher throughput, but you'll still only get ~100 seeks/second.
Quoted 3 lines
Clearly when you're doing this over 4 drives you can have ~400 seeks/second but that's still limiting you to ~400 reads/second for smallish block sizes.
John is perfectly correct, although of course a 10ms seek is a
fairly slow one. The point is, it is drive dependent, and there may not be much one can do about it at the software layer. That said, you might try a different scheduler, as the seek order can make a difference. Drives with larger caches may help some, although the increase in performance with larger cache sizes diminishes rapidly beyond a certain point. As one would infer from John's post, increasing the number of drives in the array will help a lot, since increasing the number of drives raises the limit on the number of seeks / second.What file system are you using? It can make a difference, and
surely has a bigger impact than most tweaks to the RAID subsystem.
Quoted 9 lines
The biggest question in my mind, however, is why is random access a
big issue for you? Are you running a very large relational database with tens of thousands of tiny files? For most systems, high volume accesses consist mostly of large sequential I/O. The majority of random I/O is of rather short duration, meaning even with comparatively poor performance, it doesn't take long to get the job done. Fifty to eighty Megabits per second is nothing at which to sneeze for random access of small files. A few years ago, many drives would have been barely able to manage that on a sustained basis for sequential I/O.
I thought the numbers were way too low. But I guess I was wrong. I really only have three use cases for my arrays. One will be hosting VM images/volumes, and iso disk images, while another will be hosting large media which will be streaming off, p2p downloads, amd rsync/rsnapshot backups of several machines. I imagine the vm array will appreciate faster random io (boot times will improve, as will things like database and http disk access), and the p2p surely will appreciate faster random io.
I currently have them all on one disk array, but I'm thinking its a good idea to separate the media from the VMs. when ktorrent is downloading a linux iso or something similar atop shows very high disk utilization for ktorrent, same goes for booting VMs. and the backups, oh my lord does that take a while, I even tell it to skip a lot of stuff I don't need to backup.
When I get around to it I may utilize the raid10 module for the VM's and backups. Though that may decrease performance a little bit in the small random io case.
Quoted 5 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 79 lines · Thomas Fjellstrom
Quoted 54 lines · Leslie Rhorer
Quoted 37 lines
Quoted 1 line
I happen to be the friend Maurice was talking about. I let the raid
layer keep
Quoted 1 line
its default chunk size of 64K. The smaller size (below like 2MB) tests
in
Quoted 1 line
iozone are very very slow. I recently tried disabling readahead,
Acoustic
Quoted 1 line
Management, and played with the io scheduler and all any of it has done
is
Quoted 1 line
make the sequential access slower and has barely touched the smaller
sized
Quoted 1 line
random access test results. Even with the 64K iozone test random
read/write is
Quoted 3 lines
only in the 7 and 11MB/s range.
It just seems too low to me.
I don't think so; can you try a similar test on single drives not using md RAID-5?
The killer is seeks, which is what random I/O uses lots of; with a 10ms seek time you're only going to get ~100 seeks/second and if you're only reading 512 bytes after each seek you're only going to get ~500 kbytes/second. Bigger block sizes will show higher throughput, but you'll still only get ~100 seeks/second.
Quoted 3 lines
Clearly when you're doing this over 4 drives you can have ~400 seeks/second but that's still limiting you to ~400 reads/second for smallish block sizes.
John is perfectly correct, although of course a 10ms seek is a
fairly slow one. The point is, it is drive dependent, and there may not be much one can do about it at the software layer. That said, you might try a different scheduler, as the seek order can make a difference. Drives with larger caches may help some, although the increase in performance with larger cache sizes diminishes rapidly beyond a certain point. As one would infer from John's post, increasing the number of drives in the array will help a lot, since increasing the number of drives raises the limit on the number of seeks / second.What file system are you using? It can make a difference, and
surely has a bigger impact than most tweaks to the RAID subsystem.Quoted 9 lines
The biggest question in my mind, however, is why is random access a
big issue for you? Are you running a very large relational database with tens of thousands of tiny files? For most systems, high volume accesses consist mostly of large sequential I/O. The majority of random I/O is of rather short duration, meaning even with comparatively poor performance, it doesn't take long to get the job done. Fifty to eighty Megabits per second is nothing at which to sneeze for random access of small files. A few years ago, many drives would have been barely able to manage that on a sustained basis for sequential I/O.I thought the numbers were way too low. But I guess I was wrong. I really only have three use cases for my arrays. One will be hosting VM images/volumes, and iso disk images, while another will be hosting large media which will be streaming off, p2p downloads, amd rsync/rsnapshot backups of several machines. I imagine the vm array will appreciate faster random io (boot times will improve, as will things like database and http disk access), and the p2p surely will appreciate faster random io.
I currently have them all on one disk array, but I'm thinking its a good idea to separate the media from the VMs. when ktorrent is downloading a linux iso or something similar atop shows very high disk utilization for ktorrent, same goes for booting VMs. and the backups, oh my lord does that take a while, I even tell it to skip a lot of stuff I don't need to backup.
Quoted 3 lines
When I get around to it I may utilize the raid10 module for the VM's and backups. Though that may decrease performance a little bit in the small random io case.
raid10,f2 may actually speed up random i/o as seeks are in essence confined to the faster outer sectors of the disk, and thereby about halfing the access times.
best regards
keld
Quoted 73 lines · Thomas Fjellstrom
Quoted 60 lines · Leslie Rhorer
Quoted 43 lines
Quoted 2 lines
I happen to be the friend Maurice was talking about. I let the raid
layer keep
Quoted 2 lines
its default chunk size of 64K. The smaller size (below like 2MB) tests
in
Quoted 2 lines
iozone are very very slow. I recently tried disabling readahead,
Acoustic
Quoted 2 lines
Management, and played with the io scheduler and all any of it has done
is
Quoted 2 lines
make the sequential access slower and has barely touched the smaller
sized
Quoted 2 lines
random access test results. Even with the 64K iozone test random
read/write is
Quoted 4 lines
only in the 7 and 11MB/s range.
It just seems too low to me.
I don't think so; can you try a similar test on single drives not using md RAID-5?
The killer is seeks, which is what random I/O uses lots of; with a 10ms seek time you're only going to get ~100 seeks/second and if you're only reading 512 bytes after each seek you're only going to get ~500 kbytes/second. Bigger block sizes will show higher throughput, but you'll still only get ~100 seeks/second.
Quoted 4 lines
Clearly when you're doing this over 4 drives you can have ~400 seeks/second but that's still limiting you to ~400 reads/second for smallish block sizes.
John is perfectly correct, although of course a 10ms seek is a
fairly slow one. The point is, it is drive dependent, and there may not be much one can do about it at the software layer. That said, you might try a different scheduler, as the seek order can make a difference. Drives with larger caches may help some, although the increase in performance with larger cache sizes diminishes rapidly beyond a certain point. As one would infer from John's post, increasing the number of drives in the array will help a lot, since increasing the number of drives raises the limit on the number of seeks / second.What file system are you using? It can make a difference, and
surely has a bigger impact than most tweaks to the RAID subsystem.Quoted 10 lines
The biggest question in my mind, however, is why is random access a
big issue for you? Are you running a very large relational database with tens of thousands of tiny files? For most systems, high volume accesses consist mostly of large sequential I/O. The majority of random I/O is of rather short duration, meaning even with comparatively poor performance, it doesn't take long to get the job done. Fifty to eighty Megabits per second is nothing at which to sneeze for random access of small files. A few years ago, many drives would have been barely able to manage that on a sustained basis for sequential I/O.
Quoted 6 lines
I thought the numbers were way too low. But I guess I was wrong. I really only have three use cases for my arrays. One will be hosting VM images/volumes, and iso disk images, while another will be hosting large media which will be streaming off, p2p downloads, amd rsync/rsnapshot backups of several machines. I imagine the vm array will appreciate faster random io (boot times will improve, as will things like database and http disk access), and the p2p surely will appreciate faster random io.
I currently have them all on one disk array, but I'm thinking its a good idea to separate the media from the VMs. when ktorrent is downloading a linux iso or something similar atop shows very high disk utilization for ktorrent, same goes for booting VMs. and the backups, oh my lord does that take a while, I even tell it to skip a lot of stuff I don't need to backup.
When I get around to it I may utilize the raid10 module for the VM's and backups. Though that may decrease performance a little bit in the small random io case.
The accesses on the VM will be similar to a real disk, so you want the VM on whatever you would use for bare iron. I run on raid10, many of my machines are on VM (including this one, my main desktop). Raid10 is a good general use array, I use it for a lot, other than cases where I need cheap space and use raid[56] to get more bytes/$ and don't need blinding speed. Archival storage, for instance.
Signature · 7 lines
Bill Davidsen <[email protected]> Even purely technical things can appear to be magic, if the documentation is obscure enough. For example, PulseAudio is configured by dancing naked around a fire at midnight, shaking a rattle with one hand and a LISP manual with the other, while reciting the GNU manifesto in hexadecimal. The documentation fails to note that you must circle the fire counter-clockwise in the southern hemisphere.
Quoted 6 lines · Bill Davidsen
Quoted 4 lines · Thomas Fjellstrom
When I get around to it I may utilize the raid10 module for the VM's and backups. Though that may decrease performance a little bit in the small random io case.
Quoted 6 lines
The accesses on the VM will be similar to a real disk, so you want the VM on whatever you would use for bare iron. I run on raid10, many of my machines are on VM (including this one, my main desktop). Raid10 is a good general use array, I use it for a lot, other than cases where I need cheap space and use raid[56] to get more bytes/$ and don't need blinding speed. Archival storage, for instance.
My perception is that raid10,f2 is probably the fastest also for small random reads because of the lower latency, and faster transfer times due to only using the outer disk sectors. For writes the elevator evens out the ramdom access. Benchmarks may not show this effect as they are often done on clean file systems, where the files are allocated in the beginning of the fs.
For cases where you need cheap disk space, and have big files like .iso's then raid5 could be a good choice because it has the most space while maintaining fair to good performance for big files.
In your case, using 3 disks, raid5 should give about 210 % of the nominal single disk speed for big file reads, and maybe 180 % for big file writes. raid10,f2 should give about 290 % for big file reads and 140% for big file writes. Random reads should be about the same for raid5 and raid10,f2 - raid10,f2 maybe 15 % faster, while random writes should be mediocre for raid5, and good for raid10,f2.
best regards
keld
On 02/06/2009 20:47, Keld Jørn Simonsen wrote:
[...]
Quoted 11 lines
My perception is that raid10,f2 is probably the fastest also for small random reads because of the lower latency, and faster transfer times due to only using the outer disk sectors. For writes the elevator evens out the ramdom access. Benchmarks may not show this effect as they are often done on clean file systems, where the files are allocated in the beginning of the fs.
For cases where you need cheap disk space, and have big files like .iso's then raid5 could be a good choice because it has the most space while maintaining fair to good performance for big files.
Quoted 6 lines
In your case, using 3 disks, raid5 should give about 210 % of the nominal single disk speed for big file reads, and maybe 180 % for big file writes. raid10,f2 should give about 290 % for big file reads and 140% for big file writes. Random reads should be about the same for raid5 and raid10,f2 - raid10,f2 maybe 15 % faster, while random writes should be mediocre for raid5, and good for raid10,f2.
I'd be interested in reading about where you got these figures from and/or the rationale behind them; I'd have guessed differently...
Cheers,
John.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 19 lines · John Robinson
On 02/06/2009 20:47, Keld Jørn Simonsen wrote:
[...]Quoted 7 lines
My perception is that raid10,f2 is probably the fastest also for small random reads because of the lower latency, and faster transfer times due to only using the outer disk sectors. For writes the elevator evens out the ramdom access. Benchmarks may not show this effect as they are often done on clean file systems, where the files are allocated in the beginning of the fs.
Quoted 9 lines
For cases where you need cheap disk space, and have big files like .iso's then raid5 could be a good choice because it has the most space while maintaining fair to good performance for big files.
In your case, using 3 disks, raid5 should give about 210 % of the nominal single disk speed for big file reads, and maybe 180 % for big file writes. raid10,f2 should give about 290 % for big file reads and 140% for big file writes. Random reads should be about the same for raid5 and raid10,f2 - raid10,f2 maybe 15 % faster, while random writes should be mediocre for raid5, and good for raid10,f2.
Quoted 1 line
I'd be interested in reading about where you got these figures from and/or the rationale behind them; I'd have guessed differently...
For small values of N, 10,f2 generally comes quite close to N*Sr, where N is # of disks and Sr is single drive read speed. This is assuming fiarly large reads and adequate stripe buffer space. Obviously for larger values of N that saturates something else in the system, like the bus, before N gets too large. I don't generally see more than (N/2-1)*Sw for write, at least for large writes. I came up with those numbers based on testing 3-4-5 drive arrays which do large file transfers. If you want to read more than large file speed into them, feel free.
All tests done on raw devices and raw arrays, and ext3 devices and arrays. The ratios stay about the same, tuning stripe size (stride) can be helpful for improving write speed.
Short summary - the numbers look close enough to mine that I would say they are at least useful approximations.
Signature · 14 lines
Bill Davidsen <[email protected]> Even purely technical things can appear to be magic, if the documentation is obscure enough. For example, PulseAudio is configured by dancing naked around a fire at midnight, shaking a rattle with one hand and a LISP manual with the other, while reciting the GNU manifesto in hexadecimal. The documentation fails to note that you must circle the fire counter-clockwise in the southern hemisphere. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 2 lines · Bill Davidsen
Quoted 1 line · John Robinson
On 02/06/2009 20:47, Keld Jørn Simonsen wrote:
[...]
Quoted 10 lines
Quoted 8 lines
Quoted 6 lines
In your case, using 3 disks, raid5 should give about 210 % of the nominal single disk speed for big file reads, and maybe 180 % for big file writes. raid10,f2 should give about 290 % for big file reads and 140% for big file writes. Random reads should be about the same for raid5 and raid10,f2 - raid10,f2 maybe 15 % faster, while random writes should be mediocre for raid5, and good for raid10,f2.
I'd be interested in reading about where you got these figures from and/or the rationale behind them; I'd have guessed differently...
For small values of N, 10,f2 generally comes quite close to N*Sr, where N is # of disks and Sr is single drive read speed. This is assuming fiarly large reads and adequate stripe buffer space. Obviously for larger values of N that saturates something else in the system, like the bus, before N gets too large. I don't generally see more than (N/2-1)*Sw for write, at least for large writes. I came up with those numbers based on testing 3-4-5 drive arrays which do large file transfers. If you want to read more than large file speed into them, feel free.
Actually it was the RAID-5 figures I'd have guessed differently. I'd expect ~290% (rather than 210%) for big 3-disc RAID-5 reads, and ~140% (rather than "mediocre") for random small writes. But of course I haven't tested.
Cheers,
John.
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 15 lines · John Robinson
Quoted 2 lines · Bill Davidsen
Quoted 1 line · John Robinson
On 02/06/2009 20:47, Keld Jørn Simonsen wrote:
[...]
Quoted 11 lines
Quoted 10 lines
Quoted 7 lines
In your case, using 3 disks, raid5 should give about 210 % of the nominal
single disk speed for big file reads, and maybe 180 % for big file writes. raid10,f2 should give about 290 % for big file reads and 140% for big file writes. Random reads should be about the same for raid5 and raid10,f2 - raid10,f2 maybe 15 % faster, while random writes should be mediocre for raid5, and good for raid10,f2.I'd be interested in reading about where you got these figures from and/or the rationale behind them; I'd have guessed differently...
Quoted 9 lines
Quoted 9 lines
For small values of N, 10,f2 generally comes quite close to N*Sr, where N is # of disks and Sr is single drive read speed. This is assuming fiarly large reads and adequate stripe buffer space. Obviously for larger values of N that saturates something else in the system, like the bus, before N gets too large. I don't generally see more than (N/2-1)*Sw for write, at least for large writes. I came up with those numbers based on testing 3-4-5 drive arrays which do large file transfers. If you want to read more than large file speed into them, feel free.
With far copies reading is like reading raid0 and writing is like raid0 but writing twice with a seek between each. So (N/2) and (N/2-a bit) are the theoretical maximums and raid10 comes damn close to those.
Quoted 4 lines
Actually it was the RAID-5 figures I'd have guessed differently. I'd expect ~290% (rather than 210%) for big 3-disc RAID-5 reads, and ~140% (rather than "mediocre") for random small writes. But of course I haven't tested.
That kind of depends on the chunk size I think.
Say you have a raid 5 with chunk size << size of 1 track. Then on each disk you read 2 chunks, skip a chunk, read 2 chunks, skip a chunk. But skipping a chunk means waiting for the disk to rotate over it. That takes as long as reading it. You shouldn't even get 210% speed.
Only if chunk size >> size of 1 track could you seek over a chunk. And you have to hope that by the time you have seeked the start of the next chunk hasn't rotated past the head yet.
Anyone know what the size of a track is on modern disks? How many sectors/track do they have?
MfG
Goswin
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 14 lines · Goswin von Brederlow
Quoted 12 lines · John Robinson
Quoted 2 lines · Bill Davidsen
Quoted 1 line · John Robinson
On 02/06/2009 20:47, Keld Jørn Simonsen wrote:
[...]
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
In your case, using 3 disks, raid5 should give about 210 % of the nominal
single disk speed for big file reads, and maybe 180 % for big file writes. raid10,f2 should give about 290 % for big file reads and 140% for big file writes. Random reads should be about the same for raid5 and raid10,f2 - raid10,f2 maybe 15 % faster, while random writes should be mediocre for raid5, and good for raid10,f2.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
I'd be interested in reading about where you got these figures from and/or the rationale behind them; I'd have guessed differently...
See more on our wiki for actual benchmarks,
http://linux-raid.osdl.org/index.php/Performance
http://blog.jamponi.net/2008/07/raid56-and-10-benchmarks-on-26255_10.html
The latter reports on arrays with 4 disks, som downscale it and you get a good idea of expected values for 3 disks.
Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
For small values of N, 10,f2 generally comes quite close to N*Sr, where N is # of disks and Sr is single drive read speed. This is assuming fiarly large reads and adequate stripe buffer space. Obviously for larger values of N that saturates something else in the system, like the bus, before N gets too large. I don't generally see more than (N/2-1)*Sw for write, at least for large writes. I came up with those numbers based on testing 3-4-5 drive arrays which do large file transfers. If you want to read more than large file speed into them, feel free.
Quoted 3 lines
With far copies reading is like reading raid0 and writing is like raid0 but writing twice with a seek between each. So (N/2) and (N/2-a bit) are the theoretical maximums and raid10 comes damn close to those.
My take on theoretical maxima is:
raid10,f2 for sequential reads: N * Sr
Raid10,f2 for sequential writes: N/2 * Sw
Quoted 13 lines
Quoted 5 lines
Actually it was the RAID-5 figures I'd have guessed differently. I'd expect ~290% (rather than 210%) for big 3-disc RAID-5 reads, and ~140% (rather than "mediocre") for random small writes. But of course I haven't tested.
That kind of depends on the chunk size I think.
Say you have a raid 5 with chunk size << size of 1 track. Then on each disk you read 2 chunks, skip a chunk, read 2 chunks, skip a chunk. But skipping a chunk means waiting for the disk to rotate over it. That takes as long as reading it. You shouldn't even get 210% speed.
Quoted 6 lines
Only if chunk size >> size of 1 track could you seek over a chunk. And you have to hope that by the time you have seeked the start of the next chunk hasn't rotated past the head yet.
Anyone know what the size of a track is on modern disks? How many sectors/track do they have?
I believe Goswins analyses here is valid, skipping sectors is as expensive as reading them.
Anyway, using somewhat bigger chunk sizes you may get into the effect of not reading/seeking over data, and thus go beyond the N-1 mark. As I was trying to report best values obtainable, then I chose to report this factor also. Actually some figures show a loss of only 0.50 for sequential reads on raid5 with a chunk size of 2 MB.
For sequential writes I was asuming that you were writing 2 data stripes and 1 parity stripe, and that the theoretical effective writing speed would get close to 2 (for a 3 disk raid5). Jon's benchmark does not support this. His best figures for raid5 is a loss of 2.25 write speed, where I would expect somethng like a little more than 1. Maybe the fact that the test is on raw partitions, and not on a file system with an active elevator is in play here. Maybe it is because there is quite some calculations involved for the parity calculation, and because of no elevator, the system have to wait for completion of parity calculation before parity writes can be done.
For random writes on raid5 I reported "mediocre". This is because that if you write randomly in raid5, you need to first read the chunk, read the parity chunk, do updating and then write the chunk and the parity chunk again. And you need to read full chuncks. So at most you will something like N/4 if your data size is close to the chunk size. If you have a big chunk size and smallish payload size than a lot of read/writes are done on uninteresting data. This probably also goes for other raid types, and the fs elevator may help a little here, especially for writing.
In general I think raid5 random writes would be in the order of N/4 where mirrored raid types would be N/2 (with 2 copies) - making raid5 half speed of mirrored raid types like raid1 and raid10. I am not sure I have data to back that statement up.
best regards
keld
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
On Thu, Jun 04, 2009 at 12:21:02AM +0200, Goswin von Brederlow wrote:
.....mi
Quoted 3 lines
Anyone know what the size of a track is on modern disks? How many sectors/track do they have?
The number will differ from the inside to the outside of the disk. The number of zones will differ from drive to drive too....
Some diagnostic software will either know this or have vendor specific ways to get it live from the disk.
Data sheets now report average... At one time vendors made a big deal on this...
http://www.impediment.com/seagate/s2000/spec_318436lcv.shtml
Sectors/Track (avg) 426
Bytes/Track (avg) 218,112If you take numbers like:
Track Density (TPI) 18,145 tracks/inch
Recording Density (BPI, max) 328,272 bits/inch and dust off some geometry you might discover how many more bits are possible on the outside tracks vs the inside tracks and then take those bits and estimate the number of additional blocks. An estimate may all you get because spares and other uses for bits will be hidden.
Signature · 2 lines
T o m M i t c h e l l Found me a new hat, now what?
Quoted 51 lines · Goswin von Brederlow
Quoted 19 lines · John Robinson
Quoted 4 lines · Bill Davidsen
Quoted 2 lines · John Robinson
On 02/06/2009 20:47, Keld Jørn Simonsen wrote:
[...]
Quoted 11 lines
Quoted 11 lines
Quoted 8 lines
In your case, using 3 disks, raid5 should give about 210 % of the nominal
single disk speed for big file reads, and maybe 180 % for big file writes. raid10,f2 should give about 290 % for big file reads and 140% for big file writes. Random reads should be about the same for raid5 and raid10,f2 - raid10,f2 maybe 15 % faster, while random writes should be mediocre for raid5, and good for raid10,f2.I'd be interested in reading about where you got these figures from and/or the rationale behind them; I'd have guessed differently...
Quoted 10 lines
Quoted 10 lines
For small values of N, 10,f2 generally comes quite close to N*Sr, where N is # of disks and Sr is single drive read speed. This is assuming fiarly large reads and adequate stripe buffer space. Obviously for larger values of N that saturates something else in the system, like the bus, before N gets too large. I don't generally see more than (N/2-1)*Sw for write, at least for large writes. I came up with those numbers based on testing 3-4-5 drive arrays which do large file transfers. If you want to read more than large file speed into them, feel free.
With far copies reading is like reading raid0 and writing is like raid0 but writing twice with a seek between each. So (N/2) and (N/2-a bit) are the theoretical maximums and raid10 comes damn close to those.
Quoted 5 lines
Actually it was the RAID-5 figures I'd have guessed differently. I'd expect ~290% (rather than 210%) for big 3-disc RAID-5 reads, and ~140% (rather than "mediocre") for random small writes. But of course I haven't tested.
That kind of depends on the chunk size I think.
Say you have a raid 5 with chunk size << size of 1 track. Then on each disk you read 2 chunks, skip a chunk, read 2 chunks, skip a chunk. But skipping a chunk means waiting for the disk to rotate over it. That takes as long as reading it. You shouldn't even get 210% speed.
Quoted 7 lines
Only if chunk size >> size of 1 track could you seek over a chunk. And you have to hope that by the time you have seeked the start of the next chunk hasn't rotated past the head yet.
Anyone know what the size of a track is on modern disks? How many sectors/track do they have?
It varies to keep the bpi constant, so there are more sectors on outer tracks and transfer rate is higher. raid10 can use outer tracks in more cases (with the "far" layout) and thus delivers a higher transfer rate. Or so the theory goes, in practice raid10 *does* give a higher transfer rate, so the above is theory to explain the observed facts.
Signature · 14 lines
Bill Davidsen <[email protected]> Even purely technical things can appear to be magic, if the documentation is obscure enough. For example, PulseAudio is configured by dancing naked around a fire at midnight, shaking a rattle with one hand and a LISP manual with the other, while reciting the GNU manifesto in hexadecimal. The documentation fails to note that you must circle the fire counter-clockwise in the southern hemisphere. -- To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Leslie Rhorer ([email protected]) wrote on 31 May 2009 10:41:
Quoted 25 lines
Quoted 25 lines
Quoted 1 line
I happen to be the friend Maurice was talking about. I let the raid
layer keep
Quoted 1 line
its default chunk size of 64K. The smaller size (below like 2MB) tests
in
Quoted 1 line
iozone are very very slow. I recently tried disabling readahead,
Acoustic
Quoted 1 line
Management, and played with the io scheduler and all any of it has done
is
Quoted 1 line
make the sequential access slower and has barely touched the smaller
sized
Quoted 1 line
random access test results. Even with the 64K iozone test random
read/write is
Quoted 3 lines
only in the 7 and 11MB/s range.
It just seems too low to me.
I don't think so; can you try a similar test on single drives not using md RAID-5?
The killer is seeks, which is what random I/O uses lots of; with a 10ms seek time you're only going to get ~100 seeks/second and if you're only reading 512 bytes after each seek you're only going to get ~500 kbytes/second. Bigger block sizes will show higher throughput, but you'll still only get ~100 seeks/second.
Quoted 6 lines
Quoted 3 lines
Clearly when you're doing this over 4 drives you can have ~400 seeks/second but that's still limiting you to ~400 reads/second for smallish block sizes.
John is perfectly correct, although of course a 10ms seek is a
fairly slow one.
Unfortunately it doesn't seem to be. Take a well-considered drive such as the WD RE3; it's spec for average latency is 4.2ms. However does it include the rotational latency (the time the head takes to reach the sector once it's on the track)? I bet it doesn't. Taking it to be only the average seek time, this drive is still among the fastest. For a 7200rpm drive this latency is just 4.2ms, so we'd have for this fast drive an average total latency of 8.4ms.
Quoted 4 lines
The biggest question in my mind, however, is why is random access a
big issue for you? Are you running a very large relational database with tens of thousands of tiny files? For most systems, high volume accesses consist mostly of large sequential I/O.
No, random I/O is the most common case for busy servers, when there are lots of processes doing uncorrelated reads and writes. Even if a single application does sequential access the head will likely have moved between them. The only solution is to have lots of ram for cache, and/or lots of disks. It'd be better if they were connected to several controllers...
Quoted 10 lines
Quoted 2 lines
John is perfectly correct, although of course a 10ms seek is a
fairly slow one.Unfortunately it doesn't seem to be. Take a well-considered drive such as the WD RE3; it's spec for average latency is 4.2ms. However does it include the rotational latency (the time the head takes to reach the sector once it's on the track)? I bet it doesn't. Taking it to be only the average seek time, this drive is still among the fastest. For a 7200rpm drive this latency is just 4.2ms, so we'd have for this fast drive an average total latency of 8.4ms.
That's an average. For a random seek to exceed that, it's going to have to span many cylinders. Give the container size of a modern cylinder, that's a pretty big jump. Single applications will tend to have their data lumped somewhat together on the drive.
Quoted 8 lines
Quoted 2 lines
The biggest question in my mind, however, is why is random access a
big issue for you? Are you running a very large relational databasewith
Quoted 2 lines
tens of thousands of tiny files? For most systems, high volume accesses consist mostly of large sequential I/O.
No, random I/O is the most common case for busy servers, when there are lots of processes doing uncorrelated reads and writes. Even if a
Yes, exactly. By definition, such a scenario represents a multithreaded set of seeks, and as we already established, multithreaded seeks are vastly more efficient than serial random seeks. The 400 seeks per second number for 4 drives applies. I don't know the details of the Linux schedulers, but most schedulers employ some variation of an elevator seek to maximize seek efficiency. The brings the average latency way down and brings the seek frequency way up.
Quoted 4 lines
single application does sequential access the head will likely have moved between them. The only solution is to have lots of ram for cache, and/or lots of disks. It'd be better if they were connected to several controllers...
A large RAM cache will help, but as I already pointed out, the increases in returns for increasing cache size diminish rapidly past a certain point. Most quality drives these days have a 32MB cache, or 128M for a 4 drive array. Add the Linux cache on top of that, and it should be sufficient for most purposes. Remember, random seeks implies small data extents. Lots of disks will bring the biggest benefit, and disks are cheap. Multiple controllers really are not necessary, especially if the controller and drives support NCQ , but having multiple controllers certainly doesn't hurt.
Quoted 23 lines · Leslie Rhorer
Quoted 8 lines
Unfortunately it doesn't seem to be. Take a well-considered drive such as the WD RE3; it's spec for average latency is 4.2ms. However does it include the rotational latency (the time the head takes to reach the sector once it's on the track)? I bet it doesn't. Taking it to be only the average seek time, this drive is still among the fastest. For a 7200rpm drive this latency is just 4.2ms, so we'd have for this fast drive an average total latency of 8.4ms.
That's an average. For a random seek to exceed that, it's going to have to span many cylinders. Give the container size of a modern cylinder, that's a pretty big jump. Single applications will tend to have their data lumped somewhat together on the drive.
Quoted 7 lines
with
No, random I/O is the most common case for busy servers, when there are lots of processes doing uncorrelated reads and writes. Even if a
Quoted 7 lines
Yes, exactly. By definition, such a scenario represents a multithreaded set of seeks, and as we already established, multithreaded seeks are vastly more efficient than serial random seeks. The 400 seeks per second number for 4 drives applies. I don't know the details of the Linux schedulers, but most schedulers employ some variation of an elevator seek to maximize seek efficiency. The brings the average latency way down and brings the seek frequency way up.
Ah, I never really understood how adding more random load could increase performance. Now I get it :)
Quoted 5 lines
Quoted 4 lines
single application does sequential access the head will likely have moved between them. The only solution is to have lots of ram for cache, and/or lots of disks. It'd be better if they were connected to several controllers...
Quoted 9 lines
A large RAM cache will help, but as I already pointed out, the increases in returns for increasing cache size diminish rapidly past a certain point. Most quality drives these days have a 32MB cache, or 128M for a 4 drive array. Add the Linux cache on top of that, and it should be sufficient for most purposes. Remember, random seeks implies small data extents. Lots of disks will bring the biggest benefit, and disks are cheap. Multiple controllers really are not necessary, especially if the controller and drives support NCQ , but having multiple controllers certainly doesn't hurt.
Yet I've heard NCQ makes some things worse. Some raid tweaking pages tell you to try disabling NCQ.
I've actually been thinking about trying md-cache with an SSD on top of my new raid and see how that works long term. But I can't really think of a good benchmark that actually imitates my particular use cases well enough to show me if it'd help me at all ::)
I doubt my punny little 30G OCZ Vertex would really help all that much any how.
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Thomas Fjellstrom [email protected]
Quoted 3 lines · Thomas Fjellstrom
..
Yet I've heard NCQ makes some things worse. Some raid tweaking pages tell you to try disabling NCQ.
Not so relevant here.
This recommendation to disable drive NCQ is generally due tot he fact that NCQ implementations by various manufacturers tend to differ in behaviour, and, in some cases , is just plain "broken".
Signature · 1 line
Regards, Maurice
Quoted 11 lines · Maurice Hilarius
Quoted 5 lines · Thomas Fjellstrom
..
Yet I've heard NCQ makes some things worse. Some raid tweaking pages tell you to try disabling NCQ.Not so relevant here.
This recommendation to disable drive NCQ is generally due tot he fact that NCQ implementations by various manufacturers tend
to differ in behaviour, and, in some cases , is just plain "broken".
Not trying to start a flamewar on whos hardware is better, but I'd love to get clarification on that last statement. I've always been under the impression Intel gets their NCQ right (works for me). All the reviews I've read on the current set of AMD southbridges indicate they don't but I've no personal experience there. No idea about nvidia or the cheap add-in sata cards like Silicon Image.
Wil
--
To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 15 lines · Wil Reichert
Quoted 7 lines · Maurice Hilarius
Quoted 5 lines · Thomas Fjellstrom
..
Yet I've heard NCQ makes some things worse. Some raid tweaking pages tell you to try disabling NCQ.Quoted 5 lines
Not so relevant here.
This recommendation to disable drive NCQ is generally due tot he fact that NCQ implementations by various manufacturers tend
to differ in behaviour, and, in some cases , is just plain "broken".
Quoted 9 lines
Not trying to start a flamewar on whos hardware is better, but I'd love to get clarification on that last statement. I've always been under the impression Intel gets their NCQ right (works for me). All the reviews I've read on the current set of AMD southbridges indicate they don't but I've no personal experience there. No idea about nvidia or the cheap add-in sata cards like Silicon Image.
Wil
From speaking with hardware RAID card manufacturers support engineers over the past few years, there have been several instances where NCQ implementations were incomplete, or faulty, or both.
This seems to have improved a lot over the last year or two, so I think it is likely now "OK".
However, from the earlier history I think there is a tendency for people now to assume it is "broken", and to turn it off.
Signature · 1 line
Regards, Maurice
Quoted 55 lines · Thomas Fjellstrom
Quoted 24 lines · Leslie Rhorer
Quoted 8 lines
Unfortunately it doesn't seem to be. Take a well-considered drive such as the WD RE3; it's spec for average latency is 4.2ms. However does it include the rotational latency (the time the head takes to reach the sector once it's on the track)? I bet it doesn't. Taking it to be only the average seek time, this drive is still among the fastest. For a 7200rpm drive this latency is just 4.2ms, so we'd have for this fast drive an average total latency of 8.4ms.
That's an average. For a random seek to exceed that, it's going to have to span many cylinders. Give the container size of a modern cylinder, that's a pretty big jump. Single applications will tend to have their data lumped somewhat together on the drive.
Quoted 8 lines
with
No, random I/O is the most common case for busy servers, when there are lots of processes doing uncorrelated reads and writes. Even if a
Quoted 8 lines
Yes, exactly. By definition, such a scenario represents a multithreaded set of seeks, and as we already established, multithreaded seeks are vastly more efficient than serial random seeks. The 400 seeks per second number for 4 drives applies. I don't know the details of the Linux schedulers, but most schedulers employ some variation of an elevator seek to maximize seek efficiency. The brings the average latency way down and brings the seek frequency way up.
Ah, I never really understood how adding more random load could increase performance. Now I get it :)
Quoted 5 lines
Quoted 5 lines
single application does sequential access the head will likely have moved between them. The only solution is to have lots of ram for cache, and/or lots of disks. It'd be better if they were connected to several controllers...
Quoted 10 lines
A large RAM cache will help, but as I already pointed out, the increases in returns for increasing cache size diminish rapidly past a certain point. Most quality drives these days have a 32MB cache, or 128M for a 4 drive array. Add the Linux cache on top of that, and it should be sufficient for most purposes. Remember, random seeks implies small data extents. Lots of disks will bring the biggest benefit, and disks are cheap. Multiple controllers really are not necessary, especially if the controller and drives support NCQ , but having multiple controllers certainly doesn't hurt.
Quoted 6 lines
Yet I've heard NCQ makes some things worse. Some raid tweaking pages tell you to try disabling NCQ.
I've actually been thinking about trying md-cache with an SSD on top of my new raid and see how that works long term. But I can't really think of a good benchmark that actually imitates my particular use cases well enough to show me if it'd help me at all ::)
I doubt my punny little 30G OCZ Vertex would really help all that much any how.
For ext[34] you might want to put the journal on SSD, if you are doing any significant write that will help. Mounting data=journal may also help write, supposedly the write will complete when the data hits the journal, and not wait for the platter.
Signature · 7 lines
Bill Davidsen <[email protected]> Even purely technical things can appear to be magic, if the documentation is obscure enough. For example, PulseAudio is configured by dancing naked around a fire at midnight, shaking a rattle with one hand and a LISP manual with the other, while reciting the GNU manifesto in hexadecimal. The documentation fails to note that you must circle the fire counter-clockwise in the southern hemisphere.
Quoted 11 lines · Leslie Rhorer
Quoted 10 lines
Quoted 2 lines
John is perfectly correct, although of course a 10ms seek is a
fairly slow one.Unfortunately it doesn't seem to be. Take a well-considered drive such as the WD RE3; it's spec for average latency is 4.2ms. However does it include the rotational latency (the time the head takes to reach the sector once it's on the track)? I bet it doesn't. Taking it to be only the average seek time, this drive is still among the fastest. For a 7200rpm drive this latency is just 4.2ms, so we'd have for this fast drive an average total latency of 8.4ms.
Quoted 4 lines
That's an average. For a random seek to exceed that, it's going to have to span many cylinders. Give the container size of a modern cylinder, that's a pretty big jump. Single applications will tend to have their data lumped somewhat together on the drive.
Only at the start, which is usualy when people benchmark. But after a while filesystem fragment. Files get distributed all over the disk, files themself get spread out as they grow. And suddenly an FS that was fine month ago is too slow.
The worst you can do to an FS is run mldonkey/rtorrent on it with lots of downloads. I've managed to get an ext2 to the point where copying a file from the FS to another disk only managed <100kiB/s.
In conclusion: Seek times can not be ignored and they should be avoided.
MfG
Goswin
Quoted 4 lines
Quoted 1 line
That's an average. For a random seek to exceed that, it's going to have
to
Quoted 1 line
span many cylinders. Give the container size of a modern cylinder,
that's a
Quoted 8 lines
Quoted 1 line
pretty big jump. Single applications will tend to have their data
lumped
Quoted 1 line
somewhat together on the drive.
Only at the start, which is usualy when people benchmark. But after a while filesystem fragment. Files get distributed all over the disk, files themself get spread out as they grow. And suddenly an FS that was fine month ago is too slow.
There can be a lot of application dependent variation, of course, but even with a fragmented disk, many applications still tend to wind up with their files clustered together on the disk. If the application writes each file once and never updates it, creating many more files as time goes by, then indeed the database will grow ever more scattered. Random access files, of course, may wind up scattered all over the drive, even if there is only one file use by the app. If the application tends to update the majority of its files on a regular basis, however, then the file updates tend to fall in little pools across the disk, rather than being scattered in a perfectly random fashion. One's mileage will definitely vary.
Quoted 7 lines · John Robinson
Clearly when you're doing this over 4 drives you can have ~400 seeks/second but that's still limiting you to ~400 reads/second for smallish block sizes.
Cheers,
John.
Note that that only holds true for writes or multithreaded reads. Reading from a single thread will randomly pick one drive (depending on where it wants to read), wait for it to seek, read one block of data and repeat. So you get the speed of a single drive no matter how many drives there are in the raid.
Single thread seek times are only improved with raid1 (raid10) because there linux can choose the drive with shorter seek time. And that only saves a tiny amount.
MfG
Goswin
Quoted 10 lines · Goswin von Brederlow
Quoted 3 lines · John Robinson
Clearly when you're doing this over 4 drives you can have ~400 seeks/second but that's still limiting you to ~400 reads/second for smallish block sizes.
Note that that only holds true for writes or multithreaded reads. Reading from a single thread will randomly pick one drive (depending on where it wants to read), wait for it to seek, read one block of data and repeat. So you get the speed of a single drive no matter how many drives there are in the raid.
Sure, that's why I said "can", but I thought iozone was multi-threaded. Maybe it needs an option specified, in which case using more threads than there are discs would be a good idea.
Cheers,
John.