Tiobench results LOWER with more threads
OK, just joined the list and rad the faq, and something caught my eye. Tiobench results are apparently supposed to INCREASE when there are more threads.
What I have is Tiobench results decreasing with more threads. This is my setup:
Celeron 633
196 MB PC 133
Adaptec 29160 SCSI controller (PCI)
5 IBM Ultrastar 18XP (18gig, SCSI-3) disks hanging off the Adaptec controller Red Hat 7.3 Linux (2.4.18-3)
Experimenting with different RAID configurations, I have found that I can not get more than 32 MB/s from this array with 4 disks, one spare. I have actually found out that the disks set the SCSI bus at 40 MB/s (since the disks are old) and that in RAID 0 it scales well, the speed doubles for two disks, and then the third disk brings in a little more performance, and then things topp off at 32 MB/s with four disks. Adding the fifth disk gains no extra performance.
Apparently VIA chipsets have problems with PCI bursting, so that is why I can't see the full 40 MB/s. That's fine.
But, my tests with tiobench also show that performance decreases as extra threads are added. I am testing with a file of 800 MB (approx. 4x size of RAM, to get meaningful results) and the decrease with threads, while READING only, is consistent in all RAID levels. The write performance will sometimes increase, sometimes stay the same.
WHY is this happening? Is it something I have not set up right , or what? I am not so much interested in getting more speed out of these old disks, they are gonna be replaced soon anyway, but I REALLY want to know WHY is this??
Can anyone give me any pointers?
Thanks,
Vlad.
Quoted 3 lines · Vladimir Milovanovic
OK, just joined the list and rad the faq, and something caught my eye. Tiobench results are apparently supposed to INCREASE when there are more threads.
No, what gave you that idea?
It is so much easier for the kernel to handle one sequential stream of I/O, instead of many streams.
If you have more than one stream, you need to seek. Seeking is bad. One sequential I/O is almost always (with the notable exception of RAID-1 reads) faster in total sustained throughput, and always (as in really always) faster in per-thread sustained throughput.
Quoted 3 lines
What I have is Tiobench results decreasing with more threads. This is my setup:
Good, tiobench works :)
Quoted 16 lines
Celeron 633
196 MB PC 133
Adaptec 29160 SCSI controller (PCI)
5 IBM Ultrastar 18XP (18gig, SCSI-3) disks hanging off the Adaptec controller
Red Hat 7.3 Linux (2.4.18-3)Experimenting with different RAID configurations, I have found that I can not get more than 32 MB/s from this array with 4 disks, one spare. I have actually found out that the disks set the SCSI bus at 40 MB/s (since the disks are old) and that in RAID 0 it scales well, the speed doubles for two disks, and then the third disk brings in a little more performance, and then things topp off at 32 MB/s with four disks. Adding the fifth disk gains no extra performance.
Quoted 2 lines
Apparently VIA chipsets have problems with PCI bursting, so that is why I can't see the full 40 MB/s. That's fine.
There's some SCSI overhead as well. And probably you have some RAM bandwidth limitation also - although that is probably not very important at the speed you're seeing. But it all adds up.
Quoted 10 lines
But, my tests with tiobench also show that performance decreases as extra threads are added. I am testing with a file of 800 MB (approx. 4x size of RAM, to get meaningful results) and the decrease with threads, while READING only, is consistent in all RAID levels. The write performance will sometimes increase, sometimes stay the same.
WHY is this happening? Is it something I have not set up right , or what? I am not so much interested in getting more speed out of these old disks, they are gonna be replaced soon anyway, but I REALLY want to know WHY is this??
If you used SDRAM instead of disks with actual spindles, you should see almost the same total sustained I/O going from 1 to some handfull or two of threads.
But you use real disks. Those have heads that need to move when seeking. The average seek time for your disks is probably around 7 ms. Let's say you can do 10 MB/sec sequential reading from a disk (low number probably), then *one* seek costs you the equivalent of 72 kB (low number again) of transfer that you do not get while seeking.
The kernel will have to do a lot of seeks to satisfy your multiple readers. That's many times 72kB. And that's why you're losing performance using more readers (or writers).
The more you add, the more you lose :)
Signature · 6 lines
................................................................ : [email protected] : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............:
Quoted 7 lines · Jakob Oestergaard
Quoted 3 lines · Vladimir Milovanovic
OK, just joined the list and rad the faq, and something caught my eye. Tiobench results are apparently supposed to INCREASE when there are more threads.
No, what gave you that idea?
Quoted 7 lines
It is so much easier for the kernel to handle one sequential stream of I/O, instead of many streams.
If you have more than one stream, you need to seek. Seeking is bad. One sequential I/O is almost always (with the notable exception of RAID-1 reads) faster in total sustained throughput, and always (as in really always) faster in per-thread sustained throughput.
Err, so are you saying that a single sequential I/O is slower on RAID 1 when compared with a single disk? That doesn't make a lot of sense. There -are- instances where parallel I/O is required. In these cases, any RAID-1 should be much faster than a single disk, as should RAID 10. I'm not sure that RAID 5 should give a similar benefit, but given the cost of disks, I don't care about RAID 5.
Quoted 7 lines
Quoted 7 lines
Celeron 633
196 MB PC 133
Adaptec 29160 SCSI controller (PCI)
5 IBM Ultrastar 18XP (18gig, SCSI-3) disks hanging off the Adaptec controller
Red Hat 7.3 Linux (2.4.18-3)
Quoted 7 lines
Quoted 7 lines
Experimenting with different RAID configurations, I have found that I can not get more than 32 MB/s from this array with 4 disks, one spare. I have actually found out that the disks set the SCSI bus at 40 MB/s (since the disks are old) and that in RAID 0 it scales well, the speed doubles for two disks, and then the third disk brings in a little more performance, and then things topp off at 32 MB/s with four disks. Adding the fifth disk gains no extra performance.
Interesting. How fast are you getting single-threaded reads from a single disk? And what are the real specs on the drives (RPM, cache, SCSI version support (SCSI 3 is a family, not a version)).
Quoted 6 lines
Quoted 2 lines
Apparently VIA chipsets have problems with PCI bursting, so that is why I can't see the full 40 MB/s. That's fine.
There's some SCSI overhead as well. And probably you have some RAM bandwidth limitation also - although that is probably not very important at the speed you're seeing. But it all adds up.
Yeah, if this is a 40MB/sec bus, you shouldn't expect to get more than 35MB/sec out of the bus. It'd be slick to get more, but overhead eats the rest.
Greg
With regards to your message at 08:31 PM 10/17/02, Gregory Leblanc. Where you stated:
Quoted 6 lines
Err, so are you saying that a single sequential I/O is slower on RAID 1 when compared with a single disk? That doesn't make a lot of sense. There -are- instances where parallel I/O is required. In these cases, any RAID-1 should be much faster than a single disk, as should RAID 10. I'm not sure that RAID 5 should give a similar benefit, but given the cost of disks, I don't care about RAID 5.
Of course RAID1 should be slower.
We are doing more WORK!
We have to generate two writes, and manage them and the RAID.
With a single disk we are doing much less, and there is _much_ less overhead.
In an ideal world RAID10 will gain us back the loss because of the RAID0 element which DOES gain us some performance back. But even then the CPU load will be much more, as we are still writing more data.
With our best regards,
Maurice W. Hilarius Telephone: 01-780-456-9771
Hard Data Ltd. FAX: 01-780-456-9772
11060 - 166 Avenue mailto:[email protected]
Edmonton, AB, Canada http://www.harddata.com/
T5X 1Y3
Ask me about NAS and near-line storage
Quoted 6 lines · Gregory Leblanc
Interesting. How fast are you getting single-threaded reads from a single disk? And what are the real specs on the drives (RPM, cache, SCSI version support (SCSI 3 is a family, not a version)).
I am getting about 12 MB/s for sequential reads single threaded per disk. The disks are 18GB, 7200 rpm, ultra fast/wide scsi disks. I am really not sure about the cache.
So what is the consensus now? Let's just stick to RAID 0 - should I see more when going multithreaded or should I not be worried that the transfer rates are dropping off? Note that the rates drop off even if I am only running a 2 disk RAID 0 i.e. my bus is not being maxed out then.
Cheers, Vlad.
Quoted 16 lines · Gregory Leblanc
Quoted 7 lines · Jakob Oestergaard
Quoted 3 lines · Vladimir Milovanovic
OK, just joined the list and rad the faq, and something caught my eye. Tiobench results are apparently supposed to INCREASE when there are more threads.
No, what gave you that idea?
Quoted 7 lines
It is so much easier for the kernel to handle one sequential stream of I/O, instead of many streams.
If you have more than one stream, you need to seek. Seeking is bad. One sequential I/O is almost always (with the notable exception of RAID-1 reads) faster in total sustained throughput, and always (as in really always) faster in per-thread sustained throughput.
Quoted 6 lines
Err, so are you saying that a single sequential I/O is slower on RAID 1 when compared with a single disk? That doesn't make a lot of sense. There -are- instances where parallel I/O is required. In these cases, any RAID-1 should be much faster than a single disk, as should RAID 10. I'm not sure that RAID 5 should give a similar benefit, but given the cost of disks, I don't care about RAID 5.
The exception I made was:
"One sequential I/O is almost always (with the notable exception of RAID-1 reads) faster in total sustained throughput,..."
What I wanted to say was: One sequential reader is usually faster than N readers are in total. Except on an N-disk RAID-1, where up to N readers will be faster in total than one reader.
An N-disk RAID-1 will scale with up to N readers.
There's some fuzz on these measurements, because a sequential read on a filesystem is not a sequential read on the disk (because of fs metadata), so therefore often a single threaded read on an N-disk RAID-1 will be faster than on a single disk - and maybe RAID-1 will not scale up to N readers, but only N-1, but that all depends on the fs - and let's not go too deep into that for now :)
Signature · 10 lines
................................................................ : [email protected] : And I see the elder races, : :.........................: putrid forms of man : : Jakob Østergaard : See him rise and claim the earth, : : OZ9ABN : his downfall is at hand. : :.........................:............{Konkhra}...............: - To unsubscribe from this list: send the line "unsubscribe linux-raid" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html