per-backing dev unplugging #2
| Hi, | | Final version, unless something stupid pops up. Changes: | | - Adapt to 2.6.4-mm1 | - Cleaned up the dm bits, much nicer with the lockless unplugging | (thanks Joe) | - md and loop unplugging, stacked devices should unplug their targets. | Otherwise they'll end up waiting for the unplug timer, which sucks. | - XFS fixed up, I hope. XFS folks still encouraged to look at this, | looks better this time around though (and works, I tested). | - blk_run_* inlined in blkdev.h | | Against 2.6.4-mm1 (note you need other attached patch to boot it).
I got a chance to try this.
It makes a huge improvement.
Prior to the last per-cpu patch, I was getting about 75000 to 80000 IOPS at 100% cpu usage.
With the per-cpu patch, that went up to 110000 IOPS at 100% CPU.
With this patch, I'm seeing 200000 IOPS at about 65% CPU usage.
So it makes a tremendous improvement in I/O scalability, dramatically improving performance in small size I/O, high I/O count workloads.
My tests were on an 8 CPU x 1300 MHz Altix with 64 disks.
jeremy
Quoted 10 lines · Jeremy Higdon
Quoted 8 lines · Nick Piggin
Quoted 4 lines · Jeremy Higdon
My tests were on an 8 CPU x 1300 MHz Altix with 64 disks.
Quoted 10 lines
Quoted 4 lines
Nice - so if you had enough IO capacity to saturate the CPUs it might come close to a 4x improvement - and this sounds like one of your baby systems?
Baby by cpu count, mid size by I/O capability. Extrapolations of this sort are fraught with peril. However, it is conceivable that we would end up with 4X the IOPS.
Well, it is more than 2x! What does a profile look like after the patch, I wonder?
Quoted 7 lines
Quoted 2 lines
I wonder why nobody's complained about this before?
Well, some of us have, but probably not very loudly. I had naively believed that the global unplug was gone in 2.6.
I wasn't talking about SGI in particular, I just remember people being very worried about making sure we can support *thousands* of queues quite a while back.
Anyway, it's done now. And Jens' patch turned out to be pretty simple and probably makes the code cleaner, anyway.
Quoted 19 lines · Jeremy Higdon
| Hi, | | Final version, unless something stupid pops up. Changes: | | - Adapt to 2.6.4-mm1 | - Cleaned up the dm bits, much nicer with the lockless unplugging | (thanks Joe) | - md and loop unplugging, stacked devices should unplug their targets. | Otherwise they'll end up waiting for the unplug timer, which sucks. | - XFS fixed up, I hope. XFS folks still encouraged to look at this, | looks better this time around though (and works, I tested). | - blk_run_* inlined in blkdev.h | | Against 2.6.4-mm1 (note you need other attached patch to boot it).I got a chance to try this.
It makes a huge improvement.
Prior to the last per-cpu patch, I was getting about 75000 to 80000 IOPS at 100% cpu usage.
Quoted 10 lines
With the per-cpu patch, that went up to 110000 IOPS at 100% CPU.
With this patch, I'm seeing 200000 IOPS at about 65% CPU usage.
So it makes a tremendous improvement in I/O scalability, dramatically improving performance in small size I/O, high I/O count workloads.
My tests were on an 8 CPU x 1300 MHz Altix with 64 disks.
Nice - so if you had enough IO capacity to saturate the CPUs it might come close to a 4x improvement - and this sounds like one of your baby systems?
I wonder why nobody's complained about this before?
Quoted 4 lines · Jeremy Higdon
Quoted 1 line
I wonder why nobody's complained about this before?
Well, some of us have, but probably not very loudly. I had naively believed that the global unplug was gone in 2.6.
Ditching plugging at the water cooler doesn't count as complain, it needs to get out in the open :-). When Intel posted their patch and numbers, that was the first I heard of it.
Signature · 1 line
Jens Axboe
Quoted 22 lines · Jeremy Higdon
| Hi, | | Final version, unless something stupid pops up. Changes: | | - Adapt to 2.6.4-mm1 | - Cleaned up the dm bits, much nicer with the lockless unplugging | (thanks Joe) | - md and loop unplugging, stacked devices should unplug their targets. | Otherwise they'll end up waiting for the unplug timer, which sucks. | - XFS fixed up, I hope. XFS folks still encouraged to look at this, | looks better this time around though (and works, I tested). | - blk_run_* inlined in blkdev.h | | Against 2.6.4-mm1 (note you need other attached patch to boot it).I got a chance to try this.
It makes a huge improvement.
Prior to the last per-cpu patch, I was getting about 75000 to 80000 IOPS at 100% cpu usage.
Quoted 8 lines
With the per-cpu patch, that went up to 110000 IOPS at 100% CPU.
With this patch, I'm seeing 200000 IOPS at about 65% CPU usage.
So it makes a tremendous improvement in I/O scalability, dramatically improving performance in small size I/O, high I/O count workloads.
My tests were on an 8 CPU x 1300 MHz Altix with 64 disks.
That's awesome results, thanks for sharing! And I'm happy to see it's so worth it, even with the various minor issues we've had with the patch (hope they are all resolved now).
Signature · 1 line
Jens Axboe
Quoted 11 lines · Nick Piggin
Quoted 3 lines · Jeremy Higdon
My tests were on an 8 CPU x 1300 MHz Altix with 64 disks.
Nice - so if you had enough IO capacity to saturate the CPUs it might come close to a 4x improvement - and this sounds like one of your baby systems?
Baby by cpu count, mid size by I/O capability. Extrapolations of this sort are fraught with peril. However, it is conceivable that we would end up with 4X the IOPS.
Quoted 1 line
I wonder why nobody's complained about this before?
Well, some of us have, but probably not very loudly. I had naively believed that the global unplug was gone in 2.6.
jeremy
Quoted 6 lines · Jeremy Higdon
Prior to the last per-cpu patch, I was getting about 75000 to 80000 IOPS at 100% cpu usage.
With the per-cpu patch, that went up to 110000 IOPS at 100% CPU.
With this patch, I'm seeing 200000 IOPS at about 65% CPU usage.
Well, that settles it, I'm going to have to start using your machine instead. :)
Jesse
Quoted 8 lines · Jens Axboe
Quoted 4 lines · Jeremy Higdon
Quoted 1 line
I wonder why nobody's complained about this before?
Well, some of us have, but probably not very loudly. I had naively believed that the global unplug was gone in 2.6.
Ditching plugging at the water cooler doesn't count as complain, it needs to get out in the open :-). When Intel posted their patch and
I agree. It's one of those little assumptions that one should not make, but one ends up making anyway. :-) Jbarnes did a prototype of per-queue plug/unplug a couple of years ago for LK2.4, but we didn't see much benefit. Obviously, we gave up too soon.
Quoted 1 line
numbers, that was the first I heard of it.
One problem we had is that the profiling tools on IA64 are very poor, so we couldn't really tell where the problem was.
jeremy