submit_bh leaves interrupts on upon return
Any reason why submit_bh should turn on interrupts after being called by a process with ints off in 2.4.20? I see it's possible to sleep during elevatoring, but why does it need to leave interrupts on if the calling state was with ints off.
I'm back BTW. Hope no one missed me. I can be reached at the email address above.
:-)
Jeff
Quoted 5 lines · Jeff V. Merkey
Any reason why
submit_bhshould turn on interrupts after being called by a process with ints off in 2.4.20? I see it's possible to sleep during elevatoring, but why does it need to leave interrupts on if the calling state was with ints off.
It's illegal to call it with interrupts off, so... __make_request() doesn't save interrupt state, so you will always leave with interrupts enabled.
Signature · 1 line
Jens Axboe
Quoted 13 lines · Jens Axboe
Quoted 3 lines · Jeff V. Merkey
Any reason why
submit_bhshould turn on interrupts after being called by a process with ints off in 2.4.20? I see it's possible to sleep during elevatoring, but why does it need to leave interrupts on if the calling state was with ints off.It's illegal to call it with interrupts off, so...
__make_request()doesn't save interrupt state, so you will always leave with interrupts enabled.
Jens
I noticed in the code it does not check for this when make_request is called, so I altered the calling sequence to call with ints on. I don't see much of a performance difference either way, so calling with ints on was easy to instrument. I am posting about 80,000+ buffer heads per second in with what I am doing, so filling out buffer_head structures and submitting them ad hoc was causing some interrupt windows where the chains were getting corrupted. I altered the calling sequence and added atomic counters so I can submit and call with ints on to avoid the corruption. One of the troublesome aspects of the manner in which make_request is implemented in always needing a context of a thread for sleeping to submit asynch I/O limits the ability to gang schedule large disk I/O from the b_end_io callback. Would make performance a lot more spectacular if it worked this way, but I am seeing good enough performance with it left the way it is. 3Ware's 66Mhz ATA adapter in this implementation is reaching almost 400 MB/S throughput on 2.4.20. I have not tried this on 2.6 yet, but will later this month.
Also, I ported the kernel debugger from MANOS to Linux and made a lot of significant enhancements and www.devicelogics.com is distributing it from their website. If anyone wants a more pleasant debugger for the kernel to work with, they are allowing downloads of the modules with a patch. Not free (what is in this world) but very nice to work with.
Thanks for the response. Sorry I was out of touch for about a year. I was going through a very nasty divorce with my wife of 24 years and I discovered when something like that is happening in your life, you don't have much attention for much else.
Jeff
Quoted 20 lines · Jeff V. Merkey
Quoted 10 lines · Jens Axboe
Quoted 6 lines · Jeff V. Merkey
Any reason why
submit_bhshould turn on interrupts after being called by a process with ints off in 2.4.20? I see it's possible to sleep during elevatoring, but why does it need to leave interrupts on if the calling state was with ints off.Quoted 6 lines
It's illegal to call it with interrupts off, so...
__make_request()doesn't save interrupt state, so you will always leave with interrupts enabled.Jens
Quoted 16 lines
I noticed in the code it does not check for this when
make_requestis called, so I altered the calling sequence to call with ints on. I don't see much of a performance difference either way, so calling with ints on was easy to instrument. I am posting about 80,000+ buffer heads per second in with what I am doing, so filling outbuffer_headstructures and submitting them ad hoc was causing some interrupt windows where the chains were getting corrupted. I altered the calling sequence and added atomic counters so I can submit and call with ints on to avoid the corruption. One of the troublesome aspects of the manner in whichmake_requestis implemented in always needing a context of a thread for sleeping to submit asynch I/O limits the ability to gang schedule large disk I/O from theb_end_iocallback. Would make performance a lot more spectacular if it worked this way, but I am seeing good enough performance with it left the way it is. 3Ware's 66Mhz ATA adapter in this implementation is reaching almost 400 MB/S throughput on 2.4.20. I have not tried this on 2.6 yet, but will later this month.
Submitting large numbers of buffer_heads from b_end_io is _nasty_, 2.4 io scheduler runtime isn't exactly world champion and you are doing this at hard irq time. Not a good idea. Definitely not the true path to performance, unless you don't care about anything else in the system.
At least in 2.6 you have a much faster io scheduler and the additionally large bio, so you wont spend nearly as much time there if you are clever. You still need process context, though, that hasn't changed.
Signature · 1 line
Jens Axboe
Quoted 11 lines · Jens Axboe
Submitting large numbers of
buffer_headsfromb_end_iois _nasty_, 2.4 io scheduler runtime isn't exactly world champion and you are doing this at hard irq time. Not a good idea. Definitely not the true path to performance, unless you don't care about anything else in the system.At least in 2.6 you have a much faster io scheduler and the additionally large bio, so you wont spend nearly as much time there if you are clever. You still need process context, though, that hasn't changed.
Sounds like I need to move to 2.6. I noticed the elevator is coalescing quite well, and since I am posting mostly continguous runs of sectors, what ends up at the adapter level would probably not change much much between 2.4 and 2.6 since I am maxing out the driver request queues as it is (255 pending requests of 32 scatter/gather elements of 256 sector runs). 2.6 might help but I suspect it will only help alleviate the submission overhead, and not make much difference on performance since the 3Ware card does have an upward limit on outstanding I/O requests.
Jeff
Quoted 13 lines · Jeff V. Merkey
Quoted 11 lines · Jens Axboe
Submitting large numbers of
buffer_headsfromb_end_iois _nasty_, 2.4 io scheduler runtime isn't exactly world champion and you are doing this at hard irq time. Not a good idea. Definitely not the true path to performance, unless you don't care about anything else in the system.At least in 2.6 you have a much faster io scheduler and the additionally large bio, so you wont spend nearly as much time there if you are clever. You still need process context, though, that hasn't changed.
Quoted 8 lines
Sounds like I need to move to 2.6. I noticed the elevator is coalescing quite well, and since I am posting mostly continguous runs of sectors, what ends up at the adapter level would probably not change much much between 2.4 and 2.6 since I am maxing out the driver request queues as it is (255 pending requests of 32 scatter/gather elements of 256 sector runs). 2.6 might help but I suspect it will only help alleviate the submission overhead, and not make much difference on performance since the 3Ware card does have an upward limit on outstanding I/O requests.
That's correct, it just helps you diminish the submission overhead by pushing down 256 sector entities in one go. So as long as you're io bound it won't give you better io performance, of course. If you are doing 400MiB/sec it should help you out, though.
Signature · 1 line
Jens Axboe
Quoted 11 lines · Jens Axboe
Quoted 9 lines · Jeff V. Merkey
Sounds like I need to move to 2.6. I noticed the elevator is coalescing quite well, and since I am posting mostly continguous runs of sectors, what ends up at the adapter level would probably not change much much between 2.4 and 2.6 since I am maxing out the driver request queues as it is (255 pending requests of 32 scatter/gather elements of 256 sector runs). 2.6 might help but I suspect it will only help alleviate the submission overhead, and not make much difference on performance since the 3Ware card does have an upward limit on outstanding I/O requests.
Quoted 4 lines
That's correct, it just helps you diminish the submission overhead by pushing down 256 sector entities in one go. So as long as you're io bound it won't give you better io performance, of course. If you are doing 400MiB/sec it should help you out, though.
Well, if Jeff does almost exclusively contiguous stuff and submits them in order, then the coalescing will make sure that even on 2.4.x the queues don't get too long, and he probably won't see the pathological cases.
Linus
Quoted 15 lines · Linus Torvalds
Quoted 11 lines · Jens Axboe
Quoted 9 lines · Jeff V. Merkey
Sounds like I need to move to 2.6. I noticed the elevator is coalescing quite well, and since I am posting mostly continguous runs of sectors, what ends up at the adapter level would probably not change much much between 2.4 and 2.6 since I am maxing out the driver request queues as it is (255 pending requests of 32 scatter/gather elements of 256 sector runs). 2.6 might help but I suspect it will only help alleviate the submission overhead, and not make much difference on performance since the 3Ware card does have an upward limit on outstanding I/O requests.
Quoted 8 lines
Quoted 4 lines
That's correct, it just helps you diminish the submission overhead by pushing down 256 sector entities in one go. So as long as you're io bound it won't give you better io performance, of course. If you are doing 400MiB/sec it should help you out, though.
Well, if Jeff does almost exclusively contiguous stuff and submits them in order, then the coalescing will make sure that even on 2.4.x the queues don't get too long, and he probably won't see the pathological cases.
At 255 pending requests (see above), it will get somewhat bad. So even if 2.4 copes, it will spend more sys time than 2.6 for this work load for sure. The queues will get long even for mainly contig submissions, but of course the time spent in the io scheduler will not be too bad since it's scanned backwards. And call frequency will be 32 times as big...
Signature · 1 line
Jens Axboe
Quoted 17 lines · Linus Torvalds
Quoted 12 lines · Jens Axboe
Quoted 3 lines · Jeff V. Merkey
Sounds like I need to move to 2.6. I noticed the elevator is coalescing quite well, and since I am posting mostly continguous runs of sectors, what ends up at the adapter level would probably not change much much between 2.4 and 2.6 since I am maxing out the driver request queues as it is (255 pending requests of 32 scatter/gather elements of 256 sector runs). 2.6 might help but I suspect it will only help alleviate the submission overhead, and not make much difference on performance since the 3Ware card does have an upward limit on outstanding I/O requests.
That's correct, it just helps you diminish the submission overhead by pushing down 256 sector entities in one go. So as long as you're io bound it won't give you better io performance, of course. If you are doing 400MiB/sec it should help you out, though.
Quoted 7 lines
Well, if Jeff does almost exclusively contiguous stuff and submits them in order, then the coalescing will make sure that even on 2.4.x the queues don't get too long, and he probably won't see the pathological cases.
Linus
-
Linus,
This seems to be the case.
Jeff
Quoted 7 lines
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 20 lines · Jens Axboe
Quoted 17 lines · Jeff V. Merkey
Quoted 13 lines · Jens Axboe
Submitting large numbers of
buffer_headsfromb_end_iois _nasty_, 2.4 io scheduler runtime isn't exactly world champion and you are doing this at hard irq time. Not a good idea. Definitely not the true path to performance, unless you don't care about anything else in the system.At least in 2.6 you have a much faster io scheduler and the additionally large bio, so you wont spend nearly as much time there if you are clever. You still need process context, though, that hasn't changed.
Quoted 11 lines
Quoted 3 lines
Sounds like I need to move to 2.6. I noticed the elevator is coalescing quite well, and since I am posting mostly continguous runs of sectors, what ends up at the adapter level would probably not change much much between 2.4 and 2.6 since I am maxing out the driver request queues as it is (255 pending requests of 32 scatter/gather elements of 256 sector runs). 2.6 might help but I suspect it will only help alleviate the submission overhead, and not make much difference on performance since the 3Ware card does have an upward limit on outstanding I/O requests.
That's correct, it just helps you diminish the submission overhead by pushing down 256 sector entities in one go. So as long as you're io bound it won't give you better io performance, of course. If you are doing 400MiB/sec it should help you out, though.
I'll give it a try n 2.6.
:-)
Jeff