Crash in elevator_dispatch_fn() (e.g. deadline_dispatch()) when changing elevators.
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 19 lines · Tejun Heo
Hello, Frank.
Quoted 4 lines · Frank Mayhar
After investigation, it's clear that the elevator switch code is trying to quiesce the request queue and sets the bypass flag. Unfortunately,
scsi_mode_sense()and friends callblk_execute_rq()directly, which pays no attention to said flag.Ouch.
Quoted 2 lines
In fact, the bypass flag, as far as I can tell, is only checked in the
blk_cgroup.c, inblkg_lookup()andblkg_lookup_create().Hmmm? IIRC the main place is in
__get_request(). If the queue is bypassing, the request doesn't getREQ_ELVPRIVwhich basically indicates that the request shouldn't go through elevator. I don't think that part is broken even forscsi_execute_rq()- it usesblk_get_request()which should handle it properly.
Quoted 3 lines
Quoted 1 line
So my question is: Is this a simple oversight? Should
blk_execute_rq()Yeah, it's a bug. It's not supposed to crash like that.
Clearly. :-)
Quoted 17 lines
Quoted 4 lines
care about the bypass flag? Should it perhaps hold off the I/O until the bypass flag is cleared? Since at that level it has nothing to do with cgroups I kinda don't think so but I'm still trying to get my head around how all this stuff goes together.
The cgroup part isn't really relevant. That's just because cgroup also uses bypassing to quiesce the queue when it's changing blkcg policies. The thing relevant to the elveator is the check in
__get_request().Hmm.... the culprit is that we don't have any check in the dispatch path. It doesn't really matter who's calling
blk_peek_request(), that function is called as part of all request dispatches and should never crash. I think the only reason we haven't noticed yet is because calling evelatordispatch_fndoesn't crash in most cases even if the elevator isn't in fully working condition.
Quoted 3 lines
Probably what we need is replacing
blk_queue_dying()withblk_queue_bypass()test in__elv_next_request()before invoking the elevator dispatch function.
Replacing? Or adding to? Is BYPASS always set when DYING is set? (My guess is not but I haven't done an exhaustive analysis.) So the relevant code snippet in __elv_next_request() would be:
if (unlikely(blk_queue_dying(q)) || unlikely(blk_queue_bypass(q)) || !q->elevator->type->ops.elevator_dispatch_fn(q, 0)) return NULL;
Quoted 2 lines
Can you please reply w/ Jens and lkml cc'd with the original description and my response? No reason to keep this private.
Sure, doing so. I just wanted to make sure my understanding was correct before bringing others into it.
Quoted 1 line
Thanks a lot for the report!
No problem. Thanks for the quick response. Sorry for my three-day-weekend-delayed reply. :-)
Jens, others, my original email follows:
Quoted 28 lines
Hey, Tejun. I have a question about stuff going on in the block layer that's not quite as straightforward as one might wish. We have a situation in which we need to use something other than the CFQ I/O scheduler so after we create a block device (iSCSI, as it happens) we switch it to use the deadline scheduler. Unfortunately we've run into a crash when we do this in which a request completion (in some cases) or a sync I/O (in others) calls
blk_peek_request()which in turn calls__elv_next_request()which in its turn callsq->elevator->type->ops.elevator_dispatch_fn(), which happens at the point of the call bedeadline_dispatch_requests(), which promptly falls over because we're in the middle of the elevator switch andq->elevator->elevator_datais NULL.After investigation, it's clear that the elevator switch code is trying to quiesce the request queue and sets the bypass flag. Unfortunately,
scsi_mode_sense()and friends callblk_execute_rq()directly, which pays no attention to said flag.In fact, the bypass flag, as far as I can tell, is only checked in the
blk_cgroup.c, inblkg_lookup()andblkg_lookup_create().So my question is: Is this a simple oversight? Should
blk_execute_rq()care about the bypass flag? Should it perhaps hold off the I/O until the bypass flag is cleared? Since at that level it has nothing to do with cgroups I kinda don't think so but I'm still trying to get my head around how all this stuff goes together.Any hints would be _greatly_ appreciated. Thanks!
Signature · 2 lines
Frank Mayhar 310-460-4042
Quoted 7 lines · Frank Mayhar
Replacing? Or adding to? Is BYPASS always set when DYING is set? (My guess is not but I haven't done an exhaustive analysis.) So the relevant code snippet in
__elv_next_request()would be:if (unlikely(blk_queue_dying(q)) || unlikely(blk_queue_bypass(q)) || !q->elevator->type->ops.elevator_dispatch_fn(q, 0)) return NULL;
FYI, I've made this change and tested it. I can't say for certain that it fixes the crash (since it's one of those races that's difficult to reproduce), but it does seem to pass all the tests I've thrown at it so far.
Signature · 2 lines
Frank Mayhar 310-460-4042
Quoted 9 lines · Frank Mayhar
Quoted 7 lines · Frank Mayhar
Replacing? Or adding to? Is BYPASS always set when DYING is set? (My guess is not but I haven't done an exhaustive analysis.) So the relevant code snippet in
__elv_next_request()would be:if (unlikely(blk_queue_dying(q)) || unlikely(blk_queue_bypass(q)) || !q->elevator->type->ops.elevator_dispatch_fn(q, 0)) return NULL;
Quoted 4 lines
FYI, I've made this change and tested it. I can't say for certain that it fixes the crash (since it's one of those races that's difficult to reproduce), but it does seem to pass all the tests I've thrown at it so far.
Um, does anyone care about this? Tejun? Jens? Anyone?
This is a real crash; it would be nice if someone would weigh in.
Signature · 2 lines
Frank Mayhar 310-460-4042
Quoted 10 lines · Frank Mayhar
Quoted 9 lines · Frank Mayhar
Quoted 7 lines · Frank Mayhar
Replacing? Or adding to? Is BYPASS always set when DYING is set? (My guess is not but I haven't done an exhaustive analysis.) So the relevant code snippet in
__elv_next_request()would be:if (unlikely(blk_queue_dying(q)) || unlikely(blk_queue_bypass(q)) || !q->elevator->type->ops.elevator_dispatch_fn(q, 0)) return NULL;
Quoted 8 lines
Quoted 4 lines
FYI, I've made this change and tested it. I can't say for certain that it fixes the crash (since it's one of those races that's difficult to reproduce), but it does seem to pass all the tests I've thrown at it so far.
Um, does anyone care about this? Tejun? Jens? Anyone?
This is a real crash; it would be nice if someone would weigh in.
Yeah, we're gonna fix this and I *think* replacing dying with bypass is the right thing to do as a queue is always bypassing when killed. It's probably just that we're in the earlier part of the merge window and I have some other things on my plate. Will post a patch in a couple days.
Thanks.
Signature · 1 line
tejun
Quoted 18 lines · Tejun Heo
Quoted 17 lines · Frank Mayhar
Quoted 9 lines · Frank Mayhar
Quoted 7 lines · Frank Mayhar
Replacing? Or adding to? Is BYPASS always set when DYING is set? (My guess is not but I haven't done an exhaustive analysis.) So the relevant code snippet in
__elv_next_request()would be:if (unlikely(blk_queue_dying(q)) || unlikely(blk_queue_bypass(q)) || !q->elevator->type->ops.elevator_dispatch_fn(q, 0)) return NULL;Quoted 4 lines
FYI, I've made this change and tested it. I can't say for certain that it fixes the crash (since it's one of those races that's difficult to reproduce), but it does seem to pass all the tests I've thrown at it so far.
Um, does anyone care about this? Tejun? Jens? Anyone?
Quoted 7 lines
Quoted 1 line
This is a real crash; it would be nice if someone would weigh in.
Yeah, we're gonna fix this and I *think* replacing dying with bypass is the right thing to do as a queue is always bypassing when killed. It's probably just that we're in the earlier part of the merge window and I have some other things on my plate. Will post a patch in a couple days.
Thanks!
For the record, I've seriously beaten on the change above (despite it maybe being redundant, it seems to do the right thing) and have seen no problems so far. I've changed it to _just_ check bypass and will beat on that now. If you're correct that a dying queue always has bypass set then I anticipate no problems.
Signature · 2 lines
Frank Mayhar 310-460-4042