Deadlock during multipath failover
Subsystems (2)
During failover tests on a current distribution kernel, we found this problem. From reading the code, the upstream kernel has the same problem:
During multipath failover tests with SCSI on System z, the kernel deadlocks in this situation:
Quoted 10 lines
STACK:
0 blk_add_timer+206 [0x2981ea]
1 blk_rq_timed_out+132 [0x2982a8]
2 blk_abort_request+114 [0x29833e]
3 blk_abort_queue+92 [0x2983a8]
4 deactivate_path+74 [0x3e00009625a]
5 run_workqueue+236 [0x149e04]
6 worker_thread+294 [0x149fce]
7 kthread+110 [0x14f436]
8 kernel_thread_starter+6 [0x10941a]
blk_abort_queue takes the queue_lock with spinlock_irqsave and walks the timer_list with list_for_each_entry_safe. Since a path to a SCSI device just failed, the rport state is FC_PORTSTATE_BLOCKED. This rport state triggers blk_add_timer that calls list_add_tail to move the request to the end of timer_list. Thus, the list_for_each_entry_safe never reaches the end of the timer_list, it continously moves the requests to the end of the list.
The rport state FC_PORTSTATE_BLOCKED would end, when the function fc_timeout_deleted_rport would run to remove the rport. But this function was schedules from queue_delayed_work. The timer already expired, but the timer function does not run, because the timer interrupt is disabled from the spinlock_irqsave call.
Christof Schmitt
Hi Christof,
Quoted 18 lines · Christof Schmitt
During failover tests on a current distribution kernel, we found this problem. From reading the code, the upstream kernel has the same problem:
During multipath failover tests with SCSI on System z, the kernel deadlocks in this situation:
Quoted 10 lines
STACK:
0 blk_add_timer+206 [0x2981ea]
1 blk_rq_timed_out+132 [0x2982a8]
2 blk_abort_request+114 [0x29833e]
3 blk_abort_queue+92 [0x2983a8]
4 deactivate_path+74 [0x3e00009625a]
5 run_workqueue+236 [0x149e04]
6 worker_thread+294 [0x149fce]
7 kthread+110 [0x14f436]
8 kernel_thread_starter+6 [0x10941a]
Quoted 8 lines
blk_abort_queuetakes thequeue_lockwithspinlock_irqsaveand walks thetimer_listwithlist_for_each_entry_safe. Since a path to a SCSI device just failed, the rport state isFC_PORTSTATE_BLOCKED. This rport state triggersblk_add_timerthat callslist_add_tailto move the request to the end oftimer_list. Thus, thelist_for_each_entry_safenever reaches the end of thetimer_list, it continously moves the requests to the end of the list.
Hmm. That would be fixes by using list_splice() here:
block/blk-timeout.c +0 -0
@@ -209,12 +209,15 @@ void blk_abort_queue(struct request_queue *q)
{
unsigned long flags;
struct request *rq, *tmp;+ LIST_HEAD(list);
spin_lock_irqsave(q->queue_lock, flags);
elv_abort_queue(q);- list_for_each_entry_safe(rq, tmp, &q->timeout_list, timeout_list)
+ list_splice_init(&q->timeout_list, &list);
+
+ list_for_each_entry_safe(rq, tmp, &list, timeout_list)
blk_abort_request(rq);
spin_unlock_irqrestore(q->queue_lock, flags);Quoted 6 lines
The rport state
FC_PORTSTATE_BLOCKEDwould end, when the functionfc_timeout_deleted_rportwould run to remove the rport. But this function was schedules fromqueue_delayed_work. The timer already expired, but the timer function does not run, because the timer interrupt is disabled from thespinlock_irqsavecall.
.. but this shouldn't happen anymore when using splice, as the timer will be called _after_ the irqrestore above.
Cheers,
Hannes
Signature · 8 lines
Dr. Hannes Reinecke zSeries & Storage [email protected] +49 911 74053 688 SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg GF: Markus Rex, HRB 16746 (AG Nürnberg) -- To unsubscribe from this list: send the line "unsubscribe linux-scsi" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 53 lines · Hannes Reinecke
Hi Christof,
Quoted 18 lines · Christof Schmitt
During failover tests on a current distribution kernel, we found this problem. From reading the code, the upstream kernel has the same problem:
During multipath failover tests with SCSI on System z, the kernel deadlocks in this situation:
Quoted 10 lines
STACK:
0 blk_add_timer+206 [0x2981ea]
1 blk_rq_timed_out+132 [0x2982a8]
2 blk_abort_request+114 [0x29833e]
3 blk_abort_queue+92 [0x2983a8]
4 deactivate_path+74 [0x3e00009625a]
5 run_workqueue+236 [0x149e04]
6 worker_thread+294 [0x149fce]
7 kthread+110 [0x14f436]
8 kernel_thread_starter+6 [0x10941a]Quoted 8 lines
blk_abort_queuetakes thequeue_lockwithspinlock_irqsaveand walks thetimer_listwithlist_for_each_entry_safe. Since a path to a SCSI device just failed, the rport state isFC_PORTSTATE_BLOCKED. This rport state triggersblk_add_timerthat callslist_add_tailto move the request to the end oftimer_list. Thus, thelist_for_each_entry_safenever reaches the end of thetimer_list, it continously moves the requests to the end of the list.Hmm. That would be fixes by using
list_splice()here:block/blk-timeout.c +4
-1@@ -209,12 +209,15 @@ void blk_abort_queue(struct request_queue *q) {unsigned long flags;struct request *rq, *tmp; +LIST_HEAD (list);spin_lock_irqsave (q->queue_lock, flags);elv_abort_queue (q); -list_for_each_entry_safe (rq, tmp, &q->timeout_list, timeout_list) +list_splice_init (&q->timeout_list, &list); + +list_for_each_entry_safe (rq, tmp, &list, timeout_list)blk_abort_request (rq);spin_unlock_irqrestore (q->queue_lock, flags);
Quoted 8 lines
Quoted 6 lines
The rport state
FC_PORTSTATE_BLOCKEDwould end, when the functionfc_timeout_deleted_rportwould run to remove the rport. But this function was schedules fromqueue_delayed_work. The timer already expired, but the timer function does not run, because the timer interrupt is disabled from thespinlock_irqsavecall... but this shouldn't happen anymore when using splice, as the timer will be called _after_ the irqrestore above.
If this patch does not address the deadlock another option to look into would be to run some testing without blk_abort_request (just using elv_abort_queue) and not try to abort in flight IOs at this time.
We observed reduced IO delays during storage failover testing (target responsive but timing out IOs) with this code, but I do not have good breakdown data on the number of IOs handled by elv_abort_queue vs blk_abort_request vs IO delay (It is also config dependent).
-andmike
--
Michael Anderson
[email protected]
Quoted 63 lines · Mike Anderson
Quoted 53 lines · Hannes Reinecke
Hi Christof,
Quoted 18 lines · Christof Schmitt
During failover tests on a current distribution kernel, we found this problem. From reading the code, the upstream kernel has the same problem:
During multipath failover tests with SCSI on System z, the kernel deadlocks in this situation:
Quoted 10 lines
STACK:
0 blk_add_timer+206 [0x2981ea]
1 blk_rq_timed_out+132 [0x2982a8]
2 blk_abort_request+114 [0x29833e]
3 blk_abort_queue+92 [0x2983a8]
4 deactivate_path+74 [0x3e00009625a]
5 run_workqueue+236 [0x149e04]
6 worker_thread+294 [0x149fce]
7 kthread+110 [0x14f436]
8 kernel_thread_starter+6 [0x10941a]Quoted 8 lines
blk_abort_queuetakes thequeue_lockwithspinlock_irqsaveand walks thetimer_listwithlist_for_each_entry_safe. Since a path to a SCSI device just failed, the rport state isFC_PORTSTATE_BLOCKED. This rport state triggersblk_add_timerthat callslist_add_tailto move the request to the end oftimer_list. Thus, thelist_for_each_entry_safenever reaches the end of thetimer_list, it continously moves the requests to the end of the list.Hmm. That would be fixes by using
list_splice()here:block/blk-timeout.c +4
-1@@ -209,12 +209,15 @@ void blk_abort_queue(struct request_queue *q) {unsigned long flags;struct request *rq, *tmp; +LIST_HEAD (list);spin_lock_irqsave (q->queue_lock, flags);elv_abort_queue (q); -list_for_each_entry_safe (rq, tmp, &q->timeout_list, timeout_list) +list_splice_init (&q->timeout_list, &list); + +list_for_each_entry_safe (rq, tmp, &list, timeout_list)blk_abort_request (rq);spin_unlock_irqrestore (q->queue_lock, flags);Quoted 8 lines
Quoted 6 lines
The rport state
FC_PORTSTATE_BLOCKEDwould end, when the functionfc_timeout_deleted_rportwould run to remove the rport. But this function was schedules fromqueue_delayed_work. The timer already expired, but the timer function does not run, because the timer interrupt is disabled from thespinlock_irqsavecall... but this shouldn't happen anymore when using splice, as the timer will be called _after_ the irqrestore above.
Quoted 8 lines
If this patch does not address the deadlock another option to look into would be to run some testing without
blk_abort_request(just usingelv_abort_queue) and not try to abort in flight IOs at this time.We observed reduced IO delays during storage failover testing (target responsive but timing out IOs) with this code, but I do not have good breakdown data on the number of IOs handled by
elv_abort_queuevsblk_abort_requestvs IO delay (It is also config dependent).
The patch fixes the observed deadlock. While the rport is BLOCKED, blk_abort_request only resets the timer for each request, so i would guess there is no big difference in calling blk_abort_request or not, at least in this scenario.
Christof Schmitt
Quoted 73 lines · Christof Schmitt
Quoted 63 lines · Mike Anderson
Quoted 53 lines · Hannes Reinecke
Hi Christof,
Quoted 18 lines · Christof Schmitt
During failover tests on a current distribution kernel, we found this problem. From reading the code, the upstream kernel has the same problem:
During multipath failover tests with SCSI on System z, the kernel deadlocks in this situation:
Quoted 10 lines
STACK:
0 blk_add_timer+206 [0x2981ea]
1 blk_rq_timed_out+132 [0x2982a8]
2 blk_abort_request+114 [0x29833e]
3 blk_abort_queue+92 [0x2983a8]
4 deactivate_path+74 [0x3e00009625a]
5 run_workqueue+236 [0x149e04]
6 worker_thread+294 [0x149fce]
7 kthread+110 [0x14f436]
8 kernel_thread_starter+6 [0x10941a]Quoted 8 lines
blk_abort_queuetakes thequeue_lockwithspinlock_irqsaveand walks thetimer_listwithlist_for_each_entry_safe. Since a path to a SCSI device just failed, the rport state isFC_PORTSTATE_BLOCKED. This rport state triggersblk_add_timerthat callslist_add_tailto move the request to the end oftimer_list. Thus, thelist_for_each_entry_safenever reaches the end of thetimer_list, it continously moves the requests to the end of the list.Hmm. That would be fixes by using
list_splice()here:block/blk-timeout.c +4
-1@@ -209,12 +209,15 @@ void blk_abort_queue(struct request_queue *q) {unsigned long flags;struct request *rq, *tmp; +LIST_HEAD (list);spin_lock_irqsave (q->queue_lock, flags);elv_abort_queue (q); -list_for_each_entry_safe (rq, tmp, &q->timeout_list, timeout_list) +list_splice_init (&q->timeout_list, &list); + +list_for_each_entry_safe (rq, tmp, &list, timeout_list)blk_abort_request (rq);spin_unlock_irqrestore (q->queue_lock, flags);Quoted 8 lines
Quoted 6 lines
The rport state
FC_PORTSTATE_BLOCKEDwould end, when the functionfc_timeout_deleted_rportwould run to remove the rport. But this function was schedules fromqueue_delayed_work. The timer already expired, but the timer function does not run, because the timer interrupt is disabled from thespinlock_irqsavecall... but this shouldn't happen anymore when using splice, as the timer will be called _after_ the irqrestore above.
Quoted 8 lines
If this patch does not address the deadlock another option to look into would be to run some testing without
blk_abort_request(just usingelv_abort_queue) and not try to abort in flight IOs at this time.We observed reduced IO delays during storage failover testing (target responsive but timing out IOs) with this code, but I do not have good breakdown data on the number of IOs handled by
elv_abort_queuevsblk_abort_requestvs IO delay (It is also config dependent).
Quoted 4 lines
The patch fixes the observed deadlock. While the rport is BLOCKED,
blk_abort_requestonly resets the timer for each request, so i would guess there is no big difference in callingblk_abort_requestor not, at least in this scenario.
That approach does look safer to me. Can the original author send a proper patch so I get attributions correct, then I'll merge it.
Signature · 1 line
Jens Axboe