Filesystem lockup with CONFIG_PREEMPT_RT

DISCUSSION John Blackwood 3 participants 3 messages May 21, 2014 – Jul 5, 2014 Scheduler

Share & tools

John Blackwood

author

2014-05-21 19:30 UTC #0 lore raw

Quoted 7 lines

Date: Wed, 21 May 2014 03:33:49 -0400
From: Richard Weinberger <[email protected]>
To: Austin Schuh <[email protected]>
CC: LKML <[email protected]>, xfs <[email protected]>, rt-users
<[email protected]>
Subject: Re: Filesystem lockup with CONFIG_PREEMPT_RT

Quoted 28 lines

CC'ing RT folks
Quoted 25 lines · Austin Schuh
Quoted 25 lines · Austin Schuh
Quoted 11 lines
Quoted 11 lines
Hi,
I am observing a filesystem lockup with XFS on a CONFIG_PREEMPT_RT patched kernel. I have currently only triggered it using dpkg. Dave Chinner on the XFS mailing list suggested that it was a rt-kernel workqueue issue as opposed to a XFS problem after looking at the kernel messages.
The only modification to the kernel besides the RT patch is that I have applied tglx's "genirq: Sanitize spurious interrupt detection of threaded irqs" patch.
I upgraded to 3.14.3-rt4, and the problem still persists.
I turned on event tracing and tracked it down further. I'm able to lock it up by scping a new kernel debian package to /tmp/ on the machine. scp is locking the inode, and then scheduling xfs_bmapi_allocate_worker in the work queue. The work then never gets run. The kworkers then lock up waiting for the inode lock.
Here are the relevant events from the trace. ffff8803e9f10288 (blk_delay_work) gets run later on in the trace, but ffff8803b4c158d0 (xfs_bmapi_allocate_worker) never does. The kernel then warns about blocked tasks 120 seconds later.

Austin and Richard,

I'm not 100% sure that the patch below will fix your problem, but we saw something that sounds pretty familiar to your issue involving the nvidia driver and the preempt-rt patch. The nvidia driver uses the completion support to create their own driver's notion of an internally used semaphore.

Some tasks were failing to ever wakeup from wait_for_completion() calls due to a race in the underlying do_wait_for_common() routine.

This is the patch that we used to fix this issue:

------------------- -------------------

Fix a race in the PRT wait for completion simple wait code.

A wait_for_completion() waiter task can be awoken by a task calling complete(), but fail to consume the 'done' completion resource if it looses a race with another task calling wait_for_completion() just as it is waking up.

In this case, the awoken task will call schedule_timeout() again without being in the simple wait queue.

So if the awoken task is unable to claim the 'done' completion resource, check to see if it needs to be re-inserted into the wait list before waiting again in schedule_timeout().

Fix-by John Blackwood <[email protected]>

kernel/sched/core.c +9 -0

Index: b/kernel/sched/core.c
===================================================================
@@ -3529,11 +3529,19 @@ static inline long __sched
 do_wait_for_common(struct completion *x,
 		   long (*action)(long), long timeout, int state)
 {
+	int again = 0;
+
 	if (!x->done) {
 		DEFINE_SWAITER(wait);
 
 		swait_prepare_locked(&x->wait, &wait);
 		do {
+			/* Check to see if we lost race for 'done' and are
+			 * no longer in the wait list.
+			 */
+			if (unlikely(again) && list_empty(&wait.node))
+				swait_prepare_locked(&x->wait, &wait);
+
 			if (signal_pending_state(state, current)) {
 				timeout = -ERESTARTSYS;
 				break;

@@ -3542,6 +3550,7 @@ do_wait_for_common(struct completion *x,
 			raw_spin_unlock_irq(&x->wait.lock);
 			timeout = action(timeout);
 			raw_spin_lock_irq(&x->wait.lock);
+			again = 1;
 		} while (!x->done && timeout);
 		swait_finish_locked(&x->wait, &wait);
 		if (!x->done)

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Austin Schuh

2014-05-21 21:59 UTC #1 lore raw

Quoted 43 lines · John Blackwood

Quoted 8 lines
Date: Wed, 21 May 2014 03:33:49 -0400
From: Richard Weinberger <[email protected]>
To: Austin Schuh <[email protected]>
CC: LKML <[email protected]>, xfs <[email protected]>, rt-users
<[email protected]>
Subject: Re: Filesystem lockup with CONFIG_PREEMPT_RT
Quoted 32 lines
CC'ing RT folks
Quoted 28 lines · Austin Schuh
Quoted 28 lines
Quoted 9 lines · Austin Schuh
Quoted 9 lines
Hi,
I am observing a filesystem lockup with XFS on a CONFIG_PREEMPT_RT patched kernel. I have currently only triggered it using dpkg. Dave
Chinner on the XFS mailing list suggested that it was a rt-kernel workqueue issue as opposed to a XFS problem after looking at the kernel messages.
Quoted 4 lines
Quoted 4 lines
The only modification to the kernel besides the RT patch is that I have applied tglx's "genirq: Sanitize spurious interrupt detection of
threaded irqs" patch.
I upgraded to 3.14.3-rt4, and the problem still persists.
I turned on event tracing and tracked it down further. I'm able to lock it up by scping a new kernel debian package to /tmp/ on the machine. scp is locking the inode, and then scheduling xfs_bmapi_allocate_worker in the work queue. The work then never gets run. The kworkers then lock up waiting for the inode lock.
Here are the relevant events from the trace. ffff8803e9f10288 (blk_delay_work) gets run later on in the trace, but ffff8803b4c158d0 (xfs_bmapi_allocate_worker) never does. The kernel then warns about blocked tasks 120 seconds later.
Austin and Richard,

Quoted 8 lines

I'm not 100% sure that the patch below will fix your problem, but we saw something that sounds pretty familiar to your issue involving the nvidia driver and the preempt-rt patch. The nvidia driver uses the completion support to create their own driver's notion of an internally used semaphore.
Some tasks were failing to ever wakeup from wait_for_completion() calls due to a race in the underlying do_wait_for_common() routine.

Hi John,

Thanks for the suggestion and patch. The issue is that the work never gets run, not that the work finishes but the waiter never gets woken. I applied it anyways to see if it helps, but I still get the lockup.

Thanks,
Austin

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs

Thomas Gleixner

2014-07-05 20:36 UTC #2 lore raw

In reply to John Blackwood

Quoted 5 lines · John Blackwood

I'm not 100% sure that the patch below will fix your problem, but we saw something that sounds pretty familiar to your issue involving the nvidia driver and the preempt-rt patch. The nvidia driver uses the completion support to create their own driver's notion of an internally used semaphore.

And why should we care about that? If they abuse the completion code then it's their problem. And it has nothing to do at all with the problem Austin is observing.

Quoted 2 lines

Some tasks were failing to ever wakeup from wait_for_completion() calls due to a race in the underlying do_wait_for_common() routine.

Why are you not fixing the nvidia crap instead of hacking utter shite into the core kernel?

Thanks,

tglx

_______________________________________________
xfs mailing list
[email protected]
http://oss.sgi.com/mailman/listinfo/xfs