2.6.33.[56]-rt23: howto create repeatable explosion in wakeup_next_waiter()

FWIW, 2.6.31.14-rt21 explodes too (darn).

Quoted 2 lines · Mike Galbraith

FWIW, 2.6.31.14-rt21 explodes too (darn).

What about .29-rt ?

Greetings,

Stress testing, looking to trigger RCU stalls, I've managed to find a way to repeatably create fireworks. (got RCU stall, see attached)

1. download ltp-full-20100630. Needs to be this version because of testcase bustage in earlier versions, and must be built with gcc > 4.3, else testcases will segfault due to a gcc bug.

2. apply patchlet so you can run testcases/realtime/perf/latency/run.sh at all.

pthread_cond_many.c renamed from pthread_cond_many.c.org +1 -1

@@ -259,7 +259,7 @@ void usage(void)
 
 int parse_args(int c, char *v)
 {
-	int handled;
+	int handled = 1;
         switch (c) {
 		case 'h':
 			usage();

3. add --realtime for no particular reason.

run.sh renamed from run.sh.org +2 -2

@@ -22,7 +22,7 @@ make
 # process to run realtime.  The remainder of the processes (if any)
 # will run non-realtime in any case.
 
-nthread=5000
+nthread=500
 iter=400
 nproc=5
 

@@ -39,7 +39,7 @@ i=0
 i=1
 while test $i -lt $nproc
 do
-        ./pthread_cond_many --broadcast -i $iter -n $nthread > $nthread.$iter.$nproc.$i.out &
+        ./pthread_cond_many --realtime --broadcast -i $iter -n $nthread > $nthread.$iter.$nproc.$i.out &
         i=`expr $i + 1`
 done
 wait

4. run it.

What happens here is we hit WARN_ON(pendowner->pi_blocked_on != waiter), this does not make it to consoles (poking sysrq-foo doesn't either). Next comes WARN_ON(!pendowner->pi_blocked_on), followed by the NULL explosion, which does make it to consoles.

With explosion avoidance, I also see pendowner->pi_blocked_on->task == NULL at times, but that, as !pendowner->pi_blocked_on, seems to be fallout. The start of bad juju is always pi_blocked_on != waiter.

[  141.609268] BUG: unable to handle kernel NULL pointer dereference at 0000000000000058
[  141.609268] IP: [<ffffffff8106856d>] wakeup_next_waiter+0x12c/0x177
[  141.609268] PGD 20e174067 PUD 20e253067 PMD 0 
[  141.609268] Oops: 0000 [#1] PREEMPT SMP 
[  141.609268] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[  141.609268] CPU 0 
[  141.609268] Pid: 8154, comm: pthread_cond_ma Tainted: G        W  2.6.33.6-rt23 #12 MS-7502/MS-7502
[  141.609268] RIP: 0010:[<ffffffff8106856d>]  [<ffffffff8106856d>] wakeup_next_waiter+0x12c/0x177
[  141.609268] RSP: 0018:ffff88020e3cdd78  EFLAGS: 00010097
[  141.609268] RAX: 0000000000000000 RBX: ffff8801e8eba5c0 RCX: 0000000000000000
[  141.609268] RDX: ffff880028200000 RSI: 0000000000000046 RDI: 0000000000000009
[  141.609268] RBP: ffff88020e3cdda8 R08: 0000000000000002 R09: 0000000000000000
[  141.609268] R10: 0000000000000005 R11: 0000000000000000 R12: ffffffff81659068
[  141.609268] R13: ffff8801e8ebdb58 R14: 0000000000000000 R15: ffff8801e8ebac08
[  141.609268] FS:  00007f664d539700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
[  141.609268] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  141.609268] CR2: 0000000000000058 CR3: 0000000214266000 CR4: 00000000000006f0
[  141.609268] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  141.609268] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  141.609268] Process pthread_cond_ma (pid: 8154, threadinfo ffff88020e3cc000, task ffff88020e2a4700)
[  141.609268] Stack:
[  141.609268]  0000000000000000 ffffffff81659068 0000000000000202 0000000000000000
[  141.609268] <0> 0000000000000000 0000000080001fda ffff88020e3cddc8 ffffffff812fec48
[  141.609268] <0> ffffffff81659068 0000000000606300 ffff88020e3cddd8 ffffffff812ff1b9
[  141.609268] Call Trace:
[  141.609268]  [<ffffffff812fec48>] rt_spin_lock_slowunlock+0x43/0x61
[  141.609268]  [<ffffffff812ff1b9>] rt_spin_unlock+0x46/0x48
[  141.609268]  [<ffffffff81067d7f>] do_futex+0x83c/0x935
[  141.609268]  [<ffffffff810c26ce>] ? handle_mm_fault+0x6de/0x6f1
[  141.609268]  [<ffffffff81067e36>] ? do_futex+0x8f3/0x935
[  141.609268]  [<ffffffff81067fba>] sys_futex+0x142/0x154
[  141.609268]  [<ffffffff81020eb0>] ? do_page_fault+0x23e/0x28e
[  141.609268]  [<ffffffff81004aa7>] ? math_state_restore+0x3d/0x3f
[  141.609268]  [<ffffffff81004b08>] ? do_device_not_available+0xe/0x12
[  141.609268]  [<ffffffff81002c5b>] system_call_fastpath+0x16/0x1b
[  141.609268] Code: c7 09 6d 41 81 e8 ac 34 fd ff 4c 39 ab 70 06 00 00 74 11 be 47 02 00 00 48 c7 c7 09 6d 41 81 e8 92 34 fd ff 48 8b 83 70 06 00 00 <4c> 39 60 58 74 11 be 48 02 00 00 48 c7 c7 09 6d 41 81 e8 74 34 
[  141.609268] RIP  [<ffffffff8106856d>] wakeup_next_waiter+0x12c/0x177
[  141.609268]  RSP <ffff88020e3cdd78>
[  141.609268] CR2: 0000000000000058
[  141.609268] ---[ end trace 58805b944e6f93ce ]---
[  141.609268] note: pthread_cond_ma[8154] exited with preempt_count 2

(5. eyeball locks.. -> zzzzt -> report -> eyeball..)

-Mike

Cc'ing Darren.

On Wed, 7 Jul 2010, Mike Galbraith wrote:

Quoted 8 lines

Greetings,

Stress testing, looking to trigger RCU stalls, I've managed to find a way to repeatably create fireworks. (got RCU stall, see attached)

1. download ltp-full-20100630. Needs to be this version because of testcase bustage in earlier versions, and must be built with gcc> 4.3, else testcases will segfault due to a gcc bug.

Quoted 18 lines

2. apply patchlet so you can run testcases/realtime/perf/latency/run.sh at all.

pthread_cond_many.c renamed from pthread_cond_many.c.org +1 -1

@@ -259,7 +259,7 @@ void usage(void)
 
  int parse_args(int c, char *v)
  {
-	int handled;
+	int handled = 1;
          switch (c) {
  		case 'h':
  			usage();

3. add --realtime for no particular reason.

run.sh make · line 25

Quoted 8 lines

run.sh renamed from run.sh.org +1 -1

@@ -22,7 +22,7 @@ make
  # process to run realtime.  The remainder of the processes (if any)
  # will run non-realtime in any case.
 
-nthread=5000
+nthread=500

run.sh i=0 · line 45

Quoted 13 lines

run.sh renamed from run.sh.org +1 -1

  iter=400
  nproc=5
 

@@ -39,7 +39,7 @@ i=0
  i=1
  while test $i -lt $nproc
  do
-        ./pthread_cond_many --broadcast -i $iter -n $nthread>  $nthread.$iter.$nproc.$i.out&
+        ./pthread_cond_many --realtime --broadcast -i $iter -n $nthread>  $nthread.$iter.$nproc.$i.out&
          i=`expr $i + 1`
  done
  wait

Quoted 2 lines

4. run it.

Quoted 55 lines

What happens here is we hit WARN_ON(pendowner->pi_blocked_on != waiter), this does not make it to consoles (poking sysrq-foo doesn't either). Next comes WARN_ON(!pendowner->pi_blocked_on), followed by the NULL explosion, which does make it to consoles.

With explosion avoidance, I also see pendowner->pi_blocked_on->task == NULL at times, but that, as !pendowner->pi_blocked_on, seems to be fallout. The start of bad juju is always pi_blocked_on != waiter.

[  141.609268] BUG: unable to handle kernel NULL pointer dereference at 0000000000000058
[  141.609268] IP: [<ffffffff8106856d>] wakeup_next_waiter+0x12c/0x177
[  141.609268] PGD 20e174067 PUD 20e253067 PMD 0
[  141.609268] Oops: 0000 [#1] PREEMPT SMP
[  141.609268] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[  141.609268] CPU 0
[  141.609268] Pid: 8154, comm: pthread_cond_ma Tainted: G        W  2.6.33.6-rt23 #12 MS-7502/MS-7502
[  141.609268] RIP: 0010:[<ffffffff8106856d>]  [<ffffffff8106856d>] wakeup_next_waiter+0x12c/0x177
[  141.609268] RSP: 0018:ffff88020e3cdd78  EFLAGS: 00010097
[  141.609268] RAX: 0000000000000000 RBX: ffff8801e8eba5c0 RCX: 0000000000000000
[  141.609268] RDX: ffff880028200000 RSI: 0000000000000046 RDI: 0000000000000009
[  141.609268] RBP: ffff88020e3cdda8 R08: 0000000000000002 R09: 0000000000000000
[  141.609268] R10: 0000000000000005 R11: 0000000000000000 R12: ffffffff81659068
[  141.609268] R13: ffff8801e8ebdb58 R14: 0000000000000000 R15: ffff8801e8ebac08
[  141.609268] FS:  00007f664d539700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
[  141.609268] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  141.609268] CR2: 0000000000000058 CR3: 0000000214266000 CR4: 00000000000006f0
[  141.609268] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  141.609268] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  141.609268] Process pthread_cond_ma (pid: 8154, threadinfo ffff88020e3cc000, task ffff88020e2a4700)
[  141.609268] Stack:
[  141.609268]  0000000000000000 ffffffff81659068 0000000000000202 0000000000000000
[  141.609268]<0>  0000000000000000 0000000080001fda ffff88020e3cddc8 ffffffff812fec48
[  141.609268]<0>  ffffffff81659068 0000000000606300 ffff88020e3cddd8 ffffffff812ff1b9
[  141.609268] Call Trace:
[  141.609268]  [<ffffffff812fec48>] rt_spin_lock_slowunlock+0x43/0x61
[  141.609268]  [<ffffffff812ff1b9>] rt_spin_unlock+0x46/0x48
[  141.609268]  [<ffffffff81067d7f>] do_futex+0x83c/0x935
[  141.609268]  [<ffffffff810c26ce>] ? handle_mm_fault+0x6de/0x6f1
[  141.609268]  [<ffffffff81067e36>] ? do_futex+0x8f3/0x935
[  141.609268]  [<ffffffff81067fba>] sys_futex+0x142/0x154
[  141.609268]  [<ffffffff81020eb0>] ? do_page_fault+0x23e/0x28e
[  141.609268]  [<ffffffff81004aa7>] ? math_state_restore+0x3d/0x3f
[  141.609268]  [<ffffffff81004b08>] ? do_device_not_available+0xe/0x12
[  141.609268]  [<ffffffff81002c5b>] system_call_fastpath+0x16/0x1b
[  141.609268] Code: c7 09 6d 41 81 e8 ac 34 fd ff 4c 39 ab 70 06 00 00 74 11 be 47 02 00 00 48 c7 c7 09 6d 41 81 e8 92 34 fd ff 48 8b 83 70 06 00 00<4c>  39 60 58 74 11 be 48 02 00 00 48 c7 c7 09 6d 41 81 e8 74 34
[  141.609268] RIP  [<ffffffff8106856d>] wakeup_next_waiter+0x12c/0x177
[  141.609268]  RSP<ffff88020e3cdd78>
[  141.609268] CR2: 0000000000000058
[  141.609268] ---[ end trace 58805b944e6f93ce ]---
[  141.609268] note: pthread_cond_ma[8154] exited with preempt_count 2

(5. eyeball locks.. -> zzzzt -> report -> eyeball..)

-Mike

Quoted 4 lines · Thomas Gleixner

Cc'ing Darren.

On Wed, 7 Jul 2010, Mike Galbraith wrote:

Hi Mike,

Quoted 5 lines

Quoted 5 lines

Greetings,

Stress testing, looking to trigger RCU stalls, I've managed to find a way to repeatably create fireworks. (got RCU stall, see attached)

Quoted 3 lines

Quoted 3 lines

1. download ltp-full-20100630. Needs to be this version because of testcase bustage in earlier versions, and must be built with gcc> 4.3, else testcases will segfault due to a gcc bug.

Interesting, I had not hit any gcc specific issues with this. Can you point me to the bug?

Quoted 8 lines

run.sh make · line 25

Quoted 8 lines

run.sh renamed from run.sh.org +1 -1

@@ -22,7 +22,7 @@ make
  # process to run realtime.  The remainder of the processes (if any)
  # will run non-realtime in any case.
 
-nthread=5000
+nthread=500

Was this just to lighten the load, or was it required to reproduce?

Which architecture?

Glibc version?

I see kernel version is: 2.6.33.6-rt23, have you reproduced this on earlier kernel versions as well? Any 2.6.31 rt kernel would be a good data point.

Is this immediately reproducible for you?

I see a possibly fault occuring in the stack, if you run with mlockall(), does the problem go away? (assuming not, but an easy thing to test).

I see kernel version is: 2.6.33.6-rt23, have you reproduced this on earlier kernel versions as well? Any 2.6.31 rt kernel would be a good data point.

Quoted 3 lines · Darren Hart

I see kernel version is: 2.6.33.6-rt23, have you reproduced this on earlier kernel versions as well? Any 2.6.31 rt kernel would be a good data point.

FUTEX_CMP/WAIT_REQUEUE_PI first appeared in 2.6.29-rt, as does the explosion.

Cc'ing Darren.

Quoted 4 lines · Mike Galbraith

Greetings,

Stress testing, looking to trigger RCU stalls, I've managed to find a way to repeatably create fireworks. (got RCU stall, see attached)

Quoted 6 lines

4. run it.

What happens here is we hit WARN_ON(pendowner->pi_blocked_on != waiter), this does not make it to consoles (poking sysrq-foo doesn't either). Next comes WARN_ON(!pendowner->pi_blocked_on), followed by the NULL explosion, which does make it to consoles.

From fa6a6bee6e467d12d3774612c838703acd265ea6 Mon Sep 17 00:00:00 2001

So the WARN_ON sequence is obviously wrong, if it's critical it should be a BUG(), if not we shouldn't dereference what we know to be null. The following patch avoids the NULL pointer dereference in the WARN_ON. With this patch the NULL WARN_ON makes it to the console, and test runs to completion with no obvious negative side effects. I'm only posting for reference at this point, as while this may be necessary, it isn't the right "solution".

Tonight/Tomorrow I'll review the rtmutex and futex code to try and fully understand (again) the usage of pi_blocked_on and if we need to avoid this scenario, or if we need to handle it "gracefully".

I'll keep on looking, but I wanted to share what I've learned in case anyone in another timezone is hoping to continue debugging should I fall asleep at the helm ;-)

Walking through it:

First the dumps:

------------[ cut here ]------------
WARNING: at kernel/rtmutex.c:583 wakeup_next_waiter+0x1ad/0x220()

WARN_ON(pendowner->pi_blocked_on != waiter);
The pi_blocked_on is not NULL, but it isn't the expected waiter either. This means that the top waiter selected at the beginning of wakeup_next_waiter() is now blocked on a lock with a different waiter structure, possibly on a different lock.

Quoted 7 lines · Darren Hart

Walking through it:

First the dumps:

------------[ cut here ]------------
WARNING: at kernel/rtmutex.c:583 wakeup_next_waiter+0x1ad/0x220()

Quoted 5 lines

WARN_ON(pendowner->pi_blocked_on != waiter);
The pi_blocked_on is not NULL, but it isn't the expected waiter either. This means that the top waiter selected at the beginning of wakeup_next_waiter() is now blocked on a lock with a different waiter structure, possibly on a different lock.

pendowner->pi_blocked_on changes while we're in wakeup_next_waiter(). The below fi^Wmade it not do that any more. We hold the wait_lock for this lock, but if the wakee blocks on another, what's protecting us?

bandaid-by: /me

kernel/rtmutex.c +2 -2

index 23dd443..dd91ede 100644
@@ -525,6 +525,8 @@ static void wakeup_next_waiter(struct rt_mutex *lock, int savestate)
  	pendowner = waiter->task;
  	waiter->task = NULL;
 
+	raw_spin_lock(&pendowner->pi_lock);
+
  	/*
  	 * Do the wakeup before the ownership change to give any spinning
  	 * waiter grantees a headstart over the other threads that will

@@ -577,8 +579,6 @@ static void wakeup_next_waiter(struct rt_mutex *lock, int savestate)
  	else
  		next = NULL;
 
-	raw_spin_lock(&pendowner->pi_lock);
-
  	WARN_ON(!pendowner->pi_blocked_on);
  	WARN_ON(pendowner->pi_blocked_on != waiter);
  	WARN_ON(pendowner->pi_blocked_on->lock != lock);

If pendowner is blocked on "lock" to begin with (he should be as his waiter struct in in the rtmutex waiters list) then he can't block on someone else until he either acquires this one or removes himself as a waiter (due to a timeout for instance) - both of these operations require holding lock->wait_lock, which is held by the caller of wakeup_next_waiter().

Cc'ing Darren.

Quoted 61 lines · Mike Galbraith

Greetings,

Stress testing, looking to trigger RCU stalls, I've managed to find a way to repeatably create fireworks. (got RCU stall, see attached)
4. run it.

What happens here is we hit WARN_ON(pendowner->pi_blocked_on != waiter), this does not make it to consoles (poking sysrq-foo doesn't either). Next comes WARN_ON(!pendowner->pi_blocked_on), followed by the NULL explosion, which does make it to consoles.

With explosion avoidance, I also see pendowner->pi_blocked_on->task == NULL at times, but that, as !pendowner->pi_blocked_on, seems to be fallout. The start of bad juju is always pi_blocked_on != waiter.

[  141.609268] BUG: unable to handle kernel NULL pointer dereference at 0000000000000058
[  141.609268] IP: [<ffffffff8106856d>] wakeup_next_waiter+0x12c/0x177
[  141.609268] PGD 20e174067 PUD 20e253067 PMD 0
[  141.609268] Oops: 0000 [#1] PREEMPT SMP
[  141.609268] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[  141.609268] CPU 0
[  141.609268] Pid: 8154, comm: pthread_cond_ma Tainted: G        W  2.6.33.6-rt23 #12 MS-7502/MS-7502
[  141.609268] RIP: 0010:[<ffffffff8106856d>]  [<ffffffff8106856d>] wakeup_next_waiter+0x12c/0x177
[  141.609268] RSP: 0018:ffff88020e3cdd78  EFLAGS: 00010097
[  141.609268] RAX: 0000000000000000 RBX: ffff8801e8eba5c0 RCX: 0000000000000000
[  141.609268] RDX: ffff880028200000 RSI: 0000000000000046 RDI: 0000000000000009
[  141.609268] RBP: ffff88020e3cdda8 R08: 0000000000000002 R09: 0000000000000000
[  141.609268] R10: 0000000000000005 R11: 0000000000000000 R12: ffffffff81659068
[  141.609268] R13: ffff8801e8ebdb58 R14: 0000000000000000 R15: ffff8801e8ebac08
[  141.609268] FS:  00007f664d539700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
[  141.609268] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  141.609268] CR2: 0000000000000058 CR3: 0000000214266000 CR4: 00000000000006f0
[  141.609268] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  141.609268] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  141.609268] Process pthread_cond_ma (pid: 8154, threadinfo ffff88020e3cc000, task ffff88020e2a4700)
[  141.609268] Stack:
[  141.609268]  0000000000000000 ffffffff81659068 0000000000000202 0000000000000000
[  141.609268]<0>  0000000000000000 0000000080001fda ffff88020e3cddc8 ffffffff812fec48
[  141.609268]<0>  ffffffff81659068 0000000000606300 ffff88020e3cddd8 ffffffff812ff1b9
[  141.609268] Call Trace:
[  141.609268]  [<ffffffff812fec48>] rt_spin_lock_slowunlock+0x43/0x61
[  141.609268]  [<ffffffff812ff1b9>] rt_spin_unlock+0x46/0x48
[  141.609268]  [<ffffffff81067d7f>] do_futex+0x83c/0x935
[  141.609268]  [<ffffffff810c26ce>] ? handle_mm_fault+0x6de/0x6f1
[  141.609268]  [<ffffffff81067e36>] ? do_futex+0x8f3/0x935
[  141.609268]  [<ffffffff81067fba>] sys_futex+0x142/0x154
[  141.609268]  [<ffffffff81020eb0>] ? do_page_fault+0x23e/0x28e
[  141.609268]  [<ffffffff81004aa7>] ? math_state_restore+0x3d/0x3f
[  141.609268]  [<ffffffff81004b08>] ? do_device_not_available+0xe/0x12
[  141.609268]  [<ffffffff81002c5b>] system_call_fastpath+0x16/0x1b
[  141.609268] Code: c7 09 6d 41 81 e8 ac 34 fd ff 4c 39 ab 70 06 00 00 74 11 be 47 02 00 00 48 c7 c7 09 6d 41 81 e8 92 34 fd ff 48 8b 83 70 06 00 00<4c>  39 60 58 74 11 be 48 02 00 00 48 c7 c7 09 6d 41 81 e8 74 34
[  141.609268] RIP  [<ffffffff8106856d>] wakeup_next_waiter+0x12c/0x177
[  141.609268]  RSP<ffff88020e3cdd78>
[  141.609268] CR2: 0000000000000058
[  141.609268] ---[ end trace 58805b944e6f93ce ]---
[  141.609268] note: pthread_cond_ma[8154] exited with preempt_count 2

(5. eyeball locks.. -> zzzzt -> report -> eyeball..)

-Mike

From 93fd3bb97800ebf5e5c1a6a85937bab93256dd42 Mon Sep 17 00:00:00 2001

Thanks to Thomas, Steven, and Mike for hashing this over me. After an IRC discussion with Thomas, I put the following together. It resolves the issue for me, Mike please test and let us know if it fixes it for you. A couple of points of discussion before we commit this:

The use of the new state flag, PI_WAKEUP_INPROGRESS, is pretty ugly. Would a new task_pi_blocked_on_valid() method be preferred (in rtmutex.c)?

The new WARN_ON() in task_blocks_on_rt_mutex() is complex. It didn't exist before and we've now closed this gap, should we just drop it?

I've added a couple BUG_ON()s in futex_wait_requeue_pi() dealing with the race with requeue and q.lock_ptr. I'd like to leave this for the time being if nobody strongly objects.

kernel/futex.c +0 -1

-

kernel/rtmutex.c +3 -0

+  		WARN_ON(pendowner->pi_blocked_on != waiter);
+  		WARN_ON(pendowner->pi_blocked_on->lock != lock);
+  	}
 
 	pendowner->pi_blocked_on = NULL;
 

kernel/rtmutex.c static int rt_mutex_adjust_prio_chain(struct task_struct *task, · line 231

Quoted 9 lines · Darren Hart

kernel/rtmutex.c +1 -1

@@ -227,7 +227,7 @@ static int rt_mutex_adjust_prio_chain(struct task_struct *task,
 	 * reached or the state of the chain has changed while we
 	 * dropped the locks.
 	 */
-	if (!waiter || !waiter->task)
+	if (!waiter || (long)waiter == PI_WAKEUP_INPROGRESS || !waiter->task)
 		goto out_unlock_pi;

Why do we need that check ? Either the requeue succeeded then task->pi_blocked_on is set to the real waiter or the wakeup won and we are in no lock chain.

If we ever find a waiter with PI_WAKEUP_INPROGRESS set in rt_mutex_adjust_prio_chain() then it's a bug nothing else.

kernel/rtmutex.c void task_setprio(struct task_struct *p, int prio) · line 6383

Quoted 7 lines

kernel/rtmutex.c +2 -1

@@ -6377,7 +6379,8 @@ void task_setprio(struct task_struct *p, int prio)
 	 */
 	if (unlikely(p == rq->idle)) {
 		WARN_ON(p != rq->curr);
-		WARN_ON(p->pi_blocked_on);
+		WARN_ON(p->pi_blocked_on &&
+		        (long)p->pi_blocked_on != PI_WAKEUP_INPROGRESS);

Yuck. Paranoia ? If we ever requeue idle, then .....

This code causes braindamage. I really wonder whether we need to remove it according to the "United Nations Convention against Torture and Other Cruel, Inhuman or Degrading Treatment or Punishment".

Quoted 5 lines · Thomas Gleixner

This code causes braindamage. I really wonder whether we need to remove it according to the "United Nations Convention against Torture and Other Cruel, Inhuman or Degrading Treatment or Punishment".

Ok, finally managed to twist my brain around it. Mike, can you give it a test ride ?

Quoted 5 lines · Thomas Gleixner

This code causes braindamage. I really wonder whether we need to remove it according to the "United Nations Convention against Torture and Other Cruel, Inhuman or Degrading Treatment or Punishment".

Ok, finally managed to twist my brain around it. Mike, can you give it a test ride ?

kernel/futex.c +0 -3

@@ -2255,18 +2265,51 @@ static int futex_wait_requeue_pi(u32 __u
  	/* Queue the futex_q, drop the hb lock, wait for wakeup. */
  	futex_wait_queue_me(hb,&q, to);
 
-	spin_lock(&hb->lock);
-	ret = handle_early_requeue_pi_wakeup(hb,&q,&key2, to);
-	spin_unlock(&hb->lock);

On Tue, 13 Jul 2010, Thomas Gleixner wrote:

Ok, finally managed to twist my brain around it. Mike, can you give it a test ride ?

Greetings,

Stress testing, looking to trigger RCU stalls, I've managed to find a way to repeatably create fireworks. (got RCU stall, see attached)

1. download ltp-full-20100630. Needs to be this version because of testcase bustage in earlier versions, and must be built with gcc> 4.3, else testcases will segfault due to a gcc bug.

2. apply patchlet so you can run testcases/realtime/perf/latency/run.sh at all.

pthread_cond_many.c renamed from pthread_cond_many.c.org +1 -1

@@ -259,7 +259,7 @@ void usage(void)
 
  int parse_args(int c, char *v)
  {
-	int handled;
+	int handled = 1;
          switch (c) {
  		case 'h':
  			usage();

3. add --realtime for no particular reason.

run.sh renamed from run.sh.org +2 -2

@@ -22,7 +22,7 @@ make
  # process to run realtime.  The remainder of the processes (if any)
  # will run non-realtime in any case.
 
-nthread=5000
+nthread=500
  iter=400
  nproc=5
 

@@ -39,7 +39,7 @@ i=0
  i=1
  while test $i -lt $nproc
  do
-        ./pthread_cond_many --broadcast -i $iter -n $nthread>  $nthread.$iter.$nproc.$i.out&
+        ./pthread_cond_many --realtime --broadcast -i $iter -n $nthread>  $nthread.$iter.$nproc.$i.out&
          i=`expr $i + 1`
  done
  wait

4. run it.

What happens here is we hit WARN_ON(pendowner->pi_blocked_on != waiter), this does not make it to consoles (poking sysrq-foo doesn't either). Next comes WARN_ON(!pendowner->pi_blocked_on), followed by the NULL explosion, which does make it to consoles.

With explosion avoidance, I also see pendowner->pi_blocked_on->task == NULL at times, but that, as !pendowner->pi_blocked_on, seems to be fallout. The start of bad juju is always pi_blocked_on != waiter.

[  141.609268] BUG: unable to handle kernel NULL pointer dereference at 0000000000000058
[  141.609268] IP: [<ffffffff8106856d>] wakeup_next_waiter+0x12c/0x177
[  141.609268] PGD 20e174067 PUD 20e253067 PMD 0
[  141.609268] Oops: 0000 [#1] PREEMPT SMP
[  141.609268] last sysfs file: /sys/devices/system/cpu/cpu3/cache/index2/shared_cpu_map
[  141.609268] CPU 0
[  141.609268] Pid: 8154, comm: pthread_cond_ma Tainted: G        W  2.6.33.6-rt23 #12 MS-7502/MS-7502
[  141.609268] RIP: 0010:[<ffffffff8106856d>]  [<ffffffff8106856d>] wakeup_next_waiter+0x12c/0x177
[  141.609268] RSP: 0018:ffff88020e3cdd78  EFLAGS: 00010097
[  141.609268] RAX: 0000000000000000 RBX: ffff8801e8eba5c0 RCX: 0000000000000000
[  141.609268] RDX: ffff880028200000 RSI: 0000000000000046 RDI: 0000000000000009
[  141.609268] RBP: ffff88020e3cdda8 R08: 0000000000000002 R09: 0000000000000000
[  141.609268] R10: 0000000000000005 R11: 0000000000000000 R12: ffffffff81659068
[  141.609268] R13: ffff8801e8ebdb58 R14: 0000000000000000 R15: ffff8801e8ebac08
[  141.609268] FS:  00007f664d539700(0000) GS:ffff880028200000(0000) knlGS:0000000000000000
[  141.609268] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  141.609268] CR2: 0000000000000058 CR3: 0000000214266000 CR4: 00000000000006f0
[  141.609268] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  141.609268] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  141.609268] Process pthread_cond_ma (pid: 8154, threadinfo ffff88020e3cc000, task ffff88020e2a4700)
[  141.609268] Stack:
[  141.609268]  0000000000000000 ffffffff81659068 0000000000000202 0000000000000000
[  141.609268]<0>  0000000000000000 0000000080001fda ffff88020e3cddc8 ffffffff812fec48
[  141.609268]<0>  ffffffff81659068 0000000000606300 ffff88020e3cddd8 ffffffff812ff1b9
[  141.609268] Call Trace:
[  141.609268]  [<ffffffff812fec48>] rt_spin_lock_slowunlock+0x43/0x61
[  141.609268]  [<ffffffff812ff1b9>] rt_spin_unlock+0x46/0x48
[  141.609268]  [<ffffffff81067d7f>] do_futex+0x83c/0x935
[  141.609268]  [<ffffffff810c26ce>] ? handle_mm_fault+0x6de/0x6f1
[  141.609268]  [<ffffffff81067e36>] ? do_futex+0x8f3/0x935
[  141.609268]  [<ffffffff81067fba>] sys_futex+0x142/0x154
[  141.609268]  [<ffffffff81020eb0>] ? do_page_fault+0x23e/0x28e
[  141.609268]  [<ffffffff81004aa7>] ? math_state_restore+0x3d/0x3f
[  141.609268]  [<ffffffff81004b08>] ? do_device_not_available+0xe/0x12
[  141.609268]  [<ffffffff81002c5b>] system_call_fastpath+0x16/0x1b
[  141.609268] Code: c7 09 6d 41 81 e8 ac 34 fd ff 4c 39 ab 70 06 00 00 74 11 be 47 02 00 00 48 c7 c7 09 6d 41 81 e8 92 34 fd ff 48 8b 83 70 06 00 00<4c>  39 60 58 74 11 be 48 02 00 00 48 c7 c7 09 6d 41 81 e8 74 34
[  141.609268] RIP  [<ffffffff8106856d>] wakeup_next_waiter+0x12c/0x177
[  141.609268]  RSP<ffff88020e3cdd78>
[  141.609268] CR2: 0000000000000058
[  141.609268] ---[ end trace 58805b944e6f93ce ]---
[  141.609268] note: pthread_cond_ma[8154] exited with preempt_count 2

(5. eyeball locks.. -> zzzzt -> report -> eyeball..)

-Mike

Quoted 10 lines · Mike Galbraith

Greetings,

Stress testing, looking to trigger RCU stalls, I've managed to find a way to repeatably create fireworks. (got RCU stall, see attached)

1. download ltp-full-20100630. Needs to be this version because of testcase bustage in earlier versions, and must be built with gcc> 4.3, else testcases will segfault due to a gcc bug.

Hi Mike,
I have seen this segfault esp with GCC v4.3.4. I am about to post this patch
in ltp:

Signed-off-by Gowrishankar <[email protected]>

---
  testcases/realtime/include/librttest.h |    6 +++---
  1 files changed, 3 insertions(+), 3 deletions(-)

testcases/realtime/include/librttest.h +3 -4

index e526ab4..273de6f 100644
@@ -118,9 +118,9 @@ static inline int atomic_add(int i, atomic_t *v)
         int __i;
         __i = i;
         asm volatile(
-                       "lock; xaddl %0, %1;"
-                       :"=r"(i)
-                       :"m"(v->counter), "0"(i));
+                       "lock; xaddl %1, %0;"
+                       :"=m"(v->counter)
+                       :"r"(i), "m" (v->counter));
         return i + __i;
  #elif defined(__powerpc__)
  #define ISYNC_ON_SMP   "\n\tisync\n"
--

Please let me know if this patch helps.

Quoted 41 lines · gowrishankar

Quoted 10 lines · Mike Galbraith

Greetings,

Stress testing, looking to trigger RCU stalls, I've managed to find a way to repeatably create fireworks. (got RCU stall, see attached)

1. download ltp-full-20100630. Needs to be this version because of
testcase bustage in earlier versions, and must be built with gcc> 4.3,
else testcases will segfault due to a gcc bug.

Hi Mike,
I have seen this segfault esp with GCC v4.3.4. I am about to post this patch
in ltp:

Signed-off-by Gowrishankar<[email protected]>

---
   testcases/realtime/include/librttest.h |    6 +++---
   1 files changed, 3 insertions(+), 3 deletions(-)

testcases/realtime/include/librttest.h +3 -4

index e526ab4..273de6f 100644
@@ -118,9 +118,9 @@ static inline int atomic_add(int i, atomic_t *v)
          int __i;
          __i = i;
          asm volatile(
-                       "lock; xaddl %0, %1;"
-                       :"=r"(i)
-                       :"m"(v->counter), "0"(i));
+                       "lock; xaddl %1, %0;"
+                       :"=m"(v->counter)
+                       :"r"(i), "m" (v->counter));
          return i + __i;
   #elif defined(__powerpc__)
   #define ISYNC_ON_SMP   "\n\tisync\n"
--

Quoted 1 line

Please let me know if this patch helps.

Yup, all better.

-Mike

Quoted 1 line · Mike Galbraith

Yup, all better.

So with this, the "volatile struct" patch isn't necessary?