3.14-rt1

Dear RT folks!

I'm pleased to announce the v3.14-rt1 patch set.

Changes since v3.12.15-rt25
- I dropped the sparc64 patches I had in the queue. They did not apply cleanly, the code in v3.14 changed in the MMU area. Here is where I remembered that it was not working perfectly either.

- Scott Wood pointed out that I forgot a return value in the latest gianfar fixup for RT. Return value added, thanks Scott.

This -RT series didn't crashed within ~4h testing on my ARM and x86-32.
x86-64 crashed after I started hackbench. I figured out that the crash does not happen with lazy-preempt disabled. Therefore the last but one patch in the queue disables lazy preempt on x86-64. With this change the test box survived ~2h without a crash. I look at this later but it looks good now.

I decided to release it now because otherwise it would be lying around probably the next two weeks or so. So here it is, enjoy.

Known issues:

- bcache is disabled.

The RT patch against 3.14 can be found here:

https://www.kernel.org/pub/linux/kernel/projects/rt/3.14/patch-3.14.0-rt1.patch.xz

The split quilt queue is available at:

https://www.kernel.org/pub/linux/kernel/projects/rt/3.14/patches-3.14.0-rt1.tar.xz

Sebastian

11.04.2014 22:57, Sebastian Andrzej Siewior пишет:

Hooooooooooooooooray!

                                                         Pavel.

11.04.2014 22:57, Sebastian Andrzej Siewior пишет:

AMD64 (Dual AMD Opteron 285), 8Gb RAM, MB: TYAN Thunder KW8E 2895

WORK ONLY IN UniProcessor mode (maxcpus=0)

Normal boot: Debian 7 x64, Xorg, Google Chrome, Firefox/Thunderbird, Unigine Heaven Benchmark: 39 FPS, Nvidia driver 334.21, watch youtube.

./hackbench -T 4 -l 10000
Time: 162.643

./cyclictest -p 99 -D 60
policy: fifo: loadavg: 0.27 19.88 19.17 1/311 4481
T: 0 ( 4336) P:99 I:1000 C: 59995 Min: 19 Act: 23 Avg: 24 Max: 61

./signaltest -p99 -m -l 10000
0.25 13.60 16.96 3/312 4831

T: 0 ( 4812) P:99 C: 10000 Min: 9 Act: 9 Avg:10 Max: 35

---

In SMP mode boot only with kernel option init=/bin/bash After remount /dev/sda2 to RW mode, and run mc (midnight commander) system freezeeeee! But SysRQ key works!

/proc/cmdline:
root=/dev/sda2 quiet vga=0 pci=routeirq nmi_watchdog=0 iommu=noagp mce=dont_log_ce tsc=reliable clocksource=hpet workqueue.power_efficient=0 agp=off reset_devices iommu=memaper=4 amd_iommu=fullflush pci=skip_isa_align sata_nv.adma=1 hpet=force zswap.enabled=1 maxcpus=0

.config: http://pastebin.com/2GmGc3Ra
dmesg: http://pastebin.com/gxZXHK2x

                                                         Pavel.
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi Sebastian,

This hunk in hotplug-light-get-online-cpus.patch looks like a bug.

@@ -333,7 +449,7 @@ static int __ref _cpu_down(unsigned int

                /* CPU didn't die: tell everyone.  Can't complain. */
                smpboot_unpark_threads(cpu);
                cpu_notify_nofail(CPU_DOWN_FAILED | mod, hcpu);
-               goto out_release;
+               goto out_cancel;
        }
        BUG_ON(cpu_online(cpu));

Ah, I had trouble there a while back too. I'll try to scrape up cycles for a round 2, see who begs for mercy this time, it or me again.

-Mike

Thanks, fixed up.

Sebastian

Box actually deadlocks like so.

                                                CPU3 boot.kdump
                                                sys_wait4
                                                do_wait
                                                   read_lock(&tasklist_lock)
                                                      rt_read_lock
                                                         __rt_spin_lock(lock)
                                                            migrate_disable()
                                                               pin_current_cpu()
                                                                  if (hp->grab_lock) {
                                                                     preempt_enable(); <== hmm
                                                                     hotplug_lock(hp); 
                                                                       hp = &__get_cpu_var(hotplug_pcp);  <== hmm
                                                                       struct hotplug_pcp {
                                                                          unplug = 0xffff8800b7d0e540,
                                                                          sync_tsk = 0x0,
                                                                          refcount = 0, <== hmm
                                                                          grab_lock = 1,
                                                                          ...
                                                                          lock = {
                                                                             ...
                                                                             owner = 0xffff8802039f0001,
                                                                             stress-cpu-hotplug_stress.sh?!?
                                                                             <=== he's way over yonder.
                                                                             Yo, dude, would you please NOT
                                                                             take percpu locks with you?

CPU0 stress-cpu-hotplug_stress.sh
sysfs_write_file
dev_attr_store
online_store
device_offline
cpu_subsys_offline
cpu_down
_cpu_down
cpu_hotplug_begin

       mutex_lock(&cpu_hotplug.lock);
       ...
      check_for_tasks
         write_lock_irq(&tasklist_lock);
         held by CPU3 boot.kdump over there ===>

CPU0 kworker/0:0

cpuset_hotplug_workfn+0x23e/0x380
rebuild_sched_domains+0x15/0x30
rebuild_sched_domains_locked+0x17/0x80
get_online_cpus+0x35/0x50
   mutex_lock(&cpu_hotplug.lock);
   held by stress-cpu-hotplug_stress.sh

twiddle twiddle twiddle...

INFO: task kworker/0:0:4 blocked for more than 120 seconds.

...

3.12-rt looks a bit busted migrate_disable/enable() wise.

/me eyeballs 3.10-rt (looks better), confirms 3.10-rt hotplug works, swipes working code, confirms 3.12-rt now works. Yup, that was it.

When I fix lg_global_lock() (I think it and Medusa are both busted) I bet a nickle 3.14-rt will work.

Hm, actually, rt_write_trylock() in swiped 3.10-rt code below (and some others) look busted to me. migrate_disable() _after_ grabbing a lock is too late, no?

---
 include/linux/rwlock_rt.h |   32 ++++++++++++++++++++++++++++----
 kernel/rt.c               |   21 +++++++++++----------
 2 files changed, 39 insertions(+), 14 deletions(-)

@@ -33,50 +33,72 @@ extern void __rt_rwlock_init(rwlock_t *r
 #define read_lock_irqsave(lock, flags)			\
 	do {						\
 		typecheck(unsigned long, flags);	\
+		migrate_disable();			\
 		flags = rt_read_lock_irqsave(lock);	\
 	} while (0)
 
 #define write_lock_irqsave(lock, flags)			\
 	do {						\
 		typecheck(unsigned long, flags);	\
+		migrate_disable();			\
 		flags = rt_write_lock_irqsave(lock);	\
 	} while (0)
 
-#define read_lock(lock)		rt_read_lock(lock)
+#define read_lock(lock)					\
+	do {						\
+		migrate_disable();			\
+		rt_read_lock(lock);			\
+	} while (0)
 
 #define read_lock_bh(lock)				\
 	do {						\
 		local_bh_disable();			\
+		migrate_disable();			\
 		rt_read_lock(lock);			\
 	} while (0)
 
 #define read_lock_irq(lock)	read_lock(lock)
 
-#define write_lock(lock)	rt_write_lock(lock)
+#define write_lock(lock)				\
+	do {						\
+		migrate_disable();			\
+		rt_write_lock(lock);			\
+	} while (0)
 
 #define write_lock_bh(lock)				\
 	do {						\
 		local_bh_disable();			\
+		migrate_disable();			\
 		rt_write_lock(lock);			\
 	} while (0)
 
 #define write_lock_irq(lock)	write_lock(lock)
 
-#define read_unlock(lock)	rt_read_unlock(lock)
+#define read_unlock(lock)				\
+	do {						\
+		rt_read_unlock(lock);			\
+		migrate_enable();			\
+	} while (0)
 
 #define read_unlock_bh(lock)				\
 	do {						\
 		rt_read_unlock(lock);			\
+		migrate_enable();			\
 		local_bh_enable();			\
 	} while (0)
 
 #define read_unlock_irq(lock)	read_unlock(lock)
 
-#define write_unlock(lock)	rt_write_unlock(lock)
+#define write_unlock(lock)				\
+	do {						\
+		rt_write_unlock(lock);			\
+		migrate_enable();			\
+	} while (0)
 
 #define write_unlock_bh(lock)				\
 	do {						\
 		rt_write_unlock(lock);			\
+		migrate_enable();			\
 		local_bh_enable();			\
 	} while (0)
 

@@ -87,6 +109,7 @@ extern void __rt_rwlock_init(rwlock_t *r
 		typecheck(unsigned long, flags);	\
 		(void) flags;				\
 		rt_read_unlock(lock);			\
+		migrate_enable();			\
 	} while (0)
 
 #define write_unlock_irqrestore(lock, flags) \

@@ -94,6 +117,7 @@ extern void __rt_rwlock_init(rwlock_t *r
 		typecheck(unsigned long, flags);	\
 		(void) flags;				\
 		rt_write_unlock(lock);			\
+		migrate_enable();			\
 	} while (0)
 
 #endif

@@ -182,10 +182,11 @@ int __lockfunc rt_write_trylock(rwlock_t
 {
 	int ret = rt_mutex_trylock(&rwlock->lock);
 
-	if (ret) {
+	migrate_disable();
+	if (ret)
 		rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
-		migrate_disable();
-	}
+	else
+		migrate_enable();
 
 	return ret;
 }

@@ -196,7 +197,10 @@ int __lockfunc rt_write_trylock_irqsave(
 	int ret;
 
 	*flags = 0;
+	migrate_disable();
 	ret = rt_write_trylock(rwlock);
+	if (!ret)
+		migrate_enable();
 	return ret;
 }
 EXPORT_SYMBOL(rt_write_trylock_irqsave);

@@ -211,18 +215,19 @@ int __lockfunc rt_read_trylock(rwlock_t
 	 * but not when read_depth == 0 which means that the lock is
 	 * write locked.
 	 */
+	migrate_disable();
 	if (rt_mutex_owner(lock) != current) {
 		ret = rt_mutex_trylock(lock);
-		if (ret) {
+		if (ret)
 			rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
-			migrate_disable();
-		}
 	} else if (!rwlock->read_depth) {
 		ret = 0;
 	}
 
 	if (ret)
 		rwlock->read_depth++;
+	else
+		migrate_enable();
 
 	return ret;
 }

@@ -231,7 +236,6 @@ EXPORT_SYMBOL(rt_read_trylock);
 void __lockfunc rt_write_lock(rwlock_t *rwlock)
 {
 	rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
-	migrate_disable();
 	__rt_spin_lock(&rwlock->lock);
 }
 EXPORT_SYMBOL(rt_write_lock);

@@ -246,7 +250,6 @@ void __lockfunc rt_read_lock(rwlock_t *r
 	if (rt_mutex_owner(lock) != current) {
 		rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
 		__rt_spin_lock(lock);
-		migrate_disable();
 	}
 	rwlock->read_depth++;
 }

@@ -258,7 +261,6 @@ void __lockfunc rt_write_unlock(rwlock_t
 	/* NOTE: we always pass in '1' for nested, for simplicity */
 	rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
 	__rt_spin_unlock(&rwlock->lock);
-	migrate_enable();
 }
 EXPORT_SYMBOL(rt_write_unlock);
 

@@ -268,7 +270,6 @@ void __lockfunc rt_read_unlock(rwlock_t
 	if (--rwlock->read_depth == 0) {
 		rwlock_release(&rwlock->dep_map, 1, _RET_IP_);
 		__rt_spin_unlock(&rwlock->lock);
-		migrate_enable();
 	}
 }
 EXPORT_SYMBOL(rt_read_unlock);

Hi Nicholas,

My boxen, including 64 core DL980 that ran hotplug stress for 3 hours yesterday with pre-pushdown rwlocks, say the migrate_disable/enable pushdown patches are very definitely busted.

Instead of whacking selective bits, as I did to verify that the rwlock changes were indeed causing hotplug stress deadlock woes, I'm eyeballing the lot, twiddling primitives to look like I think they should, after which I'll let my boxen express their opinions of the result.

-Mike

migrate_disable-pushd-down-in-atomic_dec_and_spin_lo.patch

bug: migrate_disable() after blocking is too late.

@@ -1028,12 +1028,12 @@ int atomic_dec_and_spin_lock(atomic_t *a

        /* Subtract 1 from counter unless that drops it to 0 (ie. it was 1) */
        if (atomic_add_unless(atomic, -1, 1))
                return 0;
-       migrate_disable();
        rt_spin_lock(lock);
-       if (atomic_dec_and_test(atomic))
+       if (atomic_dec_and_test(atomic)){
+               migrate_disable();
                return 1;
+       }
        rt_spin_unlock(lock);
-       migrate_enable();
        return 0;

}
EXPORT_SYMBOL(atomic_dec_and_spin_lock);

read_lock-migrate_disable-pushdown-to-rt_read_lock.patch

bug: ditto.

@@ -244,8 +246,10 @@ void __lockfunc rt_read_lock(rwlock_t *r

        /*
         * recursive read locks succeed when current owns the lock
         */
-       if (rt_mutex_owner(lock) != current)
+       if (rt_mutex_owner(lock) != current) {
                __rt_spin_lock(lock);
+               migrate_disable();
+       }
        rwlock->read_depth++;

}

Moving that migrate_disable() up will likely fix my hotplug troubles. I'll find out when I get back from physical torture (therapy) session.

-Mike

And for tasklist_lock, seems you also MUST do that prior to trylock as well, else you'll run afoul of the hotplug beast.

-Mike

This lockdep gripe is from the deadlocked crashdump with only the clearly busted bits patched up.

[  193.033224] ======================================================
[  193.033225] [ INFO: possible circular locking dependency detected ]
[  193.033227] 3.12.18-rt25 #19 Not tainted
[  193.033227] -------------------------------------------------------
[  193.033228] boot.kdump/5422 is trying to acquire lock:
[  193.033237]  (&hp->lock){+.+...}, at: [<ffffffff81044974>] pin_current_cpu+0x84/0x1d0
[  193.033238] 
               but task is already holding lock:
[  193.033241]  (tasklist_lock){+.+...}, at: [<ffffffff81046a5b>] do_wait+0xbb/0x2a0
[  193.033242] 
               which lock already depends on the new lock.

[  193.033242] 
               the existing dependency chain (in reverse order) is:
[  193.033244] 
               -> #1 (tasklist_lock){+.+...}:
[  193.033248]        [<ffffffff810ae4a8>] check_prevs_add+0xf8/0x180
[  193.033250]        [<ffffffff810aeada>] validate_chain.isra.45+0x5aa/0x750
[  193.033252]        [<ffffffff810af4f6>] __lock_acquire+0x3f6/0x9f0
[  193.033253]        [<ffffffff810b01bc>] lock_acquire+0x8c/0x160
[  193.033257]        [<ffffffff8155e99c>] rt_write_lock+0x2c/0x40
[  193.033260]        [<ffffffff81548169>] _cpu_down+0x219/0x440
[  193.033261]        [<ffffffff815483c0>] cpu_down+0x30/0x50
[  193.033264]        [<ffffffff813711dc>] cpu_subsys_offline+0x1c/0x30
[  193.033267]        [<ffffffff8136c2d5>] device_offline+0x95/0xc0
[  193.033269]        [<ffffffff8136c3e0>] online_store+0x40/0x80
[  193.033271]        [<ffffffff81369813>] dev_attr_store+0x13/0x30
[  193.033274]        [<ffffffff811c8820>] sysfs_write_file+0xf0/0x170
[  193.033277]        [<ffffffff8115a068>] vfs_write+0xc8/0x1d0
[  193.033279]        [<ffffffff8115a500>] SyS_write+0x50/0xa0
[  193.033282]        [<ffffffff81566ca2>] system_call_fastpath+0x16/0x1b
[  193.033284] 
               -> #0 (&hp->lock){+.+...}:
[  193.033286]        [<ffffffff810ae39d>] check_prev_add+0x7bd/0x7d0
[  193.033287]        [<ffffffff810ae4a8>] check_prevs_add+0xf8/0x180
[  193.033289]        [<ffffffff810aeada>] validate_chain.isra.45+0x5aa/0x750
[  193.033291]        [<ffffffff810af4f6>] __lock_acquire+0x3f6/0x9f0
[  193.033293]        [<ffffffff810b01bc>] lock_acquire+0x8c/0x160
[  193.033295]        [<ffffffff8155e6a5>] rt_spin_lock+0x55/0x70
[  193.033296]        [<ffffffff81044974>] pin_current_cpu+0x84/0x1d0
[  193.033299]        [<ffffffff81079ef1>] migrate_disable+0x81/0x100
[  193.033301]        [<ffffffff8155e947>] rt_read_lock+0x47/0x60
[  193.033303]        [<ffffffff81046a5b>] do_wait+0xbb/0x2a0
[  193.033305]        [<ffffffff8104777e>] SyS_wait4+0x9e/0x100
[  193.033307]        [<ffffffff81566ca2>] system_call_fastpath+0x16/0x1b
[  193.033307] 
               other info that might help us debug this:

[  193.033308]  Possible unsafe locking scenario:

[  193.033309]        CPU0                    CPU1
[  193.033309]        ----                    ----
[  193.033310]   lock(tasklist_lock);
[  193.033312]                                lock(&hp->lock);
[  193.033313]                                lock(tasklist_lock);
[  193.033314]   lock(&hp->lock);
[  193.033315] 
                *** DEADLOCK ***

[  193.033316] 1 lock held by boot.kdump/5422:
[  193.033319]  #0:  (tasklist_lock){+.+...}, at: [<ffffffff81046a5b>] do_wait+0xbb/0x2a0
[  193.033320] 
               stack backtrace:
[  193.033322] CPU: 0 PID: 5422 Comm: boot.kdump Not tainted 3.12.18-rt25 #19
[  193.033323] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[  193.033326]  ffff880200550818 ffff8802004e5ad8 ffffffff8155538c 0000000000000000
[  193.033328]  0000000000000000 ffff8802004e5b28 ffffffff8154d0df ffff8802004e5b18
[  193.033330]  ffff8802004e5b50 ffff880200550818 ffff8802005507e0 ffff880200550818
[  193.033331] Call Trace:
[  193.033335]  [<ffffffff8155538c>] dump_stack+0x4f/0x91
[  193.033337]  [<ffffffff8154d0df>] print_circular_bug+0xd3/0xe4
[  193.033339]  [<ffffffff810ae39d>] check_prev_add+0x7bd/0x7d0
[  193.033342]  [<ffffffff8107e1f5>] ? sched_clock_local+0x25/0x90
[  193.033344]  [<ffffffff8107e388>] ? sched_clock_cpu+0xa8/0x120
[  193.033346]  [<ffffffff810ae4a8>] check_prevs_add+0xf8/0x180
[  193.033348]  [<ffffffff810aeada>] validate_chain.isra.45+0x5aa/0x750
[  193.033350]  [<ffffffff810af4f6>] __lock_acquire+0x3f6/0x9f0
[  193.033352]  [<ffffffff8155da11>] ? rt_spin_lock_slowlock+0x231/0x280
[  193.033354]  [<ffffffff8155d911>] ? rt_spin_lock_slowlock+0x131/0x280
[  193.033356]  [<ffffffff81044974>] ? pin_current_cpu+0x84/0x1d0
[  193.033358]  [<ffffffff810b01bc>] lock_acquire+0x8c/0x160
[  193.033360]  [<ffffffff81044974>] ? pin_current_cpu+0x84/0x1d0
[  193.033362]  [<ffffffff8155e6a5>] rt_spin_lock+0x55/0x70
[  193.033363]  [<ffffffff81044974>] ? pin_current_cpu+0x84/0x1d0
[  193.033365]  [<ffffffff81044974>] pin_current_cpu+0x84/0x1d0
[  193.033367]  [<ffffffff81079ef1>] migrate_disable+0x81/0x100
[  193.033369]  [<ffffffff8155e947>] rt_read_lock+0x47/0x60
[  193.033371]  [<ffffffff81046a5b>] ? do_wait+0xbb/0x2a0
[  193.033373]  [<ffffffff8155cd39>] ? schedule+0x29/0x90
[  193.033374]  [<ffffffff81046a5b>] do_wait+0xbb/0x2a0
[  193.033378]  [<ffffffff8112ded6>] ? might_fault+0x56/0xb0
[  193.033380]  [<ffffffff8104777e>] SyS_wait4+0x9e/0x100
[  193.033382]  [<ffffffff81566cc7>] ? sysret_check+0x1b/0x56
[  193.033384]  [<ffffffff81045d50>] ? task_stopped_code+0xa0/0xa0
[  193.033386]  [<ffffffff81566ca2>] system_call_fastpath+0x16/0x1b
[  193.033845] SMP alternatives: lockdep: fixing up alternatives

Bah. Futzing with dmesg while stress script is running is either a very bad idea, or a very good test. Both virgin 3.10-rt and 3.12-rt with new bugs squashed will deadlock.

Too bad I kept on testing, I liked the notion that hotplug was solid ;-)

-Mike

I was able to stress cpu hotplug on 3.12-rt after applying the following patch.

If there's no complaints about it. I'm going to add this to the 3.12-rt stable tree. As without it, it fails horribly with the cpu hotplug stress test, and I wont release a stable kernel that does that.

-- Steve

index bb72347..4f2a613 100644
@@ -180,12 +180,15 @@ EXPORT_SYMBOL(_mutex_unlock);
  */
 int __lockfunc rt_write_trylock(rwlock_t *rwlock)
 {
-	int ret = rt_mutex_trylock(&rwlock->lock);
+	int ret;
+
+	migrate_disable();
+	ret = rt_mutex_trylock(&rwlock->lock);
 
-	if (ret) {
+	if (ret)
 		rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
-		migrate_disable();
-	}
+	else
+		migrate_enable();
 
 	return ret;
 }

@@ -212,11 +215,12 @@ int __lockfunc rt_read_trylock(rwlock_t *rwlock)
 	 * write locked.
 	 */
 	if (rt_mutex_owner(lock) != current) {
+		migrate_disable();
 		ret = rt_mutex_trylock(lock);
-		if (ret) {
+		if (ret)
 			rwlock_acquire(&rwlock->dep_map, 0, 1, _RET_IP_);
-			migrate_disable();
-		}
+		else
+			migrate_enable();
 	} else if (!rwlock->read_depth) {
 		ret = 0;
 	}

@@ -245,8 +249,8 @@ void __lockfunc rt_read_lock(rwlock_t *rwlock)
 	 */
 	if (rt_mutex_owner(lock) != current) {
 		rwlock_acquire(&rwlock->dep_map, 0, 0, _RET_IP_);
-		__rt_spin_lock(lock);
 		migrate_disable();
+		__rt_spin_lock(lock);
 	}
 	rwlock->read_depth++;
 }

My local boxen are happy, 64 core box with 14-rt seems happy as well, though I couldn't let it burn for long.

BTW, that dmesg business went into hiding. I didn't have time to put virgin 10-rt back on and play around poking both kernels this that and the other way again, but seems there's some phase-of-moon factor there.

-Mike

And 3.12 looks stable on 64 core DL980 as well. (If it survived a 24 hour busy+stress session I'd still likely fall outta my chair though)

My kinda sorta 3.12-rt enterprise to be kernel wasn't stable on DL980, while appearing just fine on small boxen, which made me suspect that there was still a big box something lurking, only raising its ugly head in the fatter kernel. That wasn't an rt problem after all, someone in enterprise land just didn't stack their goody pile quite high enough while wedging upstream into the stable base kernel, which bent rt.

The End.. I hope. I've had enough hotplug entertainment for a while.

-Mike

Not for me. 3.14-rt stress-cpu-hotplug crashes quickly. But it's a different issues than what my patch addressed. I'm still debugging it.

-- Steve

If you didn't fix the two bugs I showed, and (wisely) didn't look at the beautiful lglock patches I posted (no frozen shark, I'm disappointed;), your patch won't help.

-Mike

Mike,

I'm testing it now. But could you please post them as regular patches. They were attachments to this thread, and were not something that stood out.

Thanks,

-- Steve

With your two patches, it still crashes exactly the same way. I probably should remove my debug just in case, but I think this box has another problem with it.

-- Steve

You killed this hunk of hotplug-light-get-online-cpus.patch

@@ -333,7 +449,7 @@ static int __ref _cpu_down(unsigned int

                /* CPU didn't die: tell everyone.  Can't complain. */
                smpboot_unpark_threads(cpu);
                cpu_notify_nofail(CPU_DOWN_FAILED | mod, hcpu);
-               goto out_release;
+               goto out_cancel;
        }
        BUG_ON(cpu_online(cpu));

..and fixed this too?

Another little bug. This hunk of patches/stomp-machine-raw-lock.patch should be while (atomic_read(&done.nr_todo))

@@ -647,7 +671,7 @@ int stop_machine_from_inactive_cpu(int (

        ret = multi_cpu_stop(&msdata);

        /* Busy wait for completion. */
-       while (!completion_done(&done.completion))
+       while (!atomic_read(&done.nr_todo))
                cpu_relax();

        mutex_unlock(&stop_cpus_mutex);

^--- that ! needs to go away

Yes, but it should read "while (atomic_read(&done.nr_todo))"

-Mike

Ah, this would have been better if you had sent a patch. I misread what you talked about.

Yes, this was the culprit of my failures. After removing the '!', it worked.

Care to send a patch :-)

-- Steve

I figured those two were just edit patch, done, but yeah, I can do that.

-Mike

Hah! I knew you were just hiding, you sneaky little SOB ;-)

[50661.070049] smpboot: Booting Node 0 Processor 15 APIC 0x36
[50661.142381] kvm: enabling virtualization on CPU15
[50661.142397] BUG: unable to handle kernel NULL pointer dereference at           (null)
[50661.142417] IP: [<ffffffff810922f1>] wake_up_process+0x1/0x40
[50661.142420] PGD 0
[50661.142422] Oops: 0000 [#1] PREEMPT SMP
[50661.142470] Modules linked in: nfsd(F) lockd(F) nfs_acl(F) auth_rpcgss(F) sunrpc(F) autofs4(F) binfmt_misc(F) edd(F) af_packet(F) bridge(F) stp(F) llc(F) cpufreq_conservative(F) cpufreq_ondemand(F) cpufreq_userspace(F) cpufreq_powersave(F) pcc_cpufreq(F) fuse(F) loop(F) md_mod(F) dm_mod(F) iTCO_wdt(F) iTCO_vendor_support(F) gpio_ich(F) vhost_net(F) macvtap(F) macvlan(F) vhost(F) tun(F) i7core_edac(F) netxen_nic(F) kvm_intel(F) joydev(F) shpchp(F) edac_core(F) hid_generic(F) kvm(F) ipmi_si(F) sr_mod(F) ipmi_msghandler(F) bnx2(F) cdrom(F) sg(F) hpilo(F) hpwdt(F) ehci_pci(F) lpc_ich(F) mfd_core(F) acpi_power_meter(F) pcspkr(F) button(F) ext4(F) jbd2(F) mbcache(F) crc16(F) usbhid(F) uhci_hcd(F) ehci_hcd(F) usbcore(F) sd_mod(F) usb_common(F) thermal(F) processor(F) scsi_dh_rdac(F) scsi_dh_al
 ua(F) scsi_dh_emc(F)
[50661.142475]  scsi_dh_hp_sw(F) scsi_dh(F) ata_generic(F) ata_piix(F) libata(F) cciss(F) hpsa(F) scsi_mod(F)
[50661.142479] CPU: 39 PID: 283 Comm: migration/39 Tainted: GF            3.14.2-rt1 #667
[50661.142481] Hardware name: Hewlett-Packard ProLiant DL980 G7, BIOS P66 07/07/2010
[50661.142482] task: ffff880274515bb0 ti: ffff88027454e000 task.ti: ffff88027454e000
[50661.142486] RIP: 0010:[<ffffffff810922f1>]  [<ffffffff810922f1>] wake_up_process+0x1/0x40
[50661.142487] RSP: 0018:ffff88027454fda8  EFLAGS: 00010002
[50661.142488] RAX: 0000000080000001 RBX: ffff880275581eb8 RCX: 0000000000000000
[50661.142488] RDX: ffffffff81aacec0 RSI: 0000000000000100 RDI: 0000000000000000
[50661.142489] RBP: ffff8802772ee9b0 R08: 0000000000000000 R09: ffffffff81aacec0
[50661.142490] R10: 0000000000000000 R11: ffffffff8103d640 R12: ffffffff810f26c0
[50661.142490] R13: ffff880275581e88 R14: ffff8802772ee9b8 R15: ffff88027454e010
[50661.142492] FS:  0000000000000000(0000) GS:ffff8802772e0000(0000) knlGS:0000000000000000
[50661.142493] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[50661.142494] CR2: 0000000000000000 CR3: 0000000001a0f000 CR4: 00000000000007e0
[50661.142494] Stack:
[50661.142505]  ffff880275581eb8 ffffffff810f2555 ffff880274515bb0 0000000000000005
[50661.142508]  0000000000000001 0000000000000001 0140000000000000 0000000000000001
[50661.142512]  ffff880274515bb0 ffff88027454e000 ffff8802772f4020 0000000000000005
[50661.142512] Call Trace:
[50661.142526]  [<ffffffff810f2555>] ? cpu_stopper_thread+0x125/0x1a0
[50661.142530]  [<ffffffff8108ba2d>] ? smpboot_thread_fn+0x23d/0x320
[50661.142533]  [<ffffffff8108b7f0>] ? smpboot_create_threads+0x70/0x70
[50661.142535]  [<ffffffff8108b7f0>] ? smpboot_create_threads+0x70/0x70
[50661.142543]  [<ffffffff81083c32>] ? kthread+0xd2/0xe0
[50661.142545]  [<ffffffff81083b60>] ? kthreadd+0x330/0x330
[50661.142553]  [<ffffffff815337cc>] ? ret_from_fork+0x7c/0xb0
[50661.142555]  [<ffffffff81083b60>] ? kthreadd+0x330/0x330
[50661.142568] Code: fd ff ff 0f 1f 80 00 00 00 00 31 d2 e9 09 fd ff ff 66 0f 1f 84 00 00 00 00 00 ba 08 00 00 00 be 0f 00 00 00 e9 f1 fc ff ff 90 53 <48> 8b 07 48 89 fb a8 0c 75 08 48 8b 47 08 a8 0c 74 11 be ba 06
[50661.142570] RIP  [<ffffffff810922f1>] wake_up_process+0x1/0x40
[50661.142570]  RSP <ffff88027454fda8>
[50661.142571] CR2: 0000000000000000

What's this from? A new bug that had all the patches applied? Or was this without one of the patches?

-- Steve

alua(F) scsi_dh_emc(F)

It's with all patches applied. It's not new, it has muddied the water during other hunting expeditions. You may never see it in a box with a sane topology, that box is kinda special.

-Mike

I fired off a 100 iteration run on 64 core box. If it's still alive in the morning, it should still be busy as hell.

-Mike

They were meant to not stick out :) I showed what I did to deal with that damn lglock, but showing them at all felt more akin to chumming the waters for frozen sharks than posting patches.

'spose I could try to muster up some courage, showing them put a pretty big dent in my supply.

-Mike

Hi Mike,

I just sent a set of dinky patches with which patch to fold them into, along with my way of dealing with the new stopper_lock.

-Mike

I think the below fixes it (in a more or less minimalist way), but it's not very pretty. Methinks it would be prettier to either clone the x86 percpu + fold logic, or neutralize that optimization completely when PREEMPT_LAZY is enabled.

x86_32 bit is completely untested, x86_64 hasn't exploded.. yet :)

---
 include/linux/preempt.h        |    3 +--
 arch/x86/include/asm/preempt.h |    8 ++++++++
 arch/x86/kernel/asm-offsets.c  |    1 +
 arch/x86/kernel/entry_32.S     |    9 ++++++---
 arch/x86/kernel/entry_64.S     |    7 +++++--
 5 files changed, 21 insertions(+), 7 deletions(-)

@@ -126,8 +126,7 @@ do { \
 #define preempt_enable_notrace() \
 do { \
 	barrier(); \
-	if (unlikely(__preempt_count_dec_and_test() || \
-				test_thread_flag(TIF_NEED_RESCHED_LAZY))) \
+	if (unlikely(__preempt_count_dec_and_test())) \
 		__preempt_schedule_context(); \
 } while (0)
 #else

@@ -94,7 +94,11 @@ static __always_inline bool __preempt_co
 {
 	if (____preempt_count_dec_and_test())
 		return true;
+#ifdef CONFIG_PREEMPT_LAZY
 	return test_thread_flag(TIF_NEED_RESCHED_LAZY);
+#else
+	return false;
+#endif
 }
 
 /*

@@ -102,8 +106,12 @@ static __always_inline bool __preempt_co
  */
 static __always_inline bool should_resched(void)
 {
+#ifdef CONFIG_PREEMPT_LAZY
 	return unlikely(!__this_cpu_read_4(__preempt_count) || \
 			test_thread_flag(TIF_NEED_RESCHED_LAZY));
+#else
+	return unlikely(!__this_cpu_read_4(__preempt_count));
+#endif
 }
 
 #ifdef CONFIG_PREEMPT

@@ -72,4 +72,5 @@ void common(void) {
 
 	BLANK();
 	DEFINE(PTREGS_SIZE, sizeof(struct pt_regs));
+	DEFINE(_PREEMPT_ENABLED, PREEMPT_ENABLED);
 }

@@ -365,19 +365,22 @@ ENTRY(resume_kernel)
 need_resched:
 	# preempt count == 0 + NEED_RS set?
 	cmpl $0,PER_CPU_VAR(__preempt_count)
+#ifndef CONFIG_PREEMPT_LAZY
+	jnz restore_all
+#else
 	jz test_int_off
 
 	# atleast preempt count == 0 ?
-	cmpl $_TIF_NEED_RESCHED,PER_CPU_VAR(__preempt_count)
+	cmpl $_PREEMPT_ENABLED,PER_CPU_VAR(__preempt_count)
 	jne restore_all
 
 	cmpl $0,TI_preempt_lazy_count(%ebp)	# non-zero preempt_lazy_count ?
 	jnz restore_all
 
-	testl $_TIF_NEED_RESCHED_LAZY, %ecx
+	testl $_TIF_NEED_RESCHED_LAZY, TI_flags(%ebp)
 	jz restore_all
-
 test_int_off:
+#endif
 	testl $X86_EFLAGS_IF,PT_EFLAGS(%esp)	# interrupts off (exception path) ?
 	jz restore_all
 	call preempt_schedule_irq

@@ -1104,10 +1104,13 @@ ENTRY(native_iret)
 	/* rcx:	 threadinfo. interrupts off. */
 ENTRY(retint_kernel)
 	cmpl $0,PER_CPU_VAR(__preempt_count)
+#ifndef CONFIG_PREEMPT_LAZY
+	jnz  retint_restore_args
+#else
 	jz  check_int_off
 
 	# atleast preempt count == 0 ?
-	cmpl $_TIF_NEED_RESCHED,PER_CPU_VAR(__preempt_count)
+	cmpl $_PREEMPT_ENABLED,PER_CPU_VAR(__preempt_count)
 	jnz retint_restore_args
 
 	cmpl $0, TI_preempt_lazy_count(%rcx)

@@ -1115,8 +1118,8 @@ ENTRY(retint_kernel)
 
 	bt $TIF_NEED_RESCHED_LAZY,TI_flags(%rcx)
 	jnc  retint_restore_args

This patch makes sense to me.

-- Steve

Turning lockdep on, it says it's busted.

(I'll go stare at it, maybe the beast will blink first for a change)

[    0.000000] Lock dependency validator: Copyright (c) 2006 Red Hat, Inc., Ingo Molnar
[    0.000000] ... MAX_LOCKDEP_SUBCLASSES:  8
[    0.000000] ... MAX_LOCK_DEPTH:          48
[    0.000000] ... MAX_LOCKDEP_KEYS:        8191
[    0.000000] ... CLASSHASH_SIZE:          4096
[    0.000000] ... MAX_LOCKDEP_ENTRIES:     16384
[    0.000000] ... MAX_LOCKDEP_CHAINS:      32768
[    0.000000] ... CHAINHASH_SIZE:          16384
[    0.000000]  memory used by lock dependency info: 6367 kB
[    0.000000]  per task-struct memory footprint: 2688 bytes
[    0.000000] ------------------------
[    0.000000] | Locking API testsuite:
[    0.000000] ----------------------------------------------------------------------------
[    0.000000]                                  | spin |wlock |rlock |mutex | wsem | rsem |
[    0.000000]   --------------------------------------------------------------------------
[    0.000000]                      A-A deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e710f>] locking_selftest+0xdf/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]                  A-B-B-A deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e719e>] locking_selftest+0x16e/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]              A-B-B-C-C-A deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e722d>] locking_selftest+0x1fd/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]              A-B-C-A-B-C deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e72bc>] locking_selftest+0x28c/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]          A-B-B-C-C-D-D-A deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e734b>] locking_selftest+0x31b/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]          A-B-C-D-B-D-D-A deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e73da>] locking_selftest+0x3aa/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]          A-B-C-D-B-C-D-A deadlock:  ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e7469>] locking_selftest+0x439/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |  ok  |
[    0.000000]                     double unlock:  ok  |
[    0.000000] ------------[ cut here ]------------
[    0.000000] WARNING: CPU: 0 PID: 0 at kernel/sched/core.c:2660 migrate_disable+0xbd/0xd0()
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000a64 ffffffff81a01d38 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000000 ffffffff81a01d78 ffffffff8104f0cc ffffffff81a01d78
[    0.000000]  ffffffff81a194c0 0000000000000027 0000000000000006 0000000000000001
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff8104f0cc>] warn_slowpath_common+0x8c/0xc0
[    0.000000]  [<ffffffff8104f11a>] warn_slowpath_null+0x1a/0x20
[    0.000000]  [<ffffffff8108795d>] migrate_disable+0xbd/0xd0
[    0.000000]  [<ffffffff810b7a7f>] call_console_drivers.constprop.20+0x4f/0x140
[    0.000000]  [<ffffffff810b823f>] console_unlock.part.15+0x1df/0x2e0
[    0.000000]  [<ffffffff815ec835>] ? _raw_spin_unlock+0x35/0x60
[    0.000000]  [<ffffffff810b8358>] console_unlock+0x18/0x30
[    0.000000]  [<ffffffff810b8771>] vprintk_emit+0x231/0x400
[    0.000000]  [<ffffffff815e9aca>] ? preempt_schedule+0x4a/0x70
[    0.000000]  [<ffffffff812e1320>] ? bad_unlock_order_spin+0x40/0x40
[    0.000000]  [<ffffffff815d8519>] printk+0x4d/0x4f
[    0.000000]  [<ffffffff815f11d9>] ? preempt_count_sub+0x29/0x70
[    0.000000]  [<ffffffff812e1348>] ? double_unlock_spin+0x28/0x30
[    0.000000]  [<ffffffff815e1dc8>] dotest+0x75/0xc7
[    0.000000]  [<ffffffff812e74cf>] locking_selftest+0x49f/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000] ---[ end trace 0000000000000001 ]---
[    0.000000] ------------[ cut here ]------------
[    0.000000] WARNING: CPU: 0 PID: 0 at kernel/sched/core.c:2693 migrate_enable+0xf6/0x1a0()
[    0.000000] Modules linked in:
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000a85 ffffffff81a01cb8 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000000 ffffffff81a01cf8 ffffffff8104f0cc ffffffff81a01d18
[    0.000000]  ffffffff81a194c0 0000000000000000 000000000000000a ffffffff8275f367
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff8104f0cc>] warn_slowpath_common+0x8c/0xc0
[    0.000000]  [<ffffffff8104f11a>] warn_slowpath_null+0x1a/0x20
[    0.000000]  [<ffffffff810877f6>] migrate_enable+0xf6/0x1a0
[    0.000000]  [<ffffffff813aab31>] vt_console_print+0x2f1/0x3d0
[    0.000000]  [<ffffffff810b7ae7>] call_console_drivers.constprop.20+0xb7/0x140
[    0.000000]  [<ffffffff810b823f>] console_unlock.part.15+0x1df/0x2e0
[    0.000000]  [<ffffffff815ec835>] ? _raw_spin_unlock+0x35/0x60
[    0.000000]  [<ffffffff810b8358>] console_unlock+0x18/0x30
[    0.000000]  [<ffffffff810b8771>] vprintk_emit+0x231/0x400
[    0.000000]  [<ffffffff815e9aca>] ? preempt_schedule+0x4a/0x70
[    0.000000]  [<ffffffff812e1320>] ? bad_unlock_order_spin+0x40/0x40
[    0.000000]  [<ffffffff815d8519>] printk+0x4d/0x4f
[    0.000000]  [<ffffffff815f11d9>] ? preempt_count_sub+0x29/0x70
[    0.000000]  [<ffffffff812e1348>] ? double_unlock_spin+0x28/0x30
[    0.000000]  [<ffffffff815e1dc8>] dotest+0x75/0xc7
[    0.000000]  [<ffffffff812e74cf>] locking_selftest+0x49f/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000] ---[ end trace 0000000000000002 ]---
[    0.000000]   ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 0000000000000046
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e74f5>] locking_selftest+0x4c5/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]   ok  |  ok  |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000008 ffffffff81a01f28 ffffffff815e12a5 0000000000000046
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e752e>] locking_selftest+0x4fe/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000] 
[    0.000000]                   initialize held:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
[    0.000000]                  bad unlock order:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
[    0.000000]   --------------------------------------------------------------------------
[    0.000000]               recursive read-lock:             |  ok  |             |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000008 ffffffff81a01f28 ffffffff815e12a5 0000000000000006
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e76c5>] locking_selftest+0x695/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000] 
[    0.000000]            recursive read-lock #2:             |FAILED|
[    0.000000] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        W    3.14.1-rt1 #16
[    0.000000] Hardware name: MEDIONPC MS-7502/MS-7502, BIOS 6.00 PG 12/26/2007
[    0.000000]  0000000000000002 ffffffff81a01f28 ffffffff815e12a5 ffffffff810b7727
[    0.000000]  0000000000000001 ffffffff81a01f58 ffffffff815e1db2 0000000000000000
[    0.000000]  0000000000000000 0000000000000000 0000000000000000 ffffffff81a01f68
[    0.000000] Call Trace:
[    0.000000]  [<ffffffff815e12a5>] dump_stack+0x4f/0x7c
[    0.000000]  [<ffffffff810b7727>] ? console_trylock_for_printk+0x37/0xf0
[    0.000000]  [<ffffffff815e1db2>] dotest+0x5f/0xc7
[    0.000000]  [<ffffffff812e7703>] locking_selftest+0x6d3/0xb30
[    0.000000]  [<ffffffff81d51d8f>] start_kernel+0x215/0x327
[    0.000000]  [<ffffffff81d51a0c>] ? repair_env_string+0x5a/0x5a
[    0.000000]  [<ffffffff81d9f4c1>] ? memblock_reserve+0x49/0x4e
[    0.000000]  [<ffffffff81d515b2>] x86_64_start_reservations+0x2a/0x2c
[    0.000000]  [<ffffffff81d516a4>] x86_64_start_kernel+0xf0/0xf7
[    0.000000]              |  ok  |
[    0.000000]             mixed read-write-lock:             |  ok  |             |  ok  |
[    0.000000]             mixed write-read-lock:             |  ok  |             |  ok  |
[    0.000000]   --------------------------------------------------------------------------
[    0.000000]      hard-irqs-on + irq-safe-A/12:  ok  |
[    0.000000]      hard-irqs-on + irq-safe-A/21:  ok  |
[    0.000000]          hard-safe-A + irqs-on/12:  ok  |
[    0.000000]          hard-safe-A + irqs-on/21:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #1/123:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #1/132:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #1/213:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #1/231:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #1/312:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #1/321:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #2/123:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #2/132:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #2/213:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #2/231:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #2/312:  ok  |
[    0.000000]     hard-safe-A + unsafe-B #2/321:  ok  |
[    0.000000]   --------------------------------------------------------------------------
[    0.000000]   | Wound/wait tests |
[    0.000000]   ---------------------
[    0.000000]                   ww api failures:  ok  |  ok  |  ok  |
[    0.000000]                ww contexts mixing:  ok  |  ok  |
[    0.000000]              finishing ww context:  ok  |  ok  |  ok  |  ok  |
[    0.000000]                locking mismatches:  ok  |  ok  |  ok  |
[    0.000000]                  EDEADLK handling:  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |  ok  |
[    0.000000]            spinlock nest unlocked:  ok  |
[    0.000000]   -----------------------------------------------------
[    0.000000]                                  |block | try  |context|
[    0.000000]   -----------------------------------------------------
[    0.000000]                           context:  ok  |  ok  |  ok  |
[    0.000000]                               try:  ok  |  ok  |  ok  |
[    0.000000]                             block:  ok  |  ok  |  ok  |
[    0.000000]                          spinlock:  ok  |  ok  |  ok  |
[    0.000000] -----------------------------------------------------------------
[    0.000000] BUG:  11 unexpected failures (out of 119) - debugging disabled! |
[    0.000000] -----------------------------------------------------------------

http://www.spinics.net/lists/linux-rt-users/msg11179.html

Sebastian

I was heading toward the same conclusion while regression testing. Guess I can stop that.

-Mike

Saw this a moment ago (3.14.1 + rt1, Fedora 19 laptop - I think I have seen something similar in 3.12.x-r):

Apr 26 11:16:11 localhost kernel: [   96.323248] ------------[ cut here ]------------
Apr 26 11:16:11 localhost kernel: [   96.323262] WARNING: CPU: 0 PID: 2051 at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0()
Apr 26 11:16:11 localhost kernel: [   96.323264] list_del corruption. prev->next should be ffff8802101196a0, but was 0000000000000001
Apr 26 11:16:11 localhost kernel: [   96.323266] Modules linked in: fuse ipt_MASQUERADE xt_CHECKSUM tun ip6t_rpfilter ip6t_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw rfcomm ip6table_filter bnep ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel uvcvideo videobuf2_vmalloc microcode videobuf2_memops snd_hda_codec_hdmi videobuf2_core videodev media serio_raw btusb bluetooth intel_ips i2c_i801 6lowpan_iphc snd_hda_codec_conexant snd_hda_codec_generic arc4 iwldvm mac80211 iwlwifi lpc_ich sdhci_pci mfd_core sdhci cfg80211 mmc_core snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm e1000e snd_timer ptp mei_me pps_core mei shpchp thinkpad_acpi snd ppdev soundcore rfkill parport_pc parport acpi_cpufreq uinput firewire_ohci nouveau firewire_core crc_itu_t i2c_algo_bit drm_kms_helper ttm drm mxm_wmi i2c_core wmi video
Apr 26 11:16:11 localhost kernel: [   96.323331] CPU: 0 PID: 2051 Comm: cinnamon Not tainted 3.14.1-200.rt1.1.fc19.ccrma.x86_64+rt #1
Apr 26 11:16:11 localhost kernel: [   96.323332] Hardware name: LENOVO 4313CTO/4313CTO, BIOS 6MET64WW (1.27 ) 07/15/2010
Apr 26 11:16:11 localhost kernel: [   96.323334]  0000000000000000 000000008a5c11dc ffff8800ae715a88 ffffffff81707fca
Apr 26 11:16:11 localhost kernel: [   96.323336]  ffff8800ae715ad0 ffff8800ae715ac0 ffffffff8108d03d ffff8802101196a0
Apr 26 11:16:11 localhost kernel: [   96.323337]  ffff880210119b50 ffff880210119b50 ffff880210119b40 ffff88021a615648
Apr 26 11:16:11 localhost kernel: [   96.323338] Call Trace:
Apr 26 11:16:11 localhost kernel: [   96.323345]  [<ffffffff81707fca>] dump_stack+0x4d/0x82
Apr 26 11:16:11 localhost kernel: [   96.323351]  [<ffffffff8108d03d>] warn_slowpath_common+0x7d/0xc0
Apr 26 11:16:11 localhost kernel: [   96.323352]  [<ffffffff8108d0dc>] warn_slowpath_fmt+0x5c/0x80
Apr 26 11:16:11 localhost kernel: [   96.323354]  [<ffffffff8137c551>] __list_del_entry+0xa1/0xd0
Apr 26 11:16:11 localhost kernel: [   96.323355]  [<ffffffff8137c58d>] list_del+0xd/0x30
Apr 26 11:16:11 localhost kernel: [   96.323393]  [<ffffffffa0135593>] nouveau_fence_signal+0x53/0x80 [nouveau]
Apr 26 11:16:11 localhost kernel: [   96.323414]  [<ffffffffa0135678>] nouveau_fence_update+0x48/0xa0 [nouveau]
Apr 26 11:16:11 localhost kernel: [   96.323435]  [<ffffffffa0135f85>] nouveau_fence_sync+0x45/0x80 [nouveau]
Apr 26 11:16:11 localhost kernel: [   96.323456]  [<ffffffffa013aea8>] validate_list+0xd8/0x2e0 [nouveau]
Apr 26 11:16:11 localhost kernel: [   96.323478]  [<ffffffffa013c3d3>] nouveau_gem_ioctl_pushbuf+0xaa3/0x13e0 [nouveau]
Apr 26 11:16:11 localhost kernel: [   96.323500]  [<ffffffffa002ad02>] drm_ioctl+0x4f2/0x620 [drm]
Apr 26 11:16:11 localhost kernel: [   96.323506]  [<ffffffff810c1af4>] ? migrate_enable+0x94/0x1c0
Apr 26 11:16:11 localhost kernel: [   96.323527]  [<ffffffffa0132cfe>] nouveau_drm_ioctl+0x4e/0x90 [nouveau]
Apr 26 11:16:11 localhost kernel: [   96.323530]  [<ffffffff81203480>] do_vfs_ioctl+0x2e0/0x4c0
Apr 26 11:16:11 localhost kernel: [   96.323533]  [<ffffffff812fd8d6>] ? file_has_perm+0xa6/0xb0
Apr 26 11:16:11 localhost kernel: [   96.323535]  [<ffffffff812036e1>] SyS_ioctl+0x81/0xa0
Apr 26 11:16:11 localhost kernel: [   96.323538]  [<ffffffff81716769>] system_call_fastpath+0x16/0x1b
Apr 26 11:16:11 localhost kernel: [   96.323569] ---[ end trace 0000000000000002 ]---

-- Fernando

Yes, you did: https://lkml.org/lkml/2014/3/7/163
You did not test I've sent. Care to do so?

and please send backtrace information properly formatted. This is terrible hard to read.

Sebastian

I did patch my kernel and (I think) I did not see the problem again. I did get some very occassional hangs that seemed to be video related but I think I could not see what had caused them.

Sorry about that, I will attach files in the future.

I re-patched 3.14.3-rt5 with a slightly tweaked version of you patch. Will see what happens and report back. -- Fernando

...

I got another freeze a moment ago. I'm attaching a text file with the oops, the patch I added to rt5, and the kernel configuration...

-- Fernando

May 15 12:51:27 localhost kernel: [ 9282.171060] ------------[ cut here ]------------
May 15 12:51:27 localhost kernel: [ 9282.171079] WARNING: CPU: 0 PID: 2049 at lib/list_debug.c:59 __list_del_entry+0xa1/0xd0()
May 15 12:51:27 localhost kernel: [ 9282.171082] list_del corruption. prev->next should be ffff880211174c60, but was dead000000100100
May 15 12:51:27 localhost kernel: [ 9282.171083] Modules linked in: fuse ipt_MASQUERADE xt_CHECKSUM tun ip6t_rpfilter ip6t_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat rfcomm nf_conntrack iptable_mangle bnep iptable_security iptable_raw iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel microcode uvcvideo serio_raw videobuf2_vmalloc videobuf2_memops snd_hda_codec_hdmi videobuf2_core videodev intel_ips i2c_i801 media btusb bluetooth 6lowpan_iphc arc4 iwldvm snd_hda_codec_conexant snd_hda_codec_generic mac80211 iwlwifi sdhci_pci sdhci cfg80211 mmc_core lpc_ich mfd_core snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer e1000e mei_me ptp mei thinkpad_acpi pps_core shpchpnouveau E[  PGRAPH][0000:01:00.0] TRAP_PROP - TP 0 - ppdev snd soundcore rfkill parport_pc parport acpi_cpufreq uinput nouveau i2c_algo_bit drm_kms_helper firewire_ohci mxm_wmi firewire_core ttm
May 15 12:51:27 localhost kernel: [ 9282.171136]  DST2D_FAULT
May 15 12:51:27 localhost kernel: [ 9282.171139]  crc_itu_t drm i2c_core
May 15 12:51:27 localhost kernel: [ 9282.171139]  - Address 0020a92000
May 15 12:51:27 localhost kernel: [ 9282.171141]  wmi video
May 15 12:51:27 localhost kernel: [ 9282.171144] CPU: 0 PID: 2049 Comm: cinnamon Tainted: G        W    3.14.3-200.rt5.1.fc19.ccrma.x86_64+rt #1
May 15 12:51:27 localhost kernel: [ 9282.171145] nouveau E[  PGRAPH][0000:01:00.0] TRAP_PROP - TP 0 - e0c: 00000000, e18: 00000000, e1c: 00000000, e20: 00000011, e24: 0c030000
May 15 12:51:27 localhost kernel: [ 9282.171146] Hardware name: LENOVO 4313CTO/4313CTO, BIOS 6MET64WW (1.27 ) 07/15/2010
May 15 12:51:27 localhost kernel: [ 9282.171149]  0000000000000000 000000006a3e8229 ffff8800ada91a88 ffffffff81703b57
May 15 12:51:27 localhost kernel: [ 9282.171151]  ffff8800ada91ad0 ffff8800ada91ac0
May 15 12:51:27 localhost kernel: [ 9282.171151] nouveau E[  PGRAPH][0000:01:00.0] ch 3 [0x001fb14000 Xorg[1396]] subc 2 class 0x502d mthd 0x060c data 0x00000022
May 15 12:51:27 localhost kernel: [ 9282.171153]  ffffffff8108bc2d ffff880211174c60
May 15 12:51:27 localhost kernel: [ 9282.171156]  ffff880211174d50 ffff880211174d50 ffff880211174d40 ffff88021b73be48
May 15 12:51:27 localhost kernel: [ 9282.171156] Call Trace:
May 15 12:51:27 localhost kernel: [ 9282.171164]  [<ffffffff81703b57>] dump_stack+0x4d/0x6f
May 15 12:51:27 localhost kernel: [ 9282.171170]  [<ffffffff8108bc2d>] warn_slowpath_common+0x7d/0xc0
May 15 12:51:27 localhost kernel: [ 9282.171171] nouveau E[     PFB][0000:01:00.0] trapped write at 0x0020a92000 on channel 0x0001fb14 [Xorg[1396]] 
May 15 12:51:27 localhost kernel: [ 9282.171173]  [<ffffffff8108bccc>] warn_slowpath_fmt+0x5c/0x80
May 15 12:51:27 localhost kernel: [ 9282.171176]  [<ffffffff81376581>] __list_del_entry+0xa1/0xd0
May 15 12:51:27 localhost kernel: [ 9282.171178]  [<ffffffff813765bd>] list_del+0xd/0x30
May 15 12:51:27 localhost kernel: [ 9282.171181] PGRAPH/
May 15 12:51:27 localhost kernel: [ 9282.171181]  [<ffffffffa0145593>] nouveau_fence_signal+0x53/0x80 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.171219] PROP/ [<ffffffffa0145678>] nouveau_fence_update+0x48/0xa0 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.171260]  [<ffffffffa0145fb5>] nouveau_fence_sync+0x45/0x80 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.171282] DST2D
May 15 12:51:27 localhost kernel: [ 9282.171282]  [<ffffffffa014aed8>] validate_list+0xd8/0x2e0 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.171304]  [<ffffffffa014c403>] nouveau_gem_ioctl_pushbuf+0xaa3/0x13e0 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.171305]  reason: PAGE_NOT_PRESENT
May 15 12:51:27 localhost kernel: [ 9282.171318]  [<ffffffffa002ad02>] drm_ioctl+0x4f2/0x620 [drm]
May 15 12:51:27 localhost kernel: [ 9282.171323]  [<ffffffff810bcf44>] ? migrate_enable+0x94/0x1a0
May 15 12:51:27 localhost kernel: [ 9282.171344]  [<ffffffffa0142cfe>] nouveau_drm_ioctl+0x4e/0x90 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.171347]  [<ffffffff811feb50>] do_vfs_ioctl+0x2e0/0x4c0
May 15 12:51:27 localhost kernel: [ 9282.171349]  [<ffffffff812f7da6>] ? file_has_perm+0xa6/0xb0
May 15 12:51:27 localhost kernel: [ 9282.171351]  [<ffffffff810bb701>] ? __sched_fork+0x171/0x220
May 15 12:51:27 localhost kernel: [ 9282.171352]  [<ffffffff811fedb1>] SyS_ioctl+0x81/0xa0
May 15 12:51:27 localhost kernel: [ 9282.171355]  [<ffffffff81712069>] system_call_fastpath+0x16/0x1b
May 15 12:51:27 localhost kernel: [ 9282.171356] ---[ end trace 0000000000000006 ]---
May 15 12:51:27 localhost kernel: [ 9282.172554] nouveau E[  PGRAPH][0000:01:00.0] TRAP_PROP - TP 0 - RT_FAULT - Address 0020a92000
May 15 12:51:27 localhost kernel: [ 9282.172561] nouveau E[  PGRAPH][0000:01:00.0] TRAP_PROP - TP 0 - e0c: 00000000, e18: 00000000, e1c: 00000000, e20: 00002a00, e24: 00030000
May 15 12:51:27 localhost kernel: [ 9282.172566] nouveau E[  PGRAPH][0000:01:00.0] ch 3 [0x001fb14000 Xorg[1396]] subc 7 class 0x8597 mthd 0x15e0 data 0x00000000
May 15 12:51:27 localhost kernel: [ 9282.172575] nouveau E[     PFB][0000:01:00.0] trapped write at 0x0020a92000 on channel 0x0001fb14 [Xorg[1396]] PGRAPH/PROP/RT0 reason: PAGE_NOT_PRESENT
May 15 12:51:27 localhost kernel: [ 9282.172616] nouveau E[  PGRAPH][0000:01:00.0] TRAP_PROP - TP 0 - RT_FAULT - Address 0020a92000
May 15 12:51:27 localhost kernel: [ 9282.172621] nouveau E[  PGRAPH][0000:01:00.0] TRAP_PROP - TP 0 - e0c: 00000000, e18: 00000000, e1c: 00000000, e20: 00002a00, e24: 00030000
May 15 12:51:27 localhost kernel: [ 9282.172625] nouveau E[  PGRAPH][0000:01:00.0] ch 3 [0x001fb14000 Xorg[1396]] subc 7 class 0x8597 mthd 0x15e0 data 0x00000000
May 15 12:51:27 localhost kernel: [ 9282.172635] nouveau E[     PFB][0000:01:00.0] trapped write at 0x0020a92000 on channel 0x0001fb14 [Xorg[1396]] PGRAPH/PROP/RT0 reason: PAGE_NOT_PRESENT
May 15 12:51:27 localhost kernel: [ 9282.172656] nouveau E[  PGRAPH][0000:01:00.0] magic set 0:
May 15 12:51:27 localhost kernel: [ 9282.172659] nouveau E[  PGRAPH][0000:01:00.0] 	0x00408604: 0x20095203
May 15 12:51:27 localhost kernel: [ 9282.172662] nouveau E[  PGRAPH][0000:01:00.0] 	0x00408608: 0x0020a958
May 15 12:51:27 localhost kernel: [ 9282.172665] nouveau E[  PGRAPH][0000:01:00.0] 	0x0040860c: 0x80000432
May 15 12:51:27 localhost kernel: [ 9282.172668] nouveau E[  PGRAPH][0000:01:00.0] 	0x00408610: 0xa9200000
May 15 12:51:27 localhost kernel: [ 9282.172671] nouveau E[  PGRAPH][0000:01:00.0] TRAP_TEXTURE - TP0: Unhandled ustatus 0x00000003
May 15 12:51:27 localhost kernel: [ 9282.172674] nouveau E[  PGRAPH][0000:01:00.0] ch 3 [0x001fb14000 Xorg[1396]] subc 2 class 0x502d mthd 0x08dc data 0x00000000
May 15 12:51:27 localhost kernel: [ 9282.172685] nouveau E[     PFB][0000:01:00.0] trapped read at 0x0020a92200 on channel 0x0001fb14 [Xorg[1396]] PGRAPH/TEXTURE/00 reason: PAGE_NOT_PRESENT
May 15 12:51:27 localhost kernel: [ 9282.240139] BUG: unable to handle kernel NULL pointer dereference at 0000000000000009
May 15 12:51:27 localhost kernel: [ 9282.240149] IP: [<ffffffff81376511>] __list_del_entry+0x31/0xd0
May 15 12:51:27 localhost kernel: [ 9282.240152] PGD 227e4e067 PUD 227e4d067 PMD 0 
May 15 12:51:27 localhost kernel: [ 9282.240154] Oops: 0000 [#1] PREEMPT SMP 
May 15 12:51:27 localhost kernel: [ 9282.240188] Modules linked in: fuse ipt_MASQUERADE xt_CHECKSUM tun ip6t_rpfilter ip6t_REJECT xt_conntrack ebtable_nat ebtable_broute bridge stp llc ebtable_filter ebtables ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw ip6table_filter ip6_tables iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat rfcomm nf_conntrack iptable_mangle bnep iptable_security iptable_raw iTCO_wdt iTCO_vendor_support coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel microcode uvcvideo serio_raw videobuf2_vmalloc videobuf2_memops snd_hda_codec_hdmi videobuf2_core videodev intel_ips i2c_i801 media btusb bluetooth 6lowpan_iphc arc4 iwldvm snd_hda_codec_conexant snd_hda_codec_generic mac80211 iwlwifi sdhci_pci sdhci cfg80211 mmc_core lpc_ich mfd_core snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_timer e1000e mei_me ptp mei thinkpad_acpi pps_core shpchp ppdev snd soundcore rfkill parport_pc parport acpi_cpufreq uinput nouveau i2c_algo_bit drm_kms_helper firewire_ohci mxm_wmi firewire_core ttm crc_itu_t drm i2c_core wmi video
May 15 12:51:27 localhost kernel: [ 9282.240206] CPU: 0 PID: 2049 Comm: cinnamon Tainted: G        W    3.14.3-200.rt5.1.fc19.ccrma.x86_64+rt #1
May 15 12:51:27 localhost kernel: [ 9282.240207] Hardware name: LENOVO 4313CTO/4313CTO, BIOS 6MET64WW (1.27 ) 07/15/2010
May 15 12:51:27 localhost kernel: [ 9282.240208] task: ffff880227cfc750 ti: ffff8800ada90000 task.ti: ffff8800ada90000
May 15 12:51:27 localhost kernel: [ 9282.240211] RIP: 0010:[<ffffffff81376511>]  [<ffffffff81376511>] __list_del_entry+0x31/0xd0
May 15 12:51:27 localhost kernel: [ 9282.240212] RSP: 0018:ffff8800ada91b38  EFLAGS: 00010246
May 15 12:51:27 localhost kernel: [ 9282.240213] RAX: ffff8800a79fcf50 RBX: ffff8800a79fc3a0 RCX: dead000000200200
May 15 12:51:27 localhost kernel: [ 9282.240214] RDX: 0000000000000001 RSI: 0000000000000286 RDI: ffff8800a79fc3a0
May 15 12:51:27 localhost kernel: [ 9282.240214] RBP: ffff8800ada91b38 R08: ffff8800a79fc3a0 R09: 0000000000000000
May 15 12:51:27 localhost kernel: [ 9282.240215] R10: ffffea00029e2d40 R11: ffffffffa01456c8 R12: ffff8800a79fc820
May 15 12:51:27 localhost kernel: [ 9282.240216] R13: ffff8800a79fcf50 R14: ffff8800a79fcf40 R15: ffff88021b73be48
May 15 12:51:27 localhost kernel: [ 9282.240217] FS:  00007fc079062a40(0000) GS:ffff88023bc00000(0000) knlGS:0000000000000000
May 15 12:51:27 localhost kernel: [ 9282.240218] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
May 15 12:51:27 localhost kernel: [ 9282.240219] CR2: 0000000000000009 CR3: 000000022834f000 CR4: 00000000000007f0
May 15 12:51:27 localhost kernel: [ 9282.240220] Stack:
May 15 12:51:27 localhost kernel: [ 9282.240222]  ffff8800ada91b50 ffffffff813765bd ffff8800a79fc800 ffff8800ada91b80
May 15 12:51:27 localhost kernel: [ 9282.240223]  ffffffffa0145593 ffff8800a79fcf40 ffff8800a79fc340 ffff88021b73be00
May 15 12:51:27 localhost kernel: [ 9282.240225]  ffff88021742ab40 ffff8800ada91bb8 ffffffffa0145678 ffff8800a79fc340
May 15 12:51:27 localhost kernel: [ 9282.240225] Call Trace:
May 15 12:51:27 localhost kernel: [ 9282.240229]  [<ffffffff813765bd>] list_del+0xd/0x30
May 15 12:51:27 localhost kernel: [ 9282.240268]  [<ffffffffa0145593>] nouveau_fence_signal+0x53/0x80 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.240295]  [<ffffffffa0145678>] nouveau_fence_update+0x48/0xa0 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.240321]  [<ffffffffa0145fb5>] nouveau_fence_sync+0x45/0x80 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.240347]  [<ffffffffa014aed8>] validate_list+0xd8/0x2e0 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.240374]  [<ffffffffa014c403>] nouveau_gem_ioctl_pushbuf+0xaa3/0x13e0 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.240386]  [<ffffffffa002ad02>] drm_ioctl+0x4f2/0x620 [drm]
May 15 12:51:27 localhost kernel: [ 9282.240393]  [<ffffffff810bcf44>] ? migrate_enable+0x94/0x1a0
May 15 12:51:27 localhost kernel: [ 9282.240418]  [<ffffffffa0142cfe>] nouveau_drm_ioctl+0x4e/0x90 [nouveau]
May 15 12:51:27 localhost kernel: [ 9282.240421]  [<ffffffff811feb50>] do_vfs_ioctl+0x2e0/0x4c0
May 15 12:51:27 localhost kernel: [ 9282.240424]  [<ffffffff812f7da6>] ? file_has_perm+0xa6/0xb0
May 15 12:51:27 localhost kernel: [ 9282.240426]  [<ffffffff811fedb1>] SyS_ioctl+0x81/0xa0
May 15 12:51:27 localhost kernel: [ 9282.240429]  [<ffffffff81129566>] ? __audit_syscall_exit+0x1f6/0x2a0
May 15 12:51:27 localhost kernel: [ 9282.240432]  [<ffffffff81712069>] system_call_fastpath+0x16/0x1b
May 15 12:51:27 localhost kernel: [ 9282.240451] Code: 00 01 10 00 00 00 ad de 48 8b 47 08 48 89 e5 48 39 ca 74 29 48 b9 00 02 20 00 00 00 ad de 48 39 c8 74 7a 4c 8b 00 4c 39 c7 75 53 <4c> 8b 42 08 4c 39 c7 75 2b 48 89 42 08 48 89 10 5d c3 49 89 d0 
May 15 12:51:27 localhost kernel: [ 9282.240453] RIP  [<ffffffff81376511>] __list_del_entry+0x31/0xd0
May 15 12:51:27 localhost kernel: [ 9282.240453]  RSP <ffff8800ada91b38>
May 15 12:51:27 localhost kernel: [ 9282.240454] CR2: 0000000000000009
May 15 12:51:27 localhost kernel: [ 9282.247071] ---[ end trace 0000000000000007 ]---

@@ -178,11 +178,17 @@
 
 {
 	struct nouveau_channel *chan = fence->channel;
-	struct nouveau_fifo *pfifo = nouveau_fifo(chan->drm->device);
-	struct nouveau_fence_priv *priv = chan->drm->fence;
+	struct nouveau_fifo *pfifo;
+	struct nouveau_fence_priv *priv;
 	struct nouveau_eventh *handler;
 	int ret = 0;
 
+	if (WARN_ON_ONCE(!chan))
+		return 0;
+
+	pfifo = nouveau_fifo(chan->drm->device);
+	priv = chan->drm->fence;
+
 	ret = nouveau_event_new(pfifo->uevent, 0,
 				nouveau_fence_wait_uevent_handler,
 				priv, &handler);

Hello,

After applied the 3.14-rt1 patch set, one compiling error occurred.

Compile failed with the default arch/x86/configs/x86_64_defconfig on my laptop(Intel(R) Core(TM) i5-3230M CPU). The configure command is 'make defconfig'.
Error messages:
linux-3.14/kernel/softirq.c: In function ‘__local_bh_enable_ip’:
linux-3.14/kernel/softirq.c:328:2: error: implicit declaration of fu nction ‘preempt_check_resched’ [-Werror=implicit-function-declaration]
cc1: some warnings being treated as errors

The reason is when configured with 'CONFIG_PREEMPT_VOLUNTARY', the compiler can not find the definiation of function preempt_check_resched. This patch is supplied to fix this problem.

---
 include/linux/preempt.h |    1 +
 1 file changed, 1 insertion(+)

index 116af6a..7d4f557 100644
@@ -155,6 +155,7 @@ do { \
 #define preempt_enable_no_resched_notrace()    barrier()
 #define preempt_enable_notrace()        barrier()
 #define preempt_check_resched_rt()        barrier()
+#define preempt_check_resched()  barrier()
 
 #endif /* CONFIG_PREEMPT_COUNT */
 

1.7.10.4


Best regards,

Yang Honggang
--
To unsubscribe from this list: send the line "unsubscribe linux-rt-users" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi,

…