SGI Origin 200 (ip27) with DEBUG_SPINLOCK

Greetings SGI Origin wizards,

I'm doing some SMP testing on an SGI Origin 200 (ip27).

I started with a 2.6.15 vintage kernel and added changes from here:
ftp://ftp.linux-mips.org/pub/linux/mips/people/ralf/ip27/

It boots both processors and runs OK.

Then I turn on CONFIG_DEBUG_SPINLOCK and CONFIG_DEBUG_SPINLOCK_SLEEP, and get lots of lockup messages. A typical one is below.

Anyone seen this? Some of the low-level lock code has R10000_LLSC_WAR versions, but I don't see anything wrong there.

Sep 26 16:08:15 localhost BUG: spinlock lockup on CPU#0, sshd/1876, a8000000003c7868
Sep 26 16:08:15 localhost loops-per-jiffy 110592 hz 1000
Sep 26 16:08:15 localhost owner ffffffffffffffff cpu -1 locked 0
Sep 26 16:08:15 localhost Call Trace:
Sep 26 16:08:15 localhost [<a8000000001e5da0>] _raw_spin_lock+0x220/0x2c8
Sep 26 16:08:15 localhost [<a80000000032c63c>] _spin_lock_irqsave+0x24/0x38
Sep 26 16:08:15 localhost [<a8000000001f4d60>] tty_ldisc_try+0x38/0x80
Sep 26 16:08:15 localhost [<a8000000001f634c>] tty_ldisc_ref_wait+0x1c/0x100
Sep 26 16:08:15 localhost [<a80000000032c63c>] _spin_lock_irqsave+0x24/0x38
Sep 26 16:08:15 localhost [<a8000000001f64b8>] tty_poll+0x88/0xd8
Sep 26 16:08:15 localhost [<a8000000000df9d0>] do_select+0x298/0x4c0
Sep 26 16:08:15 localhost [<a8000000000dfbf8>] __pollwait+0x0/0x118
Sep 26 16:08:15 localhost [<a8000000000de9d8>] vfs_ioctl+0x80/0x400
Sep 26 16:08:15 localhost [<a800000000100e48>] compat_sys_select+0x318/0x710
Sep 26 16:08:15 localhost [<a8000000000396b4>] handle_sysn32+0x54/0xa4
Sep 26 16:08:15 localhost

There have been a few significant fixes to the Origin code IP27 since 2.6.15 ...

The R10000_LLSC_WAR is a workaround for a CPU bug in certain relativly old version of the R10000 processor. Version 2.6 or older were affected but the cutoff version number could have been 2.7. Anyway, the sympthom was that possibly multiple processors were taking a able to grab a spinlock which obvious is the way to disaster. I originally found the problem when analyzing why rebuilding a MD RAID array was resulting in a crash.

Even with that fix applied I found a MD RAID 5 / 6 not very stable as of last week; it seems this instability is limited to IP27 and IP30 and it seemed like the various kernel debuging options I tried were aggrevating the problem significantly.

Ralf