sg_open: leaving the kernel with locks still held!

Hello!

While trying to figure out why the request queue to sda (ext4) was clogging up on one of our btrfs backup boxes, I noticed a megarc process in D state, so enabled locking debugging, and got this (on 3.12-rc6):

[  205.372823] ================================================
[  205.372901] [ BUG: lock held when returning to user space! ]
[  205.372979] 3.12.0-rc6-hw-debug-pagealloc+ #67 Not tainted
[  205.373055] ------------------------------------------------
[  205.373132] megarc.bin/5283 is leaving the kernel with locks still held!
[  205.373212] 1 lock held by megarc.bin/5283:
[  205.373285]  #0:  (&sdp->o_sem){.+.+..}, at: [<ffffffff8161e650>] sg_open+0x3a0/0x4d0

Vaughan, it seems you touched this area last in 15b06f9a02406e, and git tag --contains says this went in for 3.12-rc. We didn't see this on 3.11, though I haven't tried with lockdep.

This is caused by some of our internal RAID monitoring scripts that run "megarc.bin -dispCfg -a0" (even though that controller isn't present on this server -- a PowerEdge 2950 w/Perc 5).

strace output of the program execution that causes the above message is
here: http://0x.ca/sim/ref/3.12-rc6/megarc_strace.txt

Quoted 22 lines · Simon Kirby

Hello!

While trying to figure out why the request queue to sda (ext4) was clogging up on one of our btrfs backup boxes, I noticed a megarc process in D state, so enabled locking debugging, and got this (on 3.12-rc6):

[  205.372823] ================================================
[  205.372901] [ BUG: lock held when returning to user space! ]
[  205.372979] 3.12.0-rc6-hw-debug-pagealloc+ #67 Not tainted
[  205.373055] ------------------------------------------------
[  205.373132] megarc.bin/5283 is leaving the kernel with locks still held!
[  205.373212] 1 lock held by megarc.bin/5283:
[  205.373285]  #0:  (&sdp->o_sem){.+.+..}, at: [<ffffffff8161e650>] sg_open+0x3a0/0x4d0

Vaughan, it seems you touched this area last in 15b06f9a02406e, and git tag --contains says this went in for 3.12-rc. We didn't see this on 3.11, though I haven't tried with lockdep.

This is caused by some of our internal RAID monitoring scripts that run "megarc.bin -dispCfg -a0" (even though that controller isn't present on this server -- a PowerEdge 2950 w/Perc 5).

Quoted 2 lines

strace output of the program execution that causes the above message is
here: http://0x.ca/sim/ref/3.12-rc6/megarc_strace.txt

This has been reported. That patch will be reverted or, if there is enough time, a fix will (or at least should) go in before the release of lk 3.12 .

Quoted 26 lines · Douglas Gilbert

Quoted 24 lines · Simon Kirby

Hello!

While trying to figure out why the request queue to sda (ext4) was clogging up on one of our btrfs backup boxes, I noticed a megarc process in D state, so enabled locking debugging, and got this (on 3.12-rc6):

[  205.372823] ================================================
[  205.372901] [ BUG: lock held when returning to user space! ]
[  205.372979] 3.12.0-rc6-hw-debug-pagealloc+ #67 Not tainted
[  205.373055] ------------------------------------------------
[  205.373132] megarc.bin/5283 is leaving the kernel with locks still held!
[  205.373212] 1 lock held by megarc.bin/5283:
[  205.373285]  #0:  (&sdp->o_sem){.+.+..}, at: [<ffffffff8161e650>] sg_open+0x3a0/0x4d0

Vaughan, it seems you touched this area last in 15b06f9a02406e, and git tag --contains says this went in for 3.12-rc. We didn't see this on 3.11, though I haven't tried with lockdep.

This is caused by some of our internal RAID monitoring scripts that run "megarc.bin -dispCfg -a0" (even though that controller isn't present on this server -- a PowerEdge 2950 w/Perc 5).

strace output of the program execution that causes the above message is
here: http://0x.ca/sim/ref/3.12-rc6/megarc_strace.txt

Quoted 3 lines

This has been reported. That patch will be reverted or, if there is enough time, a fix will (or at least should) go in before the release of lk 3.12 .

I think you've got about a week to prove you can fix it (before 3.12 goes final). I'll send my current set of fixes to Linus without doing anything about sg.

Quoted 31 lines · James Bottomley

Quoted 29 lines · Douglas Gilbert

Quoted 18 lines · Simon Kirby

Hello!

While trying to figure out why the request queue to sda (ext4) was clogging up on one of our btrfs backup boxes, I noticed a megarc process in D state, so enabled locking debugging, and got this (on 3.12-rc6):

[  205.372823] ================================================
[  205.372901] [ BUG: lock held when returning to user space! ]
[  205.372979] 3.12.0-rc6-hw-debug-pagealloc+ #67 Not tainted
[  205.373055] ------------------------------------------------
[  205.373132] megarc.bin/5283 is leaving the kernel with locks still held!
[  205.373212] 1 lock held by megarc.bin/5283:
[  205.373285]  #0:  (&sdp->o_sem){.+.+..}, at: [<ffffffff8161e650>] sg_open+0x3a0/0x4d0

Vaughan, it seems you touched this area last in 15b06f9a02406e, and git tag --contains says this went in for 3.12-rc. We didn't see this on 3.11, though I haven't tried with lockdep.

Quoted 6 lines

This is caused by some of our internal RAID monitoring scripts that run "megarc.bin -dispCfg -a0" (even though that controller isn't present on this server -- a PowerEdge 2950 w/Perc 5).

strace output of the program execution that causes the above message is
here: http://0x.ca/sim/ref/3.12-rc6/megarc_strace.txt

This has been reported. That patch will be reverted or, if there is enough time, a fix will (or at least should) go in before the release of lk 3.12 .

Quoted 3 lines

I think you've got about a week to prove you can fix it (before 3.12 goes final). I'll send my current set of fixes to Linus without doing anything about sg.

In the event that a suitable fix isn't found, are you going to revert the commit(s) that caused the issue?

Quoted 26 lines · Douglas Gilbert

Quoted 24 lines · Simon Kirby

Hello!

While trying to figure out why the request queue to sda (ext4) was clogging up on one of our btrfs backup boxes, I noticed a megarc process in D state, so enabled locking debugging, and got this (on 3.12-rc6):

[  205.372823] ================================================
[  205.372901] [ BUG: lock held when returning to user space! ]
[  205.372979] 3.12.0-rc6-hw-debug-pagealloc+ #67 Not tainted
[  205.373055] ------------------------------------------------
[  205.373132] megarc.bin/5283 is leaving the kernel with locks still held!
[  205.373212] 1 lock held by megarc.bin/5283:
[  205.373285]  #0:  (&sdp->o_sem){.+.+..}, at: [<ffffffff8161e650>] sg_open+0x3a0/0x4d0

Vaughan, it seems you touched this area last in 15b06f9a02406e, and git tag --contains says this went in for 3.12-rc. We didn't see this on 3.11, though I haven't tried with lockdep.

This is caused by some of our internal RAID monitoring scripts that run "megarc.bin -dispCfg -a0" (even though that controller isn't present on this server -- a PowerEdge 2950 w/Perc 5).

strace output of the program execution that causes the above message is
here: http://0x.ca/sim/ref/3.12-rc6/megarc_strace.txt

Quoted 3 lines

This has been reported. That patch will be reverted or, if there is enough time, a fix will (or at least should) go in before the release of lk 3.12 .

I think you've got about a week to prove you can fix it (before 3.12 goes final). I'll send my current set of fixes to Linus without doing anything about sg.

Quoted 27 lines · James Bottomley

Quoted 26 lines · Douglas Gilbert

Quoted 24 lines · Simon Kirby

Hello!

While trying to figure out why the request queue to sda (ext4) was clogging up on one of our btrfs backup boxes, I noticed a megarc process in D state, so enabled locking debugging, and got this (on 3.12-rc6):

[  205.372823] ================================================
[  205.372901] [ BUG: lock held when returning to user space! ]
[  205.372979] 3.12.0-rc6-hw-debug-pagealloc+ #67 Not tainted
[  205.373055] ------------------------------------------------
[  205.373132] megarc.bin/5283 is leaving the kernel with locks still held!
[  205.373212] 1 lock held by megarc.bin/5283:
[  205.373285]  #0:  (&sdp->o_sem){.+.+..}, at: [<ffffffff8161e650>] sg_open+0x3a0/0x4d0

Vaughan, it seems you touched this area last in 15b06f9a02406e, and git tag --contains says this went in for 3.12-rc. We didn't see this on 3.11, though I haven't tried with lockdep.

This is caused by some of our internal RAID monitoring scripts that run "megarc.bin -dispCfg -a0" (even though that controller isn't present on this server -- a PowerEdge 2950 w/Perc 5).

strace output of the program execution that causes the above message is
here: http://0x.ca/sim/ref/3.12-rc6/megarc_strace.txt

Quoted 7 lines

Quoted 3 lines

This has been reported. That patch will be reverted or, if there is enough time, a fix will (or at least should) go in before the release of lk 3.12 .

I think you've got about a week to prove you can fix it (before 3.12 goes final). I'll send my current set of fixes to Linus without doing anything about sg.

"prove" is a big ask, especially coming from a mathematician. I consider it more hacking (in the golf sense) on my part to tweak well-meaning patches to the sg driver that cause collateral damage. Further, I suspect Vaughan's patch was an attempt to fix damage left be a previous sg_open() hacker.

I have asked Simon Kirby to apply the patch:
http://marc.info/?l=linux-scsi&m=138237283432010&w=2
and report if it fixes his problems. Further I have written three test programs to test O_EXCL handling on SCSI devices, two of which are in the examples directory of sg3_utils version 1.37 . The latest one (single exclusive writer, multiple readers) can be found in the News section of:

   http://sg.danny.cz/sg/

These tests don't check all possibilities (e.g. random signals, ml error processing and detached devices) but they are better than nothing. And, as a side issue, they break bsg (cause it ignores O_EXCL) and break the block layer (e.g. /dev/sdb) so perhaps it should be reverted :-)

Quoted 51 lines · Douglas Gilbert

Quoted 27 lines · James Bottomley

Quoted 26 lines · Douglas Gilbert

Quoted 24 lines · Simon Kirby

Hello!

While trying to figure out why the request queue to sda (ext4) was clogging up on one of our btrfs backup boxes, I noticed a megarc process in D state, so enabled locking debugging, and got this (on 3.12-rc6):

[  205.372823] ================================================
[  205.372901] [ BUG: lock held when returning to user space! ]
[  205.372979] 3.12.0-rc6-hw-debug-pagealloc+ #67 Not tainted
[  205.373055] ------------------------------------------------
[  205.373132] megarc.bin/5283 is leaving the kernel with locks still held!
[  205.373212] 1 lock held by megarc.bin/5283:
[  205.373285]  #0:  (&sdp->o_sem){.+.+..}, at: [<ffffffff8161e650>] sg_open+0x3a0/0x4d0

Vaughan, it seems you touched this area last in 15b06f9a02406e, and git tag --contains says this went in for 3.12-rc. We didn't see this on 3.11, though I haven't tried with lockdep.

This is caused by some of our internal RAID monitoring scripts that run "megarc.bin -dispCfg -a0" (even though that controller isn't present on this server -- a PowerEdge 2950 w/Perc 5).

strace output of the program execution that causes the above message is
here: http://0x.ca/sim/ref/3.12-rc6/megarc_strace.txt

Quoted 7 lines

Quoted 3 lines

This has been reported. That patch will be reverted or, if there is enough time, a fix will (or at least should) go in before the release of lk 3.12 .

I think you've got about a week to prove you can fix it (before 3.12 goes final). I'll send my current set of fixes to Linus without doing anything about sg.

"prove" is a big ask, especially coming from a mathematician. I consider it more hacking (in the golf sense) on my part to tweak well-meaning patches to the sg driver that cause collateral damage. Further, I suspect Vaughan's patch was an attempt to fix damage left be a previous sg_open() hacker.

I have asked Simon Kirby to apply the patch:
http://marc.info/?l=linux-scsi&m=138237283432010&w=2
and report if it fixes his problems. Further I have written three test programs to test O_EXCL handling on SCSI devices, two of which are in the examples directory of sg3_utils version 1.37 . The latest one (single exclusive writer, multiple readers) can be found in the News section of:

Quoted 6 lines

   http://sg.danny.cz/sg/

These tests don't check all possibilities (e.g. random signals, ml error processing and detached devices) but they are better than nothing. And, as a side issue, they break bsg (cause it ignores O_EXCL) and break the block layer (e.g. /dev/sdb) so perhaps it should be reverted :-)

Well, this patch works for me in that I see no more lockdep warnings or unintended consequences when running the same "megarc.bin -dispCfg -a0" command.