2.6.39-rc5+ BUG at scsi_run_queue+0x24/0xe3

DISCUSSION Jim Schutt 2 participants 7 messages May 3, 2011 SCSI

lore

Share & tools

lore

Jim Schutt

author

2011-05-03 16:54 UTC #0 lore raw

Hi,

I'm getting this BUG on ~20% of boots with 2.6.39-rc5+:

[   22.607020] BUG: unable to handle kernel NULL pointer dereference at           (null)
[   22.608004] IP: [<ffffffffa019b8c5>] scsi_run_queue+0x24/0xe3 [scsi_mod]
[   22.608004] PGD 22564b067 PUD 222e93067 PMD 0
[   22.608004] Oops: 0000 [#1] SMP
[   22.608004] last sysfs file: /sys/devices/pci0000:00/0000:00:1d.7/usb1/usb_device/usbdev1.1/dev
[   22.608004] CPU 0
[   22.608004] Modules linked in: megaraid_sas ide_cd_mod ib_mthca(+) cdrom ib_mad qla2xxx(+) ib_core button scsi_transport_fc scsi_tgt serio_raw ata_piix i5k_amb tpm_tis libata hwmon tpm i5000_edac floppy(+) tpm_bios scsi_mod dcdbas edac_core pcspkr uhci_hcd ehci_hcd iTCO_wdt iTCO_vendor_support rtc nfs nfs_acl auth_rpcgss fscache lockd sunrpc tg3 bnx2 e1000
[   22.608004]
[   22.608004] Pid: 1820, comm: path_id Not tainted 2.6.39-rc5-00139-g9fbc674 #23 Dell Inc. PowerEdge 1950/0DT097
[   22.608004] RIP: 0010:[<ffffffffa019b8c5>]  [<ffffffffa019b8c5>] scsi_run_queue+0x24/0xe3 [scsi_mod]
[   22.608004] RSP: 0000:ffff88022fc03d10  EFLAGS: 00010282
[   22.608004] RAX: ffff8802240ece00 RBX: ffff88022fc03d20 RCX: ffff88022f002900
[   22.608004] RDX: 0000000000000000 RSI: 0000000000000037 RDI: 0000000000000000
[   22.608004] RBP: ffff88022fc03d60 R08: 0000000000000286 R09: ffffea00077e33a0
[   22.608004] R10: ffff88022f002900 R11: ffff88022fc03cf0 R12: ffff880223977740
[   22.608004] R13: ffff8802254b2938 R14: 0000000000000000 R15: ffff880223977740
[   22.608004] FS:  00007f2e084d26e0(0000) GS:ffff88022fc00000(0000) knlGS:0000000000000000
[   22.608004] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   22.608004] CR2: 0000000000000000 CR3: 0000000222e03000 CR4: 00000000000006f0
[   22.608004] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   22.608004] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   22.608004] Process path_id (pid: 1820, threadinfo ffff8802236a8000, task ffff8802234d1690)
[   22.608004] Stack:
[   22.608004]  0000000000000282 ffff8802254b2800 0000000000000000 ffff880223977740
[   22.608004]  ffff88022fc03d60 ffff8802240ece00 ffff880223977740 ffff8802254b2938
[   22.608004]  0000000000000000 ffff880223977740 ffff88022fc03d90 ffffffffa019c205
[   22.608004] Call Trace:
[   22.608004]  <IRQ>
[   22.608004]  [<ffffffffa019c205>] scsi_next_command+0x3b/0x4c [scsi_mod]
[   22.608004]  [<ffffffffa019c84a>] scsi_end_request+0x83/0x94 [scsi_mod]
[   22.608004]  [<ffffffffa019cbea>] scsi_io_completion+0x1b0/0x3fb [scsi_mod]
[   22.608004]  [<ffffffffa019b635>] ? spin_unlock_irqrestore+0xe/0x10 [scsi_mod]
[   22.608004]  [<ffffffffa0195159>] scsi_finish_command+0xeb/0xf4 [scsi_mod]
[   22.608004]  [<ffffffffa019d9df>] scsi_softirq_done+0x112/0x11b [scsi_mod]
[   22.608004]  [<ffffffff811c727e>] blk_done_softirq+0x4b/0x61
[   22.608004]  [<ffffffff8104f74c>] __do_softirq+0xbf/0x16e
[   22.608004]  [<ffffffff813b354c>] call_softirq+0x1c/0x30
[   22.608004]  [<ffffffff810041a3>] do_softirq+0x3d/0x86
[   22.608004]  [<ffffffff8104f44a>] invoke_softirq+0x17/0x20
[   22.608004]  [<ffffffff8104fa19>] irq_exit+0x57/0x98
[   22.608004]  [<ffffffff813b3c81>] do_IRQ+0x91/0xa8
[   22.608004]  [<ffffffff813abc53>] common_interrupt+0x13/0x13
[   22.608004]  <EOI>
[   22.608004]  [<ffffffff813abc9a>] ? retint_swapgs+0xe/0x13
[   22.608004] Code: ff ff 5b 41 5c c9 c3 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 28 0f 1f 44 00 00 49 89 ff 48 8b bf 40 03 00 00 48 8d 5d c0 <4c> 8b 37 48 89 5d c0 48 89 5d c8 48 8b 87 38 01 00 00 f6 80 a4
[   22.608004] RIP  [<ffffffffa019b8c5>] scsi_run_queue+0x24/0xe3 [scsi_mod]
[   22.608004]  RSP <ffff88022fc03d10>
[   22.608004] CR2: 0000000000000000
[   22.929460] ---[ end trace f9ecaaa16661ec4a ]---
[   22.934070] Kernel panic - not syncing: Fatal exception in interrupt
[   22.940410] Pid: 1820, comm: path_id Tainted: G      D     2.6.39-rc5-00139-g9fbc674 #23
[   22.948483] Call Trace:
[   22.950923]  <IRQ>  [<ffffffff8104953e>] ? panic+0xbc/0x1c3
[   22.953064] qla2xxx 0000:0e:00.0: Allocated (64 KB) for EFT...
[   22.953217] qla2xxx 0000:0e:00.0: Allocated (1285 KB) for firmware dump...
[   22.969169]  [<ffffffff813aba57>] ? _raw_spin_unlock_irqrestore+0xe/0x10
[   22.970176] scsi0 : qla2xxx
[   22.970505] qla2xxx 0000:0e:00.0:
[   22.970506]  QLogic Fibre Channel HBA Driver: 8.03.07.00
[   22.970507]   QLogic QLE2462 - PCI-Express Dual Channel 4Gb Fibre Channel HBA
[   22.970508]   ISP2432: PCIe (2.5GT/s x4) @ 0000:0e:00.0 hdma+, host#=0, fw=5.03.13 (496)
[   22.970540] qla2xxx 0000:0e:00.1: PCI INT B -> GSI 17 (level, low) -> IRQ 17
[   23.009538]  [<ffffffff81049a32>] ? spin_unlock_irqrestore+0xe/0x10
[   23.009544] qla2xxx 0000:0e:00.1: Found an ISP2432, irq 17, iobase 0xffffc9001178c000
[   23.009787] qla2xxx 0000:0e:00.1: irq 107 for MSI/MSI-X
[   23.009856] qla2xxx 0000:0e:00.1: Configuring PCI space...
[   23.009862] qla2xxx 0000:0e:00.1: setting latency timer to 64
[   23.040006]  [<ffffffff8104ada0>] ? kmsg_dump+0x4f/0xe6
[   23.040531] qla2xxx 0000:0e:00.1: Configure NVRAM parameters...
[   23.051121]  [<ffffffff813aca70>] ? oops_end+0xaf/0xbf
[   23.056249]  [<ffffffff8102b04c>] ? no_context+0xea/0xf6
[   23.061550]  [<ffffffff8102b227>] ? __bad_area_nosemaphore+0x107/0x114
[   23.068063]  [<ffffffff8102b2be>] ? bad_area_nosemaphore+0x13/0x15
[   23.074231]  [<ffffffff813aeb28>] ? do_page_fault+0x192/0x331
[   23.076259] qla2xxx 0000:0e:00.1: Verifying loaded RISC code...
[   23.085867]  [<ffffffff8101c7eb>] ? apic_write+0x16/0x18
[   23.086060] qla2xxx 0000:0e:00.1: FW: Loading via request-firmware...
[   23.097590]  [<ffffffff8101ca74>] ? lapic_next_event+0x15/0x19
[   23.103413]  [<ffffffff81072fa9>] ? clockevents_program_event+0x78/0x81
[   23.110014]  [<ffffffff81074428>] ? tick_dev_program_event+0x2f/0x8e
[   23.116357]  [<ffffffff810adeae>] ? trace_hardirqs_off_caller+0x11/0x25
[   23.122959]  [<ffffffff8106bb66>] ? sched_clock_local+0x11/0x76
[   23.128867]  [<ffffffff811dd81a>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[   23.135381]  [<ffffffff813abe8f>] ? page_fault+0x1f/0x30
[   23.140692]  [<ffffffffa019b8c5>] ? scsi_run_queue+0x24/0xe3 [scsi_mod]
[   23.147300]  [<ffffffffa019c205>] ? scsi_next_command+0x3b/0x4c [scsi_mod]
[   23.154166]  [<ffffffffa019c84a>] ? scsi_end_request+0x83/0x94 [scsi_mod]
[   23.160946]  [<ffffffffa019cbea>] ? scsi_io_completion+0x1b0/0x3fb [scsi_mod]
[   23.168072]  [<ffffffffa019b635>] ? spin_unlock_irqrestore+0xe/0x10 [scsi_mod]
[   23.175283]  [<ffffffffa0195159>] ? scsi_finish_command+0xeb/0xf4 [scsi_mod]
[   23.182329]  [<ffffffffa019d9df>] ? scsi_softirq_done+0x112/0x11b [scsi_mod]
[   23.189363]  [<ffffffff811c727e>] ? blk_done_softirq+0x4b/0x61
[   23.195185]  [<ffffffff8104f74c>] ? __do_softirq+0xbf/0x16e
[   23.200745]  [<ffffffff813b354c>] ? call_softirq+0x1c/0x30
[   23.206219]  [<ffffffff810041a3>] ? do_softirq+0x3d/0x86
[   23.211519]  [<ffffffff8104f44a>] ? invoke_softirq+0x17/0x20
[   23.217167]  [<ffffffff8104fa19>] ? irq_exit+0x57/0x98
[   23.222293]  [<ffffffff813b3c81>] ? do_IRQ+0x91/0xa8
[   23.227247]  [<ffffffff813abc53>] ? common_interrupt+0x13/0x13
[   23.233067]  <EOI>  [<ffffffff813abc9a>] ? retint_

I get no BUGs in dozens of boots if I revert commit 86cbfb5607d:

[SCSI] put stricter guards on queue dead checks

SCSI uses request_queue->queuedata == NULL as a signal that the queue is dying. We set this state in the sdev release function. However, this allows a small window where we release the last reference but haven't quite got to this stage yet and so something will try to take a reference in scsi_request_fn and oops. It's very rare, but we had a report here, so we're pushing this as a bug fix

The actual fix is to set request_queue->queuedata to NULL in scsi_remove_device() before we drop the reference. This causes correct automatic rejects from scsi_request_fn as people who hold additional references try to submit work and prevents anything from getting a new reference to the sdev that way.

Cc [email protected]

Signed-off-by James Bottomley <[email protected]>

Please let me know if what further information you need, or if there is anything I can do, to help resolve this.

Thanks -- Jim

James Bottomley

2011-05-03 17:00 UTC #1 lore raw

Quoted 2 lines · Jim Schutt

Please let me know if what further information you need, or if there is anything I can do, to help resolve this.

I think this is the fix (already in rc-fixes):

James

---

Quoted 1 line

From 3e85ea868dbd60a84240be5c1eebc36841b9c568 Mon Sep 17 00:00:00 2001

From: James Bottomley <[email protected]>
Date: Sun, 1 May 2011 09:42:07 -0500
Subject: [PATCH] [SCSI] fix oops in scsi_run_queue()

The recent commit closing the race window in device teardown:

commit 86cbfb5607d4b81b1a993ff689bbd2addd5d3a9b

Author: James Bottomley <[email protected]>
Date:   Fri Apr 22 10:39:59 2011 -0500

[SCSI] put stricter guards on queue dead checks

is causing a potential NULL deref in scsi_run_queue() because the q->queuedata may already be NULL by the time this function is called. Since we shouldn't be running a queue that is being torn down, simply add a NULL check in scsi_run_queue() to forestall this.

Signed-off-by James Bottomley <[email protected]>

drivers/scsi/scsi_lib.c +4 -0

index e9901b8..03979f4 100644
@@ -404,6 +404,10 @@ static void scsi_run_queue(struct request_queue *q)
 	LIST_HEAD(starved_list);
 	unsigned long flags;
 
+	/* if the device is dead, sdev will be NULL, so no queue to run */
+	if (!sdev)
+		return;
+
 	if (scsi_target(sdev)->single_lun)
 		scsi_single_lun_run(sdev);

Jim Schutt

author

2011-05-03 17:28 UTC #2 lore raw

Quoted 34 lines · James Bottomley

Quoted 2 lines · Jim Schutt
Please let me know if what further information you need, or if there is anything I can do, to help resolve this.
I think this is the fix (already in rc-fixes):
James
---
From 3e85ea868dbd60a84240be5c1eebc36841b9c568 Mon Sep 17 00:00:00 2001
From: James Bottomley <[email protected]>
Date: Sun, 1 May 2011 09:42:07 -0500
Subject: [PATCH] [SCSI] fix oops in scsi_run_queue()
The recent commit closing the race window in device teardown:
commit 86cbfb5607d4b81b1a993ff689bbd2addd5d3a9b
Author: James Bottomley <[email protected]>
Date:   Fri Apr 22 10:39:59 2011 -0500
[SCSI] put stricter guards on queue dead checks
is causing a potential NULL deref in scsi_run_queue() because the q->queuedata may already be NULL by the time this function is called. Since we shouldn't be running a queue that is being torn down, simply add a NULL check in scsi_run_queue() to forestall this.
Signed-off-by James Bottomley <[email protected]>
drivers/scsi/scsi_lib.c +0 -0
index e9901b8..03979f4 100644

drivers/scsi/scsi_lib.c static void scsi_run_queue(struct request_queue *q) · line 413

Quoted 10 lines

drivers/scsi/scsi_lib.c +4 -0

@@ -404,6 +404,10 @@ static void scsi_run_queue(struct request_queue *q)
 	LIST_HEAD(starved_list);
 	unsigned long flags;
 
+	/* if the device is dead, sdev will be NULL, so no queue to run */
+	if (!sdev)
+		return;
+
 	if (scsi_target(sdev)->single_lun)
 		scsi_single_lun_run(sdev);

Hmmm, with the above added, I still get BUGs. Here's an example:

[   17.142931] BUG: unable to handle kernel NULL pointer dereference at           (null)
[   17.143002] IP: [<ffffffffa01cf8c5>] scsi_run_queue+0x24/0xec [scsi_mod]
[   17.143002] PGD 128257067 PUD 129da5067 PMD 0
[   17.143002] Oops: 0000 [#1] SMP
[   17.143002] last sysfs file: /sys/devices/platform/pcspkr/input/input0/event0/dev
[   17.143002] CPU 1
[   17.143002] Modules linked in: megaraid_sas ide_cd_mod cdrom button ib_mthca(+) ib_mad ib_core serio_raw floppy(+) dcdbas tpm_tis ata_piix tpm tpm_bios libata i5k_amb hwmon iTCO_wdt scsi_mod iTCO_vendor_support i5000_edac ehci_hcd pcspkr edac_core uhci_hcd rtc nfs nfs_acl auth_rpcgss fscache lockd sunrpc tg3 bnx2 e1000
[   17.143002]
[   17.143002] Pid: 1751, comm: path_id Not tainted 2.6.39-rc5-00140-g6a9a2d5 #24 Dell Inc. PowerEdge 1950/0DT097
[   17.143002] RIP: 0010:[<ffffffffa01cf8c5>]  [<ffffffffa01cf8c5>] scsi_run_queue+0x24/0xec [scsi_mod]
[   17.143002] RSP: 0000:ffff88012fc43d10  EFLAGS: 00010246
[   17.143002] RAX: ffff880127393700 RBX: ffff880127393700 RCX: ffff88012f002900
[   17.143002] RDX: 0000000000000000 RSI: 0000000000000037 RDI: 0000000000000000
[   17.143002] RBP: ffff88012fc43d60 R08: 0000000000000286 R09: ffffea00040947f0
[   17.143002] R10: ffff88012f002900 R11: ffff88012fc43cf0 R12: ffff880126cdcf80
[   17.143002] R13: ffff880126d45138 R14: 0000000000000000 R15: ffff880126cdcf80
[   17.143002] FS:  0000000000000000(0000) GS:ffff88012fc40000(0000) knlGS:0000000000000000
[   17.143002] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[   17.143002] CR2: 0000000000000000 CR3: 0000000126d0f000 CR4: 00000000000006e0
[   17.143002] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   17.143002] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   17.143002] Process path_id (pid: 1751, threadinfo ffff880127a10000, task ffff880129c02d20)
[   17.143002] Stack:
[   17.143002]  0000000000000282 ffff880126cdcf80 0000000000000000 ffff880126cdcf80
[   17.143002]  ffff88012fc43d60 ffff880127393700 ffff880126cdcf80 ffff880126d45138
[   17.143002]  0000000000000000 ffff880126cdcf80 ffff88012fc43d90 ffffffffa01d020e
[   17.143002] Call Trace:
[   17.143002]  <IRQ>
[   17.143002]  [<ffffffffa01d020e>] scsi_next_command+0x3b/0x4c [scsi_mod]
[   17.143002]  [<ffffffffa01d0853>] scsi_end_request+0x83/0x94 [scsi_mod]
[   17.143002]  [<ffffffffa01d0bf3>] scsi_io_completion+0x1b0/0x3fb [scsi_mod]
[   17.143002]  [<ffffffffa01cf635>] ? spin_unlock_irqrestore+0xe/0x10 [scsi_mod]
[   17.143002]  [<ffffffffa01c9159>] scsi_finish_command+0xeb/0xf4 [scsi_mod]
[   17.143002]  [<ffffffffa01d19e8>] scsi_softirq_done+0x112/0x11e [scsi_mod]
[   17.143002]  [<ffffffff811c727e>] blk_done_softirq+0x4b/0x61
[   17.143002]  [<ffffffff8104f74c>] __do_softirq+0xbf/0x16e
[   17.143002]  [<ffffffff813b354c>] call_softirq+0x1c/0x30
[   17.143002]  [<ffffffff810041a3>] do_softirq+0x3d/0x86
[   17.143002]  [<ffffffff8104f44a>] invoke_softirq+0x17/0x20
[   17.143002]  [<ffffffff8104fa19>] irq_exit+0x57/0x98
[   17.143002]  [<ffffffff813b3c81>] do_IRQ+0x91/0xa8
[   17.143002]  [<ffffffff813abc53>] common_interrupt+0x13/0x13
[   17.143002]  <EOI>
[   17.143002]  [<ffffffff811078c0>] ? kmem_cache_create+0x175/0x175
[   17.143002]  [<ffffffff810eeb06>] ? anon_vma_alloc+0x1a/0x2b
[   17.143002]  [<ffffffff810eec0a>] anon_vma_prepare+0x60/0xfe
[   17.143002]  [<ffffffff810e2f59>] __do_fault+0xc8/0x360
[   17.143002]  [<ffffffff810e3227>] do_linear_fault+0x36/0x38
[   17.143002]  [<ffffffff8102ea08>] ? pgtable_page_ctor+0x1a/0x1c
[   17.143002]  [<ffffffff810e4112>] handle_pte_fault+0x6a/0x170
[   17.143002]  [<ffffffff810e2a1e>] ? spin_lock+0xe/0x10
[   17.143002]  [<ffffffff810e56e6>] handle_mm_fault+0x15f/0x177
[   17.143002]  [<ffffffff813aebda>] do_page_fault+0x244/0x331
[   17.143002]  [<ffffffff810eb086>] ? do_mmap_pgoff+0x267/0x2cc
[   17.143002]  [<ffffffff810adeae>] ? trace_hardirqs_off_caller+0x11/0x25
[   17.143002]  [<ffffffff811dd81a>] ? trace_hardirqs_off_thunk+0x3a/0x6c
[   17.143002]  [<ffffffff813abe8f>] page_fault+0x1f/0x30
[   17.143002] Code: ff ff 5b 41 5c c9 c3 55 48 89 e5 41 57 41 56 41 55 41 54 53 48 83 ec 28 0f 1f 44 00 00 48 89 7d b8 48 8b bf 40 03 00 00 48 85 ff <4c> 8b 37 0f 84 b0 00 00 00 48 8d 5d c0 48 89 5d c0 48 89 5d c8
[   17.143002] RIP  [<ffffffffa01cf8c5>] scsi_run_queue+0x24/0xec [scsi_mod]
[   17.143002]  RSP <ffff88012fc43d10>
[   17.143002] CR2: 0000000000000000
[   17.535741] ---[ end trace 97dde672b920540a ]---

Please let me know what else I can do to help sort this out.

-- Jim

James Bottomley

2011-05-03 17:37 UTC #3 lore raw

Quoted 49 lines · Jim Schutt

Quoted 34 lines · James Bottomley
Quoted 2 lines · Jim Schutt
Please let me know if what further information you need, or if there is anything I can do, to help resolve this.
I think this is the fix (already in rc-fixes):
James
---
From 3e85ea868dbd60a84240be5c1eebc36841b9c568 Mon Sep 17 00:00:00 2001
From: James Bottomley <[email protected]>
Date: Sun, 1 May 2011 09:42:07 -0500
Subject: [PATCH] [SCSI] fix oops in scsi_run_queue()
The recent commit closing the race window in device teardown:
commit 86cbfb5607d4b81b1a993ff689bbd2addd5d3a9b
Author: James Bottomley <[email protected]>
Date:   Fri Apr 22 10:39:59 2011 -0500
[SCSI] put stricter guards on queue dead checks
is causing a potential NULL deref in scsi_run_queue() because the q->queuedata may already be NULL by the time this function is called. Since we shouldn't be running a queue that is being torn down, simply add a NULL check in scsi_run_queue() to forestall this.
Signed-off-by James Bottomley <[email protected]>
drivers/scsi/scsi_lib.c +0 -0
index e9901b8..03979f4 100644
drivers/scsi/scsi_lib.c static void scsi_run_queue(struct request_queue *q) · line 413
Quoted 10 lines
drivers/scsi/scsi_lib.c +4 -0
@@ -404,6 +404,10 @@ static void scsi_run_queue(struct request_queue *q)
 	LIST_HEAD(starved_list);
 	unsigned long flags;
 
+	/* if the device is dead, sdev will be NULL, so no queue to run */
+	if (!sdev)
+		return;
+
 	if (scsi_target(sdev)->single_lun)
 		scsi_single_lun_run(sdev);
 
Hmmm, with the above added, I still get BUGs. Here's an example:

Quoted 2 lines

[   17.142931] BUG: unable to handle kernel NULL pointer dereference at           (null)
[   17.143002] IP: [<ffffffffa01cf8c5>] scsi_run_queue+0x24/0xec [scsi_mod]

Ooh, compiler optimisation, I think; try this instead

James

---

drivers/scsi/scsi_lib.c +6 -1

index e9901b8..0bac91e 100644
@@ -400,10 +400,15 @@ static inline int scsi_host_is_busy(struct Scsi_Host *shost)
 static void scsi_run_queue(struct request_queue *q)
 {
 	struct scsi_device *sdev = q->queuedata;
-	struct Scsi_Host *shost = sdev->host;
+	struct Scsi_Host *shost;
 	LIST_HEAD(starved_list);
 	unsigned long flags;
 
+	/* if the device is dead, sdev will be NULL, so no queue to run */
+	if (!sdev)
+		return;
+
+	shost = sdev->host;
 	if (scsi_target(sdev)->single_lun)
 		scsi_single_lun_run(sdev);

Jim Schutt

author

2011-05-03 17:55 UTC #4 lore raw

Quoted 68 lines · James Bottomley

Quoted 47 lines · Jim Schutt
Quoted 33 lines · James Bottomley
Quoted 2 lines · Jim Schutt
Please let me know if what further information you need, or if there is anything I can do, to help resolve this.
I think this is the fix (already in rc-fixes):
James
---
From 3e85ea868dbd60a84240be5c1eebc36841b9c568 Mon Sep 17 00:00:00 2001
From: James Bottomley <[email protected]>
Date: Sun, 1 May 2011 09:42:07 -0500
Subject: [PATCH] [SCSI] fix oops in scsi_run_queue()
The recent commit closing the race window in device teardown:
commit 86cbfb5607d4b81b1a993ff689bbd2addd5d3a9b
Author: James Bottomley <[email protected]>
Date:   Fri Apr 22 10:39:59 2011 -0500
[SCSI] put stricter guards on queue dead checks
is causing a potential NULL deref in scsi_run_queue() because the q->queuedata may already be NULL by the time this function is called. Since we shouldn't be running a queue that is being torn down, simply add a NULL check in scsi_run_queue() to forestall this.
Signed-off-by James Bottomley <[email protected]>
drivers/scsi/scsi_lib.c +0 -0
index e9901b8..03979f4 100644
drivers/scsi/scsi_lib.c static void scsi_run_queue(struct request_queue *q) · line 413
Quoted 10 lines
drivers/scsi/scsi_lib.c +4 -0
@@ -404,6 +404,10 @@ static void scsi_run_queue(struct request_queue *q)
 	LIST_HEAD(starved_list);
 	unsigned long flags;
 
+	/* if the device is dead, sdev will be NULL, so no queue to run */
+	if (!sdev)
+		return;
+
 	if (scsi_target(sdev)->single_lun)
 		scsi_single_lun_run(sdev);
 
Hmmm, with the above added, I still get BUGs. Here's an example:
Quoted 2 lines
[   17.142931] BUG: unable to handle kernel NULL pointer dereference at           (null)
[   17.143002] IP: [<ffffffffa01cf8c5>] scsi_run_queue+0x24/0xec [scsi_mod]
Ooh, compiler optimisation, I think; try this instead
James
---
drivers/scsi/scsi_lib.c +1 -1
index e9901b8..0bac91e 100644
@@ -400,10 +400,15 @@ static inline int scsi_host_is_busy(struct Scsi_Host *shost)
 static void scsi_run_queue(struct request_queue *q)
 {
 	struct scsi_device *sdev = q->queuedata;
-	struct Scsi_Host *shost = sdev->host;
+	struct Scsi_Host *shost;
 	LIST_HEAD(starved_list);

drivers/scsi/scsi_lib.c static inline int scsi_host_is_busy(struct Scsi_Host *shost) · line 414

Quoted 10 lines

drivers/scsi/scsi_lib.c +5 -0

 	unsigned long flags;
 
+	/* if the device is dead, sdev will be NULL, so no queue to run */
+	if (!sdev)
+		return;
+
+	shost = sdev->host;
 	if (scsi_target(sdev)->single_lun)
 		scsi_single_lun_run(sdev);

Yes, that definitely fixes things for me.

Thanks!!

-- Jim

Jim Schutt

author

2011-05-03 18:52 UTC #5 lore raw

Hi James,

FWIW, I noticed that commit 99f3c722e23 in scsi-rc-fixes-2.6 might want a Cc: [email protected], since the commit it's fixing, 86cbfb5607d4, had one.

I'm not sure how that all works, but I thought I'd mention it just in case.

-- Jim

James Bottomley

2011-05-03 20:36 UTC #6 lore raw

Quoted 6 lines · Jim Schutt

FWIW, I noticed that commit 99f3c722e23 in scsi-rc-fixes-2.6 might want a Cc: [email protected], since the commit it's fixing, 86cbfb5607d4, had one.
I'm not sure how that all works, but I thought I'd mention it just in case.

Yes, good point; I'll add it ... destabilising the stable kernels wouldn't be a very good idea.

James