Skip to contentSkip to conversation

NULL pointer dereference at blk_drain_queue

DISCUSSION Jiri Slaby 3 participants 9 messages Jun 14, 2012 – Jun 15, 2012 Block layer

lore
Share & tools
lore

Jiri Slaby

author
#0 lore raw

Hi,

with today's -next I'm (reproducibly) getting this while updating packages:

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<ffffffff8108cd16>] __wake_up_common+0x26/0x90
PGD 463f1067 PUD 463f2067 PMD 0
Oops: 0000 [#1] SMP
CPU 1
Modules linked in:
Pid: 2711, comm: kworker/1:0 Not tainted 3.5.0-rc2-next-20120614_64+
#1752 Bochs Bochs
RIP: 0010:[<ffffffff8108cd16>]  [<ffffffff8108cd16>]
__wake_up_common+0x26/0x90
RSP: 0018:ffff880047221cb0  EFLAGS: 00010082
RAX: 0000000000000086 RBX: ffff880046350888 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880046350888
RBP: ffff880047221cf0 R08: 0000000000000000 R09: 00000001000c0009
R10: ffff880047804480 R11: 0000000000000000 R12: ffff880046350890
R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffff880049700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000045ced000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 2711, threadinfo ffff880047220000, task
ffff8800435bc5c0)
Stack:
 000000004628da68 0000000000000000 ffff88004970d340 ffff880046350888
 0000000000000086 0000000000000003 0000000000000000 0000000000000000
 ffff880047221d30 ffffffff8108d9a3 ffff88004970d340 ffff880046350848
Call Trace:
 [<ffffffff8108d9a3>] __wake_up+0x43/0x70
 [<ffffffff81267f96>] blk_drain_queue+0xf6/0x120
 [<ffffffff8126803f>] blk_cleanup_queue+0x7f/0xd0
 [<ffffffff814a9a80>] md_free+0x50/0x70
 [<ffffffff8127b3c2>] kobject_cleanup+0x82/0x1b0
 [<ffffffff8127b24b>] kobject_put+0x2b/0x60
 [<ffffffff814a97ef>] mddev_delayed_delete+0x2f/0x40
 [<ffffffff8107e1ab>] process_one_work+0x11b/0x3f0
 [<ffffffff814a97c0>] ? restart_array+0xc0/0xc0
 [<ffffffff8107f94e>] worker_thread+0x12e/0x340
 [<ffffffff8107f820>] ? manage_workers.isra.29+0x1f0/0x1f0
 [<ffffffff81084e1e>] kthread+0x8e/0xa0
 [<ffffffff8160add4>] kernel_thread_helper+0x4/0x10
 [<ffffffff81084d90>] ? flush_kthread_worker+0x70/0x70
 [<ffffffff8160add0>] ? gs_change+0xb/0xb
Code: 80 00 00 00 00 55 48 89 e5 41 57 41 89 f7 41 56 41 89 ce 41 55 41

54 4c 8d 67 08 53 48 83 ec 18 89 55 c4 48 8b 57 08 4c 89 45 c8 <4c> 8b 2a 48 8d 42 e8 49 83 ed 18 49 39 d4 75 0d eb 40 0f 1f 84

regards,

Signature · 2 lines
js
suse labs
maintainer
#1 lore raw
Quoted 47 lines · Jiri Slaby

Hi,

with today's -next I'm (reproducibly) getting this while updating packages:

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<ffffffff8108cd16>] __wake_up_common+0x26/0x90
PGD 463f1067 PUD 463f2067 PMD 0
Oops: 0000 [#1] SMP
CPU 1
Modules linked in:
Pid: 2711, comm: kworker/1:0 Not tainted 3.5.0-rc2-next-20120614_64+
#1752 Bochs Bochs
RIP: 0010:[<ffffffff8108cd16>]  [<ffffffff8108cd16>]
__wake_up_common+0x26/0x90
RSP: 0018:ffff880047221cb0  EFLAGS: 00010082
RAX: 0000000000000086 RBX: ffff880046350888 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880046350888
RBP: ffff880047221cf0 R08: 0000000000000000 R09: 00000001000c0009
R10: ffff880047804480 R11: 0000000000000000 R12: ffff880046350890
R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffff880049700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000045ced000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 2711, threadinfo ffff880047220000, task
ffff8800435bc5c0)
Stack:
 000000004628da68 0000000000000000 ffff88004970d340 ffff880046350888
 0000000000000086 0000000000000003 0000000000000000 0000000000000000
 ffff880047221d30 ffffffff8108d9a3 ffff88004970d340 ffff880046350848
Call Trace:
 [<ffffffff8108d9a3>] __wake_up+0x43/0x70
 [<ffffffff81267f96>] blk_drain_queue+0xf6/0x120
 [<ffffffff8126803f>] blk_cleanup_queue+0x7f/0xd0
 [<ffffffff814a9a80>] md_free+0x50/0x70
 [<ffffffff8127b3c2>] kobject_cleanup+0x82/0x1b0
 [<ffffffff8127b24b>] kobject_put+0x2b/0x60
 [<ffffffff814a97ef>] mddev_delayed_delete+0x2f/0x40
 [<ffffffff8107e1ab>] process_one_work+0x11b/0x3f0
 [<ffffffff814a97c0>] ? restart_array+0xc0/0xc0
 [<ffffffff8107f94e>] worker_thread+0x12e/0x340
 [<ffffffff8107f820>] ? manage_workers.isra.29+0x1f0/0x1f0
 [<ffffffff81084e1e>] kthread+0x8e/0xa0
 [<ffffffff8160add4>] kernel_thread_helper+0x4/0x10
 [<ffffffff81084d90>] ? flush_kthread_worker+0x70/0x70
 [<ffffffff8160add0>] ? gs_change+0xb/0xb
Code: 80 00 00 00 00 55 48 89 e5 41 57 41 89 f7 41 56 41 89 ce 41 55 41
Quoted 2 lines

54 4c 8d 67 08 53 48 83 ec 18 89 55 c4 48 8b 57 08 4c 89 45 c8 <4c> 8b 2a 48 8d 42 e8 49 83 ed 18 49 39 d4 75 0d eb 40 0f 1f 84

It's a bug in local commit bc85cf83, for stacked devices we have not initialized the wait queues. So the below should fix it, as would always initializing all queue structures even for the partial use case.

block/blk-core.c +6 -4
index b477fa0..93eb3e4 100644
@@ -415,10 +415,12 @@ void blk_drain_queue(struct request_queue *q, bool drain_all)
 	 * allocation path, so the wakeup chaining is lost and we're
 	 * left with hung waiters. We need to wake up those waiters.
 	 */
-	spin_lock_irq(q->queue_lock);
-	for (i = 0; i < ARRAY_SIZE(q->rq.wait); i++)
-		wake_up_all(&q->rq.wait[i]);
-	spin_unlock_irq(q->queue_lock);
+	if (q->request_fn) {
+		spin_lock_irq(q->queue_lock);
+		for (i = 0; i < ARRAY_SIZE(q->rq.wait); i++)
+			wake_up_all(&q->rq.wait[i]);
+		spin_unlock_irq(q->queue_lock);
+	}
 }
 
 /**
Signature · 1 line
Jens Axboe
maintainer
#2 lore raw
In reply to Jiri Slaby
Quoted 47 lines · Jiri Slaby

Hi,

with today's -next I'm (reproducibly) getting this while updating packages:

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<ffffffff8108cd16>] __wake_up_common+0x26/0x90
PGD 463f1067 PUD 463f2067 PMD 0
Oops: 0000 [#1] SMP
CPU 1
Modules linked in:
Pid: 2711, comm: kworker/1:0 Not tainted 3.5.0-rc2-next-20120614_64+
#1752 Bochs Bochs
RIP: 0010:[<ffffffff8108cd16>]  [<ffffffff8108cd16>]
__wake_up_common+0x26/0x90
RSP: 0018:ffff880047221cb0  EFLAGS: 00010082
RAX: 0000000000000086 RBX: ffff880046350888 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880046350888
RBP: ffff880047221cf0 R08: 0000000000000000 R09: 00000001000c0009
R10: ffff880047804480 R11: 0000000000000000 R12: ffff880046350890
R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffff880049700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000045ced000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 2711, threadinfo ffff880047220000, task
ffff8800435bc5c0)
Stack:
 000000004628da68 0000000000000000 ffff88004970d340 ffff880046350888
 0000000000000086 0000000000000003 0000000000000000 0000000000000000
 ffff880047221d30 ffffffff8108d9a3 ffff88004970d340 ffff880046350848
Call Trace:
 [<ffffffff8108d9a3>] __wake_up+0x43/0x70
 [<ffffffff81267f96>] blk_drain_queue+0xf6/0x120
 [<ffffffff8126803f>] blk_cleanup_queue+0x7f/0xd0
 [<ffffffff814a9a80>] md_free+0x50/0x70
 [<ffffffff8127b3c2>] kobject_cleanup+0x82/0x1b0
 [<ffffffff8127b24b>] kobject_put+0x2b/0x60
 [<ffffffff814a97ef>] mddev_delayed_delete+0x2f/0x40
 [<ffffffff8107e1ab>] process_one_work+0x11b/0x3f0
 [<ffffffff814a97c0>] ? restart_array+0xc0/0xc0
 [<ffffffff8107f94e>] worker_thread+0x12e/0x340
 [<ffffffff8107f820>] ? manage_workers.isra.29+0x1f0/0x1f0
 [<ffffffff81084e1e>] kthread+0x8e/0xa0
 [<ffffffff8160add4>] kernel_thread_helper+0x4/0x10
 [<ffffffff81084d90>] ? flush_kthread_worker+0x70/0x70
 [<ffffffff8160add0>] ? gs_change+0xb/0xb
Code: 80 00 00 00 00 55 48 89 e5 41 57 41 89 f7 41 56 41 89 ce 41 55 41
Quoted 2 lines

54 4c 8d 67 08 53 48 83 ec 18 89 55 c4 48 8b 57 08 4c 89 45 c8 <4c> 8b 2a 48 8d 42 e8 49 83 ed 18 49 39 d4 75 0d eb 40 0f 1f 84

It's a bug in local commit bc85cf83, for stacked devices we have not initialized the wait queues. So the below should fix it, as would always initializing all queue structures even for the partial use case.

block/blk-core.c +6 -4
index b477fa0..93eb3e4 100644
@@ -415,10 +415,12 @@ void blk_drain_queue(struct request_queue *q, bool drain_all)
 	 * allocation path, so the wakeup chaining is lost and we're
 	 * left with hung waiters. We need to wake up those waiters.
 	 */
-	spin_lock_irq(q->queue_lock);
-	for (i = 0; i < ARRAY_SIZE(q->rq.wait); i++)
-		wake_up_all(&q->rq.wait[i]);
-	spin_unlock_irq(q->queue_lock);
+	if (q->request_fn) {
+		spin_lock_irq(q->queue_lock);
+		for (i = 0; i < ARRAY_SIZE(q->rq.wait); i++)
+			wake_up_all(&q->rq.wait[i]);
+		spin_unlock_irq(q->queue_lock);
+	}
 }
 
 /**
Signature · 1 line
Jens Axboe
author
#3 lore raw
Quoted 64 lines · Jens Axboe
Quoted 47 lines · Jiri Slaby

Hi,

with today's -next I'm (reproducibly) getting this while updating packages:

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<ffffffff8108cd16>] __wake_up_common+0x26/0x90
PGD 463f1067 PUD 463f2067 PMD 0
Oops: 0000 [#1] SMP
CPU 1
Modules linked in:
Pid: 2711, comm: kworker/1:0 Not tainted 3.5.0-rc2-next-20120614_64+
#1752 Bochs Bochs
RIP: 0010:[<ffffffff8108cd16>]  [<ffffffff8108cd16>]
__wake_up_common+0x26/0x90
RSP: 0018:ffff880047221cb0  EFLAGS: 00010082
RAX: 0000000000000086 RBX: ffff880046350888 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880046350888
RBP: ffff880047221cf0 R08: 0000000000000000 R09: 00000001000c0009
R10: ffff880047804480 R11: 0000000000000000 R12: ffff880046350890
R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffff880049700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000045ced000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 2711, threadinfo ffff880047220000, task
ffff8800435bc5c0)
Stack:
 000000004628da68 0000000000000000 ffff88004970d340 ffff880046350888
 0000000000000086 0000000000000003 0000000000000000 0000000000000000
 ffff880047221d30 ffffffff8108d9a3 ffff88004970d340 ffff880046350848
Call Trace:
 [<ffffffff8108d9a3>] __wake_up+0x43/0x70
 [<ffffffff81267f96>] blk_drain_queue+0xf6/0x120
 [<ffffffff8126803f>] blk_cleanup_queue+0x7f/0xd0
 [<ffffffff814a9a80>] md_free+0x50/0x70
 [<ffffffff8127b3c2>] kobject_cleanup+0x82/0x1b0
 [<ffffffff8127b24b>] kobject_put+0x2b/0x60
 [<ffffffff814a97ef>] mddev_delayed_delete+0x2f/0x40
 [<ffffffff8107e1ab>] process_one_work+0x11b/0x3f0
 [<ffffffff814a97c0>] ? restart_array+0xc0/0xc0
 [<ffffffff8107f94e>] worker_thread+0x12e/0x340
 [<ffffffff8107f820>] ? manage_workers.isra.29+0x1f0/0x1f0
 [<ffffffff81084e1e>] kthread+0x8e/0xa0
 [<ffffffff8160add4>] kernel_thread_helper+0x4/0x10
 [<ffffffff81084d90>] ? flush_kthread_worker+0x70/0x70
 [<ffffffff8160add0>] ? gs_change+0xb/0xb
Code: 80 00 00 00 00 55 48 89 e5 41 57 41 89 f7 41 56 41 89 ce 41 55 41
Quoted 2 lines

54 4c 8d 67 08 53 48 83 ec 18 89 55 c4 48 8b 57 08 4c 89 45 c8 <4c> 8b 2a 48 8d 42 e8 49 83 ed 18 49 39 d4 75 0d eb 40 0f 1f 84

It's a bug in local commit bc85cf83, for stacked devices we have not initialized the wait queues. So the below should fix it, as would always initializing all queue structures even for the partial use case.

block/blk-core.c +0 -0
index b477fa0..93eb3e4 100644
@@ -415,10 +415,12 @@ void blk_drain_queue(struct request_queue *q, bool drain_all)
 	 * allocation path, so the wakeup chaining is lost and we're
 	 * left with hung waiters. We need to wake up those waiters.
 	 */

block/blk-core.c void blk_drain_queue(struct request_queue *q, bool drain_all) · line 423

Quoted 10 lines
block/blk-core.c +6 -4
-	spin_lock_irq(q->queue_lock);
-	for (i = 0; i < ARRAY_SIZE(q->rq.wait); i++)
-		wake_up_all(&q->rq.wait[i]);
-	spin_unlock_irq(q->queue_lock);
+	if (q->request_fn) {
+		spin_lock_irq(q->queue_lock);
+		for (i = 0; i < ARRAY_SIZE(q->rq.wait); i++)
+			wake_up_all(&q->rq.wait[i]);
+		spin_unlock_irq(q->queue_lock);
+	}

Yes, that fixed it.

thanks,

Signature · 2 lines
js
suse labs
#4 lore raw
Quoted 69 lines · Jiri Slaby
Quoted 68 lines · Jens Axboe
Quoted 47 lines · Jiri Slaby

Hi,

with today's -next I'm (reproducibly) getting this while updating packages:

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<ffffffff8108cd16>] __wake_up_common+0x26/0x90
PGD 463f1067 PUD 463f2067 PMD 0
Oops: 0000 [#1] SMP
CPU 1
Modules linked in:
Pid: 2711, comm: kworker/1:0 Not tainted 3.5.0-rc2-next-20120614_64+
#1752 Bochs Bochs
RIP: 0010:[<ffffffff8108cd16>]  [<ffffffff8108cd16>]
__wake_up_common+0x26/0x90
RSP: 0018:ffff880047221cb0  EFLAGS: 00010082
RAX: 0000000000000086 RBX: ffff880046350888 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880046350888
RBP: ffff880047221cf0 R08: 0000000000000000 R09: 00000001000c0009
R10: ffff880047804480 R11: 0000000000000000 R12: ffff880046350890
R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffff880049700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000045ced000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 2711, threadinfo ffff880047220000, task
ffff8800435bc5c0)
Stack:
  000000004628da68 0000000000000000 ffff88004970d340 ffff880046350888
  0000000000000086 0000000000000003 0000000000000000 0000000000000000
  ffff880047221d30 ffffffff8108d9a3 ffff88004970d340 ffff880046350848
Call Trace:
  [<ffffffff8108d9a3>] __wake_up+0x43/0x70
  [<ffffffff81267f96>] blk_drain_queue+0xf6/0x120
  [<ffffffff8126803f>] blk_cleanup_queue+0x7f/0xd0
  [<ffffffff814a9a80>] md_free+0x50/0x70
  [<ffffffff8127b3c2>] kobject_cleanup+0x82/0x1b0
  [<ffffffff8127b24b>] kobject_put+0x2b/0x60
  [<ffffffff814a97ef>] mddev_delayed_delete+0x2f/0x40
  [<ffffffff8107e1ab>] process_one_work+0x11b/0x3f0
  [<ffffffff814a97c0>] ? restart_array+0xc0/0xc0
  [<ffffffff8107f94e>] worker_thread+0x12e/0x340
  [<ffffffff8107f820>] ? manage_workers.isra.29+0x1f0/0x1f0
  [<ffffffff81084e1e>] kthread+0x8e/0xa0
  [<ffffffff8160add4>] kernel_thread_helper+0x4/0x10
  [<ffffffff81084d90>] ? flush_kthread_worker+0x70/0x70
  [<ffffffff8160add0>] ? gs_change+0xb/0xb
Code: 80 00 00 00 00 55 48 89 e5 41 57 41 89 f7 41 56 41 89 ce 41 55 41
Quoted 2 lines

54 4c 8d 67 08 53 48 83 ec 18 89 55 c4 48 8b 57 08 4c 89 45 c8 <4c> 8b 2a 48 8d 42 e8 49 83 ed 18 49 39 d4 75 0d eb 40 0f 1f 84

It's a bug in local commit bc85cf83, for stacked devices we have not initialized the wait queues. So the below should fix it, as would always initializing all queue structures even for the partial use case.

block/blk-core.c +0 -4
index b477fa0..93eb3e4 100644
@@ -415,10 +415,12 @@ void blk_drain_queue(struct request_queue *q, bool drain_all)
  	 * allocation path, so the wakeup chaining is lost and we're
  	 * left with hung waiters. We need to wake up those waiters.
  	 */
-	spin_lock_irq(q->queue_lock);
-	for (i = 0; i < ARRAY_SIZE(q->rq.wait); i++)
-		wake_up_all(&q->rq.wait[i]);
-	spin_unlock_irq(q->queue_lock);
Quoted 8 lines

block/blk-core.c void blk_drain_queue(struct request_queue *q, bool drain_all) · line 423

Quoted 6 lines
block/blk-core.c +6 -0
+	if (q->request_fn) {
+		spin_lock_irq(q->queue_lock);
+		for (i = 0; i < ARRAY_SIZE(q->rq.wait); i++)
+			wake_up_all(&q->rq.wait[i]);
+		spin_unlock_irq(q->queue_lock);
+	}

Yes, that fixed it.

Jiri, good to hear this fixes for you. BTW. How do you trigger this issue?

Jens, do you prefer to fix it up in your tree yourself or wait a patch from me?

Signature · 1 line
Asias
author
#5 lore raw
Quoted 5 lines · Asias He
Quoted 4 lines · Jiri Slaby
Quoted 3 lines · Jens Axboe
Quoted 2 lines · Jiri Slaby

with today's -next I'm (reproducibly) getting this while updating packages:

...

Quoted 1 line

Jiri, good to hear this fixes for you. BTW. How do you trigger this issue?

On opensuse factory, by zypper dup. It always died at the point of updating udev or multipath package.

regards,

Signature · 2 lines
js
suse labs
#6 lore raw
Quoted 11 lines · Jiri Slaby
Quoted 5 lines · Asias He
Quoted 4 lines · Jiri Slaby
Quoted 3 lines · Jens Axboe
Quoted 2 lines · Jiri Slaby

with today's -next I'm (reproducibly) getting this while updating packages:

...

Quoted 1 line

Jiri, good to hear this fixes for you. BTW. How do you trigger this issue?

On opensuse factory, by zypper dup. It always died at the point of updating udev or multipath package.

OK.

Signature · 1 line
Asias
maintainer
#7 lore raw
In reply to Asias He
Quoted 79 lines · Asias He
Quoted 69 lines · Jiri Slaby
Quoted 68 lines · Jens Axboe
Quoted 47 lines · Jiri Slaby

Hi,

with today's -next I'm (reproducibly) getting this while updating packages:

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<ffffffff8108cd16>] __wake_up_common+0x26/0x90
PGD 463f1067 PUD 463f2067 PMD 0
Oops: 0000 [#1] SMP
CPU 1
Modules linked in:
Pid: 2711, comm: kworker/1:0 Not tainted 3.5.0-rc2-next-20120614_64+
#1752 Bochs Bochs
RIP: 0010:[<ffffffff8108cd16>]  [<ffffffff8108cd16>]
__wake_up_common+0x26/0x90
RSP: 0018:ffff880047221cb0  EFLAGS: 00010082
RAX: 0000000000000086 RBX: ffff880046350888 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880046350888
RBP: ffff880047221cf0 R08: 0000000000000000 R09: 00000001000c0009
R10: ffff880047804480 R11: 0000000000000000 R12: ffff880046350890
R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffff880049700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000045ced000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 2711, threadinfo ffff880047220000, task
ffff8800435bc5c0)
Stack:
  000000004628da68 0000000000000000 ffff88004970d340 ffff880046350888
  0000000000000086 0000000000000003 0000000000000000 0000000000000000
  ffff880047221d30 ffffffff8108d9a3 ffff88004970d340 ffff880046350848
Call Trace:
  [<ffffffff8108d9a3>] __wake_up+0x43/0x70
  [<ffffffff81267f96>] blk_drain_queue+0xf6/0x120
  [<ffffffff8126803f>] blk_cleanup_queue+0x7f/0xd0
  [<ffffffff814a9a80>] md_free+0x50/0x70
  [<ffffffff8127b3c2>] kobject_cleanup+0x82/0x1b0
  [<ffffffff8127b24b>] kobject_put+0x2b/0x60
  [<ffffffff814a97ef>] mddev_delayed_delete+0x2f/0x40
  [<ffffffff8107e1ab>] process_one_work+0x11b/0x3f0
  [<ffffffff814a97c0>] ? restart_array+0xc0/0xc0
  [<ffffffff8107f94e>] worker_thread+0x12e/0x340
  [<ffffffff8107f820>] ? manage_workers.isra.29+0x1f0/0x1f0
  [<ffffffff81084e1e>] kthread+0x8e/0xa0
  [<ffffffff8160add4>] kernel_thread_helper+0x4/0x10
  [<ffffffff81084d90>] ? flush_kthread_worker+0x70/0x70
  [<ffffffff8160add0>] ? gs_change+0xb/0xb
Code: 80 00 00 00 00 55 48 89 e5 41 57 41 89 f7 41 56 41 89 ce 41 55 41
Quoted 2 lines

54 4c 8d 67 08 53 48 83 ec 18 89 55 c4 48 8b 57 08 4c 89 45 c8 <4c> 8b 2a 48 8d 42 e8 49 83 ed 18 49 39 d4 75 0d eb 40 0f 1f 84

It's a bug in local commit bc85cf83, for stacked devices we have not initialized the wait queues. So the below should fix it, as would always initializing all queue structures even for the partial use case.

block/blk-core.c +0 -4
index b477fa0..93eb3e4 100644
@@ -415,10 +415,12 @@ void blk_drain_queue(struct request_queue *q, bool drain_all)
       * allocation path, so the wakeup chaining is lost and we're
       * left with hung waiters. We need to wake up those waiters.
       */
-    spin_lock_irq(q->queue_lock);
-    for (i = 0; i < ARRAY_SIZE(q->rq.wait); i++)
-        wake_up_all(&q->rq.wait[i]);
-    spin_unlock_irq(q->queue_lock);
Quoted 8 lines

block/blk-core.c void blk_drain_queue(struct request_queue *q, bool drain_all) · line 423

Quoted 6 lines
block/blk-core.c +6 -0
+    if (q->request_fn) {
+        spin_lock_irq(q->queue_lock);
+        for (i = 0; i < ARRAY_SIZE(q->rq.wait); i++)
+            wake_up_all(&q->rq.wait[i]);
+        spin_unlock_irq(q->queue_lock);
+    }

Yes, that fixed it.

Quoted 5 lines

Jiri, good to hear this fixes for you. BTW. How do you trigger this issue?

Jens, do you prefer to fix it up in your tree yourself or wait a patch from me?

I will fixup the existing patch, so we don't have this problem in a bisection point after it's merged.

Signature · 1 line
Jens Axboe
#8 lore raw
Quoted 80 lines · Jens Axboe
Quoted 79 lines · Asias He
Quoted 77 lines · Jiri Slaby
Quoted 64 lines · Jens Axboe
Quoted 47 lines · Jiri Slaby

Hi,

with today's -next I'm (reproducibly) getting this while updating packages:

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<ffffffff8108cd16>] __wake_up_common+0x26/0x90
PGD 463f1067 PUD 463f2067 PMD 0
Oops: 0000 [#1] SMP
CPU 1
Modules linked in:
Pid: 2711, comm: kworker/1:0 Not tainted 3.5.0-rc2-next-20120614_64+
#1752 Bochs Bochs
RIP: 0010:[<ffffffff8108cd16>]  [<ffffffff8108cd16>]
__wake_up_common+0x26/0x90
RSP: 0018:ffff880047221cb0  EFLAGS: 00010082
RAX: 0000000000000086 RBX: ffff880046350888 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000003 RDI: ffff880046350888
RBP: ffff880047221cf0 R08: 0000000000000000 R09: 00000001000c0009
R10: ffff880047804480 R11: 0000000000000000 R12: ffff880046350890
R13: 0000000000000003 R14: 0000000000000000 R15: 0000000000000003
FS:  0000000000000000(0000) GS:ffff880049700000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000045ced000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/1:0 (pid: 2711, threadinfo ffff880047220000, task
ffff8800435bc5c0)
Stack:
   000000004628da68 0000000000000000 ffff88004970d340 ffff880046350888
   0000000000000086 0000000000000003 0000000000000000 0000000000000000
   ffff880047221d30 ffffffff8108d9a3 ffff88004970d340 ffff880046350848
Call Trace:
   [<ffffffff8108d9a3>] __wake_up+0x43/0x70
   [<ffffffff81267f96>] blk_drain_queue+0xf6/0x120
   [<ffffffff8126803f>] blk_cleanup_queue+0x7f/0xd0
   [<ffffffff814a9a80>] md_free+0x50/0x70
   [<ffffffff8127b3c2>] kobject_cleanup+0x82/0x1b0
   [<ffffffff8127b24b>] kobject_put+0x2b/0x60
   [<ffffffff814a97ef>] mddev_delayed_delete+0x2f/0x40
   [<ffffffff8107e1ab>] process_one_work+0x11b/0x3f0
   [<ffffffff814a97c0>] ? restart_array+0xc0/0xc0
   [<ffffffff8107f94e>] worker_thread+0x12e/0x340
   [<ffffffff8107f820>] ? manage_workers.isra.29+0x1f0/0x1f0
   [<ffffffff81084e1e>] kthread+0x8e/0xa0
   [<ffffffff8160add4>] kernel_thread_helper+0x4/0x10
   [<ffffffff81084d90>] ? flush_kthread_worker+0x70/0x70
   [<ffffffff8160add0>] ? gs_change+0xb/0xb
Code: 80 00 00 00 00 55 48 89 e5 41 57 41 89 f7 41 56 41 89 ce 41 55 41
Quoted 2 lines

54 4c 8d 67 08 53 48 83 ec 18 89 55 c4 48 8b 57 08 4c 89 45 c8 <4c> 8b 2a 48 8d 42 e8 49 83 ed 18 49 39 d4 75 0d eb 40 0f 1f 84

It's a bug in local commit bc85cf83, for stacked devices we have not initialized the wait queues. So the below should fix it, as would always initializing all queue structures even for the partial use case.

block/blk-core.c +0 -0
index b477fa0..93eb3e4 100644
@@ -415,10 +415,12 @@ void blk_drain_queue(struct request_queue *q, bool drain_all)
        * allocation path, so the wakeup chaining is lost and we're
        * left with hung waiters. We need to wake up those waiters.
        */

block/blk-core.c void blk_drain_queue(struct request_queue *q, bool drain_all) · line 423

Quoted 10 lines
block/blk-core.c +6 -4
-    spin_lock_irq(q->queue_lock);
-    for (i = 0; i < ARRAY_SIZE(q->rq.wait); i++)
-        wake_up_all(&q->rq.wait[i]);
-    spin_unlock_irq(q->queue_lock);
+    if (q->request_fn) {
+        spin_lock_irq(q->queue_lock);
+        for (i = 0; i < ARRAY_SIZE(q->rq.wait); i++)
+            wake_up_all(&q->rq.wait[i]);
+        spin_unlock_irq(q->queue_lock);
+    }

Yes, that fixed it.

Quoted 8 lines
Quoted 5 lines

Jiri, good to hear this fixes for you. BTW. How do you trigger this issue?

Jens, do you prefer to fix it up in your tree yourself or wait a patch from me?

I will fixup the existing patch, so we don't have this problem in a bisection point after it's merged.

OK. Thanks.

Signature · 1 line
Asias