Skip to contentSkip to conversation

2.6.24-rc1-gb4f5550 oops

DISCUSSION Oleg Nesterov 3 participants 7 messages Nov 14, 2007

lore
Share & tools
lore
#0 lore raw
Quoted 18 lines · Grant Wilson
[18073.371126] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP:
[18073.371134]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
[18073.371144] PGD 81f9067 PUD 81c8067 PMD 0
[18073.371151] Oops: 0000 [1] PREEMPT SMP
[18073.371157] CPU 2
[18073.371161] Modules linked in: vfat fat
[18073.371168] Pid: 4639, comm: kwin Not tainted 2.6.24-rc1 #1
[18073.371171] RIP: 0010:[<ffffffff8023572e>]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
[18073.371177] RSP: 0018:ffff810008531a78  EFLAGS: 00010006
[18073.371179] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[18073.371183] RDX: ffff810004441bf0 RSI: ffff81000801e860 RDI: ffff81000444ab80
[18073.371186] RBP: ffff810008531aa8 R08: 000000d0d47a4a90 R09: 0000000000000000
[18073.371188] R10: ffff810004441bf0 R11: 0000000000000001 R12: ffff810006520400
[18073.371190] R13: ffff81000801e860 R14: ffff81000a63a000 R15: ffff81000443d8e0
[18073.371193] FS:  00002b7d646a86f0(0000) GS:ffff810004c11780(0000) knlGS:0000000000000000
[18073.371196] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[18073.371199] CR2: 0000000000000120 CR3: 0000000008495000 CR4: 00000000000006e0
Quoted 8 lines
[18073.371202] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[18073.371211] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[18073.371214] Process kwin (pid: 4639, threadinfo ffff810008530000, task ffff81000840a860)
[18073.371216] Stack:  ffff81000444ab80 0000000000000001 ffff81000801e860 ffff81000444ab80
[18073.371231]  0000000000000002 ffff81000443d8e0 ffff810008531b38 ffffffff8023061e
[18073.371238]  0000000000000000 ffff810004441b80 0000000000000002 0000000100000000
[18073.371245] Call Trace:
[18073.371250]  [<ffffffff8023061e>] try_to_wake_up+0x2fe/0x3a0

I suspect I see the bug in that area, but I am not sure it can explain this trace completely.

Suppose that the SCHED_FIFO task does

	switch_uid(new_user);

Now, p->se.cfs_rq and p->se.parent both point into the old user_struct->tg because sched_move_task() doesn't call set_task_cfs_rq() for !fair_sched_class case.

Suppose that old user_struct/task_group is freed/reused, and the task does

	sched_setscheduler(SCHED_NORMAL);

__setscheduler() sets fair_sched_class, but doesn't update ->se.cfs_rq/parent which point to the freed memory.

This means that check_preempt_wakeup() doing

		while (!is_same_group(se, pse)) {
			se = parent_entity(se);
			pse = parent_entity(pse);
		}

may OOPS in a similar way if rq->curr or p did something like above.

Perhaps we need something like the patch below, note that __setscheduler() can't do set_task_cfs_rq().

Oleg.

- renamed from sched.c +3 -1
@@ -7068,8 +7068,10 @@ void sched_move_task(struct task_struct 
 
 	rq = task_rq_lock(tsk, &flags);
 
-	if (tsk->sched_class != &fair_sched_class)
+	if (tsk->sched_class != &fair_sched_class) {
+		set_task_cfs_rq(tsk);
 		goto done;
+	}
 
 	update_rq_clock(rq);
 
#1 lore raw
Quoted 26 lines · Oleg Nesterov
Quoted 17 lines
[18073.371126] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP:
[18073.371134]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
[18073.371144] PGD 81f9067 PUD 81c8067 PMD 0
[18073.371151] Oops: 0000 [1] PREEMPT SMP
[18073.371157] CPU 2
[18073.371161] Modules linked in: vfat fat
[18073.371168] Pid: 4639, comm: kwin Not tainted 2.6.24-rc1 #1
[18073.371171] RIP: 0010:[<ffffffff8023572e>]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
[18073.371177] RSP: 0018:ffff810008531a78  EFLAGS: 00010006
[18073.371179] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[18073.371183] RDX: ffff810004441bf0 RSI: ffff81000801e860 RDI: ffff81000444ab80
[18073.371186] RBP: ffff810008531aa8 R08: 000000d0d47a4a90 R09: 0000000000000000
[18073.371188] R10: ffff810004441bf0 R11: 0000000000000001 R12: ffff810006520400
[18073.371190] R13: ffff81000801e860 R14: ffff81000a63a000 R15: ffff81000443d8e0
[18073.371193] FS:  00002b7d646a86f0(0000) GS:ffff810004c11780(0000) knlGS:0000000000000000
[18073.371196] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[18073.371199] CR2: 0000000000000120 CR3: 0000000008495000 CR4: 00000000000006e0
Quoted 8 lines
[18073.371202] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[18073.371211] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[18073.371214] Process kwin (pid: 4639, threadinfo ffff810008530000, task ffff81000840a860)
[18073.371216] Stack:  ffff81000444ab80 0000000000000001 ffff81000801e860 ffff81000444ab80
[18073.371231]  0000000000000002 ffff81000443d8e0 ffff810008531b38 ffffffff8023061e
[18073.371238]  0000000000000000 ffff810004441b80 0000000000000002 0000000100000000
[18073.371245] Call Trace:
[18073.371250]  [<ffffffff8023061e>] try_to_wake_up+0x2fe/0x3a0
Quoted 2 lines

I suspect I see the bug in that area, but I am not sure it can explain this trace completely.

there's a fix pending from Dmitry - please see below. It took days for Grant to trigger the crash so it needs some time to be confirmed but it could explain the crash in theory.

Ingo

---------------------->
Subject: sched: fix __set_task_cpu() SMP race
From: Dmitry Adamushko <[email protected]>

Grant Wilson has reported rare SCHED_FAIR_USER crashes on his quad-core system, which crashes can only be explained via runqueue corruption.

there is a narrow SMP race in __set_task_cpu(): after ->cpu is set up to a new value, task_rq_lock(p, ...) can be successfuly executed on another CPU. We must ensure that updates of per-task data have been completed by this moment.

this bug has been hiding in the Linux scheduler for an eternity (we never had any explicit barrier for task->cpu in set_task_cpu() - so the bug was introduced in 2.5.1), but only became visible via set_task_cfs_rq() being accidentally put after the task->cpu update. It also probably needs a sufficiently out-of-order CPU to trigger.

---
 kernel/sched.c |   18 ++++++++++++------
 1 file changed, 12 insertions(+), 6 deletions(-)
kernel/sched.c +12 -6
Index: linux/kernel/sched.c
===================================================================
@@ -217,15 +217,15 @@ static inline struct task_group *task_gr
 }
 
 /* Change a task's cfs_rq and parent entity if it moves across CPUs/groups */
-static inline void set_task_cfs_rq(struct task_struct *p)
+static inline void set_task_cfs_rq(struct task_struct *p, unsigned int cpu)
 {
-	p->se.cfs_rq = task_group(p)->cfs_rq[task_cpu(p)];
-	p->se.parent = task_group(p)->se[task_cpu(p)];
+	p->se.cfs_rq = task_group(p)->cfs_rq[cpu];
+	p->se.parent = task_group(p)->se[cpu];
 }
 
 #else
 
-static inline void set_task_cfs_rq(struct task_struct *p) { }
+static inline void set_task_cfs_rq(struct task_struct *p, unsigned int cpu) { }
 
 #endif	/* CONFIG_FAIR_GROUP_SCHED */
 
@@ -1023,10 +1023,16 @@ unsigned long weighted_cpuload(const int
 
 static inline void __set_task_cpu(struct task_struct *p, unsigned int cpu)
 {
+	set_task_cfs_rq(p, cpu);
 #ifdef CONFIG_SMP
+	/*
+	 * After ->cpu is set up to a new value, task_rq_lock(p, ...) can be
+	 * successfuly executed on another CPU. We must ensure that updates of
+	 * per-task data have been completed by this moment.
+	 */
+	smp_wmb();
 	task_thread_info(p)->cpu = cpu;
 #endif
-	set_task_cfs_rq(p);
 }
 
 #ifdef CONFIG_SMP
@@ -7111,7 +7117,7 @@ void sched_move_task(struct task_struct 
 			tsk->sched_class->put_prev_task(rq, tsk);
 	}
 
-	set_task_cfs_rq(tsk);
+	set_task_cfs_rq(tsk, task_cpu(tsk));
 
 	if (on_rq) {
 		if (unlikely(running))
#2 lore raw
Quoted 29 lines · Ingo Molnar
Quoted 26 lines · Oleg Nesterov
Quoted 25 lines
[18073.371126] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP:
[18073.371134]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
[18073.371144] PGD 81f9067 PUD 81c8067 PMD 0
[18073.371151] Oops: 0000 [1] PREEMPT SMP
[18073.371157] CPU 2
[18073.371161] Modules linked in: vfat fat
[18073.371168] Pid: 4639, comm: kwin Not tainted 2.6.24-rc1 #1
[18073.371171] RIP: 0010:[<ffffffff8023572e>]  [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110
[18073.371177] RSP: 0018:ffff810008531a78  EFLAGS: 00010006
[18073.371179] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
[18073.371183] RDX: ffff810004441bf0 RSI: ffff81000801e860 RDI: ffff81000444ab80
[18073.371186] RBP: ffff810008531aa8 R08: 000000d0d47a4a90 R09: 0000000000000000
[18073.371188] R10: ffff810004441bf0 R11: 0000000000000001 R12: ffff810006520400
[18073.371190] R13: ffff81000801e860 R14: ffff81000a63a000 R15: ffff81000443d8e0
[18073.371193] FS:  00002b7d646a86f0(0000) GS:ffff810004c11780(0000) knlGS:0000000000000000
[18073.371196] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[18073.371199] CR2: 0000000000000120 CR3: 0000000008495000 CR4: 00000000000006e0
[18073.371202] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[18073.371211] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[18073.371214] Process kwin (pid: 4639, threadinfo ffff810008530000, task ffff81000840a860)
[18073.371216] Stack:  ffff81000444ab80 0000000000000001 ffff81000801e860 ffff81000444ab80
[18073.371231]  0000000000000002 ffff81000443d8e0 ffff810008531b38 ffffffff8023061e
[18073.371238]  0000000000000000 ffff810004441b80 0000000000000002 0000000100000000
[18073.371245] Call Trace:
[18073.371250]  [<ffffffff8023061e>] try_to_wake_up+0x2fe/0x3a0
Quoted 6 lines
Quoted 2 lines

I suspect I see the bug in that area, but I am not sure it can explain this trace completely.

there's a fix pending from Dmitry - please see below. It took days for Grant to trigger the crash so it needs some time to be confirmed but it could explain the crash in theory.

Yes I agree, it can explain the crash.

However, this patch can't fix the bug I was talking about (of course, unless I missed something). It is still possible that the "fair_sched_class" task can have ->se.cfs_rq/parent pointing to the freed memory, no?

Oleg.

#3 lore raw
Quoted 7 lines · Oleg Nesterov
Quoted 6 lines
Quoted 2 lines

I suspect I see the bug in that area, but I am not sure it can explain this trace completely.

there's a fix pending from Dmitry - please see below. It took days for Grant to trigger the crash so it needs some time to be confirmed but it could explain the crash in theory.

Quoted 6 lines

Yes I agree, it can explain the crash.

However, this patch can't fix the bug I was talking about (of course, unless I missed something). It is still possible that the "fair_sched_class" task can have ->se.cfs_rq/parent pointing to the freed memory, no?

yeah, agreed. I've queued up your fix for merging - thanks!

Ingo

#4 lore raw

btw., could you get your Signed-off-by line for that fix?

Ingo

#5 lore raw
In reply to Oleg Nesterov
Quoted 1 line · Oleg Nesterov

Suppose that old user_struct/task_group is freed/reused, and the task does

Shouldn't this old user actually be the root user_struct?

Signature · 2 lines
Regards,
vatsa
#6 lore raw
Quoted 5 lines · Srivatsa Vaddagiri
Quoted 1 line · Oleg Nesterov

Suppose that old user_struct/task_group is freed/reused, and the task does

Shouldn't this old user actually be the root user_struct?

Well, the S_ISUID application can switch ->user from !root_user (note that setuid exec doesn't change ->user), and we also have CAP_SETUID...

But yes, I agree, this all is very unlikely.

Oleg.