2.6.24-rc1-gb4f5550 oops
Quoted 18 lines · Grant Wilson
[18073.371126] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP: [18073.371134] [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110 [18073.371144] PGD 81f9067 PUD 81c8067 PMD 0 [18073.371151] Oops: 0000 [1] PREEMPT SMP [18073.371157] CPU 2 [18073.371161] Modules linked in: vfat fat [18073.371168] Pid: 4639, comm: kwin Not tainted 2.6.24-rc1 #1 [18073.371171] RIP: 0010:[<ffffffff8023572e>] [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110 [18073.371177] RSP: 0018:ffff810008531a78 EFLAGS: 00010006 [18073.371179] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [18073.371183] RDX: ffff810004441bf0 RSI: ffff81000801e860 RDI: ffff81000444ab80 [18073.371186] RBP: ffff810008531aa8 R08: 000000d0d47a4a90 R09: 0000000000000000 [18073.371188] R10: ffff810004441bf0 R11: 0000000000000001 R12: ffff810006520400 [18073.371190] R13: ffff81000801e860 R14: ffff81000a63a000 R15: ffff81000443d8e0 [18073.371193] FS: 00002b7d646a86f0(0000) GS:ffff810004c11780(0000) knlGS:0000000000000000 [18073.371196] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [18073.371199] CR2: 0000000000000120 CR3: 0000000008495000 CR4: 00000000000006e0
Quoted 8 lines
[18073.371202] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18073.371211] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18073.371214] Process kwin (pid: 4639, threadinfo ffff810008530000, task ffff81000840a860) [18073.371216] Stack: ffff81000444ab80 0000000000000001 ffff81000801e860 ffff81000444ab80 [18073.371231] 0000000000000002 ffff81000443d8e0 ffff810008531b38 ffffffff8023061e [18073.371238] 0000000000000000 ffff810004441b80 0000000000000002 0000000100000000 [18073.371245] Call Trace: [18073.371250] [<ffffffff8023061e>] try_to_wake_up+0x2fe/0x3a0
I suspect I see the bug in that area, but I am not sure it can explain this trace completely.
Suppose that the SCHED_FIFO task does
switch_uid(new_user);
Now, p->se.cfs_rq and p->se.parent both point into the old user_struct->tg because sched_move_task() doesn't call set_task_cfs_rq() for !fair_sched_class case.
Suppose that old user_struct/task_group is freed/reused, and the task does
sched_setscheduler(SCHED_NORMAL);
__setscheduler() sets fair_sched_class, but doesn't update ->se.cfs_rq/parent which point to the freed memory.
This means that check_preempt_wakeup() doing
while (!is_same_group(se, pse)) {
se = parent_entity(se);
pse = parent_entity(pse);
}may OOPS in a similar way if rq->curr or p did something like above.
Perhaps we need something like the patch below, note that __setscheduler() can't do set_task_cfs_rq().
Oleg.
- renamed from sched.c +3 -1
@@ -7068,8 +7068,10 @@ void sched_move_task(struct task_struct rq = task_rq_lock(tsk, &flags); - if (tsk->sched_class != &fair_sched_class) + if (tsk->sched_class != &fair_sched_class) { + set_task_cfs_rq(tsk); goto done; + } update_rq_clock(rq);
Quoted 26 lines · Oleg Nesterov
Quoted 17 lines
[18073.371126] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP: [18073.371134] [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110 [18073.371144] PGD 81f9067 PUD 81c8067 PMD 0 [18073.371151] Oops: 0000 [1] PREEMPT SMP [18073.371157] CPU 2 [18073.371161] Modules linked in: vfat fat [18073.371168] Pid: 4639, comm: kwin Not tainted 2.6.24-rc1 #1 [18073.371171] RIP: 0010:[<ffffffff8023572e>] [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110 [18073.371177] RSP: 0018:ffff810008531a78 EFLAGS: 00010006 [18073.371179] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [18073.371183] RDX: ffff810004441bf0 RSI: ffff81000801e860 RDI: ffff81000444ab80 [18073.371186] RBP: ffff810008531aa8 R08: 000000d0d47a4a90 R09: 0000000000000000 [18073.371188] R10: ffff810004441bf0 R11: 0000000000000001 R12: ffff810006520400 [18073.371190] R13: ffff81000801e860 R14: ffff81000a63a000 R15: ffff81000443d8e0 [18073.371193] FS: 00002b7d646a86f0(0000) GS:ffff810004c11780(0000) knlGS:0000000000000000 [18073.371196] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [18073.371199] CR2: 0000000000000120 CR3: 0000000008495000 CR4: 00000000000006e0Quoted 8 lines
[18073.371202] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18073.371211] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18073.371214] Process kwin (pid: 4639, threadinfo ffff810008530000, task ffff81000840a860) [18073.371216] Stack: ffff81000444ab80 0000000000000001 ffff81000801e860 ffff81000444ab80 [18073.371231] 0000000000000002 ffff81000443d8e0 ffff810008531b38 ffffffff8023061e [18073.371238] 0000000000000000 ffff810004441b80 0000000000000002 0000000100000000 [18073.371245] Call Trace: [18073.371250] [<ffffffff8023061e>] try_to_wake_up+0x2fe/0x3a0
Quoted 2 lines
I suspect I see the bug in that area, but I am not sure it can explain this trace completely.
there's a fix pending from Dmitry - please see below. It took days for Grant to trigger the crash so it needs some time to be confirmed but it could explain the crash in theory.
Ingo
---------------------->
Subject: sched: fix __set_task_cpu() SMP race
From: Dmitry Adamushko <[email protected]>
Grant Wilson has reported rare SCHED_FAIR_USER crashes on his quad-core system, which crashes can only be explained via runqueue corruption.
there is a narrow SMP race in __set_task_cpu(): after ->cpu is set up to a new value, task_rq_lock(p, ...) can be successfuly executed on another CPU. We must ensure that updates of per-task data have been completed by this moment.
this bug has been hiding in the Linux scheduler for an eternity (we never had any explicit barrier for task->cpu in set_task_cpu() - so the bug was introduced in 2.5.1), but only became visible via set_task_cfs_rq() being accidentally put after the task->cpu update. It also probably needs a sufficiently out-of-order CPU to trigger.
--- kernel/sched.c | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-)
kernel/sched.c +12 -6
@@ -217,15 +217,15 @@ static inline struct task_group *task_gr }/* Change a task's cfs_rq and parent entity if it moves across CPUs/groups */ -static inline void set_task_cfs_rq (struct task_struct *p) +static inline void set_task_cfs_rq (struct task_struct *p,unsigned int cpu) { - p->se.cfs_rq =task_group (p)->cfs_rq[task_cpu (p)]; - p->se.parent =task_group (p)->se[task_cpu (p)]; + p->se.cfs_rq =task_group (p)->cfs_rq[cpu]; + p->se.parent =task_group (p)->se[cpu]; }#else -static inline void set_task_cfs_rq (struct task_struct *p) { } +static inline void set_task_cfs_rq (struct task_struct *p,unsigned int cpu) { }#endif /* CONFIG_FAIR_GROUP_SCHED */
@@ -1023,10 +1023,16 @@ unsigned long weighted_cpuload(const intstatic inline void __set_task_cpu (struct task_struct *p,unsigned int cpu) { +set_task_cfs_rq (p, cpu);#ifdef CONFIG_SMP +/* +* After ->cpu is set up to a new value, task_rq_lock(p, ...) can be +* successfuly executed on another CPU. We must ensure that updates of +* per-task data have been completed by this moment. +*/ +smp_wmb ();task_thread_info (p)->cpu = cpu;#endif -set_task_cfs_rq (p); }#ifdef CONFIG_SMP
@@ -7111,7 +7117,7 @@ void sched_move_task(struct task_struct tsk->sched_class->put_prev_task (rq, tsk); } -set_task_cfs_rq (tsk); +set_task_cfs_rq (tsk,task_cpu (tsk));if (on_rq) {if (unlikely (running))
Quoted 29 lines · Ingo Molnar
Quoted 26 lines · Oleg Nesterov
Quoted 25 lines
[18073.371126] Unable to handle kernel NULL pointer dereference at 0000000000000120 RIP: [18073.371134] [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110 [18073.371144] PGD 81f9067 PUD 81c8067 PMD 0 [18073.371151] Oops: 0000 [1] PREEMPT SMP [18073.371157] CPU 2 [18073.371161] Modules linked in: vfat fat [18073.371168] Pid: 4639, comm: kwin Not tainted 2.6.24-rc1 #1 [18073.371171] RIP: 0010:[<ffffffff8023572e>] [<ffffffff8023572e>] check_preempt_wakeup+0x6e/0x110 [18073.371177] RSP: 0018:ffff810008531a78 EFLAGS: 00010006 [18073.371179] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000 [18073.371183] RDX: ffff810004441bf0 RSI: ffff81000801e860 RDI: ffff81000444ab80 [18073.371186] RBP: ffff810008531aa8 R08: 000000d0d47a4a90 R09: 0000000000000000 [18073.371188] R10: ffff810004441bf0 R11: 0000000000000001 R12: ffff810006520400 [18073.371190] R13: ffff81000801e860 R14: ffff81000a63a000 R15: ffff81000443d8e0 [18073.371193] FS: 00002b7d646a86f0(0000) GS:ffff810004c11780(0000) knlGS:0000000000000000 [18073.371196] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [18073.371199] CR2: 0000000000000120 CR3: 0000000008495000 CR4: 00000000000006e0 [18073.371202] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [18073.371211] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 [18073.371214] Process kwin (pid: 4639, threadinfo ffff810008530000, task ffff81000840a860) [18073.371216] Stack: ffff81000444ab80 0000000000000001 ffff81000801e860 ffff81000444ab80 [18073.371231] 0000000000000002 ffff81000443d8e0 ffff810008531b38 ffffffff8023061e [18073.371238] 0000000000000000 ffff810004441b80 0000000000000002 0000000100000000 [18073.371245] Call Trace: [18073.371250] [<ffffffff8023061e>] try_to_wake_up+0x2fe/0x3a0
Quoted 6 lines
Quoted 2 lines
I suspect I see the bug in that area, but I am not sure it can explain this trace completely.
there's a fix pending from Dmitry - please see below. It took days for Grant to trigger the crash so it needs some time to be confirmed but it could explain the crash in theory.
Yes I agree, it can explain the crash.
However, this patch can't fix the bug I was talking about (of course, unless I missed something). It is still possible that the "fair_sched_class" task can have ->se.cfs_rq/parent pointing to the freed memory, no?
Oleg.
Quoted 7 lines · Oleg Nesterov
Quoted 6 lines
Quoted 2 lines
I suspect I see the bug in that area, but I am not sure it can explain this trace completely.
there's a fix pending from Dmitry - please see below. It took days for Grant to trigger the crash so it needs some time to be confirmed but it could explain the crash in theory.
Quoted 6 lines
Yes I agree, it can explain the crash.
However, this patch can't fix the bug I was talking about (of course, unless I missed something). It is still possible that the "
fair_sched_class" task can have ->se.cfs_rq/parent pointing to the freed memory, no?
yeah, agreed. I've queued up your fix for merging - thanks!
Ingo
btw., could you get your Signed-off-by line for that fix?
Ingo
Quoted 1 line · Oleg Nesterov
Suppose that old user_struct/task_group is freed/reused, and the task does
Shouldn't this old user actually be the root user_struct?
Signature · 2 lines
Regards, vatsa
Quoted 5 lines · Srivatsa Vaddagiri
Quoted 1 line · Oleg Nesterov
Suppose that old user_struct/task_group is freed/reused, and the task does
Shouldn't this old user actually be the root
user_struct?
Well, the S_ISUID application can switch ->user from !root_user (note that setuid exec doesn't change ->user), and we also have CAP_SETUID...
But yes, I agree, this all is very unlikely.
Oleg.