crash in pick_next_task_fair
hi,
we see crashes on older kernel (2.6.32 based) in pick_next_task_fair:
...
#8 [ffff8819335efa20] page_fault at ffffffff8152d375
[exception RIP: rb_next+1]
RIP: ffffffff81292ae1 RSP: ffff8819335efad8 RFLAGS: 00010046
RAX: 0000000000000000 RBX: ffff8810b8a96928 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880ff7602ae0 RDI: 0000000000000010
RBP: ffff8819335efb28 R8: 0000000000000000 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000
R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018#9 [ffff8819335efae0] pick_next_task_fair at ffffffff8106c511 #10 [ffff8819335efb30] schedule at ffffffff81529746 #11 [ffff8819335efc00] futex_wait_queue_me at ffffffff810b226a #12 [ffff8819335efc40] futex_wait at ffffffff810b33a0 #13 [ffff8819335efdb0] do_futex at ffffffff810b4c91 #14 [ffff8819335efef0] sys_futex at ffffffff810b56cb #15 [ffff8819335eff80] system_call_fastpath at ffffffff8100b072
...
- pick_next_task_fair calls pick_next_entity
- pick_next_entity calls __pick_first_entity and gets NULL from cfs_rq->rb_leftmost
- cfs_rq->skip is NULL so it gets through (cfs_rq->skip == se) condition
and calls __pick_next_entity(se) which fails on rb_next(&se->run_node)
it seems that upstream could fail also via:
- pick_next_task_fair calls pick_next_entity with curr == NULL (simple case)
- __pick_first_entity calls __pick_first_entity and gets NULL from cfs_rq->rb_leftmost
- pick_next_entity returns NULL
- pick_next_task_fair calls set_next_entity with se = NULL
- set_next_entity fails on se->on_rq
would attached patch make sense or do I miss some rb_leftmost rules/behaviour?
thanks,
jirka
---
kernel/sched/fair.c +1 -1
@@ -538,7 +538,7 @@ static void __dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)struct sched_entity *__pick_first_entity (struct cfs_rq *cfs_rq) { -struct rb_node *left = cfs_rq->rb_leftmost; +struct rb_node *left = cfs_rq->rb_leftmost ? :rb_first (&cfs_rq->tasks_timeline);if (!left)return NULL ;
Peter Zijlstra
maintainerQuoted 2 lines · Jiri Olsa
hi,
we see crashes on older kernel (2.6.32 based) inpick_next_task_fair:
This is a RHEL (6?) kernel, right? So not really anything like an actual 2.6.32.
Lemme use a random RHEL6 kernel I found on github instead of staring at a local v2.6.32...
https://github.com/dduval/kernel-rhel6
Quoted 20 lines
...
#8 [ffff8819335efa20]page_faultatffffffff8152d375[exception RIP: rb_next+1] RIP: ffffffff81292ae1 RSP: ffff8819335efad8 RFLAGS: 00010046 RAX: 0000000000000000 RBX: ffff8810b8a96928 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff880ff7602ae0 RDI: 0000000000000010 RBP: ffff8819335efb28 R8: 0000000000000000 R9: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000 ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018#9 [
ffff8819335efae0]pick_next_task_fairatffffffff8106c511#10 [ffff8819335efb30] schedule atffffffff81529746#11 [ffff8819335efc00]futex_wait_queue_meatffffffff810b226a#12 [ffff8819335efc40]futex_waitatffffffff810b33a0#13 [ffff8819335efdb0]do_futexatffffffff810b4c91#14 [ffff8819335efef0]sys_futexatffffffff810b56cb#15 [ffff8819335eff80]system_call_fastpathatffffffff8100b072
...
Quoted 4 lines
-
pick_next_task_faircallspick_next_entity
-pick_next_entitycalls__pick_first_entityand gets NULL fromcfs_rq->rb_leftmost
-cfs_rq->skipis NULL so it gets through (cfs_rq->skip== se) conditionand calls __pick_next_entity(se) which fails on rb_next(&se->run_node)
So that code is a whole lot simpler than what we have upstream, but indeed comparable to the 'simple' branch.
Quoted 3 lines
it seems that upstream could fail also via:
-pick_next_task_faircallspick_next_entitywith curr == NULL (simple case)
-__pick_first_entitycalls__pick_first_entityand gets NULL fromcfs_rq->rb_leftmost
This _should_ be impossible:
- if !cfs_rq->nr_running, we'll never get here;
- if cfs_rq->nr_running == 1 && prev->sched_class == &fair_sched_class, put_prev_entity() will have put prev back in the rb-tree and we have a leftmost;
- if cfs_rq->nr_running >= 1, there are entries in the tree, therefore leftmost must not be NULL.
Quoted 1 line
would attached patch make sense or do I miss some
rb_leftmostrules/behaviour?
If there is _anything_ in the tree, there must be a leftmost. If the tree is empty, we should not be trying to find a task in it due to nr_running checks prior to attempting to do so.