Skip to contentSkip to conversation

sched,numa: invalid memory access in account_entity_dequeue

DISCUSSION Sasha Levin 2 participants 5 messages May 3, 2014 – May 6, 2014

lore
Share & tools
lore
#0 lore raw

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel I've stumbled on the following:

[ 1796.591361] BUG: unable to handle kernel paging request at fffffffedf97f040
[ 1796.592665] IP: __cpu_to_node (arch/x86/mm/numa.c:777)
[ 1796.593710] PGD 21e30067 PUD 0
[ 1796.594174] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 1796.594937] Dumping ftrace buffer:
[ 1796.595678]    (ftrace buffer empty)
[ 1796.596329] Modules linked in:
[ 1796.596733] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W     3.15.0-rc3-next-20140502-sasha-00019-g5cb1c98 #431
[ 1796.598143] task: ffff8803345b8000 ti: ffff880035fc0000 task.ti: ffff880035fc0000
[ 1796.598975] RIP: __cpu_to_node (arch/x86/mm/numa.c:777)
[ 1796.600093] RSP: 0018:ffff8800a6c03b88  EFLAGS: 00010046
[ 1796.600197] RAX: ffff8806e791a000 RBX: ffffffffe791a028 RCX: 0000000000000000
[ 1796.600197] RDX: 0000000000000001 RSI: ffff8806cdc68068 RDI: 00000000e791a028
[ 1796.600197] RBP: ffff8800a6c03b98 R08: ffff880496183078 R09: 00000000000151c6
[ 1796.600197] R10: 000000000000b731 R11: 0000000000000001 R12: ffff8801b4dd7840
[ 1796.600197] R13: 0000000000000000 R14: 000000000000001e R15: ffff8801b34ac1a0
[ 1796.600197] FS:  0000000000000000(0000) GS:ffff8800a6c00000(0000) knlGS:0000000000000000
[ 1796.600197] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1796.600197] CR2: fffffffedf97f040 CR3: 0000000021e2d000 CR4: 00000000000006a0
[ 1796.610323] Stack:
[ 1796.610323]  0000000000000000 ffff8801b34ac1a0 ffff8800a6c03bd8 ffffffff9d1a9646
[ 1796.610323]  ffff8800a6c03bd8 ffff8806cdc68068 ffff8806cdc68068 ffff8801b34ac1a0
[ 1796.610323]  0000000000000000 000000000000b7db ffff8800a6c03c38 ffffffff9d1ae987
[ 1796.610323] Call Trace:
[ 1796.610323]  <IRQ>
[ 1796.610323] account_entity_dequeue (kernel/sched/fair.c:859 kernel/sched/fair.c:2009)
[ 1796.610323] dequeue_entity (kernel/sched/fair.c:2827)
[ 1796.610323] dequeue_task_fair (kernel/sched/fair.c:3907 include/linux/jump_label.h:105 kernel/sched/fair.c:3041 kernel/sched/fair.c:3217 kernel/sched/fair.c:3915)
[ 1796.610323] dequeue_task (kernel/sched/core.c:793)
[ 1796.610323] deactivate_task (kernel/sched/core.c:809)
[ 1796.610323] move_task (kernel/sched/fair.c:5032)
[ 1796.610323] load_balance (kernel/sched/fair.c:5305 kernel/sched/fair.c:6485)
[ 1796.610323] ? debug_smp_processor_id (lib/smp_processor_id.c:57)
[ 1796.610323] rebalance_domains (kernel/sched/fair.c:7032)
[ 1796.610323] ? rebalance_domains (kernel/sched/fair.c:6975)
[ 1796.610323] run_rebalance_domains (kernel/sched/fair.c:7105 kernel/sched/fair.c:7198)
[ 1796.610323] __do_softirq (kernel/softirq.c:269 include/linux/jump_label.h:105 include/trace/events/irq.h:126 kernel/softirq.c:270)
[ 1796.610323] ? irq_exit (include/linux/vtime.h:82 include/linux/vtime.h:121 kernel/softirq.c:384)
[ 1796.610323] irq_exit (kernel/softirq.c:346 kernel/softirq.c:387)
[ 1796.610323] scheduler_ipi (kernel/sched/core.c:1545)
[ 1796.610323] smp_reschedule_interrupt (arch/x86/kernel/smp.c:266)
[ 1796.610323] reschedule_interrupt (arch/x86/kernel/entry_64.S:1178)
[ 1796.610323]  <EOI>
[ 1796.610323] ? native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 1796.610323] ? trace_hardirqs_on (kernel/locking/lockdep.c:2607)
[ 1796.637135] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 1796.637135] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 1796.637135] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 1796.637135] cpu_startup_entry (??:?)
[ 1796.637135] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 1796.637135] Code: 3a ea 05 00 74 25 89 de 48 c7 c7 08 b4 6c a1 31 c0 e8 99 6c 45 03 e8 7c 39 46 03 48 8b 05 71 3a ea 05 8b 04 98 eb 16 0f 1f 40 00 <48> 8b 14 dd 00 ef 0a a3 48 c7 c0 d8 f4 00 00 8b 04 10 48 83 c4
[ 1796.637135] RIP __cpu_to_node (arch/x86/mm/numa.c:777)
[ 1796.637135]  RSP <ffff8800a6c03b88>
[ 1796.637135] CR2: fffffffedf97f040

Thanks,
Sasha

#1 lore raw
Quoted 5 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel I've stumbled on the following:

Cute.. not making sense.. :-)

Quoted 2 lines
[ 1796.591361] BUG: unable to handle kernel paging request at fffffffedf97f040
[ 1796.592665] IP: __cpu_to_node (arch/x86/mm/numa.c:777)

I suppose you've scripted this addr2line -ie vmlinux for all addresses in this splat?

Quoted 41 lines
[ 1796.593710] PGD 21e30067 PUD 0
[ 1796.594174] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 1796.594937] Dumping ftrace buffer:
[ 1796.595678]    (ftrace buffer empty)
[ 1796.596329] Modules linked in:
[ 1796.596733] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W     3.15.0-rc3-next-20140502-sasha-00019-g5cb1c98 #431
[ 1796.598143] task: ffff8803345b8000 ti: ffff880035fc0000 task.ti: ffff880035fc0000
[ 1796.598975] RIP: __cpu_to_node (arch/x86/mm/numa.c:777)
[ 1796.600093] RSP: 0018:ffff8800a6c03b88  EFLAGS: 00010046
[ 1796.600197] RAX: ffff8806e791a000 RBX: ffffffffe791a028 RCX: 0000000000000000
[ 1796.600197] RDX: 0000000000000001 RSI: ffff8806cdc68068 RDI: 00000000e791a028
[ 1796.600197] RBP: ffff8800a6c03b98 R08: ffff880496183078 R09: 00000000000151c6
[ 1796.600197] R10: 000000000000b731 R11: 0000000000000001 R12: ffff8801b4dd7840
[ 1796.600197] R13: 0000000000000000 R14: 000000000000001e R15: ffff8801b34ac1a0
[ 1796.600197] FS:  0000000000000000(0000) GS:ffff8800a6c00000(0000) knlGS:0000000000000000
[ 1796.600197] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1796.600197] CR2: fffffffedf97f040 CR3: 0000000021e2d000 CR4: 00000000000006a0
[ 1796.610323] Stack:
[ 1796.610323]  0000000000000000 ffff8801b34ac1a0 ffff8800a6c03bd8 ffffffff9d1a9646
[ 1796.610323]  ffff8800a6c03bd8 ffff8806cdc68068 ffff8806cdc68068 ffff8801b34ac1a0
[ 1796.610323]  0000000000000000 000000000000b7db ffff8800a6c03c38 ffffffff9d1ae987
[ 1796.610323] Call Trace:
[ 1796.610323]  <IRQ>
[ 1796.610323] account_entity_dequeue (kernel/sched/fair.c:859 kernel/sched/fair.c:2009)
[ 1796.610323] dequeue_entity (kernel/sched/fair.c:2827)
[ 1796.610323] dequeue_task_fair (kernel/sched/fair.c:3907 include/linux/jump_label.h:105 kernel/sched/fair.c:3041 kernel/sched/fair.c:3217 kernel/sched/fair.c:3915)
[ 1796.610323] dequeue_task (kernel/sched/core.c:793)
[ 1796.610323] deactivate_task (kernel/sched/core.c:809)
[ 1796.610323] move_task (kernel/sched/fair.c:5032)
[ 1796.610323] load_balance (kernel/sched/fair.c:5305 kernel/sched/fair.c:6485)
[ 1796.610323] ? debug_smp_processor_id (lib/smp_processor_id.c:57)
[ 1796.610323] rebalance_domains (kernel/sched/fair.c:7032)
[ 1796.610323] ? rebalance_domains (kernel/sched/fair.c:6975)
[ 1796.610323] run_rebalance_domains (kernel/sched/fair.c:7105 kernel/sched/fair.c:7198)
[ 1796.610323] __do_softirq (kernel/softirq.c:269 include/linux/jump_label.h:105 include/trace/events/irq.h:126 kernel/softirq.c:270)
[ 1796.610323] ? irq_exit (include/linux/vtime.h:82 include/linux/vtime.h:121 kernel/softirq.c:384)
[ 1796.610323] irq_exit (kernel/softirq.c:346 kernel/softirq.c:387)
[ 1796.610323] scheduler_ipi (kernel/sched/core.c:1545)
[ 1796.610323] smp_reschedule_interrupt (arch/x86/kernel/smp.c:266)
[ 1796.610323] reschedule_interrupt (arch/x86/kernel/entry_64.S:1178)
[ 1796.610323]  <EOI>
Quoted 8 lines
[ 1796.610323] ? native_safe_halt (arch/x86/include/asm/irqflags.h:50)
[ 1796.610323] ? trace_hardirqs_on (kernel/locking/lockdep.c:2607)
[ 1796.637135] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310)
[ 1796.637135] arch_cpu_idle (arch/x86/kernel/process.c:302)
[ 1796.637135] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226)
[ 1796.637135] cpu_startup_entry (??:?)
[ 1796.637135] start_secondary (arch/x86/kernel/smpboot.c:267)
[ 1796.637135] Code: 3a ea 05 00 74 25 89 de 48 c7 c7 08 b4 6c a1 31 c0 e8 99 6c 45 03 e8 7c 39 46 03 48 8b 05 71 3a ea 05 8b 04 98 eb 16 0f 1f 40 00 <48> 8b 14 dd 00 ef 0a a3 48 c7 c0 d8 f4 00 00 8b 04 10 48 83 c4

Could you maybe also do the same with the Code? -- that is, script an auto-decode for it?

Obviously scripts/decodecode doesn't actually work right anymore:

# echo [ 1796.637135] Code: 3a ea 05 00 74 25 89 de 48 c7 c7 08 b4 6c a1 31 c0 e8 99 6c 45 03 e8 7c 39 46 03 48 8b 05 71 3a ea 05 8b 04 98 eb 16 0f 1f 40 00 <48> 8b 14 dd 00 ef 0a a3 48 c7 c0 d8 00 00 8b 04 10 48 83 c4 | ./scripts/decodecode 
-bash: syntax error near unexpected token `48'

But if I remove the <> by hand I get:

# echo [ 1796.637135] Code: 3a ea 05 00 74 25 89 de 48 c7 c7 08 b4 6c a1 31 c0 e8 99 6c 45 03 e8 7c 39 46 03 48 8b 05 71 3a ea 05 8b 04 98 eb 16 0f 1f 40 00 48 8b 14 dd 00 ef 0a a3 48 c7 c0 d8 00 00 8b 04 10 48 83 c4 | ./scripts/decodecode 
[ 1796.637135] Code: 3a ea 05 00 74 25 89 de 48 c7 c7 08 b4 6c a1 31 c0 e8 99 6c 45 03 e8 7c 39 46 03 48 8b 05 71 3a ea 05 8b 04 98 eb 16 0f 1f 40 00 48 8b 14 dd 00 ef 0a a3 48 c7 c0 d8 00 00 8b 04 10 48 83 c4
sed: -e expression #1, char 1: unknown command: `-'

Code starting with the faulting instruction
===========================================

   0:   3a ea                   cmp    %dl,%ch
   2:   05 00 74 25 89          add    $0x89257400,%eax
   7:   de 48 c7                fimul  -0x39(%rax)
   a:   c7                      (bad)  
   b:   08 b4 6c a1 31 c0 e8    or     %dh,-0x173fce5f(%rsp,%rbp,2)
  12:   99                      cltd   
  13:   6c                      insb   (%dx),%es:(%rdi)
  14:   45 03 e8                add    %r8d,%r13d
  17:   7c 39                   jl     0x52
  19:   46 03 48 8b             rex.RX add -0x75(%rax),%r9d
  1d:   05 71 3a ea 05          add    $0x5ea3a71,%eax
  22:   8b 04 98                mov    (%rax,%rbx,4),%eax
  25:   eb 16                   jmp    0x3d
  27:   0f 1f 40 00             nopl   0x0(%rax)
  2b:   48 8b 14 dd 00 ef 0a    mov    -0x5cf51100(,%rbx,8),%rdx
  32:   a3 
  33:   48 c7 c0 d8 00 00 8b    mov    $0xffffffff8b0000d8,%rax
  3a:   04 10                   add    $0x10,%al
  3c:   48                      rex.W
  3d:   83                      .byte 0x83
  3e:   c4                      .byte 0xc4

And 2b is the offset where the <> was.

Anyway, the reason I did this was that I was hoping to find the cpu argument in one of the registers, but looking at your RBX value doesn't really help.

If I compile this function with a defconfig based .config, I get something like:

00000000000000a0 <__cpu_to_node>:

  a0:   48 83 3d 00 00 00 00    cmpq   $0x0,0x0(%rip)        # a8 <__cpu_to_node+0x8>
  a7:   00 
  a8:   55                      push   %rbp
  a9:   48 89 e5                mov    %rsp,%rbp
  ac:   53                      push   %rbx
  ad:   48 63 df                movslq %edi,%rbx
  b0:   75 15                   jne    c7 <__cpu_to_node+0x27>
  b2:   48 8b 14 dd 00 00 00    mov    0x0(,%rbx,8),%rdx
  b9:   00 
  ba:   48 c7 c0 00 00 00 00    mov    $0x0,%rax
  c1:   8b 04 10                mov    (%rax,%rdx,1),%eax
  c4:   5b                      pop    %rbx
  c5:   5d                      pop    %rbp
  c6:   c3                      retq   
  c7:   89 de                   mov    %ebx,%esi
  c9:   48 c7 c7 00 00 00 00    mov    $0x0,%rdi
  d0:   31 c0                   xor    %eax,%eax
  d2:   e8 00 00 00 00          callq  d7 <__cpu_to_node+0x37>
  d7:   e8 00 00 00 00          callq  dc <__cpu_to_node+0x3c>
  dc:   48 8b 05 00 00 00 00    mov    0x0(%rip),%rax        # e3 <__cpu_to_node+0x43>
  e3:   8b 04 98                mov    (%rax,%rbx,4),%eax
  e6:   eb dc                   jmp    c4 <__cpu_to_node+0x24>
  e8:   0f 1f 84 00 00 00 00    nopl   0x0(%rax,%rax,1)
  ef:   00

And the b2 offset matches up fairly nicely, although the rest of the decode appears to be crap. Still no hints though.

However, calling convention puts the first argument in EAX, and at b2 EAX should still contain the original value, however your RAX value is complete nonsense again :/

Of course, the cpu argument being complete crap is a good reason for this to happen. Which would make thread_info::cpu of the task in question be complete crap.. and I'm not sure I can explain that either.

la-la-la..

#2 lore raw
Quoted 11 lines · Peter Zijlstra
Quoted 4 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel I've stumbled on the following:

Cute.. not making sense.. :-)

Quoted 1 line
[ 1796.591361] BUG: unable to handle kernel paging request at fffffffedf97f040 [ 1796.592665] IP: __cpu_to_node (arch/x86/mm/numa.c:777)

I suppose you've scripted this addr2line -ie vmlinux for all addresses in this splat?

Yeah, I'm trying to get that script upstream (https://lkml.org/lkml/2014/3/29/1)
since it seems to simplify looking at stack traces.

Quoted 1 line
Quoted 1 line
[ 1796.593710] PGD 21e30067 PUD 0 [ 1796.594174] Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC [ 1796.594937] Dumping ftrace buffer: [ 1796.595678]    (ftrace buffer empty) [ 1796.596329] Modules linked in: [ 1796.596733] CPU: 2 PID: 0 Comm: swapper/2 Tainted: G        W     3.15.0-rc3-next-20140502-sasha-00019-g5cb1c98 #431 [ 1796.598143] task: ffff8803345b8000 ti: ffff880035fc0000 task.ti: ffff880035fc0000 [ 1796.598975] RIP: __cpu_to_node (arch/x86/mm/numa.c:777) [ 1796.600093] RSP: 0018:ffff8800a6c03b88  EFLAGS: 00010046 [ 1796.600197] RAX: ffff8806e791a000 RBX: ffffffffe791a028 RCX: 0000000000000000 [ 1796.600197] RDX: 0000000000000001 RSI: ffff8806cdc68068 RDI: 00000000e791a028 [ 1796.600197] RBP: ffff8800a6c03b98 R08: ffff880496183078 R09: 00000000000151c6 [ 1796.600197] R10: 000000000000b731 R11: 0000000000000001 R12: ffff8801b4dd7840 [ 1796.600197] R13: 0000000000000000 R14: 000000000000001e R15: ffff8801b34ac1a0 [ 1796.600197] FS:  0000000000000000(0000) GS:ffff88!

00a6c00000
(0000) knlGS:0000000000000000 [ 1796.600197] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b [ 1796.600197] CR2: fffffffedf97f040 CR3: 0000000021e2d000 CR4: 00000000000006a0 [ 1796.610323] Stack: [ 1796.610323] 0000000000000000 ffff8801b34ac1a0 ffff8800a6c03bd8 ffffffff9d1a9646 [ 1796.610323] ffff8800a6c03bd8 ffff8806cdc68068 ffff8806cdc68068 ffff8801b34ac1a0 [ 1796.610323] 0000000000000000 000000000000b7db ffff8800a6c03c38 ffffffff9d1ae987 [ 1796.610323] Call Trace: [ 1796.610323] <IRQ> [ 1796.610323] account_entity_dequeue (kernel/sched/fair.c:859 kernel/sched/fair.c:2009) [ 1796.610323] dequeue_entity (kernel/sched/fair.c:2827) [ 1796.610323] dequeue_task_fair (kernel/sched/fair.c:3907 include/linux/jump_label.h:105 kernel/sched/fair.c:3041 kernel/sched/fair.c:3217 kernel/sched/fair.c:3915) [ 1796.610323] dequeue_task (kernel/sched/core.c:793) [ 1796.610323] deactivate_task (kernel/sched/core.c:809) [ 1796.610323] move_task (kernel/sched/fair.c:5032) [ 1796.610323] !
load_balan

ce (kernel/sched/fair.c:5305 kernel/sched/fair.c:6485) [ 1796.610323] ? debug_smp_processor_id (lib/smp_processor_id.c:57) [ 1796.610323] rebalance_domains (kernel/sched/fair.c:7032) [ 1796.610323] ? rebalance_domains (kernel/sched/fair.c:6975) [ 1796.610323] run_rebalance_domains (kernel/sched/fair.c:7105 kernel/sched/fair.c:7198) [ 1796.610323] __do_softirq (kernel/softirq.c:269 include/linux/jump_label.h:105 include/trace/events/irq.h:126 kernel/softirq.c:270) [ 1796.610323] ? irq_exit (include/linux/vtime.h:82 include/linux/vtime.h:121 kernel/softirq.c:384) [ 1796.610323] irq_exit (kernel/softirq.c:346 kernel/softirq.c:387) [ 1796.610323] scheduler_ipi (kernel/sched/core.c:1545) [ 1796.610323] smp_reschedule_interrupt (arch/x86/kernel/smp.c:266) [ 1796.610323] reschedule_interrupt (arch/x86/kernel/entry_64.S:1178) [ 1796.610323]  <EOI> [ 1796.610323] ? native_safe_halt (arch/x86/include/asm/irqflags.h:50) [ 1796.610323] ? trace_hardirqs_on (kernel/locking/lockdep.c:2607)!
  [ 1796.63

7135] default_idle (arch/x86/include/asm/paravirt.h:111 arch/x86/kernel/process.c:310) [ 1796.637135] arch_cpu_idle (arch/x86/kernel/process.c:302) [ 1796.637135] cpu_idle_loop (kernel/sched/idle.c:179 kernel/sched/idle.c:226) [ 1796.637135] cpu_startup_entry (??:?) [ 1796.637135] start_secondary (arch/x86/kernel/smpboot.c:267) [ 1796.637135] Code: 3a ea 05 00 74 25 89 de 48 c7 c7 08 b4 6c a1 31 c0 e8 99 6c 45 03 e8 7c 39 46 03 48 8b 05 71 3a ea 05 8b 04 98 eb 16 0f 1f 40 00 <48> 8b 14 dd 00 ef 0a a3 48 c7 c0 d8 f4 00 00 8b 04 10 48 83 c4

Quoted 6 lines

Could you maybe also do the same with the Code? -- that is, script an auto-decode for it?

Obviously scripts/decodecode doesn't actually work right anymore:

Quoted 7 lines
# echo [ 1796.637135] Code: 3a ea 05 00 74 25 89 de 48 c7 c7 08 b4 6c a1 31 c0 e8 99 6c 45 03 e8 7c 39 46 03 48 8b 05 71 3a ea 05 8b 04 98 eb 16 0f 1f 40 00 <48> 8b 14 dd 00 ef 0a a3 48 c7 c0 d8 00 00 8b 04 10 48 83 c4 | ./scripts/decodecode -bash: syntax error near unexpected token `48'

But if I remove the <> by hand I get:

# echo [ 1796.637135] Code: 3a ea 05 00 74 25 89 de 48 c7 c7 08 b4 6c a1 31 c0 e8 99 6c 45 03 e8 7c 39 46 03 48 8b 05 71 3a ea 05 8b 04 98 eb 16 0f 1f 40 00 48 8b 14 dd 00 ef 0a a3 48 c7 c0 d8 00 00 8b 04 10 48 83 c4 | ./scripts/decodecode [ 1796.637135] Code: 3a ea 05 00 74 25 89 de 48 c7 c7 08 b4 6c a1 31 c0 e8 99 6c 45 03 e8 7c 39 46 03 48 8b 05 71 3a ea 05 8b 04 98 eb 16 0f 1f 40 00 48 8b 14 dd 00 ef 0a a3 48 c7 c0 d8 00 00 8b 04 10 48 83 c4 sed: -e expression #1, char 1: unknown command: `-'

Code starting with the faulting instruction =========================================== 0: 3a ea cmp %dl,%ch 2: 05 00 74 25 89 add $0x89257400,%eax 7: de 48 c7 fimul -0x39(%rax) a: c7 (bad) b: 08 b4 6c a1 31 c0 e8 or %dh,-0x173fce5f(%rsp,%rbp,2) 12: 99 cltd 13: 6c insb (%dx),%es:(%rdi) 14: 45 03 e8 add %r8d,%r13d 17: 7c 39 jl 0x52 19: 46 03 48 8b rex.RX add -0x75(%rax),%r9d 1d: 05 71 3a ea 05 add $0x5ea3a71,%eax 22: 8b 04 98 mov (%rax,%rbx,4),%eax 25: eb 16 jmp 0x3d 27: 0f 1f 40 00 nopl 0x0(%rax) 2b: 48 8b 14 dd 00 ef 0a mov -0x5cf51100(,%rbx,8),%rdx 32: a3 33: 48 c7 c0 d8 00 00 8b mov $0xffffffff8b0000d8,%rax 3a: 04 10 add $0x10,%al 3c: 48 rex.W 3d!

: 83

                  .byte 0x83 3e:   c4                      .byte 0xc4
Quoted 2 lines

And 2b is the offset where the <> was.

Sure, I can look into that.

Quoted 6 lines

Anyway, the reason I did this was that I was hoping to find the cpu argument in one of the registers, but looking at your RBX value doesn't really help.

If I compile this function with a defconfig based .config, I get something like:

00000000000000a0 <__cpu_to_node>: a0: 48 83 3d 00 00 00 00 cmpq $0x0,0x0(%rip) # a8 <__cpu_to_node+0x8> a7: 00 a8: 55 push %rbp a9: 48 89 e5 mov %rsp,%rbp ac: 53 push %rbx ad: 48 63 df movslq %edi,%rbx b0: 75 15 jne c7 <__cpu_to_node+0x27> b2: 48 8b 14 dd 00 00 00 mov 0x0(,%rbx,8),%rdx b9: 00 ba: 48 c7 c0 00 00 00 00 mov $0x0,%rax c1: 8b 04 10 mov (%rax,%rdx,1),%eax c4: 5b pop %rbx c5: 5d pop %rbp c6: c3 retq c7: 89 de mov %ebx,%esi c9: 48 c7 c7 00 00 00 00 mov $0x0,%rdi d0: 31 c0 xor %eax,%eax d2: e8 00 00 00 00 callq d7 <__cpu_to_node+0x37> d7: e8 00 00 00 00 callq dc <__cpu_to_node+0x3c> dc: 48 8b 05 00 00 00 00 mov 0x0(%rip),%rax # e3 <__cpu_t!

o_node+0x4
3> e3: 8b 04 98 mov (%rax,%rbx,4),%eax e6: eb dc jmp c4 <__cpu_to_node+0x24> e8: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) ef: 00

Quoted 10 lines

And the b2 offset matches up fairly nicely, although the rest of the decode appears to be crap. Still no hints though.

However, calling convention puts the first argument in EAX, and at b2 EAX should still contain the original value, however your RAX value is complete nonsense again :/

Of course, the cpu argument being complete crap is a good reason for this to happen. Which would make thread_info::cpu of the task in question be complete crap.. and I'm not sure I can explain that either.

la-la-la..

I haven't seen it happening again, so maybe an unrelated memory corruption?

Thanks,
Sasha

#3 lore raw
Quoted 4 lines · Sasha Levin
Quoted 1 line

I suppose you've scripted this addr2line -ie vmlinux for all addresses in this splat?

Yeah, I'm trying to get that script upstream (https://lkml.org/lkml/2014/3/29/1)
since it seems to simplify looking at stack traces.

Seems like a nice addition to scripts/ indeed. Although Linus doesn't seem to like it much:

http://marc.info/?l=linux-kernel&m=139862933908922

Quoted 3 lines
Quoted 1 line

Could you maybe also do the same with the Code? -- that is, script an auto-decode for it?

Sure, I can look into that.

Thanks!

Quoted 4 lines
Quoted 2 lines

la-la-la..

I haven't seen it happening again, so maybe an unrelated memory corruption?

Yeah, just the thing we need :/ In any case, let me know if you do hit it again this side of the Sun burning up.

#4 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 7 lines · Peter Zijlstra
Quoted 3 lines · Sasha Levin
Quoted 3 lines
Quoted 1 line
Quoted 1 line

I suppose you've scripted this addr2line -ie vmlinux for all addresses in this splat?

Yeah, I'm trying to get that script upstream (https://lkml.org/lkml/2014/3/29/1) since it seems to simplify looking at stack traces.

Seems like a nice addition to scripts/ indeed. Although Linus doesn't seem to like it much:

http://marc.info/?l=linux-kernel&m=139862933908922

Linus objected to the original version of the script using the hex numbers he wants to throw out instead of resolving it from 'nm vmlinux', which was fixed in later versions.

Since there were no objections from Linus after all the fixes that were asked for I'll try pinging him again next merge window, but until then I'll add some more features that were asked for like caching and code decoding and try again :)

Thanks,
Sasha
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQIcBAEBAgAGBQJTaO+fAAoJEN6mb/eXdyzcNFQP/1cSr+MVTKBLQdI0p6esCSx0 nEMdSbJpU/P98cK2kuWffYHL4IUs9cfAeumOrUbWaDWBatetDxH1e557k53VW7aQ mmaNXrtaDQBFc3lzicDXxdjgjttf+ljkLFzBUexGhw01DsgRHa1DodD9ONcZ5Dav bsy9sWtEfUyiDNln6NtGxTSlMUXehIpLDVoqKcxzOg5wm6sSs+JF9w6YvohNuEtS OvrHKZv0XiV+Stv/496lntbmyVoVNpuhpzm3RmAxGy6VspRJLWlc8srIHWVY6KZG zP/5WvrgGawQqbMGfxKUYDplRSGzupm8xDdpVSZ8RQLfWVEqswEHjO0wRt5+8yqt r62nXPLcepjJcYppN7t6QqJ18zWTDGAOhvjNb8Yl0+YkLO/9URFetAiYq+frHVYN WAjEBBQdWiBS3akllFkf8hGEnF4vbhBHdZfBS0irbGG8eyZH3kaLhS5lJzyTnWKr oHrQ2YPrtDJjTQ1WWNiKU23GouFUHsRS/dK51VPQrUVKn5p8T6O9JSVMC0HMJkji lRfgOJM3YTB51Y3CJ9N5t8jP0l8ynLllzj4hrLN0KzfkdmYGk0yLzCuSy8BlPzyL bgN0u2ZEIEWyhlB/RVv/NA2gC6yC3xmypQgAbyQyIjqFWH6+s1hhcV5D+QnW6kN0 UZVTCyFUo4XatOhKqnj6
=6rXX
-----END PGP SIGNATURE-----