cpuidle: Kernel panics with AMD Opteron 6300 entering C2 - clock related

DISCUSSION Sebastian Parschauer 2 participants 7 messages Jun 17, 2015 – Jun 18, 2015

Share & tools

Sebastian Parschauer

author

2015-06-17 10:06 UTC #0 lore raw

Hi cpuidle maintainers,

we notice kernel panics with CPUs from the AMD Opteron 6300 series and kernel 3.12 when entering C2. In that C-state the clock is shut down but the flag CPUIDLE_FLAG_TIMER_STOP isn't set. We use the TSC clock source for performance as our servers host KVM VMs. During the panics interrupts are enabled again and the timer interrupt corrupts the instruction pointer and/or the stack pointer.

Would it help to set the flag CPUIDLE_FLAG_TIMER_STOP for C2? Or how to fix this?

Thanks,
Sebastian

==========
Additional debug info:

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<          (null)>]           (null)
...
Call trace:
[<ffffffff815af9b5>] cpuidle_idle_call+0xc5/0x150
[<ffffffff8100b529>] arch_cpu_idle+0x9/0x20
[<ffffffff81092e6f>] cpu_startup_entry+0xaf/0x240
[<ffffffff8102df4b>] start_secondary+0x1db/0x240

The CPUs provide three C-states:
0: POLL
1: C1
2: C2

C2 information from the crash dump:

Quoted 11 lines

{
      name = "C2\000\000\000\000\000\000\000\000\000\000\000\000\000", 
      desc = "ACPI IOPORT 0x815\000\000\000\000\000\000\000\000\000\000\000\000\000\000", 
      flags = 1, 
      exit_latency = 100, 
      power_usage = 0, 
      target_residency = 200, 
      disabled = false, 
      enter = 0xffffffffa00ab026 <acpi_idle_enter_simple>, 
      enter_dead = 0xffffffffa00aa39c <acpi_idle_play_dead>
}

Assembly level analysis:

Quoted 1 line

RDX: 0000000225c17d03

So EDX is 00000002 and that's the entered state C2.

Quoted 11 lines

RDI: ffffffff81c15540
..
crash> info symbol 0xffffffff81c15540
clocksource_tsc in section .data

crash> disassemble cpuidle_enter_state
...

   0xffffffff815af5fc <+60>:    callq  0xffffffff8109b360 <ktime_get>
   0xffffffff815af601 <+65>:    sti    
   0xffffffff815af602 <+66>:    sub    %r13,%rax <- here rdi still points to clocksource_tsc
   0xffffffff815af605 <+69>:    mov    %rax,%rdi <- rdi is overwritten by the ktime_get return address

Daniel Lezcano

2015-06-18 09:22 UTC #1 lore raw

Quoted 11 lines · Sebastian Parschauer

Hi cpuidle maintainers,
we notice kernel panics with CPUs from the AMD Opteron 6300 series and kernel 3.12 when entering C2. In that C-state the clock is shut down but the flag CPUIDLE_FLAG_TIMER_STOP isn't set. We use the TSC clock source for performance as our servers host KVM VMs. During the panics interrupts are enabled again and the timer interrupt corrupts the instruction pointer and/or the stack pointer.
Would it help to set the flag CPUIDLE_FLAG_TIMER_STOP for C2? Or how to fix this?

Did you try the flag ? Does it fix it ?

Quoted 48 lines

==========
Additional debug info:

BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<          (null)>]           (null)
...
Call trace:
[<ffffffff815af9b5>] cpuidle_idle_call+0xc5/0x150
[<ffffffff8100b529>] arch_cpu_idle+0x9/0x20
[<ffffffff81092e6f>] cpu_startup_entry+0xaf/0x240
[<ffffffff8102df4b>] start_secondary+0x1db/0x240

The CPUs provide three C-states:
0: POLL
1: C1
2: C2

C2 information from the crash dump:

Quoted 11 lines

{
       name = "C2\000\000\000\000\000\000\000\000\000\000\000\000\000",
       desc = "ACPI IOPORT 0x815\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
       flags = 1,
       exit_latency = 100,
       power_usage = 0,
       target_residency = 200,
       disabled = false,
       enter = 0xffffffffa00ab026 <acpi_idle_enter_simple>,
       enter_dead = 0xffffffffa00aa39c <acpi_idle_play_dead>
}

Assembly level analysis:

Quoted 1 line

RDX: 0000000225c17d03

So EDX is 00000002 and that's the entered state C2.

Quoted 11 lines

RDI: ffffffff81c15540
..
crash> info symbol 0xffffffff81c15540
clocksource_tsc in section .data

crash> disassemble cpuidle_enter_state
...

    0xffffffff815af5fc <+60>:    callq  0xffffffff8109b360 <ktime_get>
    0xffffffff815af601 <+65>:    sti
    0xffffffff815af602 <+66>:    sub    %r13,%rax <- here rdi still points to clocksource_tsc
    0xffffffff815af605 <+69>:    mov    %rax,%rdi <- rdi is overwritten by the ktime_get return address

Signature · 5 lines

 <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Sebastian Parschauer

author

2015-06-18 10:52 UTC #2 lore raw

Quoted 10 lines · Daniel Lezcano

Quoted 9 lines · Sebastian Parschauer
Hi cpuidle maintainers,
we notice kernel panics with CPUs from the AMD Opteron 6300 series and kernel 3.12 when entering C2. In that C-state the clock is shut down but the flag CPUIDLE_FLAG_TIMER_STOP isn't set. We use the TSC clock source for performance as our servers host KVM VMs. During the panics interrupts are enabled again and the timer interrupt corrupts the instruction pointer and/or the stack pointer.

Quoted 4 lines

Quoted 2 lines
Would it help to set the flag CPUIDLE_FLAG_TIMER_STOP for C2? Or how to fix this?
Did you try the flag ? Does it fix it ?

Thanks for your reply. Unfortunately we can't roll out new kernels fast (VMs have to be migrated). But we've disabled the C2 state via sysfs for all CPU cores and all servers and had one more kernel panic with the same call trace although C2 was (or should have been) disabled. We use the menu governor and a v3.12.40 kernel.

It's strange to me coming into the same code path with state index 2 as parameter again. I think I'll prepare a kernel with some debug messages when transitioning from one state to another and deploy it to a test system.

Is there any better method to debug the cpuidle driver?

How do you guys test it?

Can we provide any missing additional information?

Maybe something else corrupts the memory in an interrupt and the cpuidle driver is just the one noticing an unrelated problem.

Quoted 53 lines

Quoted 51 lines
==========
Additional debug info:
BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<          (null)>]           (null)
...
Call trace:
[<ffffffff815af9b5>] cpuidle_idle_call+0xc5/0x150
[<ffffffff8100b529>] arch_cpu_idle+0x9/0x20
[<ffffffff81092e6f>] cpu_startup_entry+0xaf/0x240
[<ffffffff8102df4b>] start_secondary+0x1db/0x240
The CPUs provide three C-states:
0: POLL
1: C1
2: C2
C2 information from the crash dump:
Quoted 4 lines
{
name = "C2\000\000\000\000\000\000\000\000\000\000\000\000\000", desc = "ACPI IOPORT
0x815\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
Quoted 8 lines
       flags = 1,
       exit_latency = 100,
       power_usage = 0,
       target_residency = 200,
       disabled = false,
       enter = 0xffffffffa00ab026 <acpi_idle_enter_simple>,
       enter_dead = 0xffffffffa00aa39c <acpi_idle_play_dead>
}
Assembly level analysis:
Quoted 1 line
RDX: 0000000225c17d03
So EDX is 00000002 and that's the entered state C2.
Quoted 13 lines
RDI: ffffffff81c15540
..
crash> info symbol 0xffffffff81c15540
clocksource_tsc in section .data
crash> disassemble cpuidle_enter_state
...
    0xffffffff815af5fc <+60>:    callq  0xffffffff8109b360 <ktime_get>
    0xffffffff815af601 <+65>:    sti
    0xffffffff815af602 <+66>:    sub    %r13,%rax <- here rdi still
points to clocksource_tsc
    0xffffffff815af605 <+69>:    mov    %rax,%rdi <- rdi is
overwritten by the ktime_get return address

Sebastian Parschauer

author

2015-06-18 11:21 UTC #3 lore raw

Quoted 26 lines · Sebastian Parschauer

Quoted 10 lines · Daniel Lezcano
Quoted 9 lines · Sebastian Parschauer
Hi cpuidle maintainers,
we notice kernel panics with CPUs from the AMD Opteron 6300 series and kernel 3.12 when entering C2. In that C-state the clock is shut down but the flag CPUIDLE_FLAG_TIMER_STOP isn't set. We use the TSC clock source for performance as our servers host KVM VMs. During the panics interrupts are enabled again and the timer interrupt corrupts the instruction pointer and/or the stack pointer.
Quoted 4 lines
Quoted 2 lines
Would it help to set the flag CPUIDLE_FLAG_TIMER_STOP for C2? Or how to fix this?
Did you try the flag ? Does it fix it ?
Thanks for your reply. Unfortunately we can't roll out new kernels fast (VMs have to be migrated). But we've disabled the C2 state via sysfs for all CPU cores and all servers and had one more kernel panic with the same call trace although C2 was (or should have been) disabled. We use the menu governor and a v3.12.40 kernel.
It's strange to me coming into the same code path with state index 2 as parameter again. I think I'll prepare a kernel with some debug messages when transitioning from one state to another and deploy it to a test system.

Quoted 8 lines

Is there any better method to debug the cpuidle driver?
How do you guys test it?
Can we provide any missing additional information?
Maybe something else corrupts the memory in an interrupt and the cpuidle driver is just the one noticing an unrelated problem.

Sorry, I had a closer look at the most recent crash again. It happened at entering C1 with disabled C2. So maybe our problem is not cpuidle related.

Quoted 55 lines

Quoted 54 lines
Quoted 52 lines
==========
Additional debug info:
BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<          (null)>]           (null)
...
Call trace:
[<ffffffff815af9b5>] cpuidle_idle_call+0xc5/0x150
[<ffffffff8100b529>] arch_cpu_idle+0x9/0x20
[<ffffffff81092e6f>] cpu_startup_entry+0xaf/0x240
[<ffffffff8102df4b>] start_secondary+0x1db/0x240
The CPUs provide three C-states:
0: POLL
1: C1
2: C2
C2 information from the crash dump:
Quoted 4 lines
{
name = "C2\000\000\000\000\000\000\000\000\000\000\000\000\000", desc = "ACPI IOPORT
0x815\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
Quoted 8 lines
       flags = 1,
       exit_latency = 100,
       power_usage = 0,
       target_residency = 200,
       disabled = false,
       enter = 0xffffffffa00ab026 <acpi_idle_enter_simple>,
       enter_dead = 0xffffffffa00aa39c <acpi_idle_play_dead>
}
Assembly level analysis:
Quoted 1 line
RDX: 0000000225c17d03
So EDX is 00000002 and that's the entered state C2.
Quoted 13 lines
RDI: ffffffff81c15540
..
crash> info symbol 0xffffffff81c15540
clocksource_tsc in section .data
crash> disassemble cpuidle_enter_state
...
    0xffffffff815af5fc <+60>:    callq  0xffffffff8109b360 <ktime_get>
    0xffffffff815af601 <+65>:    sti
    0xffffffff815af602 <+66>:    sub    %r13,%rax <- here rdi still
points to clocksource_tsc
    0xffffffff815af605 <+69>:    mov    %rax,%rdi <- rdi is
overwritten by the ktime_get return address

Daniel Lezcano

2015-06-18 13:27 UTC #4 lore raw

Quoted 31 lines · Sebastian Parschauer

Quoted 30 lines · Sebastian Parschauer
Quoted 10 lines · Daniel Lezcano
Quoted 9 lines · Sebastian Parschauer
Hi cpuidle maintainers,
we notice kernel panics with CPUs from the AMD Opteron 6300 series and kernel 3.12 when entering C2. In that C-state the clock is shut down but the flag CPUIDLE_FLAG_TIMER_STOP isn't set. We use the TSC clock source for performance as our servers host KVM VMs. During the panics interrupts are enabled again and the timer interrupt corrupts the instruction pointer and/or the stack pointer.
Quoted 4 lines
Quoted 2 lines
Would it help to set the flag CPUIDLE_FLAG_TIMER_STOP for C2? Or how to fix this?
Did you try the flag ? Does it fix it ?
Thanks for your reply. Unfortunately we can't roll out new kernels fast (VMs have to be migrated). But we've disabled the C2 state via sysfs for all CPU cores and all servers and had one more kernel panic with the same call trace although C2 was (or should have been) disabled. We use the menu governor and a v3.12.40 kernel.
It's strange to me coming into the same code path with state index 2 as parameter again. I think I'll prepare a kernel with some debug messages when transitioning from one state to another and deploy it to a test system.
Is there any better method to debug the cpuidle driver?
How do you guys test it?

Quoted 8 lines

Quoted 4 lines
Can we provide any missing additional information?
Maybe something else corrupts the memory in an interrupt and the cpuidle driver is just the one noticing an unrelated problem.
Sorry, I had a closer look at the most recent crash again. It happened at entering C1 with disabled C2. So maybe our problem is not cpuidle related.

As mentioned in the previous email, disabling the idle state index 2 in the kernel does not prevent the firmware to auto-promote to this state.

By the way, I am not sure this is really the C2 state but the idle state index 2. Could you give the C state name you have in the sysfs directory ?

Signature · 5 lines

 <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog

Sebastian Parschauer

author

2015-06-18 14:09 UTC #5 lore raw

Quoted 44 lines · Daniel Lezcano

Quoted 34 lines · Sebastian Parschauer
Quoted 33 lines · Sebastian Parschauer
Quoted 12 lines · Daniel Lezcano
Quoted 11 lines · Sebastian Parschauer
Hi cpuidle maintainers,
we notice kernel panics with CPUs from the AMD Opteron 6300 series and kernel 3.12 when entering C2. In that C-state the clock is shut down but
the flag CPUIDLE_FLAG_TIMER_STOP isn't set. We use the TSC clock source
for performance as our servers host KVM VMs. During the panics interrupts are enabled again and the timer interrupt corrupts the instruction pointer and/or the stack pointer.
Quoted 4 lines
Quoted 2 lines
Would it help to set the flag CPUIDLE_FLAG_TIMER_STOP for C2? Or how to fix this?
Did you try the flag ? Does it fix it ?
Thanks for your reply. Unfortunately we can't roll out new kernels fast (VMs have to be migrated). But we've disabled the C2 state via sysfs for all CPU cores and all servers and had one more kernel panic with the same call trace although C2 was (or should have been) disabled. We use the menu governor and a v3.12.40 kernel.
It's strange to me coming into the same code path with state index 2 as parameter again. I think I'll prepare a kernel with some debug messages when transitioning from one state to another and deploy it to a test system.
Is there any better method to debug the cpuidle driver?
How do you guys test it?
Quoted 8 lines
Quoted 4 lines
Can we provide any missing additional information?
Maybe something else corrupts the memory in an interrupt and the cpuidle driver is just the one noticing an unrelated problem.
Sorry, I had a closer look at the most recent crash again. It happened at entering C1 with disabled C2. So maybe our problem is not cpuidle related.

Quoted 5 lines

As mentioned in the previous email, disabling the idle state index 2 in the kernel does not prevent the firmware to auto-promote to this state.
By the way, I am not sure this is really the C2 state but the idle state index 2. Could you give the C state name you have in the sysfs directory ?

state0: POLL
state1: C1
state2: C2

We also see it in the crash dump that C2 is drv->states[2].

Thanks for the detailed information! That helps a lot. Yes, we have the possibility to set the max. allowed C-state in the BIOS. We'll do a research on this. We are only rarely in C2 and can't afford to disable C1. So setting C1 as the maximum C-state in the BIOS could be an option. Thanks!

Cheers,
Sebastian

Daniel Lezcano

2015-06-18 13:23 UTC #6 lore raw

In reply to Sebastian Parschauer

Quoted 22 lines · Sebastian Parschauer

Quoted 10 lines · Daniel Lezcano
Quoted 9 lines · Sebastian Parschauer
Hi cpuidle maintainers,
we notice kernel panics with CPUs from the AMD Opteron 6300 series and kernel 3.12 when entering C2. In that C-state the clock is shut down but the flag CPUIDLE_FLAG_TIMER_STOP isn't set. We use the TSC clock source for performance as our servers host KVM VMs. During the panics interrupts are enabled again and the timer interrupt corrupts the instruction pointer and/or the stack pointer.
Quoted 4 lines
Quoted 2 lines
Would it help to set the flag CPUIDLE_FLAG_TIMER_STOP for C2? Or how to fix this?
Did you try the flag ? Does it fix it ?
Thanks for your reply. Unfortunately we can't roll out new kernels fast (VMs have to be migrated). But we've disabled the C2 state via sysfs for all CPU cores and all servers and had one more kernel panic with the same call trace although C2 was (or should have been) disabled. We use the menu governor and a v3.12.40 kernel.

Quoted 3 lines

It's strange to me coming into the same code path with state index 2 as parameter again. I think I'll prepare a kernel with some debug messages when transitioning from one state to another and deploy it to a test system.

It is weird you disabled the state index 2 and the system enters with this index again.

Are you sure you disabled effectively for all cores on the system this state ?

Furthermore, if I am not wrong the C state on AMD differs a bit from the C-state intel's semantic.

The firmware will put the cluster down if all core go to the C1 state, no ?

Quoted 1 line

Is there any better method to debug the cpuidle driver?

You can try by passing to the kernel command line:

processor.max_cstate=1

Note, that does not guarantee the firmware won't promote to a deeper idle state.

If the kernel panics again, may be in the BIOS, there is an option to set max idle states for the firmware.

Quoted 1 line

How do you guys test it?

On the x86 platform, most of the magic is in the firmware, so if there is a bug there, hmm ... that will be hard to spot.

Quoted 61 lines

Can we provide any missing additional information?
Maybe something else corrupts the memory in an interrupt and the cpuidle driver is just the one noticing an unrelated problem.
Quoted 53 lines
Quoted 51 lines
==========
Additional debug info:
BUG: unable to handle kernel NULL pointer dereference at           (null)
IP: [<          (null)>]           (null)
...
Call trace:
[<ffffffff815af9b5>] cpuidle_idle_call+0xc5/0x150
[<ffffffff8100b529>] arch_cpu_idle+0x9/0x20
[<ffffffff81092e6f>] cpu_startup_entry+0xaf/0x240
[<ffffffff8102df4b>] start_secondary+0x1db/0x240
The CPUs provide three C-states:
0: POLL
1: C1
2: C2
C2 information from the crash dump:
Quoted 4 lines
{
name = "C2\000\000\000\000\000\000\000\000\000\000\000\000\000", desc = "ACPI IOPORT
0x815\000\000\000\000\000\000\000\000\000\000\000\000\000\000",
Quoted 8 lines
        flags = 1,
        exit_latency = 100,
        power_usage = 0,
        target_residency = 200,
        disabled = false,
        enter = 0xffffffffa00ab026 <acpi_idle_enter_simple>,
        enter_dead = 0xffffffffa00aa39c <acpi_idle_play_dead>
}
Assembly level analysis:
Quoted 1 line
RDX: 0000000225c17d03
So EDX is 00000002 and that's the entered state C2.
Quoted 13 lines
RDI: ffffffff81c15540
..
crash> info symbol 0xffffffff81c15540
clocksource_tsc in section .data
crash> disassemble cpuidle_enter_state
...
     0xffffffff815af5fc <+60>:    callq  0xffffffff8109b360 <ktime_get>
     0xffffffff815af601 <+65>:    sti
     0xffffffff815af602 <+66>:    sub    %r13,%rax <- here rdi still
points to clocksource_tsc
     0xffffffff815af605 <+69>:    mov    %rax,%rdi <- rdi is
overwritten by the ktime_get return address

Signature · 5 lines

 <http://www.linaro.org/> Linaro.org │ Open source software for ARM SoCs

Follow Linaro:  <http://www.facebook.com/pages/Linaro> Facebook |
<http://twitter.com/#!/linaroorg> Twitter |
<http://www.linaro.org/linaro-blog/> Blog