Skip to contentSkip to conversation

target/ppc: cpu_init: Clean up stop state on cpu reset

PATCH Superseded Frederic Barrat 3 participants 5 messages Jun 14, 2022 – Jun 15, 2022

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 target/ppc: cpu_init: Clean up stop state on cpu reset

The 'resume_as_sreset' attribute of a cpu can be set when a thread is entering a stop state on ppc books. It causes the thread to be re-routed to vector 0x100 when woken up by an exception. So it must be cleaned on reset or a thread might be re-routed unexpectedly after a reset, when it was not in a stop state and/or when the appropriate exception handler isn't set up yet.

---

I didn't find an appropriate commit to add a "Fixes:". It originates when adding support for power management states but the code looked quite different in 2016 and it's not clear whether we were supporting reset then.

target/ppc/cpu_init.c | 3 +++

 1 file changed, 3 insertions(+)
target/ppc/cpu_init.c +3 -0
index 0f891afa04..c16cb8dbe7 100644
@@ -7186,6 +7186,9 @@ static void ppc_cpu_reset(DeviceState *dev)
         }
         pmu_update_summaries(env);
     }
+
+    /* clean any pending stop state */
+    env->resume_as_sreset = 0;
 #endif
     hreg_compute_hflags(env);
     env->reserve_addr = (target_ulong)-1ULL;
Signature · 1 line
2.35.3
Reviewed-by on v1 1/1
#1 lore raw
Quoted 8 lines · Frederic Barrat

The 'resume_as_sreset' attribute of a cpu can be set when a thread is entering a stop state on ppc books. It causes the thread to be re-routed to vector 0x100 when woken up by an exception. So it must be cleaned on reset or a thread might be re-routed unexpectedly after a reset, when it was not in a stop state and/or when the appropriate exception handler isn't set up yet.

Reviewed-by on v1 1/1
#2 lore raw
In reply to Frederic Barrat
Quoted 6 lines · Frederic Barrat

The 'resume_as_sreset' attribute of a cpu can be set when a thread is entering a stop state on ppc books. It causes the thread to be re-routed to vector 0x100 when woken up by an exception. So it must be cleaned on reset or a thread might be re-routed unexpectedly after a reset, when it was not in a stop state and/or when the appropriate exception handler isn't set up yet.

What is the test scenario ? and what are the symptoms ?

Quoted 1 line
Quoted 6 lines

---

I didn't find an appropriate commit to add a "Fixes:". It originates when adding support for power management states but the code looked quite different in 2016 and it's not clear whether we were supporting reset then.

It was added when we needed some support for the POWER8 stop states. About that time.

Thanks,

C.

Quoted 18 lines

target/ppc/cpu_init.c | 3 +++

  1 file changed, 3 insertions(+)
target/ppc/cpu_init.c +3 -0
index 0f891afa04..c16cb8dbe7 100644
@@ -7186,6 +7186,9 @@ static void ppc_cpu_reset(DeviceState *dev)
          }
          pmu_update_summaries(env);
      }
+
+    /* clean any pending stop state */
+    env->resume_as_sreset = 0;
  #endif
      hreg_compute_hflags(env);
      env->reserve_addr = (target_ulong)-1ULL;
#3 lore raw
Quoted 9 lines · Cédric Le Goater
Quoted 6 lines · Frederic Barrat

The 'resume_as_sreset' attribute of a cpu can be set when a thread is entering a stop state on ppc books. It causes the thread to be re-routed to vector 0x100 when woken up by an exception. So it must be cleaned on reset or a thread might be re-routed unexpectedly after a reset, when it was not in a stop state and/or when the appropriate exception handler isn't set up yet.

What is the test scenario ? and what are the symptoms ?

I was hitting it because of another bug in skiboot: if you have many chips, we spend way too much time in add_opal_interrupts(), especially on powernv10 (I'm working on a separate patch in skiboot to fix that). Sufficiently so that the watchdog timer resets the system. When it happens, all the secondary threads are in stopped state, only the main thread is working. That's how I was reproducing.

What happens after the reset can vary a bit due to timing, but the most likely scenario is that we go through another primary thread election in skiboot. If the primary thread is the same as before, then there's no problem. If it's a different primary, then it will enter main_cpu_entry() while the other threads wait as secondaries. At some point, the primary thread (which still carries the wrong resume_as_sreset value from before reset) will enable the decrementer interrupt. The vector for the decrementer exception 0x900 is defined, so that shouldn't be a problem. However, because of the wrong resume_as_sreset value, it is re-routed to vector 0x100, which is still defined as the default boot-time handler, which is the entry point for BML. So the thread restarts as new, but this time it will be elected secondary. And we end up with all threads waiting as secondaries and a system stuck. All that happen before we've init the uart, so there's not a single trace on the console. Fun :-)

Fred

Quoted 41 lines
Quoted 3 lines
Quoted 6 lines

---

I didn't find an appropriate commit to add a "Fixes:". It originates when adding support for power management states but the code looked quite different in 2016 and it's not clear whether we were supporting reset then.

It was added when we needed some support for the POWER8 stop states. About that time.

Thanks,

C.

Quoted 18 lines

target/ppc/cpu_init.c | 3 +++

  1 file changed, 3 insertions(+)
target/ppc/cpu_init.c +3 -0
index 0f891afa04..c16cb8dbe7 100644
@@ -7186,6 +7186,9 @@ static void ppc_cpu_reset(DeviceState *dev)
           }
           pmu_update_summaries(env);
       }
+
+    /* clean any pending stop state */
+    env->resume_as_sreset = 0;
   #endif
       hreg_compute_hflags(env);
       env->reserve_addr = (target_ulong)-1ULL;
#4 lore raw
Quoted 11 lines · Frederic Barrat
Quoted 8 lines · Cédric Le Goater
Quoted 6 lines · Frederic Barrat

The 'resume_as_sreset' attribute of a cpu can be set when a thread is entering a stop state on ppc books. It causes the thread to be re-routed to vector 0x100 when woken up by an exception. So it must be cleaned on reset or a thread might be re-routed unexpectedly after a reset, when it was not in a stop state and/or when the appropriate exception handler isn't set up yet.

Quoted 6 lines
Quoted 1 line

What is the test scenario ? and what are the symptoms ?

I was hitting it because of another bug in skiboot: if you have many chips, we spend way too much time in add_opal_interrupts(), especially on powernv10 (I'm working on a separate patch in skiboot to fix that). Sufficiently so that the watchdog timer resets the system. When it happens, all the secondary threads are in stopped state, only the main thread is working. That's how I was reproducing.

What happens after the reset can vary a bit due to timing, but the most likely scenario is that we go through another primary thread election in skiboot. If the primary thread is the same as before, then there's no problem. If it's a different primary, then it will enter main_cpu_entry() while the other threads wait as secondaries. At some point, the primary thread (which still carries the wrong resume_as_sreset value from before reset) will enable the decrementer interrupt. The vector for the decrementer exception 0x900 is defined, so that shouldn't be a problem. However, because of the wrong resume_as_sreset value, it is re-routed to vector 0x100, which is still defined as the default boot-time handler, which is the entry point for BML. So the thread restarts as new, but this time it will be elected secondary. And we end up with all threads waiting as secondaries and a system stuck. All that happen before we've init the uart, so there's not a single trace on the console. Fun :-)

Great analysis !

I think this deserve a v2 just to put in the commit log what you just wrote :)

Thanks,

C.