Skip to contentSkip to conversation

ppc: Fix migration issue with quiesced state

PATCH In review Fabiano Rosas 5 participants 8 messages Jan 9 – Jan 20 PowerPC

lore
Share & tools
lore
#0 lore raw

v3 cover ppc: Fix migration issue with quiesced state

Changes in v3:
- removed extra include
- changed the test to set the machine earlier

CI run: https://gitlab.com/farosas/qemu/-/pipelines/2252213960

v2:
https://lore.kernel.org/r/[email protected]

A while back, Fabian reported two issues with migration on ppc64le, one of which was already fixed. There remains this one from the previous series:

pseries migration causes a guest hang after migration due to a new variable used to track the stopped state of vcpus, which is not migrated. Fix is to migrate the new variable. To avoid breaking backward migration, a compat property is added. For forward migration, a heuritics based on the known QEMU code is used.

Thanks

CI run: https://gitlab.com/farosas/qemu/-/pipelines/2220496446

v1:
https://lore.kernel.org/r/[email protected]

Fabiano Rosas (2):
target/ppc: Fix env->quiesced migration tests/functional: Add a OS level migration test for pseries

 hw/ppc/spapr.c                           |  6 +++
 target/ppc/cpu.h                         |  1 +
 target/ppc/cpu_init.c                    |  7 +++
 target/ppc/machine.c                     | 62 ++++++++++++++++++++++++
 tests/functional/ppc64/test_migration.py | 12 +++++
 tests/functional/ppc64/test_pseries.py   | 35 +++++++++++++
 6 files changed, 123 insertions(+)
Signature · 1 line
2.51.0
#1 lore raw

PATCH v3 1/2 target/ppc: Fix env->quiesced migration

The commit referenced (from QEMU 10.0) has changed the way the pseries machine marks a cpu as quiesced. Previously, the cpu->halted value from QEMU common cpu code was (incorrectly) used. With the fix, the env->quiesced variable starts being used, which improves on the original situation, but also causes a side effect after migration:

The env->quiesced is set at reset and never migrated, which causes the destination QEMU to stop delivering interrupts and hang the machine.

To fix the issue from this point on, start migrating the env->quiesced value.

For QEMU versions < 10.0, sending the new element on the stream would cause migration to be aborted, so add the appropriate compatibility property to omit the new subsection.

Independently of this patch, all migrations from QEMU versions < 10.0 would result in a hang since the older QEMU never migrates env->quiesced. This is bad because it leaves machines already running on the old QEMU without a migration path into newer versions.

As a workaround, use a few heuristics to infer the new value of env->quiesced based on cpu->halted, LPCR and PSSCR bits that are usually set/cleared along with quiesced.

Note that this was tested with -cpu power9 and -machine ic-mode=xive due to another bug affecting migration of XICS guests. Tested both forward and backward migration and savevm/loadvm from 9.2 and 10.0.

Also tested loadvm of a savevm image that contains a mix of cpus both halted and not halted.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3079

Fixes fb802acdc8b ("ppc/spapr: Fix RTAS stopped state")
---
 hw/ppc/spapr.c        |  6 +++++
 target/ppc/cpu.h      |  1 +
 target/ppc/cpu_init.c |  7 +++++
 target/ppc/machine.c  | 62 +++++++++++++++++++++++++++++++++++++++++++
 4 files changed, 76 insertions(+)
hw/ppc/spapr.c +6 -0
index 3d87450be5..5e8b7ab1aa 100644
@@ -4788,8 +4788,14 @@ DEFINE_SPAPR_MACHINE(10, 1);
  */
 static void spapr_machine_10_0_class_options(MachineClass *mc)
 {
+    static GlobalProperty spapr_compat_10_0[] = {
+        { TYPE_POWERPC_CPU, "rtas-stopped-state", "false" },
+    };
+
     spapr_machine_10_1_class_options(mc);
     compat_props_add(mc->compat_props, hw_compat_10_0, hw_compat_10_0_len);
+    compat_props_add(mc->compat_props, spapr_compat_10_0,
+                     G_N_ELEMENTS(spapr_compat_10_0));
 }
 
 DEFINE_SPAPR_MACHINE(10, 0);
target/ppc/cpu.h +1 -0
index dbebae89dc..49445eb4ca 100644
@@ -1529,6 +1529,7 @@ struct ArchCPU {
     void *machine_data;
     int32_t node_id; /* NUMA node this CPU belongs to */
     PPCHash64Options *hash64_opts;
+    bool rtas_stopped_state;
 
     /* Those resources are used only during code translation */
     /* opcode handlers */
target/ppc/cpu_init.c +7 -0
index 929254827d..58816c51a7 100644
@@ -55,6 +55,11 @@
 /* #define PPC_DEBUG_SPR */
 /* #define USE_APPLE_GDB */
 
+static const Property powerpc_cpu_properties[] = {
+    DEFINE_PROP_BOOL("rtas-stopped-state", PowerPCCPU,
+                      rtas_stopped_state, true),
+};
+
 static inline void vscr_init(CPUPPCState *env, uint32_t val)
 {
     /* Altivec always uses round-to-nearest */
@@ -7529,6 +7534,8 @@ static void ppc_cpu_class_init(ObjectClass *oc, const void *data)
                                       &pcc->parent_unrealize);
     pcc->pvr_match = ppc_pvr_match_default;
 
+    device_class_set_props(dc, powerpc_cpu_properties);
+
     resettable_class_set_parent_phases(rc, NULL, ppc_cpu_reset_hold, NULL,
                                        &pcc->parent_phases);
 
target/ppc/machine.c +62 -0

80 lines not on this page · raw message

Signature · 1 line
2.51.0
Acked-by on v3 1/2
#2 lore raw

Re: [PATCH v3 1/2] target/ppc: Fix env->quiesced migration

Quoted 29 lines · Fabiano Rosas

The commit referenced (from QEMU 10.0) has changed the way the pseries machine marks a cpu as quiesced. Previously, the cpu->halted value from QEMU common cpu code was (incorrectly) used. With the fix, the env->quiesced variable starts being used, which improves on the original situation, but also causes a side effect after migration:

The env->quiesced is set at reset and never migrated, which causes the destination QEMU to stop delivering interrupts and hang the machine.

To fix the issue from this point on, start migrating the env->quiesced value.

For QEMU versions < 10.0, sending the new element on the stream would cause migration to be aborted, so add the appropriate compatibility property to omit the new subsection.

Independently of this patch, all migrations from QEMU versions < 10.0 would result in a hang since the older QEMU never migrates env->quiesced. This is bad because it leaves machines already running on the old QEMU without a migration path into newer versions.

As a workaround, use a few heuristics to infer the new value of env->quiesced based on cpu->halted, LPCR and PSSCR bits that are usually set/cleared along with quiesced.

Note that this was tested with -cpu power9 and -machine ic-mode=xive due to another bug affecting migration of XICS guests. Tested both forward and backward migration and savevm/loadvm from 9.2 and 10.0.

Quoted 7 lines

Also tested loadvm of a savevm image that contains a mix of cpus both halted and not halted.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3079

Fixes fb802acdc8b ("ppc/spapr: Fix RTAS stopped state")
Quoted 150 lines
---
  hw/ppc/spapr.c        |  6 +++++
  target/ppc/cpu.h      |  1 +
  target/ppc/cpu_init.c |  7 +++++
  target/ppc/machine.c  | 62 +++++++++++++++++++++++++++++++++++++++++++
  4 files changed, 76 insertions(+)
hw/ppc/spapr.c +6 -0
index 3d87450be5..5e8b7ab1aa 100644
@@ -4788,8 +4788,14 @@ DEFINE_SPAPR_MACHINE(10, 1);
   */
  static void spapr_machine_10_0_class_options(MachineClass *mc)
  {
+    static GlobalProperty spapr_compat_10_0[] = {
+        { TYPE_POWERPC_CPU, "rtas-stopped-state", "false" },
+    };
+
      spapr_machine_10_1_class_options(mc);
      compat_props_add(mc->compat_props, hw_compat_10_0, hw_compat_10_0_len);
+    compat_props_add(mc->compat_props, spapr_compat_10_0,
+                     G_N_ELEMENTS(spapr_compat_10_0));
  }
 
  DEFINE_SPAPR_MACHINE(10, 0);
target/ppc/cpu.h +1 -0
index dbebae89dc..49445eb4ca 100644
@@ -1529,6 +1529,7 @@ struct ArchCPU {
      void *machine_data;
      int32_t node_id; /* NUMA node this CPU belongs to */
      PPCHash64Options *hash64_opts;
+    bool rtas_stopped_state;
 
      /* Those resources are used only during code translation */
      /* opcode handlers */
target/ppc/cpu_init.c +7 -0
index 929254827d..58816c51a7 100644
@@ -55,6 +55,11 @@
  /* #define PPC_DEBUG_SPR */
  /* #define USE_APPLE_GDB */
 
+static const Property powerpc_cpu_properties[] = {
+    DEFINE_PROP_BOOL("rtas-stopped-state", PowerPCCPU,
+                      rtas_stopped_state, true),
+};
+
  static inline void vscr_init(CPUPPCState *env, uint32_t val)
  {
      /* Altivec always uses round-to-nearest */
@@ -7529,6 +7534,8 @@ static void ppc_cpu_class_init(ObjectClass *oc, const void *data)
                                        &pcc->parent_unrealize);
      pcc->pvr_match = ppc_pvr_match_default;
 
+    device_class_set_props(dc, powerpc_cpu_properties);
+
      resettable_class_set_parent_phases(rc, NULL, ppc_cpu_reset_hold, NULL,
                                         &pcc->parent_phases);
 
target/ppc/machine.c +62 -0
index d72e5ecb94..49cfdc6d67 100644
@@ -257,6 +257,45 @@ static int cpu_post_load(void *opaque, int version_id)
          ppc_store_sdr1(env, env->spr[SPR_SDR1]);
      }
 
+    if (!cpu->rtas_stopped_state) {
+        /*
+         * The source QEMU doesn't have fb802acdc8 and still uses halt +
+         * PM bits in LPCR to implement RTAS stopped state. The new (this)
+         * QEMU will have put the secondary vcpus in stopped state,
+         * waiting for the start-cpu RTAS call. That call will never come
+         * if the source cpus were already running. Try to infer the cpus
+         * state and set env->quiesced accordingly.
+         *
+         * env->quiesced = true  ==> the cpu is waiting to start
+         * env->quiesced = false ==> the cpu is running (unless halted)
+         */
+
+        /*
+         * Halted _could_ mean quiesced, but it could also be cede,
+         * confer_self, power management, etc.
+         */
+        if (CPU(cpu)->halted) {
+            PowerPCCPUClass *pcc = POWERPC_CPU_GET_CLASS(cpu);
+            /*
+             * Both the PSSCR_EC bit and LPCR PM bits set at cpu reset
+             * and rtas_stop and cleared at rtas_start, it's a good
+             * heuristic.
+             */
+            if ((env->spr[SPR_PSSCR] & PSSCR_EC) &&
+                (env->spr[SPR_LPCR] & pcc->lpcr_pm)) {
+                env->quiesced = true;
+            } else {
+                env->quiesced = false;
+            }
+        } else {
+            /*
+             * Old QEMU sets halted during rtas_stop_self. Not halted,
+             * therefore definitely not quiesced.
+             */
+            env->quiesced = false;
+        }
+    }
+
      post_load_update_msr(env);
 
      if (tcg_enabled()) {
@@ -649,6 +688,28 @@ static const VMStateDescription vmstate_reservation = {
      }
  };
 
+static bool rtas_stopped_needed(void *opaque)
+{
+    PowerPCCPU *cpu = opaque;
+
+    return cpu->rtas_stopped_state;
+}
+
+static const VMStateDescription vmstate_rtas_stopped = {
+    .name = "cpu/rtas_stopped",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = rtas_stopped_needed,
+    .fields = (const VMStateField[]) {
+        /*
+         * "RTAS stopped" state, independent of halted state. For QEMU
+         * < 10.0, this is taken from cpu->halted at cpu_post_load()
+         */
+        VMSTATE_BOOL(env.quiesced, PowerPCCPU),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
  #ifdef TARGET_PPC64
  static bool bhrb_needed(void *opaque)
  {
@@ -715,6 +776,7 @@ const VMStateDescription vmstate_ppc_cpu = {
          &vmstate_tlbmas,
          &vmstate_compat,
          &vmstate_reservation,
+        &vmstate_rtas_stopped,
          NULL
      }
  };
Reviewed-by on v3 1/2
#3 lore raw

Re: [PATCH v3 1/2] target/ppc: Fix env->quiesced migration

In reply to Fabiano Rosas
Quoted 29 lines · Fabiano Rosas

The commit referenced (from QEMU 10.0) has changed the way the pseries machine marks a cpu as quiesced. Previously, the cpu->halted value from QEMU common cpu code was (incorrectly) used. With the fix, the env->quiesced variable starts being used, which improves on the original situation, but also causes a side effect after migration:

The env->quiesced is set at reset and never migrated, which causes the destination QEMU to stop delivering interrupts and hang the machine.

To fix the issue from this point on, start migrating the env->quiesced value.

For QEMU versions < 10.0, sending the new element on the stream would cause migration to be aborted, so add the appropriate compatibility property to omit the new subsection.

Independently of this patch, all migrations from QEMU versions < 10.0 would result in a hang since the older QEMU never migrates env->quiesced. This is bad because it leaves machines already running on the old QEMU without a migration path into newer versions.

As a workaround, use a few heuristics to infer the new value of env->quiesced based on cpu->halted, LPCR and PSSCR bits that are usually set/cleared along with quiesced.

Note that this was tested with -cpu power9 and -machine ic-mode=xive due to another bug affecting migration of XICS guests. Tested both forward and backward migration and savevm/loadvm from 9.2 and 10.0.

Quoted 7 lines

Also tested loadvm of a savevm image that contains a mix of cpus both halted and not halted.

Resolves:https://gitlab.com/qemu-project/qemu/-/issues/3079

Fixes fb802acdc8b ("ppc/spapr: Fix RTAS stopped state")

and Queued.

#4 lore raw

PATCH v3 2/2 tests/functional: Add a OS level migration test for pseries

In reply to Fabiano Rosas

There's currently no OS level test for ppc64le. Add one such test by reusing the boot level tests that are already present.

The test boots the source machine, waits for it to reach a mid-boot message, migrates and checks that the destination has reached the final boot message (VFS error due to no disk).

---
 tests/functional/ppc64/test_migration.py | 12 ++++++++
 tests/functional/ppc64/test_pseries.py   | 35 ++++++++++++++++++++++++
 2 files changed, 47 insertions(+)
tests/functional/ppc64/test_migration.py +12 -0
index 5dfdaaf709..a3b819680b 100755
@@ -4,6 +4,7 @@
 #
 # ppc migration test
 
+from qemu_test.ports import Ports
 from migration import MigrationTest
 
 
@@ -21,6 +22,17 @@ def test_migration_with_exec(self):
         self.set_machine('mac99')
         self.migration_with_exec()
 
+    def do_migrate_ppc64_linux(self, source_vm, dest_vm):
+        with Ports() as ports:
+            port = ports.find_free_port()
+            if port is None:
+                self.skipTest('Failed to find a free port')
+            uri = 'tcp:localhost:%u' % port
+
+            dest_vm.qmp('migrate-incoming', uri=uri)
+            source_vm.qmp('migrate', uri=uri)
+            self.assert_migration(source_vm, dest_vm)
+
 
 if __name__ == '__main__':
     MigrationTest.main()
tests/functional/ppc64/test_pseries.py +35 -0
index 7840c4e3ff..b45763c305 100755
@@ -9,6 +9,7 @@
 
 from qemu_test import QemuSystemTest, Asset
 from qemu_test import wait_for_console_pattern
+from test_migration import PpcMigrationTest
 
 class PseriesMachine(QemuSystemTest):
 
@@ -87,5 +88,39 @@ def test_ppc64_linux_big_boot(self):
         wait_for_console_pattern(self, console_pattern, self.panic_message)
         wait_for_console_pattern(self, self.good_message, self.panic_message)
 
+    def test_ppc64_linux_migration(self):
+        self.set_machine('pseries')
+
+        kernel_path = self.ASSET_KERNEL.fetch()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE
+
+        dest_vm = self.get_vm(name="dest-qemu")
+        dest_vm.add_args('-incoming', 'defer')
+        dest_vm.add_args('-smp', '4')
+        dest_vm.add_args('-nodefaults')
+        dest_vm.add_args('-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        dest_vm.set_console()
+        dest_vm.launch()
+
+        source_vm = self.get_vm(name="source-qemu")
+        source_vm.add_args('-smp', '4')
+        source_vm.add_args('-nodefaults')
+        source_vm.add_args('-kernel', kernel_path,
+                           '-append', kernel_command_line)
+        source_vm.set_console()
+        source_vm.launch()
+
+        # ensure the boot has reached Linux
+        console_pattern = 'smp: Brought up 1 node, 4 CPUs'
+        wait_for_console_pattern(self, console_pattern, self.panic_message,
+                                 vm=source_vm)
+
+        PpcMigrationTest().do_migrate_ppc64_linux(source_vm, dest_vm);
+
+        # ensure the boot proceeds after migration
+        wait_for_console_pattern(self, self.good_message, self.panic_message,
+                                 vm=dest_vm)
+
 if __name__ == '__main__':
     QemuSystemTest.main()
Signature · 1 line
2.51.0
Reviewed-by on v3 2/2
#5 lore raw

Re: [PATCH v3 2/2] tests/functional: Add a OS level migration test for pseries

Quoted 27 lines · Fabiano Rosas

There's currently no OS level test for ppc64le. Add one such test by reusing the boot level tests that are already present.

The test boots the source machine, waits for it to reach a mid-boot message, migrates and checks that the destination has reached the final boot message (VFS error due to no disk).

---
 tests/functional/ppc64/test_migration.py | 12 ++++++++
 tests/functional/ppc64/test_pseries.py   | 35 ++++++++++++++++++++++++
 2 files changed, 47 insertions(+)
tests/functional/ppc64/test_migration.py +1 -0
index 5dfdaaf709..a3b819680b 100755
@@ -4,6 +4,7 @@
 #
 # ppc migration test
 
+from qemu_test.ports import Ports
 from migration import MigrationTest
 
 

tests/functional/ppc64/test_migration.py def test_migration_with_exec(self): · line 30

Quoted 9 lines
tests/functional/ppc64/test_migration.py +6 -0
@@ -21,6 +22,17 @@ def test_migration_with_exec(self):
         self.set_machine('mac99')
         self.migration_with_exec()
 
+    def do_migrate_ppc64_linux(self, source_vm, dest_vm):
+        with Ports() as ports:
+            port = ports.find_free_port()
+            if port is None:
+                self.skipTest('Failed to find a free port')
+            uri = 'tcp:localhost:%u' % port

* When port is None, shouldn't it return after the skipTest() call? With port = None, uri will become -> 'tcp:localhost:None' OR maybe port should have default value?

Quoted 50 lines
tests/functional/ppc64/test_migration.py +4 -0
+            dest_vm.qmp('migrate-incoming', uri=uri)
+            source_vm.qmp('migrate', uri=uri)
+            self.assert_migration(source_vm, dest_vm)
+
 
 if __name__ == '__main__':
     MigrationTest.main()
tests/functional/ppc64/test_pseries.py +28 -0
index 7840c4e3ff..b45763c305 100755
@@ -9,6 +9,7 @@
 
 from qemu_test import QemuSystemTest, Asset
 from qemu_test import wait_for_console_pattern
+from test_migration import PpcMigrationTest
 
 class PseriesMachine(QemuSystemTest):
 
@@ -87,5 +88,39 @@ def test_ppc64_linux_big_boot(self):
         wait_for_console_pattern(self, console_pattern, self.panic_message)
         wait_for_console_pattern(self, self.good_message, self.panic_message)
 
+    def test_ppc64_linux_migration(self):
+        self.set_machine('pseries')
+
+        kernel_path = self.ASSET_KERNEL.fetch()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE
+
+        dest_vm = self.get_vm(name="dest-qemu")
+        dest_vm.add_args('-incoming', 'defer')
+        dest_vm.add_args('-smp', '4')
+        dest_vm.add_args('-nodefaults')
+        dest_vm.add_args('-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        dest_vm.set_console()
+        dest_vm.launch()
+
+        source_vm = self.get_vm(name="source-qemu")
+        source_vm.add_args('-smp', '4')
+        source_vm.add_args('-nodefaults')
+        source_vm.add_args('-kernel', kernel_path,
+                           '-append', kernel_command_line)
+        source_vm.set_console()
+        source_vm.launch()
+
+        # ensure the boot has reached Linux
+        console_pattern = 'smp: Brought up 1 node, 4 CPUs'
+        wait_for_console_pattern(self, console_pattern, self.panic_message,
+                                 vm=source_vm)

tests/functional/ppc64/test_pseries.py def test_ppc64_linux_big_boot(self):

Quoted 10 lines
tests/functional/ppc64/test_pseries.py +7 -1
+
+        PpcMigrationTest().do_migrate_ppc64_linux(source_vm, dest_vm);
+
+        # ensure the boot proceeds after migration
+        wait_for_console_pattern(self, self.good_message, self.panic_message,
+                                 vm=dest_vm)
+
 if __name__ == '__main__':
     QemuSystemTest.main()
--

* Looks okay.

Thank you.
---
- Prasad

#6 lore raw

Re: [PATCH v3 2/2] tests/functional: Add a OS level migration test for pseries

Quoted 31 lines · Prasad Pandit
Quoted 30 lines · Fabiano Rosas

There's currently no OS level test for ppc64le. Add one such test by reusing the boot level tests that are already present.

The test boots the source machine, waits for it to reach a mid-boot message, migrates and checks that the destination has reached the final boot message (VFS error due to no disk).

---
  tests/functional/ppc64/test_migration.py | 12 ++++++++
  tests/functional/ppc64/test_pseries.py   | 35 ++++++++++++++++++++++++
  2 files changed, 47 insertions(+)
tests/functional/ppc64/test_migration.py +1 -0
index 5dfdaaf709..a3b819680b 100755
@@ -4,6 +4,7 @@
  #
  # ppc migration test
 
+from qemu_test.ports import Ports
  from migration import MigrationTest
 
 
@@ -21,6 +22,17 @@ def test_migration_with_exec(self):
          self.set_machine('mac99')
          self.migration_with_exec()
 
Quoted 10 lines

tests/functional/ppc64/test_migration.py def test_migration_with_exec(self): · line 30

Quoted 6 lines
tests/functional/ppc64/test_migration.py +6 -0
+    def do_migrate_ppc64_linux(self, source_vm, dest_vm):
+        with Ports() as ports:
+            port = ports.find_free_port()
+            if port is None:
+                self.skipTest('Failed to find a free port')
+            uri = 'tcp:localhost:%u' % port

* When port is None, shouldn't it return after the skipTest() call? With port = None, uri will become -> 'tcp:localhost:None' OR maybe port should have default value?

skipTest() aborts the test immediately, the remaining code after this statement is not executed anymore. (I think it internally raises an exception that is caught by the unittest code to mark the test as skipped, but you might better ask a Python wizard if you want to know the gory details. See https://docs.python.org/3/library/unittest.html#unittest-skipping for example)

HTH,
Thomas

#7 lore raw

Re: [PATCH v3 2/2] tests/functional: Add a OS level migration test for pseries

Quoted 6 lines · Thomas Huth

skipTest() aborts the test immediately, the remaining code after this statement is not executed anymore. (I think it internally raises an exception that is caught by the unittest code to mark the test as skipped, but you might better ask a Python wizard if you want to know the gory details. See
https://docs.python.org/3/library/unittest.html#unittest-skipping for example)

* I see, got it. Thank you for the link.
---
- Prasad