Skip to contentSkip to conversation

ppc: Fix migration issues with XICS and quiesce

PATCH In review Fabiano Rosas 3 participants 12 messages Aug 19, 2025 – Sep 18, 2025 PowerPC

lore
Share & tools
lore
#0 lore raw

cover ppc: Fix migration issues with XICS and quiesce

Fabian reports two issues with migration on ppc64le, one of which with a proposed fix, which I include in this series.

1) XICS migration causes a guest hang after migration due to missing ICP server state. Fix is to bring back the vmstate_register call for that device. Breaks backward migration, but it was already non-functional anyway.

2) pseries migration causes a guest hang after migration due to a new variable used to track the stopped state of vcpus, which is not migrated. Fix is to migrate the new variable. To avoid breaking backward migration, a compat property is added. Breaks forward migration, a workaround is proposed.

I also added some functional tests changes because there are currently no test that can detect the kind of hangs seen here. RFC on those, feel free to nitpick.

Thanks

CI run: https://gitlab.com/farosas/qemu/-/pipelines/1992482993

Fabian Vogt (1):
hw/intc/xics: Add missing call to register vmstate_icp_server

Fabiano Rosas (3):
tests/functional: Extract migration code into a new class tests/functional: Add a OS level migration test for pseries target/ppc: Fix env->quiesced migration

 hw/core/machine.c                       |  1 +
 hw/intc/xics.c                          |  2 ++
 target/ppc/cpu.h                        |  1 +
 target/ppc/cpu_init.c                   |  7 +++++
 target/ppc/machine.c                    | 40 ++++++++++++++++++++++++
 tests/functional/qemu_test/migration.py | 40 ++++++++++++++++++++++++
 tests/functional/test_migration.py      | 24 ++-------------
 tests/functional/test_ppc64_pseries.py  | 41 +++++++++++++++++++++++++
 8 files changed, 135 insertions(+), 21 deletions(-)
 create mode 100644 tests/functional/qemu_test/migration.py
Signature · 1 line
2.35.3
#1 lore raw

PATCH 1/4 hw/intc/xics: Add missing call to register vmstate_icp_server

From: Fabian Vogt <[email protected]>

An obsolete wrapper function with a workaround was removed entirely, without restoring the call it wrapped.

Without this, the guest is stuck after savevm/loadvm.

---
 hw/intc/xics.c | 2 ++
 1 file changed, 2 insertions(+)
hw/intc/xics.c +2 -0
index d9a199e883..200710eb6c 100644
@@ -335,6 +335,8 @@ static void icp_realize(DeviceState *dev, Error **errp)
             return;
         }
     }
+
+    vmstate_register(NULL, icp->cs->cpu_index, &vmstate_icp_server, icp);
 }
 
 static void icp_unrealize(DeviceState *dev)
Signature · 1 line
2.35.3
Reviewed-by on v1 1/4
#2 lore raw

Re: [PATCH 1/4] hw/intc/xics: Add missing call to register vmstate_icp_server

Quoted 33 lines · Fabiano Rosas

From: Fabian Vogt <[email protected]>

An obsolete wrapper function with a workaround was removed entirely, without restoring the call it wrapped.

Without this, the guest is stuck after savevm/loadvm.

---
 hw/intc/xics.c | 2 ++
 1 file changed, 2 insertions(+)
hw/intc/xics.c +2 -0
index d9a199e883..200710eb6c 100644
@@ -335,6 +335,8 @@ static void icp_realize(DeviceState *dev, Error **errp)
             return;
         }
     }
+
+    vmstate_register(NULL, icp->cs->cpu_index, &vmstate_icp_server, icp);
 }
 
 static void icp_unrealize(DeviceState *dev)
Signature · 1 line
2.35.3

I did some testing with QEMU-9.2.0 and 10.1.0 and my observations are:
1. QEMU-9.2.0
With XICS, both snapshots and migrations were broken and they work fine with this patch

2. QEMU-10.1.0
With XICS, snapshot and migration both are broken - lockups are observed (with and without this patch)

The 10.1.0 can be fixed in a follow up patch.

For now since 9.2 works fine, please feel free to add

Thanks,
Gautam

#3 lore raw

RFC PATCH 2/4 tests/functional: Extract migration code into a new class

In reply to Fabiano Rosas

Move some of the code from test_migration.py to a new class so it can be reused to invoke migrations from other tests.

---
I see this conflicts with Thomas' series, I'll update accordingly.

---
 tests/functional/qemu_test/migration.py | 40 +++++++++++++++++++++++++
 tests/functional/test_migration.py      | 24 ++-------------
 2 files changed, 43 insertions(+), 21 deletions(-)
 create mode 100644 tests/functional/qemu_test/migration.py
tests/functional/qemu_test/migration.py new file +40 -0
new file mode 100644
index 0000000000..37988704e8
@@ -0,0 +1,40 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Migration test
+#
+# Copyright (c) 2019 Red Hat, Inc.
+#
+# Authors:
+#  Cleber Rosa <[email protected]>
+#  Caio Carrara <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import time
+
+
+class Migration():
+
+    @staticmethod
+    def migration_finished(vm):
+        return vm.cmd('query-migrate')['status'] in ('completed', 'failed')
+
+    def assert_migration(self, test, src_vm, dst_vm, timeout):
+
+        end = time.monotonic() + timeout
+        while time.monotonic() < end and not self.migration_finished(src_vm):
+           time.sleep(0.1)
+
+        end = time.monotonic() + timeout
+        while time.monotonic() < end and not self.migration_finished(dst_vm):
+           time.sleep(0.1)
+
+        test.assertEqual(src_vm.cmd('query-migrate')['status'], 'completed')
+        test.assertEqual(dst_vm.cmd('query-migrate')['status'], 'completed')
+        test.assertEqual(dst_vm.cmd('query-status')['status'], 'running')
+        test.assertEqual(src_vm.cmd('query-status')['status'],'postmigrate')
+
+    def migrate(self, test, source_vm, dest_vm, src_uri, timeout):
+        source_vm.qmp('migrate', uri=src_uri)
+        self.assert_migration(test, source_vm, dest_vm, timeout)
tests/functional/test_migration.py +3 -21
index c4393c3543..1c75a98330 100755
@@ -15,6 +15,7 @@
 import time
 
 from qemu_test import QemuSystemTest, skipIfMissingCommands
+from qemu_test.migration import Migration
 from qemu_test.ports import Ports
 
 
@@ -22,25 +23,6 @@ class MigrationTest(QemuSystemTest):
 
     timeout = 10
 
-    @staticmethod
-    def migration_finished(vm):
-        return vm.cmd('query-migrate')['status'] in ('completed', 'failed')
-
-    def assert_migration(self, src_vm, dst_vm):
-
-        end = time.monotonic() + self.timeout
-        while time.monotonic() < end and not self.migration_finished(src_vm):
-           time.sleep(0.1)
-
-        end = time.monotonic() + self.timeout
-        while time.monotonic() < end and not self.migration_finished(dst_vm):
-           time.sleep(0.1)
-
-        self.assertEqual(src_vm.cmd('query-migrate')['status'], 'completed')
-        self.assertEqual(dst_vm.cmd('query-migrate')['status'], 'completed')
-        self.assertEqual(dst_vm.cmd('query-status')['status'], 'running')
-        self.assertEqual(src_vm.cmd('query-status')['status'],'postmigrate')
-
     def select_machine(self):
         target_machine = {
             'aarch64': 'quanta-gsj',
@@ -67,8 +49,8 @@ def do_migrate(self, dest_uri, src_uri=None):
         source_vm = self.get_vm(name="source-qemu")
         source_vm.add_args('-nodefaults')
         source_vm.launch()
-        source_vm.qmp('migrate', uri=src_uri)
-        self.assert_migration(source_vm, dest_vm)
+
+        Migration().migrate(self, source_vm, dest_vm, src_uri, self.timeout)
 
     def _get_free_port(self, ports):
         port = ports.find_free_port()
Signature · 1 line
2.35.3
Reviewed-by on v1 2/4
#4 lore raw

Re: [RFC PATCH 2/4] tests/functional: Extract migration code into a new class

Quoted 63 lines · Fabiano Rosas

Move some of the code from test_migration.py to a new class so it can be reused to invoke migrations from other tests.

---
I see this conflicts with Thomas' series, I'll update accordingly.

---
  tests/functional/qemu_test/migration.py | 40 +++++++++++++++++++++++++
  tests/functional/test_migration.py      | 24 ++-------------
  2 files changed, 43 insertions(+), 21 deletions(-)

create mode 100644 tests/functional/qemu_test/migration.py

tests/functional/qemu_test/migration.py new file +40 -0
new file mode 100644
index 0000000000..37988704e8
@@ -0,0 +1,40 @@
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# Migration test
+#
+# Copyright (c) 2019 Red Hat, Inc.
+#
+# Authors:
+#  Cleber Rosa <[email protected]>
+#  Caio Carrara <[email protected]>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later.  See the COPYING file in the top-level directory.
+
+import time
+
+
+class Migration():
+
+    @staticmethod
+    def migration_finished(vm):
+        return vm.cmd('query-migrate')['status'] in ('completed', 'failed')
+
+    def assert_migration(self, test, src_vm, dst_vm, timeout):
+
+        end = time.monotonic() + timeout
+        while time.monotonic() < end and not self.migration_finished(src_vm):
+           time.sleep(0.1)
+
+        end = time.monotonic() + timeout
+        while time.monotonic() < end and not self.migration_finished(dst_vm):
+           time.sleep(0.1)
+
+        test.assertEqual(src_vm.cmd('query-migrate')['status'], 'completed')
+        test.assertEqual(dst_vm.cmd('query-migrate')['status'], 'completed')
+        test.assertEqual(dst_vm.cmd('query-status')['status'], 'running')
+        test.assertEqual(src_vm.cmd('query-status')['status'],'postmigrate')
+
+    def migrate(self, test, source_vm, dest_vm, src_uri, timeout):
+        source_vm.qmp('migrate', uri=src_uri)
+        self.assert_migration(test, source_vm, dest_vm, timeout)
tests/functional/test_migration.py +0 -0
index c4393c3543..1c75a98330 100755

tests/functional/test_migration.py line 15

Quoted 1 line
tests/functional/test_migration.py +0 -0
@@ -15,6 +15,7 @@
  import time

I guess you could drop the "import time" here now?

Apart from that:

Quoted 42 lines
tests/functional/test_migration.py +3 -21
  from qemu_test import QemuSystemTest, skipIfMissingCommands
+from qemu_test.migration import Migration
  from qemu_test.ports import Ports
 
 
@@ -22,25 +23,6 @@ class MigrationTest(QemuSystemTest):
 
      timeout = 10
 
-    @staticmethod
-    def migration_finished(vm):
-        return vm.cmd('query-migrate')['status'] in ('completed', 'failed')
-
-    def assert_migration(self, src_vm, dst_vm):
-
-        end = time.monotonic() + self.timeout
-        while time.monotonic() < end and not self.migration_finished(src_vm):
-           time.sleep(0.1)
-
-        end = time.monotonic() + self.timeout
-        while time.monotonic() < end and not self.migration_finished(dst_vm):
-           time.sleep(0.1)
-
-        self.assertEqual(src_vm.cmd('query-migrate')['status'], 'completed')
-        self.assertEqual(dst_vm.cmd('query-migrate')['status'], 'completed')
-        self.assertEqual(dst_vm.cmd('query-status')['status'], 'running')
-        self.assertEqual(src_vm.cmd('query-status')['status'],'postmigrate')
-
      def select_machine(self):
          target_machine = {
              'aarch64': 'quanta-gsj',
@@ -67,8 +49,8 @@ def do_migrate(self, dest_uri, src_uri=None):
          source_vm = self.get_vm(name="source-qemu")
          source_vm.add_args('-nodefaults')
          source_vm.launch()
-        source_vm.qmp('migrate', uri=src_uri)
-        self.assert_migration(source_vm, dest_vm)
+
+        Migration().migrate(self, source_vm, dest_vm, src_uri, self.timeout)
 
      def _get_free_port(self, ports):
          port = ports.find_free_port()
#5 lore raw

RFC PATCH 3/4 tests/functional: Add a OS level migration test for pseries

In reply to Fabiano Rosas

There's currently no OS level test for ppc64le. Add one such test by reusing the boot level tests that are already present.

The test boots the source machine, waits for it to reach a mid-boot message, migrates and checks that the destination has reached the final boot message (VFS error due to no disk).

---
 tests/functional/test_ppc64_pseries.py | 41 ++++++++++++++++++++++++++
 1 file changed, 41 insertions(+)
tests/functional/test_ppc64_pseries.py +41 -0
index 67057934e8..7a7e0fe8ae 100755
@@ -9,6 +9,8 @@
 
 from qemu_test import QemuSystemTest, Asset
 from qemu_test import wait_for_console_pattern
+from qemu_test.migration import Migration
+from qemu_test.ports import Ports
 
 class pseriesMachine(QemuSystemTest):
 
@@ -87,5 +89,44 @@ def test_ppc64_linux_big_boot(self):
         wait_for_console_pattern(self, console_pattern, self.panic_message)
         wait_for_console_pattern(self, self.good_message, self.panic_message)
 
+    def test_ppc64_linux_migration(self):
+        with Ports() as ports:
+            port = ports.find_free_port()
+            if port is None:
+                self.skipTest('Failed to find a free port')
+        uri = 'tcp:localhost:%u' % port
+
+        kernel_path = self.ASSET_KERNEL.fetch()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE
+
+        self.set_machine('pseries')
+
+        dest_vm = self.get_vm('-incoming', uri, name="dest-qemu")
+        dest_vm.add_args('-smp', '4')
+        dest_vm.add_args('-nodefaults')
+        dest_vm.add_args('-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        dest_vm.set_console()
+        dest_vm.launch()
+
+        source_vm = self.get_vm(name="source-qemu")
+        source_vm.add_args('-smp', '4')
+        source_vm.add_args('-nodefaults')
+        source_vm.add_args('-kernel', kernel_path,
+                           '-append', kernel_command_line)
+        source_vm.set_console()
+        source_vm.launch()
+
+        # ensure the boot has reached Linux
+        console_pattern = 'smp: Brought up 1 node, 4 CPUs'
+        wait_for_console_pattern(self, console_pattern, self.panic_message,
+                                 vm=source_vm)
+
+        Migration().migrate(self, source_vm, dest_vm, uri, self.timeout)
+
+        # ensure the boot proceeds after migration
+        wait_for_console_pattern(self, self.good_message, self.panic_message,
+                                 vm=dest_vm)
+
 if __name__ == '__main__':
     QemuSystemTest.main()
Signature · 1 line
2.35.3
#6 lore raw

Re: [RFC PATCH 3/4] tests/functional: Add a OS level migration test for pseries

Quoted 26 lines · Fabiano Rosas

There's currently no OS level test for ppc64le. Add one such test by reusing the boot level tests that are already present.

The test boots the source machine, waits for it to reach a mid-boot message, migrates and checks that the destination has reached the final boot message (VFS error due to no disk).

---
  tests/functional/test_ppc64_pseries.py | 41 ++++++++++++++++++++++++++
  1 file changed, 41 insertions(+)
tests/functional/test_ppc64_pseries.py +2 -0
index 67057934e8..7a7e0fe8ae 100755
@@ -9,6 +9,8 @@
 
  from qemu_test import QemuSystemTest, Asset
  from qemu_test import wait_for_console_pattern
+from qemu_test.migration import Migration
+from qemu_test.ports import Ports
 
  class pseriesMachine(QemuSystemTest):
 

tests/functional/test_ppc64_pseries.py def test_ppc64_linux_big_boot(self): · line 97

Quoted 9 lines
tests/functional/test_ppc64_pseries.py +6 -0
@@ -87,5 +89,44 @@ def test_ppc64_linux_big_boot(self):
          wait_for_console_pattern(self, console_pattern, self.panic_message)
          wait_for_console_pattern(self, self.good_message, self.panic_message)
 
+    def test_ppc64_linux_migration(self):
+        with Ports() as ports:
+            port = ports.find_free_port()
+            if port is None:
+                self.skipTest('Failed to find a free port')
+        uri = 'tcp:localhost:%u' % port

Hi,

this is not how to use the context for Ports: Once the "with" block is left, the locking for the free port will be gone and you're subject to a race condition with other tests running in parallel (see the __enter__ and __exit__ methods in tests/functional/qemu_test/ports.py ... and yes, there should be more documentation for this).

You've got to put everything up to the point where QEMU takes the port into the "with" block, i.e. everything up to including the Migration().migrate() line.

Thomas

Quoted 34 lines
tests/functional/test_ppc64_pseries.py +32 -0
+        kernel_path = self.ASSET_KERNEL.fetch()
+        kernel_command_line = self.KERNEL_COMMON_COMMAND_LINE
+
+        self.set_machine('pseries')
+
+        dest_vm = self.get_vm('-incoming', uri, name="dest-qemu")
+        dest_vm.add_args('-smp', '4')
+        dest_vm.add_args('-nodefaults')
+        dest_vm.add_args('-kernel', kernel_path,
+                         '-append', kernel_command_line)
+        dest_vm.set_console()
+        dest_vm.launch()
+
+        source_vm = self.get_vm(name="source-qemu")
+        source_vm.add_args('-smp', '4')
+        source_vm.add_args('-nodefaults')
+        source_vm.add_args('-kernel', kernel_path,
+                           '-append', kernel_command_line)
+        source_vm.set_console()
+        source_vm.launch()
+
+        # ensure the boot has reached Linux
+        console_pattern = 'smp: Brought up 1 node, 4 CPUs'
+        wait_for_console_pattern(self, console_pattern, self.panic_message,
+                                 vm=source_vm)
+
+        Migration().migrate(self, source_vm, dest_vm, uri, self.timeout)
+
+        # ensure the boot proceeds after migration
+        wait_for_console_pattern(self, self.good_message, self.panic_message,
+                                 vm=dest_vm)
+
  if __name__ == '__main__':
      QemuSystemTest.main()
#7 lore raw

Re: [RFC PATCH 3/4] tests/functional: Add a OS level migration test for pseries

Quoted 39 lines · Thomas Huth
Quoted 26 lines · Fabiano Rosas

There's currently no OS level test for ppc64le. Add one such test by reusing the boot level tests that are already present.

The test boots the source machine, waits for it to reach a mid-boot message, migrates and checks that the destination has reached the final boot message (VFS error due to no disk).

---
  tests/functional/test_ppc64_pseries.py | 41 ++++++++++++++++++++++++++
  1 file changed, 41 insertions(+)
tests/functional/test_ppc64_pseries.py +2 -0
index 67057934e8..7a7e0fe8ae 100755
@@ -9,6 +9,8 @@
 
  from qemu_test import QemuSystemTest, Asset
  from qemu_test import wait_for_console_pattern
+from qemu_test.migration import Migration
+from qemu_test.ports import Ports
 
  class pseriesMachine(QemuSystemTest):
 

tests/functional/test_ppc64_pseries.py def test_ppc64_linux_big_boot(self): · line 97

Quoted 9 lines
tests/functional/test_ppc64_pseries.py +6 -0
@@ -87,5 +89,44 @@ def test_ppc64_linux_big_boot(self):
          wait_for_console_pattern(self, console_pattern, self.panic_message)
          wait_for_console_pattern(self, self.good_message, self.panic_message)
 
+    def test_ppc64_linux_migration(self):
+        with Ports() as ports:
+            port = ports.find_free_port()
+            if port is None:
+                self.skipTest('Failed to find a free port')
+        uri = 'tcp:localhost:%u' % port

Hi,

Quoted 6 lines

this is not how to use the context for Ports: Once the "with" block is left, the locking for the free port will be gone and you're subject to a race condition with other tests running in parallel (see the __enter__ and __exit__ methods in tests/functional/qemu_test/ports.py ... and yes, there should be more documentation for this).

Haha, I'm dumb. It never crossed my mind.

Thanks

#8 lore raw

PATCH 4/4 target/ppc: Fix env->quiesced migration

In reply to Fabiano Rosas

The commit referenced (from QEMU 10.0) has changed the way the pseries machine marks a cpu as quiesced. Previously, the cpu->halted value from QEMU common cpu code was (incorrectly) used. With the fix, the env->quiesced variable starts being used, which improves on the original situation, but also causes a side effect after migration:

The env->quiesced is set at reset and never migrated, which causes the destination QEMU to stop delivering interrupts and hang the machine.

To fix the issue from this point on, start migrating the env->quiesced value.

For QEMU versions < 10.0, sending the new element on the stream would cause migration to be aborted, so add the appropriate compatibility property to omit the new subsection.

Independently of this patch, all migrations from QEMU versions < 10.0 will result in a hang since the older QEMU never migrates env->quiesced. This is bad because it leaves machines already running on the old QEMU without a migration path into newer versions.

As a workaround, clear env->quiesced in the new QEMU whenever cpu->halted is also clear. This assumes rtas_stop_self() always sets both flags at the same time. Migrations during secondaries bringup (i.e. before rtas-start-cpu) will still cause a hang, but those are early enough that requiring reboot would not be unreasonable.

Note that this was tested with -cpu power9 and -machine ic-mode=xive due to another bug affecting migration of XICS guests. Tested both forward and backward migration and savevm/loadvm from 9.2 and 10.0.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3079

Fixes fb802acdc8b ("ppc/spapr: Fix RTAS stopped state")

---
The choice of PowerPCCPU to hold the compat property is dubious. This only affects pseries, but it seems like a layering violation to access SpaprMachine from target/ppc/, suggestions welcome.

---
 hw/core/machine.c     |  1 +
 target/ppc/cpu.h      |  1 +
 target/ppc/cpu_init.c |  7 +++++++
 target/ppc/machine.c  | 40 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 49 insertions(+)
hw/core/machine.c +1 -0
index bd47527479..ea83c0876b 100644
@@ -42,6 +42,7 @@ GlobalProperty hw_compat_10_0[] = {
     { "vfio-pci", "x-migration-load-config-after-iter", "off" },
     { "ramfb", "use-legacy-x86-rom", "true"},
     { "vfio-pci-nohotplug", "use-legacy-x86-rom", "true" },
+    { "powerpc64-cpu", "rtas-stopped-state", "false" },
 };
 const size_t hw_compat_10_0_len = G_N_ELEMENTS(hw_compat_10_0);
 
target/ppc/cpu.h +1 -0
index 6b90543811..8ff453024b 100644
@@ -1470,6 +1470,7 @@ struct ArchCPU {
     void *machine_data;
     int32_t node_id; /* NUMA node this CPU belongs to */
     PPCHash64Options *hash64_opts;
+    bool rtas_stopped_state;
 
     /* Those resources are used only during code translation */
     /* opcode handlers */
target/ppc/cpu_init.c +7 -0
index a0e77f2673..4380c6eb14 100644
@@ -55,6 +55,11 @@
 /* #define PPC_DEBUG_SPR */
 /* #define USE_APPLE_GDB */
 
+static const Property powerpc_cpu_properties[] = {
+    DEFINE_PROP_BOOL("rtas-stopped-state", PowerPCCPU,
+                      rtas_stopped_state, true),
+};
+
 static inline void vscr_init(CPUPPCState *env, uint32_t val)
 {
     /* Altivec always uses round-to-nearest */
@@ -7525,6 +7530,8 @@ static void ppc_cpu_class_init(ObjectClass *oc, const void *data)
                                       &pcc->parent_unrealize);
     pcc->pvr_match = ppc_pvr_match_default;
 
+    device_class_set_props(dc, powerpc_cpu_properties);
+
     resettable_class_set_parent_phases(rc, NULL, ppc_cpu_reset_hold, NULL,
                                        &pcc->parent_phases);
 
target/ppc/machine.c +40 -0
index d72e5ecb94..8797233ebe 100644
@@ -257,6 +257,23 @@ static int cpu_post_load(void *opaque, int version_id)
         ppc_store_sdr1(env, env->spr[SPR_SDR1]);
     }
 
+    if (!cpu->rtas_stopped_state) {
+        /*
+         * The source QEMU doesn't have fb802acdc8 and still uses halt
+         * + PM bits in LPCR to implement RTAS stopped state. The new
+         * QEMU will have put the newly created vcpus in that state,
+         * waiting for the start-cpu RTAS call. Clear the quiesced
+         * flag if possible, otherwise the newly-loaded machine will
+         * hang indefinitely due to quiesced state ignoring
+         * interrupts.
+         */
+
+        if (!CPU(cpu)->halted) {
+            /* not halted, so definitely not in RTAS stopped state */
+            env->quiesced = 0;
+        }
+    }
+
     post_load_update_msr(env);
 
     if (tcg_enabled()) {
@@ -649,6 +666,28 @@ static const VMStateDescription vmstate_reservation = {
     }
 };
 
+static bool rtas_stopped_needed(void *opaque)
+{
+    PowerPCCPU *cpu = opaque;
+
+    return cpu->rtas_stopped_state && !cpu->env.quiesced;
+}
+
+static const VMStateDescription vmstate_rtas_stopped = {
+    .name = "cpu/rtas_stopped",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = rtas_stopped_needed,
+    .fields = (const VMStateField[]) {
+        /*
+         * "RTAS stopped" state, independent of halted state. For QEMU
+         * < 10.0, this is taken from cpu->halted at cpu_post_load()
+         */
+        VMSTATE_BOOL(env.quiesced, PowerPCCPU),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
 #ifdef TARGET_PPC64
 static bool bhrb_needed(void *opaque)
 {
@@ -715,6 +754,7 @@ const VMStateDescription vmstate_ppc_cpu = {
         &vmstate_tlbmas,
         &vmstate_compat,
         &vmstate_reservation,
+        &vmstate_rtas_stopped,
         NULL
     }
 };
Signature · 1 line
2.35.3
#9 lore raw

Re: [PATCH 4/4] target/ppc: Fix env->quiesced migration

Quoted 51 lines · Fabiano Rosas

The commit referenced (from QEMU 10.0) has changed the way the pseries machine marks a cpu as quiesced. Previously, the cpu->halted value from QEMU common cpu code was (incorrectly) used. With the fix, the env->quiesced variable starts being used, which improves on the original situation, but also causes a side effect after migration:

The env->quiesced is set at reset and never migrated, which causes the destination QEMU to stop delivering interrupts and hang the machine.

To fix the issue from this point on, start migrating the env->quiesced value.

For QEMU versions < 10.0, sending the new element on the stream would cause migration to be aborted, so add the appropriate compatibility property to omit the new subsection.

Independently of this patch, all migrations from QEMU versions < 10.0 will result in a hang since the older QEMU never migrates env->quiesced. This is bad because it leaves machines already running on the old QEMU without a migration path into newer versions.

As a workaround, clear env->quiesced in the new QEMU whenever cpu->halted is also clear. This assumes rtas_stop_self() always sets both flags at the same time. Migrations during secondaries bringup (i.e. before rtas-start-cpu) will still cause a hang, but those are early enough that requiring reboot would not be unreasonable.

Note that this was tested with -cpu power9 and -machine ic-mode=xive due to another bug affecting migration of XICS guests. Tested both forward and backward migration and savevm/loadvm from 9.2 and 10.0.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3079

Fixes fb802acdc8b ("ppc/spapr: Fix RTAS stopped state")

---
The choice of PowerPCCPU to hold the compat property is dubious. This only affects pseries, but it seems like a layering violation to access SpaprMachine from target/ppc/, suggestions welcome.

---
  hw/core/machine.c     |  1 +
  target/ppc/cpu.h      |  1 +
  target/ppc/cpu_init.c |  7 +++++++
  target/ppc/machine.c  | 40 ++++++++++++++++++++++++++++++++++++++++
  4 files changed, 49 insertions(+)
hw/core/machine.c +0 -0
index bd47527479..ea83c0876b 100644

hw/core/machine.c GlobalProperty hw_compat_10_0[] = { · line 45

Quoted 4 lines
hw/core/machine.c +1 -0
@@ -42,6 +42,7 @@ GlobalProperty hw_compat_10_0[] = {
      { "vfio-pci", "x-migration-load-config-after-iter", "off" },
      { "ramfb", "use-legacy-x86-rom", "true"},
      { "vfio-pci-nohotplug", "use-legacy-x86-rom", "true" },
+    { "powerpc64-cpu", "rtas-stopped-state", "false" },

This is specific to ppc, so it should not go into the generic hw_compat_* array.

Please define a spapr_compat_10_0 array in spapr_machine_10_0_class_options() and do another compat_props_add() for that array there. (Similar to what is done for TYPE_SPAPR_PCI_HOST_BRIDGE in spapr_machine_5_0_class_options() for example)

Thanks,
Thomas

Quoted 105 lines
hw/core/machine.c +0 -0
  };
  const size_t hw_compat_10_0_len = G_N_ELEMENTS(hw_compat_10_0);
 
target/ppc/cpu.h +1 -0
index 6b90543811..8ff453024b 100644
@@ -1470,6 +1470,7 @@ struct ArchCPU {
      void *machine_data;
      int32_t node_id; /* NUMA node this CPU belongs to */
      PPCHash64Options *hash64_opts;
+    bool rtas_stopped_state;
 
      /* Those resources are used only during code translation */
      /* opcode handlers */
target/ppc/cpu_init.c +7 -0
index a0e77f2673..4380c6eb14 100644
@@ -55,6 +55,11 @@
  /* #define PPC_DEBUG_SPR */
  /* #define USE_APPLE_GDB */
 
+static const Property powerpc_cpu_properties[] = {
+    DEFINE_PROP_BOOL("rtas-stopped-state", PowerPCCPU,
+                      rtas_stopped_state, true),
+};
+
  static inline void vscr_init(CPUPPCState *env, uint32_t val)
  {
      /* Altivec always uses round-to-nearest */
@@ -7525,6 +7530,8 @@ static void ppc_cpu_class_init(ObjectClass *oc, const void *data)
                                        &pcc->parent_unrealize);
      pcc->pvr_match = ppc_pvr_match_default;
 
+    device_class_set_props(dc, powerpc_cpu_properties);
+
      resettable_class_set_parent_phases(rc, NULL, ppc_cpu_reset_hold, NULL,
                                         &pcc->parent_phases);
 
target/ppc/machine.c +40 -0
index d72e5ecb94..8797233ebe 100644
@@ -257,6 +257,23 @@ static int cpu_post_load(void *opaque, int version_id)
          ppc_store_sdr1(env, env->spr[SPR_SDR1]);
      }
 
+    if (!cpu->rtas_stopped_state) {
+        /*
+         * The source QEMU doesn't have fb802acdc8 and still uses halt
+         * + PM bits in LPCR to implement RTAS stopped state. The new
+         * QEMU will have put the newly created vcpus in that state,
+         * waiting for the start-cpu RTAS call. Clear the quiesced
+         * flag if possible, otherwise the newly-loaded machine will
+         * hang indefinitely due to quiesced state ignoring
+         * interrupts.
+         */
+
+        if (!CPU(cpu)->halted) {
+            /* not halted, so definitely not in RTAS stopped state */
+            env->quiesced = 0;
+        }
+    }
+
      post_load_update_msr(env);
 
      if (tcg_enabled()) {
@@ -649,6 +666,28 @@ static const VMStateDescription vmstate_reservation = {
      }
  };
 
+static bool rtas_stopped_needed(void *opaque)
+{
+    PowerPCCPU *cpu = opaque;
+
+    return cpu->rtas_stopped_state && !cpu->env.quiesced;
+}
+
+static const VMStateDescription vmstate_rtas_stopped = {
+    .name = "cpu/rtas_stopped",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = rtas_stopped_needed,
+    .fields = (const VMStateField[]) {
+        /*
+         * "RTAS stopped" state, independent of halted state. For QEMU
+         * < 10.0, this is taken from cpu->halted at cpu_post_load()
+         */
+        VMSTATE_BOOL(env.quiesced, PowerPCCPU),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
  #ifdef TARGET_PPC64
  static bool bhrb_needed(void *opaque)
  {
@@ -715,6 +754,7 @@ const VMStateDescription vmstate_ppc_cpu = {
          &vmstate_tlbmas,
          &vmstate_compat,
          &vmstate_reservation,
+        &vmstate_rtas_stopped,
          NULL
      }
  };
#10 lore raw

Re: [PATCH 4/4] target/ppc: Fix env->quiesced migration

Quoted 47 lines · Thomas Huth
Quoted 46 lines · Fabiano Rosas

The commit referenced (from QEMU 10.0) has changed the way the pseries machine marks a cpu as quiesced. Previously, the cpu->halted value from QEMU common cpu code was (incorrectly) used. With the fix, the env->quiesced variable starts being used, which improves on the original situation, but also causes a side effect after migration:

The env->quiesced is set at reset and never migrated, which causes the destination QEMU to stop delivering interrupts and hang the machine.

To fix the issue from this point on, start migrating the env->quiesced value.

For QEMU versions < 10.0, sending the new element on the stream would cause migration to be aborted, so add the appropriate compatibility property to omit the new subsection.

Independently of this patch, all migrations from QEMU versions < 10.0 will result in a hang since the older QEMU never migrates env->quiesced. This is bad because it leaves machines already running on the old QEMU without a migration path into newer versions.

As a workaround, clear env->quiesced in the new QEMU whenever cpu->halted is also clear. This assumes rtas_stop_self() always sets both flags at the same time. Migrations during secondaries bringup (i.e. before rtas-start-cpu) will still cause a hang, but those are early enough that requiring reboot would not be unreasonable.

Note that this was tested with -cpu power9 and -machine ic-mode=xive due to another bug affecting migration of XICS guests. Tested both forward and backward migration and savevm/loadvm from 9.2 and 10.0.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3079

Fixes fb802acdc8b ("ppc/spapr: Fix RTAS stopped state")

---
The choice of PowerPCCPU to hold the compat property is dubious. This only affects pseries, but it seems like a layering violation to access SpaprMachine from target/ppc/, suggestions welcome.

---
  hw/core/machine.c     |  1 +
  target/ppc/cpu.h      |  1 +
  target/ppc/cpu_init.c |  7 +++++++
  target/ppc/machine.c  | 40 ++++++++++++++++++++++++++++++++++++++++
  4 files changed, 49 insertions(+)
Quoted 12 lines

hw/core/machine.c GlobalProperty hw_compat_10_0[] = { · line 45

Quoted 9 lines
hw/core/machine.c +1 -0
index bd47527479..ea83c0876b 100644
@@ -42,6 +42,7 @@ GlobalProperty hw_compat_10_0[] = {
      { "vfio-pci", "x-migration-load-config-after-iter", "off" },
      { "ramfb", "use-legacy-x86-rom", "true"},
      { "vfio-pci-nohotplug", "use-legacy-x86-rom", "true" },
+    { "powerpc64-cpu", "rtas-stopped-state", "false" },

This is specific to ppc, so it should not go into the generic hw_compat_* array.

So arm-cpu in hw_compat_9_2 should not be there?

Quoted 5 lines

Please define a spapr_compat_10_0 array in spapr_machine_10_0_class_options() and do another compat_props_add() for that array there. (Similar to what is done for TYPE_SPAPR_PCI_HOST_BRIDGE in spapr_machine_5_0_class_options() for example)

Ok, thanks for the pointer.

Quoted 109 lines

Thanks,
Thomas

Quoted 105 lines
hw/core/machine.c +0 -0
  };
  const size_t hw_compat_10_0_len = G_N_ELEMENTS(hw_compat_10_0);
 
target/ppc/cpu.h +1 -0
index 6b90543811..8ff453024b 100644
@@ -1470,6 +1470,7 @@ struct ArchCPU {
      void *machine_data;
      int32_t node_id; /* NUMA node this CPU belongs to */
      PPCHash64Options *hash64_opts;
+    bool rtas_stopped_state;
 
      /* Those resources are used only during code translation */
      /* opcode handlers */
target/ppc/cpu_init.c +7 -0
index a0e77f2673..4380c6eb14 100644
@@ -55,6 +55,11 @@
  /* #define PPC_DEBUG_SPR */
  /* #define USE_APPLE_GDB */
 
+static const Property powerpc_cpu_properties[] = {
+    DEFINE_PROP_BOOL("rtas-stopped-state", PowerPCCPU,
+                      rtas_stopped_state, true),
+};
+
  static inline void vscr_init(CPUPPCState *env, uint32_t val)
  {
      /* Altivec always uses round-to-nearest */
@@ -7525,6 +7530,8 @@ static void ppc_cpu_class_init(ObjectClass *oc, const void *data)
                                        &pcc->parent_unrealize);
      pcc->pvr_match = ppc_pvr_match_default;
 
+    device_class_set_props(dc, powerpc_cpu_properties);
+
      resettable_class_set_parent_phases(rc, NULL, ppc_cpu_reset_hold, NULL,
                                         &pcc->parent_phases);
 
target/ppc/machine.c +40 -0
index d72e5ecb94..8797233ebe 100644
@@ -257,6 +257,23 @@ static int cpu_post_load(void *opaque, int version_id)
          ppc_store_sdr1(env, env->spr[SPR_SDR1]);
      }
 
+    if (!cpu->rtas_stopped_state) {
+        /*
+         * The source QEMU doesn't have fb802acdc8 and still uses halt
+         * + PM bits in LPCR to implement RTAS stopped state. The new
+         * QEMU will have put the newly created vcpus in that state,
+         * waiting for the start-cpu RTAS call. Clear the quiesced
+         * flag if possible, otherwise the newly-loaded machine will
+         * hang indefinitely due to quiesced state ignoring
+         * interrupts.
+         */
+
+        if (!CPU(cpu)->halted) {
+            /* not halted, so definitely not in RTAS stopped state */
+            env->quiesced = 0;
+        }
+    }
+
      post_load_update_msr(env);
 
      if (tcg_enabled()) {
@@ -649,6 +666,28 @@ static const VMStateDescription vmstate_reservation = {
      }
  };
 
+static bool rtas_stopped_needed(void *opaque)
+{
+    PowerPCCPU *cpu = opaque;
+
+    return cpu->rtas_stopped_state && !cpu->env.quiesced;
+}
+
+static const VMStateDescription vmstate_rtas_stopped = {
+    .name = "cpu/rtas_stopped",
+    .version_id = 1,
+    .minimum_version_id = 1,
+    .needed = rtas_stopped_needed,
+    .fields = (const VMStateField[]) {
+        /*
+         * "RTAS stopped" state, independent of halted state. For QEMU
+         * < 10.0, this is taken from cpu->halted at cpu_post_load()
+         */
+        VMSTATE_BOOL(env.quiesced, PowerPCCPU),
+        VMSTATE_END_OF_LIST()
+    }
+};
+
  #ifdef TARGET_PPC64
  static bool bhrb_needed(void *opaque)
  {
@@ -715,6 +754,7 @@ const VMStateDescription vmstate_ppc_cpu = {
          &vmstate_tlbmas,
          &vmstate_compat,
          &vmstate_reservation,
+        &vmstate_rtas_stopped,
          NULL
      }
  };
#11 lore raw

Re: [PATCH 4/4] target/ppc: Fix env->quiesced migration

Quoted 62 lines · Fabiano Rosas
Quoted 47 lines · Thomas Huth
Quoted 46 lines · Fabiano Rosas

The commit referenced (from QEMU 10.0) has changed the way the pseries machine marks a cpu as quiesced. Previously, the cpu->halted value from QEMU common cpu code was (incorrectly) used. With the fix, the env->quiesced variable starts being used, which improves on the original situation, but also causes a side effect after migration:

The env->quiesced is set at reset and never migrated, which causes the destination QEMU to stop delivering interrupts and hang the machine.

To fix the issue from this point on, start migrating the env->quiesced value.

For QEMU versions < 10.0, sending the new element on the stream would cause migration to be aborted, so add the appropriate compatibility property to omit the new subsection.

Independently of this patch, all migrations from QEMU versions < 10.0 will result in a hang since the older QEMU never migrates env->quiesced. This is bad because it leaves machines already running on the old QEMU without a migration path into newer versions.

As a workaround, clear env->quiesced in the new QEMU whenever cpu->halted is also clear. This assumes rtas_stop_self() always sets both flags at the same time. Migrations during secondaries bringup (i.e. before rtas-start-cpu) will still cause a hang, but those are early enough that requiring reboot would not be unreasonable.

Note that this was tested with -cpu power9 and -machine ic-mode=xive due to another bug affecting migration of XICS guests. Tested both forward and backward migration and savevm/loadvm from 9.2 and 10.0.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/3079

Fixes fb802acdc8b ("ppc/spapr: Fix RTAS stopped state")

---
The choice of PowerPCCPU to hold the compat property is dubious. This only affects pseries, but it seems like a layering violation to access SpaprMachine from target/ppc/, suggestions welcome.

---
   hw/core/machine.c     |  1 +
   target/ppc/cpu.h      |  1 +
   target/ppc/cpu_init.c |  7 +++++++
   target/ppc/machine.c  | 40 ++++++++++++++++++++++++++++++++++++++++
   4 files changed, 49 insertions(+)
Quoted 12 lines

hw/core/machine.c GlobalProperty hw_compat_10_0[] = { · line 45

Quoted 9 lines
hw/core/machine.c +1 -0
index bd47527479..ea83c0876b 100644
@@ -42,6 +42,7 @@ GlobalProperty hw_compat_10_0[] = {
       { "vfio-pci", "x-migration-load-config-after-iter", "off" },
       { "ramfb", "use-legacy-x86-rom", "true"},
       { "vfio-pci-nohotplug", "use-legacy-x86-rom", "true" },
+    { "powerpc64-cpu", "rtas-stopped-state", "false" },

This is specific to ppc, so it should not go into the generic hw_compat_* array.

Quoted 1 line

So arm-cpu in hw_compat_9_2 should not be there?

Right, this should get moved to the code in hw/arm/virt.c.

Same for arm-cpu in hw_compat_9_0 and for arm-gicv3-common in hw_compat_7_0.

Thomas