Skip to contentSkip to conversation

accel: Only include qdev-realized vCPUs in global &cpus_queue

PATCH In review Igor Mammedov 5 participants 26 messages Feb 7, 2025 – Mar 3, 2025

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

v2 cover accel: Only include qdev-realized vCPUs in global &cpus_queue

Changelog:
* drop wire/unwire hooks patches
* drop unrealize related patches
* include fixed up patches from
[PATCH 0/6] tcg: fix qemu crash when add assert_cpu_is_self() is enabled
and cleanups related to cpu->created check
https://patchew.org/QEMU/[email protected]/
as it's related to the topic (well, modulo bsd cleanup)
* CI mostly green modulo rust failure on Ubuntu

     https://gitlab.com/imammedo/qemu/-/pipelines/1660855467

The goal of this series is to expose vCPUs in a stable state to the accelerators, in particular the QDev 'REALIZED' step.

To do this we split out cpu_index assignment into a separate step, and move call cpu_list_add() to the end of CPU realize stage.

I expect these changes to allow CPUState::cpu_index clarifications and simplifications, but this will be addressed (and commented) in a separate series.

As result, the series also
* fix regression intoroduced by
30933c4fb4f3d ("tcg/cputlb: remove other-cpu capability from TLB flushing")
for deatials see 'tcg:tlb: use tcg_debug_assert() in assert_cpu_is_self()'
* drops no longer needed workaround 'cpu->check' due to vCPU being exposed too early in cpus_queue.

CC Paolo Bonzini <[email protected]>
CC Richard Henderson <[email protected]>
CC "Philippe Mathieu-Daudé" <[email protected]>
CC Alex Bennée <[email protected]>

Igor Mammedov (7):
bsd-user: drop not longer used target_reset_cpu()
loongarch: reset vcpu after it's created
m68k: reset vcpu after it's created
tcg:tlb: use tcg_debug_assert() in assert_cpu_is_self() Revert "tcg/cputlb: remove other-cpu capability from TLB flushing"
tcg: drop cpu->created check
cpus: expose only realized vCPUs to global &cpus_queue

Philippe Mathieu-Daudé (3):
accel/tcg: Simplify use of &first_cpu in rr_cpu_thread_fn() accel/kvm: Assert vCPU is created when calling kvm_dirty_ring_reap*() accel/kvm: Remove unreachable assertion in kvm_dirty_ring_reap*()

 bsd-user/aarch64/target_arch_cpu.h |  5 ---
 bsd-user/arm/target_arch_cpu.h     |  4 ---
 bsd-user/i386/target_arch_cpu.h    |  5 ---
 bsd-user/riscv/target_arch_cpu.h   |  4 ---
 bsd-user/x86_64/target_arch_cpu.h  |  5 ---
 include/hw/core/cpu.h              |  6 ++++
 accel/kvm/kvm-all.c                |  9 ------
 accel/tcg/cputlb.c                 | 49 +++++++++++++++++++++---------
 accel/tcg/tcg-accel-ops-rr.c       | 13 +++++---
 cpu-common.c                       | 23 ++++++++------
 cpu-target.c                       |  2 +-
 hw/core/cpu-common.c               |  2 ++
 target/loongarch/cpu.c             |  2 +-
 target/m68k/cpu.c                  |  2 +-
 14 files changed, 68 insertions(+), 63 deletions(-)
Signature · 1 line
2.43.0
#1 lore raw

PATCH v2 1/10 bsd-user: drop not longer used target_reset_cpu()

target_reset_cpu() static inlines have no user,
remove them.

---
CC: Warner Losh <[email protected]>
CC: Kyle Evans <[email protected]>

---
 bsd-user/aarch64/target_arch_cpu.h | 5 -----
 bsd-user/arm/target_arch_cpu.h     | 4 ----
 bsd-user/i386/target_arch_cpu.h    | 5 -----
 bsd-user/riscv/target_arch_cpu.h   | 4 ----
 bsd-user/x86_64/target_arch_cpu.h  | 5 -----
 5 files changed, 23 deletions(-)
bsd-user/aarch64/target_arch_cpu.h +0 -5
index 87fbf6d677..46a448e93f 100644
@@ -181,9 +181,4 @@ static inline void target_cpu_clone_regs(CPUARMState *env, target_ulong newsp)
     pstate_write(env, 0);
 }
 
-static inline void target_cpu_reset(CPUArchState *env)
-{
-}
-
-
 #endif /* TARGET_ARCH_CPU_H */
bsd-user/arm/target_arch_cpu.h +0 -4
index bc2eaa0bf4..b9583b0f92 100644
@@ -206,8 +206,4 @@ static inline void target_cpu_clone_regs(CPUARMState *env, target_ulong newsp)
     env->regs[0] = 0;
 }
 
-static inline void target_cpu_reset(CPUArchState *env)
-{
-}
-
 #endif /* TARGET_ARCH_CPU_H */
bsd-user/i386/target_arch_cpu.h +0 -5
index 5d4c931dec..371e702799 100644
@@ -194,9 +194,4 @@ static inline void target_cpu_clone_regs(CPUX86State *env, target_ulong newsp)
     env->regs[R_EAX] = 0;
 }
 
-static inline void target_cpu_reset(CPUArchState *env)
-{
-    cpu_reset(env_cpu(env));
-}
-
 #endif /* TARGET_ARCH_CPU_H */
bsd-user/riscv/target_arch_cpu.h +0 -4
index ef92f00480..d3cc5adbf4 100644
@@ -141,8 +141,4 @@ static inline void target_cpu_clone_regs(CPURISCVState *env, target_ulong newsp)
     env->gpr[xT0] = 0;
 }
 
-static inline void target_cpu_reset(CPUArchState *env)
-{
-}
-
 #endif /* TARGET_ARCH_CPU_H */
bsd-user/x86_64/target_arch_cpu.h +0 -5
index f82042e30a..8ec5c65fab 100644
@@ -169,9 +169,4 @@ static inline void target_cpu_clone_regs(CPUX86State *env, target_ulong newsp)
     env->regs[R_EAX] = 0;
 }
 
-static inline void target_cpu_reset(CPUArchState *env)
-{
-    cpu_reset(env_cpu(env));
-}
-
 #endif /* TARGET_ARCH_CPU_H */
Signature · 1 line
2.43.0
Reviewed-by on v2 1/10
#3 lore raw

Re: [PATCH v2 01/10] bsd-user: drop not longer used target_reset_cpu()

In reply to Igor Mammedov
Quoted 7 lines · Igor Mammedov

target_reset_cpu() static inlines have no user,
remove them.

---
CC: Warner Losh <[email protected]>
CC: Kyle Evans <[email protected]>

Quoted 8 lines
---
 bsd-user/aarch64/target_arch_cpu.h | 5 -----
 bsd-user/arm/target_arch_cpu.h     | 4 ----
 bsd-user/i386/target_arch_cpu.h    | 5 -----
 bsd-user/riscv/target_arch_cpu.h   | 4 ----
 bsd-user/x86_64/target_arch_cpu.h  | 5 -----
 5 files changed, 23 deletions(-)

Warner

Quoted 82 lines
bsd-user/aarch64/target_arch_cpu.h +0 -5
index 87fbf6d677..46a448e93f 100644
@@ -181,9 +181,4 @@ static inline void target_cpu_clone_regs(CPUARMState
 *env, target_ulong newsp)
     pstate_write(env, 0);
 }
 
-static inline void target_cpu_reset(CPUArchState *env)
-{
-}
-
-

#endif /* TARGET_ARCH_CPU_H */

bsd-user/arm/target_arch_cpu.h +0 -4
index bc2eaa0bf4..b9583b0f92 100644
@@ -206,8 +206,4 @@ static inline void target_cpu_clone_regs(CPUARMState
 *env, target_ulong newsp)
     env->regs[0] = 0;
 }
 
-static inline void target_cpu_reset(CPUArchState *env)
-{
-}
-

#endif /* TARGET_ARCH_CPU_H */

bsd-user/i386/target_arch_cpu.h +0 -5
index 5d4c931dec..371e702799 100644
@@ -194,9 +194,4 @@ static inline void target_cpu_clone_regs(CPUX86State
 *env, target_ulong newsp)
     env->regs[R_EAX] = 0;
 }
 
-static inline void target_cpu_reset(CPUArchState *env)
-{
-    cpu_reset(env_cpu(env));
-}
-

#endif /* TARGET_ARCH_CPU_H */

bsd-user/riscv/target_arch_cpu.h +0 -4
index ef92f00480..d3cc5adbf4 100644
@@ -141,8 +141,4 @@ static inline void target_cpu_clone_regs(CPURISCVState
 *env, target_ulong newsp)
     env->gpr[xT0] = 0;
 }
 
-static inline void target_cpu_reset(CPUArchState *env)
-{
-}
-

#endif /* TARGET_ARCH_CPU_H */

bsd-user/x86_64/target_arch_cpu.h +0 -5
index f82042e30a..8ec5c65fab 100644
@@ -169,9 +169,4 @@ static inline void target_cpu_clone_regs(CPUX86State
 *env, target_ulong newsp)
     env->regs[R_EAX] = 0;
 }
 
-static inline void target_cpu_reset(CPUArchState *env)
-{
-    cpu_reset(env_cpu(env));
-}
-

#endif /* TARGET_ARCH_CPU_H */
--
2.43.0

#4 lore raw

PATCH v2 2/10 loongarch: reset vcpu after it's created

In reply to Igor Mammedov

Reseting vcpu before its thread is created, caused various issues in the past for other targets. It doesn't cause issues for loongarch at the moment but to be consistent with the rest of targets, move reset during realize time after qemu_init_vcpu().

That basically prevents reset being run when when vCPU is in incositent state (i.e. accelerator hasn't initialized vCPU yet).

---
CC: Song Gao <[email protected]>

---
 target/loongarch/cpu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
target/loongarch/cpu.c +1 -1
index e91f4a5239..15018d43ae 100644
@@ -634,8 +634,8 @@ static void loongarch_cpu_realizefn(DeviceState *dev, Error **errp)
 
     loongarch_cpu_register_gdb_regs_for_features(cs);
 
-    cpu_reset(cs);
     qemu_init_vcpu(cs);
+    cpu_reset(cs);
 
     lacc->parent_realize(dev, errp);
 }
Signature · 1 line
2.43.0
Reviewed-by on v2 2/10
#5 lore raw

Re: [PATCH v2 02/10] loongarch: reset vcpu after it's created

Quoted 21 lines · Igor Mammedov

Reseting vcpu before its thread is created, caused various issues in the past for other targets. It doesn't cause issues for loongarch at the moment but to be consistent with the rest of targets, move reset during realize time after qemu_init_vcpu().

That basically prevents reset being run when when vCPU is in incositent state (i.e. accelerator hasn't initialized vCPU yet).

---
CC: Song Gao <[email protected]>

---
  target/loongarch/cpu.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
target/loongarch/cpu.c +0 -0
index e91f4a5239..15018d43ae 100644

target/loongarch/cpu.c static void loongarch_cpu_realizefn(DeviceState *dev, Error **errp) · line 641

Quoted 9 lines
target/loongarch/cpu.c +1 -1
@@ -634,8 +634,8 @@ static void loongarch_cpu_realizefn(DeviceState *dev, Error **errp)
 
      loongarch_cpu_register_gdb_regs_for_features(cs);
 
-    cpu_reset(cs);
      qemu_init_vcpu(cs);
+    cpu_reset(cs);
 
      lacc->parent_realize(dev, errp);
  }

yes, that is actually a problem in kvm mode. vCPU thread should be created and kvm_fd for the vCPU need valid before cpu_reset().

Reviewed-by on v2 2/10
#6 lore raw

Re: [PATCH v2 02/10] loongarch: reset vcpu after it's created

In reply to Igor Mammedov
Quoted 6 lines · Igor Mammedov

Reseting vcpu before its thread is created, caused various issues in the past for other targets. It doesn't cause issues for loongarch at the moment but to be consistent with the rest of targets, move reset during realize time after qemu_init_vcpu().

That basically prevents reset being run when when vCPU is in incositent state

"inconsistent"

Quoted 3 lines

(i.e. accelerator hasn't initialized vCPU yet).

Quoted 20 lines

---
CC: Song Gao <[email protected]>

---
  target/loongarch/cpu.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
target/loongarch/cpu.c +1 -1
index e91f4a5239..15018d43ae 100644
@@ -634,8 +634,8 @@ static void loongarch_cpu_realizefn(DeviceState *dev, Error **errp)
 
      loongarch_cpu_register_gdb_regs_for_features(cs);
 
-    cpu_reset(cs);
      qemu_init_vcpu(cs);
+    cpu_reset(cs);
 
      lacc->parent_realize(dev, errp);
  }
#7 lore raw

PATCH v2 3/10 m68k: reset vcpu after it's created

In reply to Igor Mammedov

Reseting vcpu before its thread is created, caused various issues in the past for other targets. It doesn't cause issues for m68k at the moment but to be consistent with the rest of targets, move reset during realize time after qemu_init_vcpu().

That basically prevents reset being run when when vCPU is in incositent state (i.e. accelerator hasn't initialized vCPU yet).

---
CC: Laurent Vivier <[email protected]>

---
 target/m68k/cpu.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
target/m68k/cpu.c +1 -1
index 41dfdf5804..3fd2663fb0 100644
@@ -334,8 +334,8 @@ static void m68k_cpu_realizefn(DeviceState *dev, Error **errp)
 
     m68k_cpu_init_gdb(cpu);
 
-    cpu_reset(cs);
     qemu_init_vcpu(cs);
+    cpu_reset(cs);
 
     mcc->parent_realize(dev, errp);
 }
Signature · 1 line
2.43.0
Reviewed-by on v2 3/10
#8 lore raw

Re: [PATCH v2 03/10] m68k: reset vcpu after it's created

Quoted 9 lines · Igor Mammedov

Reseting vcpu before its thread is created, caused various issues in the past for other targets. It doesn't cause issues for m68k at the moment but to be consistent with the rest of targets, move reset during realize time after qemu_init_vcpu().

That basically prevents reset being run when when vCPU is in incositent state (i.e. accelerator hasn't initialized vCPU yet).

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
Reviewed-by on v2 3/10
#9 lore raw

Re: [PATCH v2 03/10] m68k: reset vcpu after it's created

In reply to Igor Mammedov
Quoted 6 lines · Igor Mammedov

Reseting vcpu before its thread is created, caused various issues in the past for other targets. It doesn't cause issues for m68k at the moment but to be consistent with the rest of targets, move reset during realize time after qemu_init_vcpu().

That basically prevents reset being run when when vCPU is in incositent state

(typo)

Quoted 3 lines

(i.e. accelerator hasn't initialized vCPU yet).

Quoted 5 lines

---
CC: Laurent Vivier <[email protected]>

---
  target/m68k/cpu.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
#10 lore raw

PATCH v2 4/10 tcg:tlb: use tcg_debug_assert() in assert_cpu_is_self()

In reply to Igor Mammedov

that will enable assert_cpu_is_self when QEMU is configured with
--enable-debug
without need for manual patching DEBUG_TLB_GATE define.

Need to manually path DEBUG_TLB_GATE define to enable assert, let regression caused by [1] creep in unnoticed.

1) 30933c4fb4f3d ("tcg/cputlb: remove other-cpu capability from TLB flushing")

---
v2: revert DEBUG_TLB_GATE/DEBUG_TLB_LOG_GATE to 0 as it used to be

---
 accel/tcg/cputlb.c | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)
accel/tcg/cputlb.c +2 -5
index b4ccf0cdcb..7380b29da3 100644
@@ -74,11 +74,8 @@
     } \
 } while (0)
 
-#define assert_cpu_is_self(cpu) do {                              \
-        if (DEBUG_TLB_GATE) {                                     \
-            g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));   \
-        }                                                         \
-    } while (0)
+#define assert_cpu_is_self(cpu)                             \
+    tcg_debug_assert(!(cpu)->created || qemu_cpu_is_self(cpu))
 
 /* run_on_cpu_data.target_ptr should always be big enough for a
  * vaddr even on 32 bit builds
Signature · 1 line
2.43.0
Reviewed-by on v2 4/10
#11 lore raw

Re: [PATCH v2 04/10] tcg:tlb: use tcg_debug_assert() in assert_cpu_is_self()

Quoted 10 lines · Igor Mammedov

that will enable assert_cpu_is_self when QEMU is configured with
--enable-debug
without need for manual patching DEBUG_TLB_GATE define.

Need to manually path DEBUG_TLB_GATE define to enable assert, let regression caused by [1] creep in unnoticed.

1) 30933c4fb4f3d ("tcg/cputlb: remove other-cpu capability from TLB flushing")

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#12 lore raw

PATCH v2 5/10 Revert "tcg/cputlb: remove other-cpu capability from TLB flushing"

In reply to Igor Mammedov

1)
This reverts commit 30933c4fb4f3df95ae44c4c3c86a5df049852c01.
("tcg/cputlb: remove other-cpu capability from TLB flushing")

The commit caused a regression which went unnoticed due to affected being disabled by default (DEBUG_TLB_GATE 0) Previous patch switched to using tcg_debug_assert() so that at least on debug builds assert_cpu_is_self() path would be exercised.

And that lead to exposing regression introduced by [1] with abort during tests. to reproduce:

  $ configure  --target-list=x86_64-softmmu --enable-debug
  $ make && ./qemu-system-x86_64

accel/tcg/cputlb.c:419: tlb_flush_by_mmuidx:

    Assertion `!(cpu)->created || qemu_cpu_is_self(cpu)' failed.

which is triggered by usage outside of cpu thread:

    x86_cpu_new -> ... ->
      x86_cpu_realizefn -> cpu_reset -> ... ->
          tcg_cpu_reset_hold

Drop offending commit for now, until a propper fix that doesn't break 'make check' is available.

PS:
fixup g_memdup() checkpatch error s/g_memdup/g_memdup2/

---
I'll leave it upto TCG folz to fix it up propperly.

---
 accel/tcg/cputlb.c | 42 +++++++++++++++++++++++++++++++++---------
 1 file changed, 33 insertions(+), 9 deletions(-)
accel/tcg/cputlb.c +33 -9
index 7380b29da3..3d1d7d2409 100644
@@ -416,9 +416,12 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
 {
     tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    assert_cpu_is_self(cpu);
-
-    tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
+    if (cpu->created && !qemu_cpu_is_self(cpu)) {
+        async_run_on_cpu(cpu, tlb_flush_by_mmuidx_async_work,
+                         RUN_ON_CPU_HOST_INT(idxmap));
+    } else {
+        tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
+    }
 }
 
 void tlb_flush(CPUState *cpu)
@@ -607,12 +610,28 @@ void tlb_flush_page_by_mmuidx(CPUState *cpu, vaddr addr, uint16_t idxmap)
 {
     tlb_debug("addr: %016" VADDR_PRIx " mmu_idx:%" PRIx16 "\n", addr, idxmap);
 
-    assert_cpu_is_self(cpu);
-
     /* This should already be page aligned */
     addr &= TARGET_PAGE_MASK;
 
-    tlb_flush_page_by_mmuidx_async_0(cpu, addr, idxmap);
+    if (qemu_cpu_is_self(cpu)) {
+        tlb_flush_page_by_mmuidx_async_0(cpu, addr, idxmap);
+    } else if (idxmap < TARGET_PAGE_SIZE) {
+        /*
+         * Most targets have only a few mmu_idx.  In the case where
+         * we can stuff idxmap into the low TARGET_PAGE_BITS, avoid
+         * allocating memory for this operation.
+         */
+        async_run_on_cpu(cpu, tlb_flush_page_by_mmuidx_async_1,
+                         RUN_ON_CPU_TARGET_PTR(addr | idxmap));
+    } else {
+        TLBFlushPageByMMUIdxData *d = g_new(TLBFlushPageByMMUIdxData, 1);
+
+        /* Otherwise allocate a structure, freed by the worker.  */
+        d->addr = addr;
+        d->idxmap = idxmap;
+        async_run_on_cpu(cpu, tlb_flush_page_by_mmuidx_async_2,
+                         RUN_ON_CPU_HOST_PTR(d));
+    }
 }
 
 void tlb_flush_page(CPUState *cpu, vaddr addr)
@@ -775,8 +794,6 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr,
 {
     TLBFlushRangeData d;
 
-    assert_cpu_is_self(cpu);
-
     /*
      * If all bits are significant, and len is small,
      * this devolves to tlb_flush_page.
@@ -797,7 +814,14 @@ void tlb_flush_range_by_mmuidx(CPUState *cpu, vaddr addr,
     d.idxmap = idxmap;
     d.bits = bits;
 
-    tlb_flush_range_by_mmuidx_async_0(cpu, d);
+    if (qemu_cpu_is_self(cpu)) {
+        tlb_flush_range_by_mmuidx_async_0(cpu, d);
+    } else {
+        /* Otherwise allocate a structure, freed by the worker.  */
+        TLBFlushRangeData *p = g_memdup2(&d, sizeof(d));
+        async_run_on_cpu(cpu, tlb_flush_range_by_mmuidx_async_1,
+                         RUN_ON_CPU_HOST_PTR(p));
+    }
 }
 
 void tlb_flush_page_bits_by_mmuidx(CPUState *cpu, vaddr addr,
Signature · 1 line
2.43.0
#13 lore raw

Re: [PATCH v2 05/10] Revert "tcg/cputlb: remove other-cpu capability from TLB flushing"

Quoted 14 lines · Igor Mammedov

1)
This reverts commit 30933c4fb4f3df95ae44c4c3c86a5df049852c01.
("tcg/cputlb: remove other-cpu capability from TLB flushing")

The commit caused a regression which went unnoticed due to affected being disabled by default (DEBUG_TLB_GATE 0) Previous patch switched to using tcg_debug_assert() so that at least on debug builds assert_cpu_is_self() path would be exercised.

And that lead to exposing regression introduced by [1] with abort during tests. to reproduce:

  $ configure  --target-list=x86_64-softmmu --enable-debug
  $ make && ./qemu-system-x86_64
Quoted 7 lines

accel/tcg/cputlb.c:419: tlb_flush_by_mmuidx:

    Assertion `!(cpu)->created || qemu_cpu_is_self(cpu)' failed.

which is triggered by usage outside of cpu thread:

    x86_cpu_new -> ... ->
      x86_cpu_realizefn -> cpu_reset -> ... ->
          tcg_cpu_reset_hold

If the reset case is the only one I don't think we need to revert the rest of the changes as only tlb_flush needs calling. How about something like:

--8<---------------cut here---------------start------------->8---
cputlb: introduce tlb_flush_other_cpu for reset use

The commit 30933c4fb4 (tcg/cputlb: remove other-cpu capability from TLB flushing) introduced a regression that only shows up when --enable-debug-tcg is used. The main use case of tlb_flush outside of the current_cpu context is for handling reset and CPU creation. Rather than revert the commit introduce a new helper and tweak the documentation to make it clear where it should be used.

3 files changed, 26 insertions(+), 5 deletions(-)
include/exec/exec-all.h   | 20 ++++++++++++++++----
accel/tcg/cputlb.c        |  9 +++++++++
accel/tcg/tcg-accel-ops.c |  2 +-

modified include/exec/exec-all.h
@@ -64,12 +64,24 @@ void tlb_flush_page_all_cpus_synced(CPUState *src, vaddr addr);
* tlb_flush:
* @cpu: CPU whose TLB should be flushed
*
- * Flush the entire TLB for the specified CPU. Most CPU architectures
- * allow the implementation to drop entries from the TLB at any time
- * so this is generally safe. If more selective flushing is required
- * use one of the other functions for efficiency.
+ * Flush the entire TLB for the specified current CPU.
+ *
+ * Most CPU architectures allow the implementation to drop entries
+ * from the TLB at any time so this is generally safe. If more
+ * selective flushing is required use one of the other functions for
+ * efficiency.
*/
void tlb_flush(CPUState *cpu);
+/**
+ * tlb_flush_other_cpu:
+ * @cpu: CPU whose TLB should be flushed
+ *
+ * Flush the entire TLB for a specified CPU. For cross vCPU flushes
+ * you shuld be using a more selective function. This is really only
+ * used for flushing CPUs being reset from outside their current
+ * context.
+ */
+void tlb_flush_other_cpu(CPUState *cpu);
/**
* tlb_flush_all_cpus_synced:
* @cpu: src CPU of the flush
modified accel/tcg/cputlb.c
@@ -414,6 +414,15 @@ void tlb_flush(CPUState *cpu)

     tlb_flush_by_mmuidx(cpu, ALL_MMUIDX_BITS);

}

+void tlb_flush_other_cpu(CPUState *cpu)
+{
+ g_assert(!qemu_cpu_is_self(cpu));
+
+ async_run_on_cpu(cpu,
+ tlb_flush_by_mmuidx_async_work,
+ RUN_ON_CPU_HOST_INT(ALL_MMUIDX_BITS));
+}
+
void tlb_flush_by_mmuidx_all_cpus_synced(CPUState *src_cpu, uint16_t idxmap)

 {
     const run_on_cpu_func fn = tlb_flush_by_mmuidx_async_work;
modified   accel/tcg/tcg-accel-ops.c
@@ -85,7 +85,7 @@ static void tcg_cpu_reset_hold(CPUState *cpu)
 {
     tcg_flush_jmp_cache(cpu);

- tlb_flush(cpu);
+ tlb_flush_other_cpu(cpu);
}

--8<---------------cut here---------------end--------------->8---

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#14 lore raw

Re: [PATCH v2 05/10] Revert "tcg/cputlb: remove other-cpu capability from TLB flushing"

Quoted 24 lines · On Tue, 25 Feb 2025 12:42:24 +0000 Alex Bennée <[email protected]> wrote:
Quoted 14 lines · Igor Mammedov

1)
This reverts commit 30933c4fb4f3df95ae44c4c3c86a5df049852c01.
("tcg/cputlb: remove other-cpu capability from TLB flushing")

The commit caused a regression which went unnoticed due to affected being disabled by default (DEBUG_TLB_GATE 0) Previous patch switched to using tcg_debug_assert() so that at least on debug builds assert_cpu_is_self() path would be exercised.

And that lead to exposing regression introduced by [1] with abort during tests. to reproduce:

  $ configure  --target-list=x86_64-softmmu --enable-debug
  $ make && ./qemu-system-x86_64
Quoted 7 lines

accel/tcg/cputlb.c:419: tlb_flush_by_mmuidx:

    Assertion `!(cpu)->created || qemu_cpu_is_self(cpu)' failed.

which is triggered by usage outside of cpu thread:

    x86_cpu_new -> ... ->
      x86_cpu_realizefn -> cpu_reset -> ... ->
          tcg_cpu_reset_hold  
Quoted 6 lines

If the reset case is the only one I don't think we need to revert the rest of the changes as only tlb_flush needs calling. How about something like:

--8<---------------cut here---------------start------------->8---
cputlb: introduce tlb_flush_other_cpu for reset use

while that works for my reproducer it's not sufficient, 'make check' is some tests fails anyway

ex:

10/378 qemu:qtest+qtest-x86_64 / qtest-x86_64/ahci-test ERROR 13.47s killed by signal 6 SIGABRT stderr:
qemu-system-x86_64: qemu/accel/tcg/cputlb.c:419: tlb_flush_by_mmuidx: Assertion `qemu_cpu_is_self(cpu)' failed. Broken pipe
qemu/tests/qtest/libqtest.c:208: kill_qemu() detected QEMU death from signal 6 (Aborted) (core dumped)

Quoted 73 lines

The commit 30933c4fb4 (tcg/cputlb: remove other-cpu capability from TLB flushing) introduced a regression that only shows up when --enable-debug-tcg is used. The main use case of tlb_flush outside of the current_cpu context is for handling reset and CPU creation. Rather than revert the commit introduce a new helper and tweak the documentation to make it clear where it should be used.

3 files changed, 26 insertions(+), 5 deletions(-)
include/exec/exec-all.h   | 20 ++++++++++++++++----
accel/tcg/cputlb.c        |  9 +++++++++
accel/tcg/tcg-accel-ops.c |  2 +-

modified include/exec/exec-all.h

(quoted diff) +16 -4
@@ -64,12 +64,24 @@ void tlb_flush_page_all_cpus_synced(CPUState *src, vaddr addr);
  * tlb_flush:
  * @cpu: CPU whose TLB should be flushed
  *
- * Flush the entire TLB for the specified CPU. Most CPU architectures
- * allow the implementation to drop entries from the TLB at any time
- * so this is generally safe. If more selective flushing is required
- * use one of the other functions for efficiency.
+ * Flush the entire TLB for the specified current CPU.
+ *
+ * Most CPU architectures allow the implementation to drop entries
+ * from the TLB at any time so this is generally safe. If more
+ * selective flushing is required use one of the other functions for
+ * efficiency.
  */
 void tlb_flush(CPUState *cpu);
+/**
+ * tlb_flush_other_cpu:
+ * @cpu: CPU whose TLB should be flushed
+ *
+ * Flush the entire TLB for a specified CPU. For cross vCPU flushes
+ * you shuld be using a more selective function. This is really only
+ * used for flushing CPUs being reset from outside their current
+ * context.
+ */
+void tlb_flush_other_cpu(CPUState *cpu);
 /**
  * tlb_flush_all_cpus_synced:
  * @cpu: src CPU of the flush

modified accel/tcg/cputlb.c

(quoted diff) +9 -0
@@ -414,6 +414,15 @@ void tlb_flush(CPUState *cpu)
     tlb_flush_by_mmuidx(cpu, ALL_MMUIDX_BITS);
 }
 
+void tlb_flush_other_cpu(CPUState *cpu)
+{
+    g_assert(!qemu_cpu_is_self(cpu));
+
+    async_run_on_cpu(cpu,
+                     tlb_flush_by_mmuidx_async_work,
+                     RUN_ON_CPU_HOST_INT(ALL_MMUIDX_BITS));
+}
+
 void tlb_flush_by_mmuidx_all_cpus_synced(CPUState *src_cpu, uint16_t idxmap)
 {
     const run_on_cpu_func fn = tlb_flush_by_mmuidx_async_work;

modified accel/tcg/tcg-accel-ops.c

(quoted diff) +1 -2
@@ -85,7 +85,7 @@ static void tcg_cpu_reset_hold(CPUState *cpu)
 {
     tcg_flush_jmp_cache(cpu);
 
-    tlb_flush(cpu);
+    tlb_flush_other_cpu(cpu);
 }
 
--8<---------------cut here---------------end--------------->8---
#15 lore raw

Re: [PATCH v2 05/10] Revert "tcg/cputlb: remove other-cpu capability from TLB flushing"

Quoted 37 lines · On Tue, 25 Feb 2025 18:19:03 +0100 Igor Mammedov <[email protected]> wrote:
Quoted 24 lines · On Tue, 25 Feb 2025 12:42:24 +0000 Alex Bennée <[email protected]> wrote:
Quoted 14 lines · Igor Mammedov

1)
This reverts commit 30933c4fb4f3df95ae44c4c3c86a5df049852c01.
("tcg/cputlb: remove other-cpu capability from TLB flushing")

The commit caused a regression which went unnoticed due to affected being disabled by default (DEBUG_TLB_GATE 0) Previous patch switched to using tcg_debug_assert() so that at least on debug builds assert_cpu_is_self() path would be exercised.

And that lead to exposing regression introduced by [1] with abort during tests. to reproduce:

  $ configure  --target-list=x86_64-softmmu --enable-debug
  $ make && ./qemu-system-x86_64
Quoted 7 lines

accel/tcg/cputlb.c:419: tlb_flush_by_mmuidx:

    Assertion `!(cpu)->created || qemu_cpu_is_self(cpu)' failed.

which is triggered by usage outside of cpu thread:

    x86_cpu_new -> ... ->
      x86_cpu_realizefn -> cpu_reset -> ... ->
          tcg_cpu_reset_hold    
Quoted 6 lines

If the reset case is the only one I don't think we need to revert the rest of the changes as only tlb_flush needs calling. How about something like:

--8<---------------cut here---------------start------------->8---
cputlb: introduce tlb_flush_other_cpu for reset use

while that works for my reproducer it's not sufficient, 'make check' is some tests fails anyway

Quoted 7 lines

ex:

10/378 qemu:qtest+qtest-x86_64 / qtest-x86_64/ahci-test ERROR 13.47s killed by signal 6 SIGABRT stderr:
qemu-system-x86_64: qemu/accel/tcg/cputlb.c:419: tlb_flush_by_mmuidx: Assertion `qemu_cpu_is_self(cpu)' failed. Broken pipe
qemu/tests/qtest/libqtest.c:208: kill_qemu() detected QEMU death from signal 6 (Aborted) (core dumped)

here is v3 rebased on top of the patch

https://gitlab.com/imammedo/qemu/-/commits/qemu_cpu_cond_v3?ref_type=heads

it seems that reset path is not the only one that relied on 'cpu->created' workaround

Quoted 77 lines
Quoted 76 lines

The commit 30933c4fb4 (tcg/cputlb: remove other-cpu capability from TLB flushing) introduced a regression that only shows up when --enable-debug-tcg is used. The main use case of tlb_flush outside of the current_cpu context is for handling reset and CPU creation. Rather than revert the commit introduce a new helper and tweak the documentation to make it clear where it should be used.

3 files changed, 26 insertions(+), 5 deletions(-)
include/exec/exec-all.h   | 20 ++++++++++++++++----
accel/tcg/cputlb.c        |  9 +++++++++
accel/tcg/tcg-accel-ops.c |  2 +-

modified include/exec/exec-all.h

(quoted diff) +16 -4
@@ -64,12 +64,24 @@ void tlb_flush_page_all_cpus_synced(CPUState *src, vaddr addr);
  * tlb_flush:
  * @cpu: CPU whose TLB should be flushed
  *
- * Flush the entire TLB for the specified CPU. Most CPU architectures
- * allow the implementation to drop entries from the TLB at any time
- * so this is generally safe. If more selective flushing is required
- * use one of the other functions for efficiency.
+ * Flush the entire TLB for the specified current CPU.
+ *
+ * Most CPU architectures allow the implementation to drop entries
+ * from the TLB at any time so this is generally safe. If more
+ * selective flushing is required use one of the other functions for
+ * efficiency.
  */
 void tlb_flush(CPUState *cpu);
+/**
+ * tlb_flush_other_cpu:
+ * @cpu: CPU whose TLB should be flushed
+ *
+ * Flush the entire TLB for a specified CPU. For cross vCPU flushes
+ * you shuld be using a more selective function. This is really only
+ * used for flushing CPUs being reset from outside their current
+ * context.
+ */
+void tlb_flush_other_cpu(CPUState *cpu);
 /**
  * tlb_flush_all_cpus_synced:
  * @cpu: src CPU of the flush

modified accel/tcg/cputlb.c

(quoted diff) +9 -0
@@ -414,6 +414,15 @@ void tlb_flush(CPUState *cpu)
     tlb_flush_by_mmuidx(cpu, ALL_MMUIDX_BITS);
 }
 
+void tlb_flush_other_cpu(CPUState *cpu)
+{
+    g_assert(!qemu_cpu_is_self(cpu));
+
+    async_run_on_cpu(cpu,
+                     tlb_flush_by_mmuidx_async_work,
+                     RUN_ON_CPU_HOST_INT(ALL_MMUIDX_BITS));
+}
+
 void tlb_flush_by_mmuidx_all_cpus_synced(CPUState *src_cpu, uint16_t idxmap)
 {
     const run_on_cpu_func fn = tlb_flush_by_mmuidx_async_work;

modified accel/tcg/tcg-accel-ops.c

(quoted diff) +1 -2
@@ -85,7 +85,7 @@ static void tcg_cpu_reset_hold(CPUState *cpu)
 {
     tcg_flush_jmp_cache(cpu);
 
-    tlb_flush(cpu);
+    tlb_flush_other_cpu(cpu);
 }
 
--8<---------------cut here---------------end--------------->8---
#16 lore raw

PATCH v2 6/10 tcg: drop cpu->created check

In reply to Igor Mammedov

previous commits fixed 2 remaining cases where vcpu might have had 'cpu->created == false' during 1st vcpu reset (at realize time) that leads to call chain

      tcg_cpu_reset_hold() => tlb_flush_by_mmuidx()

remove not need anymore check, with cpu->created always being true.

---
CC: Nicholas Piggin <[email protected]>

---
 accel/tcg/cputlb.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
accel/tcg/cputlb.c +2 -2
index 3d1d7d2409..6ccb173960 100644
@@ -75,7 +75,7 @@
 } while (0)
 
 #define assert_cpu_is_self(cpu)                             \
-    tcg_debug_assert(!(cpu)->created || qemu_cpu_is_self(cpu))
+    tcg_debug_assert(qemu_cpu_is_self(cpu))
 
 /* run_on_cpu_data.target_ptr should always be big enough for a
  * vaddr even on 32 bit builds
@@ -416,7 +416,7 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
 {
     tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    if (cpu->created && !qemu_cpu_is_self(cpu)) {
+    if (!qemu_cpu_is_self(cpu)) {
         async_run_on_cpu(cpu, tlb_flush_by_mmuidx_async_work,
                          RUN_ON_CPU_HOST_INT(idxmap));
     } else {
Signature · 1 line
2.43.0
#17 lore raw

PATCH v2 7/10 accel/tcg: Simplify use of &first_cpu in rr_cpu_thread_fn()

In reply to Igor Mammedov

From: Philippe Mathieu-Daudé <[email protected]>

Let vCPUs wait for themselves being ready first, then other ones. This allows the first thread to starts without the global vcpu queue (thus &first_cpu) being populated.

---
 accel/tcg/tcg-accel-ops-rr.c | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)
accel/tcg/tcg-accel-ops-rr.c +8 -5
index 028b385af9..d9eadd5ec4 100644
@@ -197,16 +197,19 @@ static void *rr_cpu_thread_fn(void *arg)
     qemu_guest_random_seed_thread_part2(cpu->random_seed);
 
     /* wait for initial kick-off after machine start */
-    while (first_cpu->stopped) {
-        qemu_cond_wait_bql(first_cpu->halt_cond);
+    while (cpu->stopped) {
+        CPUState *iter_cpu;
+
+        qemu_cond_wait_bql(cpu->halt_cond);
 
         /* process any pending work */
-        CPU_FOREACH(cpu) {
-            current_cpu = cpu;
-            qemu_wait_io_event_common(cpu);
+        CPU_FOREACH(iter_cpu) {
+            current_cpu = iter_cpu;
+            qemu_wait_io_event_common(iter_cpu);
         }
     }
 
+    g_assert(first_cpu);
     rr_start_kick_timer();
 
     cpu = first_cpu;
Signature · 1 line
2.43.0
Acked-by on v2 7/10
#18 lore raw

Re: [PATCH v2 07/10] accel/tcg: Simplify use of &first_cpu in rr_cpu_thread_fn()

Quoted 8 lines · Igor Mammedov

From: Philippe Mathieu-Daudé <[email protected]>

Let vCPUs wait for themselves being ready first, then other ones. This allows the first thread to starts without the global vcpu queue (thus &first_cpu) being populated.

Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#19 lore raw

PATCH v2 8/10 cpus: expose only realized vCPUs to global &cpus_queue

In reply to Igor Mammedov

cpu_list_add() was doing 2 distinct things:
- assign some index to vCPU
- add unrealized (thus in inconsistent state) vCPU to &cpus_queue

Code using CPU_FOREACH() macro would iterate over possibly unrealized vCPUs, often dealt with special casing.

Instead of working around of vCPU existence in cpus_queue, split out cpu_index assignment from cpu_list_add(), and move the later to the end of realize stage, right before vCPU is let run.

---
CC: Yanan Wang <[email protected]>
CC: Zhao Liu <[email protected]>

---
 include/hw/core/cpu.h |  6 ++++++
 cpu-common.c          | 23 ++++++++++++++---------
 cpu-target.c          |  2 +-
 hw/core/cpu-common.c  |  2 ++
 4 files changed, 23 insertions(+), 10 deletions(-)
include/hw/core/cpu.h +6 -0
index fb397cdfc5..c338fd31bd 100644
@@ -750,6 +750,12 @@ bool cpu_virtio_is_big_endian(CPUState *cpu);
 
 #endif /* CONFIG_USER_ONLY */
 
+/**
+ * cpu_auto_assign_cpu_index:
+ * @cpu: The CPU to be assigned a cpu_index
+ */
+void cpu_auto_assign_cpu_index(CPUState *cpu);
+
 /**
  * cpu_list_add:
  * @cpu: The CPU to be added to the list of CPUs.
cpu-common.c +14 -9
index 4248b2d727..92f3d00e56 100644
@@ -71,15 +71,7 @@ int cpu_get_free_index(void)
     return max_cpu_index;
 }
 
-CPUTailQ cpus_queue = QTAILQ_HEAD_INITIALIZER(cpus_queue);
-static unsigned int cpu_list_generation_id;
-
-unsigned int cpu_list_generation_id_get(void)
-{
-    return cpu_list_generation_id;
-}
-
-void cpu_list_add(CPUState *cpu)
+void cpu_auto_assign_cpu_index(CPUState *cpu)
 {
     static bool cpu_index_auto_assigned;
 
@@ -91,6 +83,19 @@ void cpu_list_add(CPUState *cpu)
     } else {
         assert(!cpu_index_auto_assigned);
     }
+}
+
+CPUTailQ cpus_queue = QTAILQ_HEAD_INITIALIZER(cpus_queue);
+static unsigned int cpu_list_generation_id;
+
+unsigned int cpu_list_generation_id_get(void)
+{
+    return cpu_list_generation_id;
+}
+
+void cpu_list_add(CPUState *cpu)
+{
+    QEMU_LOCK_GUARD(&qemu_cpu_list_lock);
     QTAILQ_INSERT_TAIL_RCU(&cpus_queue, cpu, node);
     cpu_list_generation_id++;
 }
cpu-target.c +1 -1
index 667688332c..0c86c18a50 100644
@@ -142,7 +142,7 @@ bool cpu_exec_realizefn(CPUState *cpu, Error **errp)
     }
 
     /* Wait until cpu initialization complete before exposing cpu. */
-    cpu_list_add(cpu);
+    cpu_auto_assign_cpu_index(cpu);
 
 #ifdef CONFIG_USER_ONLY
     assert(qdev_get_vmsd(DEVICE(cpu)) == NULL ||
hw/core/cpu-common.c +2 -0
index cb79566cc5..c29737e5e3 100644
@@ -211,6 +211,8 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
         }
     }
 
+    cpu_list_add(cpu);
+
     if (dev->hotplugged) {
         cpu_synchronize_post_init(cpu);
         cpu_resume(cpu);
Signature · 1 line
2.43.0
#20 lore raw

Re: [PATCH v2 08/10] cpus: expose only realized vCPUs to global &cpus_queue

Quoted 9 lines · Igor Mammedov

cpu_list_add() was doing 2 distinct things:
- assign some index to vCPU
- add unrealized (thus in inconsistent state) vCPU to &cpus_queue

Code using CPU_FOREACH() macro would iterate over possibly unrealized vCPUs, often dealt with special casing.

Instead of working around of vCPU existence in cpus_queue, split out cpu_index assignment from cpu_list_add(),

Better split 2 distinct changes in 2 patches for clarity.

Quoted 98 lines

and move the later to the end of realize stage, right before vCPU is let run.

---
CC: Yanan Wang <[email protected]>
CC: Zhao Liu <[email protected]>

---
  include/hw/core/cpu.h |  6 ++++++
  cpu-common.c          | 23 ++++++++++++++---------
  cpu-target.c          |  2 +-
  hw/core/cpu-common.c  |  2 ++
  4 files changed, 23 insertions(+), 10 deletions(-)
include/hw/core/cpu.h +6 -0
index fb397cdfc5..c338fd31bd 100644
@@ -750,6 +750,12 @@ bool cpu_virtio_is_big_endian(CPUState *cpu);
 
  #endif /* CONFIG_USER_ONLY */
 
+/**
+ * cpu_auto_assign_cpu_index:
+ * @cpu: The CPU to be assigned a cpu_index
+ */
+void cpu_auto_assign_cpu_index(CPUState *cpu);
+
  /**
   * cpu_list_add:
   * @cpu: The CPU to be added to the list of CPUs.
cpu-common.c +14 -9
index 4248b2d727..92f3d00e56 100644
@@ -71,15 +71,7 @@ int cpu_get_free_index(void)
      return max_cpu_index;
  }
 
-CPUTailQ cpus_queue = QTAILQ_HEAD_INITIALIZER(cpus_queue);
-static unsigned int cpu_list_generation_id;
-
-unsigned int cpu_list_generation_id_get(void)
-{
-    return cpu_list_generation_id;
-}
-
-void cpu_list_add(CPUState *cpu)
+void cpu_auto_assign_cpu_index(CPUState *cpu)
  {
      static bool cpu_index_auto_assigned;
 
@@ -91,6 +83,19 @@ void cpu_list_add(CPUState *cpu)
      } else {
          assert(!cpu_index_auto_assigned);
      }
+}
+
+CPUTailQ cpus_queue = QTAILQ_HEAD_INITIALIZER(cpus_queue);
+static unsigned int cpu_list_generation_id;
+
+unsigned int cpu_list_generation_id_get(void)
+{
+    return cpu_list_generation_id;
+}
+
+void cpu_list_add(CPUState *cpu)
+{
+    QEMU_LOCK_GUARD(&qemu_cpu_list_lock);
      QTAILQ_INSERT_TAIL_RCU(&cpus_queue, cpu, node);
      cpu_list_generation_id++;
  }
cpu-target.c +1 -1
index 667688332c..0c86c18a50 100644
@@ -142,7 +142,7 @@ bool cpu_exec_realizefn(CPUState *cpu, Error **errp)
      }
 
      /* Wait until cpu initialization complete before exposing cpu. */
-    cpu_list_add(cpu);
+    cpu_auto_assign_cpu_index(cpu);
 
  #ifdef CONFIG_USER_ONLY
      assert(qdev_get_vmsd(DEVICE(cpu)) == NULL ||
hw/core/cpu-common.c +2 -0
index cb79566cc5..c29737e5e3 100644
@@ -211,6 +211,8 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
          }
      }
 
+    cpu_list_add(cpu);
+
      if (dev->hotplugged) {
          cpu_synchronize_post_init(cpu);
          cpu_resume(cpu);
#21 lore raw

Re: [PATCH v2 08/10] cpus: expose only realized vCPUs to global &cpus_queue

Quoted 5 lines · On Wed, 26 Feb 2025 08:16:52 +0100 Philippe Mathieu-Daudé <[email protected]>…
Quoted 4 lines · Igor Mammedov

cpu_list_add() was doing 2 distinct things:
- assign some index to vCPU
- add unrealized (thus in inconsistent state) vCPU to &cpus_queue

Quoted 7 lines
Quoted 5 lines

Code using CPU_FOREACH() macro would iterate over possibly unrealized vCPUs, often dealt with special casing.

Instead of working around of vCPU existence in cpus_queue, split out cpu_index assignment from cpu_list_add(),

Better split 2 distinct changes in 2 patches for clarity.

Will do it later, once folks decide how to fix broken TCG reset path.

do you mean:
#1 - introduce cpu_auto_assign_cpu_index()
#2 - move cpu_list_add() to later stage but keep cpu_auto_assign_cpu_index()
where it's now?

Quoted 100 lines
Quoted 99 lines

and move the later to the end of realize stage, right before vCPU is let run.

---
CC: Yanan Wang <[email protected]>
CC: Zhao Liu <[email protected]>

---
  include/hw/core/cpu.h |  6 ++++++
  cpu-common.c          | 23 ++++++++++++++---------
  cpu-target.c          |  2 +-
  hw/core/cpu-common.c  |  2 ++
  4 files changed, 23 insertions(+), 10 deletions(-)
include/hw/core/cpu.h +6 -0
index fb397cdfc5..c338fd31bd 100644
@@ -750,6 +750,12 @@ bool cpu_virtio_is_big_endian(CPUState *cpu);
 
  #endif /* CONFIG_USER_ONLY */
 
+/**
+ * cpu_auto_assign_cpu_index:
+ * @cpu: The CPU to be assigned a cpu_index
+ */
+void cpu_auto_assign_cpu_index(CPUState *cpu);
+
  /**
   * cpu_list_add:
   * @cpu: The CPU to be added to the list of CPUs.
cpu-common.c +14 -9
index 4248b2d727..92f3d00e56 100644
@@ -71,15 +71,7 @@ int cpu_get_free_index(void)
      return max_cpu_index;
  }
 
-CPUTailQ cpus_queue = QTAILQ_HEAD_INITIALIZER(cpus_queue);
-static unsigned int cpu_list_generation_id;
-
-unsigned int cpu_list_generation_id_get(void)
-{
-    return cpu_list_generation_id;
-}
-
-void cpu_list_add(CPUState *cpu)
+void cpu_auto_assign_cpu_index(CPUState *cpu)
  {
      static bool cpu_index_auto_assigned;
 
@@ -91,6 +83,19 @@ void cpu_list_add(CPUState *cpu)
      } else {
          assert(!cpu_index_auto_assigned);
      }
+}
+
+CPUTailQ cpus_queue = QTAILQ_HEAD_INITIALIZER(cpus_queue);
+static unsigned int cpu_list_generation_id;
+
+unsigned int cpu_list_generation_id_get(void)
+{
+    return cpu_list_generation_id;
+}
+
+void cpu_list_add(CPUState *cpu)
+{
+    QEMU_LOCK_GUARD(&qemu_cpu_list_lock);
      QTAILQ_INSERT_TAIL_RCU(&cpus_queue, cpu, node);
      cpu_list_generation_id++;
  }
cpu-target.c +1 -1
index 667688332c..0c86c18a50 100644
@@ -142,7 +142,7 @@ bool cpu_exec_realizefn(CPUState *cpu, Error **errp)
      }
 
      /* Wait until cpu initialization complete before exposing cpu. */
-    cpu_list_add(cpu);
+    cpu_auto_assign_cpu_index(cpu);
 
  #ifdef CONFIG_USER_ONLY
      assert(qdev_get_vmsd(DEVICE(cpu)) == NULL ||
hw/core/cpu-common.c +2 -0
index cb79566cc5..c29737e5e3 100644
@@ -211,6 +211,8 @@ static void cpu_common_realizefn(DeviceState *dev, Error **errp)
          }
      }
 
+    cpu_list_add(cpu);
+
      if (dev->hotplugged) {
          cpu_synchronize_post_init(cpu);
          cpu_resume(cpu);  
#22 lore raw

Re: [PATCH v2 08/10] cpus: expose only realized vCPUs to global &cpus_queue

Quoted 17 lines · Igor Mammedov
Quoted 5 lines · On Wed, 26 Feb 2025 08:16:52 +0100 Philippe Mathieu-Daudé <[email protected]>…
Quoted 4 lines · Igor Mammedov

cpu_list_add() was doing 2 distinct things:
- assign some index to vCPU
- add unrealized (thus in inconsistent state) vCPU to &cpus_queue

Quoted 7 lines
Quoted 5 lines

Code using CPU_FOREACH() macro would iterate over possibly unrealized vCPUs, often dealt with special casing.

Instead of working around of vCPU existence in cpus_queue, split out cpu_index assignment from cpu_list_add(),

Better split 2 distinct changes in 2 patches for clarity.

Quoted 6 lines

Will do it later, once folks decide how to fix broken TCG reset path.

do you mean:
#1 - introduce cpu_auto_assign_cpu_index()
#2 - move cpu_list_add() to later stage but keep cpu_auto_assign_cpu_index()
where it's now?

Exactly, thanks!

#23 lore raw

PATCH v2 9/10 accel/kvm: Assert vCPU is created when calling kvm_dirty_ring_reap*()

In reply to Igor Mammedov

From: Philippe Mathieu-Daudé <[email protected]>

Previous commits made sure vCPUs are realized before accelerators (such KVM) use them. Ensure that by asserting the vCPU is created, no need to return.

For more context, see commit 56adee407fc ("kvm: dirty-ring: Fix race with vcpu creation").

---
CC: [email protected]
CC: [email protected]

---
 accel/kvm/kvm-all.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)
accel/kvm/kvm-all.c +3 -5
index c65b790433..cb56d120a9 100644
@@ -831,13 +831,11 @@ static uint32_t kvm_dirty_ring_reap_one(KVMState *s, CPUState *cpu)
     uint32_t count = 0, fetch = cpu->kvm_fetch_index;
 
     /*
-     * It's possible that we race with vcpu creation code where the vcpu is
+     * It's not possible that we race with vcpu creation code where the vcpu is
      * put onto the vcpus list but not yet initialized the dirty ring
-     * structures.  If so, skip it.
+     * structures.
      */
-    if (!cpu->created) {
-        return 0;
-    }
+    assert(cpu->created);
 
     assert(dirty_gfns && ring_size);
     trace_kvm_dirty_ring_reap_vcpu(cpu->cpu_index);
Signature · 1 line
2.43.0
#24 lore raw

PATCH v2 10/10 accel/kvm: Remove unreachable assertion in kvm_dirty_ring_reap*()

In reply to Igor Mammedov

From: Philippe Mathieu-Daudé <[email protected]>

Previous commit passed all our CI tests, this assertion being never triggered. Remove it as dead code.

---
CC: [email protected]

---
 accel/kvm/kvm-all.c | 7 -------
 1 file changed, 7 deletions(-)
accel/kvm/kvm-all.c +0 -7
index cb56d120a9..814b1a53eb 100644
@@ -830,13 +830,6 @@ static uint32_t kvm_dirty_ring_reap_one(KVMState *s, CPUState *cpu)
     uint32_t ring_size = s->kvm_dirty_ring_size;
     uint32_t count = 0, fetch = cpu->kvm_fetch_index;
 
-    /*
-     * It's not possible that we race with vcpu creation code where the vcpu is
-     * put onto the vcpus list but not yet initialized the dirty ring
-     * structures.
-     */
-    assert(cpu->created);
-
     assert(dirty_gfns && ring_size);
     trace_kvm_dirty_ring_reap_vcpu(cpu->cpu_index);
 
Signature · 1 line
2.43.0
#25 lore raw
In reply to Igor Mammedov
Quoted 21 lines · On Fri, 7 Feb 2025 17:20:38 +0100 Igor Mammedov <[email protected]> wrote:

Changelog:
* drop wire/unwire hooks patches
* drop unrealize related patches
* include fixed up patches from
[PATCH 0/6] tcg: fix qemu crash when add assert_cpu_is_self() is enabled
and cleanups related to cpu->created check
https://patchew.org/QEMU/[email protected]/
as it's related to the topic (well, modulo bsd cleanup)
* CI mostly green modulo rust failure on Ubuntu

     https://gitlab.com/imammedo/qemu/-/pipelines/1660855467

The goal of this series is to expose vCPUs in a stable state to the accelerators, in particular the QDev 'REALIZED' step.

To do this we split out cpu_index assignment into a separate step, and move call cpu_list_add() to the end of CPU realize stage.

I expect these changes to allow CPUState::cpu_index clarifications and simplifications, but this will be addressed (and commented) in a separate series.

Quoted 6 lines

As result, the series also
* fix regression intoroduced by
30933c4fb4f3d ("tcg/cputlb: remove other-cpu capability from TLB flushing")
for deatials see 'tcg:tlb: use tcg_debug_assert() in assert_cpu_is_self()'
* drops no longer needed workaround 'cpu->check' due to vCPU being exposed too early in cpus_queue.

Richard,
gentle ping,
can you pick it up is it looks reasonable.

Quoted 35 lines
CC Paolo Bonzini <[email protected]>
CC Richard Henderson <[email protected]>
CC "Philippe Mathieu-Daudé" <[email protected]>
CC Alex Bennée <[email protected]>

Igor Mammedov (7):
bsd-user: drop not longer used target_reset_cpu()
loongarch: reset vcpu after it's created
m68k: reset vcpu after it's created
tcg:tlb: use tcg_debug_assert() in assert_cpu_is_self() Revert "tcg/cputlb: remove other-cpu capability from TLB flushing"
tcg: drop cpu->created check
cpus: expose only realized vCPUs to global &cpus_queue

Philippe Mathieu-Daudé (3):
accel/tcg: Simplify use of &first_cpu in rr_cpu_thread_fn() accel/kvm: Assert vCPU is created when calling kvm_dirty_ring_reap*() accel/kvm: Remove unreachable assertion in kvm_dirty_ring_reap*()

 bsd-user/aarch64/target_arch_cpu.h |  5 ---
 bsd-user/arm/target_arch_cpu.h     |  4 ---
 bsd-user/i386/target_arch_cpu.h    |  5 ---
 bsd-user/riscv/target_arch_cpu.h   |  4 ---
 bsd-user/x86_64/target_arch_cpu.h  |  5 ---
 include/hw/core/cpu.h              |  6 ++++
 accel/kvm/kvm-all.c                |  9 ------
 accel/tcg/cputlb.c                 | 49 +++++++++++++++++++++---------
 accel/tcg/tcg-accel-ops-rr.c       | 13 +++++---
 cpu-common.c                       | 23 ++++++++------
 cpu-target.c                       |  2 +-
 hw/core/cpu-common.c               |  2 ++
 target/loongarch/cpu.c             |  2 +-
 target/m68k/cpu.c                  |  2 +-
 14 files changed, 68 insertions(+), 63 deletions(-)