Skip to contentSkip to conversation

tcg: drop qemu_cpu_is_self() in tlb_flush_by_mmuidx[_async_work]

PATCH In review Igor Mammedov 3 participants 10 messages Jan 23, 2025 – Jan 27, 2025

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tcg: drop qemu_cpu_is_self() in tlb_flush_by_mmuidx[_async_work]

QEMU will crash with following debug enabled

  # define DEBUG_TLB_GATE 1
  # define DEBUG_TLB_LOG_GATE 1
due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work]
functions are called not only from vcpu thread but also from reset handler
that is called from main thread at cpu realize time when vcpu is already
created
  x86_cpu_new -> ... ->
      x86_cpu_realizefn -> cpu_reset -> ... ->
          tcg_cpu_reset_hold

drop assert to fix crash.

1)

Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")
---
 accel/tcg/cputlb.c | 4 ----
 1 file changed, 4 deletions(-)
accel/tcg/cputlb.c +0 -4
index b26c0e088f..2da803103c 100644
@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)
     uint16_t all_dirty, work, to_clean;
     int64_t now = get_clock_realtime();
 
-    assert_cpu_is_self(cpu);
-
     tlb_debug("mmu_idx:0x%04" PRIx16 "\n", asked);
 
     qemu_spin_lock(&cpu->neg.tlb.c.lock);
@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
 {
     tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    assert_cpu_is_self(cpu);
-
     tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
 }
 
Signature · 1 line
2.43.0
#1 lore raw
Quoted 11 lines · Igor Mammedov

QEMU will crash with following debug enabled

  # define DEBUG_TLB_GATE 1
  # define DEBUG_TLB_LOG_GATE 1
due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work]
functions are called not only from vcpu thread but also from reset handler
that is called from main thread at cpu realize time when vcpu is already
created
  x86_cpu_new -> ... ->
      x86_cpu_realizefn -> cpu_reset -> ... ->
          tcg_cpu_reset_hold
Quoted 1 line

drop assert to fix crash.

Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:

  g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));

was trying to account for pre-initialised vCPUs. What has changed?

cpu_thread_signal_created(cpu) is called just before we start running the main loop in mttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.

Quoted 29 lines

1)

Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")
---
 accel/tcg/cputlb.c | 4 ----
 1 file changed, 4 deletions(-)
accel/tcg/cputlb.c +0 -4
index b26c0e088f..2da803103c 100644
@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)
     uint16_t all_dirty, work, to_clean;
     int64_t now = get_clock_realtime();
 
-    assert_cpu_is_self(cpu);
-
     tlb_debug("mmu_idx:0x%04" PRIx16 "\n", asked);
 
     qemu_spin_lock(&cpu->neg.tlb.c.lock);
@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
 {
     tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    assert_cpu_is_self(cpu);
-
     tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
 }
Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#2 lore raw
Quoted 18 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov

QEMU will crash with following debug enabled

  # define DEBUG_TLB_GATE 1
  # define DEBUG_TLB_LOG_GATE 1
due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work]
functions are called not only from vcpu thread but also from reset handler
that is called from main thread at cpu realize time when vcpu is already
created
  x86_cpu_new -> ... ->
      x86_cpu_realizefn -> cpu_reset -> ... ->
          tcg_cpu_reset_hold
Quoted 1 line

drop assert to fix crash.

Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:

Quoted 7 lines
  g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));

was trying to account for pre-initialised vCPUs. What has changed?

cpu_thread_signal_created(cpu) is called just before we start running the main loop in mttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.

it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of cpu->created usage.

Quoted 29 lines
Quoted 28 lines

1)

Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")
---
 accel/tcg/cputlb.c | 4 ----
 1 file changed, 4 deletions(-)
accel/tcg/cputlb.c +0 -4
index b26c0e088f..2da803103c 100644
@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)
     uint16_t all_dirty, work, to_clean;
     int64_t now = get_clock_realtime();
 
-    assert_cpu_is_self(cpu);
-
     tlb_debug("mmu_idx:0x%04" PRIx16 "\n", asked);
 
     qemu_spin_lock(&cpu->neg.tlb.c.lock);
@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
 {
     tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    assert_cpu_is_self(cpu);
-
     tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
 }  
#3 lore raw
Quoted 25 lines · Igor Mammedov
Quoted 22 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov

QEMU will crash with following debug enabled

  # define DEBUG_TLB_GATE 1
  # define DEBUG_TLB_LOG_GATE 1
due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work]
functions are called not only from vcpu thread but also from reset handler
that is called from main thread at cpu realize time when vcpu is already
created
  x86_cpu_new -> ... ->
      x86_cpu_realizefn -> cpu_reset -> ... ->
          tcg_cpu_reset_hold
Quoted 1 line

drop assert to fix crash.

Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:

  g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));

was trying to account for pre-initialised vCPUs. What has changed?

Quoted 7 lines
Quoted 3 lines

cpu_thread_signal_created(cpu) is called just before we start running the main loop in mttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.

it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of cpu->created usage.

Why the drive to get rid of cpu->created?

I guess we could assert:

  g_assert(!current_cpu || qemu_cpu_is_self(cpu);

as current_cpu should only be set as we go into the main thread. However there is a sketchy setting of current_cpu in cpu_exec() that I'm not sure should be there. Also do_run_on_cpu() messes with current_cpu in a way I don't fully understand either.

Quoted 31 lines
Quoted 31 lines
Quoted 30 lines

1)

Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")
---
 accel/tcg/cputlb.c | 4 ----
 1 file changed, 4 deletions(-)
accel/tcg/cputlb.c +0 -4
index b26c0e088f..2da803103c 100644
@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)
     uint16_t all_dirty, work, to_clean;
     int64_t now = get_clock_realtime();
 
-    assert_cpu_is_self(cpu);
-
     tlb_debug("mmu_idx:0x%04" PRIx16 "\n", asked);
 
     qemu_spin_lock(&cpu->neg.tlb.c.lock);
@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
 {
     tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    assert_cpu_is_self(cpu);
-
     tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
 }  
Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro
#4 lore raw
Quoted 31 lines · On Thu, 23 Jan 2025 12:09:59 +0000 Alex Bennée <[email protected]> wrote:
Quoted 29 lines · Igor Mammedov
Quoted 25 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov

QEMU will crash with following debug enabled

  # define DEBUG_TLB_GATE 1
  # define DEBUG_TLB_LOG_GATE 1
due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work]
functions are called not only from vcpu thread but also from reset handler
that is called from main thread at cpu realize time when vcpu is already
created
  x86_cpu_new -> ... ->
      x86_cpu_realizefn -> cpu_reset -> ... ->
          tcg_cpu_reset_hold
Quoted 1 line

drop assert to fix crash.

Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:

  g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));

was trying to account for pre-initialised vCPUs. What has changed?

cpu_thread_signal_created(cpu) is called just before we start running the main loop in mttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.

Quoted 5 lines
Quoted 3 lines

it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of cpu->created usage.

Why the drive to get rid of cpu->created?

During review of Philippe's cpu cleanups,
I've noticed that cpu->created is mostly used for signaling main loop thread we've started vcpu thread with a couple of odd cases in tcg and kvm.
- 1st silently bit-rotted being behind ifdefs
- 2nd is work around CPU being prematurely visible to others

Quoted 8 lines

I guess we could assert:

  g_assert(!current_cpu || qemu_cpu_is_self(cpu);

as current_cpu should only be set as we go into the main thread. However there is a sketchy setting of current_cpu in cpu_exec() that I'm not sure should be there. Also do_run_on_cpu() messes with current_cpu in a way I don't fully understand either.

I'd rather not rely on that, even if it works it would be subject to to the same kind of breakage.

How about instead of workaround check we would have 2 variants of tlb_flush_by_mmuidx[_async_work], one that have self check and other for usage externally (i.e. from reset handler). That won't have to rely on sketchy globals (which becomes more sketchy in context of Philippes's multi accel work), and it would clearly document what can be used externally.

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines

1)

Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")
---
 accel/tcg/cputlb.c | 4 ----
 1 file changed, 4 deletions(-)
accel/tcg/cputlb.c +0 -4
index b26c0e088f..2da803103c 100644
@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)
     uint16_t all_dirty, work, to_clean;
     int64_t now = get_clock_realtime();
 
-    assert_cpu_is_self(cpu);
-
     tlb_debug("mmu_idx:0x%04" PRIx16 "\n", asked);
 
     qemu_spin_lock(&cpu->neg.tlb.c.lock);
@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
 {
     tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    assert_cpu_is_self(cpu);
-
     tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
 }    
#5 lore raw

+Peter

Quoted 38 lines · Igor Mammedov
Quoted 35 lines · On Thu, 23 Jan 2025 12:09:59 +0000 Alex Bennée <[email protected]> wrote:
Quoted 32 lines · Igor Mammedov
Quoted 18 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov

QEMU will crash with following debug enabled

   # define DEBUG_TLB_GATE 1
   # define DEBUG_TLB_LOG_GATE 1
due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work]
functions are called not only from vcpu thread but also from reset handler
that is called from main thread at cpu realize time when vcpu is already
created
   x86_cpu_new -> ... ->
       x86_cpu_realizefn -> cpu_reset -> ... ->
           tcg_cpu_reset_hold
Quoted 1 line

drop assert to fix crash.

Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:

Quoted 7 lines
   g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));

was trying to account for pre-initialised vCPUs. What has changed?

cpu_thread_signal_created(cpu) is called just before we start running the main loop in mttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.

it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of cpu->created usage.

Quoted 8 lines
Quoted 1 line

Why the drive to get rid of cpu->created?

During review of Philippe's cpu cleanups,
I've noticed that cpu->created is mostly used for signaling main loop thread we've started vcpu thread with a couple of odd cases in tcg and kvm.
- 1st silently bit-rotted being behind ifdefs
- 2nd is work around CPU being prematurely visible to others

2nd is:

commit 56adee407fc564da19e49cfe18e20e3da92320be

Author: Peter Xu <[email protected]>
Date:   Fri Feb 17 00:18:32 2023 +0800

kvm: dirty-ring: Fix race with vcpu creation

It's possible that we want to reap a dirty ring on a vcpu that is
during
creation, because the vcpu is put onto list (CPU_FOREACH visible)
before
initialization of the structures. In this case:

qemu_init_vcpu
x86_cpu_realizefn
cpu_exec_realizefn
cpu_list_add <---- can be probed by CPU_FOREACH
qemu_init_vcpu
cpus_accel->create_vcpu_thread(cpu);
kvm_init_vcpu
map kvm_dirty_gfns <--- kvm_dirty_gfns valid

Don't try to reap dirty ring on vcpus during creation or it'll crash.

Looking at cpu_list_add() in cpu_exec_realizefn():

hw/core/cpu-common.c-190- /* Wait until cpu initialization complete before exposing cpu. */
hw/core/cpu-common.c:191: cpu_list_add(cpu);

IMO the problem is with cpu_list_add(), we shouldn't expose the vCPU until it is realized.

cpu_list_add() seems to be doing 2 things, auto-assign CPU index if UNASSIGNED_CPU_INDEX, then insert to global cpus_queue.

IIRC cpu_list_add() is called early because various cpu init code expects cpu->index to be assigned.

Maybe we could extract the 'safely assign an unique cpu index' part (guarding by qemu_cpu_list_lock), having cpu_list_add() just add to the global queue. I'll give it a try...

Quoted 55 lines
Quoted 9 lines

I guess we could assert:

   g_assert(!current_cpu || qemu_cpu_is_self(cpu);

as current_cpu should only be set as we go into the main thread. However there is a sketchy setting of current_cpu in cpu_exec() that I'm not sure should be there. Also do_run_on_cpu() messes with current_cpu in a way I don't fully understand either.

I'd rather not rely on that, even if it works it would be subject to to the same kind of breakage.

How about instead of workaround check we would have 2 variants of tlb_flush_by_mmuidx[_async_work], one that have self check and other for usage externally (i.e. from reset handler). That won't have to rely on sketchy globals (which becomes more sketchy in context of Philippes's multi accel work), and it would clearly document what can be used externally.

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines

1)

Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")
---
  accel/tcg/cputlb.c | 4 ----
  1 file changed, 4 deletions(-)
accel/tcg/cputlb.c +0 -4
index b26c0e088f..2da803103c 100644
@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)
      uint16_t all_dirty, work, to_clean;
      int64_t now = get_clock_realtime();
 
-    assert_cpu_is_self(cpu);
-
      tlb_debug("mmu_idx:0x%04" PRIx16 "\n", asked);
 
      qemu_spin_lock(&cpu->neg.tlb.c.lock);
@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
  {
      tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    assert_cpu_is_self(cpu);
-
      tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
  }
#6 lore raw
Quoted 75 lines · On Mon, 27 Jan 2025 14:24:56 +0100 Philippe Mathieu-Daudé <[email protected]>…

+Peter

Quoted 38 lines · Igor Mammedov
Quoted 35 lines · On Thu, 23 Jan 2025 12:09:59 +0000 Alex Bennée <[email protected]> wrote:
Quoted 32 lines · Igor Mammedov
Quoted 18 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov

QEMU will crash with following debug enabled

   # define DEBUG_TLB_GATE 1
   # define DEBUG_TLB_LOG_GATE 1
due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work]
functions are called not only from vcpu thread but also from reset handler
that is called from main thread at cpu realize time when vcpu is already
created
   x86_cpu_new -> ... ->
       x86_cpu_realizefn -> cpu_reset -> ... ->
           tcg_cpu_reset_hold
Quoted 1 line

drop assert to fix crash.

Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:

Quoted 7 lines
   g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));

was trying to account for pre-initialised vCPUs. What has changed?

cpu_thread_signal_created(cpu) is called just before we start running the main loop in mttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.

it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of cpu->created usage.

Quoted 8 lines
Quoted 1 line

Why the drive to get rid of cpu->created?

During review of Philippe's cpu cleanups,
I've noticed that cpu->created is mostly used for signaling main loop thread we've started vcpu thread with a couple of odd cases in tcg and kvm.
- 1st silently bit-rotted being behind ifdefs
- 2nd is work around CPU being prematurely visible to others

2nd is:

commit 56adee407fc564da19e49cfe18e20e3da92320be

Author: Peter Xu <[email protected]>
Date:   Fri Feb 17 00:18:32 2023 +0800

kvm: dirty-ring: Fix race with vcpu creation

It's possible that we want to reap a dirty ring on a vcpu that is
during
creation, because the vcpu is put onto list (CPU_FOREACH visible)
before
initialization of the structures. In this case:

qemu_init_vcpu
x86_cpu_realizefn
cpu_exec_realizefn
cpu_list_add <---- can be probed by CPU_FOREACH
qemu_init_vcpu
cpus_accel->create_vcpu_thread(cpu);
kvm_init_vcpu
map kvm_dirty_gfns <--- kvm_dirty_gfns valid

Don't try to reap dirty ring on vcpus during creation or it'll crash.

Quoted 8 lines

Looking at cpu_list_add() in cpu_exec_realizefn():

hw/core/cpu-common.c-190- /* Wait until cpu initialization complete
before exposing cpu. */
hw/core/cpu-common.c:191: cpu_list_add(cpu);

IMO the problem is with cpu_list_add(), we shouldn't expose the vCPU until it is realized.

that was my reasoning as well

Quoted 10 lines

cpu_list_add() seems to be doing 2 things, auto-assign CPU index if UNASSIGNED_CPU_INDEX, then insert to global cpus_queue.

IIRC cpu_list_add() is called early because various cpu init code expects cpu->index to be assigned.

Maybe we could extract the 'safely assign an unique cpu index' part (guarding by qemu_cpu_list_lock), having cpu_list_add() just add to the global queue. I'll give it a try...

ok,
lets see if we can postpone cpu_list_add() till the end of realize time.

Quoted 57 lines
Quoted 56 lines
Quoted 10 lines

I guess we could assert:

   g_assert(!current_cpu || qemu_cpu_is_self(cpu);

as current_cpu should only be set as we go into the main thread. However there is a sketchy setting of current_cpu in cpu_exec() that I'm not sure should be there. Also do_run_on_cpu() messes with current_cpu in a way I don't fully understand either.

I'd rather not rely on that, even if it works it would be subject to to the same kind of breakage.

How about instead of workaround check we would have 2 variants of tlb_flush_by_mmuidx[_async_work], one that have self check and other for usage externally (i.e. from reset handler). That won't have to rely on sketchy globals (which becomes more sketchy in context of Philippes's multi accel work), and it would clearly document what can be used externally.

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines

1)

Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")
---
  accel/tcg/cputlb.c | 4 ----
  1 file changed, 4 deletions(-)
accel/tcg/cputlb.c +0 -4
index b26c0e088f..2da803103c 100644
@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)
      uint16_t all_dirty, work, to_clean;
      int64_t now = get_clock_realtime();
 
-    assert_cpu_is_self(cpu);
-
      tlb_debug("mmu_idx:0x%04" PRIx16 "\n", asked);
 
      qemu_spin_lock(&cpu->neg.tlb.c.lock);
@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
  {
      tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    assert_cpu_is_self(cpu);
-
      tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
  }  
#7 lore raw

(also Cc'ing Harsh for commit 18530e7c57de)

Quoted 161 lines · Igor Mammedov
Quoted 75 lines · On Mon, 27 Jan 2025 14:24:56 +0100 Philippe Mathieu-Daudé <[email protected]>…

+Peter

Quoted 38 lines · Igor Mammedov
Quoted 35 lines · On Thu, 23 Jan 2025 12:09:59 +0000 Alex Bennée <[email protected]> wrote:
Quoted 32 lines · Igor Mammedov
Quoted 18 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov

QEMU will crash with following debug enabled

    # define DEBUG_TLB_GATE 1
    # define DEBUG_TLB_LOG_GATE 1
due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work]
functions are called not only from vcpu thread but also from reset handler
that is called from main thread at cpu realize time when vcpu is already
created
    x86_cpu_new -> ... ->
        x86_cpu_realizefn -> cpu_reset -> ... ->
            tcg_cpu_reset_hold
Quoted 1 line

drop assert to fix crash.

Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:

Quoted 7 lines
    g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));

was trying to account for pre-initialised vCPUs. What has changed?

cpu_thread_signal_created(cpu) is called just before we start running the main loop in mttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.

it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of cpu->created usage.

Quoted 8 lines
Quoted 1 line

Why the drive to get rid of cpu->created?

During review of Philippe's cpu cleanups,
I've noticed that cpu->created is mostly used for signaling main loop thread we've started vcpu thread with a couple of odd cases in tcg and kvm.
- 1st silently bit-rotted being behind ifdefs
- 2nd is work around CPU being prematurely visible to others

2nd is:

commit 56adee407fc564da19e49cfe18e20e3da92320be

Author: Peter Xu <[email protected]>
Date:   Fri Feb 17 00:18:32 2023 +0800

kvm: dirty-ring: Fix race with vcpu creation

It's possible that we want to reap a dirty ring on a vcpu that is
during
creation, because the vcpu is put onto list (CPU_FOREACH visible)
before
initialization of the structures. In this case:

qemu_init_vcpu
x86_cpu_realizefn
cpu_exec_realizefn
cpu_list_add <---- can be probed by CPU_FOREACH
qemu_init_vcpu
cpus_accel->create_vcpu_thread(cpu);
kvm_init_vcpu
map kvm_dirty_gfns <--- kvm_dirty_gfns valid

Don't try to reap dirty ring on vcpus during creation or it'll crash.

Quoted 8 lines

Looking at cpu_list_add() in cpu_exec_realizefn():

hw/core/cpu-common.c-190- /* Wait until cpu initialization complete
before exposing cpu. */
hw/core/cpu-common.c:191: cpu_list_add(cpu);

IMO the problem is with cpu_list_add(), we shouldn't expose the vCPU until it is realized.

that was my reasoning as well

Quoted 10 lines

cpu_list_add() seems to be doing 2 things, auto-assign CPU index if UNASSIGNED_CPU_INDEX, then insert to global cpus_queue.

IIRC cpu_list_add() is called early because various cpu init code expects cpu->index to be assigned.

Maybe we could extract the 'safely assign an unique cpu index' part (guarding by qemu_cpu_list_lock), having cpu_list_add() just add to the global queue. I'll give it a try...

ok,
lets see if we can postpone cpu_list_add() till the end of realize time.

Quoted 57 lines
Quoted 56 lines
Quoted 10 lines

I guess we could assert:

    g_assert(!current_cpu || qemu_cpu_is_self(cpu);

as current_cpu should only be set as we go into the main thread. However there is a sketchy setting of current_cpu in cpu_exec() that I'm not sure should be there. Also do_run_on_cpu() messes with current_cpu in a way I don't fully understand either.

I'd rather not rely on that, even if it works it would be subject to to the same kind of breakage.

How about instead of workaround check we would have 2 variants of tlb_flush_by_mmuidx[_async_work], one that have self check and other for usage externally (i.e. from reset handler). That won't have to rely on sketchy globals (which becomes more sketchy in context of Philippes's multi accel work), and it would clearly document what can be used externally.

Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines

1)

Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")
---
   accel/tcg/cputlb.c | 4 ----
   1 file changed, 4 deletions(-)
accel/tcg/cputlb.c +0 -4
index b26c0e088f..2da803103c 100644
@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)
       uint16_t all_dirty, work, to_clean;
       int64_t now = get_clock_realtime();
 
-    assert_cpu_is_self(cpu);
-
       tlb_debug("mmu_idx:0x%04" PRIx16 "\n", asked);
 
       qemu_spin_lock(&cpu->neg.tlb.c.lock);
@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
   {
       tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    assert_cpu_is_self(cpu);
-
       tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
   }
#8 lore raw
In reply to Igor Mammedov
Quoted 23 lines · On Thu, 23 Jan 2025 12:23:43 +0100 Igor Mammedov <[email protected]> wrote:
Quoted 20 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov

QEMU will crash with following debug enabled

  # define DEBUG_TLB_GATE 1
  # define DEBUG_TLB_LOG_GATE 1
due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work]
functions are called not only from vcpu thread but also from reset handler
that is called from main thread at cpu realize time when vcpu is already
created
  x86_cpu_new -> ... ->
      x86_cpu_realizefn -> cpu_reset -> ... ->
          tcg_cpu_reset_hold
Quoted 1 line

drop assert to fix crash.

Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:

  g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));
Quoted 12 lines
Quoted 5 lines

was trying to account for pre-initialised vCPUs. What has changed?

cpu_thread_signal_created(cpu) is called just before we start running the main loop in mttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.

it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of cpu->created usage.

Quoted 2 lines
Quoted 2 lines

1)

Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")

bisection points to (so above fixes was a wrong one):
30933c4fb4f3d tcg/cputlb: remove other-cpu capability from TLB flushing
which has replaced a check with assert:

- if (cpu->created && !qemu_cpu_is_self(cpu)) {
- async_run_on_cpu(cpu, tlb_flush_by_mmuidx_async_work,
- RUN_ON_CPU_HOST_INT(idxmap));
- } else {
- tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
- }
+ assert_cpu_is_self(cpu);

should we revert that instead?

perhaps also drop 'cpu->created' check in assert_cpu_is_self as it obviously doesn't work.

Quoted 28 lines
Quoted 27 lines
Quoted 26 lines
---
 accel/tcg/cputlb.c | 4 ----
 1 file changed, 4 deletions(-)
accel/tcg/cputlb.c +0 -4
index b26c0e088f..2da803103c 100644
@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)
     uint16_t all_dirty, work, to_clean;
     int64_t now = get_clock_realtime();
 
-    assert_cpu_is_self(cpu);
-
     tlb_debug("mmu_idx:0x%04" PRIx16 "\n", asked);
 
     qemu_spin_lock(&cpu->neg.tlb.c.lock);
@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
 {
     tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    assert_cpu_is_self(cpu);
-
     tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
 }    
#9 lore raw
Quoted 51 lines · Igor Mammedov
Quoted 23 lines · On Thu, 23 Jan 2025 12:23:43 +0100 Igor Mammedov <[email protected]> wrote:
Quoted 20 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov

QEMU will crash with following debug enabled

  # define DEBUG_TLB_GATE 1
  # define DEBUG_TLB_LOG_GATE 1
due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work]
functions are called not only from vcpu thread but also from reset handler
that is called from main thread at cpu realize time when vcpu is already
created
  x86_cpu_new -> ... ->
      x86_cpu_realizefn -> cpu_reset -> ... ->
          tcg_cpu_reset_hold
Quoted 1 line

drop assert to fix crash.

Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:

  g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));
Quoted 12 lines
Quoted 5 lines

was trying to account for pre-initialised vCPUs. What has changed?

cpu_thread_signal_created(cpu) is called just before we start running the main loop in mttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.

it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of cpu->created usage.

Quoted 2 lines
Quoted 2 lines

1)

Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")

bisection points to (so above fixes was a wrong one):
30933c4fb4f3d tcg/cputlb: remove other-cpu capability from TLB flushing
which has replaced a check with assert:

(quoted diff) +1 -6
-    if (cpu->created && !qemu_cpu_is_self(cpu)) {
-        async_run_on_cpu(cpu, tlb_flush_by_mmuidx_async_work,
-                         RUN_ON_CPU_HOST_INT(idxmap));
-    } else {
-        tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
-    }
+    assert_cpu_is_self(cpu);
Quoted 4 lines

should we revert that instead?

perhaps also drop 'cpu->created' check in assert_cpu_is_self as it obviously doesn't work.

That's because the asserts only check when built with debug due to concerns about the hot path. I think in the case of cputlb we should probably use tcg_debug_assert() which at least gets enabled for --enable-debug builds without needing to manually patch the define for DEBUG_TLB.

Converting the custom TLB log implementation to using tracepoints would also be nice.

Quoted 30 lines
Quoted 30 lines
Quoted 29 lines
Quoted 28 lines
---
 accel/tcg/cputlb.c | 4 ----
 1 file changed, 4 deletions(-)
accel/tcg/cputlb.c +0 -4
index b26c0e088f..2da803103c 100644
@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)
     uint16_t all_dirty, work, to_clean;
     int64_t now = get_clock_realtime();
 
-    assert_cpu_is_self(cpu);
-
     tlb_debug("mmu_idx:0x%04" PRIx16 "\n", asked);
 
     qemu_spin_lock(&cpu->neg.tlb.c.lock);
@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap)
 {
     tlb_debug("mmu_idx: 0x%" PRIx16 "\n", idxmap);
 
-    assert_cpu_is_self(cpu);
-
     tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
 }    
Signature · 2 lines
Alex Bennée
Virtualisation Tech Lead @ Linaro