tcg: drop qemu_cpu_is_self() in tlb_flush_by_mmuidx[_async_work]
PATCH 1/1 tcg: drop qemu_cpu_is_self() in tlb_flush_by_mmuidx[_async_work]
QEMU will crash with following debug enabled
# define DEBUG_TLB_GATE 1
# define DEBUG_TLB_LOG_GATE 1
due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work]
functions are called not only from vcpu thread but also from reset handler
that is called from main thread at cpu realize time when vcpu is already
created
x86_cpu_new -> ... ->
x86_cpu_realizefn -> cpu_reset -> ... ->
tcg_cpu_reset_holddrop assert to fix crash.
1)
--- accel/tcg/cputlb.c | 4 ---- 1 file changed, 4 deletions(-)
accel/tcg/cputlb.c +0 -4
@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)uint16_t all_dirty, work, to_clean;int64_t now =get_clock_realtime (); -assert_cpu_is_self (cpu); -tlb_debug ("mmu_idx:0x%04" PRIx16"\n" , asked);qemu_spin_lock (&cpu->neg.tlb.c.lock);
@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap) {tlb_debug ("mmu_idx: 0x%" PRIx16"\n" , idxmap); -assert_cpu_is_self (cpu); -tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); }
Signature · 1 line
2.43.0
Quoted 11 lines · Igor Mammedov
QEMU will crash with following debug enabled
# define DEBUG_TLB_GATE 1 # define DEBUG_TLB_LOG_GATE 1 due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work] functions are called not only from vcpu thread but also from reset handler that is called from main thread at cpu realize time when vcpu is already created x86_cpu_new -> ... -> x86_cpu_realizefn -> cpu_reset -> ... -> tcg_cpu_reset_hold
Quoted 1 line
drop assert to fix crash.
Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:
g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));
was trying to account for pre-initialised vCPUs. What has changed?
cpu_thread_signal_created(cpu) is called just before we start running the main loop in mttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.
Quoted 29 lines
1)
Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")Signed-off-by Igor Mammedov <[email protected]>--- accel/tcg/cputlb.c | 4 ---- 1 file changed, 4 deletions(-)accel/tcg/cputlb.c +0
-4@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)uint16_t all_dirty, work, to_clean;int64_t now =get_clock_realtime (); -assert_cpu_is_self (cpu); -tlb_debug ("mmu_idx:0x%04" PRIx16"\n" , asked);qemu_spin_lock (&cpu->neg.tlb.c.lock);@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap) {tlb_debug ("mmu_idx: 0x%" PRIx16"\n" , idxmap); -assert_cpu_is_self (cpu); -tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); }
Signature · 2 lines
Alex Bennée Virtualisation Tech Lead @ Linaro
Quoted 18 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov
QEMU will crash with following debug enabled
# define DEBUG_TLB_GATE 1 # define DEBUG_TLB_LOG_GATE 1 due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work] functions are called not only from vcpu thread but also from reset handler that is called from main thread at cpu realize time when vcpu is already created x86_cpu_new -> ... -> x86_cpu_realizefn -> cpu_reset -> ... -> tcg_cpu_reset_holdQuoted 1 line
drop assert to fix crash.
Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:
Quoted 7 lines
g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));was trying to account for pre-initialised vCPUs. What has changed?
cpu_thread_signal_created(cpu)is called just before we start running the main loop inmttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.
it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of cpu->created usage.
Quoted 29 lines
Quoted 28 lines
1)
Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")Signed-off-by Igor Mammedov <[email protected]>--- accel/tcg/cputlb.c | 4 ---- 1 file changed, 4 deletions(-)accel/tcg/cputlb.c +0
-4@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)uint16_t all_dirty, work, to_clean;int64_t now =get_clock_realtime (); -assert_cpu_is_self (cpu); -tlb_debug ("mmu_idx:0x%04" PRIx16"\n" , asked);qemu_spin_lock (&cpu->neg.tlb.c.lock);@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap) {tlb_debug ("mmu_idx: 0x%" PRIx16"\n" , idxmap); -assert_cpu_is_self (cpu); -tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); }
Quoted 25 lines · Igor Mammedov
Quoted 22 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov
QEMU will crash with following debug enabled
# define DEBUG_TLB_GATE 1 # define DEBUG_TLB_LOG_GATE 1 due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work] functions are called not only from vcpu thread but also from reset handler that is called from main thread at cpu realize time when vcpu is already created x86_cpu_new -> ... -> x86_cpu_realizefn -> cpu_reset -> ... -> tcg_cpu_reset_holdQuoted 1 line
drop assert to fix crash.
Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:
g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));was trying to account for pre-initialised vCPUs. What has changed?
Quoted 7 lines
Quoted 3 lines
cpu_thread_signal_created(cpu)is called just before we start running the main loop inmttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of
cpu->createdusage.
Why the drive to get rid of cpu->created?
I guess we could assert:
g_assert(!current_cpu || qemu_cpu_is_self(cpu);
as current_cpu should only be set as we go into the main thread. However there is a sketchy setting of current_cpu in cpu_exec() that I'm not sure should be there. Also do_run_on_cpu() messes with current_cpu in a way I don't fully understand either.
Quoted 31 lines
Quoted 31 lines
Quoted 30 lines
1)
Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")Signed-off-by Igor Mammedov <[email protected]>--- accel/tcg/cputlb.c | 4 ---- 1 file changed, 4 deletions(-)accel/tcg/cputlb.c +0
-4@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)uint16_t all_dirty, work, to_clean;int64_t now =get_clock_realtime (); -assert_cpu_is_self (cpu); -tlb_debug ("mmu_idx:0x%04" PRIx16"\n" , asked);qemu_spin_lock (&cpu->neg.tlb.c.lock);@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap) {tlb_debug ("mmu_idx: 0x%" PRIx16"\n" , idxmap); -assert_cpu_is_self (cpu); -tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); }
Signature · 2 lines
Alex Bennée Virtualisation Tech Lead @ Linaro
Quoted 31 lines · On Thu, 23 Jan 2025 12:09:59 +0000 Alex Bennée <[email protected]> wrote:
Quoted 29 lines · Igor Mammedov
Quoted 25 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov
QEMU will crash with following debug enabled
# define DEBUG_TLB_GATE 1 # define DEBUG_TLB_LOG_GATE 1 due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work] functions are called not only from vcpu thread but also from reset handler that is called from main thread at cpu realize time when vcpu is already created x86_cpu_new -> ... -> x86_cpu_realizefn -> cpu_reset -> ... -> tcg_cpu_reset_holdQuoted 1 line
drop assert to fix crash.
Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:
g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));was trying to account for pre-initialised vCPUs. What has changed?
cpu_thread_signal_created(cpu)is called just before we start running the main loop inmttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.
Quoted 5 lines
Quoted 3 lines
it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of
cpu->createdusage.Why the drive to get rid of
cpu->created?
During review of Philippe's cpu cleanups,
I've noticed that cpu->created is mostly used for signaling main loop thread we've started vcpu thread with a couple of odd cases in tcg and kvm.
- 1st silently bit-rotted being behind ifdefs
- 2nd is work around CPU being prematurely visible to others
Quoted 8 lines
I guess we could assert:
g_assert(!current_cpu || qemu_cpu_is_self(cpu);as
current_cpushould only be set as we go into the main thread. However there is a sketchy setting ofcurrent_cpuincpu_exec()that I'm not sure should be there. Alsodo_run_on_cpu()messes withcurrent_cpuin a way I don't fully understand either.
I'd rather not rely on that, even if it works it would be subject to to the same kind of breakage.
How about instead of workaround check we would have 2 variants of tlb_flush_by_mmuidx[_async_work], one that have self check and other for usage externally (i.e. from reset handler). That won't have to rely on sketchy globals (which becomes more sketchy in context of Philippes's multi accel work), and it would clearly document what can be used externally.
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
1)
Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")Signed-off-by Igor Mammedov <[email protected]>--- accel/tcg/cputlb.c | 4 ---- 1 file changed, 4 deletions(-)accel/tcg/cputlb.c +0
-4@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)uint16_t all_dirty, work, to_clean;int64_t now =get_clock_realtime (); -assert_cpu_is_self (cpu); -tlb_debug ("mmu_idx:0x%04" PRIx16"\n" , asked);qemu_spin_lock (&cpu->neg.tlb.c.lock);@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap) {tlb_debug ("mmu_idx: 0x%" PRIx16"\n" , idxmap); -assert_cpu_is_self (cpu); -tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); }
+Peter
Quoted 38 lines · Igor Mammedov
Quoted 35 lines · On Thu, 23 Jan 2025 12:09:59 +0000 Alex Bennée <[email protected]> wrote:
Quoted 32 lines · Igor Mammedov
Quoted 18 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov
QEMU will crash with following debug enabled
# define DEBUG_TLB_GATE 1 # define DEBUG_TLB_LOG_GATE 1 due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work] functions are called not only from vcpu thread but also from reset handler that is called from main thread at cpu realize time when vcpu is already created x86_cpu_new -> ... -> x86_cpu_realizefn -> cpu_reset -> ... -> tcg_cpu_reset_holdQuoted 1 line
drop assert to fix crash.
Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:
Quoted 7 lines
g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));was trying to account for pre-initialised vCPUs. What has changed?
cpu_thread_signal_created(cpu)is called just before we start running the main loop inmttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of
cpu->createdusage.
Quoted 8 lines
Quoted 1 line
Why the drive to get rid of
cpu->created?During review of Philippe's cpu cleanups,
I've noticed thatcpu->createdis mostly used for signaling main loop thread we've started vcpu thread with a couple of odd cases in tcg and kvm.
- 1st silently bit-rotted being behind ifdefs
- 2nd is work around CPU being prematurely visible to others
2nd is:
commit 56adee407fc564da19e49cfe18e20e3da92320be
Author: Peter Xu <[email protected]> Date: Fri Feb 17 00:18:32 2023 +0800
kvm: dirty-ring: Fix race with vcpu creation
It's possible that we want to reap a dirty ring on a vcpu that is
during
creation, because the vcpu is put onto list (CPU_FOREACH visible)
before
initialization of the structures. In this case:
qemu_init_vcpu
x86_cpu_realizefn
cpu_exec_realizefn
cpu_list_add <---- can be probed by CPU_FOREACH
qemu_init_vcpu
cpus_accel->create_vcpu_thread(cpu);
kvm_init_vcpu
map kvm_dirty_gfns <--- kvm_dirty_gfns valid
Don't try to reap dirty ring on vcpus during creation or it'll crash.
Looking at cpu_list_add() in cpu_exec_realizefn():
hw/core/cpu-common.c-190- /* Wait until cpu initialization complete before exposing cpu. */hw/core/cpu-common.c:191: cpu_list_add(cpu);
IMO the problem is with cpu_list_add(), we shouldn't expose the vCPU until it is realized.
cpu_list_add() seems to be doing 2 things, auto-assign CPU index if UNASSIGNED_CPU_INDEX, then insert to global cpus_queue.
IIRC cpu_list_add() is called early because various cpu init code expects cpu->index to be assigned.
Maybe we could extract the 'safely assign an unique cpu index' part (guarding by qemu_cpu_list_lock), having cpu_list_add() just add to the global queue. I'll give it a try...
Quoted 55 lines
Quoted 9 lines
I guess we could assert:
g_assert(!current_cpu || qemu_cpu_is_self(cpu);as
current_cpushould only be set as we go into the main thread. However there is a sketchy setting ofcurrent_cpuincpu_exec()that I'm not sure should be there. Alsodo_run_on_cpu()messes withcurrent_cpuin a way I don't fully understand either.I'd rather not rely on that, even if it works it would be subject to to the same kind of breakage.
How about instead of workaround check we would have 2 variants of tlb_flush_by_mmuidx[_async_work], one that have self check and other for usage externally (i.e. from reset handler). That won't have to rely on sketchy globals (which becomes more sketchy in context of Philippes's multi accel work), and it would clearly document what can be used externally.
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
1)
Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")Signed-off-by Igor Mammedov <[email protected]>--- accel/tcg/cputlb.c | 4 ---- 1 file changed, 4 deletions(-)accel/tcg/cputlb.c +0
-4@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)uint16_t all_dirty, work, to_clean;int64_t now =get_clock_realtime (); -assert_cpu_is_self (cpu); -tlb_debug ("mmu_idx:0x%04" PRIx16"\n" , asked);qemu_spin_lock (&cpu->neg.tlb.c.lock);@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap) {tlb_debug ("mmu_idx: 0x%" PRIx16"\n" , idxmap); -assert_cpu_is_self (cpu); -tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); }
Quoted 75 lines · On Mon, 27 Jan 2025 14:24:56 +0100 Philippe Mathieu-Daudé <[email protected]>…
+Peter
Quoted 38 lines · Igor Mammedov
Quoted 35 lines · On Thu, 23 Jan 2025 12:09:59 +0000 Alex Bennée <[email protected]> wrote:
Quoted 32 lines · Igor Mammedov
Quoted 18 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov
QEMU will crash with following debug enabled
# define DEBUG_TLB_GATE 1 # define DEBUG_TLB_LOG_GATE 1 due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work] functions are called not only from vcpu thread but also from reset handler that is called from main thread at cpu realize time when vcpu is already created x86_cpu_new -> ... -> x86_cpu_realizefn -> cpu_reset -> ... -> tcg_cpu_reset_holdQuoted 1 line
drop assert to fix crash.
Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:
Quoted 7 lines
g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));was trying to account for pre-initialised vCPUs. What has changed?
cpu_thread_signal_created(cpu)is called just before we start running the main loop inmttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of
cpu->createdusage.Quoted 8 lines
Quoted 1 line
Why the drive to get rid of
cpu->created?During review of Philippe's cpu cleanups,
I've noticed thatcpu->createdis mostly used for signaling main loop thread we've started vcpu thread with a couple of odd cases in tcg and kvm.
- 1st silently bit-rotted being behind ifdefs
- 2nd is work around CPU being prematurely visible to others2nd is:
commit
56adee407fc564da19e49cfe18e20e3da92320beAuthor: Peter Xu <[email protected]> Date: Fri Feb 17 00:18:32 2023 +0800kvm: dirty-ring: Fix race with vcpu creation
It's possible that we want to reap a dirty ring on a vcpu that is
during
creation, because the vcpu is put onto list (CPU_FOREACHvisible)
before
initialization of the structures. In this case:
qemu_init_vcpu
x86_cpu_realizefn
cpu_exec_realizefn
cpu_list_add<---- can be probed byCPU_FOREACH
qemu_init_vcpu
cpus_accel->create_vcpu_thread(cpu);
kvm_init_vcpu
mapkvm_dirty_gfns<---kvm_dirty_gfnsvalidDon't try to reap dirty ring on vcpus during creation or it'll crash.
Quoted 8 lines
Looking at
cpu_list_add()incpu_exec_realizefn():hw/core/cpu-common.c-190- /* Wait until cpu initialization complete
before exposing cpu. */hw/core/cpu-common.c:191:cpu_list_add(cpu);IMO the problem is with
cpu_list_add(), we shouldn't expose the vCPU until it is realized.
that was my reasoning as well
Quoted 10 lines
cpu_list_add()seems to be doing 2 things, auto-assign CPU index ifUNASSIGNED_CPU_INDEX, then insert to globalcpus_queue.IIRC
cpu_list_add()is called early because various cpu init code expectscpu->indexto be assigned.Maybe we could extract the 'safely assign an unique cpu index' part (guarding by
qemu_cpu_list_lock), havingcpu_list_add()just add to the global queue. I'll give it a try...
ok,
lets see if we can postpone cpu_list_add() till the end of realize time.
Quoted 57 lines
Quoted 56 lines
Quoted 10 lines
I guess we could assert:
g_assert(!current_cpu || qemu_cpu_is_self(cpu);as
current_cpushould only be set as we go into the main thread. However there is a sketchy setting ofcurrent_cpuincpu_exec()that I'm not sure should be there. Alsodo_run_on_cpu()messes withcurrent_cpuin a way I don't fully understand either.I'd rather not rely on that, even if it works it would be subject to to the same kind of breakage.
How about instead of workaround check we would have 2 variants of tlb_flush_by_mmuidx[_async_work], one that have self check and other for usage externally (i.e. from reset handler). That won't have to rely on sketchy globals (which becomes more sketchy in context of Philippes's multi accel work), and it would clearly document what can be used externally.
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
1)
Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")Signed-off-by Igor Mammedov <[email protected]>--- accel/tcg/cputlb.c | 4 ---- 1 file changed, 4 deletions(-)accel/tcg/cputlb.c +0
-4@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)uint16_t all_dirty, work, to_clean;int64_t now =get_clock_realtime (); -assert_cpu_is_self (cpu); -tlb_debug ("mmu_idx:0x%04" PRIx16"\n" , asked);qemu_spin_lock (&cpu->neg.tlb.c.lock);@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap) {tlb_debug ("mmu_idx: 0x%" PRIx16"\n" , idxmap); -assert_cpu_is_self (cpu); -tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); }
(also Cc'ing Harsh for commit 18530e7c57de)
Quoted 161 lines · Igor Mammedov
Quoted 75 lines · On Mon, 27 Jan 2025 14:24:56 +0100 Philippe Mathieu-Daudé <[email protected]>…
+Peter
Quoted 38 lines · Igor Mammedov
Quoted 35 lines · On Thu, 23 Jan 2025 12:09:59 +0000 Alex Bennée <[email protected]> wrote:
Quoted 32 lines · Igor Mammedov
Quoted 18 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov
QEMU will crash with following debug enabled
# define DEBUG_TLB_GATE 1 # define DEBUG_TLB_LOG_GATE 1 due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work] functions are called not only from vcpu thread but also from reset handler that is called from main thread at cpu realize time when vcpu is already created x86_cpu_new -> ... -> x86_cpu_realizefn -> cpu_reset -> ... -> tcg_cpu_reset_holdQuoted 1 line
drop assert to fix crash.
Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:
Quoted 7 lines
g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));was trying to account for pre-initialised vCPUs. What has changed?
cpu_thread_signal_created(cpu)is called just before we start running the main loop inmttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of
cpu->createdusage.Quoted 8 lines
Quoted 1 line
Why the drive to get rid of
cpu->created?During review of Philippe's cpu cleanups,
I've noticed thatcpu->createdis mostly used for signaling main loop thread we've started vcpu thread with a couple of odd cases in tcg and kvm.
- 1st silently bit-rotted being behind ifdefs
- 2nd is work around CPU being prematurely visible to others2nd is:
commit
56adee407fc564da19e49cfe18e20e3da92320beAuthor: Peter Xu <[email protected]> Date: Fri Feb 17 00:18:32 2023 +0800kvm: dirty-ring: Fix race with vcpu creation
It's possible that we want to reap a dirty ring on a vcpu that is
during
creation, because the vcpu is put onto list (CPU_FOREACHvisible)
before
initialization of the structures. In this case:
qemu_init_vcpu
x86_cpu_realizefn
cpu_exec_realizefn
cpu_list_add<---- can be probed byCPU_FOREACH
qemu_init_vcpu
cpus_accel->create_vcpu_thread(cpu);
kvm_init_vcpu
mapkvm_dirty_gfns<---kvm_dirty_gfnsvalidDon't try to reap dirty ring on vcpus during creation or it'll crash.
Quoted 8 lines
Looking at
cpu_list_add()incpu_exec_realizefn():hw/core/cpu-common.c-190- /* Wait until cpu initialization complete
before exposing cpu. */hw/core/cpu-common.c:191:cpu_list_add(cpu);IMO the problem is with
cpu_list_add(), we shouldn't expose the vCPU until it is realized.that was my reasoning as well
Quoted 10 lines
cpu_list_add()seems to be doing 2 things, auto-assign CPU index ifUNASSIGNED_CPU_INDEX, then insert to globalcpus_queue.IIRC
cpu_list_add()is called early because various cpu init code expectscpu->indexto be assigned.Maybe we could extract the 'safely assign an unique cpu index' part (guarding by
qemu_cpu_list_lock), havingcpu_list_add()just add to the global queue. I'll give it a try...ok,
lets see if we can postponecpu_list_add()till the end of realize time.Quoted 57 lines
Quoted 56 lines
Quoted 10 lines
I guess we could assert:
g_assert(!current_cpu || qemu_cpu_is_self(cpu);as
current_cpushould only be set as we go into the main thread. However there is a sketchy setting ofcurrent_cpuincpu_exec()that I'm not sure should be there. Alsodo_run_on_cpu()messes withcurrent_cpuin a way I don't fully understand either.I'd rather not rely on that, even if it works it would be subject to to the same kind of breakage.
How about instead of workaround check we would have 2 variants of tlb_flush_by_mmuidx[_async_work], one that have self check and other for usage externally (i.e. from reset handler). That won't have to rely on sketchy globals (which becomes more sketchy in context of Philippes's multi accel work), and it would clearly document what can be used externally.
Quoted 33 lines
Quoted 32 lines
Quoted 32 lines
Quoted 31 lines
1)
Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")Signed-off-by Igor Mammedov <[email protected]>--- accel/tcg/cputlb.c | 4 ---- 1 file changed, 4 deletions(-)accel/tcg/cputlb.c +0
-4@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)uint16_t all_dirty, work, to_clean;int64_t now =get_clock_realtime (); -assert_cpu_is_self (cpu); -tlb_debug ("mmu_idx:0x%04" PRIx16"\n" , asked);qemu_spin_lock (&cpu->neg.tlb.c.lock);@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap) {tlb_debug ("mmu_idx: 0x%" PRIx16"\n" , idxmap); -assert_cpu_is_self (cpu); -tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); }
Quoted 23 lines · On Thu, 23 Jan 2025 12:23:43 +0100 Igor Mammedov <[email protected]> wrote:
Quoted 20 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov
QEMU will crash with following debug enabled
# define DEBUG_TLB_GATE 1 # define DEBUG_TLB_LOG_GATE 1 due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work] functions are called not only from vcpu thread but also from reset handler that is called from main thread at cpu realize time when vcpu is already created x86_cpu_new -> ... -> x86_cpu_realizefn -> cpu_reset -> ... -> tcg_cpu_reset_holdQuoted 1 line
drop assert to fix crash.
Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:
g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));
Quoted 12 lines
Quoted 5 lines
was trying to account for pre-initialised vCPUs. What has changed?
cpu_thread_signal_created(cpu)is called just before we start running the main loop inmttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of
cpu->createdusage.Quoted 2 lines
Quoted 2 lines
1)
Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")
bisection points to (so above fixes was a wrong one):
30933c4fb4f3d tcg/cputlb: remove other-cpu capability from TLB flushing
which has replaced a check with assert:
- if (cpu->created && !qemu_cpu_is_self(cpu)) {
- async_run_on_cpu(cpu, tlb_flush_by_mmuidx_async_work,
- RUN_ON_CPU_HOST_INT(idxmap));
- } else {
- tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap));
- }
+ assert_cpu_is_self(cpu);
should we revert that instead?
perhaps also drop 'cpu->created' check in assert_cpu_is_self as it obviously doesn't work.
Quoted 28 lines
Quoted 27 lines
Quoted 26 lines
Signed-off-by Igor Mammedov <[email protected]>--- accel/tcg/cputlb.c | 4 ---- 1 file changed, 4 deletions(-)accel/tcg/cputlb.c +0
-4@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)uint16_t all_dirty, work, to_clean;int64_t now =get_clock_realtime (); -assert_cpu_is_self (cpu); -tlb_debug ("mmu_idx:0x%04" PRIx16"\n" , asked);qemu_spin_lock (&cpu->neg.tlb.c.lock);@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap) {tlb_debug ("mmu_idx: 0x%" PRIx16"\n" , idxmap); -assert_cpu_is_self (cpu); -tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); }
Quoted 51 lines · Igor Mammedov
Quoted 23 lines · On Thu, 23 Jan 2025 12:23:43 +0100 Igor Mammedov <[email protected]> wrote:
Quoted 20 lines · On Thu, 23 Jan 2025 10:52:15 +0000 Alex Bennée <[email protected]> wrote:
Quoted 11 lines · Igor Mammedov
QEMU will crash with following debug enabled
# define DEBUG_TLB_GATE 1 # define DEBUG_TLB_LOG_GATE 1 due to [1] introduced assert and as it happenstlb_flush_by_mmuidx[_async_work] functions are called not only from vcpu thread but also from reset handler that is called from main thread at cpu realize time when vcpu is already created x86_cpu_new -> ... -> x86_cpu_realizefn -> cpu_reset -> ... -> tcg_cpu_reset_holdQuoted 1 line
drop assert to fix crash.
Hmm the assert is there for a good reason because we do not want to be flushing another CPUs state. However the assert itself:
g_assert(!(cpu)->created || qemu_cpu_is_self(cpu));Quoted 12 lines
Quoted 5 lines
was trying to account for pre-initialised vCPUs. What has changed?
cpu_thread_signal_created(cpu)is called just before we start running the main loop inmttcg_cpu_thread_fn. So any other thread messing with the CPUs TLB can potentially mess things up.it reproduces on current master, so yes it likely has changed over time. I've just stumbled on it when attempting to get rid of
cpu->createdusage.Quoted 2 lines
Quoted 2 lines
1)
Fixes f0aff0f124028 ("cputlb: add assert_cpu_is_self checks")bisection points to (so above fixes was a wrong one):
30933c4fb4f3dtcg/cputlb: remove other-cpu capability from TLB flushing
which has replaced a check with assert:(quoted diff) +1
-6- if (cpu->created && !qemu_cpu_is_self(cpu)) { - async_run_on_cpu(cpu, tlb_flush_by_mmuidx_async_work, - RUN_ON_CPU_HOST_INT(idxmap)); - } else { - tlb_flush_by_mmuidx_async_work(cpu, RUN_ON_CPU_HOST_INT(idxmap)); - } + assert_cpu_is_self(cpu);
Quoted 4 lines
should we revert that instead?
perhaps also drop '
cpu->created' check inassert_cpu_is_selfas it obviously doesn't work.
That's because the asserts only check when built with debug due to concerns about the hot path. I think in the case of cputlb we should probably use tcg_debug_assert() which at least gets enabled for --enable-debug builds without needing to manually patch the define for DEBUG_TLB.
Converting the custom TLB log implementation to using tracepoints would also be nice.
Quoted 30 lines
Quoted 30 lines
Quoted 29 lines
Quoted 28 lines
Signed-off-by Igor Mammedov <[email protected]>--- accel/tcg/cputlb.c | 4 ---- 1 file changed, 4 deletions(-)accel/tcg/cputlb.c +0
-4@@ -381,8 +381,6 @@ static void tlb_flush_by_mmuidx_async_work(CPUState *cpu, run_on_cpu_data data)uint16_t all_dirty, work, to_clean;int64_t now =get_clock_realtime (); -assert_cpu_is_self (cpu); -tlb_debug ("mmu_idx:0x%04" PRIx16"\n" , asked);qemu_spin_lock (&cpu->neg.tlb.c.lock);@@ -419,8 +417,6 @@ void tlb_flush_by_mmuidx(CPUState *cpu, uint16_t idxmap) {tlb_debug ("mmu_idx: 0x%" PRIx16"\n" , idxmap); -assert_cpu_is_self (cpu); -tlb_flush_by_mmuidx_async_work (cpu,RUN_ON_CPU_HOST_INT (idxmap)); }
Signature · 2 lines
Alex Bennée Virtualisation Tech Lead @ Linaro