accel/tcg: cut per-block dispatch overhead
cover accel/tcg: cut per-block dispatch overhead
For guests running large amounts of code, most of what TCG executes is not translated guest work but the fixed overhead around it. Blocks are short and there are a great many of them, so the constant cost at each end of a block (the interrupt poll and the can_do_io stores on entry, the dispatch on exit) ends up dominating everything else.
The workload throughout is qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, in a --static --enable-lto --target-list=alpha-linux-user build. It executes 34.2 billion TBs at 6.04 guest instructions each, and 24.6% of its TB exits cannot use goto_tb. That is a representative shape for any guest whose text is much larger than a page: indirect calls and returns everywhere, plus direct branches that merely crossed a page boundary.
The first three patches are ordinary cleanups that stand on their own. The remaining five are marked RFC individually and are where the interesting questions are.
1 accel/tcg: cache the result of curr_cflags()
Recomputed on every one of the run's 8.4 billion dispatches, from state that changes only when gdb enables single-step or a log mask moves. Cache it in CPUState and recompute from the four places that
can change an input. -5.10%
2 accel/tcg: enlarge the TB jump cache to 64K entries
4096 entries is too small for a guest running a large program; tb_htable_lookup() is 5.73% of samples. 16 bits is the knee of the
sizing curve, at 1 MiB per vCPU. -6.02%
3 accel/tcg: skip the can_do_io stores in user-only builds
Two stores per TB that nothing in a user-only build reads: 68 billion
of them over the run. -4.55%, -4.32% wall
4 RFC: tcg: probe the TB jump cache inline instead of calling a helper
95.8% of those 8.4 billion helper_lookup_tb_ptr() calls hit the jump cache. Emit the probe inline (hash, three guarded loads, goto_ptr)
and call the helper only on a miss. -36.51%, -27.05% wall
5 RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
translator_use_goto_tb() refuses to chain across a page. In user-only builds the invalidation path already covers what that was protecting
against: every mmap/mprotect/munmap reaches page_set_flags(), which invalidates and unlinks. Lift it there, keep it for system mode.
-2.42%, -4.68% wall
6 RFC: accel/tcg: only poll for interrupts in blocks that can close a
cycle
The icount_decr poll needs to happen once per cycle in the guest CFG, not once per block, and any cycle must contain either a backward edge or an indirect one. Record both during translation and emit the check
only for blocks that have one. -6.81%, -3.10% wall
7 RFC: accel/tcg: poison the jump cache instead of polling for indirect
exits
What patch 6 leaves behind is mostly blocks flagged for an indirect exit. Give the inline probe its own jump cache base pointer and point it at zeroes when an exit is requested: every dispatch then misses into the helper, which returns the epilogue. The poll becomes a
pointer swap on the request path. -2.79%, -1.94% wall
8 RFC: tcg: fold a guest displacement into the host addressing mode
tcg_gen_qemu_ld/st cannot express a based access, so a target with a displacement in its encodings materializes the address with an lea that the host addressing mode would have done for free. Fold a preceding constant add into a new argument on the op, opt-in per
backend, wired up for x86_64 user-only. -6.29%, -3.29% wall
Each percentage is against the patch before it. End to end, measuring an unmodified build of the same base against the full series, five runs each, interleaved in one session so that host clock drift is shared rather than attributed (mean, with the run-to-run spread):
instructions retired: 1,646,129,294,236 -> 738,003,153,831 -55.17%
(0.16%) (0.03%)
wall clock: 134.934s -> 75.189s -44.28%
(0.30%) (0.99%)Both endpoints ran at the same 4.782 GHz effective clock, and the .s files they produced are identical.
The two figures do not track each other, and that is the interesting part:
what the series removes is cheap, well-predicted, highly pipelined work, so it retires far more instructions than it saves time. IPC falls from 2.55 to 2.05 as the remaining work gets less regular. Patch 4 also cuts L1-icache load misses by 39.1%, because a dispatch no longer jumps into qemu's .text and evicts translated code; qemu's own .text falls from 38.9% to 5.4% of profile samples over the series.
Every revision was built and measured separately, so the series bisects, and the emulated compiler produces byte-identical assembly output at every step, which is the correctness check these patches most need. Two new alpha tests cover the hazards the series creates: tests/tcg/alpha/test-xpage-chain.c (patch 5) and test-indirect-irq.c (patch 7). Both fail or hang if the mechanism they cover is removed, which is what makes them tests of the new behavior rather than of the old.
The RFC patches need eyes I cannot supply myself. In rough order of how much I would like someone to look at them:
- Patch 5 reverses a deliberate decision made in d3a2a1d803 on the strength of an argument about the user-only invalidation paths.
- Patch 6 moves system-mode interrupt latency from "bounded by block count" to "bounded by guest control flow". The bound is one straight-line run between cycles, but timer-driven guests want a closer look than I can give them. Its soundness also assumes every goto_tb destination passes through translator_use_goto_tb(); no target in the tree bypasses it today, but nothing enforces that.
- Patch 4 treats cpu flags and cflags as translation-time constants in its guards, reads a jump cache entry without qatomic_read(), and puts knowledge of the CPUJumpCache layout in tcg/tcg-op.c, where it does not belong.
- Patch 7's restore in cpu_handle_interrupt() races a concurrent poison from another thread. I believe the existing barrier around icount_decr.u16.high covers it, but my testing was single-threaded user mode.
- Patch 8 only examines the immediately preceding op, refuses any access with a slow path (so user-only, and no alignment check), and leaves the i128 pairs alone.
- Patch 2's 1 MiB per vCPU is easy to justify for a single-vCPU linux-user process and less obvious for system emulation with many vCPUs. It may want to be sized per target or made tunable rather than raised unconditionally.
Patches 4 and 8 are wired up for alpha and x86_64 respectively; everything else is target-independent, and no other backend changes behavior or needs touching.
Matt Turner (8):
accel/tcg: cache the result of curr_cflags()
accel/tcg: enlarge the TB jump cache to 64K entries accel/tcg: skip the can_do_io stores in user-only builds
RFC: tcg: probe the TB jump cache inline instead of calling a helper
RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
RFC: accel/tcg: only poll for interrupts in blocks that can close a
cycle
RFC: accel/tcg: poison the jump cache instead of polling for indirect
exits
RFC: tcg: fold a guest displacement into the host addressing mode
accel/tcg/cpu-exec-common.c | 48 +++++++++++- accel/tcg/cpu-exec.c | 54 ++++++++++++++ accel/tcg/internal-common.h | 22 +++++- accel/tcg/tb-jmp-cache.h | 2 +- accel/tcg/tcg-accel-ops.c | 2 + accel/tcg/tcg-all.c | 1 + accel/tcg/translator.c | 77 ++++++++++++++++++- cpu-target.c | 3 + include/exec/translation-block.h | 6 ++ include/exec/translator.h | 2 + include/hw/core/cpu.h | 23 +++++- include/system/tcg.h | 9 +++ include/tcg/tcg-op-common.h | 2 + include/tcg/tcg-opc.h | 9 ++- linux-user/main.c | 2 +- stubs/meson.build | 1 + stubs/tcg-cflags.c | 16 ++++ target/alpha/cpu.c | 2 +- target/alpha/translate.c | 6 +- tcg/tcg-op-ldst.c | 3 +- tcg/tcg-op.c | 86 +++++++++++++++++++++ tcg/tcg.c | 86 ++++++++++++++++++++- tcg/x86_64/tcg-target.c.inc | 61 +++++++++++++++ tcg/x86_64/tcg-target.h | 3 + tests/tcg/alpha/Makefile.target | 3 +- tests/tcg/alpha/test-indirect-irq.c | 53 +++++++++++++ tests/tcg/alpha/test-xpage-chain.c | 111 ++++++++++++++++++++++++++++ util/log.c | 4 + 28 files changed, 676 insertions(+), 21 deletions(-) create mode 100644 stubs/tcg-cflags.c create mode 100644 tests/tcg/alpha/test-indirect-irq.c create mode 100644 tests/tcg/alpha/test-xpage-chain.c
Signature · 1 line
2.54.0
RFC PATCH 1/8 accel/tcg: cache the result of curr_cflags()
curr_cflags() is called once per TB dispatch, from helper_lookup_tb_ptr() and from the cpu_exec() loop. It recomputes the same value every time:
uint32_t cflags = cpu->tcg_cflags;
if (unlikely(cpu_single_stepping(cpu))) { ... }
else if (qatomic_read(&one_insn_per_tb)) { ... }
else if (qemu_loglevel_mask(CPU_LOG_TB_NOCHAIN)) { ... }That is three loads and three branches on the hottest path in the interpreter, for state that changes only when gdb enables single-step, when one-insn-per-tb is toggled, or when the log mask changes.
Compute the value once into CPUState::tcg_curr_cflags and recompute it from the four places that can change an input: tcg_cflags_set(), cpu_single_step(), tcg_set_one_insn_per_tb() and qemu_set_log_internal(). curr_cflags() becomes a single load.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, in a build configured with --enable-lto:
before: 1,647,901,588,726 instructions
after: 1,563,829,403,943 instructions -5.10%That workload issues 8.4 billion dispatches, so the per-call saving is small but the aggregate is not. The emulated compiler produces byte-identical output before and after.
Wall clock does not move: 133.57s to 133.13s, a 0.33% difference against a run-to-run spread of the same size. The removed work is a few predictable loads and branches that the host executes largely in parallel with the surrounding dispatch, so this patch is worth taking for the instruction count and for what it enables, not for a time saving that can be measured on its own.
Note that tcg_set_one_insn_per_tb() does not tb_flush(), so the cache cannot piggyback on TB flushing and needs its own update call.
Caching makes the pre-computed cflags one half of a pair that has to be kept in step, and nothing in C enforces that. The cost of getting it wrong is not a crash but silently wrong code generation: a stale cache that is missing CF_PARALLEL makes TCG emit the non-atomic form of guest atomics, and the guest then corrupts its own mutexes. linux-user's cpu_copy() is exactly such a trap. do_fork() calls begin_parallel_context() on the parent before cpu_copy(), so the child inherits CF_PARALLEL and never calls tcg_cflags_set() itself; assigning the field directly would leave every cloned thread dispatching with a cflags of zero.
Close the hole from both ends.
Name the field tcg_cflags_priv and add tcg_cflags_get(), so that tcg_cflags_has()/get()/set() are the only ways to reach it. C cannot really make a struct member private, but an open-coded access now fails to compile rather than quietly going stale, which is enough to force a rebased or newly written user to look at the accessors. target/alpha's CF_PCREL setup is converted along with it: it would be benign either way today, because tcg_cpu_init_cflags() refreshes the cache afterwards in system mode, but it is the same pattern and only ordering saved it.
Then have curr_cflags() recompute the value and compare, under CONFIG_DEBUG_TCG. That is the check that catches a missed update on the first dispatch, rather than days later by way of corrupted guest mutexes. It cannot be unconditional, as recomputing on every dispatch is the very cost the cache exists to avoid.
Verified by dropping the tcg_cflags_set() call from cpu_copy() against a debug-tcg build: the assert fires immediately, reporting the cached and recomputed values and the bits that differ.
--- accel/tcg/cpu-exec-common.c | 48 +++++++++++++++++++++++++++++--- accel/tcg/internal-common.h | 19 ++++++++++++- accel/tcg/tcg-all.c | 1 + cpu-target.c | 3 ++ include/exec/translation-block.h | 6 ++++ include/hw/core/cpu.h | 13 +++++++-- include/system/tcg.h | 9 ++++++ linux-user/main.c | 2 +- stubs/meson.build | 1 + stubs/tcg-cflags.c | 16 +++++++++++ target/alpha/cpu.c | 2 +- util/log.c | 4 +++ 12 files changed, 114 insertions(+), 10 deletions(-) create mode 100644 stubs/tcg-cflags.c
accel/tcg/cpu-exec-common.c +44 -4
@@ -28,17 +28,23 @@ bool tcg_allowed;bool tcg_cflags_has (CPUState *cpu,uint32_t flags) { -return cpu->tcg_cflags & flags; +return cpu->tcg_cflags_priv & flags; +} + +uint32_t tcg_cflags_get (CPUState *cpu) +{ +return cpu->tcg_cflags_priv; }void tcg_cflags_set (CPUState *cpu,uint32_t flags) { - cpu->tcg_cflags |= flags; + cpu->tcg_cflags_priv |= flags; +tcg_update_curr_cflags (cpu); } -uint32_t curr_cflags (CPUState *cpu) +static uint32_t compute_curr_cflags (CPUState *cpu) { -uint32_t cflags = cpu->tcg_cflags; +uint32_t cflags = cpu->tcg_cflags_priv;/* * Record gdb single-step. We should be exiting the TB by raising
@@ -58,6 +64,40 @@ uint32_t curr_cflags(CPUState *cpu)return cflags; } +void tcg_update_curr_cflags (CPUState *cpu) +{ + cpu->tcg_curr_cflags =compute_curr_cflags (cpu); +} + +void tcg_update_all_curr_cflags (void ) +{ +CPUState *cpu; + +CPU_FOREACH (cpu) { +tcg_update_curr_cflags (cpu); + } +} + +#ifdef CONFIG_DEBUG_TCG +/* +* Catch a cached value that has gone stale because an input changed without +* a matching tcg_update_curr_cflags(). Called from curr_cflags() on the +* dispatch path, so it exists only in debug-tcg builds. +*/ +void tcg_assert_curr_cflags (CPUState *cpu) +{ +uint32_t cached = cpu->tcg_curr_cflags; +uint32_t fresh =compute_curr_cflags (cpu); + +if (unlikely (cached != fresh)) { +fprintf (stderr,"stale tcg_curr_cflags on CPU %d: " +"cached 0x%08x, recomputed 0x%08x (differ in 0x%08x)\n" , + cpu->cpu_index, cached, fresh, cached ^ fresh); +g_assert_not_reached (); + } +} +#endif +/* exit the current TB, but without causing any exception to be raised */ void cpu_loop_exit_noexc (CPUState *cpu) {
accel/tcg/internal-common.h +18 -1
@@ -70,7 +70,24 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp);void tcg_exec_unrealizefn (CPUState *cpu);/* current cflags for hashing/comparison */ -uint32_t curr_cflags (CPUState *cpu); +/* +* Cached by tcg_update_curr_cflags(). This is on the hot TB dispatch +* path, so it must stay a single load; see commit message. A debug-tcg +* build pays for a recompute here to prove the cache is still in step, +* which turns a missed update into a loud failure rather than subtly +* wrong code generation. +*/ +#ifdef CONFIG_DEBUG_TCG +void tcg_assert_curr_cflags (CPUState *cpu); +#endif + +static inline uint32_t curr_cflags (CPUState *cpu) +{ +#ifdef CONFIG_DEBUG_TCG +tcg_assert_curr_cflags (cpu); +#endif +return cpu->tcg_curr_cflags; +}void tb_check_watchpoint (CPUState *cpu,uintptr_t retaddr);
accel/tcg/tcg-all.c +1 -0
@@ -254,6 +254,7 @@ static void tcg_set_one_insn_per_tb(Object *obj, bool value, Error **errp) s->one_insn_per_tb = value;/* Set the global also: this changes the behaviour */ qatomic_set (&one_insn_per_tb, value); +tcg_update_all_curr_cflags (); }static void tcg_accel_class_init (ObjectClass *oc,const void *data)
cpu-target.c +3 -0
@@ -24,6 +24,7 @@#include "exec/replay-core.h" #include "exec/log.h" #include "hw/core/cpu.h" +#include "system/tcg.h" #include "trace/trace-root.h" /* enable or disable single step mode. EXCP_DEBUG is returned by the
@@ -35,6 +36,8 @@ void cpu_single_step(CPUState *cpu, unsigned flags) cpu->singlestep_flags, flags); cpu->singlestep_flags = flags; +tcg_update_curr_cflags (cpu); +#if !defined(CONFIG_USER_ONLY )const AccelOpsClass *ops =cpus_get_accel ();if (ops->update_guest_debug) {
include/exec/translation-block.h +6 -0
@@ -158,7 +158,13 @@ static inline uint32_t tb_cflags(const TranslationBlock *tb)return qatomic_read (&tb->cflags); } +/* +* CPUState::tcg_cflags_priv is reached only through these. The setter keeps +* the derived CPUState::tcg_curr_cflags in step, and assigning the field +* directly would silently leave that cache stale. +*/ bool tcg_cflags_has (CPUState *cpu,uint32_t flags); +uint32_t tcg_cflags_get (CPUState *cpu);void tcg_cflags_set (CPUState *cpu,uint32_t flags);static inline tb_page_addr_t tb_page_addr0 (const TranslationBlock *tb)
include/hw/core/cpu.h +10 -3
@@ -411,10 +411,16 @@ struct qemu_work_item;* to a cluster this will be UNASSIGNED_CLUSTER_INDEX; otherwise it will * be the same as the cluster-id property of the CPU object's TYPE_CPU_CLUSTER * QOM parent. -* Under TCG this value is propagated to @tcg_cflags. +* Under TCG this value is propagated to @tcg_cflags_priv. * See TranslationBlock::TCG CF_CLUSTER_MASK. * @start_powered_off: Indicates whether the CPU starts in powered-off state. -* @tcg_cflags: Pre-computed cflags for this cpu. +* @tcg_cflags_priv: Pre-computed cflags for this cpu. Private to +* tcg_cflags_has() and tcg_cflags_set(): @tcg_curr_cflags is derived from +* it and is refreshed by the setter, so a direct assignment here would +* leave the two out of step. The name is deliberately awkward to make an +* open-coded access fail to compile rather than silently go stale. +* @tcg_curr_cflags: Cached result of curr_cflags(), recomputed by +* tcg_update_curr_cflags() whenever any of its inputs change. * @nr_threads: Number of threads within this CPU core. * @thread: Host thread details, only live once @created is #true * @sem: WIN32 only semaphore used only for qtest
@@ -557,7 +563,8 @@ struct CPUState {/* TODO Move common fields from CPUArchState here. */ int cpu_index;int cluster_index; -uint32_t tcg_cflags; +uint32_t tcg_cflags_priv; +uint32_t tcg_curr_cflags;uint32_t halted;int32_t exception_index;
include/system/tcg.h +9 -0
@@ -17,6 +17,15 @@ extern bool tcg_allowed;#define tcg_enabled () 0#endif +/* +* Recompute CPUState::tcg_curr_cflags. Must be called whenever any input +* to the computation changes: CPUState::tcg_cflags_priv, gdb single-step +* state, one-insn-per-tb, or the CPU_LOG_TB_NOCHAIN log flag. The first of +* those is covered already, tcg_cflags_set() being the only way to change it. +*/ +void tcg_update_curr_cflags (CPUState *cpu); +void tcg_update_all_curr_cflags (void ); +/** * qemu_tcg_mttcg_enabled: * Check whether we are running MultiThread TCG or not.
linux-user/main.c +1 -1
@@ -242,7 +242,7 @@ CPUArchState *cpu_copy(CPUArchState *env)/* Reset non arch specific state */ cpu_reset (new_cpu); - new_cpu->tcg_cflags = cpu->tcg_cflags; +tcg_cflags_set (new_cpu,tcg_cflags_get (cpu));memcpy (new_env, env,sizeof (CPUArchState));#if defined(TARGET_I386 ) || defined(TARGET_X86_64 ) new_env->gdt.base = target_mmap(0, sizeof(uint64_t) * TARGET_GDT_ENTRIES,
stubs/meson.build +1 -0
@@ -3,6 +3,7 @@ # below, so that it is clear who needs the stubbed functionality. stub_ss.add(files('cpu-get-clock.c')) +stub_ss.add(files('tcg-cflags.c')) stub_ss.add(files('fdset.c')) stub_ss.add(files('iothread-lock.c')) stub_ss.add(files('is-daemonized.c'))
stubs/tcg-cflags.c new file +16 -0
@@ -0,0 +1,16 @@ +/* +* Stubs for the cached cflags update hooks, for binaries that link +* util/log.c or cpu-target.c without linking TCG. +* +* SPDX-License-Identifier: GPL-2.0-or-later +*/ +#include "qemu/osdep.h" +#include "system/tcg.h" + +void tcg_update_curr_cflags (CPUState *cpu) +{ +} + +void tcg_update_all_curr_cflags (void ) +{ +}
target/alpha/cpu.c +1 -1
@@ -114,7 +114,7 @@ static void alpha_cpu_realizefn(DeviceState *dev, Error **errp)#ifndef CONFIG_USER_ONLY /* Use pc-relative instructions in system-mode */ - cs->tcg_cflags |=CF_PCREL ; +tcg_cflags_set (cs,CF_PCREL );#endif cpu_exec_realizefn (cs, &local_err);
util/log.c +4 -0
@@ -27,6 +27,7 @@#include "qemu/thread.h" #include "qemu/lockable.h" #include "qemu/rcu.h" +#include "system/tcg.h" #ifdef CONFIG_LINUX #include <sys/syscall.h> #endif
@@ -301,6 +302,9 @@ static bool qemu_set_log_internal(const char *filename, bool changed_name,#endif qemu_loglevel = log_flags; +/* CPU_LOG_TB_NOCHAIN feeds into the per-CPU cached cflags. */ +tcg_update_all_curr_cflags (); + daemonized =is_daemonized (); need_to_open_file = false;if (!daemonized) {
Signature · 1 line
2.54.0
Re: [RFC PATCH 1/8] accel/tcg: cache the result of curr_cflags()
Quoted 11 lines · Matt Turner
curr_cflags()is called once per TB dispatch, fromhelper_lookup_tb_ptr()and from thecpu_exec()loop. It recomputes the same value every time:uint32_t cflags = cpu->tcg_cflags; if (unlikely(cpu_single_stepping(cpu))) { ... } else if (qatomic_read(&one_insn_per_tb)) { ... } else if (qemu_loglevel_mask(CPU_LOG_TB_NOCHAIN)) { ... }That is three loads and three branches on the hottest path in the interpreter, for state that changes only when gdb enables single-step, when one-insn-per-tb is toggled, or when the log mask changes.
I agree this would be useful to remove but...
Quoted 12 lines
static inline
tb_page_addr_ttb_page_addr0(const TranslationBlock *tb)include/hw/core/cpu.h +1
-1@@ -411,10 +411,16 @@ struct qemu_work_item;* to a cluster this will be UNASSIGNED_CLUSTER_INDEX; otherwise it will * be the same as the cluster-id property of the CPU object's TYPE_CPU_CLUSTER * QOM parent. -* Under TCG this value is propagated to @tcg_cflags. +* Under TCG this value is propagated to @tcg_cflags_priv.
include/hw/core/cpu.h struct qemu_work_item; · line 423
Quoted 10 lines
include/hw/core/cpu.h +7
-1* See TranslationBlock::TCG CF_CLUSTER_MASK. * @start_powered_off: Indicates whether the CPU starts in powered-off state. -* @tcg_cflags: Pre-computed cflags for this cpu. +* @tcg_cflags_priv: Pre-computed cflags for this cpu. Private to +* tcg_cflags_has() and tcg_cflags_set(): @tcg_curr_cflags is derived from +* it and is refreshed by the setter, so a direct assignment here would +* leave the two out of step. The name is deliberately awkward to make an +* open-coded access fail to compile rather than silently go stale. +* @tcg_curr_cflags: Cached result of curr_cflags(), recomputed by +* tcg_update_curr_cflags() whenever any of its inputs change.
This seems like introducing additional cache state to track. Why not just provide a helper to call when ss/gdb/log conditions change and change the current curr_cflags() code to only validate the helper has been called when TCG debugging is turned on. See ppc_get_tb_cpu_state or assert_hflags_rebuild_correctly for examples.
Quoted 13 lines
include/hw/core/cpu.h +2
-1* @nr_threads: Number of threads within this CPU core. * @thread: Host thread details, only live once @created is #true * @sem: WIN32 only semaphore used only for qtest @@ -557,7 +563,8 @@ struct CPUState {/* TODO Move common fields from CPUArchState here. */ int cpu_index;int cluster_index; -uint32_t tcg_cflags; +uint32_t tcg_cflags_priv; +uint32_t tcg_curr_cflags;uint32_t halted;int32_t exception_index;
<snip>
Signature · 2 lines
Alex Bennée Virtualisation Tech Lead @ Linaro
RFC PATCH 2/8 accel/tcg: enlarge the TB jump cache to 64K entries
In reply to Matt TurnerThe per-CPU TB jump cache has held 4096 entries since it was introduced. That is too small for guests running large programs: an emulated compiler misses often enough that the fallback qht lookup shows up prominently in a profile.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host. The compile performs 34.2 billion TB executions, of which 8.4 billion take the indirect dispatch path.
Sizing curve, on top of the preceding patch, instructions retired and wall clock:
12 bits ( 64 KiB): 1,563,829,403,943 133.13s
14 bits ( 256 KiB): 1,493,865,985,972 -4.47% 124.89s -6.19%
16 bits ( 1 MiB): 1,469,729,281,442 -6.02% 120.97s -9.13%
18 bits ( 4 MiB): 1,462,262,363,257 -6.49% 120.16s -9.74%16 bits is the knee. 18 buys another 0.47% of instructions for four times the memory, and since instructions retired does not account for the data cache pressure of a 4 MiB table, that 0.47% is probably not real: the wall clock difference between 16 and 18 bits is 0.67%, against a run-to-run spread of the same order.
In a perf profile the mechanism is visible directly: tb_htable_lookup(), which is where qht_lookup_custom() lands once it is inlined in an LTO build, falls from 5.73% of samples to 1.52%.
The cost is memory: the cache grows from 64 KiB to 1 MiB per vCPU. That is easy to justify for a single-vCPU linux-user process and less obvious for system emulation with many vCPUs, so this may want to be sized by target or made tunable rather than raised unconditionally.
--- accel/tcg/tb-jmp-cache.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
accel/tcg/tb-jmp-cache.h +1 -1
@@ -12,7 +12,7 @@#include "qemu/rcu.h" #include "exec/cpu-common.h" -#define TB_JMP_CACHE_BITS 12 +#define TB_JMP_CACHE_BITS 16#define TB_JMP_CACHE_SIZE (1 << TB_JMP_CACHE_BITS)/*
Signature · 1 line
2.54.0
Re: [RFC PATCH 2/8] accel/tcg: enlarge the TB jump cache to 64K entries
Quoted 10 lines · Matt Turner
The per-CPU TB jump cache has held 4096 entries since it was introduced. That is too small for guests running large programs: an emulated compiler misses often enough that the fallback qht lookup shows up prominently in a profile.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host. The compile performs 34.2 billion TB executions, of which 8.4 billion take the indirect dispatch path.
Quoted 7 lines
Sizing curve, on top of the preceding patch, instructions retired and wall clock:
12 bits ( 64 KiB): 1,563,829,403,943 133.13s 14 bits ( 256 KiB): 1,493,865,985,972 -4.47% 124.89s -6.19% 16 bits ( 1 MiB): 1,469,729,281,442 -6.02% 120.97s -9.13% 18 bits ( 4 MiB): 1,462,262,363,257 -6.49% 120.16s -9.74%
I'd be curious to see the system emulation numbers. You should see very different profiles as more stuff gets directly chained in user mode.
Quoted 6 lines
16 bits is the knee. 18 buys another 0.47% of instructions for four times the memory, and since instructions retired does not account for the data cache pressure of a 4 MiB table, that 0.47% is probably not real: the wall clock difference between 16 and 18 bits is 0.67%, against a run-to-run spread of the same order.
Quoted 8 lines
In a perf profile the mechanism is visible directly:
tb_htable_lookup(), which is whereqht_lookup_custom()lands once it is inlined in an LTO build, falls from 5.73% of samples to 1.52%.The cost is memory: the cache grows from 64 KiB to 1 MiB per vCPU. That is easy to justify for a single-vCPU linux-user process and less obvious for system emulation with many vCPUs, so this may want to be sized by target or made tunable rather than raised unconditionally.
Practically you wouldn't see many TCG emulations with more the 16 CPUs (although it would be interesting to see where the MTTCG gains fade with these proposals). With that in mind 1MiB doesn't see too excessive - I don't know how it compares to other big allocations QEMU makes over a typical run.
Quoted 19 lines
Signed-off-by Matt Turner <[email protected]>--- accel/tcg/tb-jmp-cache.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)accel/tcg/tb-jmp-cache.h +1
-1@@ -12,7 +12,7 @@#include "qemu/rcu.h" #include "exec/cpu-common.h" -#define TB_JMP_CACHE_BITS 12 +#define TB_JMP_CACHE_BITS 16#define TB_JMP_CACHE_SIZE (1 << TB_JMP_CACHE_BITS)/*
Signature · 2 lines
Alex Bennée Virtualisation Tech Lead @ Linaro
RFC PATCH 3/8 accel/tcg: skip the can_do_io stores in user-only builds
In reply to Matt TurnerEvery translation block stores to cpu->neg.can_do_io twice: false before the first instruction, true before the last one. Nothing reads it in a user-only build. There is no memory-mapped I/O in linux-user, and every reader is in system_ss: cputlb.c, watchpoint.c, icount-common.c and tcg-accel-ops-icount.c.
Two stores per TB is not much on its own, but TBs are short. An emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) executes 34.2 billion TBs at 6.04 guest instructions each, so this is 68 billion stores for nothing.
Measured on an x86-64 host, LTO build, on top of the preceding two patches:
before: 1,469,729,281,442 instructions
after: 1,402,816,253,499 instructions -4.55%
before: 120.97s wall clock
after: 115.75s wall clock -4.32%The emulated compiler produces byte-identical output.
--- accel/tcg/translator.c | 10 ++++++++++ 1 file changed, 10 insertions(+)
accel/tcg/translator.c +10 -0
@@ -21,12 +21,14 @@#include "disas/disas.h" #include "tb-internal.h" +#ifndef CONFIG_USER_ONLY static void set_can_do_io (DisasContextBase *db,bool val) {QEMU_BUILD_BUG_ON (sizeof_field (CPUState, neg.can_do_io) !=1 );tcg_gen_st8_i32 (tcg_constant_i32 (val), tcg_env, offsetof(CPUState, neg.can_do_io) - sizeof(CPUState)); } +#endif bool translator_io_start (DisasContextBase *db) {
@@ -210,17 +212,25 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns,/* * Manage can_do_io for the translation block: set to false before * the first insn and set to true before the last insn. +* +* Nothing reads can_do_io in user-only builds. There is no MMIO +* there, and every reader (cputlb.c, watchpoint.c, icount) is in +* system_ss, so skip the two stores per TB entirely. */ if (db->num_insns ==1 ) {tcg_debug_assert (first_insn_start == db->insn_start); }else {tcg_debug_assert (first_insn_start != db->insn_start); +#ifndef CONFIG_USER_ONLY tcg_ctx->emit_before_op = first_insn_start;set_can_do_io (db, false); +#endif } +#ifndef CONFIG_USER_ONLY tcg_ctx->emit_before_op = db->insn_start;set_can_do_io (db, true); tcg_ctx->emit_before_op =NULL ; +#endif /* May be used by disas_log or plugin callbacks. */ tb->size = db->pc_next - db->pc_first;
Signature · 1 line
2.54.0
Re: [RFC PATCH 3/8] accel/tcg: skip the can_do_io stores in user-only builds
Quoted 11 lines · Matt Turner
Every translation block stores to
cpu->neg.can_do_iotwice: false before the first instruction, true before the last one. Nothing reads it in a user-only build. There is no memory-mapped I/O in linux-user, and every reader is insystem_ss:cputlb.c,watchpoint.c,icount-common.candtcg-accel-ops-icount.c.Two stores per TB is not much on its own, but TBs are short. An emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) executes 34.2 billion TBs at 6.04 guest instructions each, so this is 68 billion stores for nothing.
Quoted 8 lines
Measured on an x86-64 host, LTO build, on top of the preceding two patches:
before: 1,469,729,281,442 instructions after: 1,402,816,253,499 instructions -4.55% before: 120.97s wall clock after: 115.75s wall clock -4.32%
Nice.
Quoted 13 lines
The emulated compiler produces byte-identical output.
Signed-off-by Matt Turner <[email protected]>--- accel/tcg/translator.c | 10 ++++++++++ 1 file changed, 10 insertions(+)accel/tcg/translator.c +0
-0@@ -21,12 +21,14 @@#include "disas/disas.h"
accel/tcg/translator.c line 31
Quoted 10 lines
accel/tcg/translator.c +2
-0#include "tb-internal.h" +#ifndef CONFIG_USER_ONLY static void set_can_do_io (DisasContextBase *db,bool val) {QEMU_BUILD_BUG_ON (sizeof_field (CPUState, neg.can_do_io) !=1 );tcg_gen_st8_i32 (tcg_constant_i32 (val), tcg_env, offsetof(CPUState, neg.can_do_io) - sizeof(CPUState)); } +#endif
I'm a bit reluctant to clutter translator.c with #ifdef'ry (in particular when we are not protecting system-specific API, so this can be evaluated at runtime).
We could have a tcg_user_emulation() or user_mode_emulation() helper hidding the #ifdef; or could move set_can_do_io() to accel/tcg/translate-all.c, or maybe others have better idea. The overall change LGTM otherwise.
Quoted 29 lines
accel/tcg/translator.c +8
-0bool translator_io_start (DisasContextBase *db) {@@ -210,17 +212,25 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns,/* * Manage can_do_io for the translation block: set to false before * the first insn and set to true before the last insn. +* +* Nothing reads can_do_io in user-only builds. There is no MMIO +* there, and every reader (cputlb.c, watchpoint.c, icount) is in +* system_ss, so skip the two stores per TB entirely. */ if (db->num_insns ==1 ) {tcg_debug_assert (first_insn_start == db->insn_start); }else {tcg_debug_assert (first_insn_start != db->insn_start); +#ifndef CONFIG_USER_ONLY tcg_ctx->emit_before_op = first_insn_start;set_can_do_io (db, false); +#endif } +#ifndef CONFIG_USER_ONLY tcg_ctx->emit_before_op = db->insn_start;set_can_do_io (db, true); tcg_ctx->emit_before_op =NULL ; +#endif /* May be used by disas_log or plugin callbacks. */ tb->size = db->pc_next - db->pc_first;
RFC PATCH 4/8 RFC: tcg: probe the TB jump cache inline instead of calling a helper
In reply to Matt TurnerEvery indirect branch that cannot use goto_tb ends in tcg_gen_lookup_and_goto_ptr(), which calls helper_lookup_tb_ptr(). For an emulated compiler that is 8.4 billion helper calls in a single translation
unit: 24.6% of all TB exits take this path, because jsr/ret/jmp have a register destination and because goto_tb is restricted to same-page targets.
The helper itself is already tight, but each call pays for a call frame, the can_do_io store, the get_tb_cpu_state() indirect call through TCGCPUOps, curr_cflags(), and a breakpoint check, before it gets to the jump cache probe that almost always hits (95.8% for this workload).
Emit the probe inline instead. The destination PC is already in a TCG temp, and the flags and cflags the destination must match are constants at translation time, so the fast path is a hash, three guarded loads and a goto_ptr. Only a miss calls the helper, which still owns filling the cache.
Two details matter for the generated code. The flags and cflags guards are folded into a single aligned 64-bit load and compare, since the fields are adjacent. And each path emits its own goto_ptr rather than branching to a shared one: a temp live across the label is spilled and reloaded on every dispatch, which cost 6.3% on its own.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO build, on top of the preceding three patches:
before: 1,402,816,253,499 instructions
after: 890,713,633,237 instructions -36.51%
before: 115.75s wall clock
after: 84.44s wall clock -27.05%The gap between the two is the point at which this stops being a straight-line win: the helper call was highly predictable work that the host pipelined well, so removing it retires far fewer instructions than it saves time. IPC falls from 2.48 to 2.15 across this patch for that reason.
Despite emitting more code, this also reduces instruction cache pressure, because a dispatch no longer jumps into qemu's .text and evicts translated code:
before: 11,476,318,964 L1-icache-load-misses
after: 6,990,186,701 L1-icache-load-misses -39.1%The mechanism is visible directly in a profile: helper_lookup_tb_ptr() falls from 30.97% of samples to 0.42%, and qemu's own .text falls from 38.9% to 5.4%, with the balance moving into generated code.
Combined with the three preceding patches, against an unmodified LTO build, 1,647,901,588,726 instructions fall to 890,713,633,237, or -45.95%. The emulated compiler produces byte-identical output throughout.
Open issues, hence RFC:
- The flags/cflags guards use the *current* TB's values as constants. That assumes the CPU flags feeding get_tb_cpu_state() cannot change within a TB, and that curr_cflags() cannot change under a running TB (gdb attaching to enable single-step would). Both need to be established or the values need to be loaded at runtime.
- tcg/tcg-op.c has no business including accel/tcg/tb-jmp-cache.h or knowing the CPUJumpCache layout. The probe likely belongs in accel/tcg with a small emit helper exported from tcg/.
- The jump cache entry is read without qatomic_read(); entries are invalidated concurrently by setting tb to NULL.
- Only wired up for alpha so far, and only for 64-bit guest PCs.
--- include/tcg/tcg-op-common.h | 2 + target/alpha/translate.c | 6 ++- tcg/tcg-op.c | 77 +++++++++++++++++++++++++++++++++++++ 3 files changed, 83 insertions(+), 2 deletions(-)
include/tcg/tcg-op-common.h +2 -0
@@ -84,6 +84,8 @@ void tcg_gen_goto_tb(unsigned idx);* this op is equivalent to calling tcg_gen_exit_tb() with 0 as the argument. */ void tcg_gen_lookup_and_goto_ptr (void ); +void tcg_gen_lookup_and_goto_ptr_inline (TCGv_i64 pc,uint32_t flags, +uint32_t cflags);void tcg_gen_plugin_cb (unsigned from);void tcg_gen_plugin_mem_cb (TCGv_i64 addr,unsigned meminfo);
target/alpha/translate.c +4 -2
@@ -449,7 +449,8 @@ static void gen_goto_tb(DisasContext *ctx, unsigned tb_slot_idx, int32_t disp)tcg_gen_exit_tb (ctx->base.tb, tb_slot_idx); }else {gen_pc_disp (ctx, cpu_pc, disp); -tcg_gen_lookup_and_goto_ptr (); +tcg_gen_lookup_and_goto_ptr_inline (cpu_pc, ctx->base.tb->flags, + ctx->base.tb->cflags); } }
@@ -2917,7 +2918,8 @@ static void alpha_tr_tb_stop(DisasContextBase *dcbase, CPUState *cpu)gen_pc_disp (ctx, cpu_pc,0 );/* FALLTHRU */ case DISAS_PC_UPDATED : -tcg_gen_lookup_and_goto_ptr (); +tcg_gen_lookup_and_goto_ptr_inline (cpu_pc, ctx->base.tb->flags, + ctx->base.tb->cflags);break ;case DISAS_PC_UPDATED_NOCHAIN :tcg_gen_exit_tb (NULL ,0 );
tcg/tcg-op.c +77 -0
86 lines not on this page · raw message
Signature · 1 line
2.54.0
Re: [RFC PATCH 4/8] RFC: tcg: probe the TB jump cache inline instead of calling a helper
Quoted 80 lines · Matt Turner
Every indirect branch that cannot use
goto_tbends intcg_gen_lookup_and_goto_ptr(), which callshelper_lookup_tb_ptr(). For an emulated compiler that is 8.4 billion helper calls in a single translation
unit: 24.6% of all TB exits take this path, because jsr/ret/jmp have a register destination and becausegoto_tbis restricted to same-page targets.The helper itself is already tight, but each call pays for a call frame, the
can_do_iostore, theget_tb_cpu_state()indirect call through TCGCPUOps,curr_cflags(), and a breakpoint check, before it gets to the jump cache probe that almost always hits (95.8% for this workload).Emit the probe inline instead. The destination PC is already in a TCG temp, and the flags and cflags the destination must match are constants at translation time, so the fast path is a hash, three guarded loads and a
goto_ptr. Only a miss calls the helper, which still owns filling the cache.Two details matter for the generated code. The flags and cflags guards are folded into a single aligned 64-bit load and compare, since the fields are adjacent. And each path emits its own
goto_ptrrather than branching to a shared one: a temp live across the label is spilled and reloaded on every dispatch, which cost 6.3% on its own.Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO build, on top of the preceding three patches:
before: 1,402,816,253,499 instructions after: 890,713,633,237 instructions -36.51% before: 115.75s wall clock after: 84.44s wall clock -27.05%The gap between the two is the point at which this stops being a straight-line win: the helper call was highly predictable work that the host pipelined well, so removing it retires far fewer instructions than it saves time. IPC falls from 2.48 to 2.15 across this patch for that reason.
Despite emitting more code, this also reduces instruction cache pressure, because a dispatch no longer jumps into qemu's .text and evicts translated code:
before: 11,476,318,964 L1-icache-load-misses after: 6,990,186,701 L1-icache-load-misses -39.1%The mechanism is visible directly in a profile:
helper_lookup_tb_ptr()falls from 30.97% of samples to 0.42%, and qemu's own .text falls from 38.9% to 5.4%, with the balance moving into generated code.Combined with the three preceding patches, against an unmodified LTO build, 1,647,901,588,726 instructions fall to 890,713,633,237, or -45.95%. The emulated compiler produces byte-identical output throughout.
Open issues, hence RFC:
- The flags/cflags guards use the *current* TB's values as constants. That assumes the CPU flags feeding
get_tb_cpu_state()cannot change within a TB, and thatcurr_cflags()cannot change under a running TB (gdb attaching to enable single-step would). Both need to be established or the values need to be loaded at runtime.
-tcg/tcg-op.chas no business includingaccel/tcg/tb-jmp-cache.hor knowing the CPUJumpCache layout. The probe likely belongs in accel/tcg with a small emit helper exported from tcg/.
- The jump cache entry is read withoutqatomic_read(); entries are invalidated concurrently by setting tb to NULL.
- Only wired up for alpha so far, and only for 64-bit guest PCs.Signed-off-by Matt Turner <[email protected]>--- include/tcg/tcg-op-common.h | 2 + target/alpha/translate.c | 6 ++- tcg/tcg-op.c | 77 +++++++++++++++++++++++++++++++++++++ 3 files changed, 83 insertions(+), 2 deletions(-)include/tcg/tcg-op-common.h +0
-0@@ -84,6 +84,8 @@ void tcg_gen_goto_tb(unsigned idx);* this op is equivalent to calling tcg_gen_exit_tb() with 0 as the argument.
include/tcg/tcg-op-common.h void tcg_gen_goto_tb(unsigned idx);
Quoted 10 lines
include/tcg/tcg-op-common.h +7
-0*/ void tcg_gen_lookup_and_goto_ptr (void ); +void tcg_gen_lookup_and_goto_ptr_inline (TCGv_i64 pc,uint32_t flags, +uint32_t cflags); +/* +* As tcg_gen_lookup_and_goto_ptr(), but probe the TB jump cache inline +* instead of calling helper_lookup_tb_ptr() unconditionally. @pc must +* hold the destination guest PC; @flags and @cflags are the values the +* destination TB must have been translated with.
Without looking at the body, this reads as these method arguments better fit:
void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc,
const TranslationBlock *tb);
Quoted 3 lines
+ */
include/tcg/tcg-op-common.h +2
-0+void tcg_gen_lookup_and_goto_ptr_inline (TCGv_i64 pc,uint32_t flags, +uint32_t cflags)
RFC PATCH 5/8 RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
In reply to Matt Turnertranslator_use_goto_tb() refuses to chain unless the destination is on the same page as the start of the TB. For guests whose text is much larger than a page this is expensive: an emulated alpha gcc compiling a 255k line translation unit takes the indirect dispatch path for 8.4 billion of its 34.2 billion TB exits, and a large share of those are ordinary direct branches that simply crossed an 8 KiB page boundary.
The restriction was made unconditional by d3a2a1d803 ("accel/tcg:
Introduce translator_use_goto_tb"), whose rationale was:
Various targets avoid the page crossing test for CONFIG_USER_ONLY, but that is wrong: mmap and mprotect can change page permissions.
That is true, but in user-only builds the invalidation path already covers it. There are no page tables: every mmap, mprotect and munmap reaches page_set_flags(), which calls tb_invalidate_phys_range() whenever the flags actually change, and tb_phys_invalidate() calls tb_jmp_unlink() to reset incoming jumps. A chained cross-page jump is therefore broken whenever the destination page's permissions change. This is not true in system mode, where TBs are keyed by physical address and a page table change invalidates nothing, so the restriction is kept there.
Add tests/tcg/alpha/test-xpage-chain.c to cover the hazard directly. It places a direct branch near the end of one page targeting the next page, runs it 200000 times so the chain is established, then checks that mprotect(PROT_NONE) makes the next call fault, and that remapping the page with different code runs the new code rather than a stale translation.
The test detects the hazard it is meant to detect: with the tb_invalidate_phys_range() call in page_set_flags() commented out, it fails both phases, executing page B after PROT_NONE and returning the stale result.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation on an x86-64 host, LTO build, on top of the preceding patches:
before: 890,713,633,237 instructions
after: 869,178,598,378 instructions -2.42%
before: 84.44s wall clock
after: 80.49s wall clock -4.68%Note that this is worth more in time than in instructions, the reverse of the preceding patch: a chained jump replaces a cache probe whose loads can miss, so the instructions it removes are more expensive than average.
Measured before the inline jump cache probe, when a missed chain cost a helper call rather than an inline probe, the same change was worth -7.9%.
RFC because this reverses a deliberate decision and the reasoning above wants review from someone who knows the invalidation paths better than I do.
--- accel/tcg/translator.c | 12 ++++ tests/tcg/alpha/Makefile.target | 2 +- tests/tcg/alpha/test-xpage-chain.c | 111 +++++++++++++++++++++++++++++ 3 files changed, 124 insertions(+), 1 deletion(-) create mode 100644 tests/tcg/alpha/test-xpage-chain.c
accel/tcg/translator.c +12 -0
@@ -117,8 +117,20 @@ bool translator_use_goto_tb(DisasContextBase *db, vaddr dest)return false; } +#ifdef CONFIG_USER_ONLY +/* +* There are no page tables in user-only mode. Every mmap, mprotect and +* munmap goes through page_set_flags(), which calls +* tb_invalidate_phys_range() whenever the flags actually change, and +* tb_phys_invalidate() unlinks incoming jumps. A chained cross-page +* jump is therefore broken whenever the destination page's permissions +* change, so the same-page restriction is not needed here. +*/ +return true; +#else /* Check for the dest on the same page as the start of the TB. */ return translator_is_same_page (db, dest); +#endif } void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns,
tests/tcg/alpha/Makefile.target +1 -1
@@ -5,7 +5,7 @@ ALPHA_SRC=$(SRC_PATH)/tests/tcg/alpha VPATH+=$(ALPHA_SRC) -ALPHA_TESTS=hello-alpha test-cond test-cmov test-ovf test-cvttq +ALPHA_TESTS=hello-alpha test-cond test-cmov test-ovf test-cvttq test-xpage-chain TESTS+=$(ALPHA_TESTS) test-cmov: EXTRA_CFLAGS=-DTEST_CMOV
tests/tcg/alpha/test-xpage-chain.c new file +111 -0
111 lines not on this page · raw message
Signature · 1 line
2.54.0
Re: [RFC PATCH 5/8] RFC: accel/tcg: allow cross-page goto_tb chaining in user-only builds
Quoted 70 lines · Matt Turner
translator_use_goto_tb()refuses to chain unless the destination is on the same page as the start of the TB. For guests whose text is much larger than a page this is expensive: an emulated alpha gcc compiling a 255k line translation unit takes the indirect dispatch path for 8.4 billion of its 34.2 billion TB exits, and a large share of those are ordinary direct branches that simply crossed an 8 KiB page boundary.The restriction was made unconditional by
d3a2a1d803("accel/tcg:
Introducetranslator_use_goto_tb"), whose rationale was:Various targets avoid the page crossing test for
CONFIG_USER_ONLY, but that is wrong: mmap and mprotect can change page permissions.That is true, but in user-only builds the invalidation path already covers it. There are no page tables: every mmap, mprotect and munmap reaches
page_set_flags(), which callstb_invalidate_phys_range()whenever the flags actually change, andtb_phys_invalidate()callstb_jmp_unlink()to reset incoming jumps. A chained cross-page jump is therefore broken whenever the destination page's permissions change. This is not true in system mode, where TBs are keyed by physical address and a page table change invalidates nothing, so the restriction is kept there.Add
tests/tcg/alpha/test-xpage-chain.cto cover the hazard directly. It places a direct branch near the end of one page targeting the next page, runs it 200000 times so the chain is established, then checks thatmprotect(PROT_NONE)makes the next call fault, and that remapping the page with different code runs the new code rather than a stale translation.The test detects the hazard it is meant to detect: with the
tb_invalidate_phys_range()call inpage_set_flags()commented out, it fails both phases, executing page B afterPROT_NONEand returning the stale result.Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation on an x86-64 host, LTO build, on top of the preceding patches:
before: 890,713,633,237 instructions after: 869,178,598,378 instructions -2.42% before: 84.44s wall clock after: 80.49s wall clock -4.68%Note that this is worth more in time than in instructions, the reverse of the preceding patch: a chained jump replaces a cache probe whose loads can miss, so the instructions it removes are more expensive than average.
Measured before the inline jump cache probe, when a missed chain cost a helper call rather than an inline probe, the same change was worth -7.9%.
RFC because this reverses a deliberate decision and the reasoning above wants review from someone who knows the invalidation paths better than I do.
Signed-off-by Matt Turner <[email protected]>--- accel/tcg/translator.c | 12 ++++ tests/tcg/alpha/Makefile.target | 2 +- tests/tcg/alpha/test-xpage-chain.c | 111 +++++++++++++++++++++++++++++ 3 files changed, 124 insertions(+), 1 deletion(-)create mode 100644
tests/tcg/alpha/test-xpage-chain.caccel/tcg/translator.c +0
-0@@ -117,8 +117,20 @@ bool translator_use_goto_tb(DisasContextBase *db, vaddr dest)return false; }
accel/tcg/translator.c bool translator_use_goto_tb(DisasContextBase *db, vaddr dest) · line 129
Quoted 10 lines
accel/tcg/translator.c +10
-0+#ifdef CONFIG_USER_ONLY +/* +* There are no page tables in user-only mode. Every mmap, mprotect and +* munmap goes through page_set_flags(), which calls +* tb_invalidate_phys_range() whenever the flags actually change, and +* tb_phys_invalidate() unlinks incoming jumps. A chained cross-page +* jump is therefore broken whenever the destination page's permissions +* change, so the same-page restriction is not needed here. +*/ +return true;
Should this be directly changed in translator_is_same_page() in "exec/translator.h" instead?
Quoted 5 lines
accel/tcg/translator.c +2
-0+#else /* Check for the dest on the same page as the start of the TB. */ return translator_is_same_page (db, dest); +#endif }
RFC PATCH 6/8 RFC: accel/tcg: only poll for interrupts in blocks that can close a cycle
In reply to Matt TurnerEvery translation block begins by loading cpu->neg.icount_decr.u32, testing it and branching to the exit path. That is three host instructions at the top of every TB. Blocks are short, so this is expensive: an emulated alpha gcc 16.2.0 compiling a 255k line translation unit executes 34.2 billion TBs at 6.04 guest instructions each. Forcing CF_NOIRQ on for the whole run, which is not correct but bounds the prize, is worth 12.0% of all instructions retired.
The check does not have to be in every block. Interrupt latency is bounded as long as every cycle in the guest control flow graph passes through at least one block that polls. Any such cycle must contain either an edge whose destination is at or below the start of the block it leaves from, or an edge whose destination is not known at translation time: take the block with the lowest start address in the cycle, and the edge entering it comes from a block at or above it.
So record, during translation, whether this TB has such an edge. translator_use_goto_tb() already sees every statically known destination, and every target that emits goto_tb reaches it, so a backward edge sets DisasContextBase::needs_exit_check there. Indirect destinations are flagged by tcg_gen_lookup_and_goto_ptr(). Blocks with neither cannot close a cycle on their own and can skip the poll.
The check is therefore emitted retroactively in gen_tb_end(), using the same emit_before_op mechanism the can_do_io stores use, and only when one of the two flags is set. icount opts out and keeps the unconditional counter.
Measured on an x86-64 host, LTO build, on top of the preceding patches:
before: 869,178,598,378 instructions
after: 809,988,851,304 instructions -6.81%
before: 80.49s wall clock
after: 78.00s wall clock -3.10%That is 57% of the 12.0% ceiling, which is about right: roughly a quarter of TB exits are indirect and are still polled, plus every loop back edge.
For the series as a whole, against an unmodified LTO build, instructions retired fall from 1,647,901,588,726 to 809,988,851,304 (-50.85%) and wall clock from 133.57s to 78.00s (-41.61%). The two do not match because what the series removes is mostly cheap, well-predicted dispatch overhead: IPC falls from 2.53 to 2.12 as the remaining work gets less regular.
tests/tcg/alpha/test-xpage-chain.c still passes, the emulated compiler still produces byte-identical output, and a tight loop under alarm(1) is still interrupted, after 897 million iterations.
RFC because:
- The soundness argument depends on every goto_tb destination passing through translator_use_goto_tb(). No target in the tree bypasses it today, but nothing enforces that.
- System mode interrupt latency now depends on guest control flow rather than on block count. The bound is one straight-line run between cycles, which should be fine, but timer-driven guests deserve a closer look than I can give them.
--- accel/tcg/translator.c | 57 ++++++++++++++++++++++++++++++++++++--- include/exec/translator.h | 2 ++ include/tcg/tcg.h | 2 ++ tcg/tcg-op.c | 2 ++ 4 files changed, 60 insertions(+), 3 deletions(-)
accel/tcg/translator.c +54 -3
@@ -42,12 +42,29 @@ bool translator_io_start(DisasContextBase *db)return true; } +/* +* Any cycle in the guest control flow graph must contain an edge whose +* destination is at or below the start of the block it leaves from, or an +* edge whose destination is not known at translation time. Only blocks with +* such an edge need the interrupt check, so defer the decision until the end +* of translation, when we know which edges this TB has. +* +* icount needs the counter unconditionally, so it opts out. +*/ +static bool defer_exit_check (uint32_t cflags) +{ +return !(cflags &CF_USE_ICOUNT ); +} +static TCGOp *gen_tb_start (DisasContextBase *db,uint32_t cflags) {TCGv_i32 count =NULL ;TCGOp *icount_start_insn =NULL ; -if ((cflags &CF_USE_ICOUNT ) || !(cflags &CF_NOIRQ )) { + tcg_ctx->exit_check_needed = false; + +if ((cflags &CF_USE_ICOUNT ) || + (!(cflags &CF_NOIRQ ) && !defer_exit_check (cflags))) { count =tcg_temp_new_i32 ();tcg_gen_ld_i32 (count, tcg_env, offsetof(CPUState, neg.icount_decr.u32) -
@@ -73,6 +90,12 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags)*/ if (cflags &CF_NOIRQ ) { tcg_ctx->exitreq_label =NULL ; + }else if (defer_exit_check (cflags)) { +/* +* Emitted retroactively by gen_tb_end(), but only if this TB can be +* part of a control flow cycle. +*/ + tcg_ctx->exitreq_label =gen_new_label (); }else { tcg_ctx->exitreq_label =gen_new_label ();tcg_gen_brcondi_i32 (TCG_COND_LT , count,0 , tcg_ctx->exitreq_label);
@@ -88,7 +111,8 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags) }static void gen_tb_end (const TranslationBlock *tb,uint32_t cflags, -TCGOp *icount_start_insn,int num_insns) +TCGOp *icount_start_insn,int num_insns, +DisasContextBase *db,TCGOp *first_insn_start) {if (cflags &CF_USE_ICOUNT ) {/*
@@ -99,6 +123,23 @@ static void gen_tb_end(const TranslationBlock *tb, uint32_t cflags, tcgv_i32_arg(tcg_constant_i32(num_insns))); } +if (tcg_ctx->exitreq_label &&defer_exit_check (cflags) && + !(cflags &CF_NOIRQ )) { +if (db->needs_exit_check || tcg_ctx->exit_check_needed) { +TCGv_i32 count =tcg_temp_new_i32 (); +TCGOp *save = tcg_ctx->emit_before_op; + + tcg_ctx->emit_before_op = first_insn_start; +tcg_gen_ld_i32 (count, tcg_env, + offsetof(CPUState, neg.icount_decr.u32) - + sizeof(CPUState)); +tcg_gen_brcondi_i32 (TCG_COND_LT , count,0 , tcg_ctx->exitreq_label); + tcg_ctx->emit_before_op = save; + }else { + tcg_ctx->exitreq_label =NULL ; + } + } +if (tcg_ctx->exitreq_label) {gen_set_label (tcg_ctx->exitreq_label);tcg_gen_exit_tb (tb,TB_EXIT_REQUESTED );
@@ -117,6 +158,14 @@ bool translator_use_goto_tb(DisasContextBase *db, vaddr dest)return false; } +/* +* A destination at or below the start of this TB can close a cycle, so +* this TB must poll for interrupts. See defer_exit_check(). +*/ +if (dest <= db->pc_first) { + db->needs_exit_check = true; + } +#ifdef CONFIG_USER_ONLY /* * There are no page tables in user-only mode. Every mmap, mprotect and
@@ -153,6 +202,7 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns, db->max_insns = *max_insns; db->insn_start =NULL ; db->fake_insn = false; + db->needs_exit_check = false; db->host_addr[0 ] = host_pc; db->host_addr[1 ] =NULL ; db->record_start =0 ;
@@ -219,7 +269,8 @@ void translator_loop(CPUState *cpu, TranslationBlock *tb, int *max_insns,/* Emit code to exit the TB, as indicated by db->is_jmp. */ ops->tb_stop (db, cpu); -gen_tb_end (tb, cflags, icount_start_insn, db->num_insns); +gen_tb_end (tb, cflags, icount_start_insn, db->num_insns, db, + first_insn_start);/* * Manage can_do_io for the translation block: set to false before
include/exec/translator.h +2 -0
@@ -74,6 +74,8 @@ struct DisasContextBase {int max_insns;bool plugin_enabled;bool fake_insn; +/* Set when this TB can be part of a control flow cycle. */ +bool needs_exit_check;uint8_t code_mmuidx;struct TCGOp *insn_start;void *host_addr[2 ];
include/tcg/tcg.h +2 -0
@@ -389,6 +389,8 @@ struct TCGContext {struct TCGLabelPoolData *pool_labels;TCGLabel *exitreq_label; +/* Set by goto_ptr emission: destination is not known statically. */ +bool exit_check_needed;#ifdef CONFIG_PLUGIN /*
tcg/tcg-op.c +2 -0
@@ -2616,6 +2616,7 @@ void tcg_gen_lookup_and_goto_ptr(void)return ; } + tcg_ctx->exit_check_needed = true;plugin_gen_disable_mem_helpers (); ptr =tcg_temp_ebb_new_ptr ();gen_helper_lookup_tb_ptr (ptr, tcg_env);
@@ -2642,6 +2643,7 @@ void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc, uint32_t flags,return ; } + tcg_ctx->exit_check_needed = true;plugin_gen_disable_mem_helpers ();QEMU_BUILD_BUG_ON (sizeof (((CPUJumpCache *)0 )->array[0 ]) !=16 );
Signature · 1 line
2.54.0
RFC PATCH 7/8 RFC: accel/tcg: poison the jump cache instead of polling for indirect exits
In reply to Matt Turnerf799aeecff dropped the icount_decr poll from blocks that cannot close a control flow cycle. What is left is dominated by blocks that end in an indirect branch: tcg_gen_lookup_and_goto_ptr{,_inline}() set exit_check_needed, because the destination is unknown at translation time and so the block might be part of a cycle. For the emulated compiler that is still 55.5% of translated blocks and 51.2% of generated-code cycles, and the two leading instructions of those blocks
mov -0x10(%rbp),%ebx
test %ebx,%ebxcarry 5.01% of all cycles spent in generated code, measured with cycles:pp so that the figure is not just skid from the dispatch that jumped there.
A block dispatching indirectly does not need to poll, because the dispatch itself can be made to notice. The out-of-line path already calls helper_lookup_tb_ptr() every time, so it only needs the helper to return the epilogue while an exit is pending. The inline probe added by 14c3e5a1f3 is the interesting case: it already loads the jump cache base from CPUState and already branches to the slow path when the entry it finds has a NULL tb. Pointing that base at a page of zeroes therefore turns every indirect dispatch into a miss, and a miss lands in the same helper. The poll becomes a pointer swap on the exit request path and costs the fast path nothing.
So give the probe its own base pointer, tb_jmp_cache_probe, that nothing else reads. The two places that set icount_decr.u16.high poison it; the place that clears the flag restores it. The real tb_jmp_cache is untouched throughout, so no cache contents are lost and the recovery is one store.
Blocks with a backward goto_tb edge still poll. Interrupt latency is unchanged in kind: an exit is noticed at the next cycle-closing edge, which is now either a poll or a dispatch, rather than only a poll.
Measured on an x86-64 host, LTO build, on top of the preceding patches. The control was measured in the same session, because the host's all-core turbo varies by ~3% between sessions and swamps the effect otherwise:
before: 810,079,619,264 instructions, 79.563s
after: 787,483,360,681 instructions, 78.019s
-2.79% instructions, -1.94% wallForcing the check off entirely, which is incorrect but is the ceiling, gives -5.64% instructions and -2.55% wall. So this takes half the instructions and three quarters of the time: what it removes sits directly after an indirect branch, where the poll's dependent load was the most expensive place a poll could be.
In a jitdump profile of the same workload, blocks opening with the poll fall from 55.5% to 25.9%, cycles in blocks that poll from 51.2% to 18.5%, and cycles on the two poll instructions from 5.01% to 1.50%.
tests/tcg/alpha/test-indirect-irq.c is added for this: a loop whose only back edge is an indirect branch, under alarm(1). It passes before and after, and hangs if the check is simply deleted, which is what makes it a test of the new mechanism rather than of the old poll. The other alpha tests still pass and the emulated compiler still produces byte-identical output.
RFC because:
- The restore in cpu_handle_interrupt() races a concurrent poison from another thread. The existing barrier around icount_decr.u16.high covers it -- a poison that lands after the restore also re-set the flag, and exit_request was stored before it -- but this deserves more eyes than the single-threaded user-mode testing I have given it.
- Only the inline probe needs the poison, and only alpha uses the inline probe today. Targets on the out-of-line path are covered by the helper check alone, but that has not been measured.
- The shared zero-filled CPUJumpCache is a 1MB allocation that is never written. A read-only mapping would express that better.
--- accel/tcg/cpu-exec.c | 54 +++++++++++++++++++++++++++++ accel/tcg/internal-common.h | 3 ++ accel/tcg/tcg-accel-ops.c | 2 ++ accel/tcg/translator.c | 4 +-- include/hw/core/cpu.h | 10 ++++++ include/tcg/tcg.h | 2 -- tcg/tcg-op.c | 13 +++++-- tests/tcg/alpha/Makefile.target | 3 +- tests/tcg/alpha/test-indirect-irq.c | 53 ++++++++++++++++++++++++++++ 9 files changed, 135 insertions(+), 9 deletions(-) create mode 100644 tests/tcg/alpha/test-indirect-irq.c
accel/tcg/cpu-exec.c +54 -0
@@ -388,6 +388,16 @@ const void *HELPER(lookup_tb_ptr)(CPUArchState *env)*/ cpu->neg.can_do_io = true; +/* +* A block that dispatches indirectly does not emit the icount_decr poll, +* so this is where a pending exit is noticed for that path: either the +* probe was poisoned and every dispatch arrives here, or the target uses +* the out-of-line lookup and always did. +*/ +if (unlikely (cpu_loop_exit_requested (cpu))) { +return tcg_code_gen_epilogue; + } +TCGTBCPUState s = cpu->cc->tcg_ops->get_tb_cpu_state (cpu); s.cflags =curr_cflags (cpu);
@@ -752,6 +762,44 @@ static inline bool cpu_handle_exception(CPUState *cpu, int *ret)return false; } +/* +* The inline jump cache probe reads cpu->tb_jmp_cache_probe and takes the +* slow path when the entry it finds has a NULL tb. Pointing the probe at a +* region that is all zeroes therefore forces every indirect dispatch into +* helper_lookup_tb_ptr(), which returns to the main loop while an exit is +* pending. That is what lets a block ending in an indirect branch skip the +* icount_decr poll: the poll's job is done by a pointer swap that costs the +* fast path nothing. +* +* Only ever read from, and only the tb field of one entry per dispatch, so +* one shared zero-filled cache is enough for every CPU. +*/ +static const CPUJumpCache *tb_jmp_cache_poison (void ) +{ +static CPUJumpCache *poison; + +if (unlikely (poison ==NULL )) { +/* Raced allocations are harmless: both are all zeroes. */ +qatomic_cmpxchg (&poison,NULL ,g_new0 (CPUJumpCache,1 )); + } +return poison; +} + +void tcg_cpu_poison_jmp_cache (CPUState *cpu) +{ +if (qatomic_read (&cpu->tb_jmp_cache_probe) !=NULL ) { +qatomic_set (&cpu->tb_jmp_cache_probe, + (CPUJumpCache *)tb_jmp_cache_poison ()); + } +} + +void tcg_cpu_restore_jmp_cache (CPUState *cpu) +{ +if (qatomic_read (&cpu->tb_jmp_cache_probe) !=NULL ) { +qatomic_set (&cpu->tb_jmp_cache_probe, cpu->tb_jmp_cache); + } +} +void tcg_kick_vcpu_thread (CPUState *cpu) {/*
@@ -764,6 +812,9 @@ void tcg_kick_vcpu_thread(CPUState *cpu)/* Ensure cpu_exec will see the exit request after TCG has exited. */ qatomic_store_release (&cpu->neg.icount_decr.u16.high,-1 ); + +/* Blocks that only dispatch indirectly do not poll; stop them chaining. */ +tcg_cpu_poison_jmp_cache (cpu); }static inline bool icount_exit_request (CPUState *cpu)
@@ -796,6 +847,7 @@ static inline bool cpu_handle_interrupt(CPUState *cpu,* tcg_kick_vcpu_thread()) */ qatomic_set_mb (&cpu->neg.icount_decr.u16.high,0 ); +tcg_cpu_restore_jmp_cache (cpu);#ifdef CONFIG_USER_ONLY assert (!cpu_test_interrupt (cpu, ~0 ));
@@ -1069,6 +1121,7 @@ bool tcg_exec_realizefn(CPUState *cpu, Error **errp) } cpu->tb_jmp_cache =g_new0 (CPUJumpCache,1 ); +qatomic_set (&cpu->tb_jmp_cache_probe, cpu->tb_jmp_cache);tlb_init (cpu);#ifndef CONFIG_USER_ONLY tcg_iommu_init_notifier_list (cpu);
@@ -1086,5 +1139,6 @@ void tcg_exec_unrealizefn(CPUState *cpu)#endif /* !CONFIG_USER_ONLY */ tlb_destroy (cpu); +qatomic_set (&cpu->tb_jmp_cache_probe,NULL );g_free_rcu (cpu->tb_jmp_cache, rcu); }
accel/tcg/internal-common.h +3 -0
@@ -154,6 +154,9 @@ void page_table_config_init(void); G_NORETURN void cpu_io_recompile(CPUState *cpu, uintptr_t retaddr);#endif /* CONFIG_USER_ONLY */ +void tcg_cpu_poison_jmp_cache (CPUState *cpu); +void tcg_cpu_restore_jmp_cache (CPUState *cpu); +void tb_phys_invalidate (TranslationBlock *tb,tb_page_addr_t page_addr);void tb_set_jmp_target (TranslationBlock *tb,int n,uintptr_t addr);
accel/tcg/tcg-accel-ops.c +2 -0
@@ -44,6 +44,7 @@#include "hw/core/cpu.h" +#include "internal-common.h" #include "tcg-accel-ops.h" #include "tcg-accel-ops-mttcg.h" #include "tcg-accel-ops-rr.h"
@@ -106,6 +107,7 @@ void tcg_handle_interrupt(CPUState *cpu, int mask)qemu_cpu_kick (cpu); }else {qatomic_set (&cpu->neg.icount_decr.u16.high,-1 ); +tcg_cpu_poison_jmp_cache (cpu); } }
accel/tcg/translator.c +1 -3
@@ -61,8 +61,6 @@ static TCGOp *gen_tb_start(DisasContextBase *db, uint32_t cflags)TCGv_i32 count =NULL ;TCGOp *icount_start_insn =NULL ; - tcg_ctx->exit_check_needed = false; -if ((cflags &CF_USE_ICOUNT ) || (!(cflags &CF_NOIRQ ) && !defer_exit_check (cflags))) { count =tcg_temp_new_i32 ();
@@ -125,7 +123,7 @@ static void gen_tb_end(const TranslationBlock *tb, uint32_t cflags,if (tcg_ctx->exitreq_label &&defer_exit_check (cflags) && !(cflags &CF_NOIRQ )) { -if (db->needs_exit_check || tcg_ctx->exit_check_needed) { +if (db->needs_exit_check) {TCGv_i32 count =tcg_temp_new_i32 ();TCGOp *save = tcg_ctx->emit_before_op;
include/hw/core/cpu.h +10 -0
@@ -526,6 +526,16 @@ struct CPUState {struct CPUJumpCache *tb_jmp_cache; +/* +* What the inline jump cache probe emitted by +* tcg_gen_lookup_and_goto_ptr_inline() reads. Normally equal to +* tb_jmp_cache; pointed at a page of zeroes while an exit is pending, so +* that every indirect dispatch misses and lands in the helper, which +* returns to the main loop. Only generated code and the two accessors in +* cpu-exec.c may touch it. +*/ +struct CPUJumpCache *tb_jmp_cache_probe; +GArray *gdb_regs;int gdb_num_regs;int gdb_num_g_regs;
include/tcg/tcg.h +0 -2
@@ -389,8 +389,6 @@ struct TCGContext {struct TCGLabelPoolData *pool_labels;TCGLabel *exitreq_label; -/* Set by goto_ptr emission: destination is not known statically. */ -bool exit_check_needed;#ifdef CONFIG_PLUGIN /*
tcg/tcg-op.c +10 -3
@@ -2616,7 +2616,10 @@ void tcg_gen_lookup_and_goto_ptr(void)return ; } - tcg_ctx->exit_check_needed = true; +/* +* No icount_decr poll is needed for this exit: the helper is called on +* every dispatch and returns to the main loop while an exit is pending. +*/ plugin_gen_disable_mem_helpers (); ptr =tcg_temp_ebb_new_ptr ();gen_helper_lookup_tb_ptr (ptr, tcg_env);
@@ -2643,7 +2646,11 @@ void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc, uint32_t flags,return ; } - tcg_ctx->exit_check_needed = true; +/* +* No icount_decr poll is needed for this exit either. A pending exit +* poisons tb_jmp_cache_probe, so the guarded load below finds a NULL tb, +* takes the slow path, and the helper returns to the main loop. +*/ plugin_gen_disable_mem_helpers ();QEMU_BUILD_BUG_ON (sizeof (((CPUJumpCache *)0 )->array[0 ]) !=16 );
@@ -2665,7 +2672,7 @@ void tcg_gen_lookup_and_goto_ptr_inline(TCGv_i64 pc, uint32_t flags,tcg_gen_shli_i64 (h, h,4 );tcg_gen_ld_ptr (jc, tcg_env, - offsetof(CPUState , tb_jmp_cache) -sizeof (CPUState)); + offsetof(CPUState , tb_jmp_cache_probe) -sizeof (CPUState));tcg_gen_trunc_i64_ptr (ent, h);tcg_gen_add_ptr (ent, jc, ent);
tests/tcg/alpha/Makefile.target +2 -1
@@ -5,7 +5,8 @@ ALPHA_SRC=$(SRC_PATH)/tests/tcg/alpha VPATH+=$(ALPHA_SRC) -ALPHA_TESTS=hello-alpha test-cond test-cmov test-ovf test-cvttq test-xpage-chain +ALPHA_TESTS=hello-alpha test-cond test-cmov test-ovf test-cvttq test-xpage-chain \ + test-indirect-irq TESTS+=$(ALPHA_TESTS) test-cmov: EXTRA_CFLAGS=-DTEST_CMOV
tests/tcg/alpha/test-indirect-irq.c new file +53 -0
53 lines not on this page · raw message
Signature · 1 line
2.54.0
RFC PATCH 8/8 RFC: tcg: fold a guest displacement into the host addressing mode
In reply to Matt TurnerNothing in the TCG frontend interface can express a based memory access. tcg_gen_qemu_ld/st take an address and nothing else, so a target with a displacement in its load and store encodings -- which is most of them -- has to materialise the address first:
ldq a1,8(a0) -> mov 0x80(%rbp),%rbx reload a0
lea 0x8(%rbx),%r12 address
mov (%r12),%r12 the load
mov %r12,0x88(%rbp) spill a1The lea is pure loss on a host whose addressing mode has a displacement field sitting empty. It also needs a register, at the point in a block where pressure is highest.
Fold it. After optimisation, look for an add of a constant immediately before a guest access, defining that access's address operand, and move the constant into a new second constant argument on the op. The add is left for liveness to remove, so nothing breaks if its result has another use. Only the immediately preceding op is examined: that is what the frontends emit, and a window of one op means the pass does not have to reason about what could have happened in between. The one thing it does check is that the add did not clobber the base it read, since the access now reads that base directly.
Targets opt in with TCG_TARGET_HAS_ldst_disp and an out_disp member on TCGOutOpQemuLdSt. Without it the pass does not run, the displacement stays zero and the existing out member is called exactly as before, so no other backend changes behaviour or needs touching.
For x86_64 the displacement goes in the disp32 that prepare_host_addr() already fills in for guest_base. The fold is refused unless there is no slow path at all, which means user-only -- softmmu compares the unadjusted address against the TLB -- and an access needing no alignment test, since the slow path hands addr_reg to the helper and that register no longer holds the full guest address. It is also refused if guest_base plus the displacement leaves disp32.
Measured with qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, LTO build, on top of the preceding patches, against a control measured in the same session:
before: 787,483,360,681 instructions, 78.250s
after: 738,020,161,466 instructions, 75.679s
-6.29% instructions, -3.29% wallEmitted code shrinks from 46.29MB to 44.53MB over the run, 153.3 to 147.5 bytes per block. Per Alpha opcode, the host bytes emitted for an access fall as expected and nothing else moves:
ldq 17.2 -> 14.3 ldah 16.9 -> 16.9
ldl 14.1 -> 11.6 lda 12.6 -> 12.6
stq 12.7 -> 9.7 mov 9.7 -> 9.7The emulated compiler produces byte-identical output and the alpha tests still pass, including with a non-zero guest_base forced via -B.
RFC because:
- Only wired up for x86_64, and only for qemu_ld and qemu_st; the i128 qemu_ld2 and qemu_st2 pairs are left alone.
- Requiring that no slow path exists is stricter than necessary. Recording the displacement in TCGLabelQemuLdst and emitting one lea on the slow path would cover alignment-checked accesses too, at no fast path cost.
- Softmmu wants the displacement folded into the TLB comparison as well, which is a bigger change than this one.
- A one op window catches everything the frontends emit today but is trivially defeated by anything scheduled in between.
--- include/tcg/tcg-opc.h | 9 +++- tcg/tcg-op-ldst.c | 3 +- tcg/tcg.c | 86 ++++++++++++++++++++++++++++++++++++- tcg/x86_64/tcg-target.c.inc | 61 ++++++++++++++++++++++++++ tcg/x86_64/tcg-target.h | 3 ++ 5 files changed, 158 insertions(+), 4 deletions(-)
include/tcg/tcg-opc.h +7 -2
@@ -118,8 +118,13 @@ DEF(goto_ptr, 0, 1, 0, TCG_OPF_BB_EXIT | TCG_OPF_BB_END) DEF(plugin_cb, 0, 0, 1, TCG_OPF_NOT_PRESENT)DEF (plugin_mem_cb,0 ,1 ,1 ,TCG_OPF_NOT_PRESENT ) -DEF(qemu_ld, 1, 1, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT) -DEF(qemu_st, 0, 2, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT) +/* +* The second constant argument is a displacement to add to the address, +* zero unless a target advertises TCG_TARGET_HAS_ldst_disp and the fold in +* fold_ldst_disp() applied. + */ +DEF(qemu_ld, 1, 1, 2, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT) +DEF(qemu_st, 0, 2, 2, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT) DEF(qemu_ld2, 2, 1, 1, TCG_OPF_CALL_CLOBBER | TCG_OPF_SIDE_EFFECTS | TCG_OPF_INT)DEF (qemu_st2,0 ,3 ,1 ,TCG_OPF_CALL_CLOBBER |TCG_OPF_SIDE_EFFECTS |TCG_OPF_INT )
tcg/tcg-op-ldst.c +2 -1
@@ -92,7 +92,8 @@ static MemOp tcg_canonicalize_memop(MemOp op, bool is64, bool st)static void gen_ldst1 (TCGOpcode opc,TCGType type,TCGTemp *v,TCGTemp *addr,MemOpIdx oi) { -TCGOp *op =tcg_gen_op3 (opc, type,temp_arg (v),temp_arg (addr), oi); +/* The trailing zero is the address displacement; see fold_ldst_disp(). */ +TCGOp *op =tcg_gen_op4 (opc, type,temp_arg (v),temp_arg (addr), oi,0 );TCGOP_FLAGS (op) =get_memop (oi) &MO_SIZE ; }
tcg/tcg.c +85 -1
110 lines not on this page · raw message
tcg/x86_64/tcg-target.c.inc +61 -0
85 lines not on this page · raw message
tcg/x86_64/tcg-target.h +3 -0
@@ -30,6 +30,9 @@#define TCG_TARGET_NB_REGS 32#define MAX_CODE_GEN_BUFFER_SIZE (2 * GiB) +/* A guest displacement can go in the disp32 of the addressing mode. */ +#define TCG_TARGET_HAS_ldst_disp 1 + typedef enum { TCG_REG_EAX = 0, TCG_REG_ECX,
Signature · 1 line
2.54.0
Hi Matt,
Quoted 176 lines · Matt Turner
For guests running large amounts of code, most of what TCG executes is not translated guest work but the fixed overhead around it. Blocks are short and there are a great many of them, so the constant cost at each end of a block (the interrupt poll and the
can_do_iostores on entry, the dispatch on exit) ends up dominating everything else.The workload throughout is qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, in a --static --enable-lto --target-list=alpha-linux-user build. It executes 34.2 billion TBs at 6.04 guest instructions each, and 24.6% of its TB exits cannot use
goto_tb. That is a representative shape for any guest whose text is much larger than a page: indirect calls and returns everywhere, plus direct branches that merely crossed a page boundary.The first three patches are ordinary cleanups that stand on their own. The remaining five are marked RFC individually and are where the interesting questions are.
1 accel/tcg: cache the result of
curr_cflags()Recomputed on every one of the run's 8.4 billion dispatches, from state that changes only when gdb enables single-step or a log mask moves. Cache it in CPUState and recompute from the four places that
can change an input. -5.10%2 accel/tcg: enlarge the TB jump cache to 64K entries
4096 entries is too small for a guest running a large program;
tb_htable_lookup()is 5.73% of samples. 16 bits is the knee of the
sizing curve, at 1 MiB per vCPU. -6.02%3 accel/tcg: skip the
can_do_iostores in user-only buildsTwo stores per TB that nothing in a user-only build reads: 68 billion
of them over the run. -4.55%, -4.32% wall4 RFC: tcg: probe the TB jump cache inline instead of calling a helper
95.8% of those 8.4 billion
helper_lookup_tb_ptr()calls hit the jump cache. Emit the probe inline (hash, three guarded loads,goto_ptr)
and call the helper only on a miss. -36.51%, -27.05% wall5 RFC: accel/tcg: allow cross-page
goto_tbchaining in user-only builds
translator_use_goto_tb()refuses to chain across a page. In user-only builds the invalidation path already covers what that was protecting
against: every mmap/mprotect/munmap reachespage_set_flags(), which invalidates and unlinks. Lift it there, keep it for system mode.
-2.42%, -4.68% wall6 RFC: accel/tcg: only poll for interrupts in blocks that can close a
cycleThe
icount_decrpoll needs to happen once per cycle in the guest CFG, not once per block, and any cycle must contain either a backward edge or an indirect one. Record both during translation and emit the check
only for blocks that have one. -6.81%, -3.10% wall7 RFC: accel/tcg: poison the jump cache instead of polling for indirect
exitsWhat patch 6 leaves behind is mostly blocks flagged for an indirect exit. Give the inline probe its own jump cache base pointer and point it at zeroes when an exit is requested: every dispatch then misses into the helper, which returns the epilogue. The poll becomes a
pointer swap on the request path. -2.79%, -1.94% wall8 RFC: tcg: fold a guest displacement into the host addressing mode
tcg_gen_qemu_ld/st cannot express a based access, so a target with a displacement in its encodings materializes the address with an lea that the host addressing mode would have done for free. Fold a preceding constant add into a new argument on the op, opt-in per
backend, wired up forx86_64user-only. -6.29%, -3.29% wallEach percentage is against the patch before it. End to end, measuring an unmodified build of the same base against the full series, five runs each, interleaved in one session so that host clock drift is shared rather than attributed (mean, with the run-to-run spread):
instructions retired: 1,646,129,294,236 -> 738,003,153,831 -55.17% (0.16%) (0.03%) wall clock: 134.934s -> 75.189s -44.28% (0.30%) (0.99%)Both endpoints ran at the same 4.782 GHz effective clock, and the .s files they produced are identical.
The two figures do not track each other, and that is the interesting part:
what the series removes is cheap, well-predicted, highly pipelined work, so it retires far more instructions than it saves time. IPC falls from 2.55 to 2.05 as the remaining work gets less regular. Patch 4 also cuts L1-icache load misses by 39.1%, because a dispatch no longer jumps into qemu's .text and evicts translated code; qemu's own .text falls from 38.9% to 5.4% of profile samples over the series.Every revision was built and measured separately, so the series bisects, and the emulated compiler produces byte-identical assembly output at every step, which is the correctness check these patches most need. Two new alpha tests cover the hazards the series creates:
tests/tcg/alpha/test-xpage-chain.c(patch 5) andtest-indirect-irq.c(patch 7). Both fail or hang if the mechanism they cover is removed, which is what makes them tests of the new behavior rather than of the old.The RFC patches need eyes I cannot supply myself. In rough order of how much I would like someone to look at them:
- Patch 5 reverses a deliberate decision made in
d3a2a1d803on the strength of an argument about the user-only invalidation paths.- Patch 6 moves system-mode interrupt latency from "bounded by block count" to "bounded by guest control flow". The bound is one straight-line run between cycles, but timer-driven guests want a closer look than I can give them. Its soundness also assumes every
goto_tbdestination passes throughtranslator_use_goto_tb(); no target in the tree bypasses it today, but nothing enforces that.- Patch 4 treats cpu flags and cflags as translation-time constants in its guards, reads a jump cache entry without
qatomic_read(), and puts knowledge of the CPUJumpCache layout intcg/tcg-op.c, where it does not belong.- Patch 7's restore in
cpu_handle_interrupt()races a concurrent poison from another thread. I believe the existing barrier around icount_decr.u16.high covers it, but my testing was single-threaded user mode.- Patch 8 only examines the immediately preceding op, refuses any access with a slow path (so user-only, and no alignment check), and leaves the i128 pairs alone.
- Patch 2's 1 MiB per vCPU is easy to justify for a single-vCPU linux-user process and less obvious for system emulation with many vCPUs. It may want to be sized per target or made tunable rather than raised unconditionally.
Patches 4 and 8 are wired up for alpha and
x86_64respectively; everything else is target-independent, and no other backend changes behavior or needs touching.Matt Turner (8):
accel/tcg: cache the result ofcurr_cflags()
accel/tcg: enlarge the TB jump cache to 64K entries accel/tcg: skip thecan_do_iostores in user-only builds
RFC: tcg: probe the TB jump cache inline instead of calling a helper
RFC: accel/tcg: allow cross-pagegoto_tbchaining in user-only builds
RFC: accel/tcg: only poll for interrupts in blocks that can close a
cycle
RFC: accel/tcg: poison the jump cache instead of polling for indirect
exits
RFC: tcg: fold a guest displacement into the host addressing modeaccel/tcg/cpu-exec-common.c | 48 +++++++++++- accel/tcg/cpu-exec.c | 54 ++++++++++++++ accel/tcg/internal-common.h | 22 +++++- accel/tcg/tb-jmp-cache.h | 2 +- accel/tcg/tcg-accel-ops.c | 2 + accel/tcg/tcg-all.c | 1 + accel/tcg/translator.c | 77 ++++++++++++++++++- cpu-target.c | 3 + include/exec/translation-block.h | 6 ++ include/exec/translator.h | 2 + include/hw/core/cpu.h | 23 +++++- include/system/tcg.h | 9 +++ include/tcg/tcg-op-common.h | 2 + include/tcg/tcg-opc.h | 9 ++- linux-user/main.c | 2 +- stubs/meson.build | 1 + stubs/tcg-cflags.c | 16 ++++ target/alpha/cpu.c | 2 +- target/alpha/translate.c | 6 +- tcg/tcg-op-ldst.c | 3 +- tcg/tcg-op.c | 86 +++++++++++++++++++++ tcg/tcg.c | 86 ++++++++++++++++++++- tcg/x86_64/tcg-target.c.inc | 61 +++++++++++++++ tcg/x86_64/tcg-target.h | 3 +
Quoted 9 lines
tests/tcg/alpha/Makefile.target | 3 +- tests/tcg/alpha/test-indirect-irq.c | 53 +++++++++++++ tests/tcg/alpha/test-xpage-chain.c | 111 ++++++++++++++++++++++++++++ util/log.c | 4 + 28 files changed, 676 insertions(+), 21 deletions(-) create mode 100644 stubs/tcg-cflags.c create mode 100644 tests/tcg/alpha/test-indirect-irq.c create mode 100644 tests/tcg/alpha/test-xpage-chain.c
I was curious to give it a try and see if everything works before helping on review, however series does not apply on master currently.
Could you please rebase it and send a v2?
Regards,
Pierrick
Quoted 190 lines · Pierrick Bouvier
Hi Matt,
Quoted 176 lines · Matt Turner
For guests running large amounts of code, most of what TCG executes is not translated guest work but the fixed overhead around it. Blocks are short and there are a great many of them, so the constant cost at each end of a block (the interrupt poll and the
can_do_iostores on entry, the dispatch on exit) ends up dominating everything else.The workload throughout is qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, in a --static --enable-lto --target-list=alpha-linux-user build. It executes 34.2 billion TBs at 6.04 guest instructions each, and 24.6% of its TB exits cannot use
goto_tb. That is a representative shape for any guest whose text is much larger than a page: indirect calls and returns everywhere, plus direct branches that merely crossed a page boundary.The first three patches are ordinary cleanups that stand on their own. The remaining five are marked RFC individually and are where the interesting questions are.
1 accel/tcg: cache the result of
curr_cflags()Recomputed on every one of the run's 8.4 billion dispatches, from state that changes only when gdb enables single-step or a log mask moves. Cache it in CPUState and recompute from the four places that
can change an input. -5.10%2 accel/tcg: enlarge the TB jump cache to 64K entries
4096 entries is too small for a guest running a large program;
tb_htable_lookup()is 5.73% of samples. 16 bits is the knee of the
sizing curve, at 1 MiB per vCPU. -6.02%3 accel/tcg: skip the
can_do_iostores in user-only buildsTwo stores per TB that nothing in a user-only build reads: 68 billion
of them over the run. -4.55%, -4.32% wall4 RFC: tcg: probe the TB jump cache inline instead of calling a helper
95.8% of those 8.4 billion
helper_lookup_tb_ptr()calls hit the jump cache. Emit the probe inline (hash, three guarded loads,goto_ptr)
and call the helper only on a miss. -36.51%, -27.05% wall5 RFC: accel/tcg: allow cross-page
goto_tbchaining in user-only builds
translator_use_goto_tb()refuses to chain across a page. In user-only builds the invalidation path already covers what that was protecting
against: every mmap/mprotect/munmap reachespage_set_flags(), which invalidates and unlinks. Lift it there, keep it for system mode.
-2.42%, -4.68% wall6 RFC: accel/tcg: only poll for interrupts in blocks that can close a
cycleThe
icount_decrpoll needs to happen once per cycle in the guest CFG, not once per block, and any cycle must contain either a backward edge or an indirect one. Record both during translation and emit the check
only for blocks that have one. -6.81%, -3.10% wall7 RFC: accel/tcg: poison the jump cache instead of polling for indirect
exitsWhat patch 6 leaves behind is mostly blocks flagged for an indirect exit. Give the inline probe its own jump cache base pointer and point it at zeroes when an exit is requested: every dispatch then misses into the helper, which returns the epilogue. The poll becomes a
pointer swap on the request path. -2.79%, -1.94% wall8 RFC: tcg: fold a guest displacement into the host addressing mode
tcg_gen_qemu_ld/st cannot express a based access, so a target with a displacement in its encodings materializes the address with an lea that the host addressing mode would have done for free. Fold a preceding constant add into a new argument on the op, opt-in per
backend, wired up forx86_64user-only. -6.29%, -3.29% wallEach percentage is against the patch before it. End to end, measuring an unmodified build of the same base against the full series, five runs each, interleaved in one session so that host clock drift is shared rather than attributed (mean, with the run-to-run spread):
instructions retired: 1,646,129,294,236 -> 738,003,153,831 -55.17% (0.16%) (0.03%) wall clock: 134.934s -> 75.189s -44.28% (0.30%) (0.99%)Both endpoints ran at the same 4.782 GHz effective clock, and the .s files they produced are identical.
The two figures do not track each other, and that is the interesting part:
what the series removes is cheap, well-predicted, highly pipelined work, so it retires far more instructions than it saves time. IPC falls from 2.55 to 2.05 as the remaining work gets less regular. Patch 4 also cuts L1-icache load misses by 39.1%, because a dispatch no longer jumps into qemu's .text and evicts translated code; qemu's own .text falls from 38.9% to 5.4% of profile samples over the series.Every revision was built and measured separately, so the series bisects, and the emulated compiler produces byte-identical assembly output at every step, which is the correctness check these patches most need. Two new alpha tests cover the hazards the series creates:
tests/tcg/alpha/test-xpage-chain.c(patch 5) andtest-indirect-irq.c(patch 7). Both fail or hang if the mechanism they cover is removed, which is what makes them tests of the new behavior rather than of the old.The RFC patches need eyes I cannot supply myself. In rough order of how much I would like someone to look at them:
- Patch 5 reverses a deliberate decision made in
d3a2a1d803on the strength of an argument about the user-only invalidation paths.- Patch 6 moves system-mode interrupt latency from "bounded by block count" to "bounded by guest control flow". The bound is one straight-line run between cycles, but timer-driven guests want a closer look than I can give them. Its soundness also assumes every
goto_tbdestination passes throughtranslator_use_goto_tb(); no target in the tree bypasses it today, but nothing enforces that.- Patch 4 treats cpu flags and cflags as translation-time constants in its guards, reads a jump cache entry without
qatomic_read(), and puts knowledge of the CPUJumpCache layout intcg/tcg-op.c, where it does not belong.- Patch 7's restore in
cpu_handle_interrupt()races a concurrent poison from another thread. I believe the existing barrier around icount_decr.u16.high covers it, but my testing was single-threaded user mode.- Patch 8 only examines the immediately preceding op, refuses any access with a slow path (so user-only, and no alignment check), and leaves the i128 pairs alone.
- Patch 2's 1 MiB per vCPU is easy to justify for a single-vCPU linux-user process and less obvious for system emulation with many vCPUs. It may want to be sized per target or made tunable rather than raised unconditionally.
Patches 4 and 8 are wired up for alpha and
x86_64respectively; everything else is target-independent, and no other backend changes behavior or needs touching.Matt Turner (8):
accel/tcg: cache the result ofcurr_cflags()
accel/tcg: enlarge the TB jump cache to 64K entries accel/tcg: skip thecan_do_iostores in user-only builds
RFC: tcg: probe the TB jump cache inline instead of calling a helper
RFC: accel/tcg: allow cross-pagegoto_tbchaining in user-only builds
RFC: accel/tcg: only poll for interrupts in blocks that can close a
cycle
RFC: accel/tcg: poison the jump cache instead of polling for indirect
exits
RFC: tcg: fold a guest displacement into the host addressing modeaccel/tcg/cpu-exec-common.c | 48 +++++++++++- accel/tcg/cpu-exec.c | 54 ++++++++++++++ accel/tcg/internal-common.h | 22 +++++- accel/tcg/tb-jmp-cache.h | 2 +- accel/tcg/tcg-accel-ops.c | 2 + accel/tcg/tcg-all.c | 1 + accel/tcg/translator.c | 77 ++++++++++++++++++- cpu-target.c | 3 + include/exec/translation-block.h | 6 ++ include/exec/translator.h | 2 + include/hw/core/cpu.h | 23 +++++- include/system/tcg.h | 9 +++ include/tcg/tcg-op-common.h | 2 + include/tcg/tcg-opc.h | 9 ++- linux-user/main.c | 2 +- stubs/meson.build | 1 + stubs/tcg-cflags.c | 16 ++++ target/alpha/cpu.c | 2 +- target/alpha/translate.c | 6 +- tcg/tcg-op-ldst.c | 3 +- tcg/tcg-op.c | 86 +++++++++++++++++++++ tcg/tcg.c | 86 ++++++++++++++++++++- tcg/x86_64/tcg-target.c.inc | 61 +++++++++++++++ tcg/x86_64/tcg-target.h | 3 +Quoted 9 lines
tests/tcg/alpha/Makefile.target | 3 +- tests/tcg/alpha/test-indirect-irq.c | 53 +++++++++++++ tests/tcg/alpha/test-xpage-chain.c | 111 ++++++++++++++++++++++++++++ util/log.c | 4 + 28 files changed, 676 insertions(+), 21 deletions(-) create mode 100644 stubs/tcg-cflags.c create mode 100644 tests/tcg/alpha/test-indirect-irq.c create mode 100644 tests/tcg/alpha/test-xpage-chain.c
Quoted 4 lines
I was curious to give it a try and see if everything works before helping on review, however series does not apply on master currently.
Could you please rebase it and send a v2?
Odd, it's based on commit af06b5df26 ("Merge tag 'qom-qdev-20260814' of https://gitlab.com/mcayland-ntx/qemu into staging") from just a couple of days ago. I'll handle feedback received so far and send a v2 today.
Quoted 196 lines · Matt Turner
Quoted 193 lines · Pierrick Bouvier
Hi Matt,
Quoted 176 lines · Matt Turner
For guests running large amounts of code, most of what TCG executes is not translated guest work but the fixed overhead around it. Blocks are short and there are a great many of them, so the constant cost at each end of a block (the interrupt poll and the
can_do_iostores on entry, the dispatch on exit) ends up dominating everything else.The workload throughout is qemu-alpha running an emulated alpha gcc 16.2.0 compiling the SQLite 3.45.1 amalgamation (255k lines, -O2) on an x86-64 host, in a --static --enable-lto --target-list=alpha-linux-user build. It executes 34.2 billion TBs at 6.04 guest instructions each, and 24.6% of its TB exits cannot use
goto_tb. That is a representative shape for any guest whose text is much larger than a page: indirect calls and returns everywhere, plus direct branches that merely crossed a page boundary.The first three patches are ordinary cleanups that stand on their own. The remaining five are marked RFC individually and are where the interesting questions are.
1 accel/tcg: cache the result of
curr_cflags()Recomputed on every one of the run's 8.4 billion dispatches, from state that changes only when gdb enables single-step or a log mask moves. Cache it in CPUState and recompute from the four places that
can change an input. -5.10%2 accel/tcg: enlarge the TB jump cache to 64K entries
4096 entries is too small for a guest running a large program;
tb_htable_lookup()is 5.73% of samples. 16 bits is the knee of the
sizing curve, at 1 MiB per vCPU. -6.02%3 accel/tcg: skip the
can_do_iostores in user-only buildsTwo stores per TB that nothing in a user-only build reads: 68 billion
of them over the run. -4.55%, -4.32% wall4 RFC: tcg: probe the TB jump cache inline instead of calling a helper
95.8% of those 8.4 billion
helper_lookup_tb_ptr()calls hit the jump cache. Emit the probe inline (hash, three guarded loads,goto_ptr)
and call the helper only on a miss. -36.51%, -27.05% wall5 RFC: accel/tcg: allow cross-page
goto_tbchaining in user-only builds
translator_use_goto_tb()refuses to chain across a page. In user-only builds the invalidation path already covers what that was protecting
against: every mmap/mprotect/munmap reachespage_set_flags(), which invalidates and unlinks. Lift it there, keep it for system mode.
-2.42%, -4.68% wall6 RFC: accel/tcg: only poll for interrupts in blocks that can close a
cycleThe
icount_decrpoll needs to happen once per cycle in the guest CFG, not once per block, and any cycle must contain either a backward edge or an indirect one. Record both during translation and emit the check
only for blocks that have one. -6.81%, -3.10% wall7 RFC: accel/tcg: poison the jump cache instead of polling for indirect
exitsWhat patch 6 leaves behind is mostly blocks flagged for an indirect exit. Give the inline probe its own jump cache base pointer and point it at zeroes when an exit is requested: every dispatch then misses into the helper, which returns the epilogue. The poll becomes a
pointer swap on the request path. -2.79%, -1.94% wall8 RFC: tcg: fold a guest displacement into the host addressing mode
tcg_gen_qemu_ld/st cannot express a based access, so a target with a displacement in its encodings materializes the address with an lea that the host addressing mode would have done for free. Fold a preceding constant add into a new argument on the op, opt-in per
backend, wired up forx86_64user-only. -6.29%, -3.29% wallEach percentage is against the patch before it. End to end, measuring an unmodified build of the same base against the full series, five runs each, interleaved in one session so that host clock drift is shared rather than attributed (mean, with the run-to-run spread):
instructions retired: 1,646,129,294,236 -> 738,003,153,831 -55.17% (0.16%) (0.03%) wall clock: 134.934s -> 75.189s -44.28% (0.30%) (0.99%)Both endpoints ran at the same 4.782 GHz effective clock, and the .s files they produced are identical.
The two figures do not track each other, and that is the interesting part:
what the series removes is cheap, well-predicted, highly pipelined work, so it retires far more instructions than it saves time. IPC falls from 2.55 to 2.05 as the remaining work gets less regular. Patch 4 also cuts L1-icache load misses by 39.1%, because a dispatch no longer jumps into qemu's .text and evicts translated code; qemu's own .text falls from 38.9% to 5.4% of profile samples over the series.Every revision was built and measured separately, so the series bisects, and the emulated compiler produces byte-identical assembly output at every step, which is the correctness check these patches most need. Two new alpha tests cover the hazards the series creates:
tests/tcg/alpha/test-xpage-chain.c(patch 5) andtest-indirect-irq.c(patch 7). Both fail or hang if the mechanism they cover is removed, which is what makes them tests of the new behavior rather than of the old.The RFC patches need eyes I cannot supply myself. In rough order of how much I would like someone to look at them:
- Patch 5 reverses a deliberate decision made in
d3a2a1d803on the strength of an argument about the user-only invalidation paths.- Patch 6 moves system-mode interrupt latency from "bounded by block count" to "bounded by guest control flow". The bound is one straight-line run between cycles, but timer-driven guests want a closer look than I can give them. Its soundness also assumes every
goto_tbdestination passes throughtranslator_use_goto_tb(); no target in the tree bypasses it today, but nothing enforces that.- Patch 4 treats cpu flags and cflags as translation-time constants in its guards, reads a jump cache entry without
qatomic_read(), and puts knowledge of the CPUJumpCache layout intcg/tcg-op.c, where it does not belong.- Patch 7's restore in
cpu_handle_interrupt()races a concurrent poison from another thread. I believe the existing barrier around icount_decr.u16.high covers it, but my testing was single-threaded user mode.- Patch 8 only examines the immediately preceding op, refuses any access with a slow path (so user-only, and no alignment check), and leaves the i128 pairs alone.
- Patch 2's 1 MiB per vCPU is easy to justify for a single-vCPU linux-user process and less obvious for system emulation with many vCPUs. It may want to be sized per target or made tunable rather than raised unconditionally.
Patches 4 and 8 are wired up for alpha and
x86_64respectively; everything else is target-independent, and no other backend changes behavior or needs touching.Matt Turner (8):
accel/tcg: cache the result ofcurr_cflags()
accel/tcg: enlarge the TB jump cache to 64K entries accel/tcg: skip thecan_do_iostores in user-only builds
RFC: tcg: probe the TB jump cache inline instead of calling a helper
RFC: accel/tcg: allow cross-pagegoto_tbchaining in user-only builds
RFC: accel/tcg: only poll for interrupts in blocks that can close a
cycle
RFC: accel/tcg: poison the jump cache instead of polling for indirect
exits
RFC: tcg: fold a guest displacement into the host addressing modeaccel/tcg/cpu-exec-common.c | 48 +++++++++++- accel/tcg/cpu-exec.c | 54 ++++++++++++++ accel/tcg/internal-common.h | 22 +++++- accel/tcg/tb-jmp-cache.h | 2 +- accel/tcg/tcg-accel-ops.c | 2 + accel/tcg/tcg-all.c | 1 + accel/tcg/translator.c | 77 ++++++++++++++++++- cpu-target.c | 3 + include/exec/translation-block.h | 6 ++ include/exec/translator.h | 2 + include/hw/core/cpu.h | 23 +++++- include/system/tcg.h | 9 +++ include/tcg/tcg-op-common.h | 2 + include/tcg/tcg-opc.h | 9 ++- linux-user/main.c | 2 +- stubs/meson.build | 1 + stubs/tcg-cflags.c | 16 ++++ target/alpha/cpu.c | 2 +- target/alpha/translate.c | 6 +- tcg/tcg-op-ldst.c | 3 +- tcg/tcg-op.c | 86 +++++++++++++++++++++ tcg/tcg.c | 86 ++++++++++++++++++++- tcg/x86_64/tcg-target.c.inc | 61 +++++++++++++++ tcg/x86_64/tcg-target.h | 3 +Quoted 9 lines
tests/tcg/alpha/Makefile.target | 3 +- tests/tcg/alpha/test-indirect-irq.c | 53 +++++++++++++ tests/tcg/alpha/test-xpage-chain.c | 111 ++++++++++++++++++++++++++++ util/log.c | 4 + 28 files changed, 676 insertions(+), 21 deletions(-) create mode 100644 stubs/tcg-cflags.c create mode 100644 tests/tcg/alpha/test-indirect-irq.c create mode 100644 tests/tcg/alpha/test-xpage-chain.cI was curious to give it a try and see if everything works before helping on review, however series does not apply on master currently.
Quoted 6 lines
Quoted 1 line
Could you please rebase it and send a v2?
Odd, it's based on commit
af06b5df26("Merge tag 'qom-qdev-20260814' of https://gitlab.com/mcayland-ntx/qemu into staging") from just a couple of days ago. I'll handle feedback received so far and send a v2 today.
I've sent v2, but v1 rebased cleanly onto master for me as well.