Skip to contentSkip to conversation

tcg: defer tb_flush when initial region alloc fails

PATCH In review Yogesh Vyas 2 participants 4 messages Aug 5 – Aug 11

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 tcg: defer tb_flush when initial region alloc fails

Under MTTCG, busy vCPUs can temporarily exhaust the fixed code-gen region pool by holding multiple regions. A newly registered vCPU thread that then fails tcg_region_initial_alloc() currently hits g_assert and aborts QEMU — seen during vCPU hotplug on ppc64 TCG (virt-type=qemu) with a large maxcpus.
Instead of asserting, mark the context with tb_flush_pending. The first tcg_tb_alloc() returns NULL so the existing tb_gen_code path queues a tb_flush; tcg_region_reset_all() then repacks the pool and the new thread obtains a region on retry.
This mirrors the graceful exhaustion handling already used by the runtime tcg_region_alloc() path.

---
 include/tcg/tcg.h |  9 +++++++++
 tcg/region.c      | 15 ++++++++++++++-
 tcg/tcg.c         | 12 ++++++++++++
 3 files changed, 35 insertions(+), 1 deletion(-)
include/tcg/tcg.h +9 -0
index 7669dc1c2d..8a05e3af7b 100644
@@ -381,6 +381,15 @@ struct TCGContext {
     /* Threshold to flush the translated code buffer.  */
     void *code_gen_highwater;
 
+    /*
+     * Set when this context (a newly registered vCPU thread) could not be
+     * given an initial code-gen region because the region pool was
+     * momentarily exhausted by other busy vCPUs. Rather than aborting, the
+     * thread defers a tb_flush: the next tcg_tb_alloc() forces a flush +
+     * retry, which resets the region pool and reclaims slack for this thread.
+     */
+    bool tb_flush_pending;
+
     /* Track which vCPU triggers events */
     CPUState *cpu;                      /* *_trans */
 
tcg/region.c +14 -1
index 5d4be1453b..1200947385 100644
@@ -393,7 +393,20 @@ bool tcg_region_alloc(TCGContext *s)
 static void tcg_region_initial_alloc__locked(TCGContext *s)
 {
     bool err = tcg_region_alloc__locked(s);
-    g_assert(!err);
+
+    /*
+     * The region pool can be momentarily exhausted when many busy vCPUs have
+     * each grabbed several regions during translation. A newly started vCPU
+     * thread that cannot get its initial region must not abort QEMU: instead
+     * flag a deferred tb_flush. The thread's first tcg_tb_alloc() will force a
+     * flush + retry (tcg_region_reset_all() resets region.current and
+     * re-packs the live contexts), which reclaims that slack and assigns this
+     * context a region. This mirrors the graceful runtime path in
+     * tcg_tb_alloc()/tcg_region_alloc().
+     */
+    if (err) {
+        s->tb_flush_pending = true;
+    }
 }
 
 void tcg_region_initial_alloc(TCGContext *s)
tcg/tcg.c +12 -0
index 1e77f2365a..ea66fb6f84 100644
@@ -1830,6 +1830,18 @@ TranslationBlock *tcg_tb_alloc(TCGContext *s)
     TranslationBlock *tb;
     void *next;
 
+    /*
+     * A newly started vCPU thread that could not obtain its initial region
+     * (region pool momentarily exhausted) deferred a flush. Trigger it now:
+     * returning NULL makes the caller queue a tb_flush and retry, which
+     * resets the region pool and gives this context a valid region. Do this
+     * before touching code_gen_ptr, which is not yet valid for this context.
+     */
+    if (unlikely(s->tb_flush_pending)) {
+        s->tb_flush_pending = false;
+        return NULL;
+    }
+
  retry:
     tb = (void *)ROUND_UP((uintptr_t)s->code_gen_ptr, align);
     next = (void *)ROUND_UP((uintptr_t)(tb + 1), align);
Signature · 1 line
2.54.0
#1 lore raw

Hi Richard, Chinmay,

Could you please share your review comments on this patch?

Regards,
Yogesh Vyas

Quoted 105 lines · On Wed, 5 Aug, 2026, 22:45 Yogesh Vyas, <[email protected]> wrote:

Under MTTCG, busy vCPUs can temporarily exhaust the fixed code-gen region pool by holding multiple regions. A newly registered vCPU thread that then fails tcg_region_initial_alloc() currently hits g_assert and aborts QEMU — seen during vCPU hotplug on ppc64 TCG (virt-type=qemu) with a large maxcpus.
Instead of asserting, mark the context with tb_flush_pending. The first tcg_tb_alloc() returns NULL so the existing tb_gen_code path queues a tb_flush; tcg_region_reset_all() then repacks the pool and the new thread obtains a region on retry.
This mirrors the graceful exhaustion handling already used by the runtime tcg_region_alloc() path.

---
 include/tcg/tcg.h |  9 +++++++++
 tcg/region.c      | 15 ++++++++++++++-
 tcg/tcg.c         | 12 ++++++++++++
 3 files changed, 35 insertions(+), 1 deletion(-)
include/tcg/tcg.h +4 -0
index 7669dc1c2d..8a05e3af7b 100644
@@ -381,6 +381,15 @@ struct TCGContext {
     /* Threshold to flush the translated code buffer.  */
     void *code_gen_highwater;
 
+    /*
+     * Set when this context (a newly registered vCPU thread) could not be
+     * given an initial code-gen region because the region pool was
+     * momentarily exhausted by other busy vCPUs. Rather than aborting,

the

include/tcg/tcg.h +2 -0
+     * thread defers a tb_flush: the next tcg_tb_alloc() forces a flush +
+     * retry, which resets the region pool and reclaims slack for this

thread.

include/tcg/tcg.h +3 -0
+     */
+    bool tb_flush_pending;
+
     /* Track which vCPU triggers events */
     CPUState *cpu;                      /* *_trans */
tcg/region.c +3 -1
index 5d4be1453b..1200947385 100644
@@ -393,7 +393,20 @@ bool tcg_region_alloc(TCGContext *s)
 static void tcg_region_initial_alloc__locked(TCGContext *s)
 {
     bool err = tcg_region_alloc__locked(s);
-    g_assert(!err);
+
+    /*
+     * The region pool can be momentarily exhausted when many busy vCPUs

have
+ * each grabbed several regions during translation. A newly started
vCPU
+ * thread that cannot get its initial region must not abort QEMU:
instead
+ * flag a deferred tb_flush. The thread's first tcg_tb_alloc() will
force a

tcg/region.c +2 -0
+     * flush + retry (tcg_region_reset_all() resets region.current and
+     * re-packs the live contexts), which reclaims that slack and assigns

this

tcg/region.c +6 -0
+     * context a region. This mirrors the graceful runtime path in
+     * tcg_tb_alloc()/tcg_region_alloc().
+     */
+    if (err) {
+        s->tb_flush_pending = true;
+    }
 }
 
 void tcg_region_initial_alloc(TCGContext *s)
tcg/tcg.c +2 -0
index 1e77f2365a..ea66fb6f84 100644
@@ -1830,6 +1830,18 @@ TranslationBlock *tcg_tb_alloc(TCGContext *s)
     TranslationBlock *tb;
     void *next;
 
+    /*
+     * A newly started vCPU thread that could not obtain its initial

region
+ * (region pool momentarily exhausted) deferred a flush. Trigger it
now:

tcg/tcg.c +2 -0
+     * returning NULL makes the caller queue a tb_flush and retry, which
+     * resets the region pool and gives this context a valid region. Do

this
+ * before touching code_gen_ptr, which is not yet valid for this
context.

tcg/tcg.c +6 -1
+     */
+    if (unlikely(s->tb_flush_pending)) {
+        s->tb_flush_pending = false;
+        return NULL;
+    }
+
  retry:
     tb = (void *)ROUND_UP((uintptr_t)s->code_gen_ptr, align);
     next = (void *)ROUND_UP((uintptr_t)(tb + 1), align);
--

2.54.0

#2 lore raw
In reply to Yogesh Vyas

Hi Yogesh,

Thanks for coming up with this solution ! It looks good to me. I had a doubt though.
After tb_flush_pending is set, tcg_region_reset_all() allocates new region to all the contexts. Is there a possibility of region not being allocated here as well and the loop repeating ?

Thanks,
Chinmay

Quoted 90 lines · Yogesh Vyas

Under MTTCG, busy vCPUs can temporarily exhaust the fixed code-gen region pool by holding multiple regions. A newly registered vCPU thread that then fails tcg_region_initial_alloc() currently hits g_assert and aborts QEMU — seen during vCPU hotplug on ppc64 TCG (virt-type=qemu) with a large maxcpus.
Instead of asserting, mark the context with tb_flush_pending. The first tcg_tb_alloc() returns NULL so the existing tb_gen_code path queues a tb_flush; tcg_region_reset_all() then repacks the pool and the new thread obtains a region on retry.
This mirrors the graceful exhaustion handling already used by the runtime tcg_region_alloc() path.

---
  include/tcg/tcg.h |  9 +++++++++
  tcg/region.c      | 15 ++++++++++++++-
  tcg/tcg.c         | 12 ++++++++++++
  3 files changed, 35 insertions(+), 1 deletion(-)
include/tcg/tcg.h +9 -0
index 7669dc1c2d..8a05e3af7b 100644
@@ -381,6 +381,15 @@ struct TCGContext {
      /* Threshold to flush the translated code buffer.  */
      void *code_gen_highwater;
 
+    /*
+     * Set when this context (a newly registered vCPU thread) could not be
+     * given an initial code-gen region because the region pool was
+     * momentarily exhausted by other busy vCPUs. Rather than aborting, the
+     * thread defers a tb_flush: the next tcg_tb_alloc() forces a flush +
+     * retry, which resets the region pool and reclaims slack for this thread.
+     */
+    bool tb_flush_pending;
+
      /* Track which vCPU triggers events */
      CPUState *cpu;                      /* *_trans */
 
tcg/region.c +14 -1
index 5d4be1453b..1200947385 100644
@@ -393,7 +393,20 @@ bool tcg_region_alloc(TCGContext *s)
  static void tcg_region_initial_alloc__locked(TCGContext *s)
  {
      bool err = tcg_region_alloc__locked(s);
-    g_assert(!err);
+
+    /*
+     * The region pool can be momentarily exhausted when many busy vCPUs have
+     * each grabbed several regions during translation. A newly started vCPU
+     * thread that cannot get its initial region must not abort QEMU: instead
+     * flag a deferred tb_flush. The thread's first tcg_tb_alloc() will force a
+     * flush + retry (tcg_region_reset_all() resets region.current and
+     * re-packs the live contexts), which reclaims that slack and assigns this
+     * context a region. This mirrors the graceful runtime path in
+     * tcg_tb_alloc()/tcg_region_alloc().
+     */
+    if (err) {
+        s->tb_flush_pending = true;
+    }
  }
 
  void tcg_region_initial_alloc(TCGContext *s)
tcg/tcg.c +12 -0
index 1e77f2365a..ea66fb6f84 100644
@@ -1830,6 +1830,18 @@ TranslationBlock *tcg_tb_alloc(TCGContext *s)
      TranslationBlock *tb;
      void *next;
 
+    /*
+     * A newly started vCPU thread that could not obtain its initial region
+     * (region pool momentarily exhausted) deferred a flush. Trigger it now:
+     * returning NULL makes the caller queue a tb_flush and retry, which
+     * resets the region pool and gives this context a valid region. Do this
+     * before touching code_gen_ptr, which is not yet valid for this context.
+     */
+    if (unlikely(s->tb_flush_pending)) {
+        s->tb_flush_pending = false;
+        return NULL;
+    }
+
   retry:
      tb = (void *)ROUND_UP((uintptr_t)s->code_gen_ptr, align);
      next = (void *)ROUND_UP((uintptr_t)(tb + 1), align);
#3 lore raw

Hi Chinmay,

Thank you for your response.
No, It appears tcg_region_reset_all() always succeeds in giving at least one region to each live context as n_ctxs <= region.n, and the hotplug path does one flush and retry, therefore should not cause a repeating alloc failure.

Regards,
Yogesh Vyas

Quoted 116 lines · On Mon, 10 Aug, 2026, 16:27 Chinmay Rath, <[email protected]> wrote:

Hi Yogesh,

Thanks for coming up with this solution ! It looks good to me. I had a doubt though.
After tb_flush_pending is set, tcg_region_reset_all() allocates new region to all the contexts. Is there a possibility of region not being allocated here as well and the loop repeating ?

Thanks,
Chinmay

Quoted 26 lines · Yogesh Vyas

Under MTTCG, busy vCPUs can temporarily exhaust the fixed code-gen region pool by holding multiple regions. A newly registered vCPU thread that then fails tcg_region_initial_alloc() currently hits g_assert and aborts QEMU — seen during vCPU hotplug on ppc64 TCG (virt-type=qemu) with a large maxcpus.
Instead of asserting, mark the context with tb_flush_pending. The first tcg_tb_alloc() returns NULL so the existing tb_gen_code path queues a tb_flush; tcg_region_reset_all() then repacks the pool and the new thread obtains a region on retry.
This mirrors the graceful exhaustion handling already used by the runtime tcg_region_alloc() path.

---
  include/tcg/tcg.h |  9 +++++++++
  tcg/region.c      | 15 ++++++++++++++-
  tcg/tcg.c         | 12 ++++++++++++
  3 files changed, 35 insertions(+), 1 deletion(-)
include/tcg/tcg.h +0 -0
index 7669dc1c2d..8a05e3af7b 100644

include/tcg/tcg.h struct TCGContext { · line 385

Quoted 5 lines
include/tcg/tcg.h +2 -0
@@ -381,6 +381,15 @@ struct TCGContext {
      /* Threshold to flush the translated code buffer.  */
      void *code_gen_highwater;
 
+    /*
+     * Set when this context (a newly registered vCPU thread) could not

be

include/tcg/tcg.h struct TCGContext { · line 387

Quoted 2 lines
include/tcg/tcg.h +2 -0
+     * given an initial code-gen region because the region pool was
+     * momentarily exhausted by other busy vCPUs. Rather than aborting,

the

include/tcg/tcg.h struct TCGContext { · line 388

Quoted 1 line
include/tcg/tcg.h +1 -0
+     * thread defers a tb_flush: the next tcg_tb_alloc() forces a flush

+

include/tcg/tcg.h struct TCGContext { · line 389

Quoted 1 line
include/tcg/tcg.h +1 -0
+     * retry, which resets the region pool and reclaims slack for this

thread.

Quoted 11 lines
include/tcg/tcg.h +3 -0
+     */
+    bool tb_flush_pending;
+
      /* Track which vCPU triggers events */
      CPUState *cpu;                      /* *_trans */
 
tcg/region.c +0 -0
index 5d4be1453b..1200947385 100644

tcg/region.c bool tcg_region_alloc(TCGContext *s) · line 398

Quoted 7 lines
tcg/region.c +3 -1
@@ -393,7 +393,20 @@ bool tcg_region_alloc(TCGContext *s)
  static void tcg_region_initial_alloc__locked(TCGContext *s)
  {
      bool err = tcg_region_alloc__locked(s);
-    g_assert(!err);
+
+    /*
+     * The region pool can be momentarily exhausted when many busy

vCPUs have

tcg/region.c bool tcg_region_alloc(TCGContext *s) · line 399

Quoted 1 line
tcg/region.c +1 -0
+     * each grabbed several regions during translation. A newly started

vCPU

tcg/region.c bool tcg_region_alloc(TCGContext *s) · line 400

Quoted 1 line
tcg/region.c +1 -0
+     * thread that cannot get its initial region must not abort QEMU:

instead

tcg/region.c bool tcg_region_alloc(TCGContext *s) · line 401

Quoted 1 line
tcg/region.c +1 -0
+     * flag a deferred tb_flush. The thread's first tcg_tb_alloc() will

force a

tcg/region.c bool tcg_region_alloc(TCGContext *s) · line 403

Quoted 2 lines
tcg/region.c +2 -0
+     * flush + retry (tcg_region_reset_all() resets region.current and
+     * re-packs the live contexts), which reclaims that slack and

assigns this

Quoted 14 lines
tcg/region.c +6 -0
+     * context a region. This mirrors the graceful runtime path in
+     * tcg_tb_alloc()/tcg_region_alloc().
+     */
+    if (err) {
+        s->tb_flush_pending = true;
+    }
  }
 
  void tcg_region_initial_alloc(TCGContext *s)
tcg/tcg.c +0 -0
index 1e77f2365a..ea66fb6f84 100644

tcg/tcg.c TranslationBlock *tcg_tb_alloc(TCGContext *s) · line 1834

Quoted 5 lines
tcg/tcg.c +2 -0
@@ -1830,6 +1830,18 @@ TranslationBlock *tcg_tb_alloc(TCGContext *s)
      TranslationBlock *tb;
      void *next;
 
+    /*
+     * A newly started vCPU thread that could not obtain its initial

region

tcg/tcg.c TranslationBlock *tcg_tb_alloc(TCGContext *s) · line 1835

Quoted 1 line
tcg/tcg.c +1 -0
+     * (region pool momentarily exhausted) deferred a flush. Trigger it

now:

tcg/tcg.c TranslationBlock *tcg_tb_alloc(TCGContext *s) · line 1837

Quoted 2 lines
tcg/tcg.c +2 -0
+     * returning NULL makes the caller queue a tb_flush and retry, which
+     * resets the region pool and gives this context a valid region. Do

this

tcg/tcg.c TranslationBlock *tcg_tb_alloc(TCGContext *s) · line 1838

Quoted 1 line
tcg/tcg.c +1 -0
+     * before touching code_gen_ptr, which is not yet valid for this

context.

Quoted 9 lines
tcg/tcg.c +6 -0
+     */
+    if (unlikely(s->tb_flush_pending)) {
+        s->tb_flush_pending = false;
+        return NULL;
+    }
+
   retry:
      tb = (void *)ROUND_UP((uintptr_t)s->code_gen_ptr, align);
      next = (void *)ROUND_UP((uintptr_t)(tb + 1), align);