Skip to contentSkip to conversation

Adjustment after introducing ASSERT_ALLOC_CONTEXT

PATCH Superseded Henry Wang 3 participants 9 messages May 5, 2022

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Henry Wang

author
#0 lore raw

cover Adjustment after introducing ASSERT_ALLOC_CONTEXT

With the enhanced ASSERT_ALLOC_CONTEXT, calling request_irq before local_irq_enable on secondary cores on Arm will lead to

(XEN) Xen call trace:
(XEN) [<000000000021d86c>] alloc_xenheap_pages+0x74/0x194 (PC) (XEN) [<000000000021d864>] alloc_xenheap_pages+0x6c/0x194 (LR) (XEN) [<0000000000229e90>] xmalloc_tlsf.c#xmalloc_pool_get+0x1c/0x28 (XEN) [<000000000022a270>] xmem_pool_alloc+0x21c/0x448 (XEN) [<000000000022a8dc>] _xmalloc+0x8c/0x290
(XEN) [<000000000026b57c>] request_irq+0x40/0xb8
(XEN) [<0000000000272780>] init_timer_interrupt+0x74/0xcc (XEN) [<000000000027212c>] start_secondary+0x1b4/0x238 (XEN) [<0000000084000200>] 0000000084000200
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 4:
(XEN) Assertion '!in_irq() && (local_irq_is_enabled() || num_online_cpus() <= 1)' failed at common/page_alloc.c:2212 (XEN) ****************************************

on systems without a big enough pool for xmalloc() to cater the requested size. To solve this issue, this series introduces two patches. The first one defers the calling of request_irq on secondary CPUs after local_irq_enable on Arm. The second one moves the definition of ASSERT_ALLOC_CONTEXT to header and uses the ASSERT_ALLOC_CONTEXT to replace the original assertion in xmalloc().

Henry Wang (2):
xen/arm: Defer request_irq on secondary CPUs after local_irq_enable xen/common: Use enhanced ASSERT_ALLOC_CONTEXT in xmalloc()

 xen/arch/arm/smpboot.c    | 7 ++++---
 xen/common/page_alloc.c   | 7 -------
 xen/common/xmalloc_tlsf.c | 2 +-
 xen/include/xen/irq.h     | 7 +++++++
 4 files changed, 12 insertions(+), 11 deletions(-)
Signature · 1 line
2.25.1
author
#1 lore raw

PATCH 1/2 xen/arm: Defer request_irq on secondary CPUs after local_irq_enable

With the enhanced ASSERT_ALLOC_CONTEXT, calling request_irq before local_irq_enable on secondary cores will lead to

(XEN) Xen call trace:
(XEN) [<000000000021d86c>] alloc_xenheap_pages+0x74/0x194 (PC) (XEN) [<000000000021d864>] alloc_xenheap_pages+0x6c/0x194 (LR) (XEN) [<0000000000229e90>] xmalloc_tlsf.c#xmalloc_pool_get+0x1c/0x28 (XEN) [<000000000022a270>] xmem_pool_alloc+0x21c/0x448 (XEN) [<000000000022a8dc>] _xmalloc+0x8c/0x290
(XEN) [<000000000026b57c>] request_irq+0x40/0xb8
(XEN) [<0000000000272780>] init_timer_interrupt+0x74/0xcc (XEN) [<000000000027212c>] start_secondary+0x1b4/0x238 (XEN) [<0000000084000200>] 0000000084000200
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 4:
(XEN) Assertion '!in_irq() && (local_irq_is_enabled() || num_online_cpus() <= 1)' failed at common/page_alloc.c:2212 (XEN) ****************************************

on systems without a big enough pool for xmalloc() to cater the requested size.

Change-Id Iebdde81f52785b0c6e037c981ff68922db016d08
---
 xen/arch/arm/smpboot.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
xen/arch/arm/smpboot.c +4 -3
index 7bfd0a73a7..a057e85ac1 100644
@@ -361,9 +361,6 @@ void start_secondary(void)
 
     init_secondary_IRQ();
 
-    init_maintenance_interrupt();
-    init_timer_interrupt();
-
     set_current(idle_vcpu[cpuid]);
 
     setup_cpu_sibling_map(cpuid);
@@ -380,6 +377,10 @@ void start_secondary(void)
     cpumask_set_cpu(cpuid, &cpu_online_map);
 
     local_irq_enable();
+
+    init_maintenance_interrupt();
+    init_timer_interrupt();
+
     local_abort_enable();
 
     check_local_cpu_errata();
Signature · 1 line
2.25.1
author
#2 lore raw

RE: [PATCH 1/2] xen/arm: Defer request_irq on secondary CPUs after local_irq_enable

Hi,

Quoted 25 lines

-----Original Message-----
Subject: [PATCH 1/2] xen/arm: Defer request_irq on secondary CPUs after local_irq_enable

With the enhanced ASSERT_ALLOC_CONTEXT, calling request_irq before local_irq_enable on secondary cores will lead to

(XEN) Xen call trace:
(XEN) [<000000000021d86c>] alloc_xenheap_pages+0x74/0x194 (PC) (XEN) [<000000000021d864>] alloc_xenheap_pages+0x6c/0x194 (LR) (XEN) [<0000000000229e90>] xmalloc_tlsf.c#xmalloc_pool_get+0x1c/0x28 (XEN) [<000000000022a270>] xmem_pool_alloc+0x21c/0x448 (XEN) [<000000000022a8dc>] _xmalloc+0x8c/0x290
(XEN) [<000000000026b57c>] request_irq+0x40/0xb8
(XEN) [<0000000000272780>] init_timer_interrupt+0x74/0xcc (XEN) [<000000000027212c>] start_secondary+0x1b4/0x238 (XEN) [<0000000084000200>] 0000000084000200
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 4:
(XEN) Assertion '!in_irq() && (local_irq_is_enabled() || num_online_cpus() <= 1)' failed at common/page_alloc.c:2212 (XEN) ****************************************

Quoted 7 lines

on systems without a big enough pool for xmalloc() to cater the requested size.

Change-Id Iebdde81f52785b0c6e037c981ff68922db016d08

Sorry about the Change-Id, forgot to remove it before sending the mail to ML, will remove it in V2. Same for PATCH 2/2.

Kind regards,
Henry

Quoted 3 lines
---
 xen/arch/arm/smpboot.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
#3 lore raw

Re: [PATCH 1/2] xen/arm: Defer request_irq on secondary CPUs after local_irq_enable

In reply to Henry Wang

Hi Henry,

The CC list doesn't seem to contain the other maintainers. Please use the scripts provided in the repo (such scripts/add_maintainers.pl) to generate the CC.

Quoted 21 lines · Henry Wang

With the enhanced ASSERT_ALLOC_CONTEXT, calling request_irq before local_irq_enable on secondary cores will lead to

(XEN) Xen call trace:
(XEN) [<000000000021d86c>] alloc_xenheap_pages+0x74/0x194 (PC) (XEN) [<000000000021d864>] alloc_xenheap_pages+0x6c/0x194 (LR) (XEN) [<0000000000229e90>] xmalloc_tlsf.c#xmalloc_pool_get+0x1c/0x28 (XEN) [<000000000022a270>] xmem_pool_alloc+0x21c/0x448 (XEN) [<000000000022a8dc>] _xmalloc+0x8c/0x290
(XEN) [<000000000026b57c>] request_irq+0x40/0xb8
(XEN) [<0000000000272780>] init_timer_interrupt+0x74/0xcc (XEN) [<000000000027212c>] start_secondary+0x1b4/0x238 (XEN) [<0000000084000200>] 0000000084000200
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 4:
(XEN) Assertion '!in_irq() && (local_irq_is_enabled() || num_online_cpus() <= 1)' failed at common/page_alloc.c:2212 (XEN) ****************************************

Quoted 2 lines

on systems without a big enough pool for xmalloc() to cater the requested size.

Can you explain in the commit message, why this is OK to move the code after enabling interrupts?

Quoted 24 lines
Change-Id Iebdde81f52785b0c6e037c981ff68922db016d08
---
  xen/arch/arm/smpboot.c | 7 ++++---
  1 file changed, 4 insertions(+), 3 deletions(-)
xen/arch/arm/smpboot.c +0 -3
index 7bfd0a73a7..a057e85ac1 100644
@@ -361,9 +361,6 @@ void start_secondary(void)
 
      init_secondary_IRQ();
 
-    init_maintenance_interrupt();
-    init_timer_interrupt();
-
      set_current(idle_vcpu[cpuid]);
 
      setup_cpu_sibling_map(cpuid);

xen/arch/arm/smpboot.c void start_secondary(void) · line 386

Quoted 10 lines
xen/arch/arm/smpboot.c +4 -0
@@ -380,6 +377,10 @@ void start_secondary(void)
      cpumask_set_cpu(cpuid, &cpu_online_map);
 
      local_irq_enable();
+
+    init_maintenance_interrupt();
+    init_timer_interrupt();
+
      local_abort_enable();
 
      check_local_cpu_errata();

Cheers,

Signature · 1 line
Julien Grall
author
#4 lore raw

RE: [PATCH 1/2] xen/arm: Defer request_irq on secondary CPUs after local_irq_enable

Hi Julien,

Quoted 30 lines

-----Original Message-----
From: Julien Grall <[email protected]>
Hi Henry,

The CC list doesn't seem to contain the other maintainers. Please use the scripts provided in the repo (such scripts/add_maintainers.pl) to generate the CC.

Quoted 21 lines · Henry Wang

With the enhanced ASSERT_ALLOC_CONTEXT, calling request_irq before local_irq_enable on secondary cores will lead to

(XEN) Xen call trace:
(XEN) [<000000000021d86c>] alloc_xenheap_pages+0x74/0x194 (PC) (XEN) [<000000000021d864>] alloc_xenheap_pages+0x6c/0x194 (LR) (XEN) [<0000000000229e90>] xmalloc_tlsf.c#xmalloc_pool_get+0x1c/0x28 (XEN) [<000000000022a270>] xmem_pool_alloc+0x21c/0x448 (XEN) [<000000000022a8dc>] _xmalloc+0x8c/0x290
(XEN) [<000000000026b57c>] request_irq+0x40/0xb8
(XEN) [<0000000000272780>] init_timer_interrupt+0x74/0xcc (XEN) [<000000000027212c>] start_secondary+0x1b4/0x238 (XEN) [<0000000084000200>] 0000000084000200
(XEN)
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 4:
(XEN) Assertion '!in_irq() && (local_irq_is_enabled() || num_online_cpus() <= 1)' failed at common/page_alloc.c:2212 (XEN) ****************************************

Quoted 5 lines
Quoted 2 lines

on systems without a big enough pool for xmalloc() to cater the requested size.

Can you explain in the commit message, why this is OK to move the code after enabling interrupts?

Sure, will correct both issues in v2.

Kind regards,
Henry

Quoted 40 lines
Quoted 25 lines
Change-Id Iebdde81f52785b0c6e037c981ff68922db016d08
---
  xen/arch/arm/smpboot.c | 7 ++++---
  1 file changed, 4 insertions(+), 3 deletions(-)
xen/arch/arm/smpboot.c +0 -3
index 7bfd0a73a7..a057e85ac1 100644
@@ -361,9 +361,6 @@ void start_secondary(void)
 
      init_secondary_IRQ();
 
-    init_maintenance_interrupt();
-    init_timer_interrupt();
-
      set_current(idle_vcpu[cpuid]);
 
      setup_cpu_sibling_map(cpuid);

xen/arch/arm/smpboot.c void start_secondary(void) · line 386

Quoted 10 lines
xen/arch/arm/smpboot.c +4 -0
@@ -380,6 +377,10 @@ void start_secondary(void)
      cpumask_set_cpu(cpuid, &cpu_online_map);
 
      local_irq_enable();
+
+    init_maintenance_interrupt();
+    init_timer_interrupt();
+
      local_abort_enable();
 
      check_local_cpu_errata();

Cheers,

--
Julien Grall

author
#5 lore raw

PATCH 2/2 xen/common: Use enhanced ASSERT_ALLOC_CONTEXT in xmalloc()

In reply to Henry Wang

xmalloc() will use a pool for allocation smaller than a page. The pool is extended only when there are no more space. At which point, alloc_xenheap_pages() is called to add more memory.

xmalloc() must be protected by ASSERT_ALLOC_CONTEXT. It should not rely on pool expanding to trigger the ASSERT_ALLOC_CONTEXT in alloc_xenheap_pages(). Hence, this commit moves the definition of ASSERT_ALLOC_CONTEXT to header and uses the ASSERT_ALLOC_CONTEXT to replace the original assertion in xmalloc().

Change-Id Ia463d2241e80e8a78d7dbb5b2318694d3ca5ed67
---
 xen/common/page_alloc.c   | 7 -------
 xen/common/xmalloc_tlsf.c | 2 +-
 xen/include/xen/irq.h     | 7 +++++++
 3 files changed, 8 insertions(+), 8 deletions(-)
xen/common/page_alloc.c +0 -7
index e866e0d864..ea59cd1a4a 100644
@@ -162,13 +162,6 @@
 static char __initdata opt_badpage[100] = "";
 string_param("badpage", opt_badpage);
 
-/*
- * Heap allocations may need TLB flushes which may require IRQs to be
- * enabled (except when only 1 PCPU is online).
- */
-#define ASSERT_ALLOC_CONTEXT() \
-    ASSERT(!in_irq() && (local_irq_is_enabled() || num_online_cpus() <= 1))
-
 /*
  * no-bootscrub -> Free pages are not zeroed during boot.
  */
xen/common/xmalloc_tlsf.c +1 -1
index d2ad909502..b8f838ae74 100644
@@ -594,7 +594,7 @@ void *_xmalloc(unsigned long size, unsigned long align)
 {
     void *p = NULL;
 
-    ASSERT(!in_irq());
+    ASSERT_ALLOC_CONTEXT();
 
     if ( !size )
         return ZERO_BLOCK_PTR;
xen/include/xen/irq.h +7 -0
index d8beadd16b..300625e56d 100644
@@ -10,6 +10,13 @@
 #include <asm/hardirq.h>
 #include <public/event_channel.h>
 
+/*
+ * Heap allocations may need TLB flushes which may require IRQs to be
+ * enabled (except when only 1 PCPU is online).
+ */
+#define ASSERT_ALLOC_CONTEXT() \
+    ASSERT(!in_irq() && (local_irq_is_enabled() || num_online_cpus() <= 1))
+
 struct irqaction {
     void (*handler)(int, void *, struct cpu_user_regs *);
     const char *name;
Signature · 1 line
2.25.1
#6 lore raw

Re: [PATCH 2/2] xen/common: Use enhanced ASSERT_ALLOC_CONTEXT in xmalloc()

Quoted 10 lines · Henry Wang

xmalloc() will use a pool for allocation smaller than a page. The pool is extended only when there are no more space. At which point, alloc_xenheap_pages() is called to add more memory.

xmalloc() must be protected by ASSERT_ALLOC_CONTEXT. It should not rely on pool expanding to trigger the ASSERT_ALLOC_CONTEXT in alloc_xenheap_pages(). Hence, this commit moves the definition of ASSERT_ALLOC_CONTEXT to header and uses the ASSERT_ALLOC_CONTEXT to replace the original assertion in xmalloc().

Quoted 4 lines
Change-Id Ia463d2241e80e8a78d7dbb5b2318694d3ca5ed67

Just two formal remarks for starters: What's this last tag? And why am I on the To: list of this patch, when ...

Quoted 5 lines
---
 xen/common/page_alloc.c   | 7 -------
 xen/common/xmalloc_tlsf.c | 2 +-
 xen/include/xen/irq.h     | 7 +++++++
 3 files changed, 8 insertions(+), 8 deletions(-)

... with this diffstat you should instead have _Cc_-ed REST maintainers?

Jan

author
#7 lore raw

RE: [PATCH 2/2] xen/common: Use enhanced ASSERT_ALLOC_CONTEXT in xmalloc()

Hi Jan,

Quoted 6 lines

-----Original Message-----
From: Jan Beulich <[email protected]>

Quoted 1 line

Change-Id: Ia463d2241e80e8a78d7dbb5b2318694d3ca5ed67

Just two formal remarks for starters: What's this last tag? And why am I on the To: list of this patch, when ...

I forgot to remove the Change-Id before sending the patch, sorry about that. This will be removed in v2.

Quoted 8 lines
Quoted 6 lines
---
 xen/common/page_alloc.c   | 7 -------
 xen/common/xmalloc_tlsf.c | 2 +-
 xen/include/xen/irq.h     | 7 +++++++
 3 files changed, 8 insertions(+), 8 deletions(-)

... with this diffstat you should instead have _Cc_-ed REST maintainers?

I think this is because by switching to add-maintainers.pl I didn't change my git send-email command line where originally it has a --suppress-cc=all. Since you are the maintainer of common code I thought I should directly "To" you.

Kind regards,
Henry

Quoted 2 lines

Jan

#8 lore raw

Re: [PATCH 2/2] xen/common: Use enhanced ASSERT_ALLOC_CONTEXT in xmalloc()

Quoted 17 lines · Henry Wang
Quoted 6 lines

-----Original Message-----
From: Jan Beulich <[email protected]>

Quoted 1 line

Change-Id: Ia463d2241e80e8a78d7dbb5b2318694d3ca5ed67

Just two formal remarks for starters: What's this last tag? And why am I on the To: list of this patch, when ...

I forgot to remove the Change-Id before sending the patch, sorry about that. This will be removed in v2.

Quoted 7 lines
Quoted 6 lines
---
 xen/common/page_alloc.c   | 7 -------
 xen/common/xmalloc_tlsf.c | 2 +-
 xen/include/xen/irq.h     | 7 +++++++
 3 files changed, 8 insertions(+), 8 deletions(-)
Quoted 6 lines
Quoted 1 line

... with this diffstat you should instead have _Cc_-ed REST maintainers?

I think this is because by switching to add-maintainers.pl I didn't change my git send-email command line where originally it has a --suppress-cc=all. Since you are the maintainer of common code I thought I should directly "To" you.

I'm sorry, but no: For one I am not "the" maintainer, but one of several. And then patches are generally only sent _To_ the list, with _Cc_ to maintainers.

Jan