Skip to contentSkip to conversation

xen/x86: Change stub page allocation/free

PATCH Superseded Roger Pau Monne 3 participants 4 messages Jun 11 – Jun 12

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 xen/x86: Change stub page allocation/free

From: Jason Andryuk <[email protected]>

Today the inline tracking of the stub page is problematic. 0xcc is used to indicate unused, but it is also a "clear value." A !CONFIG_PV build with smt=0 will bring up CPU0, bring up CPU1, bring down CPU1, and free the in-use stub page. CPU0 or subsequent onlined CPUs can write to the re-used page.

The new approach uses a global, CPU-indexed dynamically allocated array of stub addresses. However, to handle NUMA aware allocations, we cannot allocate all the memory in advance because of the NUMA dependency. Take advantage of the fact that Xen will attempt to contiguously pack CPUs on the same NUMA node (see normalise_cpu_order()), and on CPU bringup use the same stubs page the previous CPU did if suitable. Note the code would still function properly even if CPUs from NUMA nodes are not contiguously packed, it just consumes more memory.

stub pages are no longer freed. They remain referenced in the global CPU-indexed array and are re-used if the CPU is re-onlined.

The stubs array doesn't have an explicit lock. During boot it's accessed single threaded. During runtime, &cpu_add_remove_lock serializes access.

Fixes 7a66ac8d1633 ("x86: move syscall trampolines off the stack")
---
 xen/arch/x86/include/asm/stubs.h |  2 +-
 xen/arch/x86/setup.c             |  4 +-
 xen/arch/x86/smpboot.c           | 92 +++++++++++++++++---------------
 3 files changed, 50 insertions(+), 48 deletions(-)
xen/arch/x86/include/asm/stubs.h +1 -1
index a520928e9a50..467551136a2a 100644
@@ -32,6 +32,6 @@ struct stubs {
 };
 
 DECLARE_PER_CPU(struct stubs, stubs);
-unsigned long alloc_stub_page(unsigned int cpu, unsigned long *mfn);
+void init_stub(void);
 
 #endif /* X86_ASM_STUBS_H */
xen/arch/x86/setup.c +1 -3
index 4192edf635b6..0253d22c349d 100644
@@ -2089,9 +2089,7 @@ void asmlinkage __init noreturn __start_xen(void)
 
     init_idle_domain();
 
-    this_cpu(stubs.addr) = alloc_stub_page(smp_processor_id(),
-                                           &this_cpu(stubs).mfn);
-    BUG_ON(!this_cpu(stubs.addr));
+    init_stub();
 
     bsp_traps_reinit(); /* Needs stubs allocated, must be before presmp_initcalls. */
 
xen/arch/x86/smpboot.c +48 -44

142 lines not on this page · raw message

Signature · 1 line
2.53.0
Tested-by Reviewed-by on v2 1/1
#1 lore raw
Quoted 20 lines · Roger Pau Monne

From: Jason Andryuk <[email protected]>

Today the inline tracking of the stub page is problematic. 0xcc is used to indicate unused, but it is also a "clear value." A !CONFIG_PV build with smt=0 will bring up CPU0, bring up CPU1, bring down CPU1, and free the in-use stub page. CPU0 or subsequent onlined CPUs can write to the re-used page.

The new approach uses a global, CPU-indexed dynamically allocated array of stub addresses. However, to handle NUMA aware allocations, we cannot allocate all the memory in advance because of the NUMA dependency. Take advantage of the fact that Xen will attempt to contiguously pack CPUs on the same NUMA node (see normalise_cpu_order()), and on CPU bringup use the same stubs page the previous CPU did if suitable. Note the code would still function properly even if CPUs from NUMA nodes are not contiguously packed, it just consumes more memory.

stub pages are no longer freed. They remain referenced in the global CPU-indexed array and are re-used if the CPU is re-onlined.

Quoted 6 lines

The stubs array doesn't have an explicit lock. During boot it's accessed single threaded. During runtime, &cpu_add_remove_lock serializes access.

Fixes 7a66ac8d1633 ("x86: move syscall trampolines off the stack")

Thanks, Roger.

Quoted 3 lines
xen/arch/x86/smpboot.c +1 -2
-        nodeid_t node = cpu_to_node(cpu);
-        unsigned int memflags = node != NUMA_NO_NODE ? MEMF_node(node) : 0;
+        nodeid_t nid = cpu_to_node(cpu);

...

Quoted 1 line

+ struct page_info *pg = alloc_domheap_page(NULL, MEMF_node(nid));

I agree that MEMF_node(NUMA_NO_NODE) -> 0, so this is okay. I looked around, and there is a mix of places that special case NUMA_NO_NODE or don't, which makes it a little confusing.

xen/arch/x86/smpboot.c static int do_boot_cpu(int apicid, int cpu) · line 669

Quoted 4 lines
xen/arch/x86/smpboot.c +2 -1
-        unmap_domain_page(memset(__map_domain_page(pg), 0xcc, PAGE_SIZE));
+            if ( !pg )
+                return false;

Newline here?

xen/arch/x86/smpboot.c static int do_boot_cpu(int apicid, int cpu) · line 673

Quoted 4 lines
xen/arch/x86/smpboot.c +4 -0
+            unmap_domain_page(memset(__map_domain_page(pg), 0xcc, PAGE_SIZE));
+            addr = page_to_maddr(pg);
+        }
+        stubs[cpu] = addr;

!CONFIG_PV && smt=0 boots (doesn't blow up).

Thanks,
Jason

#2 lore raw
In reply to Roger Pau Monne
Quoted 26 lines · Roger Pau Monne
xen/arch/x86/smpboot.c +6 -10
@@ -641,41 +642,62 @@ static int do_boot_cpu(int apicid, int cpu)
     return rc;
 }
 
-#define STUB_BUF_CPU_OFFS(cpu) (((cpu) & (STUBS_PER_PAGE - 1)) * STUB_BUF_SIZE)
+/* Dynamically allocated, indexed by CPU.  Store physical address of stubs. */
+static paddr_t *__ro_after_init stubs;
 
-unsigned long alloc_stub_page(unsigned int cpu, unsigned long *mfn)
+static bool assign_stub_page(unsigned int cpu)
 {
     unsigned long stub_va;
-    struct page_info *pg;
+    paddr_t addr = stubs[cpu];
 
-    BUILD_BUG_ON(STUBS_PER_PAGE & (STUBS_PER_PAGE - 1));
-
-    if ( *mfn )
-        pg = mfn_to_page(_mfn(*mfn));
-    else
+    if ( addr == INVALID_PADDR )
     {
-        nodeid_t node = cpu_to_node(cpu);
-        unsigned int memflags = node != NUMA_NO_NODE ? MEMF_node(node) : 0;
+        nodeid_t nid = cpu_to_node(cpu);
 

xen/arch/x86/smpboot.c static int do_boot_cpu(int apicid, int cpu) · line 663

Quoted 10 lines
xen/arch/x86/smpboot.c +7 -3
-        pg = alloc_domheap_page(NULL, memflags);
-        if ( !pg )
-            return 0;
+        /*
+         * Attempt to use the same page as the previous CPU if possible,
+         * otherwise allocate a new one.
+         */
+        if ( cpu && nid == cpu_to_node(cpu - 1) &&
+             PAGE_OFFSET(stubs[cpu - 1] + STUB_BUF_SIZE) )
+            addr = stubs[cpu - 1] + STUB_BUF_SIZE;

Isn't this path also going to be taken if stubs[cpu - 1] is still INVALID_PADDR? That may be a purely hypothetical case right now, but imo would better be covered right away.

Jan

#3 lore raw
Quoted 27 lines · Jan Beulich
Quoted 26 lines · Roger Pau Monne
xen/arch/x86/smpboot.c +6 -10
@@ -641,41 +642,62 @@ static int do_boot_cpu(int apicid, int cpu)
     return rc;
 }
 
-#define STUB_BUF_CPU_OFFS(cpu) (((cpu) & (STUBS_PER_PAGE - 1)) * STUB_BUF_SIZE)
+/* Dynamically allocated, indexed by CPU.  Store physical address of stubs. */
+static paddr_t *__ro_after_init stubs;
 
-unsigned long alloc_stub_page(unsigned int cpu, unsigned long *mfn)
+static bool assign_stub_page(unsigned int cpu)
 {
     unsigned long stub_va;
-    struct page_info *pg;
+    paddr_t addr = stubs[cpu];
 
-    BUILD_BUG_ON(STUBS_PER_PAGE & (STUBS_PER_PAGE - 1));
-
-    if ( *mfn )
-        pg = mfn_to_page(_mfn(*mfn));
-    else
+    if ( addr == INVALID_PADDR )
     {
-        nodeid_t node = cpu_to_node(cpu);
-        unsigned int memflags = node != NUMA_NO_NODE ? MEMF_node(node) : 0;
+        nodeid_t nid = cpu_to_node(cpu);
 
Quoted 14 lines

xen/arch/x86/smpboot.c static int do_boot_cpu(int apicid, int cpu) · line 663

Quoted 10 lines
xen/arch/x86/smpboot.c +7 -3
-        pg = alloc_domheap_page(NULL, memflags);
-        if ( !pg )
-            return 0;
+        /*
+         * Attempt to use the same page as the previous CPU if possible,
+         * otherwise allocate a new one.
+         */
+        if ( cpu && nid == cpu_to_node(cpu - 1) &&
+             PAGE_OFFSET(stubs[cpu - 1] + STUB_BUF_SIZE) )
+            addr = stubs[cpu - 1] + STUB_BUF_SIZE;

Isn't this path also going to be taken if stubs[cpu - 1] is still INVALID_PADDR? That may be a purely hypothetical case right now, but imo would better be covered right away.

Yes, indeed I was assuming that stubs[cpu - 1] would always be populated, which it's the case currently, but you are right that it's trivial to also take this scenario into account here.

Thanks, Roger.