xen/x86: Change stub page allocation/free
PATCH v2 1/1 xen/x86: Change stub page allocation/free
From: Jason Andryuk <[email protected]>
Today the inline tracking of the stub page is problematic. 0xcc is used to indicate unused, but it is also a "clear value." A !CONFIG_PV build with smt=0 will bring up CPU0, bring up CPU1, bring down CPU1, and free the in-use stub page. CPU0 or subsequent onlined CPUs can write to the re-used page.
The new approach uses a global, CPU-indexed dynamically allocated array of stub addresses. However, to handle NUMA aware allocations, we cannot allocate all the memory in advance because of the NUMA dependency. Take advantage of the fact that Xen will attempt to contiguously pack CPUs on the same NUMA node (see normalise_cpu_order()), and on CPU bringup use the same stubs page the previous CPU did if suitable. Note the code would still function properly even if CPUs from NUMA nodes are not contiguously packed, it just consumes more memory.
stub pages are no longer freed. They remain referenced in the global CPU-indexed array and are re-used if the CPU is re-onlined.
The stubs array doesn't have an explicit lock. During boot it's accessed single threaded. During runtime, &cpu_add_remove_lock serializes access.
--- xen/arch/x86/include/asm/stubs.h | 2 +- xen/arch/x86/setup.c | 4 +- xen/arch/x86/smpboot.c | 92 +++++++++++++++++--------------- 3 files changed, 50 insertions(+), 48 deletions(-)
xen/arch/x86/include/asm/stubs.h +1 -1
@@ -32,6 +32,6 @@ struct stubs { }; DECLARE_PER_CPU(struct stubs, stubs); -unsigned long alloc_stub_page (unsigned int cpu,unsigned long *mfn); +void init_stub (void );#endif /* X86_ASM_STUBS_H */
xen/arch/x86/setup.c +1 -3
@@ -2089,9 +2089,7 @@ void asmlinkage __init noreturn __start_xen(void)init_idle_domain (); -this_cpu (stubs.addr) =alloc_stub_page (smp_processor_id (), - &this_cpu (stubs).mfn); -BUG_ON (!this_cpu (stubs.addr)); +init_stub ();bsp_traps_reinit ();/* Needs stubs allocated, must be before presmp_initcalls. */
xen/arch/x86/smpboot.c +48 -44
142 lines not on this page · raw message
Signature · 1 line
2.53.0
Quoted 20 lines · Roger Pau Monne
From: Jason Andryuk <[email protected]>
Today the inline tracking of the stub page is problematic.
0xccis used to indicate unused, but it is also a "clear value." A !CONFIG_PV build with smt=0 will bring up CPU0, bring up CPU1, bring down CPU1, and free the in-use stub page. CPU0 or subsequent onlined CPUs can write to the re-used page.The new approach uses a global, CPU-indexed dynamically allocated array of stub addresses. However, to handle NUMA aware allocations, we cannot allocate all the memory in advance because of the NUMA dependency. Take advantage of the fact that Xen will attempt to contiguously pack CPUs on the same NUMA node (see
normalise_cpu_order()), and on CPU bringup use the same stubs page the previous CPU did if suitable. Note the code would still function properly even if CPUs from NUMA nodes are not contiguously packed, it just consumes more memory.stub pages are no longer freed. They remain referenced in the global CPU-indexed array and are re-used if the CPU is re-onlined.
Quoted 6 lines
The stubs array doesn't have an explicit lock. During boot it's accessed single threaded. During runtime,
&cpu_add_remove_lockserializes access.Fixes 7a66ac8d1633 ("x86: move syscall trampolines off the stack")Signed-off-by Jason Andryuk <[email protected]>Signed-off-by Roger Pau Monné <[email protected]>
Thanks, Roger.
Quoted 3 lines
xen/arch/x86/smpboot.c +1
-2-nodeid_t node =cpu_to_node (cpu); -unsigned int memflags = node !=NUMA_NO_NODE ?MEMF_node (node) :0 ; +nodeid_t nid =cpu_to_node (cpu);
...
Quoted 1 line
+ struct
page_info*pg =alloc_domheap_page(NULL, MEMF_node(nid));
I agree that MEMF_node(NUMA_NO_NODE) -> 0, so this is okay. I looked around, and there is a mix of places that special case NUMA_NO_NODE or don't, which makes it a little confusing.
xen/arch/x86/smpboot.c static int do_boot_cpu(int apicid, int cpu) · line 669
Quoted 4 lines
xen/arch/x86/smpboot.c +2
-1-unmap_domain_page (memset (__map_domain_page (pg),0xcc ,PAGE_SIZE )); +if ( !pg ) +return false;
Newline here?
xen/arch/x86/smpboot.c static int do_boot_cpu(int apicid, int cpu) · line 673
Quoted 4 lines
xen/arch/x86/smpboot.c +4
-0+unmap_domain_page (memset (__map_domain_page (pg),0xcc ,PAGE_SIZE )); + addr =page_to_maddr (pg); + } + stubs[cpu] = addr;
!CONFIG_PV && smt=0 boots (doesn't blow up).
Thanks,
Jason
Quoted 26 lines · Roger Pau Monne
xen/arch/x86/smpboot.c +6
-10@@ -641,41 +642,62 @@ static int do_boot_cpu(int apicid, int cpu)return rc; } -#define STUB_BUF_CPU_OFFS (cpu) (((cpu) & (STUBS_PER_PAGE - 1)) * STUB_BUF_SIZE) +/* Dynamically allocated, indexed by CPU. Store physical address of stubs. */ +static paddr_t *__ro_after_init stubs; -unsigned long alloc_stub_page (unsigned int cpu,unsigned long *mfn) +static bool assign_stub_page (unsigned int cpu) {unsigned long stub_va; -struct page_info *pg; +paddr_t addr = stubs[cpu]; -BUILD_BUG_ON (STUBS_PER_PAGE & (STUBS_PER_PAGE -1 )); - -if ( *mfn ) - pg =mfn_to_page (_mfn (*mfn)); -else +if ( addr ==INVALID_PADDR ) { -nodeid_t node =cpu_to_node (cpu); -unsigned int memflags = node !=NUMA_NO_NODE ?MEMF_node (node) :0 ; +nodeid_t nid =cpu_to_node (cpu);
xen/arch/x86/smpboot.c static int do_boot_cpu(int apicid, int cpu) · line 663
Quoted 10 lines
xen/arch/x86/smpboot.c +7
-3- pg =alloc_domheap_page (NULL , memflags); -if ( !pg ) -return 0 ; +/* +* Attempt to use the same page as the previous CPU if possible, +* otherwise allocate a new one. +*/ +if ( cpu && nid ==cpu_to_node (cpu -1 ) && +PAGE_OFFSET (stubs[cpu -1 ] +STUB_BUF_SIZE ) ) + addr = stubs[cpu -1 ] +STUB_BUF_SIZE ;
Isn't this path also going to be taken if stubs[cpu - 1] is still INVALID_PADDR? That may be a purely hypothetical case right now, but imo would better be covered right away.
Jan
Quoted 27 lines · Jan Beulich
Quoted 26 lines · Roger Pau Monne
xen/arch/x86/smpboot.c +6
-10@@ -641,41 +642,62 @@ static int do_boot_cpu(int apicid, int cpu)return rc; } -#define STUB_BUF_CPU_OFFS (cpu) (((cpu) & (STUBS_PER_PAGE - 1)) * STUB_BUF_SIZE) +/* Dynamically allocated, indexed by CPU. Store physical address of stubs. */ +static paddr_t *__ro_after_init stubs; -unsigned long alloc_stub_page (unsigned int cpu,unsigned long *mfn) +static bool assign_stub_page (unsigned int cpu) {unsigned long stub_va; -struct page_info *pg; +paddr_t addr = stubs[cpu]; -BUILD_BUG_ON (STUBS_PER_PAGE & (STUBS_PER_PAGE -1 )); - -if ( *mfn ) - pg =mfn_to_page (_mfn (*mfn)); -else +if ( addr ==INVALID_PADDR ) { -nodeid_t node =cpu_to_node (cpu); -unsigned int memflags = node !=NUMA_NO_NODE ?MEMF_node (node) :0 ; +nodeid_t nid =cpu_to_node (cpu);
Quoted 14 lines
xen/arch/x86/smpboot.c static int do_boot_cpu(int apicid, int cpu) · line 663
Quoted 10 lines
xen/arch/x86/smpboot.c +7
-3- pg =alloc_domheap_page (NULL , memflags); -if ( !pg ) -return 0 ; +/* +* Attempt to use the same page as the previous CPU if possible, +* otherwise allocate a new one. +*/ +if ( cpu && nid ==cpu_to_node (cpu -1 ) && +PAGE_OFFSET (stubs[cpu -1 ] +STUB_BUF_SIZE ) ) + addr = stubs[cpu -1 ] +STUB_BUF_SIZE ;Isn't this path also going to be taken if stubs[cpu - 1] is still
INVALID_PADDR? That may be a purely hypothetical case right now, but imo would better be covered right away.
Yes, indeed I was assuming that stubs[cpu - 1] would always be populated, which it's the case currently, but you are right that it's trivial to also take this scenario into account here.
Thanks, Roger.