support MAXSMP
Subsystems (2)
PATCH 1/1 support MAXSMP
The MAXSMP config option requires CPUMASK_OFFSTACK, which in turn requires we init the memory for the maps while we bringing up the cpus. MAXSMP also increases NR_CPUS to 4096. This increase in size exposed an issue in the argument construction for mulitcalls from xen_flush_tlb_others. The args should only need space for the actual number of cpus, which with xen is currently only up to 32.
Andrew
The MAXSMP config option requires CPUMASK_OFFSTACK, which in turn requires we init the memory for the maps while we bringing up the cpus. MAXSMP also increases NR_CPUS to 4096. This increase in size exposed an issue in the argument construction for mulitcalls from xen_flush_tlb_others. The args should only need space for the actual number of cpus, which with xen is currently only up to 32.
--- arch/x86/xen/mmu.c | 2 +- arch/x86/xen/smp.c | 7 +++++++ 2 files changed, 8 insertions(+), 1 deletions(-)
arch/x86/xen/mmu.c +1 -1
@@ -1293,7 +1293,7 @@ static void xen_flush_tlb_others(const struct cpumask *cpus, {struct {struct mmuext_op op; - DECLARE_BITMAP(mask, NR_CPUS); + DECLARE_BITMAP(mask, num_processors); } *args;struct multicall_space mcs;
arch/x86/xen/smp.c +7 -0
@@ -178,11 +178,18 @@ static void __init xen_smp_prepare_boot_cpu(void)static void __init xen_smp_prepare_cpus (unsigned int max_cpus) {unsigned cpu; +unsigned int i;xen_init_lock_cpu (0 );smp_store_cpu_info (0 );cpu_data (0 ).x86_max_cores =1 ; + +for_each_possible_cpu (i) { +zalloc_cpumask_var (&per_cpu (cpu_sibling_map, i),GFP_KERNEL ); +zalloc_cpumask_var (&per_cpu (cpu_core_map, i),GFP_KERNEL ); +zalloc_cpumask_var (&cpu_data (i).llc_shared_map,GFP_KERNEL ); + }set_cpu_sibling_map (0 );if (xen_smp_intr_init (0 ))
Signature · 1 line
1.6.5.2
_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xensource.com/xen-devel
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Andrew Jones <[email protected]> 18.12.09 10:31 >>>
The MAXSMP config option requires
CPUMASK_OFFSTACK, which in turn requires we init the memory for the maps while we bringing up the cpus. MAXSMP also increasesNR_CPUSto 4096. This increase in size exposed an issue in the argument construction for mulitcalls fromxen_flush_tlb_others. The args should only need space for the actual number of cpus, which with xen is currently only up to 32.
I don't think new code should be making assumptions like this anymore, since Xen already supports higher numbers (it's merely the tools which so far don't). You're basically trading a compile time detectable large value on stack for one that can grow large dynamically (and hence require quite a bit more effort to debug, should it ever overrun the stack).
Jan
Quoted 8 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Andrew Jones<[email protected]> 18.12.09 10:31>>>
The MAXSMP config option requires
CPUMASK_OFFSTACK, which in turn requires we init the memory for the maps while we bringing up the cpus. MAXSMP also increasesNR_CPUSto 4096. This increase in size exposed an issue in the argument construction for mulitcalls fromxen_flush_tlb_others. The args should only need space for the actual number of cpus, which with xen is currently only up to 32.
Quoted 6 lines
I don't think new code should be making assumptions like this anymore, since Xen already supports higher numbers (it's merely the tools which so far don't). You're basically trading a compile time detectable large value on stack for one that can grow large dynamically (and hence require quite a bit more effort to debug, should it ever overrun the stack).
I say 32 cpus in my description to point out the large difference between NR_CPUS and the actual number used. However, the code shouldn't exceed the limits in multicall until something around 2000 cpus.
Keeping it compile-time is good for the stack analysis, but overly wasteful when using values like 4096, since the expected case is thousands less. If we want to keep it static then we need to change MC_ARGS to also be dependent in some way on NR_CPUS.
Andrew
Quoted 16 lines · Andrew Jones
Quoted 8 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Andrew Jones<[email protected]> 18.12.09 10:31>>>
The MAXSMP config option requires
CPUMASK_OFFSTACK, which in turn requires we init the memory for the maps while we bringing up the cpus. MAXSMP also increasesNR_CPUSto 4096. This increase in size exposed an issue in the argument construction for mulitcalls fromxen_flush_tlb_others. The args should only need space for the actual number of cpus, which with xen is currently only up to 32.Quoted 6 lines
I don't think new code should be making assumptions like this anymore, since Xen already supports higher numbers (it's merely the tools which so far don't). You're basically trading a compile time detectable large value on stack for one that can grow large dynamically (and hence require quite a bit more effort to debug, should it ever overrun the stack).
Quoted 9 lines
I say 32 cpus in my description to point out the large difference between
NR_CPUSand the actual number used. However, the code shouldn't exceed the limits in multicall until something around 2000 cpus.Keeping it compile-time is good for the stack analysis, but overly wasteful when using values like 4096, since the expected case is thousands less. If we want to keep it static then we need to change
MC_ARGSto also be dependent in some way onNR_CPUS.
Another note here is that the amount of stack allocation is the same regardless of the value of num_processors. We're just creating a pointer to this structure on the stack. sizeof is smart enough to pass the appropriate dynamic size on to the mc call for validation though. So I think all should be good with this patch.
Andrew
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Andrew Jones <[email protected]> 18.12.09 14:23 >>>
Another note here is that the amount of stack allocation is the same regardless of the value of
num_processors. We're just creating a pointer to this structure on the stack. sizeof is smart enough to pass the appropriate dynamic size on to the mc call for validation though. So I think all should be good with this patch.
Ah, okay. At least it'll be a clear BUG_ON() if an overflow happens. And that's probably what you got before your patch?
Jan
Quoted 10 lines · Jan Beulich
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Andrew Jones<[email protected]> 18.12.09 14:23>>>
Another note here is that the amount of stack allocation is the same regardless of the value of
num_processors. We're just creating a pointer to this structure on the stack. sizeof is smart enough to pass the appropriate dynamic size on to the mc call for validation though. So I think all should be good with this patch.Ah, okay. At least it'll be a clear
BUG_ON()if an overflow happens. And that's probably what you got before your patch?
That's right; "kernel BUG at arch/x86/xen/multicalls.c:209!"
Andrew
Quoted 27 lines · Andrew Jones
Quoted 16 lines · Andrew Jones
Quoted 8 lines · Jan Beulich
Quoted 7 lines
Quoted 1 line
Quoted 1 line
Andrew Jones<[email protected]> 18.12.09 10:31>>>
The MAXSMP config option requires
CPUMASK_OFFSTACK, which in turn requires we init the memory for the maps while we bringing up the cpus. MAXSMP also increasesNR_CPUSto 4096. This increase in size exposed an issue in the argument construction for mulitcalls fromxen_flush_tlb_others. The args should only need space for the actual number of cpus, which with xen is currently only up to 32.Quoted 6 lines
I don't think new code should be making assumptions like this anymore, since Xen already supports higher numbers (it's merely the tools which so far don't). You're basically trading a compile time detectable large value on stack for one that can grow large dynamically (and hence require quite a bit more effort to debug, should it ever overrun the stack).
Quoted 9 lines
I say 32 cpus in my description to point out the large difference between
NR_CPUSand the actual number used. However, the code shouldn't exceed the limits in multicall until something around 2000 cpus.Keeping it compile-time is good for the stack analysis, but overly wasteful when using values like 4096, since the expected case is thousands less. If we want to keep it static then we need to change
MC_ARGSto also be dependent in some way onNR_CPUS.
Quoted 5 lines
Another note here is that the amount of stack allocation is the same regardless of the value of
num_processors. We're just creating a pointer to this structure on the stack. sizeof is smart enough to pass the appropriate dynamic size on to the mc call for validation though. So I think all should be good with this patch.
Andrew,
Are you carrying this patch in Fedora? Is there a newer version of this patch that you have?
----- Original Message -----
Quoted 42 lines · Andrew Jones
Quoted 35 lines · Andrew Jones
Quoted 28 lines · Jan Beulich
Quoted 11 lines
Quoted 10 lines
Quoted 1 line
Quoted 1 line
Andrew Jones<[email protected]> 18.12.09 10:31>>>
The MAXSMP config option requires
CPUMASK_OFFSTACK, which in turn requires we init the memory for the maps while we bringing up the cpus.
MAXSMP also increasesNR_CPUSto 4096. This increase in size exposed an
issue in the argument construction for mulitcalls fromxen_flush_tlb_others. The args should only need space for the actual
number of cpus, which with xen is currently only up to 32.Quoted 10 lines
I don't think new code should be making assumptions like this anymore,
since Xen already supports higher numbers (it's merely the tools which
so far don't). You're basically trading a compile time detectable large
value on stack for one that can grow large dynamically (and hence require quite a bit more effort to debug, should it ever overrun the
stack).I say 32 cpus in my description to point out the large difference between
NR_CPUSand the actual number used. However, the code shouldn't
exceed the limits in multicall until something around 2000 cpus.Quoted 5 lines
Keeping it compile-time is good for the stack analysis, but overly wasteful when using values like 4096, since the expected case is thousands less. If we want to keep it static then we need to change
MC_ARGSto also be dependent in some way onNR_CPUS.Quoted 5 lines
Another note here is that the amount of stack allocation is the same regardless of the value of
num_processors. We're just creating a pointer to this structure on the stack. sizeof is smart enough to pass the appropriate dynamic size on to the mc call for validation though. So I think all should be good with this patch.
Quoted 4 lines
Andrew,
Are you carrying this patch in Fedora? Is there a newer version of this patch that you have?
Not in Fedora, it's only in RHEL. From a quick look I don't believe the patch would need to be changed for the current kernel, but I haven't tried it.
Drew
Quoted 8 lines
Quoted 4 lines
Andrew,
Are you carrying this patch in Fedora? Is there a newer version of this patch that you have?
Not in Fedora, it's only in RHEL. From a quick look I don't believe the patch would need to be changed for the current kernel, but I haven't tried it.
OK, I tried it and had to modify it a bit but so far it seems to work fine. Do you have some other patches that never go upstreamed?
----- Original Message -----
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Andrew,
Are you carrying this patch in Fedora? Is there a newer version of this patch that you have?
Quoted 8 lines
Quoted 5 lines
Not in Fedora, it's only in RHEL. From a quick look I don't believe the
patch would need to be changed for the current kernel, but I haven't tried
it.OK, I tried it and had to modify it a bit but so far it seems to work fine. Do you have some other patches that never go upstreamed?
Not that I know of. We really haven't needed to write too many xen-related patches for the RHEL6 kernel. Thanks for hunting this maxsmp one down.
Drew