: Discard reserved PXM bits for SRAT v1
Subsystems (4)
cover : Discard reserved PXM bits for SRAT v1
Hi,
ACPI specification says that the OS must disregard reserved bits. The x86_64 SRAT parser does not discard the upper 24 bits of the proximity_domain (pxm) in the acpi_srat_mem_affinity entries for SRAT v1 tables. (v2 has 32 bits wide fields.)
This can lead to problems with poor BIOS implementations that failed to set resreved bytes to zero. (The ACPI spec is a bit vague here unfortunately.)
This was also inconsistent: On x86-64 (srat_64.c), the _cpu_affinity does only use the low 8 bits of pxm, while the full 32 bits of _mem_affinity are consumed.
In srat_32.c (x86), only 8bits are used (which is OK, a 32bit system with >256 PXMs does not seem reasonable at all).
On ia64, the support of more than 8 bits was consistent between mem and cpu affinity entries, however it dependent on "sn2" platform.
The patch series has the following goals:
* Make the kernel support consistently 8bits or 32bits for the proximity domain
* Make this dependent on the SRAT version; v1 => 8bits, v2 => 32bits.
Overview over the patches:
- [1/3] Store the SRAT table version value in acpi_srat_revision
- [2/3] x86-64: Discard the upper 24 bits in mem_affinity if rev <= 1 and use upper 24bits in cpu_affinity if rev >= 2
- [3/3] ia64: Also use upper 8/24bits if rev >= 2 (but leave logic to enable on sn2 as well -- I don't know if sn2 reports v1 or v2 SRAT) Also add two __init decls in ia64 pxm accessors.
Patch has been tested on x86-64 against an 2.6.27.x kernel. (Patch is against current git.)
Thanks for James, Greg, Alexey, Norbert for comments, review and testing.
Please review and apply!
Greg, I believe this is a candidate for -stable.
Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.
PATCH 1/3 : Store SRAT revision
Hi,
This patch stores the SRAT table revision for later consumption by arch specific __init functions.
This is patch 1/3.
Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.
From: Kurt Garloff <[email protected]>
Subject: Store SRAT table revision
References: bnc#503038
In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregrard reserved fields. In order to know whether or not, we must know what version the SRAT table has.
This patch stores the SRAT table revision for later consumption by arch specific __init functions.
This is patch 1/3.
drivers/acpi/numa.c +3 -0
@@ -42,6 +42,8 @@static int node_to_pxm_map[MAX_NUMNODES ] = { [0 ...MAX_NUMNODES -1 ] =PXM_INVAL }; +unsigned char acpi_srat_revision__initdata ; +int pxm_to_node (int pxm) {if (pxm <0 )
@@ -266,6 +268,7 @@return -EINVAL ; srat = (struct acpi_table_srat *)table; + acpi_srat_revision = srat->header.revision;return 0 ; }
include/acpi/acpi_numa.h +1 -0
@@ -15,6 +15,7 @@extern int node_to_pxm (int );extern void __acpi_map_pxm_to_node (int ,int );extern int acpi_map_pxm_to_node (int ); +extern unsigned char acpi_srat_revision;#endif /* CONFIG_ACPI_NUMA */ #endif /* __ACP_NUMA_H */
Re: [PATCH 2/3]: x86-64: Handle SRAT v1 and v2 consistently
In reply to Kurt GarloffHi,
x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in cpu_affinity, but 32 bits in mem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).
This is patch 2/3.
Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.
From: Kurt Garloff <[email protected]>
Subject: Use SRAT table rev to use 8bit or 32bit PXM fields (x86-64)
References: bnc#503038
In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregrard reserved fields.
x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in cpu_affinity, but 32 bits in mem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).
This is patch 2/3.
arch/x86/mm/srat_64.c +4 -0
@@ -164,6 +164,8 @@if ((pa->flags &ACPI_SRAT_CPU_ENABLED ) ==0 )return ; pxm = pa->proximity_domain_lo; +if (acpi_srat_revision >=2 ) + pxm |= *((unsigned int *)pa->proximity_domain_hi) <<8 ; node =setup_node (pxm);if (node <0 ) {printk (KERN_ERR "SRAT: Too many proximity domains %x\n" , pxm);
@@ -275,6 +277,8 @@ start = ma->base_address; end = start + ma->length; pxm = ma->proximity_domain; +if (acpi_srat_revision <=1 ) + pxm &=0xff ; node =setup_node (pxm);if (node <0 ) {printk (KERN_ERR "SRAT: Too many proximity domains.\n" );
Yinghai Lu
Acked-byRe: [PATCH 2/3]: x86-64: Handle SRAT v1 and v2 consistently
Quoted 11 lines · Kurt Garloff
Hi,
x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in
cpu_affinity, but 32 bits inmem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).This is patch 2/3.
--
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.
From: Kurt Garloff <[email protected]>
Subject: Use SRAT table rev to use 8bit or 32bit PXM fields (x86-64)
References: bnc#503038
In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregrard reserved fields.
x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in cpu_affinity, but 32 bits in mem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).
This is patch 2/3.
--- arch/x86/mm/srat_64.c | 4 ++++ 1 file changed, 4 insertions(+)
arch/x86/mm/srat_64.c +4 -0
@@ -154,6 +154,8 @@ acpi_numa_processor_affinity_init(structif ((pa->flags &ACPI_SRAT_CPU_ENABLED ) ==0 )return ; pxm = pa->proximity_domain_lo; +if (acpi_srat_revision >=2 ) + pxm |= *((unsigned int *)pa->proximity_domain_hi) <<8 ; node =setup_node (pxm);if (node <0 ) {printk (KERN_ERR "SRAT: Too many proximity domains %x\n" , pxm);
@@ -255,6 +257,8 @@ acpi_numa_memory_affinity_init(struct ac start = ma->base_address; end = start + ma->length; pxm = ma->proximity_domain; +if (acpi_srat_revision <=1 ) + pxm &=0xff ; node =setup_node (pxm);if (node <0 ) {printk (KERN_ERR "SRAT: Too many proximity domains.\n" );
PATCH 3/3 : Consider SRAT rev on ia64
In reply to Kurt GarloffHi,
ia64 did handle the PXM fields almost consistently, but depending on sgi's sn2 platform. This patch leaves the sn2 logic in, but does also use 16/32 bits for PXM if the SRAT has rev 2 or higher.
This is patch 3/3.
Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.
From: Kurt Garloff <[email protected]>
Subject: Use SRAT table rev to use 8bit or 16/32bit PXM fields (ia64)
References: bnc#503038
In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregrard reserved fields.
ia64 did handle the PXM fields almost consistently, but depending on sgi's sn2 platform. This patch leaves the sn2 logic in, but does also use 16/32 bits for PXM if the SRAT has rev 2 or higher.
The patch also adds __init to the two pxm accessor functions, as they access __initdata now and are called from an __init function only anyway.
Note that the code only uses 16 bits for the PXM field in the processor proximity field; the patch does not address this as 16 bits are more than enough.
This is patch 3/3.
arch/ia64/kernel/acpi.c +6 -4
@@ -456,22 +456,24 @@static struct acpi_table_slit __initdata *slit_table;cpumask_t early_cpu_possible_map =CPU_MASK_NONE ; -static int get_processor_proximity_domain (struct acpi_srat_cpu_affinity *pa) +static int __init +get_processor_proximity_domain (struct acpi_srat_cpu_affinity *pa) {int pxm; pxm = pa->proximity_domain_lo; -if (ia64_platform_is ("sn2" )) +if (ia64_platform_is ("sn2" ) || acpi_srat_revision >=2 ) pxm += pa->proximity_domain_hi[0 ] <<8 ;return pxm; } -static int get_memory_proximity_domain (struct acpi_srat_mem_affinity *ma) +static int __init +get_memory_proximity_domain (struct acpi_srat_mem_affinity *ma) {int pxm; pxm = ma->proximity_domain; -if (!ia64_platform_is ("sn2" )) +if (!ia64_platform_is ("sn2" ) && acpi_srat_revision <=1 ) pxm &=0xff ;return pxm;
Quoted 31 lines · Kurt Garloff
Hi,
ACPI specification says that the OS must disregard reserved bits. The
x86_64SRAT parser does not discard the upper 24 bits of theproximity_domain(pxm) in theacpi_srat_mem_affinityentries for SRAT v1 tables. (v2 has 32 bits wide fields.)
This can lead to problems with poor BIOS implementations that failed to set resreved bytes to zero. (The ACPI spec is a bit vague here unfortunately.)This was also inconsistent: On x86-64 (
srat_64.c), the_cpu_affinitydoes only use the low 8 bits of pxm, while the full 32 bits of_mem_affinityare consumed.
Insrat_32.c(x86), only 8bits are used (which is OK, a 32bit system with >256 PXMs does not seem reasonable at all).
On ia64, the support of more than 8 bits was consistent between mem and cpu affinity entries, however it dependent on "sn2" platform.The patch series has the following goals:
* Make the kernel support consistently 8bits or 32bits for the proximity domain
* Make this dependent on the SRAT version; v1 => 8bits, v2 => 32bits.Overview over the patches:
- [1/3] Store the SRAT table version value inacpi_srat_revision
- [2/3] x86-64: Discard the upper 24 bits inmem_affinityif rev <= 1 and use upper 24bits incpu_affinityif rev >= 2
- [3/3] ia64: Also use upper 8/24bits if rev >= 2 (but leave logic to enable on sn2 as well -- I don't know if sn2 reports v1 or v2 SRAT) Also add two__initdecls in ia64 pxm accessors.
Quoted 8 lines
Patch has been tested on x86-64 against an 2.6.27.x kernel. (Patch is against current git.)
Thanks for James, Greg, Alexey, Norbert for comments, review and testing.
Please review and apply!
Greg, I believe this is a candidate for -stable.
Does it meet the rules at described in Documentation/stable_kernel_rules.txt? If so, add a
Cc: stable <[email protected]>
to the signed-off-by: area in the patch, and when it goes into Linus's tree, it will be automatically queued up to be added to the next -stable release.
thanks,
greg k-h
Hi Kurt,
We are hitting this problem on some Nehalem based platforms, that prevents a correct Numa initialization. We propsed a patch on linux_acpi to fix it.
We are OK to have your patch go into the mainstream to have this fixed quickly.
Thanks.
Xavier
Quoted 39 lines · Kurt Garloff
Hi,
ACPI specification says that the OS must disregard reserved bits. The
x86_64SRAT parser does not discard the upper 24 bits of theproximity_domain(pxm) in theacpi_srat_mem_affinityentries for SRAT v1 tables. (v2 has 32 bits wide fields.)
This can lead to problems with poor BIOS implementations that failed to set resreved bytes to zero. (The ACPI spec is a bit vague here unfortunately.)This was also inconsistent: On x86-64 (
srat_64.c), the_cpu_affinitydoes only use the low 8 bits of pxm, while the full 32 bits of_mem_affinityare consumed.
Insrat_32.c(x86), only 8bits are used (which is OK, a 32bit system
with >256 PXMs does not seem reasonable at all).
On ia64, the support of more than 8 bits was consistent between
mem and cpu affinity entries, however it dependent on "sn2" platform.The patch series has the following goals:
* Make the kernel support consistently 8bits or 32bits for the proximity domain
* Make this dependent on the SRAT version; v1 => 8bits, v2 => 32bits.Overview over the patches:
- [1/3] Store the SRAT table version value inacpi_srat_revision
- [2/3] x86-64: Discard the upper 24 bits inmem_affinityif rev <= 1 and use upper 24bits incpu_affinityif rev >= 2
- [3/3] ia64: Also use upper 8/24bits if rev >= 2 (but leave logic to enable on sn2 as well -- I don't know if sn2 reports v1 or v2 SRAT) Also add two__initdecls in ia64 pxm accessors.Patch has been tested on x86-64 against an 2.6.27.x kernel. (Patch is against current git.)
Thanks for James, Greg, Alexey, Norbert for comments, review and testing.
Please review and apply!
Greg, I believe this is a candidate for -stable.
I just wanted to post in to the thread that this patch set solved both the node numbering (where there was a node0 and a node2) and the issue where all CPUs were showing up one the first node.
I applied the three patches against 2.6.30-git14.
We look forward to seeing the changes integrated.
On this 2-socket Intel 5500 based system (Nehalem), the /sys stuff looks good:
$ ls /sys/devices/system/node/node? /sys/devices/system/node/node0: cpu0 cpu2 cpulist distance numastat cpu1 cpu3 cpumap meminfo scan_unevictable_pages
/sys/devices/system/node/node1:
cpu4 cpu6 cpulist distance numastat cpu5 cpu7 cpumap meminfo scan_unevictable_pages
Where as, before, it looked like this (non-ordered, all CPUs showing up in node 0).
# ls -ld /sys/devices/system/node/node* drwxr-xr-x 2 root root 0 2009-06-18 09:30 /sys/devices/system/node/node0 drwxr-xr-x 2 root root 0 2009-06-18 09:30 /sys/devices/system/node/node2
[root@cct201 ~]# ls /sys/devices/system/node/node0
cpu0 cpu2 cpu4 cpu6 cpulist distance numastat cpu1 cpu3 cpu5 cpu7 cpumap meminfo scan_unevictable_pages
System details:
- SGI Altix XE 270
- Supermicro X8DTN v 1.1 mainboard
- 2 sockets of Xeon X5570
/proc/cpuinfo snip
[root@cct201 erikj]# cat /proc/cpuinfo
processor : 0vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yesfpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5866.85
clflush size : 64cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 1vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
apicid : 2
initial apicid : 2
fpu : yesfpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 8212.05
clflush size : 64cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 2vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 4
apicid : 4
initial apicid : 4
fpu : yesfpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5865.76
clflush size : 64cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 3vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
apicid : 6
initial apicid : 6
fpu : yesfpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5865.76
clflush size : 64cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 4vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 0
cpu cores : 4
apicid : 16
initial apicid : 16
fpu : yesfpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5828.64
clflush size : 64cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 5vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 1
cpu cores : 4
apicid : 18
initial apicid : 18
fpu : yesfpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5865.81
clflush size : 64cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 6vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 2
cpu cores : 4
apicid : 20
initial apicid : 20
fpu : yesfpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5865.80
clflush size : 64cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
processor : 7vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 3
cpu cores : 4
apicid : 22
initial apicid : 22
fpu : yesfpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5865.80
clflush size : 64cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
cover Resend: Discard reserved PXM bits for SRAT v1
Erik,
Quoted 7 lines · Erik Jacobson
I just wanted to post in to the thread that this patch set solved both the node numbering (where there was a node0 and a node2) and the issue where all CPUs were showing up one the first node.
I applied the three patches against 2.6.30-git14.
We look forward to seeing the changes integrated.
So do I.
The patches are obviously already in SUSE kernels, I'd like to see them in mainline too.
Quoted 21 lines
On this 2-socket Intel 5500 based system (Nehalem), the /sys stuff looks good:
$ ls /sys/devices/system/node/node? /sys/devices/system/node/node0: cpu0 cpu2 cpulist distance numastat cpu1 cpu3 cpumap meminfo scan_unevictable_pages
/sys/devices/system/node/node1:cpu4 cpu6 cpulist distance numastat cpu5 cpu7 cpumap meminfo scan_unevictable_pagesWhere as, before, it looked like this (non-ordered, all CPUs showing up in node 0).
# ls -ld /sys/devices/system/node/node* drwxr-xr-x 2 root root 0 2009-06-18 09:30 /sys/devices/system/node/node0 drwxr-xr-x 2 root root 0 2009-06-18 09:30 /sys/devices/system/node/node2[root@cct201 ~]# ls
/sys/devices/system/node/node0
Quoted 8 lines
cpu0 cpu2 cpu4 cpu6 cpulist distance numastat cpu1 cpu3 cpu5 cpu7 cpumap meminfo scan_unevictable_pagesSystem details:
- SGI Altix XE 270
- Supermicro X8DTN v 1.1 mainboard
- 2 sockets of Xeon X5570
OK, you really observe the same here as did FZJ.
Thanks for reporting!
So let me resend the patches, rediffed against current 2.6.31rc git in followup mails.
Len, Linus, please adopt.
Best,
Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.
PATCH 1/3 Resend: [PATCH 1/3]: Store SRAT revision
Hi,
In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields. In order to know whether to disregard or not, we must know what version the SRAT table has.
This patch stores the SRAT table revision for later consumption by arch specific __init functions.
This is patch 1/3.
Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.
From: Kurt Garloff <[email protected]>
Subject: Store SRAT table revision
In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields. In order to know whether to disregard or not, we must know what version the SRAT table has.
This patch stores the SRAT table revision for later consumption by arch specific __init functions.
This is patch 1/3.
drivers/acpi/numa.c +3 -0
@@ -42,6 +42,8 @@static int node_to_pxm_map[MAX_NUMNODES ] = { [0 ...MAX_NUMNODES -1 ] =PXM_INVAL }; +unsigned char acpi_srat_revision__initdata ; +int pxm_to_node (int pxm) {if (pxm <0 )
@@ -266,6 +268,7 @@return -EINVAL ; srat = (struct acpi_table_srat *)table; + acpi_srat_revision = srat->header.revision;return 0 ; }
include/acpi/acpi_numa.h +1 -0
@@ -15,6 +15,7 @@extern int node_to_pxm (int );extern void __acpi_map_pxm_to_node (int ,int );extern int acpi_map_pxm_to_node (int ); +extern unsigned char acpi_srat_revision;#endif /* CONFIG_ACPI_NUMA */ #endif /* __ACP_NUMA_H */
PATCH 2/3 Resend: x86-64: Handle SRAT v1 and v2 consistently
In reply to Kurt GarloffHi,
In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields.
x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in cpu_affinity, but 32 bits in mem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).
This is patch 2/3.
Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.
From: Kurt Garloff <[email protected]>
Subject: Use SRAT table rev to use 8bit or 32bit PXM fields (x86-64)
References: bnc#503038
In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields.
x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in cpu_affinity, but 32 bits in mem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).
This is patch 2/3.
arch/x86/mm/srat_64.c +4 -0
@@ -154,6 +154,8 @@if ((pa->flags &ACPI_SRAT_CPU_ENABLED ) ==0 )return ; pxm = pa->proximity_domain_lo; +if (acpi_srat_revision >=2 ) + pxm |= *((unsigned int *)pa->proximity_domain_hi) <<8 ; node =setup_node (pxm);if (node <0 ) {printk (KERN_ERR "SRAT: Too many proximity domains %x\n" , pxm);
@@ -255,6 +257,8 @@ start = ma->base_address; end = start + ma->length; pxm = ma->proximity_domain; +if (acpi_srat_revision <=1 ) + pxm &=0xff ; node =setup_node (pxm);if (node <0 ) {printk (KERN_ERR "SRAT: Too many proximity domains.\n" );
Re: [PATCH 2/3] Resend: x86-64: Handle SRAT v1 and v2 consistently
Quoted 11 lines · Kurt Garloff
Hi,
In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields.
x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in
cpu_affinity, but 32 bits inmem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).
Quoted 4 lines
This is patch 2/3.
Signed-off-by Kurt Garloff <[email protected]>--
for 1, 2
YH
PATCH 3/3 Resend: Consider SRAT rev on ia64
In reply to Kurt GarloffHi,
In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields.
ia64 did handle the PXM fields almost consistently, but depending on sgi's sn2 platform. This patch leaves the sn2 logic in, but does also use 16/32 bits for PXM if the SRAT has rev 2 or higher.
The patch also adds __init to the two pxm accessor functions, as they access __initdata now and are called from an __init function only anyway.
Note that the code only uses 16 bits for the PXM field in the processor proximity field; the patch does not address this as 16 bits are more than enough.
This is patch 3/3.
Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.
From: Kurt Garloff <[email protected]>
Subject: Use SRAT table rev to use 8bit or 16/32bit PXM fields (ia64)
References: bnc#503038
In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields.
ia64 did handle the PXM fields almost consistently, but depending on sgi's sn2 platform. This patch leaves the sn2 logic in, but does also use 16/32 bits for PXM if the SRAT has rev 2 or higher.
The patch also adds __init to the two pxm accessor functions, as they access __initdata now and are called from an __init function only anyway.
Note that the code only uses 16 bits for the PXM field in the processor proximity field; the patch does not address this as 16 bits are more than enough.
This is patch 3/3.
arch/ia64/kernel/acpi.c +6 -4
@@ -456,22 +456,24 @@static struct acpi_table_slit __initdata *slit_table;cpumask_t early_cpu_possible_map =CPU_MASK_NONE ; -static int get_processor_proximity_domain (struct acpi_srat_cpu_affinity *pa) +static int __init +get_processor_proximity_domain (struct acpi_srat_cpu_affinity *pa) {int pxm; pxm = pa->proximity_domain_lo; -if (ia64_platform_is ("sn2" )) +if (ia64_platform_is ("sn2" ) || acpi_srat_revision >=2 ) pxm += pa->proximity_domain_hi[0 ] <<8 ;return pxm; } -static int get_memory_proximity_domain (struct acpi_srat_mem_affinity *ma) +static int __init +get_memory_proximity_domain (struct acpi_srat_mem_affinity *ma) {int pxm; pxm = ma->proximity_domain; -if (!ia64_platform_is ("sn2" )) +if (!ia64_platform_is ("sn2" ) && acpi_srat_revision <=1 ) pxm &=0xff ;return pxm;
Quoted 23 lines · On Mon, 18 May 2009 11:33:52 +0200 Kurt Garloff <[email protected]> wrote:
Hi,
ACPI specification says that the OS must disregard reserved bits. The
x86_64SRAT parser does not discard the upper 24 bits of theproximity_domain(pxm) in theacpi_srat_mem_affinityentries for SRAT v1 tables. (v2 has 32 bits wide fields.)
This can lead to problems with poor BIOS implementations that failed to set resreved bytes to zero. (The ACPI spec is a bit vague here unfortunately.)This was also inconsistent: On x86-64 (
srat_64.c), the_cpu_affinitydoes only use the low 8 bits of pxm, while the full 32 bits of_mem_affinityare consumed.
Insrat_32.c(x86), only 8bits are used (which is OK, a 32bit system with >256 PXMs does not seem reasonable at all).
On ia64, the support of more than 8 bits was consistent between mem and cpu affinity entries, however it dependent on "sn2" platform.The patch series has the following goals:
* Make the kernel support consistently 8bits or 32bits for the proximity domain
* Make this dependent on the SRAT version; v1 => 8bits, v2 => 32bits.
Quoted 8 lines
Overview over the patches:
- [1/3] Store the SRAT table version value inacpi_srat_revision
- [2/3] x86-64: Discard the upper 24 bits inmem_affinityif rev <= 1 and use upper 24bits incpu_affinityif rev >= 2
- [3/3] ia64: Also use upper 8/24bits if rev >= 2 (but leave logic to enable on sn2 as well -- I don't know if sn2 reports v1 or v2 SRAT) Also add two__initdecls in ia64 pxm accessors.
Did this patchset get lost or something? I don't see it anywhere, not even in the ACPI 'testing' tree...