Skip to contentSkip to conversation

: Discard reserved PXM bits for SRAT v1

PATCH In review Kurt Garloff 6 participants 14 messages May 18, 2009 – Aug 13, 2009

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover : Discard reserved PXM bits for SRAT v1

Hi,

ACPI specification says that the OS must disregard reserved bits. The x86_64 SRAT parser does not discard the upper 24 bits of the proximity_domain (pxm) in the acpi_srat_mem_affinity entries for SRAT v1 tables. (v2 has 32 bits wide fields.)
This can lead to problems with poor BIOS implementations that failed to set resreved bytes to zero. (The ACPI spec is a bit vague here unfortunately.)

This was also inconsistent: On x86-64 (srat_64.c), the _cpu_affinity does only use the low 8 bits of pxm, while the full 32 bits of _mem_affinity are consumed.
In srat_32.c (x86), only 8bits are used (which is OK, a 32bit system with >256 PXMs does not seem reasonable at all).
On ia64, the support of more than 8 bits was consistent between mem and cpu affinity entries, however it dependent on "sn2" platform.

The patch series has the following goals:
* Make the kernel support consistently 8bits or 32bits for the proximity domain
* Make this dependent on the SRAT version; v1 => 8bits, v2 => 32bits.

Overview over the patches:
- [1/3] Store the SRAT table version value in acpi_srat_revision
- [2/3] x86-64: Discard the upper 24 bits in mem_affinity if rev <= 1 and use upper 24bits in cpu_affinity if rev >= 2
- [3/3] ia64: Also use upper 8/24bits if rev >= 2 (but leave logic to enable on sn2 as well -- I don't know if sn2 reports v1 or v2 SRAT) Also add two __init decls in ia64 pxm accessors.

Patch has been tested on x86-64 against an 2.6.27.x kernel. (Patch is against current git.)

Thanks for James, Greg, Alexey, Norbert for comments, review and testing.

Please review and apply!

Greg, I believe this is a candidate for -stable.

Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.
#1 lore raw

PATCH 1/3 : Store SRAT revision

Hi,

This patch stores the SRAT table revision for later consumption by arch specific __init functions.

This is patch 1/3.

Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

From: Kurt Garloff <[email protected]>
Subject: Store SRAT table revision
References: bnc#503038

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregrard reserved fields. In order to know whether or not, we must know what version the SRAT table has.

This patch stores the SRAT table revision for later consumption by arch specific __init functions.

This is patch 1/3.

drivers/acpi/numa.c +3 -0
diff -r a4ed8abcc815 drivers/acpi/numa.c
@@ -42,6 +42,8 @@
 static int node_to_pxm_map[MAX_NUMNODES]
 				= { [0 ... MAX_NUMNODES - 1] = PXM_INVAL };
 
+unsigned char acpi_srat_revision __initdata;
+
 int pxm_to_node(int pxm)
 {
 	if (pxm < 0)
@@ -266,6 +268,7 @@
 		return -EINVAL;
 
 	srat = (struct acpi_table_srat *)table;
+	acpi_srat_revision = srat->header.revision;
 
 	return 0;
 }
include/acpi/acpi_numa.h +1 -0
diff -r a4ed8abcc815 include/acpi/acpi_numa.h
@@ -15,6 +15,7 @@
 extern int node_to_pxm(int);
 extern void __acpi_map_pxm_to_node(int, int);
 extern int acpi_map_pxm_to_node(int);
+extern unsigned char acpi_srat_revision;
 
 #endif				/* CONFIG_ACPI_NUMA */
 #endif				/* __ACP_NUMA_H */
#2 lore raw

Re: [PATCH 2/3]: x86-64: Handle SRAT v1 and v2 consistently

In reply to Kurt Garloff

Hi,

x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in cpu_affinity, but 32 bits in mem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).

This is patch 2/3.

Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

From: Kurt Garloff <[email protected]>
Subject: Use SRAT table rev to use 8bit or 32bit PXM fields (x86-64)
References: bnc#503038

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregrard reserved fields.

x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in cpu_affinity, but 32 bits in mem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).

This is patch 2/3.

arch/x86/mm/srat_64.c +4 -0
diff -r a4ed8abcc815 arch/x86/mm/srat_64.c
@@ -164,6 +164,8 @@
 	if ((pa->flags & ACPI_SRAT_CPU_ENABLED) == 0)
 		return;
 	pxm = pa->proximity_domain_lo;
+	if (acpi_srat_revision >= 2)
+		pxm |= *((unsigned int*)pa->proximity_domain_hi) << 8;
 	node = setup_node(pxm);
 	if (node < 0) {
 		printk(KERN_ERR "SRAT: Too many proximity domains %x\n", pxm);
@@ -275,6 +277,8 @@
 	start = ma->base_address;
 	end = start + ma->length;
 	pxm = ma->proximity_domain;
+	if (acpi_srat_revision <= 1)
+		pxm &= 0xff;
 	node = setup_node(pxm);
 	if (node < 0) {
 		printk(KERN_ERR "SRAT: Too many proximity domains.\n");
Acked-by
#3 lore raw

Re: [PATCH 2/3]: x86-64: Handle SRAT v1 and v2 consistently

Quoted 11 lines · Kurt Garloff

Hi,

x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in cpu_affinity, but 32 bits in mem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).

This is patch 2/3.
--
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

From: Kurt Garloff <[email protected]>
Subject: Use SRAT table rev to use 8bit or 32bit PXM fields (x86-64)
References: bnc#503038

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregrard reserved fields.

x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in cpu_affinity, but 32 bits in mem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).

This is patch 2/3.

---
 arch/x86/mm/srat_64.c |    4 ++++
 1 file changed, 4 insertions(+)
arch/x86/mm/srat_64.c +4 -0
Index: linux-2.6/arch/x86/mm/srat_64.c
===================================================================
@@ -154,6 +154,8 @@ acpi_numa_processor_affinity_init(struct
 	if ((pa->flags & ACPI_SRAT_CPU_ENABLED) == 0)
 		return;
 	pxm = pa->proximity_domain_lo;
+	if (acpi_srat_revision >= 2)
+		pxm |= *((unsigned int*)pa->proximity_domain_hi) << 8;
 	node = setup_node(pxm);
 	if (node < 0) {
 		printk(KERN_ERR "SRAT: Too many proximity domains %x\n", pxm);
@@ -255,6 +257,8 @@ acpi_numa_memory_affinity_init(struct ac
 	start = ma->base_address;
 	end = start + ma->length;
 	pxm = ma->proximity_domain;
+	if (acpi_srat_revision <= 1)
+		pxm &= 0xff;
 	node = setup_node(pxm);
 	if (node < 0) {
 		printk(KERN_ERR "SRAT: Too many proximity domains.\n");
#4 lore raw

PATCH 3/3 : Consider SRAT rev on ia64

In reply to Kurt Garloff

Hi,

ia64 did handle the PXM fields almost consistently, but depending on sgi's sn2 platform. This patch leaves the sn2 logic in, but does also use 16/32 bits for PXM if the SRAT has rev 2 or higher.

This is patch 3/3.

Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

From: Kurt Garloff <[email protected]>
Subject: Use SRAT table rev to use 8bit or 16/32bit PXM fields (ia64)
References: bnc#503038

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregrard reserved fields.

ia64 did handle the PXM fields almost consistently, but depending on sgi's sn2 platform. This patch leaves the sn2 logic in, but does also use 16/32 bits for PXM if the SRAT has rev 2 or higher.

The patch also adds __init to the two pxm accessor functions, as they access __initdata now and are called from an __init function only anyway.

Note that the code only uses 16 bits for the PXM field in the processor proximity field; the patch does not address this as 16 bits are more than enough.

This is patch 3/3.

arch/ia64/kernel/acpi.c +6 -4
diff -r a4ed8abcc815 arch/ia64/kernel/acpi.c
@@ -456,22 +456,24 @@
 static struct acpi_table_slit __initdata *slit_table;
 cpumask_t early_cpu_possible_map = CPU_MASK_NONE;
 
-static int get_processor_proximity_domain(struct acpi_srat_cpu_affinity *pa)
+static int __init
+get_processor_proximity_domain(struct acpi_srat_cpu_affinity *pa)
 {
 	int pxm;
 
 	pxm = pa->proximity_domain_lo;
-	if (ia64_platform_is("sn2"))
+	if (ia64_platform_is("sn2") || acpi_srat_revision >= 2)
 		pxm += pa->proximity_domain_hi[0] << 8;
 	return pxm;
 }
 
-static int get_memory_proximity_domain(struct acpi_srat_mem_affinity *ma)
+static int __init
+get_memory_proximity_domain(struct acpi_srat_mem_affinity *ma)
 {
 	int pxm;
 
 	pxm = ma->proximity_domain;
-	if (!ia64_platform_is("sn2"))
+	if (!ia64_platform_is("sn2") && acpi_srat_revision <= 1)
 		pxm &= 0xff;
 
 	return pxm;
#5 lore raw
In reply to Kurt Garloff
Quoted 31 lines · Kurt Garloff

Hi,

ACPI specification says that the OS must disregard reserved bits. The x86_64 SRAT parser does not discard the upper 24 bits of the proximity_domain (pxm) in the acpi_srat_mem_affinity entries for SRAT v1 tables. (v2 has 32 bits wide fields.)
This can lead to problems with poor BIOS implementations that failed to set resreved bytes to zero. (The ACPI spec is a bit vague here unfortunately.)

This was also inconsistent: On x86-64 (srat_64.c), the _cpu_affinity does only use the low 8 bits of pxm, while the full 32 bits of _mem_affinity are consumed.
In srat_32.c (x86), only 8bits are used (which is OK, a 32bit system with >256 PXMs does not seem reasonable at all).
On ia64, the support of more than 8 bits was consistent between mem and cpu affinity entries, however it dependent on "sn2" platform.

The patch series has the following goals:
* Make the kernel support consistently 8bits or 32bits for the proximity domain
* Make this dependent on the SRAT version; v1 => 8bits, v2 => 32bits.

Overview over the patches:
- [1/3] Store the SRAT table version value in acpi_srat_revision
- [2/3] x86-64: Discard the upper 24 bits in mem_affinity if rev <= 1 and use upper 24bits in cpu_affinity if rev >= 2
- [3/3] ia64: Also use upper 8/24bits if rev >= 2 (but leave logic to enable on sn2 as well -- I don't know if sn2 reports v1 or v2 SRAT) Also add two __init decls in ia64 pxm accessors.

Quoted 8 lines

Patch has been tested on x86-64 against an 2.6.27.x kernel. (Patch is against current git.)

Thanks for James, Greg, Alexey, Norbert for comments, review and testing.

Please review and apply!

Greg, I believe this is a candidate for -stable.

Does it meet the rules at described in Documentation/stable_kernel_rules.txt? If so, add a
Cc: stable <[email protected]>
to the signed-off-by: area in the patch, and when it goes into Linus's tree, it will be automatically queued up to be added to the next -stable release.

thanks,

greg k-h

#6 lore raw
In reply to Kurt Garloff

Hi Kurt,

We are hitting this problem on some Nehalem based platforms, that prevents a correct Numa initialization. We propsed a patch on linux_acpi to fix it.
We are OK to have your patch go into the mainstream to have this fixed quickly.
Thanks.
Xavier

Quoted 39 lines · Kurt Garloff

Hi,

ACPI specification says that the OS must disregard reserved bits. The x86_64 SRAT parser does not discard the upper 24 bits of the proximity_domain (pxm) in the acpi_srat_mem_affinity entries for SRAT v1 tables. (v2 has 32 bits wide fields.)
This can lead to problems with poor BIOS implementations that failed to set resreved bytes to zero. (The ACPI spec is a bit vague here unfortunately.)

This was also inconsistent: On x86-64 (srat_64.c), the _cpu_affinity does only use the low 8 bits of pxm, while the full 32 bits of _mem_affinity are consumed.
In srat_32.c (x86), only 8bits are used (which is OK, a 32bit system
with >256 PXMs does not seem reasonable at all).
On ia64, the support of more than 8 bits was consistent between
mem and cpu affinity entries, however it dependent on "sn2" platform.

The patch series has the following goals:
* Make the kernel support consistently 8bits or 32bits for the proximity domain
* Make this dependent on the SRAT version; v1 => 8bits, v2 => 32bits.

Overview over the patches:
- [1/3] Store the SRAT table version value in acpi_srat_revision
- [2/3] x86-64: Discard the upper 24 bits in mem_affinity if rev <= 1 and use upper 24bits in cpu_affinity if rev >= 2
- [3/3] ia64: Also use upper 8/24bits if rev >= 2 (but leave logic to enable on sn2 as well -- I don't know if sn2 reports v1 or v2 SRAT) Also add two __init decls in ia64 pxm accessors.

Patch has been tested on x86-64 against an 2.6.27.x kernel. (Patch is against current git.)

Thanks for James, Greg, Alexey, Norbert for comments, review and testing.

Please review and apply!

Greg, I believe this is a candidate for -stable.

#7 lore raw
In reply to Kurt Garloff

I just wanted to post in to the thread that this patch set solved both the node numbering (where there was a node0 and a node2) and the issue where all CPUs were showing up one the first node.

I applied the three patches against 2.6.30-git14.

We look forward to seeing the changes integrated.

On this 2-socket Intel 5500 based system (Nehalem), the /sys stuff looks good:

$ ls /sys/devices/system/node/node?
/sys/devices/system/node/node0:
cpu0  cpu2  cpulist  distance  numastat
cpu1  cpu3  cpumap   meminfo   scan_unevictable_pages

/sys/devices/system/node/node1:

cpu4  cpu6  cpulist  distance  numastat
cpu5  cpu7  cpumap   meminfo   scan_unevictable_pages

Where as, before, it looked like this (non-ordered, all CPUs showing up in node 0).

# ls -ld /sys/devices/system/node/node*
drwxr-xr-x 2 root root 0 2009-06-18 09:30 /sys/devices/system/node/node0
drwxr-xr-x 2 root root 0 2009-06-18 09:30 /sys/devices/system/node/node2

[root@cct201 ~]# ls /sys/devices/system/node/node0

cpu0  cpu2  cpu4  cpu6  cpulist  distance  numastat
cpu1  cpu3  cpu5  cpu7  cpumap   meminfo   scan_unevictable_pages

System details:
- SGI Altix XE 270
- Supermicro X8DTN v 1.1 mainboard
- 2 sockets of Xeon X5570

/proc/cpuinfo snip

[root@cct201 erikj]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5866.85
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 1
cpu cores : 4
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 8212.05
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 2
cpu cores : 4
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5865.76
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 0
siblings : 4
core id : 3
cpu cores : 4
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5865.76
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 0
cpu cores : 4
apicid : 16
initial apicid : 16
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5828.64
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 1
cpu cores : 4
apicid : 18
initial apicid : 18
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5865.81
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 2
cpu cores : 4
apicid : 20
initial apicid : 20
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5865.80
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU X5570 @ 2.93GHz
stepping : 5
cpu MHz : 1600.000
cache size : 8192 KB
physical id : 1
siblings : 4
core id : 3
cpu cores : 4
apicid : 22
initial apicid : 22
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology tsc_reliable nonstop_tsc pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida tpr_shadow vnmi flexpriority ept vpid
bogomips : 5865.80
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

#8 lore raw

cover Resend: Discard reserved PXM bits for SRAT v1

Erik,

Quoted 7 lines · Erik Jacobson

I just wanted to post in to the thread that this patch set solved both the node numbering (where there was a node0 and a node2) and the issue where all CPUs were showing up one the first node.

I applied the three patches against 2.6.30-git14.

We look forward to seeing the changes integrated.

So do I.
The patches are obviously already in SUSE kernels, I'd like to see them in mainline too.

Quoted 21 lines

On this 2-socket Intel 5500 based system (Nehalem), the /sys stuff looks good:

$ ls /sys/devices/system/node/node?
/sys/devices/system/node/node0:
cpu0  cpu2  cpulist  distance  numastat
cpu1  cpu3  cpumap   meminfo   scan_unevictable_pages

/sys/devices/system/node/node1:

cpu4  cpu6  cpulist  distance  numastat
cpu5  cpu7  cpumap   meminfo   scan_unevictable_pages

Where as, before, it looked like this (non-ordered, all CPUs showing up in node 0).

# ls -ld /sys/devices/system/node/node*
drwxr-xr-x 2 root root 0 2009-06-18 09:30 /sys/devices/system/node/node0
drwxr-xr-x 2 root root 0 2009-06-18 09:30 /sys/devices/system/node/node2

[root@cct201 ~]# ls /sys/devices/system/node/node0

Quoted 8 lines
cpu0  cpu2  cpu4  cpu6  cpulist  distance  numastat
cpu1  cpu3  cpu5  cpu7  cpumap   meminfo   scan_unevictable_pages

System details:
- SGI Altix XE 270
- Supermicro X8DTN v 1.1 mainboard
- 2 sockets of Xeon X5570

OK, you really observe the same here as did FZJ.
Thanks for reporting!

So let me resend the patches, rediffed against current 2.6.31rc git in followup mails.

Len, Linus, please adopt.

Best,

Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.
#9 lore raw

PATCH 1/3 Resend: [PATCH 1/3]: Store SRAT revision

Hi,

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields. In order to know whether to disregard or not, we must know what version the SRAT table has.

This patch stores the SRAT table revision for later consumption by arch specific __init functions.

This is patch 1/3.

Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

From: Kurt Garloff <[email protected]>
Subject: Store SRAT table revision

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields. In order to know whether to disregard or not, we must know what version the SRAT table has.

This patch stores the SRAT table revision for later consumption by arch specific __init functions.

This is patch 1/3.

drivers/acpi/numa.c +3 -0
diff -r a28eb89d1fe1 drivers/acpi/numa.c
@@ -42,6 +42,8 @@
 static int node_to_pxm_map[MAX_NUMNODES]
 				= { [0 ... MAX_NUMNODES - 1] = PXM_INVAL };
 
+unsigned char acpi_srat_revision __initdata;
+
 int pxm_to_node(int pxm)
 {
 	if (pxm < 0)
@@ -266,6 +268,7 @@
 		return -EINVAL;
 
 	srat = (struct acpi_table_srat *)table;
+	acpi_srat_revision = srat->header.revision;
 
 	return 0;
 }
include/acpi/acpi_numa.h +1 -0
diff -r a28eb89d1fe1 include/acpi/acpi_numa.h
@@ -15,6 +15,7 @@
 extern int node_to_pxm(int);
 extern void __acpi_map_pxm_to_node(int, int);
 extern int acpi_map_pxm_to_node(int);
+extern unsigned char acpi_srat_revision;
 
 #endif				/* CONFIG_ACPI_NUMA */
 #endif				/* __ACP_NUMA_H */
#10 lore raw

PATCH 2/3 Resend: x86-64: Handle SRAT v1 and v2 consistently

In reply to Kurt Garloff

Hi,

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields.

x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in cpu_affinity, but 32 bits in mem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).

This is patch 2/3.

Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

From: Kurt Garloff <[email protected]>
Subject: Use SRAT table rev to use 8bit or 32bit PXM fields (x86-64)
References: bnc#503038

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields.

x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in cpu_affinity, but 32 bits in mem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).

This is patch 2/3.

arch/x86/mm/srat_64.c +4 -0
diff -r a28eb89d1fe1 arch/x86/mm/srat_64.c
@@ -154,6 +154,8 @@
 	if ((pa->flags & ACPI_SRAT_CPU_ENABLED) == 0)
 		return;
 	pxm = pa->proximity_domain_lo;
+	if (acpi_srat_revision >= 2)
+		pxm |= *((unsigned int*)pa->proximity_domain_hi) << 8;
 	node = setup_node(pxm);
 	if (node < 0) {
 		printk(KERN_ERR "SRAT: Too many proximity domains %x\n", pxm);
@@ -255,6 +257,8 @@
 	start = ma->base_address;
 	end = start + ma->length;
 	pxm = ma->proximity_domain;
+	if (acpi_srat_revision <= 1)
+		pxm &= 0xff;
 	node = setup_node(pxm);
 	if (node < 0) {
 		printk(KERN_ERR "SRAT: Too many proximity domains.\n");
Acked-by on v1 2/3
#11 lore raw

Re: [PATCH 2/3] Resend: x86-64: Handle SRAT v1 and v2 consistently

Quoted 11 lines · Kurt Garloff

Hi,

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields.

x86-64 was rather inconsistent prior to this patch; it used 8 bits for the pxm field in cpu_affinity, but 32 bits in mem_affinity. This patch makes it consistent: Either use 8 bits consistently (SRAT rev 1 or lower) or 32 bits (SRAT rev 2 or higher).

Quoted 4 lines

This is patch 2/3.

--

for 1, 2

YH

#12 lore raw

PATCH 3/3 Resend: Consider SRAT rev on ia64

In reply to Kurt Garloff

Hi,

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields.

ia64 did handle the PXM fields almost consistently, but depending on sgi's sn2 platform. This patch leaves the sn2 logic in, but does also use 16/32 bits for PXM if the SRAT has rev 2 or higher.

The patch also adds __init to the two pxm accessor functions, as they access __initdata now and are called from an __init function only anyway.

Note that the code only uses 16 bits for the PXM field in the processor proximity field; the patch does not address this as 16 bits are more than enough.

This is patch 3/3.

Signature · 1 line
Kurt Garloff, VP OPS Partner Engineering -- Novell Inc.

From: Kurt Garloff <[email protected]>
Subject: Use SRAT table rev to use 8bit or 16/32bit PXM fields (ia64)
References: bnc#503038

In SRAT v1, we had 8bit proximity domain (PXM) fields; SRAT v2 provides 32bits for these. The new fields were reserved before. According to the ACPI spec, the OS must disregard reserved fields.

ia64 did handle the PXM fields almost consistently, but depending on sgi's sn2 platform. This patch leaves the sn2 logic in, but does also use 16/32 bits for PXM if the SRAT has rev 2 or higher.

The patch also adds __init to the two pxm accessor functions, as they access __initdata now and are called from an __init function only anyway.

Note that the code only uses 16 bits for the PXM field in the processor proximity field; the patch does not address this as 16 bits are more than enough.

This is patch 3/3.

arch/ia64/kernel/acpi.c +6 -4
diff -r a28eb89d1fe1 arch/ia64/kernel/acpi.c
@@ -456,22 +456,24 @@
 static struct acpi_table_slit __initdata *slit_table;
 cpumask_t early_cpu_possible_map = CPU_MASK_NONE;
 
-static int get_processor_proximity_domain(struct acpi_srat_cpu_affinity *pa)
+static int __init
+get_processor_proximity_domain(struct acpi_srat_cpu_affinity *pa)
 {
 	int pxm;
 
 	pxm = pa->proximity_domain_lo;
-	if (ia64_platform_is("sn2"))
+	if (ia64_platform_is("sn2") || acpi_srat_revision >= 2)
 		pxm += pa->proximity_domain_hi[0] << 8;
 	return pxm;
 }
 
-static int get_memory_proximity_domain(struct acpi_srat_mem_affinity *ma)
+static int __init
+get_memory_proximity_domain(struct acpi_srat_mem_affinity *ma)
 {
 	int pxm;
 
 	pxm = ma->proximity_domain;
-	if (!ia64_platform_is("sn2"))
+	if (!ia64_platform_is("sn2") && acpi_srat_revision <= 1)
 		pxm &= 0xff;
 
 	return pxm;
#13 lore raw
In reply to Kurt Garloff
Quoted 23 lines · On Mon, 18 May 2009 11:33:52 +0200 Kurt Garloff <[email protected]> wrote:

Hi,

ACPI specification says that the OS must disregard reserved bits. The x86_64 SRAT parser does not discard the upper 24 bits of the proximity_domain (pxm) in the acpi_srat_mem_affinity entries for SRAT v1 tables. (v2 has 32 bits wide fields.)
This can lead to problems with poor BIOS implementations that failed to set resreved bytes to zero. (The ACPI spec is a bit vague here unfortunately.)

This was also inconsistent: On x86-64 (srat_64.c), the _cpu_affinity does only use the low 8 bits of pxm, while the full 32 bits of _mem_affinity are consumed.
In srat_32.c (x86), only 8bits are used (which is OK, a 32bit system with >256 PXMs does not seem reasonable at all).
On ia64, the support of more than 8 bits was consistent between mem and cpu affinity entries, however it dependent on "sn2" platform.

The patch series has the following goals:
* Make the kernel support consistently 8bits or 32bits for the proximity domain
* Make this dependent on the SRAT version; v1 => 8bits, v2 => 32bits.

Quoted 8 lines

Overview over the patches:
- [1/3] Store the SRAT table version value in acpi_srat_revision
- [2/3] x86-64: Discard the upper 24 bits in mem_affinity if rev <= 1 and use upper 24bits in cpu_affinity if rev >= 2
- [3/3] ia64: Also use upper 8/24bits if rev >= 2 (but leave logic to enable on sn2 as well -- I don't know if sn2 reports v1 or v2 SRAT) Also add two __init decls in ia64 pxm accessors.

Did this patchset get lost or something? I don't see it anywhere, not even in the ACPI 'testing' tree...