Skip to contentSkip to conversation

x86: fix nodes_cover_memory

PATCH In review Yinghai Lu 3 participants 10 messages May 6, 2009 – May 11, 2009

Subsystems (2)
lore
Share & tools
lore

Yinghai Lu

author
#0 lore raw

PATCH 1/1 x86: fix nodes_cover_memory

found one system that missed one entry for one node in SRAT, and that SRAT is not rejected by nodes_cover_memory()

it turns out that we can not use absent_page_in_range to calaulate e820ram, bacause that will use early_node_map and that is AND result of e820 and SRAT.

revert back to use e820_hole_size instead.

also change that difference checking to 1M instead of 4G, because e820ram, and pxmram are in pages.

[ Impact: reject wrong SRAT tables ]

---
 arch/x86/mm/srat_64.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/mm/srat_64.c +2 -2
Index: linux-2.6/arch/x86/mm/srat_64.c
===================================================================
@@ -345,9 +345,9 @@ static int __init nodes_cover_memory(con
 			pxmram = 0;
 	}
 
-	e820ram = max_pfn - absent_pages_in_range(0, max_pfn);
+	e820ram = max_pfn - (e820_hole_size(0, max_pfn<<PAGE_SHIFT)>>PAGE_SHIFT);
 	/* We seem to lose 3 pages somewhere. Allow a bit of slack. */
-	if ((long)(e820ram - pxmram) >= 1*1024*1024) {
+	if ((long)(e820ram - pxmram) >= (1<<(20 - PAGE_SHIFT))) {
 		printk(KERN_ERR
 	"SRAT: PXMs only cover %luMB of your %luMB e820 RAM. Not used.\n",
 			(pxmram << PAGE_SHIFT) >> 20,
#1 lore raw
Quoted 8 lines · Yinghai Lu

found one system that missed one entry for one node in SRAT, and that SRAT is not rejected by nodes_cover_memory()

it turns out that we can not use absent_page_in_range to calaulate e820ram, bacause that will use early_node_map and that is AND result of e820 and SRAT.

Correct, good spot.

Quoted 2 lines

revert back to use e820_hole_size instead.

I think the patch fixing this part of the problem is good, but the changelog could be better. It took me a while to figure out what the problem was and why this patch addressed it.

How about something like the following?

====
Sanity check the e820 against the SRAT table using only information from the e820 map

node_cover_memory() sanity checks the SRAT table by ensuring that all PXMs cover the memory reported in the e820. However, when calculating the size of the holes in the e820, it uses the early_node_map[] which contains information taken from both SRAT and e820. If the SRAT is missing an entry, then it is not detected that the SRAT table is incorrect and missing entries.

This patch uses the e820 map to calculate the holes instead of early_node_map[].
====

As an aside, it strikes me as odd that we discard an entire SRAT because it is missing an entry in the e820. The impact may only be that the affinity for a range of memory is incorrect, but it does not necessarily mean that the entire table is incorrect. The intention of the code appears to be "if there is any error in the SRAT, it's best ignored" though so maybe it's best left alone.

Quoted 3 lines

also change that difference checking to 1M instead of 4G, because e820ram, and pxmram are in pages.

While I agree with you, this should be a separate patch with its own changelog. Something like

===
Allow 1MB of slack between the e820 map and SRAT, not 4GB

It is expected that there be slight differences between the e820 map and the SRAT table and the intention was that 1MB of slack be allowed. The calculation comparing e820ram and pxmram assumes the units are bytes, when they are in fact pages. This means 4GB of slack is being allowed, not 1MB. This patch makes the correct comparison
===

(1<<(20 - PAGE_SHIFT)) is a bit unreadable. At the very least, change the comment above from "Allow a bit of slack" to "Allow 1MB of slack" so the next reader knows what the intention of (1<<(20 - PAGE_SHIFT)) is.

Thanks

Quoted 26 lines

[ Impact: reject wrong SRAT tables ]

---
 arch/x86/mm/srat_64.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

Index: linux-2.6/arch/x86/mm/srat_64.c
===================================================================

arch/x86/mm/srat_64.c +2 -2
@@ -345,9 +345,9 @@ static int __init nodes_cover_memory(con
 			pxmram = 0;
 	}
 
-	e820ram = max_pfn - absent_pages_in_range(0, max_pfn);
+	e820ram = max_pfn - (e820_hole_size(0, max_pfn<<PAGE_SHIFT)>>PAGE_SHIFT);
 	/* We seem to lose 3 pages somewhere. Allow a bit of slack. */
-	if ((long)(e820ram - pxmram) >= 1*1024*1024) {
+	if ((long)(e820ram - pxmram) >= (1<<(20 - PAGE_SHIFT))) {
 		printk(KERN_ERR
 	"SRAT: PXMs only cover %luMB of your %luMB e820 RAM. Not used.\n",
 			(pxmram << PAGE_SHIFT) >> 20,
Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
#2 lore raw
Quoted 58 lines · Mel Gorman
Quoted 8 lines · Yinghai Lu

found one system that missed one entry for one node in SRAT, and that SRAT is not rejected by nodes_cover_memory()

it turns out that we can not use absent_page_in_range to calaulate e820ram, bacause that will use early_node_map and that is AND result of e820 and SRAT.

Correct, good spot.

Quoted 2 lines

revert back to use e820_hole_size instead.

I think the patch fixing this part of the problem is good, but the changelog could be better. It took me a while to figure out what the problem was and why this patch addressed it.

How about something like the following?

====
Sanity check the e820 against the SRAT table using only information from the e820 map

node_cover_memory() sanity checks the SRAT table by ensuring that all PXMs cover the memory reported in the e820. However, when calculating the size of the holes in the e820, it uses the early_node_map[] which contains information taken from both SRAT and e820. If the SRAT is missing an entry, then it is not detected that the SRAT table is incorrect and missing entries.

This patch uses the e820 map to calculate the holes instead of early_node_map[].
====

As an aside, it strikes me as odd that we discard an entire SRAT because it is missing an entry in the e820. The impact may only be that the affinity for a range of memory is incorrect, but it does not necessarily mean that the entire table is incorrect. The intention of the code appears to be "if there is any error in the SRAT, it's best ignored" though so maybe it's best left alone.

Quoted 3 lines

also change that difference checking to 1M instead of 4G, because e820ram, and pxmram are in pages.

While I agree with you, this should be a separate patch with its own changelog. Something like

===
Allow 1MB of slack between the e820 map and SRAT, not 4GB

It is expected that there be slight differences between the e820 map and the SRAT table and the intention was that 1MB of slack be allowed. The calculation comparing e820ram and pxmram assumes the units are bytes, when they are in fact pages. This means 4GB of slack is being allowed, not 1MB. This patch makes the correct comparison
===

Quoted 5 lines

(1<<(20 - PAGE_SHIFT)) is a bit unreadable. At the very least, change the comment above from "Allow a bit of slack" to "Allow 1MB of slack" so the next reader knows what the intention of (1<<(20 - PAGE_SHIFT)) is.

Thanks

thanks Mel!

Yinghai, mind resending the patch as two patches, with Mel's changelogs in place and with Mel's Acked-by as well?

Thanks,

Ingo

Acked-by
#3 lore raw
Quoted 65 lines · Ingo Molnar
Quoted 62 lines · Mel Gorman
Quoted 8 lines · Yinghai Lu

found one system that missed one entry for one node in SRAT, and that SRAT is not rejected by nodes_cover_memory()

it turns out that we can not use absent_page_in_range to calaulate e820ram, bacause that will use early_node_map and that is AND result of e820 and SRAT.

Correct, good spot.

Quoted 2 lines

revert back to use e820_hole_size instead.

I think the patch fixing this part of the problem is good, but the changelog could be better. It took me a while to figure out what the problem was and why this patch addressed it.

How about something like the following?

====
Sanity check the e820 against the SRAT table using only information from the e820 map

node_cover_memory() sanity checks the SRAT table by ensuring that all PXMs cover the memory reported in the e820. However, when calculating the size of the holes in the e820, it uses the early_node_map[] which contains information taken from both SRAT and e820. If the SRAT is missing an entry, then it is not detected that the SRAT table is incorrect and missing entries.

This patch uses the e820 map to calculate the holes instead of early_node_map[].
====

As an aside, it strikes me as odd that we discard an entire SRAT because it is missing an entry in the e820. The impact may only be that the affinity for a range of memory is incorrect, but it does not necessarily mean that the entire table is incorrect. The intention of the code appears to be "if there is any error in the SRAT, it's best ignored" though so maybe it's best left alone.

Quoted 3 lines

also change that difference checking to 1M instead of 4G, because e820ram, and pxmram are in pages.

While I agree with you, this should be a separate patch with its own changelog. Something like

===
Allow 1MB of slack between the e820 map and SRAT, not 4GB

It is expected that there be slight differences between the e820 map and the SRAT table and the intention was that 1MB of slack be allowed. The calculation comparing e820ram and pxmram assumes the units are bytes, when they are in fact pages. This means 4GB of slack is being allowed, not 1MB. This patch makes the correct comparison
===

(1<<(20 - PAGE_SHIFT)) is a bit unreadable. At the very least, change the comment above from "Allow a bit of slack" to "Allow 1MB of slack" so the next reader knows what the intention of (1<<(20 - PAGE_SHIFT)) is.

Quoted 7 lines
Quoted 1 line

Thanks

thanks Mel!

Yinghai, mind resending the patch as two patches, with Mel's changelogs in place and with Mel's Acked-by as well?

My bad for leaving it out, but yes. Assuming there are two patches with the updated changelog, I'm fine with adding the

Signature · 3 lines
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab
author
#4 lore raw

PATCH 1/2 x86: Sanity check the e820 against the SRAT table using e820 map only

In reply to Ingo Molnar

node_cover_memory() sanity checks the SRAT table by ensuring that all PXMs cover the memory reported in the e820. However, when calculating the size of the holes in the e820, it uses the early_node_map[] which contains information taken from both SRAT and e820. If the SRAT is missing an entry, then it is not detected that the SRAT table is incorrect and missing entries.

This patch uses the e820 map to calculate the holes instead of early_node_map[].

comment is from Mel.

[ Impact: reject wrong SRAT tables ]

---
 arch/x86/mm/srat_64.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
arch/x86/mm/srat_64.c +1 -1
Index: linux-2.6/arch/x86/mm/srat_64.c
===================================================================
@@ -345,7 +345,7 @@ static int __init nodes_cover_memory(con
 			pxmram = 0;
 	}
 
-	e820ram = max_pfn - absent_pages_in_range(0, max_pfn);
+	e820ram = max_pfn - (e820_hole_size(0, max_pfn<<PAGE_SHIFT)>>PAGE_SHIFT);
 	/* We seem to lose 3 pages somewhere. Allow a bit of slack. */
 	if ((long)(e820ram - pxmram) >= 1*1024*1024) {
 		printk(KERN_ERR
author
#5 lore raw

PATCH 2/2 x86: Allow 1MB of slack between the e820 map and SRAT, not 4GB

It is expected that there be slight differences between the e820 map and the SRAT table and the intention was that 1MB of slack be allowed. The calculation comparing e820ram and pxmram assumes the units are bytes, when they are in fact pages. This means 4GB of slack is being allowed, not 1MB. This patch makes the correct comparison

comment is from Mel.

[ Impact: don't accept the srat that could dump 2g etc ram ]

---
 arch/x86/mm/srat_64.c |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
arch/x86/mm/srat_64.c +2 -2
Index: linux-2.6/arch/x86/mm/srat_64.c
===================================================================
@@ -346,8 +346,8 @@ static int __init nodes_cover_memory(con
 	}
 
 	e820ram = max_pfn - (e820_hole_size(0, max_pfn<<PAGE_SHIFT)>>PAGE_SHIFT);
-	/* We seem to lose 3 pages somewhere. Allow a bit of slack. */
-	if ((long)(e820ram - pxmram) >= 1*1024*1024) {
+	/* We seem to lose 3 pages somewhere. Allow 1M of slack. */
+	if ((long)(e820ram - pxmram) >= (1<<(20 - PAGE_SHIFT))) {
 		printk(KERN_ERR
 	"SRAT: PXMs only cover %luMB of your %luMB e820 RAM. Not used.\n",
 			(pxmram << PAGE_SHIFT) >> 20,
#7 lore raw

Re: [PATCH 1/2] x86: Sanity check the e820 against the SRAT table using e820 map only

In reply to Yinghai Lu

arch/x86/mm/srat_64.c static int __init nodes_cover_memory(con · line 348

Quoted 2 lines · Yinghai Lu
arch/x86/mm/srat_64.c +1 -1
-	e820ram = max_pfn - absent_pages_in_range(0, max_pfn);
+	e820ram = max_pfn - (e820_hole_size(0, max_pfn<<PAGE_SHIFT)>>PAGE_SHIFT);

btw., it would be nice to have a debug check that prints a warning if the SRAT does not cover all RAM, or if it covers RAM that is not present in the e820 map. Such a warning might alert us to double check all the PXM settings in the SRAT and could uncover more quirks like the above ...

Ingo

#6 lore raw

[tip:x86/mm] x86: Allow 1MB of slack between the e820 map and SRAT, not 4GB

In reply to Yinghai Lu
Commit-ID:  0964b0562bb9c93194e852b47bab2397b9e11c18
Gitweb:     http://git.kernel.org/tip/0964b0562bb9c93194e852b47bab2397b9e11c18
Author:     Yinghai Lu <[email protected]>
AuthorDate: Fri, 8 May 2009 00:37:34 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 11 May 2009 11:38:21 +0200

x86: Allow 1MB of slack between the e820 map and SRAT, not 4GB

It is expected that there might be slight differences between the e820 map and the SRAT table and the intention was that 1MB of slack be allowed.

The calculation comparing e820ram and pxmram assumes the units are bytes, when they are in fact pages. This means 4GB of slack is being allowed, not 1MB. This patch makes the correct comparison.

comment is from Mel.

[ Impact: don't accept buggy SRATs that could dump up to 4G of RAM ]

LKML-Reference: <[email protected]>

---
 arch/x86/mm/srat_64.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)
arch/x86/mm/srat_64.c +2 -2
index c7a18aa..87b45bf 100644
@@ -346,8 +346,8 @@ static int __init nodes_cover_memory(const struct bootnode *nodes)
 	}
 
 	e820ram = max_pfn - (e820_hole_size(0, max_pfn<<PAGE_SHIFT)>>PAGE_SHIFT);
-	/* We seem to lose 3 pages somewhere. Allow a bit of slack. */
-	if ((long)(e820ram - pxmram) >= 1*1024*1024) {
+	/* We seem to lose 3 pages somewhere. Allow 1M of slack. */
+	if ((long)(e820ram - pxmram) >= (1<<(20 - PAGE_SHIFT))) {
 		printk(KERN_ERR
 	"SRAT: PXMs only cover %luMB of your %luMB e820 RAM. Not used.\n",
 			(pxmram << PAGE_SHIFT) >> 20,
#9 lore raw

[tip:x86/mm] x86: Sanity check the e820 against the SRAT table using e820 map only

In reply to Yinghai Lu
Commit-ID:  b37ab91907e9002925f4217e3bbd496aa12c2fa3
Gitweb:     http://git.kernel.org/tip/b37ab91907e9002925f4217e3bbd496aa12c2fa3
Author:     Yinghai Lu <[email protected]>
AuthorDate: Fri, 8 May 2009 00:36:44 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Mon, 11 May 2009 11:35:07 +0200

x86: Sanity check the e820 against the SRAT table using e820 map only

node_cover_memory() sanity checks the SRAT table by ensuring that all PXMs cover the memory reported in the e820.

However, when calculating the size of the holes in the e820, it uses the early_node_map[] which contains information taken from both SRAT and e820. If the SRAT is missing an entry, then it is not detected that the SRAT table is incorrect and missing entries.

This patch uses the e820 map to calculate the holes instead of early_node_map[].

comment is from Mel.

[ Impact: reject incorrect SRAT tables ]

LKML-Reference: <[email protected]>

---
 arch/x86/mm/srat_64.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
arch/x86/mm/srat_64.c +1 -1
index 0176595..c7a18aa 100644
@@ -345,7 +345,7 @@ static int __init nodes_cover_memory(const struct bootnode *nodes)
 			pxmram = 0;
 	}
 
-	e820ram = max_pfn - absent_pages_in_range(0, max_pfn);
+	e820ram = max_pfn - (e820_hole_size(0, max_pfn<<PAGE_SHIFT)>>PAGE_SHIFT);
 	/* We seem to lose 3 pages somewhere. Allow a bit of slack. */
 	if ((long)(e820ram - pxmram) >= 1*1024*1024) {
 		printk(KERN_ERR
author
#8 lore raw

Re: [PATCH 1/2] x86: Sanity check the e820 against the SRAT table using e820 map only

In reply to Ingo Molnar
Quoted 10 lines · Ingo Molnar

arch/x86/mm/srat_64.c static int __init nodes_cover_memory(con · line 348

Quoted 2 lines · Yinghai Lu
arch/x86/mm/srat_64.c +1 -1
-	e820ram = max_pfn - absent_pages_in_range(0, max_pfn);
+	e820ram = max_pfn - (e820_hole_size(0, max_pfn<<PAGE_SHIFT)>>PAGE_SHIFT);

btw., it would be nice to have a debug check that prints a warning if the SRAT does not cover all RAM, or if it covers RAM that is not present in the e820 map. Such a warning might alert us to double check all the PXM settings in the SRAT and could uncover more quirks like the above ...

we trust e820 than SRAT mem entries.

so early_node_map already are result of in e820 range AND srat. (the range in SRAT that is not in e820 is dumped)

just check srat table (without AND) with e820 will have some confusing. because SRAT would cover the hole below 4g that is not in E820.

YH