Skip to contentSkip to conversation

mm: Enable CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA

PATCH Superseded Hoan Tran OS 8 participants 29 messages Jul 11, 2019 – Sep 2, 2019

Subsystems (7)
Memory management x86 PowerPC s390 +3 more subsystems

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover mm: Enable CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA

In NUMA layout which nodes have memory ranges that span across other nodes, the mm driver can detect the memory node id incorrectly.

For example, with layout below
Node 0 address: 0000 xxxx 0000 xxxx
Node 1 address: xxxx 1111 xxxx 1111

Note:
- Memory from low to high
- 0/1: Node id
- x: Invalid memory of a node

When mm probes the memory map, without CONFIG_NODES_SPAN_OTHER_NODES config, mm only checks the memory validity but not the node id. Because of that, Node 1 also detects the memory from node 0 as below when it scans from the start address to the end address of node 1.

Node 0 address: 0000 xxxx xxxx xxxx
Node 1 address: xxxx 1111 1111 1111

This layout could occur on any architecture. This patch enables CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA to fix this issue.

V2:
* Revise the patch description

Hoan Tran (5):
mm: Enable CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA
powerpc: Kconfig: Remove CONFIG_NODES_SPAN_OTHER_NODES
x86: Kconfig: Remove CONFIG_NODES_SPAN_OTHER_NODES
sparc: Kconfig: Remove CONFIG_NODES_SPAN_OTHER_NODES
s390: Kconfig: Remove CONFIG_NODES_SPAN_OTHER_NODES

 arch/powerpc/Kconfig | 9 ---------
 arch/s390/Kconfig    | 8 --------
 arch/sparc/Kconfig   | 9 ---------
 arch/x86/Kconfig     | 9 ---------
 mm/page_alloc.c      | 2 +-
 5 files changed, 1 insertion(+), 36 deletions(-)
Signature · 1 line
2.7.4
#1 lore raw

PATCH v2 1/5 mm: Enable CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA

In NUMA layout which nodes have memory ranges that span across other nodes, the mm driver can detect the memory node id incorrectly.

For example, with layout below
Node 0 address: 0000 xxxx 0000 xxxx
Node 1 address: xxxx 1111 xxxx 1111

Note:
- Memory from low to high
- 0/1: Node id
- x: Invalid memory of a node

When mm probes the memory map, without CONFIG_NODES_SPAN_OTHER_NODES config, mm only checks the memory validity but not the node id. Because of that, Node 1 also detects the memory from node 0 as below when it scans from the start address to the end address of node 1.

Node 0 address: 0000 xxxx xxxx xxxx
Node 1 address: xxxx 1111 1111 1111

This layout could occur on any architecture. This patch enables CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA to fix this issue.

---
 mm/page_alloc.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
mm/page_alloc.c +1 -1
index d66bc8a..6335505 100644
@@ -1413,7 +1413,7 @@ int __meminit early_pfn_to_nid(unsigned long pfn)
 }
 #endif
 
-#ifdef CONFIG_NODES_SPAN_OTHER_NODES
+#ifdef CONFIG_NUMA
 /* Only safe to use early in boot when initialisation is single-threaded */
 static inline bool __meminit early_pfn_in_nid(unsigned long pfn, int node)
 {
Signature · 1 line
2.7.4
#2 lore raw

PATCH v2 2/5 powerpc: Kconfig: Remove CONFIG_NODES_SPAN_OTHER_NODES

In reply to Hoan Tran OS

Remove CONFIG_NODES_SPAN_OTHER_NODES as it's enabled by default with NUMA.

---
 arch/powerpc/Kconfig | 9 ---------
 1 file changed, 9 deletions(-)
arch/powerpc/Kconfig +0 -9
index 8c1c636..bdde8bc 100644
@@ -629,15 +629,6 @@ config ARCH_MEMORY_PROBE
 	def_bool y
 	depends on MEMORY_HOTPLUG
 
-# Some NUMA nodes have memory ranges that span
-# other nodes.  Even though a pfn is valid and
-# between a node's start and end pfns, it may not
-# reside on that node.  See memmap_init_zone()
-# for details.
-config NODES_SPAN_OTHER_NODES
-	def_bool y
-	depends on NEED_MULTIPLE_NODES
-
 config STDBINUTILS
 	bool "Using standard binutils settings"
 	depends on 44x
Signature · 1 line
2.7.4
#3 lore raw

PATCH v2 3/5 x86: Kconfig: Remove CONFIG_NODES_SPAN_OTHER_NODES

In reply to Hoan Tran OS

Remove CONFIG_NODES_SPAN_OTHER_NODES as it's enabled by default with NUMA.

---
 arch/x86/Kconfig | 9 ---------
 1 file changed, 9 deletions(-)
arch/x86/Kconfig +0 -9
index 2bbbd4d..fa9318c 100644
@@ -1567,15 +1567,6 @@ config X86_64_ACPI_NUMA
 	---help---
 	  Enable ACPI SRAT based node topology detection.
 
-# Some NUMA nodes have memory ranges that span
-# other nodes.  Even though a pfn is valid and
-# between a node's start and end pfns, it may not
-# reside on that node.  See memmap_init_zone()
-# for details.
-config NODES_SPAN_OTHER_NODES
-	def_bool y
-	depends on X86_64_ACPI_NUMA
-
 config NUMA_EMU
 	bool "NUMA emulation"
 	depends on NUMA
Signature · 1 line
2.7.4
#4 lore raw

Re: [PATCH v2 3/5] x86: Kconfig: Remove CONFIG_NODES_SPAN_OTHER_NODES

Quoted 2 lines · Hoan Tran OS

Remove CONFIG_NODES_SPAN_OTHER_NODES as it's enabled by default with NUMA.

As I told you before this does not mention that the option is now enabled even for x86(32bit) configurations which did not enable it before and does not longer depend on X86_64_ACPI_NUMA.

And there is still no rationale why this makes sense.

Thanks,

tglx

#5 lore raw

Re: [PATCH v2 3/5] x86: Kconfig: Remove CONFIG_NODES_SPAN_OTHER_NODES

SGkgVGhvbWFzLA0KDQoNCk9uIDcvMTUvMTkgMTE6NDMgQU0sIFRob21hcyBHbGVpeG5lciB3cm90 ZToNCj4gT24gVGh1LCAxMSBKdWwgMjAxOSwgSG9hbiBUcmFuIE9TIHdyb3RlOg0KPiANCj4+IFJl bW92ZSBDT05GSUdfTk9ERVNfU1BBTl9PVEhFUl9OT0RFUyBhcyBpdCdzIGVuYWJsZWQNCj4+IGJ5 IGRlZmF1bHQgd2l0aCBOVU1BLg0KPiANCj4gQXMgSSB0b2xkIHlvdSBiZWZvcmUgdGhpcyBkb2Vz IG5vdCBtZW50aW9uIHRoYXQgdGhlIG9wdGlvbiBpcyBub3cgZW5hYmxlZA0KPiBldmVuIGZvciB4 ODYoMzJiaXQpIGNvbmZpZ3VyYXRpb25zIHdoaWNoIGRpZCBub3QgZW5hYmxlIGl0IGJlZm9yZSBh bmQgZG9lcw0KPiBub3QgbG9uZ2VyIGRlcGVuZCBvbiBYODZfNjRfQUNQSV9OVU1BLg0KDQpBZ3Jl ZWQsIGxldCBtZSBhZGQgaXQgaW50byB0aGlzIHBhdGNoIGRlc2NyaXB0aW9uLg0KDQo+IA0KPiBB bmQgdGhlcmUgaXMgc3RpbGwgbm8gcmF0aW9uYWxlIHdoeSB0aGlzIG1ha2VzIHNlbnNlLg0KPiAN Cg0KQXMgd2Uga25vdyBhYm91dCB0aGUgbWVtbWFwX2luaXRfem9uZSgpIGZ1bmN0aW9uLCBpdCBp cyB1c2VkIHRvIA0KaW5pdGlhbGl6ZSBhbGwgcGFnZXMuIER1cmluZyBpbml0aWFsaXppbmcsIGVh cmx5X3Bmbl9pbl9uaWQoKSBmdW5jdGlvbiANCm1ha2VzIHN1cmUgdGhlIHBhZ2UgaXMgaW4gdGhl IHNhbWUgbm9kZSBpZC4gT3RoZXJ3aXNlLCANCm1lbW1hcF9pbml0X3pvbmUoKSBvbmx5IGNoZWNr cyB0aGUgcGFnZSB2YWxpZGl0eS4gSXQgd29uJ3Qgd29yayB3aXRoIA0Kbm9kZSBtZW1vcnkgc3Bh bnMgYWNyb3NzIHRoZSBvdGhlcnMuDQoNClRoZSBvcHRpb24gQ09ORklHX05PREVTX1NQQU5fT1RI RVJfTk9ERVMgaXMgb25seSB1c2VkIHRvIGVuYWJsZSANCmVhcmx5X3Bmbl9pbl9uaWQoKSBmdW5j dGlvbi4NCg0KSXQgb2NjdXJzIGR1cmluZyBib290LXRpbWUgYW5kIHdvbid0IGFmZmVjdCB0aGUg cnVuLXRpbWUgcGVyZm9ybWFuY2UuDQpBbmQgSSBzYXcgdGhlIG1ham9yaXR5IE5VTUEgYXJjaGl0 ZWN0dXJlcyBlbmFibGUgdGhpcyBvcHRpb24gYnkgZGVmYXVsdCANCndpdGggTlVNQS4NCg0KVGhh bmtzIGFuZCBSZWdhcmRzDQpIb2FuDQoNCg0KPiBUaGFua3MsDQo+IA0KPiAJdGdseA0KPiANCg=

#6 lore raw

PATCH v2 4/5 sparc: Kconfig: Remove CONFIG_NODES_SPAN_OTHER_NODES

In reply to Hoan Tran OS

Remove CONFIG_NODES_SPAN_OTHER_NODES as it's enabled by default with NUMA.

---
 arch/sparc/Kconfig | 9 ---------
 1 file changed, 9 deletions(-)
arch/sparc/Kconfig +0 -9
index 26ab6f5..13449ea 100644
@@ -291,15 +291,6 @@ config NODES_SHIFT
 	  Specify the maximum number of NUMA Nodes available on the target
 	  system.  Increases memory reserved to accommodate various tables.
 
-# Some NUMA nodes have memory ranges that span
-# other nodes.  Even though a pfn is valid and
-# between a node's start and end pfns, it may not
-# reside on that node.  See memmap_init_zone()
-# for details.
-config NODES_SPAN_OTHER_NODES
-	def_bool y
-	depends on NEED_MULTIPLE_NODES
-
 config ARCH_SELECT_MEMORY_MODEL
 	def_bool y if SPARC64
 
Signature · 1 line
2.7.4
#7 lore raw

PATCH v2 5/5 s390: Kconfig: Remove CONFIG_NODES_SPAN_OTHER_NODES

In reply to Hoan Tran OS

Remove CONFIG_NODES_SPAN_OTHER_NODES as it's enabled by default with NUMA.

---
 arch/s390/Kconfig | 8 --------
 1 file changed, 8 deletions(-)
arch/s390/Kconfig +0 -8
index 109243f..788a8e9 100644
@@ -438,14 +438,6 @@ config HOTPLUG_CPU
 	  can be controlled through /sys/devices/system/cpu/cpu#.
 	  Say N if you want to disable CPU hotplug.
 
-# Some NUMA nodes have memory ranges that span
-# other nodes.	Even though a pfn is valid and
-# between a node's start and end pfns, it may not
-# reside on that node.	See memmap_init_zone()
-# for details. <- They meant memory holes!
-config NODES_SPAN_OTHER_NODES
-	def_bool NUMA
-
 config NUMA
 	bool "NUMA support"
 	depends on SMP && SCHED_TOPOLOGY
Signature · 1 line
2.7.4
#8 lore raw
In reply to Hoan Tran OS
Quoted 17 lines · Hoan Tran OS

In NUMA layout which nodes have memory ranges that span across other nodes, the mm driver can detect the memory node id incorrectly.

For example, with layout below
Node 0 address: 0000 xxxx 0000 xxxx
Node 1 address: xxxx 1111 xxxx 1111

Note:
- Memory from low to high
- 0/1: Node id
- x: Invalid memory of a node

When mm probes the memory map, without CONFIG_NODES_SPAN_OTHER_NODES config, mm only checks the memory validity but not the node id. Because of that, Node 1 also detects the memory from node 0 as below when it scans from the start address to the end address of node 1.

Quoted 5 lines

Node 0 address: 0000 xxxx xxxx xxxx
Node 1 address: xxxx 1111 1111 1111

This layout could occur on any architecture. This patch enables CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA to fix this issue.

How do you know it could occur on any architecture? Surely you should just enable this for the architecture where you've noticed the problem.

#9 lore raw
In reply to Hoan Tran OS
Quoted 17 lines · Hoan Tran OS

In NUMA layout which nodes have memory ranges that span across other nodes, the mm driver can detect the memory node id incorrectly.

For example, with layout below
Node 0 address: 0000 xxxx 0000 xxxx
Node 1 address: xxxx 1111 xxxx 1111

Note:
- Memory from low to high
- 0/1: Node id
- x: Invalid memory of a node

When mm probes the memory map, without CONFIG_NODES_SPAN_OTHER_NODES config, mm only checks the memory validity but not the node id. Because of that, Node 1 also detects the memory from node 0 as below when it scans from the start address to the end address of node 1.

Quoted 5 lines

Node 0 address: 0000 xxxx xxxx xxxx
Node 1 address: xxxx 1111 1111 1111

This layout could occur on any architecture. This patch enables CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA to fix this issue.

Yes it can occur on any arch but most sane platforms simply do not overlap physical ranges. So I do not really see any reason to unconditionally enable the config for everybody. What is an advantage?

Signature · 2 lines
Michal Hocko
SUSE Labs
#10 lore raw

SGksDQoNCk9uIDcvMTIvMTkgMjowMiBQTSwgTWljaGFsIEhvY2tvIHdyb3RlOg0KPiBPbiBUaHUg MTEtMDctMTkgMjM6MjU6NDQsIEhvYW4gVHJhbiBPUyB3cm90ZToNCj4+IEluIE5VTUEgbGF5b3V0 IHdoaWNoIG5vZGVzIGhhdmUgbWVtb3J5IHJhbmdlcyB0aGF0IHNwYW4gYWNyb3NzIG90aGVyIG5v ZGVzLA0KPj4gdGhlIG1tIGRyaXZlciBjYW4gZGV0ZWN0IHRoZSBtZW1vcnkgbm9kZSBpZCBpbmNv cnJlY3RseS4NCj4+DQo+PiBGb3IgZXhhbXBsZSwgd2l0aCBsYXlvdXQgYmVsb3cNCj4+IE5vZGUg MCBhZGRyZXNzOiAwMDAwIHh4eHggMDAwMCB4eHh4DQo+PiBOb2RlIDEgYWRkcmVzczogeHh4eCAx MTExIHh4eHggMTExMQ0KPj4NCj4+IE5vdGU6DQo+PiAgIC0gTWVtb3J5IGZyb20gbG93IHRvIGhp Z2gNCj4+ICAgLSAwLzE6IE5vZGUgaWQNCj4+ICAgLSB4OiBJbnZhbGlkIG1lbW9yeSBvZiBhIG5v ZGUNCj4+DQo+PiBXaGVuIG1tIHByb2JlcyB0aGUgbWVtb3J5IG1hcCwgd2l0aG91dCBDT05GSUdf Tk9ERVNfU1BBTl9PVEhFUl9OT0RFUw0KPj4gY29uZmlnLCBtbSBvbmx5IGNoZWNrcyB0aGUgbWVt b3J5IHZhbGlkaXR5IGJ1dCBub3QgdGhlIG5vZGUgaWQuDQo+PiBCZWNhdXNlIG9mIHRoYXQsIE5v ZGUgMSBhbHNvIGRldGVjdHMgdGhlIG1lbW9yeSBmcm9tIG5vZGUgMCBhcyBiZWxvdw0KPj4gd2hl biBpdCBzY2FucyBmcm9tIHRoZSBzdGFydCBhZGRyZXNzIHRvIHRoZSBlbmQgYWRkcmVzcyBvZiBu b2RlIDEuDQo+Pg0KPj4gTm9kZSAwIGFkZHJlc3M6IDAwMDAgeHh4eCB4eHh4IHh4eHgNCj4+IE5v ZGUgMSBhZGRyZXNzOiB4eHh4IDExMTEgMTExMSAxMTExDQo+Pg0KPj4gVGhpcyBsYXlvdXQgY291 bGQgb2NjdXIgb24gYW55IGFyY2hpdGVjdHVyZS4gVGhpcyBwYXRjaCBlbmFibGVzDQo+PiBDT05G SUdfTk9ERVNfU1BBTl9PVEhFUl9OT0RFUyBieSBkZWZhdWx0IGZvciBOVU1BIHRvIGZpeCB0aGlz IGlzc3VlLg0KPiANCj4gWWVzIGl0IGNhbiBvY2N1ciBvbiBhbnkgYXJjaCBidXQgbW9zdCBzYW5l IHBsYXRmb3JtcyBzaW1wbHkgZG8gbm90DQo+IG92ZXJsYXAgcGh5c2ljYWwgcmFuZ2VzLiBTbyBJ IGRvIG5vdCByZWFsbHkgc2VlIGFueSByZWFzb24gdG8NCj4gdW5jb25kaXRpb25hbGx5IGVuYWJs ZSB0aGUgY29uZmlnIGZvciBldmVyeWJvZHkuIFdoYXQgaXMgYW4gYWR2YW50YWdlPw0KPiANCg0K QXMgSSBvYnNlcnZlZCBmcm9tIGFyY2ggZm9sZGVyLCB0aGVyZSBhcmUgOSBhcmNoIHN1cHBvcnQg TlVNQSBjb25maWcuDQoNCi4vYXJjaC9pYTY0L0tjb25maWc6Mzg3OmNvbmZpZyBOVU1BDQouL2Fy Y2gvcG93ZXJwYy9LY29uZmlnOjU4Mjpjb25maWcgTlVNQQ0KLi9hcmNoL3NwYXJjL0tjb25maWc6 MjgxOmNvbmZpZyBOVU1BDQouL2FyY2gvYWxwaGEvS2NvbmZpZzo1NTc6Y29uZmlnIE5VTUENCi4v YXJjaC9zaC9tbS9LY29uZmlnOjExMjpjb25maWcgTlVNQQ0KLi9hcmNoL2FybTY0L0tjb25maWc6 ODQxOmNvbmZpZyBOVU1BDQouL2FyY2gveDg2L0tjb25maWc6MTUzMTpjb25maWcgTlVNQQ0KLi9h cmNoL21pcHMvS2NvbmZpZzoyNjQ2OmNvbmZpZyBOVU1BDQouL2FyY2gvczM5MC9LY29uZmlnOjQ0 MTpjb25maWcgTlVNQQ0KDQpBbmQgdGhlcmUgYXJlIDUgYXJjaCBlbmFibGVzIENPTkZJR19OT0RF U19TUEFOX09USEVSX05PREVTIHdpdGggTlVNQQ0KDQphcmNoL3Bvd2VycGMvS2NvbmZpZzo2Mzc6 Y29uZmlnIE5PREVTX1NQQU5fT1RIRVJfTk9ERVMNCmFyY2gvc3BhcmMvS2NvbmZpZzoyOTk6Y29u ZmlnIE5PREVTX1NQQU5fT1RIRVJfTk9ERVMNCmFyY2gveDg2L0tjb25maWc6MTU3NTpjb25maWcg Tk9ERVNfU1BBTl9PVEhFUl9OT0RFUw0KYXJjaC9zMzkwL0tjb25maWc6NDQ2OmNvbmZpZyBOT0RF U19TUEFOX09USEVSX05PREVTDQphcmNoL2FybTY0ICh3aGljaCBJIGludGVuZGVkIHRvIGVuYWJs ZSBpbiB0aGUgb3JpZ2luYWwgcGF0Y2gpDQoNCkl0IHdvdWxkIGJlIGdvb2QgaWYgd2UgY2FuIGVu YWJsZSBpdCBieS1kZWZhdWx0LiBPdGhlcndpc2UsIGxldCBhcmNoIA0KZW5hYmxlcyBpdCBieSB0 aGVtLXNlbGYuIERvIHlvdSBoYXZlIGFueSBzdWdnZXN0aW9ucz8NCg0KVGhhbmtzDQpIb2FuDQoN Cg0K

#11 lore raw

On Fri 12-07-19 10:56:47, Hoan Tran OS wrote:
[...]

Quoted 2 lines

It would be good if we can enable it by-default. Otherwise, let arch enables it by them-self. Do you have any suggestions?

I can hardly make any suggestions when it is not really clear _why_ you want to remove this config option in the first place. Please explain what motivated you to make this change.

Signature · 2 lines
Michal Hocko
SUSE Labs
#12 lore raw

Hi all,

Quoted 8 lines · Michal Hocko

On Fri 12-07-19 10:56:47, Hoan Tran OS wrote:
[...]

Quoted 2 lines

It would be good if we can enable it by-default. Otherwise, let arch enables it by them-self. Do you have any suggestions?

I can hardly make any suggestions when it is not really clear _why_ you want to remove this config option in the first place. Please explain what motivated you to make this change.

Sorry, I think this confusion might actually be my fault and Hoan has just been implementing my vague suggestion here:

https://lore.kernel.org/linux-arm-kernel/20190625101245.s4vxfosoop52gl4e@willie-the-truck/

If the preference of the mm folks is to leave CONFIG_NODES_SPAN_OTHER_NODES as it is, then we can define it for arm64. I just find it a bit weird that the majority of NUMA-capable architectures have to add a symbol in the arch Kconfig file, for what appears to be a performance optimisation applicable only to ia64, mips and sh.

At the very least we could make the thing selectable.

Will

#13 lore raw
Quoted 17 lines · Will Deacon

Hi all,

Quoted 8 lines · Michal Hocko

On Fri 12-07-19 10:56:47, Hoan Tran OS wrote:
[...]

Quoted 2 lines

It would be good if we can enable it by-default. Otherwise, let arch enables it by them-self. Do you have any suggestions?

I can hardly make any suggestions when it is not really clear _why_ you want to remove this config option in the first place. Please explain what motivated you to make this change.

Sorry, I think this confusion might actually be my fault and Hoan has just been implementing my vague suggestion here:

https://lore.kernel.org/linux-arm-kernel/20190625101245.s4vxfosoop52gl4e@willie-the-truck/

Quoted 7 lines

If the preference of the mm folks is to leave CONFIG_NODES_SPAN_OTHER_NODES as it is, then we can define it for arm64. I just find it a bit weird that the majority of NUMA-capable architectures have to add a symbol in the arch Kconfig file, for what appears to be a performance optimisation applicable only to ia64, mips and sh.

At the very least we could make the thing selectable.

Hmm, I thought this was selectable. But I am obviously wrong here. Looking more closely, it seems that this is indeed only about __early_pfn_to_nid and as such not something that should add a config symbol. This should have been called out in the changelog though.

Also while at it, does HAVE_MEMBLOCK_NODE_MAP fall into a similar bucket? Do we have any NUMA architecture that doesn't enable it?

Thanks!

Signature · 2 lines
Michal Hocko
SUSE Labs
#14 lore raw

SGksDQoNCk9uIDcvMTIvMTkgMTA6MDAgUE0sIE1pY2hhbCBIb2NrbyB3cm90ZToNCj4gT24gRnJp IDEyLTA3LTE5IDE1OjM3OjMwLCBXaWxsIERlYWNvbiB3cm90ZToNCj4+IEhpIGFsbCwNCj4+DQo+ PiBPbiBGcmksIEp1bCAxMiwgMjAxOSBhdCAwMjoxMjoyM1BNICswMjAwLCBNaWNoYWwgSG9ja28g d3JvdGU6DQo+Pj4gT24gRnJpIDEyLTA3LTE5IDEwOjU2OjQ3LCBIb2FuIFRyYW4gT1Mgd3JvdGU6 DQo+Pj4gWy4uLl0NCj4+Pj4gSXQgd291bGQgYmUgZ29vZCBpZiB3ZSBjYW4gZW5hYmxlIGl0IGJ5 LWRlZmF1bHQuIE90aGVyd2lzZSwgbGV0IGFyY2gNCj4+Pj4gZW5hYmxlcyBpdCBieSB0aGVtLXNl bGYuIERvIHlvdSBoYXZlIGFueSBzdWdnZXN0aW9ucz8NCj4+Pg0KPj4+IEkgY2FuIGhhcmRseSBt YWtlIGFueSBzdWdnZXN0aW9ucyB3aGVuIGl0IGlzIG5vdCByZWFsbHkgY2xlYXIgX3doeV8geW91 DQo+Pj4gd2FudCB0byByZW1vdmUgdGhpcyBjb25maWcgb3B0aW9uIGluIHRoZSBmaXJzdCBwbGFj ZS4gUGxlYXNlIGV4cGxhaW4NCj4+PiB3aGF0IG1vdGl2YXRlZCB5b3UgdG8gbWFrZSB0aGlzIGNo YW5nZS4NCj4+DQo+PiBTb3JyeSwgSSB0aGluayB0aGlzIGNvbmZ1c2lvbiBtaWdodCBhY3R1YWxs eSBiZSBteSBmYXVsdCBhbmQgSG9hbiBoYXMganVzdA0KPj4gYmVlbiBpbXBsZW1lbnRpbmcgbXkg dmFndWUgc3VnZ2VzdGlvbiBoZXJlOg0KPj4NCj4+IGh0dHBzOi8vbG9yZS5rZXJuZWwub3JnL2xp bnV4LWFybS1rZXJuZWwvMjAxOTA2MjUxMDEyNDUuczR2eGZvc29vcDUyZ2w0ZUB3aWxsaWUtdGhl LXRydWNrLw0KPj4NCj4+IElmIHRoZSBwcmVmZXJlbmNlIG9mIHRoZSBtbSBmb2xrcyBpcyB0byBs ZWF2ZSBDT05GSUdfTk9ERVNfU1BBTl9PVEhFUl9OT0RFUw0KPj4gYXMgaXQgaXMsIHRoZW4gd2Ug Y2FuIGRlZmluZSBpdCBmb3IgYXJtNjQuIEkganVzdCBmaW5kIGl0IGEgYml0IHdlaXJkIHRoYXQN Cj4+IHRoZSBtYWpvcml0eSBvZiBOVU1BLWNhcGFibGUgYXJjaGl0ZWN0dXJlcyBoYXZlIHRvIGFk ZCBhIHN5bWJvbCBpbiB0aGUgYXJjaA0KPj4gS2NvbmZpZyBmaWxlLCBmb3Igd2hhdCBhcHBlYXJz IHRvIGJlIGEgcGVyZm9ybWFuY2Ugb3B0aW1pc2F0aW9uIGFwcGxpY2FibGUNCj4+IG9ubHkgdG8g aWE2NCwgbWlwcyBhbmQgc2guDQo+Pg0KPj4gQXQgdGhlIHZlcnkgbGVhc3Qgd2UgY291bGQgbWFr ZSB0aGUgdGhpbmcgc2VsZWN0YWJsZS4NCj4gDQo+IEhtbSwgSSB0aG91Z2h0IHRoaXMgd2FzIHNl bGVjdGFibGUuIEJ1dCBJIGFtIG9idmlvdXNseSB3cm9uZyBoZXJlLg0KPiBMb29raW5nIG1vcmUg Y2xvc2VseSwgaXQgc2VlbXMgdGhhdCB0aGlzIGlzIGluZGVlZCBvbmx5IGFib3V0DQo+IF9fZWFy bHlfcGZuX3RvX25pZCBhbmQgYXMgc3VjaCBub3Qgc29tZXRoaW5nIHRoYXQgc2hvdWxkIGFkZCBh IGNvbmZpZw0KPiBzeW1ib2wuIFRoaXMgc2hvdWxkIGhhdmUgYmVlbiBjYWxsZWQgb3V0IGluIHRo ZSBjaGFuZ2Vsb2cgdGhvdWdoLg0KDQpZZXMsIGRvIHlvdSBoYXZlIGFueSBvdGhlciBjb21tZW50 cyBhYm91dCBteSBwYXRjaD8NCg0KPiANCj4gQWxzbyB3aGlsZSBhdCBpdCwgZG9lcyBIQVZFX01F TUJMT0NLX05PREVfTUFQIGZhbGwgaW50byBhIHNpbWlsYXINCj4gYnVja2V0PyBEbyB3ZSBoYXZl IGFueSBOVU1BIGFyY2hpdGVjdHVyZSB0aGF0IGRvZXNuJ3QgZW5hYmxlIGl0Pw0KPiANCg0KQXMg SSBjaGVja2VkIHdpdGggYXJjaCBLY29uZmlnIGZpbGVzLCB0aGVyZSBhcmUgMiBhcmNoaXRlY3R1 cmVzLCByaXNjdiANCmFuZCBtaWNyb2JsYXplLCBkbyBub3Qgc3VwcG9ydCBOVU1BIGJ1dCBlbmFi bGUgdGhpcyBjb25maWcuDQoNCkFuZCAxIGFyY2hpdGVjdHVyZSwgYWxwaGEsIHN1cHBvcnRzIE5V TUEgYnV0IGRvZXMgbm90IGVuYWJsZSB0aGlzIGNvbmZpZy4NCg0KVGhhbmtzIGFuZCBSZWdhcmRz DQpIb2FuDQoNCj4gVGhhbmtzIQ0KPiANCg=

#15 lore raw

[Sorry for a late reply]

Quoted 3 lines · Hoan Tran OS

Hi,

On 7/12/19 10:00 PM, Michal Hocko wrote:

[...]

Quoted 6 lines
Quoted 4 lines

Hmm, I thought this was selectable. But I am obviously wrong here. Looking more closely, it seems that this is indeed only about __early_pfn_to_nid and as such not something that should add a config symbol. This should have been called out in the changelog though.

Yes, do you have any other comments about my patch?

Not really. Just make sure to explicitly state that CONFIG_NODES_SPAN_OTHER_NODES is only about __early_pfn_to_nid and that doesn't really deserve it's own config and can be pulled under NUMA.

Quoted 8 lines
Quoted 3 lines

Also while at it, does HAVE_MEMBLOCK_NODE_MAP fall into a similar bucket? Do we have any NUMA architecture that doesn't enable it?

As I checked with arch Kconfig files, there are 2 architectures, riscv and microblaze, do not support NUMA but enable this config.

And 1 architecture, alpha, supports NUMA but does not enable this config.

Care to have a look and clean this up please?

Signature · 2 lines
Michal Hocko
SUSE Labs
#16 lore raw

SGksDQoNCk9uIDcvMzAvMTkgMToxNCBBTSwgTWljaGFsIEhvY2tvIHdyb3RlOg0KPiBbU29ycnkg Zm9yIGEgbGF0ZSByZXBseV0NCj4gDQo+IE9uIE1vbiAxNS0wNy0xOSAxNzo1NTowNywgSG9hbiBU cmFuIE9TIHdyb3RlOg0KPj4gSGksDQo+Pg0KPj4gT24gNy8xMi8xOSAxMDowMCBQTSwgTWljaGFs IEhvY2tvIHdyb3RlOg0KPiBbLi4uXQ0KPj4+IEhtbSwgSSB0aG91Z2h0IHRoaXMgd2FzIHNlbGVj dGFibGUuIEJ1dCBJIGFtIG9idmlvdXNseSB3cm9uZyBoZXJlLg0KPj4+IExvb2tpbmcgbW9yZSBj bG9zZWx5LCBpdCBzZWVtcyB0aGF0IHRoaXMgaXMgaW5kZWVkIG9ubHkgYWJvdXQNCj4+PiBfX2Vh cmx5X3Bmbl90b19uaWQgYW5kIGFzIHN1Y2ggbm90IHNvbWV0aGluZyB0aGF0IHNob3VsZCBhZGQg YSBjb25maWcNCj4+PiBzeW1ib2wuIFRoaXMgc2hvdWxkIGhhdmUgYmVlbiBjYWxsZWQgb3V0IGlu IHRoZSBjaGFuZ2Vsb2cgdGhvdWdoLg0KPj4NCj4+IFllcywgZG8geW91IGhhdmUgYW55IG90aGVy IGNvbW1lbnRzIGFib3V0IG15IHBhdGNoPw0KPiANCj4gTm90IHJlYWxseS4gSnVzdCBtYWtlIHN1 cmUgdG8gZXhwbGljaXRseSBzdGF0ZSB0aGF0DQo+IENPTkZJR19OT0RFU19TUEFOX09USEVSX05P REVTIGlzIG9ubHkgYWJvdXQgX19lYXJseV9wZm5fdG9fbmlkIGFuZCB0aGF0DQo+IGRvZXNuJ3Qg cmVhbGx5IGRlc2VydmUgaXQncyBvd24gY29uZmlnIGFuZCBjYW4gYmUgcHVsbGVkIHVuZGVyIE5V TUEuDQoNClllcywgSSB3aWxsIGFkZCB0aGlzIGluZm8gaW50byB0aGUgcGF0Y2ggZGVzY3JpcHRp b24uDQoNCj4gDQo+Pj4gQWxzbyB3aGlsZSBhdCBpdCwgZG9lcyBIQVZFX01FTUJMT0NLX05PREVf TUFQIGZhbGwgaW50byBhIHNpbWlsYXINCj4+PiBidWNrZXQ/IERvIHdlIGhhdmUgYW55IE5VTUEg YXJjaGl0ZWN0dXJlIHRoYXQgZG9lc24ndCBlbmFibGUgaXQ/DQo+Pj4NCj4+DQo+PiBBcyBJIGNo ZWNrZWQgd2l0aCBhcmNoIEtjb25maWcgZmlsZXMsIHRoZXJlIGFyZSAyIGFyY2hpdGVjdHVyZXMs IHJpc2N2DQo+PiBhbmQgbWljcm9ibGF6ZSwgZG8gbm90IHN1cHBvcnQgTlVNQSBidXQgZW5hYmxl IHRoaXMgY29uZmlnLg0KPj4NCj4+IEFuZCAxIGFyY2hpdGVjdHVyZSwgYWxwaGEsIHN1cHBvcnRz IE5VTUEgYnV0IGRvZXMgbm90IGVuYWJsZSB0aGlzIGNvbmZpZy4NCj4gDQo+IENhcmUgdG8gaGF2 ZSBhIGxvb2sgYW5kIGNsZWFuIHRoaXMgdXAgcGxlYXNlPw0KDQpTdXJlLCBJJ2xsIHRha2UgYSBs b29rLg0KDQpUaGFua3MNCkhvYW4NCj4gDQoNCg0K

#17 lore raw
In reply to Michal Hocko

[ sorry for a late reply too, somehow I missed this thread before ]

Quoted 12 lines · Michal Hocko

[Sorry for a late reply]

Quoted 3 lines · Hoan Tran OS

Hi,

On 7/12/19 10:00 PM, Michal Hocko wrote:

[...]

Quoted 5 lines
Quoted 4 lines

Hmm, I thought this was selectable. But I am obviously wrong here. Looking more closely, it seems that this is indeed only about __early_pfn_to_nid and as such not something that should add a config symbol. This should have been called out in the changelog though.

Quoted 9 lines
Quoted 1 line

Yes, do you have any other comments about my patch?

Not really. Just make sure to explicitly state that CONFIG_NODES_SPAN_OTHER_NODES is only about __early_pfn_to_nid and that doesn't really deserve it's own config and can be pulled under NUMA.

Quoted 3 lines
Quoted 3 lines

Also while at it, does HAVE_MEMBLOCK_NODE_MAP fall into a similar bucket? Do we have any NUMA architecture that doesn't enable it?

HAVE_MEMBLOCK_NODE_MAP makes huge difference in node/zone initialization sequence so it's not only about a singe function.

Quoted 2 lines
Quoted 2 lines

As I checked with arch Kconfig files, there are 2 architectures, riscv and microblaze, do not support NUMA but enable this config.

My take would be that riscv will support NUMA some day.

Quoted 1 line
Quoted 1 line

And 1 architecture, alpha, supports NUMA but does not enable this config.

alpha's NUMA support is BROKEN for more than a decade now, I doubt it'll ever get fixed.

Quoted 6 lines

Care to have a look and clean this up please?

Signature · 2 lines
Michal Hocko
SUSE Labs
Signature · 2 lines
Sincerely yours,
Mike.
#18 lore raw
Quoted 21 lines · Mike Rapoport

[ sorry for a late reply too, somehow I missed this thread before ]

Quoted 18 lines · Michal Hocko

[Sorry for a late reply]

Quoted 3 lines · Hoan Tran OS

Hi,

On 7/12/19 10:00 PM, Michal Hocko wrote:

[...]

Quoted 6 lines
Quoted 4 lines

Hmm, I thought this was selectable. But I am obviously wrong here. Looking more closely, it seems that this is indeed only about __early_pfn_to_nid and as such not something that should add a config symbol. This should have been called out in the changelog though.

Yes, do you have any other comments about my patch?

Not really. Just make sure to explicitly state that CONFIG_NODES_SPAN_OTHER_NODES is only about __early_pfn_to_nid and that doesn't really deserve it's own config and can be pulled under NUMA.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Also while at it, does HAVE_MEMBLOCK_NODE_MAP fall into a similar bucket? Do we have any NUMA architecture that doesn't enable it?

HAVE_MEMBLOCK_NODE_MAP makes huge difference in node/zone initialization sequence so it's not only about a singe function.

The question is whether we want to have this a config option or enable it unconditionally for each NUMA system.

Quoted 9 lines
Quoted 2 lines
Quoted 2 lines

As I checked with arch Kconfig files, there are 2 architectures, riscv and microblaze, do not support NUMA but enable this config.

My take would be that riscv will support NUMA some day.

Quoted 1 line
Quoted 1 line

And 1 architecture, alpha, supports NUMA but does not enable this config.

alpha's NUMA support is BROKEN for more than a decade now, I doubt it'll ever get fixed.

I can see Al has marked it BROKEN in 2005. Maybe time to rip it out? Although it doesn't seem to be a lot of code in arch/alpha at first glance so maybe not worth an effort.

Signature · 2 lines
Michal Hocko
SUSE Labs
#19 lore raw
Quoted 26 lines · Michal Hocko
Quoted 25 lines · Mike Rapoport

[ sorry for a late reply too, somehow I missed this thread before ]

Quoted 21 lines · Michal Hocko

[Sorry for a late reply]

Quoted 3 lines · Hoan Tran OS

Hi,

On 7/12/19 10:00 PM, Michal Hocko wrote:

[...]

Quoted 6 lines
Quoted 4 lines

Hmm, I thought this was selectable. But I am obviously wrong here. Looking more closely, it seems that this is indeed only about __early_pfn_to_nid and as such not something that should add a config symbol. This should have been called out in the changelog though.

Yes, do you have any other comments about my patch?

Not really. Just make sure to explicitly state that CONFIG_NODES_SPAN_OTHER_NODES is only about __early_pfn_to_nid and that doesn't really deserve it's own config and can be pulled under NUMA.

Quoted 3 lines
Quoted 3 lines

Also while at it, does HAVE_MEMBLOCK_NODE_MAP fall into a similar bucket? Do we have any NUMA architecture that doesn't enable it?

Quoted 5 lines
Quoted 2 lines

HAVE_MEMBLOCK_NODE_MAP makes huge difference in node/zone initialization sequence so it's not only about a singe function.

The question is whether we want to have this a config option or enable it unconditionally for each NUMA system.

We can make it 'default NUMA', but we can't drop it completely because microblaze uses sparse_memory_present_with_active_regions() which is unavailable when HAVE_MEMBLOCK_NODE_MAP=n.

Quoted 17 lines
Quoted 9 lines
Quoted 2 lines
Quoted 2 lines

As I checked with arch Kconfig files, there are 2 architectures, riscv and microblaze, do not support NUMA but enable this config.

My take would be that riscv will support NUMA some day.

Quoted 1 line
Quoted 1 line

And 1 architecture, alpha, supports NUMA but does not enable this config.

alpha's NUMA support is BROKEN for more than a decade now, I doubt it'll ever get fixed.

I can see Al has marked it BROKEN in 2005. Maybe time to rip it out? Although it doesn't seem to be a lot of code in arch/alpha at first glance so maybe not worth an effort.

Signature · 2 lines
Michal Hocko
SUSE Labs
Signature · 2 lines
Sincerely yours,
Mike.
#20 lore raw
Quoted 30 lines · Mike Rapoport
Quoted 29 lines · Michal Hocko
Quoted 27 lines · Mike Rapoport

[ sorry for a late reply too, somehow I missed this thread before ]

Quoted 12 lines · Michal Hocko

[Sorry for a late reply]

Quoted 3 lines · Hoan Tran OS

Hi,

On 7/12/19 10:00 PM, Michal Hocko wrote:

[...]

Quoted 5 lines
Quoted 4 lines

Hmm, I thought this was selectable. But I am obviously wrong here. Looking more closely, it seems that this is indeed only about __early_pfn_to_nid and as such not something that should add a config symbol. This should have been called out in the changelog though.

Quoted 9 lines
Quoted 1 line

Yes, do you have any other comments about my patch?

Not really. Just make sure to explicitly state that CONFIG_NODES_SPAN_OTHER_NODES is only about __early_pfn_to_nid and that doesn't really deserve it's own config and can be pulled under NUMA.

Quoted 3 lines
Quoted 3 lines

Also while at it, does HAVE_MEMBLOCK_NODE_MAP fall into a similar bucket? Do we have any NUMA architecture that doesn't enable it?

HAVE_MEMBLOCK_NODE_MAP makes huge difference in node/zone initialization sequence so it's not only about a singe function.

Quoted 6 lines
Quoted 2 lines

The question is whether we want to have this a config option or enable it unconditionally for each NUMA system.

We can make it 'default NUMA', but we can't drop it completely because microblaze uses sparse_memory_present_with_active_regions() which is unavailable when HAVE_MEMBLOCK_NODE_MAP=n.

I suppose you mean that microblaze is using sparse_memory_present_with_active_regions even without CONFIG_NUMA, right? I have to confess I do not understand that code. What is the deal with setting node id there?

Signature · 2 lines
Michal Hocko
SUSE Labs
#21 lore raw
Quoted 34 lines · Michal Hocko
Quoted 33 lines · Mike Rapoport
Quoted 31 lines · Michal Hocko
Quoted 21 lines · Mike Rapoport

[ sorry for a late reply too, somehow I missed this thread before ]

Quoted 18 lines · Michal Hocko

[Sorry for a late reply]

Quoted 3 lines · Hoan Tran OS

Hi,

On 7/12/19 10:00 PM, Michal Hocko wrote:

[...]

Quoted 6 lines
Quoted 4 lines

Hmm, I thought this was selectable. But I am obviously wrong here. Looking more closely, it seems that this is indeed only about __early_pfn_to_nid and as such not something that should add a config symbol. This should have been called out in the changelog though.

Yes, do you have any other comments about my patch?

Not really. Just make sure to explicitly state that CONFIG_NODES_SPAN_OTHER_NODES is only about __early_pfn_to_nid and that doesn't really deserve it's own config and can be pulled under NUMA.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Also while at it, does HAVE_MEMBLOCK_NODE_MAP fall into a similar bucket? Do we have any NUMA architecture that doesn't enable it?

HAVE_MEMBLOCK_NODE_MAP makes huge difference in node/zone initialization sequence so it's not only about a singe function.

The question is whether we want to have this a config option or enable it unconditionally for each NUMA system.

Quoted 7 lines
Quoted 3 lines

We can make it 'default NUMA', but we can't drop it completely because microblaze uses sparse_memory_present_with_active_regions() which is unavailable when HAVE_MEMBLOCK_NODE_MAP=n.

I suppose you mean that microblaze is using sparse_memory_present_with_active_regions even without CONFIG_NUMA, right?

Yes.

Quoted 2 lines

I have to confess I do not understand that code. What is the deal with setting node id there?

The sparse_memory_present_with_active_regions() iterates over memblock.memory regions and uses the node id of each region as the parameter to memory_present(). The assumption here is that sometime before each region was assigned a proper non-negative node id.

microblaze uses device tree for memory enumeration and the current FDT code does memblock_add() that implicitly sets nid in memblock.memory regions to -1.

So in order to have proper node id passed to memory_present() microblaze has to call memblock_set_node() before it can use sparse_memory_present_with_active_regions().

Quoted 3 lines
Signature · 2 lines
Michal Hocko
SUSE Labs
Signature · 2 lines
Sincerely yours,
Mike.
#22 lore raw

microblaze HAVE_MEMBLOCK_NODE_MAP dependency (was Re: [PATCH v2 0/5] mm: Enable CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA)

Quoted 53 lines · Mike Rapoport
Quoted 34 lines · Michal Hocko
Quoted 33 lines · Mike Rapoport
Quoted 31 lines · Michal Hocko
Quoted 21 lines · Mike Rapoport

[ sorry for a late reply too, somehow I missed this thread before ]

Quoted 18 lines · Michal Hocko

[Sorry for a late reply]

Quoted 3 lines · Hoan Tran OS

Hi,

On 7/12/19 10:00 PM, Michal Hocko wrote:

[...]

Quoted 6 lines
Quoted 4 lines

Hmm, I thought this was selectable. But I am obviously wrong here. Looking more closely, it seems that this is indeed only about __early_pfn_to_nid and as such not something that should add a config symbol. This should have been called out in the changelog though.

Yes, do you have any other comments about my patch?

Not really. Just make sure to explicitly state that CONFIG_NODES_SPAN_OTHER_NODES is only about __early_pfn_to_nid and that doesn't really deserve it's own config and can be pulled under NUMA.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Also while at it, does HAVE_MEMBLOCK_NODE_MAP fall into a similar bucket? Do we have any NUMA architecture that doesn't enable it?

HAVE_MEMBLOCK_NODE_MAP makes huge difference in node/zone initialization sequence so it's not only about a singe function.

The question is whether we want to have this a config option or enable it unconditionally for each NUMA system.

Quoted 7 lines
Quoted 3 lines

We can make it 'default NUMA', but we can't drop it completely because microblaze uses sparse_memory_present_with_active_regions() which is unavailable when HAVE_MEMBLOCK_NODE_MAP=n.

I suppose you mean that microblaze is using sparse_memory_present_with_active_regions even without CONFIG_NUMA, right?

Yes.

Quoted 2 lines

I have to confess I do not understand that code. What is the deal with setting node id there?

The sparse_memory_present_with_active_regions() iterates over memblock.memory regions and uses the node id of each region as the parameter to memory_present(). The assumption here is that sometime before each region was assigned a proper non-negative node id.

Quoted 6 lines

microblaze uses device tree for memory enumeration and the current FDT code does memblock_add() that implicitly sets nid in memblock.memory regions to -1.

So in order to have proper node id passed to memory_present() microblaze has to call memblock_set_node() before it can use sparse_memory_present_with_active_regions().

I am sorry, but I still do not follow. Who is consuming that node id information when NUMA=n. In other words why cannot we simply do

arch/microblaze/mm/init.c +1 -6
index a015a951c8b7..3a47e8db8d1c 100644
@@ -175,14 +175,9 @@ void __init setup_memory(void)
 
 		start_pfn = memblock_region_memory_base_pfn(reg);
 		end_pfn = memblock_region_memory_end_pfn(reg);
-		memblock_set_node(start_pfn << PAGE_SHIFT,
-				  (end_pfn - start_pfn) << PAGE_SHIFT,
-				  &memblock.memory, 0);
+		memory_present(0, start_pfn << PAGE_SHIFT, end_pfn << PAGE_SHIFT);
 	}
 
-	/* XXX need to clip this if using highmem? */
-	sparse_memory_present_with_active_regions(0);
-
 	paging_init();
 }
 
Signature · 2 lines
Michal Hocko
SUSE Labs
#23 lore raw

Re: microblaze HAVE_MEMBLOCK_NODE_MAP dependency (was Re: [PATCH v2 0/5] mm: Enable CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA)

Quoted 57 lines · Michal Hocko
Quoted 56 lines · Mike Rapoport
Quoted 34 lines · Michal Hocko
Quoted 33 lines · Mike Rapoport
Quoted 31 lines · Michal Hocko
Quoted 21 lines · Mike Rapoport

[ sorry for a late reply too, somehow I missed this thread before ]

Quoted 18 lines · Michal Hocko

[Sorry for a late reply]

Quoted 3 lines · Hoan Tran OS

Hi,

On 7/12/19 10:00 PM, Michal Hocko wrote:

[...]

Quoted 6 lines
Quoted 4 lines

Hmm, I thought this was selectable. But I am obviously wrong here. Looking more closely, it seems that this is indeed only about __early_pfn_to_nid and as such not something that should add a config symbol. This should have been called out in the changelog though.

Yes, do you have any other comments about my patch?

Not really. Just make sure to explicitly state that CONFIG_NODES_SPAN_OTHER_NODES is only about __early_pfn_to_nid and that doesn't really deserve it's own config and can be pulled under NUMA.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Also while at it, does HAVE_MEMBLOCK_NODE_MAP fall into a similar bucket? Do we have any NUMA architecture that doesn't enable it?

HAVE_MEMBLOCK_NODE_MAP makes huge difference in node/zone initialization sequence so it's not only about a singe function.

The question is whether we want to have this a config option or enable it unconditionally for each NUMA system.

Quoted 7 lines
Quoted 3 lines

We can make it 'default NUMA', but we can't drop it completely because microblaze uses sparse_memory_present_with_active_regions() which is unavailable when HAVE_MEMBLOCK_NODE_MAP=n.

I suppose you mean that microblaze is using sparse_memory_present_with_active_regions even without CONFIG_NUMA, right?

Yes.

Quoted 2 lines

I have to confess I do not understand that code. What is the deal with setting node id there?

The sparse_memory_present_with_active_regions() iterates over memblock.memory regions and uses the node id of each region as the parameter to memory_present(). The assumption here is that sometime before each region was assigned a proper non-negative node id.

microblaze uses device tree for memory enumeration and the current FDT code does memblock_add() that implicitly sets nid in memblock.memory regions to -1.

Quoted 6 lines
Quoted 3 lines

So in order to have proper node id passed to memory_present() microblaze has to call memblock_set_node() before it can use sparse_memory_present_with_active_regions().

I am sorry, but I still do not follow. Who is consuming that node id information when NUMA=n. In other words why cannot we simply do

We can, I think nobody cared to change it.

arch/microblaze/mm/init.c void __init setup_memory(void) · line 178

Quoted 12 lines
arch/microblaze/mm/init.c +1 -3
index a015a951c8b7..3a47e8db8d1c 100644
@@ -175,14 +175,9 @@ void __init setup_memory(void)
 
 		start_pfn = memblock_region_memory_base_pfn(reg);
 		end_pfn = memblock_region_memory_end_pfn(reg);
-		memblock_set_node(start_pfn << PAGE_SHIFT,
-				  (end_pfn - start_pfn) << PAGE_SHIFT,
-				  &memblock.memory, 0);
+		memory_present(0, start_pfn << PAGE_SHIFT, end_pfn << PAGE_SHIFT);

memory_present() expects pfns, the shift is not needed.

Quoted 11 lines
arch/microblaze/mm/init.c +0 -3
 	}
 
-	/* XXX need to clip this if using highmem? */
-	sparse_memory_present_with_active_regions(0);
-
 	paging_init();
 }
 
Signature · 2 lines
Michal Hocko
SUSE Labs
Signature · 2 lines
Sincerely yours,
Mike.
#24 lore raw

Re: microblaze HAVE_MEMBLOCK_NODE_MAP dependency (was Re: [PATCH v2 0/5] mm: Enable CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA)

Quoted 58 lines · Mike Rapoport
Quoted 57 lines · Michal Hocko
Quoted 56 lines · Mike Rapoport
Quoted 34 lines · Michal Hocko
Quoted 33 lines · Mike Rapoport
Quoted 31 lines · Michal Hocko
Quoted 21 lines · Mike Rapoport

[ sorry for a late reply too, somehow I missed this thread before ]

Quoted 18 lines · Michal Hocko

[Sorry for a late reply]

Quoted 3 lines · Hoan Tran OS

Hi,

On 7/12/19 10:00 PM, Michal Hocko wrote:

[...]

Quoted 6 lines
Quoted 4 lines

Hmm, I thought this was selectable. But I am obviously wrong here. Looking more closely, it seems that this is indeed only about __early_pfn_to_nid and as such not something that should add a config symbol. This should have been called out in the changelog though.

Yes, do you have any other comments about my patch?

Not really. Just make sure to explicitly state that CONFIG_NODES_SPAN_OTHER_NODES is only about __early_pfn_to_nid and that doesn't really deserve it's own config and can be pulled under NUMA.

Quoted 6 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

Also while at it, does HAVE_MEMBLOCK_NODE_MAP fall into a similar bucket? Do we have any NUMA architecture that doesn't enable it?

HAVE_MEMBLOCK_NODE_MAP makes huge difference in node/zone initialization sequence so it's not only about a singe function.

The question is whether we want to have this a config option or enable it unconditionally for each NUMA system.

Quoted 7 lines
Quoted 3 lines

We can make it 'default NUMA', but we can't drop it completely because microblaze uses sparse_memory_present_with_active_regions() which is unavailable when HAVE_MEMBLOCK_NODE_MAP=n.

I suppose you mean that microblaze is using sparse_memory_present_with_active_regions even without CONFIG_NUMA, right?

Yes.

Quoted 2 lines

I have to confess I do not understand that code. What is the deal with setting node id there?

The sparse_memory_present_with_active_regions() iterates over memblock.memory regions and uses the node id of each region as the parameter to memory_present(). The assumption here is that sometime before each region was assigned a proper non-negative node id.

microblaze uses device tree for memory enumeration and the current FDT code does memblock_add() that implicitly sets nid in memblock.memory regions to -1.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

So in order to have proper node id passed to memory_present() microblaze has to call memblock_set_node() before it can use sparse_memory_present_with_active_regions().

I am sorry, but I still do not follow. Who is consuming that node id information when NUMA=n. In other words why cannot we simply do

We can, I think nobody cared to change it.

It would be great if somebody with the actual HW could try it out. I can throw a patch but I do not even have a cross compiler in my toolbox.

Quoted 7 lines
Quoted 7 lines
arch/microblaze/mm/init.c +0 -0
index a015a951c8b7..3a47e8db8d1c 100644
@@ -175,14 +175,9 @@ void __init setup_memory(void)
 
Quoted 8 lines

arch/microblaze/mm/init.c void __init setup_memory(void) · line 178

Quoted 6 lines
arch/microblaze/mm/init.c +1 -3
 		start_pfn = memblock_region_memory_base_pfn(reg);
 		end_pfn = memblock_region_memory_end_pfn(reg);
-		memblock_set_node(start_pfn << PAGE_SHIFT,
-				  (end_pfn - start_pfn) << PAGE_SHIFT,
-				  &memblock.memory, 0);
+		memory_present(0, start_pfn << PAGE_SHIFT, end_pfn << PAGE_SHIFT);

memory_present() expects pfns, the shift is not needed.

Right.

Signature · 2 lines
Michal Hocko
SUSE Labs
#25 lore raw

Re: microblaze HAVE_MEMBLOCK_NODE_MAP dependency (was Re: [PATCH v2 0/5] mm: Enable CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA)

Quoted 11 lines · Michal Hocko
Quoted 6 lines · Mike Rapoport
Quoted 3 lines · Michal Hocko

I am sorry, but I still do not follow. Who is consuming that node id information when NUMA=n. In other words why cannot we simply do

We can, I think nobody cared to change it.

It would be great if somebody with the actual HW could try it out. I can throw a patch but I do not even have a cross compiler in my toolbox.

Well, it compiles :)

Quoted 21 lines
Quoted 6 lines
Quoted 6 lines
arch/microblaze/mm/init.c +0 -0
index a015a951c8b7..3a47e8db8d1c 100644
@@ -175,14 +175,9 @@ void __init setup_memory(void)
 
Quoted 8 lines

arch/microblaze/mm/init.c void __init setup_memory(void) · line 178

Quoted 6 lines
arch/microblaze/mm/init.c +1 -3
 		start_pfn = memblock_region_memory_base_pfn(reg);
 		end_pfn = memblock_region_memory_end_pfn(reg);
-		memblock_set_node(start_pfn << PAGE_SHIFT,
-				  (end_pfn - start_pfn) << PAGE_SHIFT,
-				  &memblock.memory, 0);
+		memory_present(0, start_pfn << PAGE_SHIFT, end_pfn << PAGE_SHIFT);

memory_present() expects pfns, the shift is not needed.

Right.

Signature · 2 lines
Michal Hocko
SUSE Labs
Signature · 2 lines
Sincerely yours,
Mike.
#26 lore raw

Re: microblaze HAVE_MEMBLOCK_NODE_MAP dependency (was Re: [PATCH v2 0/5] mm: Enable CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA)

Quoted 7 lines · Mike Rapoport
Quoted 6 lines · Michal Hocko
Quoted 5 lines · Mike Rapoport
Quoted 3 lines · Michal Hocko

I am sorry, but I still do not follow. Who is consuming that node id information when NUMA=n. In other words why cannot we simply do

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

We can, I think nobody cared to change it.

It would be great if somebody with the actual HW could try it out. I can throw a patch but I do not even have a cross compiler in my toolbox.

Well, it compiles :)

Adding Michal Simek <[email protected]>.

It's not clear that the MICROBLAZE maintainer is still supporting MICROBLAZE.

Quoted 20 lines
Quoted 20 lines
Quoted 6 lines
Quoted 6 lines
arch/microblaze/mm/init.c +0 -0
index a015a951c8b7..3a47e8db8d1c 100644
@@ -175,14 +175,9 @@ void __init setup_memory(void)
 
Quoted 8 lines

arch/microblaze/mm/init.c void __init setup_memory(void) · line 178

Quoted 6 lines
arch/microblaze/mm/init.c +1 -3
 		start_pfn = memblock_region_memory_base_pfn(reg);
 		end_pfn = memblock_region_memory_end_pfn(reg);
-		memblock_set_node(start_pfn << PAGE_SHIFT,
-				  (end_pfn - start_pfn) << PAGE_SHIFT,
-				  &memblock.memory, 0);
+		memory_present(0, start_pfn << PAGE_SHIFT, end_pfn << PAGE_SHIFT);

memory_present() expects pfns, the shift is not needed.

Right.

Signature · 2 lines
Michal Hocko
SUSE Labs
Signature · 1 line
~Randy
maintainer
#27 lore raw

Re: microblaze HAVE_MEMBLOCK_NODE_MAP dependency (was Re: [PATCH v2 0/5] mm: Enable CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA)

In reply to Mike Rapoport
Quoted 7 lines · Mike Rapoport
Quoted 6 lines · Michal Hocko
Quoted 5 lines · Mike Rapoport
Quoted 3 lines · Michal Hocko

I am sorry, but I still do not follow. Who is consuming that node id information when NUMA=n. In other words why cannot we simply do

Quoted 24 lines
Quoted 5 lines
Quoted 1 line

We can, I think nobody cared to change it.

It would be great if somebody with the actual HW could try it out. I can throw a patch but I do not even have a cross compiler in my toolbox.

Well, it compiles :)

Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
arch/microblaze/mm/init.c +0 -0
index a015a951c8b7..3a47e8db8d1c 100644
@@ -175,14 +175,9 @@ void __init setup_memory(void)
 
Quoted 8 lines

arch/microblaze/mm/init.c void __init setup_memory(void) · line 178

Quoted 6 lines
arch/microblaze/mm/init.c +1 -3
 		start_pfn = memblock_region_memory_base_pfn(reg);
 		end_pfn = memblock_region_memory_end_pfn(reg);
-		memblock_set_node(start_pfn << PAGE_SHIFT,
-				  (end_pfn - start_pfn) << PAGE_SHIFT,
-				  &memblock.memory, 0);
+		memory_present(0, start_pfn << PAGE_SHIFT, end_pfn << PAGE_SHIFT);

memory_present() expects pfns, the shift is not needed.

Right.

Sorry for slow response on this. In general regarding this topic. Microblaze is soft core CPU (now there are hardcore versions too but not running Linux). I believe there could be Numa system with microblaze/microblazes (SMP is not supported in mainline).

This code was added in 2011 which is pretty hard to remember why it was done in this way.

It compiles but not working on HW. Please take a look at log below.

Thanks,
Michal

[    0.000000] Linux version 5.3.0-rc6-00007-g54b01939182f-dirty
(monstr@monstr-desktop3) (gcc version 8.2.0 (crosstool-NG 1.20.0)) #101
Mon Sep 2 15:44:05 CEST 2019
[    0.000000] setup_memory: max_mapnr: 0x40000
[    0.000000] setup_memory: min_low_pfn: 0x80000
[    0.000000] setup_memory: max_low_pfn: 0xb0000
[    0.000000] setup_memory: max_pfn: 0xc0000
[    0.000000] start pfn 0x80000
[    0.000000] end pfn 0xc0000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x00000000b0000000-0x00000000bfffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   1: [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] Could not find start_pfn for node 0
[    0.000000] Initmem setup node 0 [mem
0x0000000000000000-0x0000000000000000]
[    0.000000] earlycon: ns16550a0 at MMIO 0x44a01000 (options '115200n8')
[    0.000000] printk: bootconsole [ns16550a0] enabled
[    0.000000] setup_cpuinfo: initialising
[    0.000000] setup_cpuinfo: Using full CPU PVR support
[    0.000000] wt_msr_noirq
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0
[    0.000000] Built 1 zonelists, mobility grouping off.  Total pages: 0
[    0.000000] Kernel command line: earlycon
[    0.000000] Dentry cache hash table entries: -2147483648 (order: -13,
0 bytes, linear)
[    0.000000] Inode-cache hash table entries: -2147483648 (order: -13,
0 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Oops: kernel access of bad area, sig: 11
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted
5.3.0-rc6-00007-g54b01939182f-dirty #101
[    0.000000]  Registers dump: mode=805B9EA8
[    0.000000]  r1=000065A0, r2=C05B7AE6, r3=00000000, r4=00000000
[    0.000000]  r5=00080000, r6=00080B50, r7=00000000, r8=00000004
[    0.000000]  r9=00000000, r10=0000001F, r11=00000000, r12=00006666
[    0.000000]  r13=4119DCC0, r14=00000000, r15=C05EFF8C, r16=00000000
[    0.000000]  r17=C0604408, r18=FFFC0000, r19=C05B9F6C, r20=BFFEC168
[    0.000000]  r21=BFFEC168, r22=EFFF9AC0, r23=00000001, r24=C0606874
[    0.000000]  r25=BFE6B74C, r26=80000000, r27=00000000, r28=90000040
[    0.000000]  r29=01000000, r30=00000380, r31=C05C02F0, rPC=C0604408
[    0.000000]  msr=000046A0, ear=00000004, esr=00000D12, fsr=FFFFFFFF
[    0.000000] Oops: kernel access of bad area, sig: 11
Signature · 5 lines
Michal Simek, Ing. (M.Eng), OpenPGP -> KeyID: FE3D1F91
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel - Xilinx Microblaze
Maintainer of Linux kernel - Xilinx Zynq ARM and ZynqMP ARM64 SoCs
U-Boot custodian - Xilinx Microblaze/Zynq/ZynqMP/Versal SoCs
#28 lore raw

Re: microblaze HAVE_MEMBLOCK_NODE_MAP dependency (was Re: [PATCH v2 0/5] mm: Enable CONFIG_NODES_SPAN_OTHER_NODES by default for NUMA)

Quoted 65 lines · Michal Simek
Quoted 7 lines · Mike Rapoport
Quoted 6 lines · Michal Hocko
Quoted 5 lines · Mike Rapoport
Quoted 3 lines · Michal Hocko

I am sorry, but I still do not follow. Who is consuming that node id information when NUMA=n. In other words why cannot we simply do

Quoted 24 lines
Quoted 5 lines
Quoted 1 line

We can, I think nobody cared to change it.

It would be great if somebody with the actual HW could try it out. I can throw a patch but I do not even have a cross compiler in my toolbox.

Well, it compiles :)

Quoted 16 lines
Quoted 6 lines
Quoted 6 lines
arch/microblaze/mm/init.c +0 -0
index a015a951c8b7..3a47e8db8d1c 100644
@@ -175,14 +175,9 @@ void __init setup_memory(void)
 
Quoted 8 lines

arch/microblaze/mm/init.c void __init setup_memory(void) · line 178

Quoted 6 lines
arch/microblaze/mm/init.c +1 -3
 		start_pfn = memblock_region_memory_base_pfn(reg);
 		end_pfn = memblock_region_memory_end_pfn(reg);
-		memblock_set_node(start_pfn << PAGE_SHIFT,
-				  (end_pfn - start_pfn) << PAGE_SHIFT,
-				  &memblock.memory, 0);
+		memory_present(0, start_pfn << PAGE_SHIFT, end_pfn << PAGE_SHIFT);

memory_present() expects pfns, the shift is not needed.

Right.

Sorry for slow response on this. In general regarding this topic. Microblaze is soft core CPU (now there are hardcore versions too but not running Linux). I believe there could be Numa system with microblaze/microblazes (SMP is not supported in mainline).

This code was added in 2011 which is pretty hard to remember why it was done in this way.

It compiles but not working on HW. Please take a look at log below.

Thanks,
Michal

[    0.000000] Linux version 5.3.0-rc6-00007-g54b01939182f-dirty
(monstr@monstr-desktop3) (gcc version 8.2.0 (crosstool-NG 1.20.0)) #101
Mon Sep 2 15:44:05 CEST 2019
[    0.000000] setup_memory: max_mapnr: 0x40000
[    0.000000] setup_memory: min_low_pfn: 0x80000
[    0.000000] setup_memory: max_low_pfn: 0xb0000
[    0.000000] setup_memory: max_pfn: 0xc0000
[    0.000000] start pfn 0x80000
[    0.000000] end pfn 0xc0000
[    0.000000] Zone ranges:
[    0.000000]   DMA      [mem 0x0000000080000000-0x00000000afffffff]
[    0.000000]   Normal   empty
[    0.000000]   HighMem  [mem 0x00000000b0000000-0x00000000bfffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   1: [mem 0x0000000080000000-0x00000000bfffffff]
[    0.000000] Could not find start_pfn for node 0
[    0.000000] Initmem setup node 0 [mem
Quoted 1 line

0x0000000000000000-0x0000000000000000]

This does not look good :)

I think the problem is that without an explicit call to memblock_set_node() the ->nid in memblock is MAX_NUMNODES but free_area_init_nodes() presumes actual node ids are properly set.

Quoted 38 lines
[    0.000000] earlycon: ns16550a0 at MMIO 0x44a01000 (options '115200n8')
[    0.000000] printk: bootconsole [ns16550a0] enabled
[    0.000000] setup_cpuinfo: initialising
[    0.000000] setup_cpuinfo: Using full CPU PVR support
[    0.000000] wt_msr_noirq
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0
[    0.000000] Built 1 zonelists, mobility grouping off.  Total pages: 0
[    0.000000] Kernel command line: earlycon
[    0.000000] Dentry cache hash table entries: -2147483648 (order: -13,
0 bytes, linear)
[    0.000000] Inode-cache hash table entries: -2147483648 (order: -13,
0 bytes, linear)
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] Oops: kernel access of bad area, sig: 11
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted
5.3.0-rc6-00007-g54b01939182f-dirty #101
[    0.000000]  Registers dump: mode€5B9EA8
[    0.000000]  r10065A0, r2À5B7AE6, r3000000, r4000000
[    0.000000]  r5080000, r6080B50, r7000000, r8000004
[    0.000000]  r9000000, r1000001F, r11000000, r12006666
[    0.000000]  r13A19DCC0, r14000000, r15À5EFF8C, r16000000
[    0.000000]  r17À604408, r18ÿFC0000, r19À5B9F6C, r20¿FEC168
[    0.000000]  r21¿FEC168, r22ïFF9AC0, r23000001, r24À606874
[    0.000000]  r25¿E6B74C, r26€000000, r27000000, r28�000040
[    0.000000]  r29000000, r30000380, r31À5C02F0, rPCÀ604408
[    0.000000]  msr0046A0, ear000004, esr000D12, fsrÿFFFFFF
[    0.000000] Oops: kernel access of bad area, sig: 11
Signature · 5 lines
Michal Simek, Ing. (M.Eng), OpenPGP -> KeyID: FE3D1F91
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel - Xilinx Microblaze
Maintainer of Linux kernel - Xilinx Zynq ARM and ZynqMP ARM64 SoCs
U-Boot custodian - Xilinx Microblaze/Zynq/ZynqMP/Versal SoCs
Signature · 2 lines
Sincerely yours,
Mike.