mm/memory_hotplug: maintain N_NORMAL_MEMORY during hotplug

PATCH Superseded Hao Li 4 participants 11 messages Mar 27 – Mar 30

Subsystems (2)

Memory management Memory hot(un)plug

lore

Share & tools

lore

Hao Li

author

2026-03-27 12:44 UTC #0 lore raw

PATCH 1/1 mm/memory_hotplug: maintain N_NORMAL_MEMORY during hotplug

N_NORMAL_MEMORY is initialized from zone population at boot, but memory hotplug currently only updates N_MEMORY. As a result, a node that gains normal memory via hotplug can remain invisible to users iterating over N_NORMAL_MEMORY, while a node that loses its last normal memory can stay incorrectly marked as such.

Restore N_NORMAL_MEMORY maintenance directly in online_pages() and offline_pages(). Set the bit when a node that currently lacks normal memory onlines pages into a zone <= ZONE_NORMAL, and clear it when offlining removes the last present pages from zones <= ZONE_NORMAL.

This restores the intended semantics without bringing back the old status_change_nid_normal notifier plumbing which was removed in 8d2882a8edb8.

Current users that benefit include list_lru, zswap, nfsd filecache, hugetlb_cgroup, and has_normal_memory sysfs reporting.

Signed-off-by Hao Li <[email protected]>

---

This patch also prepares for a subsequent SLUB change that makes can_free_to_pcs() rely on N_NORMAL_MEMORY to decide whether an object can be freed to the sheaf.

---
 mm/memory_hotplug.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

mm/memory_hotplug.c +22 -0

index bc805029da51..5498744aa1f1 100644
@@ -1155,6 +1155,7 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 	int need_zonelists_rebuild = 0;
 	unsigned long flags;
 	int ret;
+	bool need_set_normal_memory = false;
 
 	/*
 	 * {on,off}lining is constrained to full memory sections (or more

@@ -1180,6 +1181,9 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 		if (ret)
 			goto failed_addition;
 	}
+	/* Adding normal memory to the node for the first time */
+	if (!node_state(nid, N_NORMAL_MEMORY) && zone_idx(zone) <= ZONE_NORMAL)
+		need_set_normal_memory = true;
 
 	ret = memory_notify(MEM_GOING_ONLINE, &mem_arg);
 	ret = notifier_to_errno(ret);

@@ -1209,6 +1213,8 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 
 	if (node_arg.nid >= 0)
 		node_set_state(nid, N_MEMORY);
+	if (need_set_normal_memory)
+		node_set_state(nid, N_NORMAL_MEMORY);
 	if (need_zonelists_rebuild)
 		build_all_zonelists(NULL);

@@ -1908,6 +1914,9 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 	unsigned long flags;
 	char *reason;
 	int ret;
+	bool need_clear_normal_memory = false;
+	unsigned long node_normal_pages = 0;
+	enum zone_type zt;
 
 	/*
 	 * {on,off}lining is constrained to full memory sections (or more

@@ -1977,6 +1986,13 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 			goto failed_removal_isolated;
 		}
 	}
+	/*
+	 * Check whether this operation removes the node's last normal memory.
+	 */
+	for (zt = 0; zt <= ZONE_NORMAL; zt++)
+		node_normal_pages += pgdat->node_zones[zt].present_pages;
+	if (nr_pages >= node_normal_pages && zone_idx(zone) <= ZONE_NORMAL)
+		need_clear_normal_memory = true;
 
 	ret = memory_notify(MEM_GOING_OFFLINE, &mem_arg);
 	ret = notifier_to_errno(ret);

@@ -2055,6 +2071,12 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 	/* reinitialise watermarks and update pcp limits */
 	init_per_zone_wmark_min();
 
+	/*
+	 * Clear N_NORMAL_MEMORY first to avoid the transient state
+	 * "!N_MEMORY && N_NORMAL_MEMORY".
+	 */
+	if (need_clear_normal_memory)
+		node_clear_state(node, N_NORMAL_MEMORY);
 	/*
 	 * Make sure to mark the node as memory-less before rebuilding the zone
 	 * list. Otherwise this node would still appear in the fallback lists.

Signature · 1 line

2.50.1

Joshua Hahn

2026-03-27 14:38 UTC #1 lore raw

On Fri, 27 Mar 2026 20:42:47 +0800 Hao Li <[email protected]> wrote:

Hello Hao,

I hope you are doing well, thank you for the patch!

Quoted 5 lines

N_NORMAL_MEMORY is initialized from zone population at boot, but memory hotplug currently only updates N_MEMORY. As a result, a node that gains normal memory via hotplug can remain invisible to users iterating over N_NORMAL_MEMORY, while a node that loses its last normal memory can stay incorrectly marked as such.

The second part feels more important than the second part, doing a quick glance through the code I can see a few N_NORMAL_MEMORY iterators that are in some hot paths like shrink_memcg. Iterating over nodes that don't contain any NORMAL memory seems like an inefficiency rather than a bug though.

Quoted 46 lines

Restore N_NORMAL_MEMORY maintenance directly in online_pages() and offline_pages(). Set the bit when a node that currently lacks normal memory onlines pages into a zone <= ZONE_NORMAL, and clear it when offlining removes the last present pages from zones <= ZONE_NORMAL.
This restores the intended semantics without bringing back the old status_change_nid_normal notifier plumbing which was removed in 8d2882a8edb8.
Current users that benefit include list_lru, zswap, nfsd filecache, hugetlb_cgroup, and has_normal_memory sysfs reporting.
Signed-off-by Hao Li <[email protected]>
---
This patch also prepares for a subsequent SLUB change that makes can_free_to_pcs() rely on N_NORMAL_MEMORY to decide whether an object can be freed to the sheaf.
---
 mm/memory_hotplug.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
mm/memory_hotplug.c +4 -0
index bc805029da51..5498744aa1f1 100644
@@ -1155,6 +1155,7 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 	int need_zonelists_rebuild = 0;
 	unsigned long flags;
 	int ret;
+	bool need_set_normal_memory = false;
 
 	/*
 	 * {on,off}lining is constrained to full memory sections (or more
@@ -1180,6 +1181,9 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 		if (ret)
 			goto failed_addition;
 	}
+	/* Adding normal memory to the node for the first time */
+	if (!node_state(nid, N_NORMAL_MEMORY) && zone_idx(zone) <= ZONE_NORMAL)
+		need_set_normal_memory = true;
 
 	ret = memory_notify(MEM_GOING_ONLINE, &mem_arg);
 	ret = notifier_to_errno(ret);

mm/memory_hotplug.c int online_pages(unsigned long pfn, unsigned long nr_pages, · line 1219

Quoted 7 lines

mm/memory_hotplug.c +2 -0

@@ -1209,6 +1213,8 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 
 	if (node_arg.nid >= 0)
 		node_set_state(nid, N_MEMORY);
+	if (need_set_normal_memory)
+		node_set_state(nid, N_NORMAL_MEMORY);
 	if (need_zonelists_rebuild)
 		build_all_zonelists(NULL);

Do we need the flag here? As far as I can tell, we can just skip this and just directly check whether this is the first normal memory we are adding to the node here and set the bit. Then we can remove the flag and the extraneous check. We won't do any notifier work so I think it should be OK.

Quoted 25 lines

mm/memory_hotplug.c +10 -0

@@ -1908,6 +1914,9 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 	unsigned long flags;
 	char *reason;
 	int ret;
+	bool need_clear_normal_memory = false;
+	unsigned long node_normal_pages = 0;
+	enum zone_type zt;
 
 	/*
 	 * {on,off}lining is constrained to full memory sections (or more

@@ -1977,6 +1986,13 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 			goto failed_removal_isolated;
 		}
 	}
+	/*
+	 * Check whether this operation removes the node's last normal memory.
+	 */
+	for (zt = 0; zt <= ZONE_NORMAL; zt++)
+		node_normal_pages += pgdat->node_zones[zt].present_pages;
+	if (nr_pages >= node_normal_pages && zone_idx(zone) <= ZONE_NORMAL)
+		need_clear_normal_memory = true;
 
 	ret = memory_notify(MEM_GOING_OFFLINE, &mem_arg);
 	ret = notifier_to_errno(ret);

mm/memory_hotplug.c int offline_pages(unsigned long start_pfn, unsigned long nr_pages, · line 2072

Quoted 2 lines

mm/memory_hotplug.c +0 -0

@@ -2055,6 +2071,12 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 	/* reinitialise watermarks and update pcp limits */
 	init_per_zone_wmark_min();

Same here, couldn't we just iterate through the paegs here and accumulate node_normal_pages and clear the memory here? We can get rid of the bool and also keep node_normal_pages defined inside an if (zone_idx(zone) <= ZONE_NORMAL) check as well.

Quoted 13 lines

mm/memory_hotplug.c +6 -0

+	/*
+	 * Clear N_NORMAL_MEMORY first to avoid the transient state
+	 * "!N_MEMORY && N_NORMAL_MEMORY".
+	 */
+	if (need_clear_normal_memory)
+		node_clear_state(node, N_NORMAL_MEMORY);
 	/*
 	 * Make sure to mark the node as memory-less before rebuilding the zone
 	 * list. Otherwise this node would still appear in the fallback lists.

Signature · 1 line

2.50.1

Thank you for the patch. The rest looks good to me! Have a great day : -) Joshua

Vlastimil Babka (SUSE)

2026-03-27 14:44 UTC #2 lore raw

Quoted 6 lines · Joshua Hahn

On Fri, 27 Mar 2026 20:42:47 +0800 Hao Li <[email protected]> wrote:
Hello Hao,
I hope you are doing well, thank you for the patch!

Quoted 8 lines

Quoted 5 lines
N_NORMAL_MEMORY is initialized from zone population at boot, but memory hotplug currently only updates N_MEMORY. As a result, a node that gains normal memory via hotplug can remain invisible to users iterating over N_NORMAL_MEMORY, while a node that loses its last normal memory can stay incorrectly marked as such.
The second part feels more important than the second part, doing a quick glance through the code I can see a few N_NORMAL_MEMORY iterators that

Note in practice it's unlikely that a node would hotplug normal memory, start using it, and then manage to successfully hotremove it, due to unmovable allocations. Most likely only ZONE_MOVABLE memory can get hotremoved.

Quoted 3 lines

are in some hot paths like shrink_memcg. Iterating over nodes that don't contain any NORMAL memory seems like an inefficiency rather than a bug though.

Ignoring nodes that have normal memory, just because it was hotplugged, will result also just in some form of inefficiency, or can the consequences be worse?

Quoted 8 lines

Quoted 8 lines
Restore N_NORMAL_MEMORY maintenance directly in online_pages() and offline_pages(). Set the bit when a node that currently lacks normal memory onlines pages into a zone <= ZONE_NORMAL, and clear it when offlining removes the last present pages from zones <= ZONE_NORMAL.
This restores the intended semantics without bringing back the old status_change_nid_normal notifier plumbing which was removed in 8d2882a8edb8.

But commit 8d2882a8edb8 didn't introduce the current state, or did it?

Quoted 99 lines

Quoted 37 lines
Current users that benefit include list_lru, zswap, nfsd filecache, hugetlb_cgroup, and has_normal_memory sysfs reporting.
Signed-off-by Hao Li <[email protected]>
---
This patch also prepares for a subsequent SLUB change that makes can_free_to_pcs() rely on N_NORMAL_MEMORY to decide whether an object can be freed to the sheaf.
---
 mm/memory_hotplug.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
mm/memory_hotplug.c +4 -0
index bc805029da51..5498744aa1f1 100644
@@ -1155,6 +1155,7 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 	int need_zonelists_rebuild = 0;
 	unsigned long flags;
 	int ret;
+	bool need_set_normal_memory = false;
 
 	/*
 	 * {on,off}lining is constrained to full memory sections (or more
@@ -1180,6 +1181,9 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 		if (ret)
 			goto failed_addition;
 	}
+	/* Adding normal memory to the node for the first time */
+	if (!node_state(nid, N_NORMAL_MEMORY) && zone_idx(zone) <= ZONE_NORMAL)
+		need_set_normal_memory = true;
 
 	ret = memory_notify(MEM_GOING_ONLINE, &mem_arg);
 	ret = notifier_to_errno(ret);
mm/memory_hotplug.c int online_pages(unsigned long pfn, unsigned long nr_pages, · line 1219
Quoted 7 lines
mm/memory_hotplug.c +2 -0
@@ -1209,6 +1213,8 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 
 	if (node_arg.nid >= 0)
 		node_set_state(nid, N_MEMORY);
+	if (need_set_normal_memory)
+		node_set_state(nid, N_NORMAL_MEMORY);
 	if (need_zonelists_rebuild)
 		build_all_zonelists(NULL);
Do we need the flag here? As far as I can tell, we can just skip this and just directly check whether this is the first normal memory we are adding to the node here and set the bit. Then we can remove the flag and the extraneous check. We won't do any notifier work so I think it should be OK.
Quoted 25 lines
mm/memory_hotplug.c +10 -0
@@ -1908,6 +1914,9 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 	unsigned long flags;
 	char *reason;
 	int ret;
+	bool need_clear_normal_memory = false;
+	unsigned long node_normal_pages = 0;
+	enum zone_type zt;
 
 	/*
 	 * {on,off}lining is constrained to full memory sections (or more
@@ -1977,6 +1986,13 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 			goto failed_removal_isolated;
 		}
 	}
+	/*
+	 * Check whether this operation removes the node's last normal memory.
+	 */
+	for (zt = 0; zt <= ZONE_NORMAL; zt++)
+		node_normal_pages += pgdat->node_zones[zt].present_pages;
+	if (nr_pages >= node_normal_pages && zone_idx(zone) <= ZONE_NORMAL)
+		need_clear_normal_memory = true;
 
 	ret = memory_notify(MEM_GOING_OFFLINE, &mem_arg);
 	ret = notifier_to_errno(ret);
mm/memory_hotplug.c int offline_pages(unsigned long start_pfn, unsigned long nr_pages, · line 2072
Quoted 2 lines
mm/memory_hotplug.c +0 -0
@@ -2055,6 +2071,12 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 	/* reinitialise watermarks and update pcp limits */
 	init_per_zone_wmark_min();
Same here, couldn't we just iterate through the paegs here and accumulate node_normal_pages and clear the memory here? We can get rid of the bool and also keep node_normal_pages defined inside an if (zone_idx(zone) <= ZONE_NORMAL) check as well.
Quoted 13 lines
mm/memory_hotplug.c +6 -0
+	/*
+	 * Clear N_NORMAL_MEMORY first to avoid the transient state
+	 * "!N_MEMORY && N_NORMAL_MEMORY".
+	 */
+	if (need_clear_normal_memory)
+		node_clear_state(node, N_NORMAL_MEMORY);
 	/*
 	 * Make sure to mark the node as memory-less before rebuilding the zone
 	 * list. Otherwise this node would still appear in the fallback lists.
Signature · 1 line
2.50.1
Thank you for the patch. The rest looks good to me! Have a great day : -) Joshua

David Hildenbrand (Arm)

maintainer

2026-03-27 15:22 UTC #3 lore raw

Quoted 13 lines · Vlastimil Babka (SUSE)

Quoted 12 lines · Joshua Hahn
On Fri, 27 Mar 2026 20:42:47 +0800 Hao Li <[email protected]> wrote:
Hello Hao,
I hope you are doing well, thank you for the patch!
Quoted 5 lines
N_NORMAL_MEMORY is initialized from zone population at boot, but memory hotplug currently only updates N_MEMORY. As a result, a node that gains normal memory via hotplug can remain invisible to users iterating over N_NORMAL_MEMORY, while a node that loses its last normal memory can stay incorrectly marked as such.

Quoted 6 lines

Quoted 2 lines
The second part feels more important than the second part, doing a quick glance through the code I can see a few N_NORMAL_MEMORY iterators that
Note in practice it's unlikely that a node would hotplug normal memory, start using it, and then manage to successfully hotremove it, due to unmovable allocations. Most likely only ZONE_MOVABLE memory can get hotremoved.

Yes, hotunplug of all normal memory of a node is unlikely. Possible, but nothing to optimize for.

Signature · 3 lines

Cheers,

David

Hao Li

author

2026-03-28 04:09 UTC #4 lore raw

Quoted 17 lines · David Hildenbrand (Arm)

Quoted 16 lines · Vlastimil Babka (SUSE)
Quoted 14 lines · Joshua Hahn
On Fri, 27 Mar 2026 20:42:47 +0800 Hao Li <[email protected]> wrote:
Hello Hao,
I hope you are doing well, thank you for the patch!
Quoted 5 lines
N_NORMAL_MEMORY is initialized from zone population at boot, but memory hotplug currently only updates N_MEMORY. As a result, a node that gains normal memory via hotplug can remain invisible to users iterating over N_NORMAL_MEMORY, while a node that loses its last normal memory can stay incorrectly marked as such.
The second part feels more important than the second part, doing a quick glance through the code I can see a few N_NORMAL_MEMORY iterators that

Quoted 6 lines

Quoted 3 lines
Note in practice it's unlikely that a node would hotplug normal memory, start using it, and then manage to successfully hotremove it, due to unmovable allocations. Most likely only ZONE_MOVABLE memory can get hotremoved.
Yes, hotunplug of all normal memory of a node is unlikely. Possible, but nothing to optimize for.

Yes, I agree that fully hot-unplugging the last normal-memory node is probably uncommon.

That said, once we restore the set side, clearing it feels like the symmetric counterpart as well. If we leave the clear side as-is, it seems like it could be a bit confusing for others later on.

Signature · 2 lines

Thanks,
Hao

Joshua Hahn

2026-03-27 16:35 UTC #5 lore raw

In reply to Vlastimil Babka (SUSE)

Quoted 13 lines · On Fri, 27 Mar 2026 15:44:18 +0100 "Vlastimil Babka (SUSE)"…

Quoted 12 lines · Joshua Hahn
On Fri, 27 Mar 2026 20:42:47 +0800 Hao Li <[email protected]> wrote:
Hello Hao,
I hope you are doing well, thank you for the patch!
Quoted 5 lines
N_NORMAL_MEMORY is initialized from zone population at boot, but memory hotplug currently only updates N_MEMORY. As a result, a node that gains normal memory via hotplug can remain invisible to users iterating over N_NORMAL_MEMORY, while a node that loses its last normal memory can stay incorrectly marked as such.

Quoted 6 lines

Quoted 2 lines
The second part feels more important than the second part, doing a quick glance through the code I can see a few N_NORMAL_MEMORY iterators that
Note in practice it's unlikely that a node would hotplug normal memory, start using it, and then manage to successfully hotremove it, due to unmovable allocations. Most likely only ZONE_MOVABLE memory can get hotremoved.

Hello Vlastimil! I hope you are doing well.

Yup, makes sense : -)

Quoted 6 lines

Quoted 3 lines
are in some hot paths like shrink_memcg. Iterating over nodes that don't contain any NORMAL memory seems like an inefficiency rather than a bug though.
Ignoring nodes that have normal memory, just because it was hotplugged, will result also just in some form of inefficiency, or can the consequences be worse?

Aaaahh yeah, seems like ignoring the hotplugged normal memory would be a bigger problem. struct zswap_entries allocated from slab allocator on hotplugged nodes would just get ignored by shrink_memcg, seems like a much bigger issue than iterating through extra nodes :P thanks for calling this out.

Quoted 10 lines

Quoted 8 lines
Quoted 8 lines
Restore N_NORMAL_MEMORY maintenance directly in online_pages() and offline_pages(). Set the bit when a node that currently lacks normal memory onlines pages into a zone <= ZONE_NORMAL, and clear it when offlining removes the last present pages from zones <= ZONE_NORMAL.
This restores the intended semantics without bringing back the old status_change_nid_normal notifier plumbing which was removed in 8d2882a8edb8.
But commit 8d2882a8edb8 didn't introduce the current state, or did it?

I don't mean to speak on Hao's behalf, but as far as I can tell I think this would have been the commit to introduce the state, since node_states_check_changes_online would set status_change_nid_normal to nid, then nodes_states_set_node would come around and set it to N_NORMAL_MEMORY. Maybe I'm missing something? :0

Thanks for the reply, Vlastimil. I hope you have a great day : -) Joshua

Hao Li

author

2026-03-28 04:03 UTC #6 lore raw

Quoted 18 lines · Joshua Hahn

Quoted 16 lines · On Fri, 27 Mar 2026 15:44:18 +0100 "Vlastimil Babka (SUSE)"…
Quoted 14 lines · Joshua Hahn
On Fri, 27 Mar 2026 20:42:47 +0800 Hao Li <[email protected]> wrote:
Hello Hao,
I hope you are doing well, thank you for the patch!
Quoted 5 lines
N_NORMAL_MEMORY is initialized from zone population at boot, but memory hotplug currently only updates N_MEMORY. As a result, a node that gains normal memory via hotplug can remain invisible to users iterating over N_NORMAL_MEMORY, while a node that loses its last normal memory can stay incorrectly marked as such.
The second part feels more important than the second part, doing a quick glance through the code I can see a few N_NORMAL_MEMORY iterators that

Quoted 7 lines

Quoted 3 lines
Note in practice it's unlikely that a node would hotplug normal memory, start using it, and then manage to successfully hotremove it, due to unmovable allocations. Most likely only ZONE_MOVABLE memory can get hotremoved.
Hello Vlastimil! I hope you are doing well.
Yup, makes sense : -)

Yes, indeed.

Quoted 5 lines

Quoted 5 lines
Quoted 4 lines
are in some hot paths like shrink_memcg. Iterating over nodes that don't contain any NORMAL memory seems like an inefficiency rather than a bug though.

Quoted 7 lines

Quoted 2 lines
Ignoring nodes that have normal memory, just because it was hotplugged, will result also just in some form of inefficiency, or can the consequences be worse?
Aaaahh yeah, seems like ignoring the hotplugged normal memory would be a bigger problem. struct zswap_entries allocated from slab allocator on hotplugged nodes would just get ignored by shrink_memcg, seems like a much bigger issue than iterating through extra nodes :P thanks for calling this out.

Yes, make sense to me. It could be more serious than just inefficiency. The shrinkers might even fail to visit reclaimable objects on that node.

Quoted 10 lines

Quoted 10 lines
Quoted 9 lines
Quoted 9 lines
Restore N_NORMAL_MEMORY maintenance directly in online_pages() and offline_pages(). Set the bit when a node that currently lacks normal memory onlines pages into a zone <= ZONE_NORMAL, and clear it when offlining removes the last present pages from zones <= ZONE_NORMAL.
This restores the intended semantics without bringing back the old status_change_nid_normal notifier plumbing which was removed in 8d2882a8edb8.

Quoted 7 lines

Quoted 1 line
But commit 8d2882a8edb8 didn't introduce the current state, or did it?
I don't mean to speak on Hao's behalf, but as far as I can tell I think this would have been the commit to introduce the state, since node_states_check_changes_online would set status_change_nid_normal to nid, then nodes_states_set_node would come around and set it to N_NORMAL_MEMORY. Maybe I'm missing something? :0

Thanks for this analysis. I was thinking the same. N_NORMAL_MEMORY is no longer updated after this commit.

Quoted 3 lines

Thanks for the reply, Vlastimil. I hope you have a great day : -) Joshua

Signature · 2 lines

Thanks,
Hao

Vlastimil Babka (SUSE)

2026-03-30 11:44 UTC #7 lore raw

Quoted 14 lines · Hao Li

Quoted 13 lines · Joshua Hahn
Quoted 10 lines · On Fri, 27 Mar 2026 15:44:18 +0100 "Vlastimil Babka (SUSE)"…
Quoted 8 lines
Quoted 8 lines
Restore N_NORMAL_MEMORY maintenance directly in online_pages() and offline_pages(). Set the bit when a node that currently lacks normal memory onlines pages into a zone <= ZONE_NORMAL, and clear it when offlining removes the last present pages from zones <= ZONE_NORMAL.
This restores the intended semantics without bringing back the old status_change_nid_normal notifier plumbing which was removed in 8d2882a8edb8.
But commit 8d2882a8edb8 didn't introduce the current state, or did it?

Quoted 8 lines

Quoted 5 lines
I don't mean to speak on Hao's behalf, but as far as I can tell I think this would have been the commit to introduce the state, since node_states_check_changes_online would set status_change_nid_normal to nid, then nodes_states_set_node would come around and set it to N_NORMAL_MEMORY. Maybe I'm missing something? :0
Thanks for this analysis. I was thinking the same. N_NORMAL_MEMORY is no longer updated after this commit.

You're right. Initially I found a much older commit that seemed to remove updating N_NORMAL_MEMORY shortly after it was introduced. But seems it was reintroduced later and indeed 8d2882a8edb8 removed it again. Partially my fault as I reviewed that commit and was blind to that aspect - that it's not just removing an unused status_change_nid_normal. Thanks for figuring that out!

Quoted 4 lines

Quoted 3 lines
Thanks for the reply, Vlastimil. I hope you have a great day : -) Joshua

David Hildenbrand (Arm)

maintainer

2026-03-27 15:28 UTC #8 lore raw

In reply to Joshua Hahn

Quoted 65 lines · Joshua Hahn

On Fri, 27 Mar 2026 20:42:47 +0800 Hao Li <[email protected]> wrote:
Hello Hao,
I hope you are doing well, thank you for the patch!
Quoted 5 lines
N_NORMAL_MEMORY is initialized from zone population at boot, but memory hotplug currently only updates N_MEMORY. As a result, a node that gains normal memory via hotplug can remain invisible to users iterating over N_NORMAL_MEMORY, while a node that loses its last normal memory can stay incorrectly marked as such.
The second part feels more important than the second part, doing a quick glance through the code I can see a few N_NORMAL_MEMORY iterators that are in some hot paths like shrink_memcg. Iterating over nodes that don't contain any NORMAL memory seems like an inefficiency rather than a bug though.
Quoted 47 lines
Restore N_NORMAL_MEMORY maintenance directly in online_pages() and offline_pages(). Set the bit when a node that currently lacks normal memory onlines pages into a zone <= ZONE_NORMAL, and clear it when offlining removes the last present pages from zones <= ZONE_NORMAL.
This restores the intended semantics without bringing back the old status_change_nid_normal notifier plumbing which was removed in 8d2882a8edb8.
Current users that benefit include list_lru, zswap, nfsd filecache, hugetlb_cgroup, and has_normal_memory sysfs reporting.
Signed-off-by Hao Li <[email protected]>
---
This patch also prepares for a subsequent SLUB change that makes can_free_to_pcs() rely on N_NORMAL_MEMORY to decide whether an object can be freed to the sheaf.
---
 mm/memory_hotplug.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
mm/memory_hotplug.c +4 -0
index bc805029da51..5498744aa1f1 100644
@@ -1155,6 +1155,7 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 	int need_zonelists_rebuild = 0;
 	unsigned long flags;
 	int ret;
+	bool need_set_normal_memory = false;
 
 	/*
 	 * {on,off}lining is constrained to full memory sections (or more
@@ -1180,6 +1181,9 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 		if (ret)
 			goto failed_addition;
 	}
+	/* Adding normal memory to the node for the first time */
+	if (!node_state(nid, N_NORMAL_MEMORY) && zone_idx(zone) <= ZONE_NORMAL)
+		need_set_normal_memory = true;
 
 	ret = memory_notify(MEM_GOING_ONLINE, &mem_arg);
 	ret = notifier_to_errno(ret);
@@ -1209,6 +1213,8 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 

Quoted 11 lines

mm/memory_hotplug.c int online_pages(unsigned long pfn, unsigned long nr_pages, · line 1219
Quoted 6 lines
mm/memory_hotplug.c +2 -0
 	if (node_arg.nid >= 0)
 		node_set_state(nid, N_MEMORY);
+	if (need_set_normal_memory)
+		node_set_state(nid, N_NORMAL_MEMORY);
 	if (need_zonelists_rebuild)
 		build_all_zonelists(NULL);
Do we need the flag here? As far as I can tell, we can just skip this and just directly check whether this is the first normal memory we are adding to the node here and set the bit. Then we can remove the flag and the extraneous check. We won't do any notifier work so I think it should be OK.

I assume you simply mean:

if (zone_idx(zone) <= ZONE_NORMAL && !node_state(nid, N_NORMAL_MEMORY))

	node_set_state(nid, N_NORMAL_MEMORY);

That looks m uch better!

Quoted 23 lines

mm/memory_hotplug.c +10 -0

@@ -1908,6 +1914,9 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 	unsigned long flags;
 	char *reason;
 	int ret;
+	bool need_clear_normal_memory = false;
+	unsigned long node_normal_pages = 0;
+	enum zone_type zt;
 
 	/*
 	 * {on,off}lining is constrained to full memory sections (or more

@@ -1977,6 +1986,13 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 			goto failed_removal_isolated;
 		}
 	}
+	/*
+	 * Check whether this operation removes the node's last normal memory.
+	 */
+	for (zt = 0; zt <= ZONE_NORMAL; zt++)
+		node_normal_pages += pgdat->node_zones[zt].present_pages;
+	if (nr_pages >= node_normal_pages && zone_idx(zone) <= ZONE_NORMAL)
+		need_clear_normal_memory = true;

Quoted 10 lines

mm/memory_hotplug.c int offline_pages(unsigned long start_pfn, unsigned long nr_pages, · line 2072
Quoted 5 lines
mm/memory_hotplug.c +0 -0
 	ret = memory_notify(MEM_GOING_OFFLINE, &mem_arg);
 	ret = notifier_to_errno(ret);
@@ -2055,6 +2071,12 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 	/* reinitialise watermarks and update pcp limits */
 	init_per_zone_wmark_min();
Same here, couldn't we just iterate through the paegs here and accumulate node_normal_pages and clear the memory here? We can get rid of the bool and also keep node_normal_pages defined inside an if (zone_idx(zone) <= ZONE_NORMAL) check as well.

Right, all we have to do is check for remaining present_pages in one of the nodes IIUC.

Signature · 3 lines

Cheers,

David

Hao Li

author

2026-03-28 04:12 UTC #9 lore raw

Quoted 78 lines · David Hildenbrand (Arm)

Quoted 65 lines · Joshua Hahn
On Fri, 27 Mar 2026 20:42:47 +0800 Hao Li <[email protected]> wrote:
Hello Hao,
I hope you are doing well, thank you for the patch!
Quoted 5 lines
N_NORMAL_MEMORY is initialized from zone population at boot, but memory hotplug currently only updates N_MEMORY. As a result, a node that gains normal memory via hotplug can remain invisible to users iterating over N_NORMAL_MEMORY, while a node that loses its last normal memory can stay incorrectly marked as such.
The second part feels more important than the second part, doing a quick glance through the code I can see a few N_NORMAL_MEMORY iterators that are in some hot paths like shrink_memcg. Iterating over nodes that don't contain any NORMAL memory seems like an inefficiency rather than a bug though.
Quoted 47 lines
Restore N_NORMAL_MEMORY maintenance directly in online_pages() and offline_pages(). Set the bit when a node that currently lacks normal memory onlines pages into a zone <= ZONE_NORMAL, and clear it when offlining removes the last present pages from zones <= ZONE_NORMAL.
This restores the intended semantics without bringing back the old status_change_nid_normal notifier plumbing which was removed in 8d2882a8edb8.
Current users that benefit include list_lru, zswap, nfsd filecache, hugetlb_cgroup, and has_normal_memory sysfs reporting.
Signed-off-by Hao Li <[email protected]>
---
This patch also prepares for a subsequent SLUB change that makes can_free_to_pcs() rely on N_NORMAL_MEMORY to decide whether an object can be freed to the sheaf.
---
 mm/memory_hotplug.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
mm/memory_hotplug.c +4 -0
index bc805029da51..5498744aa1f1 100644
@@ -1155,6 +1155,7 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 	int need_zonelists_rebuild = 0;
 	unsigned long flags;
 	int ret;
+	bool need_set_normal_memory = false;
 
 	/*
 	 * {on,off}lining is constrained to full memory sections (or more
@@ -1180,6 +1181,9 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 		if (ret)
 			goto failed_addition;
 	}
+	/* Adding normal memory to the node for the first time */
+	if (!node_state(nid, N_NORMAL_MEMORY) && zone_idx(zone) <= ZONE_NORMAL)
+		need_set_normal_memory = true;
 
 	ret = memory_notify(MEM_GOING_ONLINE, &mem_arg);
 	ret = notifier_to_errno(ret);
@@ -1209,6 +1213,8 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 
Quoted 11 lines
mm/memory_hotplug.c int online_pages(unsigned long pfn, unsigned long nr_pages, · line 1219
Quoted 6 lines
mm/memory_hotplug.c +2 -0
 	if (node_arg.nid >= 0)
 		node_set_state(nid, N_MEMORY);
+	if (need_set_normal_memory)
+		node_set_state(nid, N_NORMAL_MEMORY);
 	if (need_zonelists_rebuild)
 		build_all_zonelists(NULL);
Do we need the flag here? As far as I can tell, we can just skip this and just directly check whether this is the first normal memory we are adding to the node here and set the bit. Then we can remove the flag and the extraneous check. We won't do any notifier work so I think it should be OK.

Quoted 7 lines

I assume you simply mean:
if (zone_idx(zone) <= ZONE_NORMAL && !node_state(nid, N_NORMAL_MEMORY))
	node_set_state(nid, N_NORMAL_MEMORY);
That looks m uch better!

Yes, this approach is elegant! Thanks.

Quoted 35 lines

Quoted 24 lines

mm/memory_hotplug.c +10 -0

@@ -1908,6 +1914,9 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 	unsigned long flags;
 	char *reason;
 	int ret;
+	bool need_clear_normal_memory = false;
+	unsigned long node_normal_pages = 0;
+	enum zone_type zt;
 
 	/*
 	 * {on,off}lining is constrained to full memory sections (or more

@@ -1977,6 +1986,13 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 			goto failed_removal_isolated;
 		}
 	}
+	/*
+	 * Check whether this operation removes the node's last normal memory.
+	 */
+	for (zt = 0; zt <= ZONE_NORMAL; zt++)
+		node_normal_pages += pgdat->node_zones[zt].present_pages;
+	if (nr_pages >= node_normal_pages && zone_idx(zone) <= ZONE_NORMAL)
+		need_clear_normal_memory = true;

Quoted 10 lines

mm/memory_hotplug.c int offline_pages(unsigned long start_pfn, unsigned long nr_pages, · line 2072
Quoted 5 lines
mm/memory_hotplug.c +0 -0
 	ret = memory_notify(MEM_GOING_OFFLINE, &mem_arg);
 	ret = notifier_to_errno(ret);
@@ -2055,6 +2071,12 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 	/* reinitialise watermarks and update pcp limits */
 	init_per_zone_wmark_min();
Same here, couldn't we just iterate through the paegs here and accumulate node_normal_pages and clear the memory here? We can get rid of the bool and also keep node_normal_pages defined inside an if (zone_idx(zone) <= ZONE_NORMAL) check as well.

Quoted 2 lines

Right, all we have to do is check for remaining present_pages in one of the nodes IIUC.

Yes, we can simply sum up the remaining present_pages and check whether it comes out to 0. That feels more straightforward and direct.

Signature · 2 lines

Thanks,
Hao

Hao Li

author

2026-03-28 03:47 UTC #10 lore raw

In reply to Joshua Hahn

Quoted 5 lines · Joshua Hahn

On Fri, 27 Mar 2026 20:42:47 +0800 Hao Li <[email protected]> wrote:
Hello Hao,
I hope you are doing well, thank you for the patch!

Hi Joshua,

Thanks! Hope you are too.

Quoted 8 lines

Quoted 6 lines
N_NORMAL_MEMORY is initialized from zone population at boot, but memory hotplug currently only updates N_MEMORY. As a result, a node that gains normal memory via hotplug can remain invisible to users iterating over N_NORMAL_MEMORY, while a node that loses its last normal memory can stay incorrectly marked as such.
The second part feels more important

I'm guessing you mean the first part.

Quoted 5 lines

than the second part, doing a quick
glance through the code I can see a few N_NORMAL_MEMORY iterators that are in some hot paths like shrink_memcg. Iterating over nodes that don't contain any NORMAL memory seems like an inefficiency rather than a bug though.

Yes, I agree.

Quoted 48 lines

Quoted 48 lines
Restore N_NORMAL_MEMORY maintenance directly in online_pages() and offline_pages(). Set the bit when a node that currently lacks normal memory onlines pages into a zone <= ZONE_NORMAL, and clear it when offlining removes the last present pages from zones <= ZONE_NORMAL.
This restores the intended semantics without bringing back the old status_change_nid_normal notifier plumbing which was removed in 8d2882a8edb8.
Current users that benefit include list_lru, zswap, nfsd filecache, hugetlb_cgroup, and has_normal_memory sysfs reporting.
Signed-off-by Hao Li <[email protected]>
---
This patch also prepares for a subsequent SLUB change that makes can_free_to_pcs() rely on N_NORMAL_MEMORY to decide whether an object can be freed to the sheaf.
---
 mm/memory_hotplug.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)
mm/memory_hotplug.c +4 -0
index bc805029da51..5498744aa1f1 100644
@@ -1155,6 +1155,7 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 	int need_zonelists_rebuild = 0;
 	unsigned long flags;
 	int ret;
+	bool need_set_normal_memory = false;
 
 	/*
 	 * {on,off}lining is constrained to full memory sections (or more
@@ -1180,6 +1181,9 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 		if (ret)
 			goto failed_addition;
 	}
+	/* Adding normal memory to the node for the first time */
+	if (!node_state(nid, N_NORMAL_MEMORY) && zone_idx(zone) <= ZONE_NORMAL)
+		need_set_normal_memory = true;
 
 	ret = memory_notify(MEM_GOING_ONLINE, &mem_arg);
 	ret = notifier_to_errno(ret);
@@ -1209,6 +1213,8 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,
 

Quoted 11 lines

mm/memory_hotplug.c int online_pages(unsigned long pfn, unsigned long nr_pages, · line 1219
Quoted 6 lines
mm/memory_hotplug.c +2 -0
 	if (node_arg.nid >= 0)
 		node_set_state(nid, N_MEMORY);
+	if (need_set_normal_memory)
+		node_set_state(nid, N_NORMAL_MEMORY);
 	if (need_zonelists_rebuild)
 		build_all_zonelists(NULL);
Do we need the flag here? As far as I can tell, we can just skip this and just directly check whether this is the first normal memory we are adding to the node here and set the bit. Then we can remove the flag and the extraneous check. We won't do any notifier work so I think it should be OK.

Exactly, this is a good point!

Quoted 23 lines

mm/memory_hotplug.c +10 -0

@@ -1908,6 +1914,9 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 	unsigned long flags;
 	char *reason;
 	int ret;
+	bool need_clear_normal_memory = false;
+	unsigned long node_normal_pages = 0;
+	enum zone_type zt;
 
 	/*
 	 * {on,off}lining is constrained to full memory sections (or more

@@ -1977,6 +1986,13 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 			goto failed_removal_isolated;
 		}
 	}
+	/*
+	 * Check whether this operation removes the node's last normal memory.
+	 */
+	for (zt = 0; zt <= ZONE_NORMAL; zt++)
+		node_normal_pages += pgdat->node_zones[zt].present_pages;
+	if (nr_pages >= node_normal_pages && zone_idx(zone) <= ZONE_NORMAL)
+		need_clear_normal_memory = true;

Quoted 10 lines

mm/memory_hotplug.c int offline_pages(unsigned long start_pfn, unsigned long nr_pages, · line 2072
Quoted 5 lines
mm/memory_hotplug.c +0 -0
 	ret = memory_notify(MEM_GOING_OFFLINE, &mem_arg);
 	ret = notifier_to_errno(ret);
@@ -2055,6 +2071,12 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,
 	/* reinitialise watermarks and update pcp limits */
 	init_per_zone_wmark_min();
Same here, couldn't we just iterate through the paegs here and accumulate node_normal_pages and clear the memory here? We can get rid of the bool and also keep node_normal_pages defined inside an if (zone_idx(zone) <= ZONE_NORMAL) check as well.

Yes, the same to online_pages(), thanks for this idea.

Quoted 17 lines

Quoted 14 lines

mm/memory_hotplug.c +6 -0

+	/*
+	 * Clear N_NORMAL_MEMORY first to avoid the transient state
+	 * "!N_MEMORY && N_NORMAL_MEMORY".
+	 */
+	if (need_clear_normal_memory)
+		node_clear_state(node, N_NORMAL_MEMORY);
 	/*
 	 * Make sure to mark the node as memory-less before rebuilding the zone
 	 * list. Otherwise this node would still appear in the fallback lists.

Signature · 1 line

2.50.1

Thank you for the patch. The rest looks good to me! Have a great day : -) Joshua

Thanks for the review!

Signature · 2 lines

Thanks,
Hao