Skip to contentSkip to conversation

mm, memory_hotplug: remove zone onlining restriction

RFC PATCH In review Michal Hocko 4 participants 22 messages Jun 29, 2017 – Jul 12, 2017

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

cover mm, memory_hotplug: remove zone onlining restriction

Hi,
I am sending this as an RFC because this hasn't seen a lot of testing yet but I would like to see whether the semantic I came up with (see patch 2) is sensible. This work should help Joonsoo with his CMA zone based approach when reusing MOVABLE zone. I think it will also help to remove more code from the memory hotplug (e.g. zone shrinking).

Patch 1 restores original memoryXY/valid_zones semantic wrt zone ordering. This can be merged without patch 2 which removes the zone overlap restriction and defines a semantic for the default onlining. See more in the patch.

Questions, concerns, objections?

Shortlog
Michal Hocko (2):
mm, memory_hotplug: display allowed zones in the preferred ordering mm, memory_hotplug: remove zone restrictions

Diffstat

 drivers/base/memory.c          | 30 ++++++++++-----
 include/linux/memory_hotplug.h |  2 +-
 mm/memory_hotplug.c            | 87 +++++++++++++++++-------------------------
 3 files changed, 55 insertions(+), 64 deletions(-)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#1 lore raw

PATCH 1/2 mm, memory_hotplug: display allowed zones in the preferred ordering

From: Michal Hocko <[email protected]>

Prior to "mm, memory_hotplug: do not associate hotadded memory to zones until online" we used to allow to change the valid zone types of a memory block if it is adjacent to a different zone type. This fact was reflected in memoryNN/valid_zones by the ordering of printed zones. The first one was default (echo online > memoryNN/state) and the other one could be onlined explicitly by online_{movable,kernel}. This behavior was removed by the said patch and as such the ordering was not all that important. In most cases a kernel zone would be default anyway. The only exception is movable_node handled by "mm,
memory_hotplug: support movable_node for hotpluggable nodes".

Let's reintroduce this behavior again because later patch will remove the zone overlap restriction and so user will be allowed to online kernel resp. movable block regardless of its placement. Original behavior will then become significant again because it would be non-trivial for users to see what is the default zone to online into.

Implementation is really simple. Pull out zone selection out of move_pfn_range into zone_for_pfn_range helper and use it in show_valid_zones to display the zone for default onlining and then both kernel and movable if they are allowed. Default online zone is not duplicated.

fold me "mm, memory_hotplug: display allowed zones in the preferred ordering"

---
 drivers/base/memory.c          | 33 +++++++++++++------
 include/linux/memory_hotplug.h |  2 +-
 mm/memory_hotplug.c            | 73 ++++++++++++++++++++++++------------------
 3 files changed, 65 insertions(+), 43 deletions(-)
drivers/base/memory.c +23 -10
index c7c4e0325cdb..26383af9900c 100644
@@ -388,6 +388,22 @@ static ssize_t show_phys_device(struct device *dev,
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
+static void print_allowed_zone(char *buf, int nid, unsigned long start_pfn,
+		unsigned long nr_pages, int online_type,
+		struct zone *default_zone)
+{
+	struct zone *zone;
+
+	if (!allow_online_pfn_range(nid, start_pfn, nr_pages, online_type))
+		return;
+
+	zone = zone_for_pfn_range(online_type, nid, start_pfn, nr_pages);
+	if (zone != default_zone) {
+		strcat(buf, " ");
+		strcat(buf, zone->name);
+	}
+}
+
 static ssize_t show_valid_zones(struct device *dev,
 				struct device_attribute *attr, char *buf)
 {
@@ -395,7 +411,7 @@ static ssize_t show_valid_zones(struct device *dev,
 	unsigned long start_pfn = section_nr_to_pfn(mem->start_section_nr);
 	unsigned long nr_pages = PAGES_PER_SECTION * sections_per_block;
 	unsigned long valid_start_pfn, valid_end_pfn;
-	bool append = false;
+	struct zone *default_zone;
 	int nid;
 
 	/*
@@ -418,16 +434,13 @@ static ssize_t show_valid_zones(struct device *dev,
 	}
 
 	nid = pfn_to_nid(start_pfn);
-	if (allow_online_pfn_range(nid, start_pfn, nr_pages, MMOP_ONLINE_KERNEL)) {
-		strcat(buf, default_zone_for_pfn(nid, start_pfn, nr_pages)->name);
-		append = true;
-	}
+	default_zone = zone_for_pfn_range(MMOP_ONLINE_KEEP, nid, start_pfn, nr_pages);
+	strcat(buf, default_zone->name);
 
-	if (allow_online_pfn_range(nid, start_pfn, nr_pages, MMOP_ONLINE_MOVABLE)) {
-		if (append)
-			strcat(buf, " ");
-		strcat(buf, NODE_DATA(nid)->node_zones[ZONE_MOVABLE].name);
-	}
+	print_allowed_zone(buf, nid, start_pfn, nr_pages, MMOP_ONLINE_KERNEL,
+			default_zone);
+	print_allowed_zone(buf, nid, start_pfn, nr_pages, MMOP_ONLINE_MOVABLE,
+			default_zone);
 out:
 	strcat(buf, "\n");
 
include/linux/memory_hotplug.h +1 -1
index c8a5056a5ae0..5e6e4cc36ff4 100644
@@ -319,6 +319,6 @@ extern struct page *sparse_decode_mem_map(unsigned long coded_mem_map,
 					  unsigned long pnum);
 extern bool allow_online_pfn_range(int nid, unsigned long pfn, unsigned long nr_pages,
 		int online_type);
-extern struct zone *default_zone_for_pfn(int nid, unsigned long pfn,
+extern struct zone *zone_for_pfn_range(int online_type, int nid, unsigned start_pfn,
 		unsigned long nr_pages);
 #endif /* __LINUX_MEMORY_HOTPLUG_H */
mm/memory_hotplug.c +41 -32

103 lines not on this page · raw message

Signature · 7 lines
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Acked-by on v1 1/2
#2 lore raw

Re: [PATCH 1/2] mm, memory_hotplug: display allowed zones in the preferred ordering

Quoted 19 lines · Michal Hocko

From: Michal Hocko <[email protected]>

Prior to "mm, memory_hotplug: do not associate hotadded memory to zones until online" we used to allow to change the valid zone types of a memory block if it is adjacent to a different zone type. This fact was reflected in memoryNN/valid_zones by the ordering of printed zones. The first one was default (echo online > memoryNN/state) and the other one could be onlined explicitly by online_{movable,kernel}. This behavior was removed by the said patch and as such the ordering was not all that important. In most cases a kernel zone would be default anyway. The only exception is movable_node handled by "mm,
memory_hotplug: support movable_node for hotpluggable nodes".

Let's reintroduce this behavior again because later patch will remove the zone overlap restriction and so user will be allowed to online kernel resp. movable block regardless of its placement. Original behavior will then become significant again because it would be non-trivial for users to see what is the default zone to online into.

Quoted 8 lines

Implementation is really simple. Pull out zone selection out of move_pfn_range into zone_for_pfn_range helper and use it in show_valid_zones to display the zone for default onlining and then both kernel and movable if they are allowed. Default online zone is not duplicated.

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Acked-by on v1 1/2
#3 lore raw

Re: [PATCH 1/2] mm, memory_hotplug: display allowed zones in the preferred ordering

In reply to Michal Hocko
Quoted 19 lines · Michal Hocko

From: Michal Hocko <[email protected]>

Prior to "mm, memory_hotplug: do not associate hotadded memory to zones until online" we used to allow to change the valid zone types of a memory block if it is adjacent to a different zone type. This fact was reflected in memoryNN/valid_zones by the ordering of printed zones. The first one was default (echo online > memoryNN/state) and the other one could be onlined explicitly by online_{movable,kernel}. This behavior was removed by the said patch and as such the ordering was not all that important. In most cases a kernel zone would be default anyway. The only exception is movable_node handled by "mm,
memory_hotplug: support movable_node for hotpluggable nodes".

Let's reintroduce this behavior again because later patch will remove the zone overlap restriction and so user will be allowed to online kernel resp. movable block regardless of its placement. Original behavior will then become significant again because it would be non-trivial for users to see what is the default zone to online into.

Quoted 5 lines

Implementation is really simple. Pull out zone selection out of move_pfn_range into zone_for_pfn_range helper and use it in show_valid_zones to display the zone for default onlining and then both kernel and movable if they are allowed. Default online zone is not duplicated.

Hm I wouldn't call this maze of functions simple, but seems to be correct. Maybe Patch 2/2 will simplify the code...

Quoted 1 line
Quoted 182 lines

fold me "mm, memory_hotplug: display allowed zones in the preferred ordering"

---
 drivers/base/memory.c          | 33 +++++++++++++------
 include/linux/memory_hotplug.h |  2 +-
 mm/memory_hotplug.c            | 73 ++++++++++++++++++++++++------------------
 3 files changed, 65 insertions(+), 43 deletions(-)
drivers/base/memory.c +23 -10
index c7c4e0325cdb..26383af9900c 100644
@@ -388,6 +388,22 @@ static ssize_t show_phys_device(struct device *dev,
 }
 
 #ifdef CONFIG_MEMORY_HOTREMOVE
+static void print_allowed_zone(char *buf, int nid, unsigned long start_pfn,
+		unsigned long nr_pages, int online_type,
+		struct zone *default_zone)
+{
+	struct zone *zone;
+
+	if (!allow_online_pfn_range(nid, start_pfn, nr_pages, online_type))
+		return;
+
+	zone = zone_for_pfn_range(online_type, nid, start_pfn, nr_pages);
+	if (zone != default_zone) {
+		strcat(buf, " ");
+		strcat(buf, zone->name);
+	}
+}
+
 static ssize_t show_valid_zones(struct device *dev,
 				struct device_attribute *attr, char *buf)
 {
@@ -395,7 +411,7 @@ static ssize_t show_valid_zones(struct device *dev,
 	unsigned long start_pfn = section_nr_to_pfn(mem->start_section_nr);
 	unsigned long nr_pages = PAGES_PER_SECTION * sections_per_block;
 	unsigned long valid_start_pfn, valid_end_pfn;
-	bool append = false;
+	struct zone *default_zone;
 	int nid;
 
 	/*
@@ -418,16 +434,13 @@ static ssize_t show_valid_zones(struct device *dev,
 	}
 
 	nid = pfn_to_nid(start_pfn);
-	if (allow_online_pfn_range(nid, start_pfn, nr_pages, MMOP_ONLINE_KERNEL)) {
-		strcat(buf, default_zone_for_pfn(nid, start_pfn, nr_pages)->name);
-		append = true;
-	}
+	default_zone = zone_for_pfn_range(MMOP_ONLINE_KEEP, nid, start_pfn, nr_pages);
+	strcat(buf, default_zone->name);
 
-	if (allow_online_pfn_range(nid, start_pfn, nr_pages, MMOP_ONLINE_MOVABLE)) {
-		if (append)
-			strcat(buf, " ");
-		strcat(buf, NODE_DATA(nid)->node_zones[ZONE_MOVABLE].name);
-	}
+	print_allowed_zone(buf, nid, start_pfn, nr_pages, MMOP_ONLINE_KERNEL,
+			default_zone);
+	print_allowed_zone(buf, nid, start_pfn, nr_pages, MMOP_ONLINE_MOVABLE,
+			default_zone);
 out:
 	strcat(buf, "\n");
 
include/linux/memory_hotplug.h +1 -1
index c8a5056a5ae0..5e6e4cc36ff4 100644
@@ -319,6 +319,6 @@ extern struct page *sparse_decode_mem_map(unsigned long coded_mem_map,
 					  unsigned long pnum);
 extern bool allow_online_pfn_range(int nid, unsigned long pfn, unsigned long nr_pages,
 		int online_type);
-extern struct zone *default_zone_for_pfn(int nid, unsigned long pfn,
+extern struct zone *zone_for_pfn_range(int online_type, int nid, unsigned start_pfn,
 		unsigned long nr_pages);
 #endif /* __LINUX_MEMORY_HOTPLUG_H */
mm/memory_hotplug.c +34 -32
index b4015a39d108..6b9a60115e37 100644
@@ -836,31 +836,6 @@ static void node_states_set_node(int node, struct memory_notify *arg)
 	node_set_state(node, N_MEMORY);
 }
 
-bool allow_online_pfn_range(int nid, unsigned long pfn, unsigned long nr_pages, int online_type)
-{
-	struct pglist_data *pgdat = NODE_DATA(nid);
-	struct zone *movable_zone = &pgdat->node_zones[ZONE_MOVABLE];
-	struct zone *default_zone = default_zone_for_pfn(nid, pfn, nr_pages);
-
-	/*
-	 * TODO there shouldn't be any inherent reason to have ZONE_NORMAL
-	 * physically before ZONE_MOVABLE. All we need is they do not
-	 * overlap. Historically we didn't allow ZONE_NORMAL after ZONE_MOVABLE
-	 * though so let's stick with it for simplicity for now.
-	 * TODO make sure we do not overlap with ZONE_DEVICE
-	 */
-	if (online_type == MMOP_ONLINE_KERNEL) {
-		if (zone_is_empty(movable_zone))
-			return true;
-		return movable_zone->zone_start_pfn >= pfn + nr_pages;
-	} else if (online_type == MMOP_ONLINE_MOVABLE) {
-		return zone_end_pfn(default_zone) <= pfn;
-	}
-
-	/* MMOP_ONLINE_KEEP will always succeed and inherits the current zone */
-	return online_type == MMOP_ONLINE_KEEP;
-}
-
 static void __meminit resize_zone_range(struct zone *zone, unsigned long start_pfn,
 		unsigned long nr_pages)
 {
@@ -919,7 +894,7 @@ void __ref move_pfn_range_to_zone(struct zone *zone,
  * If no kernel zone covers this pfn range it will automatically go
  * to the ZONE_NORMAL.
  */
-struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn,
+static struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn,
 		unsigned long nr_pages)
 {
 	struct pglist_data *pgdat = NODE_DATA(nid);
@@ -935,6 +910,31 @@ struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn,
 	return &pgdat->node_zones[ZONE_NORMAL];
 }
 
+bool allow_online_pfn_range(int nid, unsigned long pfn, unsigned long nr_pages, int online_type)
+{
+	struct pglist_data *pgdat = NODE_DATA(nid);
+	struct zone *movable_zone = &pgdat->node_zones[ZONE_MOVABLE];
+	struct zone *default_zone = default_zone_for_pfn(nid, pfn, nr_pages);
+
+	/*
+	 * TODO there shouldn't be any inherent reason to have ZONE_NORMAL
+	 * physically before ZONE_MOVABLE. All we need is they do not
+	 * overlap. Historically we didn't allow ZONE_NORMAL after ZONE_MOVABLE
+	 * though so let's stick with it for simplicity for now.
+	 * TODO make sure we do not overlap with ZONE_DEVICE
+	 */
+	if (online_type == MMOP_ONLINE_KERNEL) {
+		if (zone_is_empty(movable_zone))
+			return true;
+		return movable_zone->zone_start_pfn >= pfn + nr_pages;
+	} else if (online_type == MMOP_ONLINE_MOVABLE) {
+		return zone_end_pfn(default_zone) <= pfn;
+	}
+
+	/* MMOP_ONLINE_KEEP will always succeed and inherits the current zone */
+	return online_type == MMOP_ONLINE_KEEP;
+}
+
 static inline bool movable_pfn_range(int nid, struct zone *default_zone,
 		unsigned long start_pfn, unsigned long nr_pages)
 {
@@ -948,12 +948,8 @@ static inline bool movable_pfn_range(int nid, struct zone *default_zone,
 	return !zone_intersects(default_zone, start_pfn, nr_pages);
 }
 
-/*
- * Associates the given pfn range with the given node and the zone appropriate
- * for the given online type.
- */
-static struct zone * __meminit move_pfn_range(int online_type, int nid,
-		unsigned long start_pfn, unsigned long nr_pages)
+struct zone * zone_for_pfn_range(int online_type, int nid, unsigned start_pfn,
+		unsigned long nr_pages)
 {
 	struct pglist_data *pgdat = NODE_DATA(nid);
 	struct zone *zone = default_zone_for_pfn(nid, start_pfn, nr_pages);
@@ -972,6 +968,19 @@ static struct zone * __meminit move_pfn_range(int online_type, int nid,
 		zone = &pgdat->node_zones[ZONE_MOVABLE];
 	}
 
+	return zone;
+}
+
+/*
+ * Associates the given pfn range with the given node and the zone appropriate
+ * for the given online type.

mm/memory_hotplug.c static struct zone * __meminit move_pfn_range(int online_type, int nid, · line 986

Quoted 10 lines
mm/memory_hotplug.c +7 -0
+ */
+static struct zone * __meminit move_pfn_range(int online_type, int nid,
+		unsigned long start_pfn, unsigned long nr_pages)
+{
+	struct zone *zone;
+
+	zone = zone_for_pfn_range(online_type, nid, start_pfn, nr_pages);
 	move_pfn_range_to_zone(zone, start_pfn, nr_pages);
 	return zone;
 }

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw

PATCH 2/2 mm, memory_hotplug: remove zone restrictions

In reply to Michal Hocko

From: Michal Hocko <[email protected]>

Historically we have enforced that any kernel zone (e.g ZONE_NORMAL) has to precede the Movable zone in the physical memory range. The purpose of the movable zone is, however, not bound to any physical memory restriction. It merely defines a class of migrateable and reclaimable memory.

There are users (e.g. CMA) who might want to reserve specific physical memory ranges for their own purpose. Moreover our pfn walkers have to be prepared for zones overlapping in the physical range already because we do support interleaving NUMA nodes and therefore zones can interleave as well. This means we can allow each memory block to be associated with a different zone.

Loosen the current onlining semantic and allow explicit onlining type on any memblock. That means that online_{kernel,movable} will be allowed regardless of the physical address of the memblock as long as it is offline of course. This might result in moveble zone overlapping with other kernel zones. Default onlining then becomes a bit tricky but still sensible. echo online > memoryXY/state will online the given block to
1) the default zone if the given range is outside of any zone
2) the enclosing zone if such a zone doesn't interleave with any other zone
3) the default zone if more zones interleave for this range
where default zone is movable zone only if movable_node is enabled otherwise it is a kernel zone.

Here is an example of the semantic with (movable_node is not present but it work in an analogous way). We start with following memblocks, all of them offline
memory34/valid_zones:Normal Movable memory35/valid_zones:Normal Movable memory36/valid_zones:Normal Movable memory37/valid_zones:Normal Movable memory38/valid_zones:Normal Movable memory39/valid_zones:Normal Movable memory40/valid_zones:Normal Movable memory41/valid_zones:Normal Movable

Now, we online block 34 in default mode and block 37 as movable

root@test1:/sys/devices/system/node/node1# echo online > memory34/state
root@test1:/sys/devices/system/node/node1# echo online_movable > memory37/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal Movable
memory40/valid_zones:Normal Movable
memory41/valid_zones:Normal Movable

As we can see all other blocks can still be onlined both into Normal and Movable zones and the Normal is default because the Movable zone spans only block37 now.

root@test1:/sys/devices/system/node/node1# echo online_movable > memory41/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Movable Normal
memory39/valid_zones:Movable Normal
memory40/valid_zones:Movable Normal
memory41/valid_zones:Movable

Now the default zone for blocks 37-41 has changed because movable zone spans that range.

root@test1:/sys/devices/system/node/node1# echo online_kernel > memory39/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal
memory40/valid_zones:Movable Normal
memory41/valid_zones:Movable

Note that the block 39 now belongs to the zone Normal and so block38 falls into Normal by default as well.

For completness

root@test1:/sys/devices/system/node/node1# for i in memory[34]?
do
	echo online > $i/state 2>/dev/null
done

memory34/valid_zones:Normal
memory35/valid_zones:Normal
memory36/valid_zones:Normal
memory37/valid_zones:Movable
memory38/valid_zones:Normal
memory39/valid_zones:Normal
memory40/valid_zones:Movable
memory41/valid_zones:Movable

Implementation wise the change is quite straightforward. We can get rid of allow_online_pfn_range altogether. online_pages allows only offline nodes already. The original default_zone_for_pfn will become default_kernel_zone_for_pfn. New default_zone_for_pfn implements the above semantic. zone_for_pfn_range is slightly reorganized to implement kernel and movable online type explicitly and MMOP_ONLINE_KEEP becomes a catch all default behavior.

---
 drivers/base/memory.c |  3 ---
 mm/memory_hotplug.c   | 74 ++++++++++++++++-----------------------------------
 2 files changed, 23 insertions(+), 54 deletions(-)
drivers/base/memory.c +0 -3
index 26383af9900c..4e3b61cda520 100644
@@ -394,9 +394,6 @@ static void print_allowed_zone(char *buf, int nid, unsigned long start_pfn,
 {
 	struct zone *zone;
 
-	if (!allow_online_pfn_range(nid, start_pfn, nr_pages, online_type))
-		return;
-
 	zone = zone_for_pfn_range(online_type, nid, start_pfn, nr_pages);
 	if (zone != default_zone) {
 		strcat(buf, " ");
mm/memory_hotplug.c +23 -51

104 lines not on this page · raw message

Signature · 7 lines
2.11.0

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Acked-by on v1 2/2
#5 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 102 lines · Michal Hocko

From: Michal Hocko <[email protected]>

Historically we have enforced that any kernel zone (e.g ZONE_NORMAL) has to precede the Movable zone in the physical memory range. The purpose of the movable zone is, however, not bound to any physical memory restriction. It merely defines a class of migrateable and reclaimable memory.

There are users (e.g. CMA) who might want to reserve specific physical memory ranges for their own purpose. Moreover our pfn walkers have to be prepared for zones overlapping in the physical range already because we do support interleaving NUMA nodes and therefore zones can interleave as well. This means we can allow each memory block to be associated with a different zone.

Loosen the current onlining semantic and allow explicit onlining type on any memblock. That means that online_{kernel,movable} will be allowed regardless of the physical address of the memblock as long as it is offline of course. This might result in moveble zone overlapping with other kernel zones. Default onlining then becomes a bit tricky but still sensible. echo online > memoryXY/state will online the given block to
1) the default zone if the given range is outside of any zone
2) the enclosing zone if such a zone doesn't interleave with any other zone
3) the default zone if more zones interleave for this range
where default zone is movable zone only if movable_node is enabled otherwise it is a kernel zone.

Here is an example of the semantic with (movable_node is not present but it work in an analogous way). We start with following memblocks, all of them offline
memory34/valid_zones:Normal Movable memory35/valid_zones:Normal Movable memory36/valid_zones:Normal Movable memory37/valid_zones:Normal Movable memory38/valid_zones:Normal Movable memory39/valid_zones:Normal Movable memory40/valid_zones:Normal Movable memory41/valid_zones:Normal Movable

Now, we online block 34 in default mode and block 37 as movable

root@test1:/sys/devices/system/node/node1# echo online > memory34/state
root@test1:/sys/devices/system/node/node1# echo online_movable > memory37/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal Movable
memory40/valid_zones:Normal Movable
memory41/valid_zones:Normal Movable

As we can see all other blocks can still be onlined both into Normal and Movable zones and the Normal is default because the Movable zone spans only block37 now.

root@test1:/sys/devices/system/node/node1# echo online_movable > memory41/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Movable Normal
memory39/valid_zones:Movable Normal
memory40/valid_zones:Movable Normal
memory41/valid_zones:Movable

Now the default zone for blocks 37-41 has changed because movable zone spans that range.

root@test1:/sys/devices/system/node/node1# echo online_kernel > memory39/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal
memory40/valid_zones:Movable Normal
memory41/valid_zones:Movable

Note that the block 39 now belongs to the zone Normal and so block38 falls into Normal by default as well.

For completness

root@test1:/sys/devices/system/node/node1# for i in memory[34]?
do
	echo online > $i/state 2>/dev/null
done

memory34/valid_zones:Normal
memory35/valid_zones:Normal
memory36/valid_zones:Normal
memory37/valid_zones:Movable
memory38/valid_zones:Normal
memory39/valid_zones:Normal
memory40/valid_zones:Movable
memory41/valid_zones:Movable

Implementation wise the change is quite straightforward. We can get rid of allow_online_pfn_range altogether. online_pages allows only offline nodes already. The original default_zone_for_pfn will become default_kernel_zone_for_pfn. New default_zone_for_pfn implements the above semantic. zone_for_pfn_range is slightly reorganized to implement kernel and movable online type explicitly and MMOP_ONLINE_KEEP becomes a catch all default behavior.

Quoted 1 line

I appreaciate your help!

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

In reply to Michal Hocko
Quoted 2 lines · Michal Hocko

From: Michal Hocko <[email protected]>

Michal,

I love the idea very much.

Quoted 12 lines

Historically we have enforced that any kernel zone (e.g ZONE_NORMAL) has to precede the Movable zone in the physical memory range. The purpose of the movable zone is, however, not bound to any physical memory restriction. It merely defines a class of migrateable and reclaimable memory.

There are users (e.g. CMA) who might want to reserve specific physical memory ranges for their own purpose. Moreover our pfn walkers have to be prepared for zones overlapping in the physical range already because we do support interleaving NUMA nodes and therefore zones can interleave as well. This means we can allow each memory block to be associated with a different zone.

Quoted 5 lines

Loosen the current onlining semantic and allow explicit onlining type on any memblock. That means that online_{kernel,movable} will be allowed regardless of the physical address of the memblock as long as it is offline of course. This might result in moveble zone overlapping with other kernel zones. Default onlining then becomes a bit tricky but still

As here mentioned, we just remove the restriction for zone_movable. For other zones, we still keep the restriction and the order as before.

Maybe the title is a little misleading. Audience may thinks no restriction for all zones.

Quoted 35 lines

sensible. echo online > memoryXY/state will online the given block to
1) the default zone if the given range is outside of any zone
2) the enclosing zone if such a zone doesn't interleave with any other zone
3) the default zone if more zones interleave for this range
where default zone is movable zone only if movable_node is enabled otherwise it is a kernel zone.

Here is an example of the semantic with (movable_node is not present but it work in an analogous way). We start with following memblocks, all of them offline
memory34/valid_zones:Normal Movable memory35/valid_zones:Normal Movable memory36/valid_zones:Normal Movable memory37/valid_zones:Normal Movable memory38/valid_zones:Normal Movable memory39/valid_zones:Normal Movable memory40/valid_zones:Normal Movable memory41/valid_zones:Normal Movable

Now, we online block 34 in default mode and block 37 as movable

root@test1:/sys/devices/system/node/node1# echo online > memory34/state
root@test1:/sys/devices/system/node/node1# echo online_movable > memory37/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal Movable
memory40/valid_zones:Normal Movable
memory41/valid_zones:Normal Movable

As we can see all other blocks can still be onlined both into Normal and Movable zones and the Normal is default because the Movable zone spans only block37 now.

Quoted 10 lines
root@test1:/sys/devices/system/node/node1# echo online_movable > memory41/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Movable Normal
memory39/valid_zones:Movable Normal
memory40/valid_zones:Movable Normal
memory41/valid_zones:Movable

As I spotted on the previous patch, after several round of online/offline, The output of valid_zones will differ.

For example in this case, after I offline memory37 and 41, I expect this:

memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Normal Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal Movable
memory40/valid_zones:Normal Movable
memory41/valid_zones:Normal Movable

While the current result would be

memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable Normal
memory38/valid_zones:Movable Normal
memory39/valid_zones:Movable Normal
memory40/valid_zones:Movable Normal
memory41/valid_zones:Movable Normal

The reason is the same, we don't adjust the zone's range when offline memory.

This is also a known issue?

Quoted 38 lines

Now the default zone for blocks 37-41 has changed because movable zone spans that range.

root@test1:/sys/devices/system/node/node1# echo online_kernel > memory39/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal
memory40/valid_zones:Movable Normal
memory41/valid_zones:Movable

Note that the block 39 now belongs to the zone Normal and so block38 falls into Normal by default as well.

For completness

root@test1:/sys/devices/system/node/node1# for i in memory[34]?
do
        echo online > $i/state 2>/dev/null
done

memory34/valid_zones:Normal
memory35/valid_zones:Normal
memory36/valid_zones:Normal
memory37/valid_zones:Movable
memory38/valid_zones:Normal
memory39/valid_zones:Normal
memory40/valid_zones:Movable
memory41/valid_zones:Movable

Implementation wise the change is quite straightforward. We can get rid of allow_online_pfn_range altogether. online_pages allows only offline nodes already. The original default_zone_for_pfn will become default_kernel_zone_for_pfn. New default_zone_for_pfn implements the above semantic. zone_for_pfn_range is slightly reorganized to implement kernel and movable online type explicitly and MMOP_ONLINE_KEEP becomes a catch all default behavior.

Quoted 2 lines

---

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 20 lines · Wei Yang
Quoted 2 lines · Michal Hocko

From: Michal Hocko <[email protected]>

Michal,

I love the idea very much.

Quoted 12 lines

Historically we have enforced that any kernel zone (e.g ZONE_NORMAL) has to precede the Movable zone in the physical memory range. The purpose of the movable zone is, however, not bound to any physical memory restriction. It merely defines a class of migrateable and reclaimable memory.

There are users (e.g. CMA) who might want to reserve specific physical memory ranges for their own purpose. Moreover our pfn walkers have to be prepared for zones overlapping in the physical range already because we do support interleaving NUMA nodes and therefore zones can interleave as well. This means we can allow each memory block to be associated with a different zone.

Quoted 8 lines
Quoted 5 lines

Loosen the current onlining semantic and allow explicit onlining type on any memblock. That means that online_{kernel,movable} will be allowed regardless of the physical address of the memblock as long as it is offline of course. This might result in moveble zone overlapping with other kernel zones. Default onlining then becomes a bit tricky but still

As here mentioned, we just remove the restriction for zone_movable. For other zones, we still keep the restriction and the order as before.

All other zones except for ZONE_NORMAL are subject of the physical memory restrictions.

Quoted 2 lines

Maybe the title is a little misleading. Audience may thinks no restriction for all zones.

I thought the context was clear from the fact that this is a hotplug related patch. As such we do not allow online_{dma,dma32,normal} we only allow to online into a kernel zone. I can update the wording but do not have a good idea how.

[...]

Quoted 16 lines

As I spotted on the previous patch, after several round of online/offline, The output of valid_zones will differ.

For example in this case, after I offline memory37 and 41, I expect this:

memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Normal Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal Movable
memory40/valid_zones:Normal Movable
memory41/valid_zones:Normal Movable

While the current result would be

Quoted 8 lines

memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable Normal
memory38/valid_zones:Movable Normal
memory39/valid_zones:Movable Normal
memory40/valid_zones:Movable Normal
memory41/valid_zones:Movable Normal

You haven't written your sequence of onlining but if you used the same one as mentioned in the patch then you should get memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable memory36/valid_zones:Normal Movable memory37/valid_zones:Normal Movable memory38/valid_zones:Normal Movable memory39/valid_zones:Normal
memory40/valid_zones:Movable Normal memory41/valid_zones:Movable Normal

Even if you kept 37 as movable and offline 38 you wouldn't get 38-41 movable by default because...

Quoted 2 lines

The reason is the same, we don't adjust the zone's range when offline memory.

.. of this.

Quoted 1 line

This is also a known issue?

yes and to be honest I do not plan to fix it unless somebody has a real life usecase for it. Now that we allow explicit onlininig type anywhere it seems like a reasonable behavior and this will allow us to remove quite some code which is always a good deal wrt longterm maintenance.

Thanks!

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#8 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 22 lines · Michal Hocko
Quoted 8 lines · Wei Yang
Quoted 2 lines · Michal Hocko

From: Michal Hocko <[email protected]>

Michal,

I love the idea very much.

You haven't written your sequence of onlining but if you used the same one as mentioned in the patch then you should get memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable memory36/valid_zones:Normal Movable memory37/valid_zones:Normal Movable memory38/valid_zones:Normal Movable memory39/valid_zones:Normal
memory40/valid_zones:Movable Normal memory41/valid_zones:Movable Normal

Quoted 3 lines

Even if you kept 37 as movable and offline 38 you wouldn't get 38-41 movable by default because...

Yes, it depends on the zone range.

Quoted 12 lines
Quoted 2 lines

The reason is the same, we don't adjust the zone's range when offline memory.

.. of this.

Quoted 1 line

This is also a known issue?

yes and to be honest I do not plan to fix it unless somebody has a real life usecase for it. Now that we allow explicit onlininig type anywhere it seems like a reasonable behavior and this will allow us to remove quite some code which is always a good deal wrt longterm maintenance.

hmm... the statistics displayed in /proc/zoneinfo would be meaningless for zone_normal and zone_movable.

I am not sure, maybe no one care about these fields.

Quoted 4 lines

Thanks!
--
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 1 line · Wei Yang

On Fri, Jun 30, 2017 at 4:39 PM, Michal Hocko <[email protected]> wrote:

[...]

Quoted 8 lines
Quoted 5 lines

yes and to be honest I do not plan to fix it unless somebody has a real life usecase for it. Now that we allow explicit onlininig type anywhere it seems like a reasonable behavior and this will allow us to remove quite some code which is always a good deal wrt longterm maintenance.

hmm... the statistics displayed in /proc/zoneinfo would be meaningless for zone_normal and zone_movable.

Why would they be meaningless? Counters will always reflect the actual use - if not then it is a bug. And wrt to zone description what is meaningless about
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable memory36/valid_zones:Movable
memory37/valid_zones:Movable Normal memory38/valid_zones:Movable Normal memory39/valid_zones:Movable Normal memory40/valid_zones:Normal
memory41/valid_zones:Movable

And
Node 1, zone Normal

  pages free     65465
        min      156
        low      221
        high     286
        spanned  229376
        present  65536
        managed  65536
[...]
  start_pfn:           1114112
Node 1, zone  Movable
  pages free     65443
        min      156
        low      221
        high     286
        spanned  196608
        present  65536
        managed  65536

[...]
start_pfn: 1179648

ranges are clearly defined as [start_pfn, start_pfn+managed] and managed matches the number of onlined pages (256MB).

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#10 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 43 lines · Michal Hocko
Quoted 1 line · Wei Yang

On Fri, Jun 30, 2017 at 4:39 PM, Michal Hocko <[email protected]> wrote:

[...]

Quoted 8 lines
Quoted 5 lines

yes and to be honest I do not plan to fix it unless somebody has a real life usecase for it. Now that we allow explicit onlininig type anywhere it seems like a reasonable behavior and this will allow us to remove quite some code which is always a good deal wrt longterm maintenance.

hmm... the statistics displayed in /proc/zoneinfo would be meaningless for zone_normal and zone_movable.

Why would they be meaningless? Counters will always reflect the actual use - if not then it is a bug. And wrt to zone description what is meaningless about
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable memory36/valid_zones:Movable
memory37/valid_zones:Movable Normal memory38/valid_zones:Movable Normal memory39/valid_zones:Movable Normal memory40/valid_zones:Normal
memory41/valid_zones:Movable

And
Node 1, zone Normal

  pages free     65465
        min      156
        low      221
        high     286
        spanned  229376
        present  65536
        managed  65536
[...]
  start_pfn:           1114112
Node 1, zone  Movable
  pages free     65443
        min      156
        low      221
        high     286
        spanned  196608
        present  65536
        managed  65536
Quoted 4 lines

[...]
start_pfn: 1179648

ranges are clearly defined as [start_pfn, start_pfn+managed] and managed

errr, this should be [start_pfn, start_pfn + spanned] of course.

Quoted 1 line

matches the number of onlined pages (256MB).

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#11 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 47 lines · Michal Hocko
Quoted 46 lines · Michal Hocko
Quoted 1 line · Wei Yang

On Fri, Jun 30, 2017 at 4:39 PM, Michal Hocko <[email protected]> wrote:

[...]

Quoted 8 lines
Quoted 5 lines

yes and to be honest I do not plan to fix it unless somebody has a real life usecase for it. Now that we allow explicit onlininig type anywhere it seems like a reasonable behavior and this will allow us to remove quite some code which is always a good deal wrt longterm maintenance.

hmm... the statistics displayed in /proc/zoneinfo would be meaningless for zone_normal and zone_movable.

Why would they be meaningless? Counters will always reflect the actual use - if not then it is a bug. And wrt to zone description what is meaningless about
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable memory36/valid_zones:Movable
memory37/valid_zones:Movable Normal memory38/valid_zones:Movable Normal memory39/valid_zones:Movable Normal memory40/valid_zones:Normal
memory41/valid_zones:Movable

And
Node 1, zone Normal

  pages free     65465
        min      156
        low      221
        high     286
        spanned  229376
        present  65536
        managed  65536
[...]
  start_pfn:           1114112
Node 1, zone  Movable
  pages free     65443
        min      156
        low      221
        high     286
        spanned  196608
        present  65536
        managed  65536

[...]
start_pfn: 1179648

Quoted 4 lines
Quoted 1 line

ranges are clearly defined as [start_pfn, start_pfn+managed] and managed

errr, this should be [start_pfn, start_pfn + spanned] of course.

The spanned is not adjusted after offline, neither does start_pfn. For example, even offline all the movable_zone range, we can still see the spanned.

Below is a result with a little changed kernel to show the start_pfn always. The sequence is:
1. bootup

Node 0, zone Movable

        spanned  65536
	present  0
	managed  0

start_pfn: 0

2. online movable 2 continuous memory_blocks

Node 0, zone Movable

        spanned  65536
	present  65536
	managed  65536

start_pfn: 1310720

3. offline 2nd memory_blocks

Node 0, zone Movable

        spanned  65536
	present  32768
	managed  32768

start_pfn: 1310720

4. offline 1st memory_blocks

Node 0, zone Movable

        spanned  65536
	present  0
	managed  0

start_pfn: 1310720

So I am not sure this is still clearly defined?

Quoted 5 lines
Quoted 1 line

matches the number of onlined pages (256MB).

Signature · 2 lines
Michal Hocko
SUSE Labs
Signature · 2 lines
Wei Yang
Help you, Help me
#12 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 48 lines · Wei Yang
Quoted 47 lines · Michal Hocko
Quoted 46 lines · Michal Hocko
Quoted 1 line · Wei Yang

On Fri, Jun 30, 2017 at 4:39 PM, Michal Hocko <[email protected]> wrote:

[...]

Quoted 8 lines
Quoted 5 lines

yes and to be honest I do not plan to fix it unless somebody has a real life usecase for it. Now that we allow explicit onlininig type anywhere it seems like a reasonable behavior and this will allow us to remove quite some code which is always a good deal wrt longterm maintenance.

hmm... the statistics displayed in /proc/zoneinfo would be meaningless for zone_normal and zone_movable.

Why would they be meaningless? Counters will always reflect the actual use - if not then it is a bug. And wrt to zone description what is meaningless about
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable memory36/valid_zones:Movable
memory37/valid_zones:Movable Normal memory38/valid_zones:Movable Normal memory39/valid_zones:Movable Normal memory40/valid_zones:Normal
memory41/valid_zones:Movable

And
Node 1, zone Normal

  pages free     65465
        min      156
        low      221
        high     286
        spanned  229376
        present  65536
        managed  65536
[...]
  start_pfn:           1114112
Node 1, zone  Movable
  pages free     65443
        min      156
        low      221
        high     286
        spanned  196608
        present  65536
        managed  65536

[...]
start_pfn: 1179648

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

ranges are clearly defined as [start_pfn, start_pfn+managed] and managed

errr, this should be [start_pfn, start_pfn + spanned] of course.

The spanned is not adjusted after offline, neither does start_pfn. For example, even offline all the movable_zone range, we can still see the spanned.

Which is completely valid. Offline only changes present/managed.

Quoted 28 lines

Below is a result with a little changed kernel to show the start_pfn always. The sequence is:
1. bootup

Node 0, zone Movable

        spanned  65536
	present  0
	managed  0

start_pfn: 0

2. online movable 2 continuous memory_blocks

Node 0, zone Movable

        spanned  65536
	present  65536
	managed  65536

start_pfn: 1310720

3. offline 2nd memory_blocks

Node 0, zone Movable

        spanned  65536
	present  32768
	managed  32768

start_pfn: 1310720

4. offline 1st memory_blocks

Quoted 7 lines

Node 0, zone Movable

        spanned  65536
	present  0
	managed  0

start_pfn: 1310720

So I am not sure this is still clearly defined?

Could you be more specific what is not clearly defined? You have offlined all online memory blocks so present/managed is 0 while the spanned is unchanged because the zone is still defined in range [1310720, 1376256].

I also do not see how this is related with the discussed patch as there is no zone interleaving involved.

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#13 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 34 lines · Michal Hocko
Quoted 34 lines

Below is a result with a little changed kernel to show the start_pfn always. The sequence is:
1. bootup

Node 0, zone Movable

        spanned  65536
	present  0
	managed  0

start_pfn: 0

2. online movable 2 continuous memory_blocks

Node 0, zone Movable

        spanned  65536
	present  65536
	managed  65536

start_pfn: 1310720

3. offline 2nd memory_blocks

Node 0, zone Movable

        spanned  65536
	present  32768
	managed  32768

start_pfn: 1310720

4. offline 1st memory_blocks

Node 0, zone Movable

        spanned  65536
	present  0
	managed  0

start_pfn: 1310720

Quoted 7 lines
Quoted 1 line

So I am not sure this is still clearly defined?

Could you be more specific what is not clearly defined? You have offlined all online memory blocks so present/managed is 0 while the spanned is unchanged because the zone is still defined in range [1310720, 1376256].

The zone is empty after remove these two memory blocks, while we still think it is defined in range [1310720, 1376256]. This is what I want to point.

Quoted 2 lines

I also do not see how this is related with the discussed patch as there is no zone interleaving involved.

I had a patch which fix the behavior, which means we can make sure the zone is empty after remove these two memory blocks. As you mentioned in the reply,
http://www.spinics.net/lists/linux-mm/msg130230.html, I thought you would have this fixed in this cycle. While it looks we will still have this behavior in this cycle and looks no intend to fix this?

Quoted 3 lines
Signature · 2 lines
Michal Hocko
SUSE Labs
Signature · 2 lines
Wei Yang
Help you, Help me
#14 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 37 lines · Wei Yang
Quoted 36 lines · Michal Hocko
Quoted 35 lines

Below is a result with a little changed kernel to show the start_pfn always. The sequence is:
1. bootup

Node 0, zone Movable

        spanned  65536
	present  0
	managed  0

start_pfn: 0

2. online movable 2 continuous memory_blocks

Node 0, zone Movable

        spanned  65536
	present  65536
	managed  65536

start_pfn: 1310720

3. offline 2nd memory_blocks

Node 0, zone Movable

        spanned  65536
	present  32768
	managed  32768

start_pfn: 1310720

4. offline 1st memory_blocks

Node 0, zone Movable

        spanned  65536
	present  0
	managed  0

start_pfn: 1310720

So I am not sure this is still clearly defined?

Quoted 8 lines
Quoted 5 lines

Could you be more specific what is not clearly defined? You have offlined all online memory blocks so present/managed is 0 while the spanned is unchanged because the zone is still defined in range [1310720, 1376256].

The zone is empty after remove these two memory blocks, while we still think it is defined in range [1310720, 1376256].

Yes and present/managed shows that the zone is empty. It's range spans some range but there are no online pages.

Quoted 1 line

This is what I want to point.

As I've said several times already. This is somemething that _could_ be fixed but I would rather not to do so until there is a _readl_ usecase which would depend on it. Especially when we can online any memory block to the zone you like. We should really strive to reduce the amount of code rather than keep it just in case without anybody actually using it.

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#15 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

In reply to Michal Hocko

[+CC linux-api]

Quoted 40 lines · Michal Hocko

From: Michal Hocko <[email protected]>

Historically we have enforced that any kernel zone (e.g ZONE_NORMAL) has to precede the Movable zone in the physical memory range. The purpose of the movable zone is, however, not bound to any physical memory restriction. It merely defines a class of migrateable and reclaimable memory.

There are users (e.g. CMA) who might want to reserve specific physical memory ranges for their own purpose. Moreover our pfn walkers have to be prepared for zones overlapping in the physical range already because we do support interleaving NUMA nodes and therefore zones can interleave as well. This means we can allow each memory block to be associated with a different zone.

Loosen the current onlining semantic and allow explicit onlining type on any memblock. That means that online_{kernel,movable} will be allowed regardless of the physical address of the memblock as long as it is offline of course. This might result in moveble zone overlapping with other kernel zones. Default onlining then becomes a bit tricky but still sensible. echo online > memoryXY/state will online the given block to
1) the default zone if the given range is outside of any zone
2) the enclosing zone if such a zone doesn't interleave with any other zone
3) the default zone if more zones interleave for this range
where default zone is movable zone only if movable_node is enabled otherwise it is a kernel zone.

Here is an example of the semantic with (movable_node is not present but it work in an analogous way). We start with following memblocks, all of them offline
memory34/valid_zones:Normal Movable memory35/valid_zones:Normal Movable memory36/valid_zones:Normal Movable memory37/valid_zones:Normal Movable memory38/valid_zones:Normal Movable memory39/valid_zones:Normal Movable memory40/valid_zones:Normal Movable memory41/valid_zones:Normal Movable

Now, we online block 34 in default mode and block 37 as movable

Quoted 10 lines
root@test1:/sys/devices/system/node/node1# echo online > memory34/state
root@test1:/sys/devices/system/node/node1# echo online_movable > memory37/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal Movable
memory40/valid_zones:Normal Movable
memory41/valid_zones:Normal Movable

Hm so previously, blocks 37-41 would only allow Movable at this point, right? Shouldn't we still default to Movable for them? We might be breaking some existing userspace here.
IMHO onlining new memory past existing blocks is more common use case than onlining memory between two blocks that are already online?

I also agree with Wei Yang that it's rather fuzzy that a zone that has been completely offlined will affect the defaults for the next onlining just because it has some spanned range, which is however empty of actual populated memory.

Maybe it would simplest for everyone to just default to Normal, except movable_node? That's if we decide that the potential breakage I described above is a non-issue.

Quoted 174 lines

As we can see all other blocks can still be onlined both into Normal and Movable zones and the Normal is default because the Movable zone spans only block37 now.

root@test1:/sys/devices/system/node/node1# echo online_movable > memory41/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Movable Normal
memory39/valid_zones:Movable Normal
memory40/valid_zones:Movable Normal
memory41/valid_zones:Movable

Now the default zone for blocks 37-41 has changed because movable zone spans that range.

root@test1:/sys/devices/system/node/node1# echo online_kernel > memory39/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal
memory40/valid_zones:Movable Normal
memory41/valid_zones:Movable

Note that the block 39 now belongs to the zone Normal and so block38 falls into Normal by default as well.

For completness

root@test1:/sys/devices/system/node/node1# for i in memory[34]?
do
	echo online > $i/state 2>/dev/null
done

memory34/valid_zones:Normal
memory35/valid_zones:Normal
memory36/valid_zones:Normal
memory37/valid_zones:Movable
memory38/valid_zones:Normal
memory39/valid_zones:Normal
memory40/valid_zones:Movable
memory41/valid_zones:Movable

Implementation wise the change is quite straightforward. We can get rid of allow_online_pfn_range altogether. online_pages allows only offline nodes already. The original default_zone_for_pfn will become default_kernel_zone_for_pfn. New default_zone_for_pfn implements the above semantic. zone_for_pfn_range is slightly reorganized to implement kernel and movable online type explicitly and MMOP_ONLINE_KEEP becomes a catch all default behavior.

---
 drivers/base/memory.c |  3 ---
 mm/memory_hotplug.c   | 74 ++++++++++++++++-----------------------------------
 2 files changed, 23 insertions(+), 54 deletions(-)
drivers/base/memory.c +0 -3
index 26383af9900c..4e3b61cda520 100644
@@ -394,9 +394,6 @@ static void print_allowed_zone(char *buf, int nid, unsigned long start_pfn,
 {
 	struct zone *zone;
 
-	if (!allow_online_pfn_range(nid, start_pfn, nr_pages, online_type))
-		return;
-
 	zone = zone_for_pfn_range(online_type, nid, start_pfn, nr_pages);
 	if (zone != default_zone) {
 		strcat(buf, " ");
mm/memory_hotplug.c +23 -48
index 6b9a60115e37..670f7acbecf4 100644
@@ -894,7 +894,7 @@ void __ref move_pfn_range_to_zone(struct zone *zone,
  * If no kernel zone covers this pfn range it will automatically go
  * to the ZONE_NORMAL.
  */
-static struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn,
+static struct zone *default_kernel_zone_for_pfn(int nid, unsigned long start_pfn,
 		unsigned long nr_pages)
 {
 	struct pglist_data *pgdat = NODE_DATA(nid);
@@ -910,65 +910,40 @@ static struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn,
 	return &pgdat->node_zones[ZONE_NORMAL];
 }
 
-bool allow_online_pfn_range(int nid, unsigned long pfn, unsigned long nr_pages, int online_type)
+static inline struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn,
+		unsigned long nr_pages)
 {
-	struct pglist_data *pgdat = NODE_DATA(nid);
-	struct zone *movable_zone = &pgdat->node_zones[ZONE_MOVABLE];
-	struct zone *default_zone = default_zone_for_pfn(nid, pfn, nr_pages);
+	struct zone *kernel_zone = default_kernel_zone_for_pfn(nid, start_pfn,
+			nr_pages);
+	struct zone *movable_zone = &NODE_DATA(nid)->node_zones[ZONE_MOVABLE];
+	bool in_kernel = zone_intersects(kernel_zone, start_pfn, nr_pages);
+	bool in_movable = zone_intersects(movable_zone, start_pfn, nr_pages);
 
 	/*
-	 * TODO there shouldn't be any inherent reason to have ZONE_NORMAL
-	 * physically before ZONE_MOVABLE. All we need is they do not
-	 * overlap. Historically we didn't allow ZONE_NORMAL after ZONE_MOVABLE
-	 * though so let's stick with it for simplicity for now.
-	 * TODO make sure we do not overlap with ZONE_DEVICE
+	 * We inherit the existing zone in a simple case where zones do not
+	 * overlap in the given range
 	 */
-	if (online_type == MMOP_ONLINE_KERNEL) {
-		if (zone_is_empty(movable_zone))
-			return true;
-		return movable_zone->zone_start_pfn >= pfn + nr_pages;
-	} else if (online_type == MMOP_ONLINE_MOVABLE) {
-		return zone_end_pfn(default_zone) <= pfn;
-	}
-
-	/* MMOP_ONLINE_KEEP will always succeed and inherits the current zone */
-	return online_type == MMOP_ONLINE_KEEP;
-}
-
-static inline bool movable_pfn_range(int nid, struct zone *default_zone,
-		unsigned long start_pfn, unsigned long nr_pages)
-{
-	if (!allow_online_pfn_range(nid, start_pfn, nr_pages,
-				MMOP_ONLINE_KERNEL))
-		return true;
-
-	if (!movable_node_is_enabled())
-		return false;
+	if (in_kernel ^ in_movable)
+		return (in_kernel) ? kernel_zone : movable_zone;
 
-	return !zone_intersects(default_zone, start_pfn, nr_pages);
+	/*
+	 * If the range doesn't belong to any zone or two zones overlap in the
+	 * given range then we use movable zone only if movable_node is
+	 * enabled because we always online to a kernel zone by default.
+	 */
+	return movable_node_enabled ? movable_zone : kernel_zone;
 }
 
 struct zone * zone_for_pfn_range(int online_type, int nid, unsigned start_pfn,
 		unsigned long nr_pages)
 {
-	struct pglist_data *pgdat = NODE_DATA(nid);
-	struct zone *zone = default_zone_for_pfn(nid, start_pfn, nr_pages);
+	if (online_type == MMOP_ONLINE_KERNEL)
+		return default_kernel_zone_for_pfn(nid, start_pfn, nr_pages);
 
-	if (online_type == MMOP_ONLINE_KEEP) {
-		struct zone *movable_zone = &pgdat->node_zones[ZONE_MOVABLE];
-		/*
-		 * MMOP_ONLINE_KEEP defaults to MMOP_ONLINE_KERNEL but use
-		 * movable zone if that is not possible (e.g. we are within
-		 * or past the existing movable zone). movable_node overrides
-		 * this default and defaults to movable zone
-		 */
-		if (movable_pfn_range(nid, zone, start_pfn, nr_pages))
-			zone = movable_zone;
-	} else if (online_type == MMOP_ONLINE_MOVABLE) {
-		zone = &pgdat->node_zones[ZONE_MOVABLE];
-	}
+	if (online_type == MMOP_ONLINE_MOVABLE)
+		return &NODE_DATA(nid)->node_zones[ZONE_MOVABLE];
 
-	return zone;
+	return default_zone_for_pfn(nid, start_pfn, nr_pages);
 }
 
 /*

mm/memory_hotplug.c int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ · line 977

Quoted 9 lines
mm/memory_hotplug.c +0 -3
@@ -997,9 +972,6 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
 	struct memory_notify arg;
 
 	nid = pfn_to_nid(pfn);
-	if (!allow_online_pfn_range(nid, pfn, nr_pages, online_type))
-		return -EINVAL;
-
 	/* associate pfn range with the zone */
 	zone = move_pfn_range(online_type, nid, pfn, nr_pages);
 

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#16 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 54 lines · Vlastimil Babka

[+CC linux-api]

Quoted 40 lines · Michal Hocko

From: Michal Hocko <[email protected]>

Historically we have enforced that any kernel zone (e.g ZONE_NORMAL) has to precede the Movable zone in the physical memory range. The purpose of the movable zone is, however, not bound to any physical memory restriction. It merely defines a class of migrateable and reclaimable memory.

There are users (e.g. CMA) who might want to reserve specific physical memory ranges for their own purpose. Moreover our pfn walkers have to be prepared for zones overlapping in the physical range already because we do support interleaving NUMA nodes and therefore zones can interleave as well. This means we can allow each memory block to be associated with a different zone.

Loosen the current onlining semantic and allow explicit onlining type on any memblock. That means that online_{kernel,movable} will be allowed regardless of the physical address of the memblock as long as it is offline of course. This might result in moveble zone overlapping with other kernel zones. Default onlining then becomes a bit tricky but still sensible. echo online > memoryXY/state will online the given block to
1) the default zone if the given range is outside of any zone
2) the enclosing zone if such a zone doesn't interleave with any other zone
3) the default zone if more zones interleave for this range
where default zone is movable zone only if movable_node is enabled otherwise it is a kernel zone.

Here is an example of the semantic with (movable_node is not present but it work in an analogous way). We start with following memblocks, all of them offline
memory34/valid_zones:Normal Movable memory35/valid_zones:Normal Movable memory36/valid_zones:Normal Movable memory37/valid_zones:Normal Movable memory38/valid_zones:Normal Movable memory39/valid_zones:Normal Movable memory40/valid_zones:Normal Movable memory41/valid_zones:Normal Movable

Now, we online block 34 in default mode and block 37 as movable

Quoted 10 lines
root@test1:/sys/devices/system/node/node1# echo online > memory34/state
root@test1:/sys/devices/system/node/node1# echo online_movable > memory37/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal Movable
memory40/valid_zones:Normal Movable
memory41/valid_zones:Normal Movable
Quoted 1 line

Hm so previously, blocks 37-41 would only allow Movable at this point, right?

yes

Quoted 2 lines

Shouldn't we still default to Movable for them? We might be breaking some existing userspace here.

I do not think so. Prior to this merge window f1dd2cd13c4b ("mm,
memory_hotplug: do not associate hotadded memory to zones until online") we allowed only the last offline or the adjacent to existing movable memory block to be onlined movable. So the above wasn't possible. I doubt we have grown a new user since the rework has been merged but if you think we should make sure nothing like that happens then we should probably merge this patch in this release cycle.

Quoted 2 lines

IMHO onlining new memory past existing blocks is more common use case than onlining memory between two blocks that are already online?

I am not really sure. It is quite common to online and offline within an existing zones for the memory ballooning. I do not know what kind of online operation they use but using the default online operation has historically preserved the zone so I would be really reluctant to change that.

Quoted 3 lines

I also agree with Wei Yang that it's rather fuzzy that a zone that has been completely offlined will affect the defaults for the next onlining just because it has some spanned range, which is however empty of actual populated memory.

I am sorry but I still do not see why. The zone is not empty. It has a range spanned. It just doesn't have any pages online. I really fail to see how that is different from zones with large offline holes.

Quoted 3 lines

Maybe it would simplest for everyone to just default to Normal, except movable_node? That's if we decide that the potential breakage I described above is a non-issue.

This would break the usecase where the memory is onlined a certain type initially and the offline/online it later on demand for ballooning.

I wish this could be more clear but the default onlining has been fuzzy since the movable online has been introduced and it is hard to buil something really clear since then. The proposed semantic is the most clean I could come up with but I am open to any suggestions that wouldn't break existing usage.

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#17 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 59 lines · Michal Hocko
Quoted 54 lines · Vlastimil Babka

[+CC linux-api]

Quoted 40 lines · Michal Hocko

From: Michal Hocko <[email protected]>

Historically we have enforced that any kernel zone (e.g ZONE_NORMAL) has to precede the Movable zone in the physical memory range. The purpose of the movable zone is, however, not bound to any physical memory restriction. It merely defines a class of migrateable and reclaimable memory.

There are users (e.g. CMA) who might want to reserve specific physical memory ranges for their own purpose. Moreover our pfn walkers have to be prepared for zones overlapping in the physical range already because we do support interleaving NUMA nodes and therefore zones can interleave as well. This means we can allow each memory block to be associated with a different zone.

Loosen the current onlining semantic and allow explicit onlining type on any memblock. That means that online_{kernel,movable} will be allowed regardless of the physical address of the memblock as long as it is offline of course. This might result in moveble zone overlapping with other kernel zones. Default onlining then becomes a bit tricky but still sensible. echo online > memoryXY/state will online the given block to
1) the default zone if the given range is outside of any zone
2) the enclosing zone if such a zone doesn't interleave with any other zone
3) the default zone if more zones interleave for this range
where default zone is movable zone only if movable_node is enabled otherwise it is a kernel zone.

Here is an example of the semantic with (movable_node is not present but it work in an analogous way). We start with following memblocks, all of them offline
memory34/valid_zones:Normal Movable memory35/valid_zones:Normal Movable memory36/valid_zones:Normal Movable memory37/valid_zones:Normal Movable memory38/valid_zones:Normal Movable memory39/valid_zones:Normal Movable memory40/valid_zones:Normal Movable memory41/valid_zones:Normal Movable

Now, we online block 34 in default mode and block 37 as movable

Quoted 10 lines
root@test1:/sys/devices/system/node/node1# echo online > memory34/state
root@test1:/sys/devices/system/node/node1# echo online_movable > memory37/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal Movable
memory40/valid_zones:Normal Movable
memory41/valid_zones:Normal Movable
Quoted 1 line

Hm so previously, blocks 37-41 would only allow Movable at this point, right?

yes

Quoted 7 lines
Quoted 2 lines

Shouldn't we still default to Movable for them? We might be breaking some existing userspace here.

I do not think so. Prior to this merge window f1dd2cd13c4b ("mm,
memory_hotplug: do not associate hotadded memory to zones until online") we allowed only the last offline or the adjacent to existing movable memory block to be onlined movable. So the above wasn't possible.

Not exactly the above, but let's say 1-34 is onlined as Normal, 35-37 is Movable. Then the only possible action before would be online 38 as Movable? Now it defaults to Normal?

Quoted 4 lines

I
doubt we have grown a new user since the rework has been merged but if you think we should make sure nothing like that happens then we should probably merge this patch in this release cycle.

If I'm right and this is a change compared to pre-rework, then it doesn't matter.

Quoted 8 lines
Quoted 2 lines

IMHO onlining new memory past existing blocks is more common use case than onlining memory between two blocks that are already online?

I am not really sure. It is quite common to online and offline within an existing zones for the memory ballooning. I do not know what kind of online operation they use but using the default online operation has historically preserved the zone so I would be really reluctant to change that.

Hmm all right, ballooning...

Quoted 12 lines
Quoted 3 lines

I also agree with Wei Yang that it's rather fuzzy that a zone that has been completely offlined will affect the defaults for the next onlining just because it has some spanned range, which is however empty of actual populated memory.

I am sorry but I still do not see why. The zone is not empty. It has a range spanned. It just doesn't have any pages online. I really fail to see how that is different from zones with large offline holes.

Quoted 3 lines

Maybe it would simplest for everyone to just default to Normal, except movable_node? That's if we decide that the potential breakage I described above is a non-issue.

Quoted 8 lines

This would break the usecase where the memory is onlined a certain type initially and the offline/online it later on demand for ballooning.

I wish this could be more clear but the default onlining has been fuzzy since the movable online has been introduced and it is hard to buil something really clear since then. The proposed semantic is the most clean I could come up with but I am open to any suggestions that wouldn't break existing usage.

OK I can live with the semantics, if we clear question of breaking existing users.

#18 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 63 lines · Vlastimil Babka
Quoted 62 lines · Michal Hocko
Quoted 54 lines · Vlastimil Babka

[+CC linux-api]

Quoted 40 lines · Michal Hocko

From: Michal Hocko <[email protected]>

Historically we have enforced that any kernel zone (e.g ZONE_NORMAL) has to precede the Movable zone in the physical memory range. The purpose of the movable zone is, however, not bound to any physical memory restriction. It merely defines a class of migrateable and reclaimable memory.

There are users (e.g. CMA) who might want to reserve specific physical memory ranges for their own purpose. Moreover our pfn walkers have to be prepared for zones overlapping in the physical range already because we do support interleaving NUMA nodes and therefore zones can interleave as well. This means we can allow each memory block to be associated with a different zone.

Loosen the current onlining semantic and allow explicit onlining type on any memblock. That means that online_{kernel,movable} will be allowed regardless of the physical address of the memblock as long as it is offline of course. This might result in moveble zone overlapping with other kernel zones. Default onlining then becomes a bit tricky but still sensible. echo online > memoryXY/state will online the given block to
1) the default zone if the given range is outside of any zone
2) the enclosing zone if such a zone doesn't interleave with any other zone
3) the default zone if more zones interleave for this range
where default zone is movable zone only if movable_node is enabled otherwise it is a kernel zone.

Here is an example of the semantic with (movable_node is not present but it work in an analogous way). We start with following memblocks, all of them offline
memory34/valid_zones:Normal Movable memory35/valid_zones:Normal Movable memory36/valid_zones:Normal Movable memory37/valid_zones:Normal Movable memory38/valid_zones:Normal Movable memory39/valid_zones:Normal Movable memory40/valid_zones:Normal Movable memory41/valid_zones:Normal Movable

Now, we online block 34 in default mode and block 37 as movable

Quoted 10 lines
root@test1:/sys/devices/system/node/node1# echo online > memory34/state
root@test1:/sys/devices/system/node/node1# echo online_movable > memory37/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal Movable
memory40/valid_zones:Normal Movable
memory41/valid_zones:Normal Movable
Quoted 1 line

Hm so previously, blocks 37-41 would only allow Movable at this point, right?

yes

Quoted 2 lines

Shouldn't we still default to Movable for them? We might be breaking some existing userspace here.

Quoted 8 lines
Quoted 4 lines

I do not think so. Prior to this merge window f1dd2cd13c4b ("mm,
memory_hotplug: do not associate hotadded memory to zones until online") we allowed only the last offline or the adjacent to existing movable memory block to be onlined movable. So the above wasn't possible.

Not exactly the above, but let's say 1-34 is onlined as Normal, 35-37 is Movable. Then the only possible action before would be online 38 as Movable? Now it defaults to Normal?

Yes. And let me repeat you couldn't onlne 35-37 as movable before. So no userspace could depend on that before the rework. Or do I still miss your point?

Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#19 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 18 lines · Michal Hocko
Quoted 17 lines · Vlastimil Babka
Quoted 15 lines · Michal Hocko
Quoted 4 lines · Vlastimil Babka

[+CC linux-api]

Hm so previously, blocks 37-41 would only allow Movable at this point, right?

yes

Quoted 2 lines

Shouldn't we still default to Movable for them? We might be breaking some existing userspace here.

I do not think so. Prior to this merge window f1dd2cd13c4b ("mm,
memory_hotplug: do not associate hotadded memory to zones until online") we allowed only the last offline or the adjacent to existing movable memory block to be onlined movable. So the above wasn't possible.

Quoted 7 lines
Quoted 3 lines

Not exactly the above, but let's say 1-34 is onlined as Normal, 35-37 is Movable. Then the only possible action before would be online 38 as Movable? Now it defaults to Normal?

Yes. And let me repeat you couldn't onlne 35-37 as movable before. So no userspace could depend on that before the rework. Or do I still miss your point?

Ah, I see. "the last offline or the adjacent to existing movable". OK then.

It would be indeed better to not change behavour twice then and merge this to 4.13, but it's the middle of merge window, so it's not simple...

#20 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

Quoted 23 lines · Vlastimil Babka
Quoted 22 lines · Michal Hocko
Quoted 20 lines · Vlastimil Babka
Quoted 8 lines · Michal Hocko
Quoted 4 lines · Vlastimil Babka

[+CC linux-api]

Hm so previously, blocks 37-41 would only allow Movable at this point, right?

yes

Quoted 7 lines
Quoted 2 lines

Shouldn't we still default to Movable for them? We might be breaking some existing userspace here.

I do not think so. Prior to this merge window f1dd2cd13c4b ("mm,
memory_hotplug: do not associate hotadded memory to zones until online") we allowed only the last offline or the adjacent to existing movable memory block to be onlined movable. So the above wasn't possible.

Not exactly the above, but let's say 1-34 is onlined as Normal, 35-37 is Movable. Then the only possible action before would be online 38 as Movable? Now it defaults to Normal?

Quoted 8 lines
Quoted 3 lines

Yes. And let me repeat you couldn't onlne 35-37 as movable before. So no userspace could depend on that before the rework. Or do I still miss your point?

Ah, I see. "the last offline or the adjacent to existing movable". OK then.

It would be indeed better to not change behavour twice then and merge this to 4.13, but it's the middle of merge window, so it's not simple...

yeah. I was thinking about about how to make the change reasonably incremental but failed to find a way. I also didn't want to bring too many changes at once (the code base is just too fragile already).

If there is a general consensus about the semantic we might want to push the patch this week. I just do not want to rush it too much as this is a users visible change and it might kick us back in future.

Signature · 2 lines
Michal Hocko
SUSE Labs
#21 lore raw

Re: [PATCH 2/2] mm, memory_hotplug: remove zone restrictions

In reply to Michal Hocko

Are there any other concerns regarding this patch? Can I repost it for inclusion?

Quoted 236 lines · Michal Hocko

From: Michal Hocko <[email protected]>

Historically we have enforced that any kernel zone (e.g ZONE_NORMAL) has to precede the Movable zone in the physical memory range. The purpose of the movable zone is, however, not bound to any physical memory restriction. It merely defines a class of migrateable and reclaimable memory.

There are users (e.g. CMA) who might want to reserve specific physical memory ranges for their own purpose. Moreover our pfn walkers have to be prepared for zones overlapping in the physical range already because we do support interleaving NUMA nodes and therefore zones can interleave as well. This means we can allow each memory block to be associated with a different zone.

Loosen the current onlining semantic and allow explicit onlining type on any memblock. That means that online_{kernel,movable} will be allowed regardless of the physical address of the memblock as long as it is offline of course. This might result in moveble zone overlapping with other kernel zones. Default onlining then becomes a bit tricky but still sensible. echo online > memoryXY/state will online the given block to
1) the default zone if the given range is outside of any zone
2) the enclosing zone if such a zone doesn't interleave with any other zone
3) the default zone if more zones interleave for this range
where default zone is movable zone only if movable_node is enabled otherwise it is a kernel zone.

Here is an example of the semantic with (movable_node is not present but it work in an analogous way). We start with following memblocks, all of them offline
memory34/valid_zones:Normal Movable memory35/valid_zones:Normal Movable memory36/valid_zones:Normal Movable memory37/valid_zones:Normal Movable memory38/valid_zones:Normal Movable memory39/valid_zones:Normal Movable memory40/valid_zones:Normal Movable memory41/valid_zones:Normal Movable

Now, we online block 34 in default mode and block 37 as movable

root@test1:/sys/devices/system/node/node1# echo online > memory34/state
root@test1:/sys/devices/system/node/node1# echo online_movable > memory37/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal Movable
memory40/valid_zones:Normal Movable
memory41/valid_zones:Normal Movable

As we can see all other blocks can still be onlined both into Normal and Movable zones and the Normal is default because the Movable zone spans only block37 now.

root@test1:/sys/devices/system/node/node1# echo online_movable > memory41/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Movable Normal
memory39/valid_zones:Movable Normal
memory40/valid_zones:Movable Normal
memory41/valid_zones:Movable

Now the default zone for blocks 37-41 has changed because movable zone spans that range.

root@test1:/sys/devices/system/node/node1# echo online_kernel > memory39/state
memory34/valid_zones:Normal
memory35/valid_zones:Normal Movable
memory36/valid_zones:Normal Movable
memory37/valid_zones:Movable
memory38/valid_zones:Normal Movable
memory39/valid_zones:Normal
memory40/valid_zones:Movable Normal
memory41/valid_zones:Movable

Note that the block 39 now belongs to the zone Normal and so block38 falls into Normal by default as well.

For completness

root@test1:/sys/devices/system/node/node1# for i in memory[34]?
do
	echo online > $i/state 2>/dev/null
done

memory34/valid_zones:Normal
memory35/valid_zones:Normal
memory36/valid_zones:Normal
memory37/valid_zones:Movable
memory38/valid_zones:Normal
memory39/valid_zones:Normal
memory40/valid_zones:Movable
memory41/valid_zones:Movable

Implementation wise the change is quite straightforward. We can get rid of allow_online_pfn_range altogether. online_pages allows only offline nodes already. The original default_zone_for_pfn will become default_kernel_zone_for_pfn. New default_zone_for_pfn implements the above semantic. zone_for_pfn_range is slightly reorganized to implement kernel and movable online type explicitly and MMOP_ONLINE_KEEP becomes a catch all default behavior.

---
 drivers/base/memory.c |  3 ---
 mm/memory_hotplug.c   | 74 ++++++++++++++++-----------------------------------
 2 files changed, 23 insertions(+), 54 deletions(-)
drivers/base/memory.c +0 -3
index 26383af9900c..4e3b61cda520 100644
@@ -394,9 +394,6 @@ static void print_allowed_zone(char *buf, int nid, unsigned long start_pfn,
 {
 	struct zone *zone;
 
-	if (!allow_online_pfn_range(nid, start_pfn, nr_pages, online_type))
-		return;
-
 	zone = zone_for_pfn_range(online_type, nid, start_pfn, nr_pages);
 	if (zone != default_zone) {
 		strcat(buf, " ");
mm/memory_hotplug.c +23 -51
index 6b9a60115e37..670f7acbecf4 100644
@@ -894,7 +894,7 @@ void __ref move_pfn_range_to_zone(struct zone *zone,
  * If no kernel zone covers this pfn range it will automatically go
  * to the ZONE_NORMAL.
  */
-static struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn,
+static struct zone *default_kernel_zone_for_pfn(int nid, unsigned long start_pfn,
 		unsigned long nr_pages)
 {
 	struct pglist_data *pgdat = NODE_DATA(nid);
@@ -910,65 +910,40 @@ static struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn,
 	return &pgdat->node_zones[ZONE_NORMAL];
 }
 
-bool allow_online_pfn_range(int nid, unsigned long pfn, unsigned long nr_pages, int online_type)
+static inline struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn,
+		unsigned long nr_pages)
 {
-	struct pglist_data *pgdat = NODE_DATA(nid);
-	struct zone *movable_zone = &pgdat->node_zones[ZONE_MOVABLE];
-	struct zone *default_zone = default_zone_for_pfn(nid, pfn, nr_pages);
+	struct zone *kernel_zone = default_kernel_zone_for_pfn(nid, start_pfn,
+			nr_pages);
+	struct zone *movable_zone = &NODE_DATA(nid)->node_zones[ZONE_MOVABLE];
+	bool in_kernel = zone_intersects(kernel_zone, start_pfn, nr_pages);
+	bool in_movable = zone_intersects(movable_zone, start_pfn, nr_pages);
 
 	/*
-	 * TODO there shouldn't be any inherent reason to have ZONE_NORMAL
-	 * physically before ZONE_MOVABLE. All we need is they do not
-	 * overlap. Historically we didn't allow ZONE_NORMAL after ZONE_MOVABLE
-	 * though so let's stick with it for simplicity for now.
-	 * TODO make sure we do not overlap with ZONE_DEVICE
+	 * We inherit the existing zone in a simple case where zones do not
+	 * overlap in the given range
 	 */
-	if (online_type == MMOP_ONLINE_KERNEL) {
-		if (zone_is_empty(movable_zone))
-			return true;
-		return movable_zone->zone_start_pfn >= pfn + nr_pages;
-	} else if (online_type == MMOP_ONLINE_MOVABLE) {
-		return zone_end_pfn(default_zone) <= pfn;
-	}
-
-	/* MMOP_ONLINE_KEEP will always succeed and inherits the current zone */
-	return online_type == MMOP_ONLINE_KEEP;
-}
-
-static inline bool movable_pfn_range(int nid, struct zone *default_zone,
-		unsigned long start_pfn, unsigned long nr_pages)
-{
-	if (!allow_online_pfn_range(nid, start_pfn, nr_pages,
-				MMOP_ONLINE_KERNEL))
-		return true;
-
-	if (!movable_node_is_enabled())
-		return false;
+	if (in_kernel ^ in_movable)
+		return (in_kernel) ? kernel_zone : movable_zone;
 
-	return !zone_intersects(default_zone, start_pfn, nr_pages);
+	/*
+	 * If the range doesn't belong to any zone or two zones overlap in the
+	 * given range then we use movable zone only if movable_node is
+	 * enabled because we always online to a kernel zone by default.
+	 */
+	return movable_node_enabled ? movable_zone : kernel_zone;
 }
 
 struct zone * zone_for_pfn_range(int online_type, int nid, unsigned start_pfn,
 		unsigned long nr_pages)
 {
-	struct pglist_data *pgdat = NODE_DATA(nid);
-	struct zone *zone = default_zone_for_pfn(nid, start_pfn, nr_pages);
+	if (online_type == MMOP_ONLINE_KERNEL)
+		return default_kernel_zone_for_pfn(nid, start_pfn, nr_pages);
 
-	if (online_type == MMOP_ONLINE_KEEP) {
-		struct zone *movable_zone = &pgdat->node_zones[ZONE_MOVABLE];
-		/*
-		 * MMOP_ONLINE_KEEP defaults to MMOP_ONLINE_KERNEL but use
-		 * movable zone if that is not possible (e.g. we are within
-		 * or past the existing movable zone). movable_node overrides
-		 * this default and defaults to movable zone
-		 */
-		if (movable_pfn_range(nid, zone, start_pfn, nr_pages))
-			zone = movable_zone;
-	} else if (online_type == MMOP_ONLINE_MOVABLE) {
-		zone = &pgdat->node_zones[ZONE_MOVABLE];
-	}
+	if (online_type == MMOP_ONLINE_MOVABLE)
+		return &NODE_DATA(nid)->node_zones[ZONE_MOVABLE];
 
-	return zone;
+	return default_zone_for_pfn(nid, start_pfn, nr_pages);
 }
 
 /*
@@ -997,9 +972,6 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
 	struct memory_notify arg;
 
 	nid = pfn_to_nid(pfn);
-	if (!allow_online_pfn_range(nid, pfn, nr_pages, online_type))
-		return -EINVAL;
-
 	/* associate pfn range with the zone */
 	zone = move_pfn_range(online_type, nid, pfn, nr_pages);
 
Signature · 1 line
2.11.0
Signature · 8 lines
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>