Skip to contentSkip to conversation

drivers/base/memory.c: indicate all memory blocks as removable

RFC PATCH In review David Hildenbrand 3 participants 8 messages Jan 24, 2020 – Jan 27, 2020

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 drivers/base/memory.c: indicate all memory blocks as removable

We see multiple issues with the implementation/interface to compute whether a memory block can be offlined (exposed via /sys/devices/system/memory/memoryX/removable) and would like to simplify it (remove the implementation).

1. It runs basically lockless. While this might be good for performance, we see possible races with memory offlining/unplug that will require at least some sort of locking to fix.

2. Nowadays, more false positives are possible. No arch-specific checks are performed that validate if memory offlining will not be denied right away (and such check will require locking). For example, arm64 won't allow to offline any memory block that was added during boot - which will imply a very high error rate. Other archs have other constraints.

3. The interface is inherently racy. E.g., if a memory block is detected to be removable (and was not a false positive at that time), there is still no guarantee that offlining will actually succeed. So any caller already has to deal with false positives.

4. It is unclear which performance benefit this interface actually provides. The introducing commit 5c755e9fd813 ("memory-hotplug: add sysfs removable attribute for hotplug memory remove") mentioned "A user-level agent must be able to identify which sections of memory are likely to be removable before attempting the potentially expensive operation."
However, no actual performance comparison was included.

Known users:
- lsmem: Will group memory blocks based on the "removable" property. [1]
- chmem: Indirect user. It has a RANGE mode where one can specify removable ranges identified via lsmem to be offlined. However, it also has a "SIZE" mode, which allows a sysadmin to skip the manual "identify removable blocks" step. [2]
- powerpc-utils: Uses the "removable" attribute to skip some memory blocks right away when trying to find some to offline+remove. However, with ballooning enabled, it already skips this information completely (because it once resulted in many false negatives). Therefore, the implementation can deal with false positives properly already. [3]

With CONFIG_MEMORY_HOTREMOVE, always indicating "removable" should not break any user space tool. We implement a very bad heuristic now. (in
contrast: always returning "not removable" would at least affect powerpc-utils)

Without CONFIG_MEMORY_HOTREMOVE we cannot offline anything, so report "not removable" as before.

Original discussion can be found in [4] ("[PATCH RFC v1] mm:
is_mem_section_removable() overhaul").

Other users of is_mem_section_removable() will be removed next, so that we can remove is_mem_section_removable() completely.

[1] http://man7.org/linux/man-pages/man1/lsmem.1.html
[2] http://man7.org/linux/man-pages/man8/chmem.8.html
[3] https://github.com/ibm-power-utilities/powerpc-utils
[4] https://lkml.kernel.org/r/[email protected]

Cc Dan Williams <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc "Rafael J. Wysocki" <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Badari Pulavarty <[email protected]>
Cc Nathan Fontenot <[email protected]>
Cc Robert Jennings <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Karel Zak <[email protected]>

---

1. Are there any use cases that really require this interface to keep producing "more reliable" results?

2. Is there any real performance advantage when using this interface to identify memory blocks to offline?

---
 drivers/base/memory.c | 27 +++++++--------------------
 1 file changed, 7 insertions(+), 20 deletions(-)
drivers/base/memory.c +7 -20
index 6503f5d0b749..d78a92f09984 100644
@@ -105,30 +105,17 @@ static ssize_t phys_index_show(struct device *dev,
 }
 
 /*
- * Show whether the memory block is likely to be offlineable (or is already
- * offline). Once offline, the memory block could be removed. The return
- * value does, however, not indicate that there is a way to remove the
- * memory block.
+ * Legacy interface that we cannot remove. Always indicate "removable"
+ * with CONFIG_MEMORY_HOTREMOVE - bad heuristic.
  */
 static ssize_t removable_show(struct device *dev, struct device_attribute *attr,
 			      char *buf)
 {
-	struct memory_block *mem = to_memory_block(dev);
-	unsigned long pfn;
-	int ret = 1, i;
-
-	if (mem->state != MEM_ONLINE)
-		goto out;
-
-	for (i = 0; i < sections_per_block; i++) {
-		if (!present_section_nr(mem->start_section_nr + i))
-			continue;
-		pfn = section_nr_to_pfn(mem->start_section_nr + i);
-		ret &= is_mem_section_removable(pfn, PAGES_PER_SECTION);
-	}
-
-out:
-	return sprintf(buf, "%d\n", ret);
+#ifdef CONFIG_MEMORY_HOTREMOVE
+	return sprintf(buf, "1\n");
+#else
+	return sprintf(buf, "0\n");
+#endif
 }
 
 /*
Signature · 1 line
2.24.1
#1 lore raw

It's been awhile since I've looked at the powerpc-utils drmgr command and pseries DLPAR code but a quick scan makes and it appears that it hasn't changed too much. Given that, some thoughts.

The sysfs 'removable' file was a great help when memory DLPAR was driven from userspace in the powerpc-utils drmgr command. Having this check did improve performance though I can't point to any numbers.

Currently, memory DLPAR is done completely in the kernel. The request is initiated from drmgr writing to /sys/kernel/dlpar (for pHyp partitions) or from a hotplug interrupt (for guests). I don't believe the 'removable' sysfs file is used in either of these paths by drmgr. The only time it is used is on older kernels that do not support in-kernel memory DLPAR.

Given this, I don't think removing the 'removable' sysfs file would cause any issues for the drmgr command. The only scenario I can think of is using an old version of drmgr that does not support in-kernel memory DLPAR on a new kernel where the 'removable' sysfs file has been removed. This doesn't seem likely though and drmgr could be updated to detect this.

-Nathan Fontenot

Quoted 131 lines · David Hildenbrand

We see multiple issues with the implementation/interface to compute whether a memory block can be offlined (exposed via /sys/devices/system/memory/memoryX/removable) and would like to simplify it (remove the implementation).

1. It runs basically lockless. While this might be good for performance, we see possible races with memory offlining/unplug that will require at least some sort of locking to fix.

2. Nowadays, more false positives are possible. No arch-specific checks are performed that validate if memory offlining will not be denied right away (and such check will require locking). For example, arm64 won't allow to offline any memory block that was added during boot - which will imply a very high error rate. Other archs have other constraints.

3. The interface is inherently racy. E.g., if a memory block is detected to be removable (and was not a false positive at that time), there is still no guarantee that offlining will actually succeed. So any caller already has to deal with false positives.

4. It is unclear which performance benefit this interface actually provides. The introducing commit 5c755e9fd813 ("memory-hotplug: add sysfs removable attribute for hotplug memory remove") mentioned "A user-level agent must be able to identify which sections of memory are likely to be removable before attempting the potentially expensive operation."
However, no actual performance comparison was included.

Known users:
- lsmem: Will group memory blocks based on the "removable" property. [1]
- chmem: Indirect user. It has a RANGE mode where one can specify removable ranges identified via lsmem to be offlined. However, it also has a "SIZE" mode, which allows a sysadmin to skip the manual "identify removable blocks" step. [2]
- powerpc-utils: Uses the "removable" attribute to skip some memory blocks right away when trying to find some to offline+remove. However, with ballooning enabled, it already skips this information completely (because it once resulted in many false negatives). Therefore, the implementation can deal with false positives properly already. [3]

With CONFIG_MEMORY_HOTREMOVE, always indicating "removable" should not break any user space tool. We implement a very bad heuristic now. (in
contrast: always returning "not removable" would at least affect powerpc-utils)

Without CONFIG_MEMORY_HOTREMOVE we cannot offline anything, so report "not removable" as before.

Original discussion can be found in [4] ("[PATCH RFC v1] mm:
is_mem_section_removable() overhaul").

Other users of is_mem_section_removable() will be removed next, so that we can remove is_mem_section_removable() completely.

[1] http://man7.org/linux/man-pages/man1/lsmem.1.html
[2] http://man7.org/linux/man-pages/man8/chmem.8.html
[3] https://github.com/ibm-power-utilities/powerpc-utils
[4] https://lkml.kernel.org/r/[email protected]

Cc Dan Williams <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc "Rafael J. Wysocki" <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Badari Pulavarty <[email protected]>
Cc Nathan Fontenot <[email protected]>
Cc Robert Jennings <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Karel Zak <[email protected]>

---

1. Are there any use cases that really require this interface to keep producing "more reliable" results?

2. Is there any real performance advantage when using this interface to identify memory blocks to offline?

---
 drivers/base/memory.c | 27 +++++++--------------------
 1 file changed, 7 insertions(+), 20 deletions(-)
drivers/base/memory.c +7 -20
index 6503f5d0b749..d78a92f09984 100644
@@ -105,30 +105,17 @@ static ssize_t phys_index_show(struct device *dev,
 }
 
 /*
- * Show whether the memory block is likely to be offlineable (or is already
- * offline). Once offline, the memory block could be removed. The return
- * value does, however, not indicate that there is a way to remove the
- * memory block.
+ * Legacy interface that we cannot remove. Always indicate "removable"
+ * with CONFIG_MEMORY_HOTREMOVE - bad heuristic.
  */
 static ssize_t removable_show(struct device *dev, struct device_attribute *attr,
 			      char *buf)
 {
-	struct memory_block *mem = to_memory_block(dev);
-	unsigned long pfn;
-	int ret = 1, i;
-
-	if (mem->state != MEM_ONLINE)
-		goto out;
-
-	for (i = 0; i < sections_per_block; i++) {
-		if (!present_section_nr(mem->start_section_nr + i))
-			continue;
-		pfn = section_nr_to_pfn(mem->start_section_nr + i);
-		ret &= is_mem_section_removable(pfn, PAGES_PER_SECTION);
-	}
-
-out:
-	return sprintf(buf, "%d\n", ret);
+#ifdef CONFIG_MEMORY_HOTREMOVE
+	return sprintf(buf, "1\n");
+#else
+	return sprintf(buf, "0\n");
+#endif
 }
 
 /*
#2 lore raw
Quoted 7 lines · Nathan

It's been awhile since I've looked at the powerpc-utils drmgr command and pseries DLPAR code but a quick scan makes and it appears that it hasn't changed too much. Given that, some thoughts.

The sysfs 'removable' file was a great help when memory DLPAR was driven from userspace in the powerpc-utils drmgr command. Having this check did improve performance though I can't point to any numbers.

Do you still have an access to the HW to give it a try?

Quoted 11 lines

Currently, memory DLPAR is done completely in the kernel. The request is initiated from drmgr writing to /sys/kernel/dlpar (for pHyp partitions) or from a hotplug interrupt (for guests). I don't believe the 'removable' sysfs file is used in either of these paths by drmgr. The only time it is used is on older kernels that do not support in-kernel memory DLPAR.

Given this, I don't think removing the 'removable' sysfs file would cause any issues for the drmgr command. The only scenario I can think of is using an old version of drmgr that does not support in-kernel memory DLPAR on a new kernel where the 'removable' sysfs file has been removed. This doesn't seem likely though and drmgr could be updated to detect this.

Thanks for the information!

Signature · 2 lines
Michal Hocko
SUSE Labs
#3 lore raw
Quoted 17 lines · Michal Hocko
Quoted 7 lines · Nathan

It's been awhile since I've looked at the powerpc-utils drmgr command and pseries DLPAR code but a quick scan makes and it appears that it hasn't changed too much. Given that, some thoughts.

The sysfs 'removable' file was a great help when memory DLPAR was driven from userspace in the powerpc-utils drmgr command. Having this check did improve performance though I can't point to any numbers.

Do you still have an access to the HW to give it a try?

Quoted 6 lines

Currently, memory DLPAR is done completely in the kernel. The request is initiated from drmgr writing to /sys/kernel/dlpar (for pHyp partitions) or from a hotplug interrupt (for guests). I don't believe the 'removable' sysfs file is used in either of these paths by drmgr. The only time it is used is on older kernels that do not support in-kernel memory DLPAR.

Quoted 8 lines
Quoted 5 lines

Given this, I don't think removing the 'removable' sysfs file would cause any issues for the drmgr command. The only scenario I can think of is using an old version of drmgr that does not support in-kernel memory DLPAR on a new kernel where the 'removable' sysfs file has been removed. This doesn't seem likely though and drmgr could be updated to detect this.

Thanks for the information!

(weird, I never received the mail from Nathan - mail deliver issues brighten my Mondays :) )

Thanks for the information! Looks like powerpc indeed can live without the interface (old userspace on shiny new kernel would in the worst case simply be slower).

Of course, the alternative to returning always "removable" would be to drop the attribute completely. So, if the "removable" attribute is not present

- powerpc-utils will fallback to "removable"
- lsmem will fallback to "not removable". Could be because it assumes "old kernel with lacking offlining capability".

I don't know how likely it is that this could break custom scripts that used the returned value for any purpose (e.g., use it as an indicator if memory offlining is supported at all etc.).

Signature · 3 lines
Thanks,

David / dhildenb
#4 lore raw
Quoted 38 lines · David Hildenbrand
Quoted 17 lines · Michal Hocko
Quoted 7 lines · Nathan

It's been awhile since I've looked at the powerpc-utils drmgr command and pseries DLPAR code but a quick scan makes and it appears that it hasn't changed too much. Given that, some thoughts.

The sysfs 'removable' file was a great help when memory DLPAR was driven from userspace in the powerpc-utils drmgr command. Having this check did improve performance though I can't point to any numbers.

Do you still have an access to the HW to give it a try?

Quoted 6 lines

Currently, memory DLPAR is done completely in the kernel. The request is initiated from drmgr writing to /sys/kernel/dlpar (for pHyp partitions) or from a hotplug interrupt (for guests). I don't believe the 'removable' sysfs file is used in either of these paths by drmgr. The only time it is used is on older kernels that do not support in-kernel memory DLPAR.

Quoted 8 lines
Quoted 5 lines

Given this, I don't think removing the 'removable' sysfs file would cause any issues for the drmgr command. The only scenario I can think of is using an old version of drmgr that does not support in-kernel memory DLPAR on a new kernel where the 'removable' sysfs file has been removed. This doesn't seem likely though and drmgr could be updated to detect this.

Thanks for the information!

(weird, I never received the mail from Nathan - mail deliver issues brighten my Mondays :) )

Thanks for the information! Looks like powerpc indeed can live without the interface (old userspace on shiny new kernel would in the worst case simply be slower).

Of course, the alternative to returning always "removable" would be to drop the attribute completely. So, if the "removable" attribute is not present

Quoted 7 lines

- powerpc-utils will fallback to "removable"
- lsmem will fallback to "not removable". Could be because it assumes "old kernel with lacking offlining capability".

I don't know how likely it is that this could break custom scripts that used the returned value for any purpose (e.g., use it as an indicator if memory offlining is supported at all etc.).

Our long term tradition with user visible knobs is to keep them in place and simply fake the answer. This seems to be a safer option and less likely to lead to failures.

Signature · 2 lines
Michal Hocko
SUSE Labs
#5 lore raw
In reply to David Hildenbrand
Quoted 11 lines · David Hildenbrand
Quoted 10 lines · Michal Hocko
Quoted 7 lines · Nathan

It's been awhile since I've looked at the powerpc-utils drmgr command and pseries DLPAR code but a quick scan makes and it appears that it hasn't changed too much. Given that, some thoughts.

The sysfs 'removable' file was a great help when memory DLPAR was driven from userspace in the powerpc-utils drmgr command. Having this check did improve performance though I can't point to any numbers.

Do you still have an access to the HW to give it a try?

No, I no longer have access to Power hardware.

-Nathan

Quoted 35 lines
Quoted 7 lines
Quoted 7 lines

Currently, memory DLPAR is done completely in the kernel. The request is initiated from drmgr writing to /sys/kernel/dlpar (for pHyp partitions) or from a hotplug interrupt (for guests). I don't believe the 'removable' sysfs file is used in either of these paths by drmgr. The only time it is used is on older kernels that do not support in-kernel memory DLPAR.

Quoted 8 lines
Quoted 5 lines

Given this, I don't think removing the 'removable' sysfs file would cause any issues for the drmgr command. The only scenario I can think of is using an old version of drmgr that does not support in-kernel memory DLPAR on a new kernel where the 'removable' sysfs file has been removed. This doesn't seem likely though and drmgr could be updated to detect this.

Thanks for the information!

(weird, I never received the mail from Nathan - mail deliver issues brighten my Mondays :) )

Thanks for the information! Looks like powerpc indeed can live without the interface (old userspace on shiny new kernel would in the worst case simply be slower).

Of course, the alternative to returning always "removable" would be to drop the attribute completely. So, if the "removable" attribute is not present

- powerpc-utils will fallback to "removable"
- lsmem will fallback to "not removable". Could be because it assumes "old kernel with lacking offlining capability".

I don't know how likely it is that this could break custom scripts that used the returned value for any purpose (e.g., use it as an indicator if memory offlining is supported at all etc.).

Acked-by on v1 1/1
#6 lore raw
In reply to David Hildenbrand
Quoted 62 lines · David Hildenbrand

We see multiple issues with the implementation/interface to compute whether a memory block can be offlined (exposed via /sys/devices/system/memory/memoryX/removable) and would like to simplify it (remove the implementation).

1. It runs basically lockless. While this might be good for performance, we see possible races with memory offlining/unplug that will require at least some sort of locking to fix.

2. Nowadays, more false positives are possible. No arch-specific checks are performed that validate if memory offlining will not be denied right away (and such check will require locking). For example, arm64 won't allow to offline any memory block that was added during boot - which will imply a very high error rate. Other archs have other constraints.

3. The interface is inherently racy. E.g., if a memory block is detected to be removable (and was not a false positive at that time), there is still no guarantee that offlining will actually succeed. So any caller already has to deal with false positives.

4. It is unclear which performance benefit this interface actually provides. The introducing commit 5c755e9fd813 ("memory-hotplug: add sysfs removable attribute for hotplug memory remove") mentioned "A user-level agent must be able to identify which sections of memory are likely to be removable before attempting the potentially expensive operation."
However, no actual performance comparison was included.

Known users:
- lsmem: Will group memory blocks based on the "removable" property. [1]
- chmem: Indirect user. It has a RANGE mode where one can specify removable ranges identified via lsmem to be offlined. However, it also has a "SIZE" mode, which allows a sysadmin to skip the manual "identify removable blocks" step. [2]
- powerpc-utils: Uses the "removable" attribute to skip some memory blocks right away when trying to find some to offline+remove. However, with ballooning enabled, it already skips this information completely (because it once resulted in many false negatives). Therefore, the implementation can deal with false positives properly already. [3]

With CONFIG_MEMORY_HOTREMOVE, always indicating "removable" should not break any user space tool. We implement a very bad heuristic now. (in
contrast: always returning "not removable" would at least affect powerpc-utils)

Without CONFIG_MEMORY_HOTREMOVE we cannot offline anything, so report "not removable" as before.

Original discussion can be found in [4] ("[PATCH RFC v1] mm:
is_mem_section_removable() overhaul").

Other users of is_mem_section_removable() will be removed next, so that we can remove is_mem_section_removable() completely.

[1] http://man7.org/linux/man-pages/man1/lsmem.1.html
[2] http://man7.org/linux/man-pages/man8/chmem.8.html
[3] https://github.com/ibm-power-utilities/powerpc-utils
[4] https://lkml.kernel.org/r/[email protected]

Quoted 13 lines
Cc Dan Williams <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc "Rafael J. Wysocki" <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Badari Pulavarty <[email protected]>
Cc Nathan Fontenot <[email protected]>
Cc Robert Jennings <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Karel Zak <[email protected]>

Please add information provided by Nathan.

Minor nit below.

drivers/base/memory.c static ssize_t phys_index_show(struct device *dev, · line 118

Quoted 5 lines
drivers/base/memory.c +5 -0
+#ifdef CONFIG_MEMORY_HOTREMOVE
+	return sprintf(buf, "1\n");
+#else
+	return sprintf(buf, "0\n");
+#endif
	int ret = IS_ENABLED(CONFIG_MEMORY_HOTREMOVE);

	return sprintf(buf, "%d\n", ret)

would be slightly nicer than explicit ifdefs.

Signature · 2 lines
Michal Hocko
SUSE Labs
#7 lore raw
Quoted 91 lines · Michal Hocko
Quoted 62 lines · David Hildenbrand

We see multiple issues with the implementation/interface to compute whether a memory block can be offlined (exposed via /sys/devices/system/memory/memoryX/removable) and would like to simplify it (remove the implementation).

1. It runs basically lockless. While this might be good for performance, we see possible races with memory offlining/unplug that will require at least some sort of locking to fix.

2. Nowadays, more false positives are possible. No arch-specific checks are performed that validate if memory offlining will not be denied right away (and such check will require locking). For example, arm64 won't allow to offline any memory block that was added during boot - which will imply a very high error rate. Other archs have other constraints.

3. The interface is inherently racy. E.g., if a memory block is detected to be removable (and was not a false positive at that time), there is still no guarantee that offlining will actually succeed. So any caller already has to deal with false positives.

4. It is unclear which performance benefit this interface actually provides. The introducing commit 5c755e9fd813 ("memory-hotplug: add sysfs removable attribute for hotplug memory remove") mentioned "A user-level agent must be able to identify which sections of memory are likely to be removable before attempting the potentially expensive operation."
However, no actual performance comparison was included.

Known users:
- lsmem: Will group memory blocks based on the "removable" property. [1]
- chmem: Indirect user. It has a RANGE mode where one can specify removable ranges identified via lsmem to be offlined. However, it also has a "SIZE" mode, which allows a sysadmin to skip the manual "identify removable blocks" step. [2]
- powerpc-utils: Uses the "removable" attribute to skip some memory blocks right away when trying to find some to offline+remove. However, with ballooning enabled, it already skips this information completely (because it once resulted in many false negatives). Therefore, the implementation can deal with false positives properly already. [3]

With CONFIG_MEMORY_HOTREMOVE, always indicating "removable" should not break any user space tool. We implement a very bad heuristic now. (in
contrast: always returning "not removable" would at least affect powerpc-utils)

Without CONFIG_MEMORY_HOTREMOVE we cannot offline anything, so report "not removable" as before.

Original discussion can be found in [4] ("[PATCH RFC v1] mm:
is_mem_section_removable() overhaul").

Other users of is_mem_section_removable() will be removed next, so that we can remove is_mem_section_removable() completely.

[1] http://man7.org/linux/man-pages/man1/lsmem.1.html
[2] http://man7.org/linux/man-pages/man8/chmem.8.html
[3] https://github.com/ibm-power-utilities/powerpc-utils
[4] https://lkml.kernel.org/r/[email protected]

Quoted 13 lines
Cc Dan Williams <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc "Rafael J. Wysocki" <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Badari Pulavarty <[email protected]>
Cc Nathan Fontenot <[email protected]>
Cc Robert Jennings <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Karel Zak <[email protected]>

Please add information provided by Nathan.

Minor nit below.

drivers/base/memory.c static ssize_t phys_index_show(struct device *dev, · line 118

Quoted 5 lines
drivers/base/memory.c +5 -0
+#ifdef CONFIG_MEMORY_HOTREMOVE
+	return sprintf(buf, "1\n");
+#else
+	return sprintf(buf, "0\n");
+#endif
	int ret = IS_ENABLED(CONFIG_MEMORY_HOTREMOVE);

	return sprintf(buf, "%d\n", ret)
Quoted 2 lines

would be slightly nicer than explicit ifdefs.

Indeed, thanks!

Signature · 3 lines
Thanks,

David / dhildenb