drivers/base/memory.c: indicate all memory blocks as removable
Subsystems (4)
PATCH 1/1 drivers/base/memory.c: indicate all memory blocks as removable
We see multiple issues with the implementation/interface to compute whether a memory block can be offlined (exposed via /sys/devices/system/memory/memoryX/removable) and would like to simplify it (remove the implementation).
1. It runs basically lockless. While this might be good for performance, we see possible races with memory offlining that will require at least some sort of locking to fix.
2. Nowadays, more false positives are possible. No arch-specific checks are performed that validate if memory offlining will not be denied right away (and such check will require locking). For example, arm64 won't allow to offline any memory block that was added during boot - which will imply a very high error rate. Other archs have other constraints.
3. The interface is inherently racy. E.g., if a memory block is detected to be removable (and was not a false positive at that time), there is still no guarantee that offlining will actually succeed. So any caller already has to deal with false positives.
4. It is unclear which performance benefit this interface actually provides. The introducing commit 5c755e9fd813 ("memory-hotplug: add sysfs removable attribute for hotplug memory remove") mentioned "A user-level agent must be able to identify which sections of memory are likely to be removable before attempting the potentially expensive operation."
However, no actual performance comparison was included.
Known users:
- lsmem: Will group memory blocks based on the "removable" property. [1]
- chmem: Indirect user. It has a RANGE mode where one can specify removable ranges identified via lsmem to be offlined. However, it also has a "SIZE" mode, which allows a sysadmin to skip the manual "identify removable blocks" step. [2]
- powerpc-utils: Uses the "removable" attribute to skip some memory blocks right away when trying to find some to offline+remove. However, with ballooning enabled, it already skips this information completely (because it once resulted in many false negatives). Therefore, the implementation can deal with false positives properly already. [3]
According to Nathan Fontenot, DLPAR on powerpc is nowadays no longer driven from userspace via the drmgr command (powerpc-utils). Nowadays it's managed in the kernel - including onlining/offlining of memory blocks - triggered by drmgr writing to /sys/kernel/dlpar. So the affected legacy userspace handling is only active on old kernels. Only very old versions of drmgr on a new kernel (unlikely) might execute slower - totally acceptable.
With CONFIG_MEMORY_HOTREMOVE, always indicating "removable" should not break any user space tool. We implement a very bad heuristic now. Without CONFIG_MEMORY_HOTREMOVE we cannot offline anything, so report "not removable" as before.
Original discussion can be found in [4] ("[PATCH RFC v1] mm:is_mem_section_removable() overhaul").
Other users of is_mem_section_removable() will be removed next, so that we can remove is_mem_section_removable() completely.
[1] http://man7.org/linux/man-pages/man1/lsmem.1.html
[2] http://man7.org/linux/man-pages/man8/chmem.8.html
[3] https://github.com/ibm-power-utilities/powerpc-utils
[4] https://lkml.kernel.org/r/[email protected]
---
RFC -> v1:
- Use IS_ENABLED() instead of ifdefs
- Add information from Nathan (thanks!)
--- drivers/base/memory.c | 23 +++-------------------- 1 file changed, 3 insertions(+), 20 deletions(-)
drivers/base/memory.c +3 -20
@@ -105,30 +105,13 @@ static ssize_t phys_index_show(struct device *dev, }/* -* Show whether the memory block is likely to be offlineable (or is already -* offline). Once offline, the memory block could be removed. The return -* value does, however, not indicate that there is a way to remove the -* memory block. +* Legacy interface that we cannot remove. Always indicate "removable" +* with CONFIG_MEMORY_HOTREMOVE - bad heuristic. */ static ssize_t removable_show (struct device *dev,struct device_attribute *attr,char *buf) { -struct memory_block *mem =to_memory_block (dev); -unsigned long pfn; -int ret =1 , i; - -if (mem->state !=MEM_ONLINE ) - goto out; - -for (i =0 ; i < sections_per_block; i++) { -if (!present_section_nr (mem->start_section_nr + i)) -continue ; - pfn =section_nr_to_pfn (mem->start_section_nr + i); - ret &=is_mem_section_removable (pfn,PAGES_PER_SECTION ); - } - -out: -return sprintf (buf,"%d\n" , ret); +return sprintf (buf,"%d\n" , (int )IS_ENABLED (CONFIG_MEMORY_HOTREMOVE )); }/*
Signature · 1 line
2.24.1
Quoted 67 lines · David Hildenbrand
We see multiple issues with the implementation/interface to compute whether a memory block can be offlined (exposed via
/sys/devices/system/memory/memoryX/removable) and would like to simplify it (remove the implementation).1. It runs basically lockless. While this might be good for performance, we see possible races with memory offlining that will require at least some sort of locking to fix.
2. Nowadays, more false positives are possible. No arch-specific checks are performed that validate if memory offlining will not be denied right away (and such check will require locking). For example, arm64 won't allow to offline any memory block that was added during boot - which will imply a very high error rate. Other archs have other constraints.
3. The interface is inherently racy. E.g., if a memory block is detected to be removable (and was not a false positive at that time), there is still no guarantee that offlining will actually succeed. So any caller already has to deal with false positives.
4. It is unclear which performance benefit this interface actually provides. The introducing commit
5c755e9fd813("memory-hotplug: add sysfs removable attribute for hotplug memory remove") mentioned "A user-level agent must be able to identify which sections of memory are likely to be removable before attempting the potentially expensive operation."
However, no actual performance comparison was included.Known users:
- lsmem: Will group memory blocks based on the "removable" property. [1]
- chmem: Indirect user. It has a RANGE mode where one can specify removable ranges identified via lsmem to be offlined. However, it also has a "SIZE" mode, which allows a sysadmin to skip the manual "identify removable blocks" step. [2]
- powerpc-utils: Uses the "removable" attribute to skip some memory blocks right away when trying to find some to offline+remove. However, with ballooning enabled, it already skips this information completely (because it once resulted in many false negatives). Therefore, the implementation can deal with false positives properly already. [3]According to Nathan Fontenot, DLPAR on powerpc is nowadays no longer driven from userspace via the drmgr command (powerpc-utils). Nowadays it's managed in the kernel - including onlining/offlining of memory blocks - triggered by drmgr writing to
/sys/kernel/dlpar. So the affected legacy userspace handling is only active on old kernels. Only very old versions of drmgr on a new kernel (unlikely) might execute slower - totally acceptable.With
CONFIG_MEMORY_HOTREMOVE, always indicating "removable" should not break any user space tool. We implement a very bad heuristic now. WithoutCONFIG_MEMORY_HOTREMOVEwe cannot offline anything, so report "not removable" as before.Original discussion can be found in [4] ("[PATCH RFC v1] mm:
is_mem_section_removable()overhaul").Other users of
is_mem_section_removable()will be removed next, so that we can removeis_mem_section_removable()completely.[1] http://man7.org/linux/man-pages/man1/lsmem.1.html
[2] http://man7.org/linux/man-pages/man8/chmem.8.html
[3] https://github.com/ibm-power-utilities/powerpc-utils
[4] https://lkml.kernel.org/r/[email protected]
Quoted 14 lines
Suggested-by Michal Hocko <[email protected]>Acked-by Michal Hocko <[email protected]>Cc Dan Williams <[email protected]>Cc Greg Kroah-Hartman <[email protected]>Cc "Rafael J. Wysocki" <[email protected]>Cc Andrew Morton <[email protected]>Cc Badari Pulavarty <[email protected]>Cc Nathan Fontenot <[email protected]>Cc Robert Jennings <[email protected]>Cc Heiko Carstens <[email protected]>Cc Karel Zak <[email protected]>Signed-off-by David Hildenbrand <[email protected]>
Quoted 49 lines
---
RFC -> v1:
- UseIS_ENABLED()instead of ifdefs
- Add information from Nathan (thanks!)--- drivers/base/memory.c | 23 +++-------------------- 1 file changed, 3 insertions(+), 20 deletions(-)drivers/base/memory.c +3
-20@@ -105,30 +105,13 @@ static ssize_t phys_index_show(struct device *dev, }/* -* Show whether the memory block is likely to be offlineable (or is already -* offline). Once offline, the memory block could be removed. The return -* value does, however, not indicate that there is a way to remove the -* memory block. +* Legacy interface that we cannot remove. Always indicate "removable" +* with CONFIG_MEMORY_HOTREMOVE - bad heuristic. */ static ssize_t removable_show (struct device *dev,struct device_attribute *attr,char *buf) { -struct memory_block *mem =to_memory_block (dev); -unsigned long pfn; -int ret =1 , i; - -if (mem->state !=MEM_ONLINE ) - goto out; - -for (i =0 ; i < sections_per_block; i++) { -if (!present_section_nr (mem->start_section_nr + i)) -continue ; - pfn =section_nr_to_pfn (mem->start_section_nr + i); - ret &=is_mem_section_removable (pfn,PAGES_PER_SECTION ); - } - -out: -return sprintf (buf,"%d\n" , ret); +return sprintf (buf,"%d\n" , (int )IS_ENABLED (CONFIG_MEMORY_HOTREMOVE )); }/*
Quoted 63 lines · David Hildenbrand
We see multiple issues with the implementation/interface to compute whether a memory block can be offlined (exposed via
/sys/devices/system/memory/memoryX/removable) and would like to simplify it (remove the implementation).1. It runs basically lockless. While this might be good for performance, we see possible races with memory offlining that will require at least some sort of locking to fix.
2. Nowadays, more false positives are possible. No arch-specific checks are performed that validate if memory offlining will not be denied right away (and such check will require locking). For example, arm64 won't allow to offline any memory block that was added during boot - which will imply a very high error rate. Other archs have other constraints.
3. The interface is inherently racy. E.g., if a memory block is detected to be removable (and was not a false positive at that time), there is still no guarantee that offlining will actually succeed. So any caller already has to deal with false positives.
4. It is unclear which performance benefit this interface actually provides. The introducing commit
5c755e9fd813("memory-hotplug: add sysfs removable attribute for hotplug memory remove") mentioned "A user-level agent must be able to identify which sections of memory are likely to be removable before attempting the potentially expensive operation."
However, no actual performance comparison was included.Known users:
- lsmem: Will group memory blocks based on the "removable" property. [1]
- chmem: Indirect user. It has a RANGE mode where one can specify removable ranges identified via lsmem to be offlined. However, it also has a "SIZE" mode, which allows a sysadmin to skip the manual "identify removable blocks" step. [2]
- powerpc-utils: Uses the "removable" attribute to skip some memory blocks right away when trying to find some to offline+remove. However, with ballooning enabled, it already skips this information completely (because it once resulted in many false negatives). Therefore, the implementation can deal with false positives properly already. [3]According to Nathan Fontenot, DLPAR on powerpc is nowadays no longer driven from userspace via the drmgr command (powerpc-utils). Nowadays it's managed in the kernel - including onlining/offlining of memory blocks - triggered by drmgr writing to
/sys/kernel/dlpar. So the affected legacy userspace handling is only active on old kernels. Only very old versions of drmgr on a new kernel (unlikely) might execute slower - totally acceptable.With
CONFIG_MEMORY_HOTREMOVE, always indicating "removable" should not break any user space tool. We implement a very bad heuristic now. WithoutCONFIG_MEMORY_HOTREMOVEwe cannot offline anything, so report "not removable" as before.Original discussion can be found in [4] ("[PATCH RFC v1] mm:
is_mem_section_removable()overhaul").Other users of
is_mem_section_removable()will be removed next, so that we can removeis_mem_section_removable()completely.
Quoted 8 lines
[1] http://man7.org/linux/man-pages/man1/lsmem.1.html
[2] http://man7.org/linux/man-pages/man8/chmem.8.html
[3] https://github.com/ibm-power-utilities/powerpc-utils
[4] https://lkml.kernel.org/r/[email protected]Suggested-by Michal Hocko <[email protected]>Acked-by Michal Hocko <[email protected]>Cc Dan Williams <[email protected]>
David, Andrew,
I'd like to recommend this patch for -stable as it likely (test underway) solves this crash report from Steve:
[ 148.796036] page dumped because: VM_BUG_ON_PAGE(PagePoisoned(p)) [ 148.796074] ------------[ cut here ]------------ [ 148.796098] kernel BUG at include/linux/mm.h:1087! [ 148.796126] invalid opcode: 0000 [#1] SMP NOPTI [ 148.796146] CPU: 63 PID: 5471 Comm: lsmem Not tainted 5.5.10-200.fc31.x8= 6_64+debug #1 [ 148.796173] Hardware name: Intel Corporation S2600WFD/S2600WFD, BIOS SE5= C620.86B.02.01.0010.010620200716 01/06/2020 [ 148.796212] RIP: 0010:is_mem_section_removable+0x1a4/0x1b0 [ 148.796561] Call Trace: [ 148.796591] removable_show+0x6e/0xa0 [ 148.796608] dev_attr_show+0x19/0x40 [ 148.796625] sysfs_kf_seq_show+0xa9/0x100 [ 148.796640] seq_read+0xd5/0x450 [ 148.796657] vfs_read+0xc5/0x180 [ 148.796672] ksys_read+0x68/0xe0 [ 148.796688] do_syscall_64+0x5c/0xa0 [ 148.796704] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 148.796721] RIP: 0033:0x7f3ab1646412
...on a non-debug kernel it just crashes.
In this case lsmem is failing when reading memory96:
openat(3, "memory96/removable", O_RDONLY|O_CLOEXEC) = 4fcntl(4, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)fstat(4, {st_mode=S_IFREG|0444, st_size=4096, ...}) = 0read(4, <unfinished ...>) = ?
+++ killed by SIGSEGV +++
Segmentation fault (core dumped)
...which is phys_index 0x60 => memory address 0x3000000000
On this platform that lands us here:
100000000-303fffffff : System RAM
291f000000-291fe00f70 : Kernel code
2920000000-292051efff : Kernel rodata
2920600000-292093b0bf : Kernel data
29214f3000-2922dfffff : Kernel bss
3040000000-305fffffff : Reserved
3060000000-1aa5fffffff : Persistent Memory
...where the last memory block of System RAM is shared with persistent memory. I.e. the block is only partially online which means that page_to_nid() in is_mem_section_removable() will assert or crash for some of the offline pages in that block.
On Thu 26-03-20 23:24:08, Dan Williams wrote:
[...]
Quoted 40 lines
David, Andrew,
I'd like to recommend this patch for -stable as it likely (test underway) solves this crash report from Steve:
[ 148.796036] page dumped because: VM_BUG_ON_PAGE(PagePoisoned(p)) [ 148.796074] ------------[ cut here ]------------ [ 148.796098] kernel BUG at include/linux/mm.h:1087! [ 148.796126] invalid opcode: 0000 [#1] SMP NOPTI [ 148.796146] CPU: 63 PID: 5471 Comm: lsmem Not tainted 5.5.10-200.fc31.x8= 6_64+debug #1 [ 148.796173] Hardware name: Intel Corporation S2600WFD/S2600WFD, BIOS SE5= C620.86B.02.01.0010.010620200716 01/06/2020 [ 148.796212] RIP: 0010:is_mem_section_removable+0x1a4/0x1b0 [ 148.796561] Call Trace: [ 148.796591] removable_show+0x6e/0xa0 [ 148.796608] dev_attr_show+0x19/0x40 [ 148.796625] sysfs_kf_seq_show+0xa9/0x100 [ 148.796640] seq_read+0xd5/0x450 [ 148.796657] vfs_read+0xc5/0x180 [ 148.796672] ksys_read+0x68/0xe0 [ 148.796688] do_syscall_64+0x5c/0xa0 [ 148.796704] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 148.796721] RIP: 0033:0x7f3ab1646412...on a non-debug kernel it just crashes.
In this case lsmem is failing when reading memory96:
openat(3, "memory96/removable", O_RDONLY|O_CLOEXEC)= 4fcntl(4, F_GETFL)=0x8000(flags O_RDONLY|O_LARGEFILE)fstat(4, {st_mode=S_IFREG|0444, st_size=4096, ...})= 0read(4, <unfinished ...>)= ?
+++ killed by SIGSEGV +++
Segmentation fault (core dumped)...which is
phys_index0x60=> memory address0x3000000000On this platform that lands us here:
Quoted 7 lines
100000000-303fffffff : System RAM
291f000000-291fe00f70 : Kernel code
2920000000-292051efff : Kernel rodata
2920600000-292093b0bf : Kernel data
29214f3000-2922dfffff : Kernel bss
3040000000-305fffffff : Reserved
3060000000-1aa5fffffff : Persistent Memory
OK, 2GB memblocks and that would mean [0x3000000000, 0x3080000000]
Quoted 4 lines
...where the last memory block of System RAM is shared with persistent memory. I.e. the block is only partially online which means that
page_to_nid()inis_mem_section_removable()will assert or crash for some of the offline pages in that block.
Yes, this patch is a simple workaround. Normal memory hotplug will not blow up because it should be able to find out that test_pages_in_a_zone is false. Who knows how other potential pfn walkers handle that.
Risking to sound like a broken record I will remind that I have been pushing for having _all_ existing struct pages initialized and we wouldn't have problems like this popping out here and there.
That being said, I do not have any objections to backporting to stable trees.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 52 lines · Michal Hocko
On Thu 26-03-20 23:24:08, Dan Williams wrote:
[...]Quoted 40 lines
David, Andrew,
I'd like to recommend this patch for -stable as it likely (test underway) solves this crash report from Steve:
[ 148.796036] page dumped because: VM_BUG_ON_PAGE(PagePoisoned(p)) [ 148.796074] ------------[ cut here ]------------ [ 148.796098] kernel BUG at include/linux/mm.h:1087! [ 148.796126] invalid opcode: 0000 [#1] SMP NOPTI [ 148.796146] CPU: 63 PID: 5471 Comm: lsmem Not tainted 5.5.10-200.fc31.x8= 6_64+debug #1 [ 148.796173] Hardware name: Intel Corporation S2600WFD/S2600WFD, BIOS SE5= C620.86B.02.01.0010.010620200716 01/06/2020 [ 148.796212] RIP: 0010:is_mem_section_removable+0x1a4/0x1b0 [ 148.796561] Call Trace: [ 148.796591] removable_show+0x6e/0xa0 [ 148.796608] dev_attr_show+0x19/0x40 [ 148.796625] sysfs_kf_seq_show+0xa9/0x100 [ 148.796640] seq_read+0xd5/0x450 [ 148.796657] vfs_read+0xc5/0x180 [ 148.796672] ksys_read+0x68/0xe0 [ 148.796688] do_syscall_64+0x5c/0xa0 [ 148.796704] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 148.796721] RIP: 0033:0x7f3ab1646412...on a non-debug kernel it just crashes.
In this case lsmem is failing when reading memory96:
openat(3, "memory96/removable", O_RDONLY|O_CLOEXEC)= 4fcntl(4, F_GETFL)=0x8000(flags O_RDONLY|O_LARGEFILE)fstat(4, {st_mode=S_IFREG|0444, st_size=4096, ...})= 0read(4, <unfinished ...>)= ?
+++ killed by SIGSEGV +++
Segmentation fault (core dumped)...which is
phys_index0x60=> memory address0x3000000000On this platform that lands us here:
Quoted 7 lines
100000000-303fffffff : System RAM
291f000000-291fe00f70 : Kernel code
2920000000-292051efff : Kernel rodata
2920600000-292093b0bf : Kernel data
29214f3000-2922dfffff : Kernel bss
3040000000-305fffffff : Reserved
3060000000-1aa5fffffff : Persistent MemoryOK, 2GB memblocks and that would mean [
0x3000000000,0x3080000000]
Quoted 8 lines
Quoted 4 lines
...where the last memory block of System RAM is shared with persistent memory. I.e. the block is only partially online which means that
page_to_nid()inis_mem_section_removable()will assert or crash for some of the offline pages in that block.Yes, this patch is a simple workaround. Normal memory hotplug will not blow up because it should be able to find out that
test_pages_in_a_zoneis false. Who knows how other potential pfn walkers handle that.
All other pfn walkers now correctly use pfn_to_online_page() - which will also result in false positives in this scenario and is still to be fixed by Dan IIRC. [1]
Quoted 4 lines
Risking to sound like a broken record I will remind that I have been pushing for having _all_ existing struct pages initialized and we wouldn't have problems like this popping out here and there.
The real issue is that we have uninitialized memmap within a section that is marked to contain initialized/online memmap. As I said, even pfn_to_online_page()/SECTION_IS_ONLINE does not help here.
Also, there is no way to mark devmem to have a initialized memmap (something like SECTION_IS_ONLINE). I expressed my feeling about that already.
[1] contains a discussion how it could be addressed.
Quoted 4 lines
That being said, I do not have any objections to backporting to stable trees.
This one is one of the remaining places where we don't use pfn_to_online_page(). So yeah, this patch shouldn't hurt.
[1] https://lkml.kernel.org/r/[email protected]
Signature · 3 lines
Thanks, David / dhildenb
Quoted 59 lines · David Hildenbrand
Quoted 57 lines · Michal Hocko
On Thu 26-03-20 23:24:08, Dan Williams wrote:
[...]Quoted 40 lines
David, Andrew,
I'd like to recommend this patch for -stable as it likely (test underway) solves this crash report from Steve:
[ 148.796036] page dumped because: VM_BUG_ON_PAGE(PagePoisoned(p)) [ 148.796074] ------------[ cut here ]------------ [ 148.796098] kernel BUG at include/linux/mm.h:1087! [ 148.796126] invalid opcode: 0000 [#1] SMP NOPTI [ 148.796146] CPU: 63 PID: 5471 Comm: lsmem Not tainted 5.5.10-200.fc31.x8= 6_64+debug #1 [ 148.796173] Hardware name: Intel Corporation S2600WFD/S2600WFD, BIOS SE5= C620.86B.02.01.0010.010620200716 01/06/2020 [ 148.796212] RIP: 0010:is_mem_section_removable+0x1a4/0x1b0 [ 148.796561] Call Trace: [ 148.796591] removable_show+0x6e/0xa0 [ 148.796608] dev_attr_show+0x19/0x40 [ 148.796625] sysfs_kf_seq_show+0xa9/0x100 [ 148.796640] seq_read+0xd5/0x450 [ 148.796657] vfs_read+0xc5/0x180 [ 148.796672] ksys_read+0x68/0xe0 [ 148.796688] do_syscall_64+0x5c/0xa0 [ 148.796704] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 148.796721] RIP: 0033:0x7f3ab1646412...on a non-debug kernel it just crashes.
In this case lsmem is failing when reading memory96:
openat(3, "memory96/removable", O_RDONLY|O_CLOEXEC)= 4fcntl(4, F_GETFL)=0x8000(flags O_RDONLY|O_LARGEFILE)fstat(4, {st_mode=S_IFREG|0444, st_size=4096, ...})= 0read(4, <unfinished ...>)= ?
+++ killed by SIGSEGV +++
Segmentation fault (core dumped)...which is
phys_index0x60=> memory address0x3000000000On this platform that lands us here:
Quoted 7 lines
100000000-303fffffff : System RAM
291f000000-291fe00f70 : Kernel code
2920000000-292051efff : Kernel rodata
2920600000-292093b0bf : Kernel data
29214f3000-2922dfffff : Kernel bss
3040000000-305fffffff : Reserved
3060000000-1aa5fffffff : Persistent MemoryOK, 2GB memblocks and that would mean [
0x3000000000,0x3080000000]Quoted 4 lines
...where the last memory block of System RAM is shared with persistent memory. I.e. the block is only partially online which means that
page_to_nid()inis_mem_section_removable()will assert or crash for some of the offline pages in that block.
Quoted 7 lines
Quoted 3 lines
Yes, this patch is a simple workaround. Normal memory hotplug will not blow up because it should be able to find out that
test_pages_in_a_zoneis false. Who knows how other potential pfn walkers handle that.All other pfn walkers now correctly use
pfn_to_online_page()- which will also result in false positives in this scenario and is still to be fixed by Dan IIRC. [1]
Sorry, it's been too long and this fell out of my cache. I also turned away once the major fire in KVM was put out with special consideration for for devmem pages. What's left these days? ...besides removable_show()?
Quoted 64 lines · Dan Williams
Quoted 63 lines · David Hildenbrand
Quoted 60 lines · Michal Hocko
On Thu 26-03-20 23:24:08, Dan Williams wrote:
[...]Quoted 40 lines
David, Andrew,
I'd like to recommend this patch for -stable as it likely (test underway) solves this crash report from Steve:
[ 148.796036] page dumped because: VM_BUG_ON_PAGE(PagePoisoned(p)) [ 148.796074] ------------[ cut here ]------------ [ 148.796098] kernel BUG at include/linux/mm.h:1087! [ 148.796126] invalid opcode: 0000 [#1] SMP NOPTI [ 148.796146] CPU: 63 PID: 5471 Comm: lsmem Not tainted 5.5.10-200.fc31.x8= 6_64+debug #1 [ 148.796173] Hardware name: Intel Corporation S2600WFD/S2600WFD, BIOS SE5= C620.86B.02.01.0010.010620200716 01/06/2020 [ 148.796212] RIP: 0010:is_mem_section_removable+0x1a4/0x1b0 [ 148.796561] Call Trace: [ 148.796591] removable_show+0x6e/0xa0 [ 148.796608] dev_attr_show+0x19/0x40 [ 148.796625] sysfs_kf_seq_show+0xa9/0x100 [ 148.796640] seq_read+0xd5/0x450 [ 148.796657] vfs_read+0xc5/0x180 [ 148.796672] ksys_read+0x68/0xe0 [ 148.796688] do_syscall_64+0x5c/0xa0 [ 148.796704] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 148.796721] RIP: 0033:0x7f3ab1646412...on a non-debug kernel it just crashes.
In this case lsmem is failing when reading memory96:
openat(3, "memory96/removable", O_RDONLY|O_CLOEXEC)= 4fcntl(4, F_GETFL)=0x8000(flags O_RDONLY|O_LARGEFILE)fstat(4, {st_mode=S_IFREG|0444, st_size=4096, ...})= 0read(4, <unfinished ...>)= ?
+++ killed by SIGSEGV +++
Segmentation fault (core dumped)...which is
phys_index0x60=> memory address0x3000000000On this platform that lands us here:
Quoted 7 lines
100000000-303fffffff : System RAM
291f000000-291fe00f70 : Kernel code
2920000000-292051efff : Kernel rodata
2920600000-292093b0bf : Kernel data
29214f3000-2922dfffff : Kernel bss
3040000000-305fffffff : Reserved
3060000000-1aa5fffffff : Persistent MemoryOK, 2GB memblocks and that would mean [
0x3000000000,0x3080000000]Quoted 4 lines
...where the last memory block of System RAM is shared with persistent memory. I.e. the block is only partially online which means that
page_to_nid()inis_mem_section_removable()will assert or crash for some of the offline pages in that block.Yes, this patch is a simple workaround. Normal memory hotplug will not blow up because it should be able to find out that
test_pages_in_a_zoneis false. Who knows how other potential pfn walkers handle that.
Quoted 8 lines
Quoted 3 lines
All other pfn walkers now correctly use
pfn_to_online_page()- which will also result in false positives in this scenario and is still to be fixed by Dan IIRC. [1]Sorry, it's been too long and this fell out of my cache. I also turned away once the major fire in KVM was put out with special consideration for for devmem pages. What's left these days? ...besides
removable_show()?
Essentially any pfn_to_online_page() is a candidate.
E.g.,
mm/memory-failure.c:memory_failure()
is obviously broken (could be worked around)
Also
mm/memory-failure.c:soft_offline_page()
is obviously broken.
Also set_zone_contiguous()->__pageblock_pfn_to_page() is broken, when it checks for "page_zone(start_page) != zone" if the memmap contains garbage.
And I only checked a handful of examples.
Signature · 3 lines
Thanks, David / dhildenb
Quoted 75 lines · David Hildenbrand
Quoted 64 lines · Dan Williams
Quoted 63 lines · David Hildenbrand
Quoted 60 lines · Michal Hocko
On Thu 26-03-20 23:24:08, Dan Williams wrote:
[...]Quoted 40 lines
David, Andrew,
I'd like to recommend this patch for -stable as it likely (test underway) solves this crash report from Steve:
[ 148.796036] page dumped because: VM_BUG_ON_PAGE(PagePoisoned(p)) [ 148.796074] ------------[ cut here ]------------ [ 148.796098] kernel BUG at include/linux/mm.h:1087! [ 148.796126] invalid opcode: 0000 [#1] SMP NOPTI [ 148.796146] CPU: 63 PID: 5471 Comm: lsmem Not tainted 5.5.10-200.fc31.x8= 6_64+debug #1 [ 148.796173] Hardware name: Intel Corporation S2600WFD/S2600WFD, BIOS SE5= C620.86B.02.01.0010.010620200716 01/06/2020 [ 148.796212] RIP: 0010:is_mem_section_removable+0x1a4/0x1b0 [ 148.796561] Call Trace: [ 148.796591] removable_show+0x6e/0xa0 [ 148.796608] dev_attr_show+0x19/0x40 [ 148.796625] sysfs_kf_seq_show+0xa9/0x100 [ 148.796640] seq_read+0xd5/0x450 [ 148.796657] vfs_read+0xc5/0x180 [ 148.796672] ksys_read+0x68/0xe0 [ 148.796688] do_syscall_64+0x5c/0xa0 [ 148.796704] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 148.796721] RIP: 0033:0x7f3ab1646412...on a non-debug kernel it just crashes.
In this case lsmem is failing when reading memory96:
openat(3, "memory96/removable", O_RDONLY|O_CLOEXEC)= 4fcntl(4, F_GETFL)=0x8000(flags O_RDONLY|O_LARGEFILE)fstat(4, {st_mode=S_IFREG|0444, st_size=4096, ...})= 0read(4, <unfinished ...>)= ?
+++ killed by SIGSEGV +++
Segmentation fault (core dumped)...which is
phys_index0x60=> memory address0x3000000000On this platform that lands us here:
Quoted 7 lines
100000000-303fffffff : System RAM
291f000000-291fe00f70 : Kernel code
2920000000-292051efff : Kernel rodata
2920600000-292093b0bf : Kernel data
29214f3000-2922dfffff : Kernel bss
3040000000-305fffffff : Reserved
3060000000-1aa5fffffff : Persistent MemoryOK, 2GB memblocks and that would mean [
0x3000000000,0x3080000000]Quoted 4 lines
...where the last memory block of System RAM is shared with persistent memory. I.e. the block is only partially online which means that
page_to_nid()inis_mem_section_removable()will assert or crash for some of the offline pages in that block.Yes, this patch is a simple workaround. Normal memory hotplug will not blow up because it should be able to find out that
test_pages_in_a_zoneis false. Who knows how other potential pfn walkers handle that.Quoted 8 lines
Quoted 3 lines
All other pfn walkers now correctly use
pfn_to_online_page()- which will also result in false positives in this scenario and is still to be fixed by Dan IIRC. [1]Sorry, it's been too long and this fell out of my cache. I also turned away once the major fire in KVM was put out with special consideration for for devmem pages. What's left these days? ...besides
removable_show()?
Quoted 7 lines
Essentially any
pfn_to_online_page()is a candidate.E.g.,
mm/memory-failure.c:memory_failure()
is obviously broken (could be worked around)
Ooh, the current state looks worse than when I looked previously. I wasn't copied on commit 96c804a6ae8c ("mm/memory-failure.c: don't access uninitialized memmaps in memory_failure()"). That commit seems to ensure the pmem errors in memory sections that overlap with System-RAM are not handled. So that change looks broken to me. Previously get_devpagemap() was sufficient protection.
Quoted 6 lines
Also
mm/memory-failure.c:soft_offline_page()
is obviously broken.
How exactly? The soft_offline_page() callers seem to already account for System-RAM vs devmem.
Quoted 6 lines
Also set_zone_contiguous()->__pageblock_pfn_to_page() is broken, when it checks for "
page_zone(start_page)!= zone" if the memmap contains garbage.And I only checked a handful of examples.
Ok, but as the first example shows in the absence of a problem report these pre-emptive changes might make things worse so I don't think it's as simple as go instrument all the pfn_to_online_page() users.
Quoted 86 lines
Am 27.03.2020 um 23:13 schrieb Dan Williams <[email protected]>:
Quoted 81 lines · David Hildenbrand
Quoted 66 lines · Dan Williams
Quoted 63 lines · David Hildenbrand
Quoted 60 lines · Michal Hocko
On Thu 26-03-20 23:24:08, Dan Williams wrote:
[...]Quoted 40 lines
David, Andrew,
I'd like to recommend this patch for -stable as it likely (test underway) solves this crash report from Steve:
[ 148.796036] page dumped because: VM_BUG_ON_PAGE(PagePoisoned(p)) [ 148.796074] ------------[ cut here ]------------ [ 148.796098] kernel BUG at include/linux/mm.h:1087! [ 148.796126] invalid opcode: 0000 [#1] SMP NOPTI [ 148.796146] CPU: 63 PID: 5471 Comm: lsmem Not tainted 5.5.10-200.fc31.x8= 6_64+debug #1 [ 148.796173] Hardware name: Intel Corporation S2600WFD/S2600WFD, BIOS SE5= C620.86B.02.01.0010.010620200716 01/06/2020 [ 148.796212] RIP: 0010:is_mem_section_removable+0x1a4/0x1b0 [ 148.796561] Call Trace: [ 148.796591] removable_show+0x6e/0xa0 [ 148.796608] dev_attr_show+0x19/0x40 [ 148.796625] sysfs_kf_seq_show+0xa9/0x100 [ 148.796640] seq_read+0xd5/0x450 [ 148.796657] vfs_read+0xc5/0x180 [ 148.796672] ksys_read+0x68/0xe0 [ 148.796688] do_syscall_64+0x5c/0xa0 [ 148.796704] entry_SYSCALL_64_after_hwframe+0x49/0xbe [ 148.796721] RIP: 0033:0x7f3ab1646412...on a non-debug kernel it just crashes.
In this case lsmem is failing when reading memory96:
openat(3, "memory96/removable", O_RDONLY|O_CLOEXEC)= 4fcntl(4, F_GETFL)=0x8000(flags O_RDONLY|O_LARGEFILE)fstat(4, {st_mode=S_IFREG|0444, st_size=4096, ...})= 0read(4, <unfinished ...>)= ?
+++ killed by SIGSEGV +++
Segmentation fault (core dumped)...which is
phys_index0x60=> memory address0x3000000000On this platform that lands us here:
Quoted 7 lines
100000000-303fffffff : System RAM
291f000000-291fe00f70 : Kernel code
2920000000-292051efff : Kernel rodata
2920600000-292093b0bf : Kernel data
29214f3000-2922dfffff : Kernel bss
3040000000-305fffffff : Reserved
3060000000-1aa5fffffff : Persistent MemoryOK, 2GB memblocks and that would mean [
0x3000000000,0x3080000000]Quoted 4 lines
...where the last memory block of System RAM is shared with persistent memory. I.e. the block is only partially online which means that
page_to_nid()inis_mem_section_removable()will assert or crash for some of the offline pages in that block.Yes, this patch is a simple workaround. Normal memory hotplug will not blow up because it should be able to find out that
test_pages_in_a_zoneis false. Who knows how other potential pfn walkers handle that.Quoted 8 lines
Quoted 3 lines
All other pfn walkers now correctly use
pfn_to_online_page()- which will also result in false positives in this scenario and is still to be fixed by Dan IIRC. [1]Sorry, it's been too long and this fell out of my cache. I also turned away once the major fire in KVM was put out with special consideration for for devmem pages. What's left these days? ...besides
removable_show()?Essentially any
pfn_to_online_page()is a candidate.E.g.,
mm/memory-failure.c:memory_failure()
Quoted 9 lines
Quoted 1 line
is obviously broken (could be worked around)
Ooh, the current state looks worse than when I looked previously. I wasn't copied on commit
96c804a6ae8c("mm/memory-failure.c: don't access uninitialized memmaps inmemory_failure()"). That commit seems to ensure the pmem errors in memory sections that overlap with System-RAM are not handled. So that change looks broken to me. Previouslyget_devpagemap()was sufficient protection.
Well, it went in before we learned that pfn_to_online_page() is now broken in corner cases since sub-section hotadd.
Quoted 9 lines
Quoted 6 lines
Also
mm/memory-failure.c:soft_offline_page()
is obviously broken.
How exactly? The
soft_offline_page()callers seem to already account for System-RAM vs devmem.
Then my quick scan was maybe wrong :)
Quoted 12 lines
Quoted 7 lines
Also set_zone_contiguous()->__pageblock_pfn_to_page() is broken, when it checks for "
page_zone(start_page)!= zone" if the memmap contains garbage.And I only checked a handful of examples.
Ok, but as the first example shows in the absence of a problem report these pre-emptive changes might make things worse so I don't think it's as simple as go instrument all the
pfn_to_online_page()users.
Fixing pfn_to_online_page() is the right thing to do, not working around it eventually having false positives IMHO.