mm/kdump: exclude reserved pages in dumps

The reserved bit once was used to hinder pages from getting swapped. While this still works,

the semantics are a little bit stronger nowadays: The
page should never be touched by anybody in the system except by the owner. The original comment already gave a hint about that.

So especially, these pages should also not be dumped by dumping tools. Let's make that more clear by updating the comment.

This will be useful especially in the future in virtual environments where pages marked with the reserved bit might no longer be accessible.

Cc Andrew Morton <[email protected]>

Cc Michal Hocko <[email protected]>

Cc Vlastimil Babka <[email protected]>

Cc "Kirill A. Shutemov" <[email protected]>

Cc Matthew Wilcox <[email protected]>

Cc Greg Kroah-Hartman <[email protected]>

Cc David Hildenbrand <[email protected]>

Cc Huang Ying <[email protected]>

Cc Pavel Tatashin <[email protected]>

Cc Miles Chen <[email protected]>

Cc Dave Young <[email protected]>

Cc Baoquan He <[email protected]>

Cc "Marc-Andre Lureau" <[email protected]>

Cc Petr Tesarik <[email protected]>

Signed-off-by David Hildenbrand <[email protected]>

---
 include/linux/page-flags.h | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

include/linux/page-flags.h +2 -2

index 901943e4754b..ba81e11a868c 100644
@@ -17,8 +17,8 @@
 /*
  * Various page->flags bits:
  *
- * PG_reserved is set for special pages, which can never be swapped out. Some
- * of them might not even exist...
+ * PG_reserved is set for special pages, which should never be touched (read/
+ * write) by anybody except their owner. Some of them might not even exist.
  *
  * The PG_private bitflag is set on pagecache pages if they contain filesystem
  * specific data (which is normally at page->private). It can be used by

Signature · 1 line

2.17.1

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

This is relevant in virtual environments where we soon might want to report certain reserved pages to the hypervisor and they might no longer be accessible - what already was documented for reserved pages a long time ago ("might not even exist").

David Hildenbrand (2):
mm: clarify semantics of reserved pages
kdump: include PG_reserved value in VMCOREINFO

 include/linux/page-flags.h | 4 ++--
 kernel/crash_core.c        | 1 +
 2 files changed, 3 insertions(+), 2 deletions(-)

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

You are right. reserve_bootmem_region will make all early bootmem allocations (including those backing memmaps) PageReserved. I have asked several times but I haven't seen a satisfactory answer yet. Why do we even care for kdump about those. If they are reserved the nobody should really look at those specific struct pages and manipulate them. Kdump tools are using a kernel interface to read the content. If the specific content is backed by a non-existing memory then they should simply not return anything.

Quoted 10 lines · Michal Hocko

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

Quoted 9 lines

You are right. reserve_bootmem_region will make all early bootmem allocations (including those backing memmaps) PageReserved. I have asked several times but I haven't seen a satisfactory answer yet. Why do we even care for kdump about those. If they are reserved the nobody should really look at those specific struct pages and manipulate them. Kdump tools are using a kernel interface to read the content. If the specific content is backed by a non-existing memory then they should simply not return anything.

"new kernel" provides an interface to read memory from "old kernel".

The new kernel has no idea about
- which memory was added/online in the old kernel
- where struct pages of the old kernel are and what their content is
- which memory is save to touch and which not

Dump tools figure all that out by interpreting the VMCORE. They e.g. identify "struct pages" and see if they should be dumped. The "new kernel" only allows to read that memory. It cannot hinder to crash the system (e.g. if a dump tool would try to read a hwpoison page).

So how should the "new kernel" know if a page can be touched or not?

The *only* way would be to have an interface to the hypervisor where we "sense" if a memory location is safe to touch. I remember that xen or hyper-v does that - they fake a zero page in that case, after querying the hypervisor. But this does not sound like a clean approach to me, especially es we need yet another hypervisor interface to sense for memory provided via "some" device.

If we can find a way to just tag pages as "don't touch", it would be the easiest and cleanest solution in my opinion.

Quoted 33 lines · David Hildenbrand

Quoted 10 lines · Michal Hocko

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

Quoted 9 lines

You are right. reserve_bootmem_region will make all early bootmem allocations (including those backing memmaps) PageReserved. I have asked several times but I haven't seen a satisfactory answer yet. Why do we even care for kdump about those. If they are reserved the nobody should really look at those specific struct pages and manipulate them. Kdump tools are using a kernel interface to read the content. If the specific content is backed by a non-existing memory then they should simply not return anything.

"new kernel" provides an interface to read memory from "old kernel".

The new kernel has no idea about
- which memory was added/online in the old kernel
- where struct pages of the old kernel are and what their content is
- which memory is save to touch and which not

Dump tools figure all that out by interpreting the VMCORE. They e.g. identify "struct pages" and see if they should be dumped. The "new kernel" only allows to read that memory. It cannot hinder to crash the system (e.g. if a dump tool would try to read a hwpoison page).

Quoted 1 line

So how should the "new kernel" know if a page can be touched or not?

I am sorry I am not familiar with kdump much. But from what I remember it reads from /proc/vmcore and implementation of this interface should simply return EINVAL or alike when you try to dump inaccessible memory range.

Quoted 34 lines · Michal Hocko

Quoted 33 lines · David Hildenbrand

Quoted 10 lines · Michal Hocko

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

Quoted 9 lines

You are right. reserve_bootmem_region will make all early bootmem allocations (including those backing memmaps) PageReserved. I have asked several times but I haven't seen a satisfactory answer yet. Why do we even care for kdump about those. If they are reserved the nobody should really look at those specific struct pages and manipulate them. Kdump tools are using a kernel interface to read the content. If the specific content is backed by a non-existing memory then they should simply not return anything.

"new kernel" provides an interface to read memory from "old kernel".

The new kernel has no idea about
- which memory was added/online in the old kernel
- where struct pages of the old kernel are and what their content is
- which memory is save to touch and which not

Dump tools figure all that out by interpreting the VMCORE. They e.g. identify "struct pages" and see if they should be dumped. The "new kernel" only allows to read that memory. It cannot hinder to crash the system (e.g. if a dump tool would try to read a hwpoison page).

Quoted 6 lines

Quoted 1 line

So how should the "new kernel" know if a page can be touched or not?

I am sorry I am not familiar with kdump much. But from what I remember it reads from /proc/vmcore and implementation of this interface should simply return EINVAL or alike when you try to dump inaccessible memory range.

I assume the main problem with this approach is that we would always have to fallback to reading old memory from vmcore page by page. e.g. makedumpfile will always try to read bigger bunches. I also assume the reason HWPOISON is handled in dump tools instead of in the kernel using the mechanism you describe is the case.

Quoted 42 lines · David Hildenbrand

Quoted 34 lines · Michal Hocko

Quoted 33 lines · David Hildenbrand

Quoted 10 lines · Michal Hocko

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

Quoted 9 lines

You are right. reserve_bootmem_region will make all early bootmem allocations (including those backing memmaps) PageReserved. I have asked several times but I haven't seen a satisfactory answer yet. Why do we even care for kdump about those. If they are reserved the nobody should really look at those specific struct pages and manipulate them. Kdump tools are using a kernel interface to read the content. If the specific content is backed by a non-existing memory then they should simply not return anything.

"new kernel" provides an interface to read memory from "old kernel".

The new kernel has no idea about
- which memory was added/online in the old kernel
- where struct pages of the old kernel are and what their content is
- which memory is save to touch and which not

Dump tools figure all that out by interpreting the VMCORE. They e.g. identify "struct pages" and see if they should be dumped. The "new kernel" only allows to read that memory. It cannot hinder to crash the system (e.g. if a dump tool would try to read a hwpoison page).

Quoted 6 lines

Quoted 1 line

So how should the "new kernel" know if a page can be touched or not?

I am sorry I am not familiar with kdump much. But from what I remember it reads from /proc/vmcore and implementation of this interface should simply return EINVAL or alike when you try to dump inaccessible memory range.

Quoted 5 lines

I assume the main problem with this approach is that we would always have to fallback to reading old memory from vmcore page by page. e.g. makedumpfile will always try to read bigger bunches. I also assume the reason HWPOISON is handled in dump tools instead of in the kernel using the mechanism you describe is the case.

Is falling back to page-by-page for some ranges a real problem? I mean most of pages will simply be there so you can go in larger chunks. Once you get EINVAL, you just fall back to page-by-page for that particular range.

Quoted 33 lines · David Hildenbrand

Quoted 10 lines · Michal Hocko

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

Quoted 9 lines

You are right. reserve_bootmem_region will make all early bootmem allocations (including those backing memmaps) PageReserved. I have asked several times but I haven't seen a satisfactory answer yet. Why do we even care for kdump about those. If they are reserved the nobody should really look at those specific struct pages and manipulate them. Kdump tools are using a kernel interface to read the content. If the specific content is backed by a non-existing memory then they should simply not return anything.

"new kernel" provides an interface to read memory from "old kernel".

The new kernel has no idea about
- which memory was added/online in the old kernel
- where struct pages of the old kernel are and what their content is
- which memory is save to touch and which not

Dump tools figure all that out by interpreting the VMCORE. They e.g. identify "struct pages" and see if they should be dumped. The "new kernel" only allows to read that memory. It cannot hinder to crash the system (e.g. if a dump tool would try to read a hwpoison page).

Quoted 1 line

So how should the "new kernel" know if a page can be touched or not?

I am sorry I am not familiar with kdump much. But from what I remember it reads from /proc/vmcore and implementation of this interface should simply return EINVAL or alike when you try to dump inaccessible memory range.

Quoted 34 lines · Michal Hocko

Quoted 33 lines · David Hildenbrand

Quoted 10 lines · Michal Hocko

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

Quoted 9 lines

You are right. reserve_bootmem_region will make all early bootmem allocations (including those backing memmaps) PageReserved. I have asked several times but I haven't seen a satisfactory answer yet. Why do we even care for kdump about those. If they are reserved the nobody should really look at those specific struct pages and manipulate them. Kdump tools are using a kernel interface to read the content. If the specific content is backed by a non-existing memory then they should simply not return anything.

"new kernel" provides an interface to read memory from "old kernel".

The new kernel has no idea about
- which memory was added/online in the old kernel
- where struct pages of the old kernel are and what their content is
- which memory is save to touch and which not

Dump tools figure all that out by interpreting the VMCORE. They e.g. identify "struct pages" and see if they should be dumped. The "new kernel" only allows to read that memory. It cannot hinder to crash the system (e.g. if a dump tool would try to read a hwpoison page).

Quoted 6 lines

Quoted 1 line

So how should the "new kernel" know if a page can be touched or not?

I am sorry I am not familiar with kdump much. But from what I remember it reads from /proc/vmcore and implementation of this interface should simply return EINVAL or alike when you try to dump inaccessible memory range.

Oh, and BTW, while something like -EINVAL could work, we usually don't want to try to read certain pages at all (e.g. ballooned pages - accessing the page might work but involves quite some overhead in the hypervisor).

So we should either handle this in dump tools (reserved + ...?) or while doing the read similar to XEN (is_ram_page()).

I wonder if we could convert the early allocated memory (PG_reserved) at some point (buddy initialized) into ordinary "simply allocated" memory.

Quoted 47 lines · David Hildenbrand

Quoted 34 lines · Michal Hocko

Quoted 33 lines · David Hildenbrand

Quoted 10 lines · Michal Hocko

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

Quoted 9 lines

You are right. reserve_bootmem_region will make all early bootmem allocations (including those backing memmaps) PageReserved. I have asked several times but I haven't seen a satisfactory answer yet. Why do we even care for kdump about those. If they are reserved the nobody should really look at those specific struct pages and manipulate them. Kdump tools are using a kernel interface to read the content. If the specific content is backed by a non-existing memory then they should simply not return anything.

"new kernel" provides an interface to read memory from "old kernel".

The new kernel has no idea about
- which memory was added/online in the old kernel
- where struct pages of the old kernel are and what their content is
- which memory is save to touch and which not

Dump tools figure all that out by interpreting the VMCORE. They e.g. identify "struct pages" and see if they should be dumped. The "new kernel" only allows to read that memory. It cannot hinder to crash the system (e.g. if a dump tool would try to read a hwpoison page).

Quoted 6 lines

Quoted 1 line

So how should the "new kernel" know if a page can be touched or not?

I am sorry I am not familiar with kdump much. But from what I remember it reads from /proc/vmcore and implementation of this interface should simply return EINVAL or alike when you try to dump inaccessible memory range.

Oh, and BTW, while something like -EINVAL could work, we usually don't want to try to read certain pages at all (e.g. ballooned pages - accessing the page might work but involves quite some overhead in the hypervisor).

Quoted 2 lines

So we should either handle this in dump tools (reserved + ...?) or while doing the read similar to XEN (is_ram_page()).

Yes, I think this is the proper way. Just test for PageOnline in read_from_oldmem/copy_oldmem_page. Btw. we already page pfn_to_online_page which performs the per-section online/offline status. This should be extendable to consider your new PageOffline state.

Quoted 48 lines · Michal Hocko

Quoted 47 lines · David Hildenbrand

Quoted 34 lines · Michal Hocko

Quoted 33 lines · David Hildenbrand

Quoted 10 lines · Michal Hocko

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

Quoted 9 lines

You are right. reserve_bootmem_region will make all early bootmem allocations (including those backing memmaps) PageReserved. I have asked several times but I haven't seen a satisfactory answer yet. Why do we even care for kdump about those. If they are reserved the nobody should really look at those specific struct pages and manipulate them. Kdump tools are using a kernel interface to read the content. If the specific content is backed by a non-existing memory then they should simply not return anything.

"new kernel" provides an interface to read memory from "old kernel".

The new kernel has no idea about
- which memory was added/online in the old kernel
- where struct pages of the old kernel are and what their content is
- which memory is save to touch and which not

Dump tools figure all that out by interpreting the VMCORE. They e.g. identify "struct pages" and see if they should be dumped. The "new kernel" only allows to read that memory. It cannot hinder to crash the system (e.g. if a dump tool would try to read a hwpoison page).

Quoted 6 lines

Quoted 1 line

So how should the "new kernel" know if a page can be touched or not?

I am sorry I am not familiar with kdump much. But from what I remember it reads from /proc/vmcore and implementation of this interface should simply return EINVAL or alike when you try to dump inaccessible memory range.

Oh, and BTW, while something like -EINVAL could work, we usually don't want to try to read certain pages at all (e.g. ballooned pages - accessing the page might work but involves quite some overhead in the hypervisor).

Quoted 8 lines

Quoted 2 lines

So we should either handle this in dump tools (reserved + ...?) or while doing the read similar to XEN (is_ram_page()).

Yes, I think this is the proper way. Just test for PageOnline in read_from_oldmem/copy_oldmem_page. Btw. we already page pfn_to_online_page which performs the per-section online/offline status. This should be extendable to consider your new PageOffline state.

That is the important bit:

What the new kernel sees is not what the old kernel saw.

Checking for pfn_to_online_page() from read_from_oldmem/copy_oldmem_page() is plain wrong.

E.g. ACPI hotplug memory is not even added in the new kernel - see "acpi_no_memhotplug" which is used in kdump environments.

The only thing we can do is
- query the hypervisor
- try to access and get an exception

Quoted 68 lines · David Hildenbrand

Quoted 48 lines · Michal Hocko

Quoted 47 lines · David Hildenbrand

Quoted 34 lines · Michal Hocko

Quoted 33 lines · David Hildenbrand

Quoted 10 lines · Michal Hocko

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

Quoted 9 lines

You are right. reserve_bootmem_region will make all early bootmem allocations (including those backing memmaps) PageReserved. I have asked several times but I haven't seen a satisfactory answer yet. Why do we even care for kdump about those. If they are reserved the nobody should really look at those specific struct pages and manipulate them. Kdump tools are using a kernel interface to read the content. If the specific content is backed by a non-existing memory then they should simply not return anything.

"new kernel" provides an interface to read memory from "old kernel".

The new kernel has no idea about
- which memory was added/online in the old kernel
- where struct pages of the old kernel are and what their content is
- which memory is save to touch and which not

Dump tools figure all that out by interpreting the VMCORE. They e.g. identify "struct pages" and see if they should be dumped. The "new kernel" only allows to read that memory. It cannot hinder to crash the system (e.g. if a dump tool would try to read a hwpoison page).

Quoted 6 lines

Quoted 1 line

So how should the "new kernel" know if a page can be touched or not?

I am sorry I am not familiar with kdump much. But from what I remember it reads from /proc/vmcore and implementation of this interface should simply return EINVAL or alike when you try to dump inaccessible memory range.

Oh, and BTW, while something like -EINVAL could work, we usually don't want to try to read certain pages at all (e.g. ballooned pages - accessing the page might work but involves quite some overhead in the hypervisor).

Quoted 8 lines

Quoted 2 lines

So we should either handle this in dump tools (reserved + ...?) or while doing the read similar to XEN (is_ram_page()).

Yes, I think this is the proper way. Just test for PageOnline in read_from_oldmem/copy_oldmem_page. Btw. we already page pfn_to_online_page which performs the per-section online/offline status. This should be extendable to consider your new PageOffline state.

That is the important bit:

What the new kernel sees is not what the old kernel saw.

Checking for pfn_to_online_page() from read_from_oldmem/copy_oldmem_page() is plain wrong.

E.g. ACPI hotplug memory is not even added in the new kernel - see "acpi_no_memhotplug" which is used in kdump environments.

Quoted 3 lines

The only thing we can do is
- query the hypervisor
- try to access and get an exception

But we do preserve struct page's (aka memmap) from the crash kernel, don't we? So you have the whole state there. Or am I missing something?

So I see right now:

- Pg_reserved + e.g. new page type (or some other unique identifier in combination with Pg_reserved)
-> Avoid reads of pages we know are offline
- extend is_ram_page()
-> Fake zero memory for pages we know are offline

Or even both (avoid reading and don't crash the kernel if it is being done).

On Tue 24-07-18 16:13:09, David Hildenbrand wrote:
[...]

Quoted 8 lines

So I see right now:

- Pg_reserved + e.g. new page type (or some other unique identifier in combination with Pg_reserved)
-> Avoid reads of pages we know are offline
- extend is_ram_page()
-> Fake zero memory for pages we know are offline

Quoted 1 line

Or even both (avoid reading and don't crash the kernel if it is being done).

I really fail to see how that can work without kernel being aware of PageOffline. What will/should happen if you run an old kdump tool on a kernel with this partially offline memory?

Quoted 10 lines · Michal Hocko

On Tue 24-07-18 16:13:09, David Hildenbrand wrote:
[...]

Quoted 8 lines

So I see right now:

- Pg_reserved + e.g. new page type (or some other unique identifier in combination with Pg_reserved)
-> Avoid reads of pages we know are offline
- extend is_ram_page()
-> Fake zero memory for pages we know are offline

Quoted 6 lines

Quoted 1 line

Or even both (avoid reading and don't crash the kernel if it is being done).

I really fail to see how that can work without kernel being aware of PageOffline. What will/should happen if you run an old kdump tool on a kernel with this partially offline memory?

New kernel with old dump tool:

a) we have not fixed up is_ram_page()

-> crash, as we access memory we shouldn't

b) we have fixed up is_ram_page()

-> We have a callback to check for applicable memory in the hypervisor whether the parts are accessible / online or not accessible / offline. (e.g. via a device driver that controls a certain memory region)

-> Don't read, but fake a page full of 0

So instead of the kernel being aware of it, it asks via is_ram_page() the hypervisor.

I don't think a) is a problem. AFAICS, we have to update makedumpfile for every new kernel. We can perform changes and update makedumpfile to be compatible with new dump tools.

E.g. remember SECTION_IS_ONLINE you introduced ? It broke dump tools and required

Quoted 18 lines · David Hildenbrand

Quoted 16 lines · Michal Hocko

On Tue 24-07-18 16:13:09, David Hildenbrand wrote:
[...]

Quoted 9 lines

So I see right now:

- Pg_reserved + e.g. new page type (or some other unique identifier in combination with Pg_reserved)
-> Avoid reads of pages we know are offline
- extend is_ram_page()
-> Fake zero memory for pages we know are offline

Or even both (avoid reading and don't crash the kernel if it is being done).

I really fail to see how that can work without kernel being aware of PageOffline. What will/should happen if you run an old kdump tool on a kernel with this partially offline memory?

Quoted 5 lines

New kernel with old dump tool:

a) we have not fixed up is_ram_page()

-> crash, as we access memory we shouldn't

this is not acceptable, right? You do not want to crash your crash kernel ;)

Quoted 7 lines

b) we have fixed up is_ram_page()

-> We have a callback to check for applicable memory in the hypervisor whether the parts are accessible / online or not accessible / offline. (e.g. via a device driver that controls a certain memory region)

Quoted 5 lines

-> Don't read, but fake a page full of 0

So instead of the kernel being aware of it, it asks via is_ram_page() the hypervisor.

I am still confused why do we even care about hypervisor. What if somebody wants to have partial memory hotplug on native OS?

Quoted 4 lines

I don't think a) is a problem. AFAICS, we have to update makedumpfile for every new kernel. We can perform changes and update makedumpfile to be compatible with new dump tools.

Not really. You simply do not crash the kernel just because you are trying to dump the already crashed kernel.

Quoted 2 lines

E.g. remember SECTION_IS_ONLINE you introduced ? It broke dump tools and required

But has it crashed the kernel when reading the dump? If yes then the whole dumping is fragile as hell...

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

You are right. reserve_bootmem_region will make all early bootmem allocations (including those backing memmaps) PageReserved. I have asked several times but I haven't seen a satisfactory answer yet. Why do we even care for kdump about those. If they are reserved the nobody should really look at those specific struct pages and manipulate them. Kdump tools are using a kernel interface to read the content. If the specific content is backed by a non-existing memory then they should simply not return anything.

Quoted 10 lines · Michal Hocko

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

Quoted 8 lines

You are right. reserve_bootmem_region will make all early bootmem allocations (including those backing memmaps) PageReserved. I have asked several times but I haven't seen a satisfactory answer yet. Why do we even care for kdump about those. If they are reserved the nobody should really look at those specific struct pages and manipulate them. Kdump tools are using a kernel interface to read the content. If the specific content is backed by a non-existing memory then they should simply not return anything.

When creating a crashdump, I definitely need the pages containing memmap included in the dump, so I can inspect the struct pages. But this is a bit recursive issue, so I'll try making it clearer:

1) there are kernel pages with data (e.g. slab) that I typically need in the dump, and are not PageReserved
2) there are struct pages for pages 1) in the memmap that physically hold the pageflags for 1), and these are PageReserved
3) there are struct pages for pages 2) somewhere else in the memmap, physically hold the pageflags for 2). They are probably also PageReserved themselves ? and self-referencing.

Excluding PageReserved from dump means there won't be cases 2) and 3) in the dump, which at least for case 2) is making such dump almost useless in many cases.

Quoted 12 lines

When creating a crashdump, I definitely need the pages containing memmap included in the dump, so I can inspect the struct pages. But this is a bit recursive issue, so I'll try making it clearer:

1) there are kernel pages with data (e.g. slab) that I typically need in the dump, and are not PageReserved
2) there are struct pages for pages 1) in the memmap that physically hold the pageflags for 1), and these are PageReserved
3) there are struct pages for pages 2) somewhere else in the memmap, physically hold the pageflags for 2). They are probably also PageReserved themselves ? and self-referencing.

Quoted 3 lines

Excluding PageReserved from dump means there won't be cases 2) and 3) in the dump, which at least for case 2) is making such dump almost useless in many cases.

Yes, we cannot simply exclude all PageReserved pages. I was merely suggesting to rule out new special PageReserved pages that are denoting offline pages. The same could be applied to HWPoison pages

Quoted 12 lines · Michal Hocko

Quoted 12 lines

When creating a crashdump, I definitely need the pages containing memmap included in the dump, so I can inspect the struct pages. But this is a bit recursive issue, so I'll try making it clearer:

1) there are kernel pages with data (e.g. slab) that I typically need in the dump, and are not PageReserved
2) there are struct pages for pages 1) in the memmap that physically hold the pageflags for 1), and these are PageReserved
3) there are struct pages for pages 2) somewhere else in the memmap, physically hold the pageflags for 2). They are probably also PageReserved themselves ? and self-referencing.

Quoted 7 lines

Quoted 3 lines

Excluding PageReserved from dump means there won't be cases 2) and 3) in the dump, which at least for case 2) is making such dump almost useless in many cases.

Yes, we cannot simply exclude all PageReserved pages. I was merely suggesting to rule out new special PageReserved pages that are denoting offline pages. The same could be applied to HWPoison pages

So how about marking them with some "page type" that we got after Matthew's struct page reorg? I assume the pages we're talking about are in a state that they don't need the mapcount/mapping field or whatever unions with the page type... but I guess some care would be needed to not have false positives when the union field is actually used but happens to look like the new type.

Quoted 12 lines · Michal Hocko

Quoted 12 lines

When creating a crashdump, I definitely need the pages containing memmap included in the dump, so I can inspect the struct pages. But this is a bit recursive issue, so I'll try making it clearer:

1) there are kernel pages with data (e.g. slab) that I typically need in the dump, and are not PageReserved
2) there are struct pages for pages 1) in the memmap that physically hold the pageflags for 1), and these are PageReserved
3) there are struct pages for pages 2) somewhere else in the memmap, physically hold the pageflags for 2). They are probably also PageReserved themselves ? and self-referencing.

Quoted 7 lines

Quoted 3 lines

Excluding PageReserved from dump means there won't be cases 2) and 3) in the dump, which at least for case 2) is making such dump almost useless in many cases.

Yes, we cannot simply exclude all PageReserved pages. I was merely suggesting to rule out new special PageReserved pages that are denoting offline pages. The same could be applied to HWPoison pages

So how about marking them with some "page type" that we got after Matthew's struct page reorg? I assume the pages we're talking about are in a state that they don't need the mapcount/mapping field or whatever unions with the page type... but I guess some care would be needed to not have false positives when the union field is actually used but happens to look like the new type.

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

Quoted 14 lines

This is relevant in virtual environments where we soon might want to report certain reserved pages to the hypervisor and they might no longer be accessible - what already was documented for reserved pages a long time ago ("might not even exist").

David Hildenbrand (2):
mm: clarify semantics of reserved pages
kdump: include PG_reserved value in VMCOREINFO

 include/linux/page-flags.h | 4 ++--
 kernel/crash_core.c        | 1 +
 2 files changed, 3 insertions(+), 2 deletions(-)

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

I proposed a new flag/value to mask pages that are logically offline but Michal wanted me to go into this direction.

While we can special case struct pages in dump tools ("we have to read/interpret them either way, so we can also dump them"), it smells like my original attempt was cleaner. Michal?

Quoted 13 lines · David Hildenbrand

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

I proposed a new flag/value to mask pages that are logically offline but Michal wanted me to go into this direction.

Quoted 3 lines

While we can special case struct pages in dump tools ("we have to read/interpret them either way, so we can also dump them"), it smells like my original attempt was cleaner. Michal?

But we do not have many page flags spare and even if we have one or two this doesn't look like the use for them. So I still think we should try the PageReserved way.

Quoted 13 lines · David Hildenbrand

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

I proposed a new flag/value to mask pages that are logically offline but Michal wanted me to go into this direction.

Quoted 3 lines

While we can special case struct pages in dump tools ("we have to read/interpret them either way, so we can also dump them"), it smells like my original attempt was cleaner. Michal?

But we do not have many page flags spare and even if we have one or two this doesn't look like the use for them. So I still think we should try the PageReserved way.

Quoted 14 lines · Michal Hocko

Quoted 13 lines · David Hildenbrand

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

I proposed a new flag/value to mask pages that are logically offline but Michal wanted me to go into this direction.

Quoted 8 lines

Quoted 3 lines

While we can special case struct pages in dump tools ("we have to read/interpret them either way, so we can also dump them"), it smells like my original attempt was cleaner. Michal?

But we do not have many page flags spare and even if we have one or two this doesn't look like the use for them. So I still think we should try the PageReserved way.

So as a summary, the only real approach that would be acceptable is using PageReserved + some other identifier to mark pages as "logically offline".

I wonder what identifier could be used, as this has to be consistent for all reserved pages (to avoid false positives).

Using other pageflags in combination might be possible, but then we have to make assumptions about all users of PageReserved right now.

As far as I can see (and as has been discussed), page_type could be used. If we don't want to consume a new bit, we could overload/reuse the "PG_balloon" bit.

E.g. "PG_balloon" set -> exclude page from dump

Quoted 39 lines · David Hildenbrand

Quoted 14 lines · Michal Hocko

Quoted 13 lines · David Hildenbrand

Quoted 8 lines · Vlastimil Babka

Quoted 3 lines · David Hildenbrand

Dumping tools (like makedumpfile) right now don't exclude reserved pages. So reserved pages might be access by dump tools although nobody except the owner should touch them.

Are you sure about that? Or maybe I understand wrong. Maybe it changed recently, but IIRC pages that are backing memmap (struct pages) are also PG_reserved. And you definitely do want those in the dump.

I proposed a new flag/value to mask pages that are logically offline but Michal wanted me to go into this direction.

Quoted 8 lines

Quoted 3 lines

While we can special case struct pages in dump tools ("we have to read/interpret them either way, so we can also dump them"), it smells like my original attempt was cleaner. Michal?

But we do not have many page flags spare and even if we have one or two this doesn't look like the use for them. So I still think we should try the PageReserved way.

So as a summary, the only real approach that would be acceptable is using PageReserved + some other identifier to mark pages as "logically offline".

I wonder what identifier could be used, as this has to be consistent for all reserved pages (to avoid false positives).

Using other pageflags in combination might be possible, but then we have to make assumptions about all users of PageReserved right now.

As far as I can see (and as has been discussed), page_type could be used. If we don't want to consume a new bit, we could overload/reuse the "PG_balloon" bit.

Quoted 1 line

E.g. "PG_balloon" set -> exclude page from dump

Does each user of PG_balloon check for PG_reserved? If this is the case then yes this would be OK.

Quoted 3 lines

Does each user of PG_balloon check for PG_reserved? If this is the case then yes this would be OK.

I can only spot one user of PageBalloon() at all (fs/proc/page.c) , which makes me wonder if this bit is actually still relevant. I think the last "real" user was removed with

commit b1123ea6d3b3da25af5c8a9d843bd07ab63213f4

Author: Minchan Kim <[email protected]>
Date:   Tue Jul 26 15:23:09 2016 -0700

mm: balloon: use general non-lru movable page feature

Now, VM has a feature to migrate non-lru movable pages so balloon doesn't need custom migration hooks in migrate.c and compaction.c.

The only user of PG_balloon in general is "include/linux/balloon_compaction.h", used effectively only by virtio_balloon.

All such pages are allocated via balloon_page_alloc() and never set reserved.

So to me it looks like PG_balloon could be easily reused, especially to also exclude virtio-balloon pages from dumps.

Quoted 25 lines · On Thu, 26 Jul 2018 10:45:54 +0200 David Hildenbrand <[email protected]> wrote:

Quoted 3 lines

Does each user of PG_balloon check for PG_reserved? If this is the case then yes this would be OK.

I can only spot one user of PageBalloon() at all (fs/proc/page.c) , which makes me wonder if this bit is actually still relevant. I think the last "real" user was removed with

commit b1123ea6d3b3da25af5c8a9d843bd07ab63213f4

Author: Minchan Kim <[email protected]>
Date:   Tue Jul 26 15:23:09 2016 -0700

mm: balloon: use general non-lru movable page feature

Now, VM has a feature to migrate non-lru movable pages so balloon doesn't need custom migration hooks in migrate.c and compaction.c.

The only user of PG_balloon in general is "include/linux/balloon_compaction.h", used effectively only by virtio_balloon.

All such pages are allocated via balloon_page_alloc() and never set reserved.

Quoted 2 lines

So to me it looks like PG_balloon could be easily reused, especially to also exclude virtio-balloon pages from dumps.

Agree. Maintaining a thingy for page-types.c which hardly anyone uses (surely) isn't sufficient justification for consuming a page flag. We should check with the virtio developers first, but this does seem to be begging to be reclaimed.