Skip to contentSkip to conversation

uninitialized pmem struct pages

DISCUSSION Michal Hocko 3 participants 32 messages Jan 4, 2021 – Jan 5, 2021

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi,
back in March [1] you have recommended 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") to be backported to stable trees and that has led to a more general discussion about the current state of pfn walkers wrt. uninitialized pmem struct pages. We haven't concluded any specific solution for that except for a general sentiment that pfn_to_online_page should be able to catch those. I might have missed any follow ups on that but I do not think we have landed on any actual solution in the end. Have I just missed any followups?

Is anybody working on that?

Also is there any reference explaining what those struct pages are and why we cannot initialize them? I am sorry if this has been explained to me but I really cannot find that in my notes anywhere. Most pmem pages should be initialized via memmap_init_zone_device, right?

I am asking mostly because we are starting to see these issues in production and while the only visible problem so far is a crash when reading sysfs (removable file) I am worried we are just lucky no other pfn walker stumble over this.

[1] http://lkml.kernel.org/r/CAPcyv4jpdaNvJ67SkjyUJLBnBnXXQv686BiVW042g03FUmWLXw@mail.gmail.com

Signature · 2 lines
Michal Hocko
SUSE Labs
#1 lore raw
Quoted 9 lines · Michal Hocko

Hi,
back in March [1] you have recommended 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") to be backported to stable trees and that has led to a more general discussion about the current state of pfn walkers wrt. uninitialized pmem struct pages. We haven't concluded any specific solution for that except for a general sentiment that pfn_to_online_page should be able to catch those. I might have missed any follow ups on that but I do not think we have landed on any actual solution in the end. Have I just missed any followups?

Thanks for raising this issue. It's still on my list of "broken and non-trivial to fix" things.

So, as far as I recall, we still have the following two issues remaining:

1. pfn_to_online_page() false positives

The semantics of pfn_to_online_page() were broken with sub-section hot-add in corner cases.

If we have ZONE_DEVICE hot-added memory that overlaps in a section with boot memory, this memory section will contain parts ZONE_DEVICE memory and parts !ZONE_DEVICE memory. This can happen in sub-section granularity (2MB IIRC). pfn_to_online_page() will succeed on ZONE_DEVICE memory parts as the whole section is marked as online. Bad.

One instance where this is still an issue is mm/memory-failure.c:memory_failure() and mm/memory-failure.c:soft_offline_page(). I thought for a while about "fixing" these, but to me it felt like fixing pfn_to_online_page() is actually the right approach.

But worse, before ZONE_DEVICE hot-add
1. The whole section memmap does already exist (early sections always have a full memmap for the whole section)
2. The whole section memmap is initialized (although eventually with dummy node/zone 0/0 for memory holes until that part is fixed) and might be accessed by pfn walkers.

So when hotadding ZONE_DEVICE we are modifying already existing and visible memmaps. Bad.

2. Deferred init of ZONE_DEVICE ranges

memmap_init_zone_device() runs after the ZONE_DEVICE zone was resized and outside the memhp lock. I did not follow if the use of get_dev_pagemap() actually makes sure that memmap_init_zone_device() in pagemap_range() actually completed. I don't think it does.

Quoted 3 lines

Is anybody working on that?

I believe Dan mentioned somewhere that he wants to see a real instance of this producing a BUG before actually moving forward with a fix. I might be wrong.

We might tackle 1. by:

a) [worked-around] doing get_dev_pagemap() before pfn_to_online_page() - just plain ugly.

b) [worked-around] using a sub-section online-map and extending pfn_to_online_page(). IMHO ugly to fix this corner case.

c) [worked-around] extending pfn_to_online_page() by zone checks. IMHO racy.

d) fixed by not allowing ZONE_DEVICE and !ZONE_DEVICE within a single section. In the worst case, don't add partially present sections that have big holes in the beginning/end. Like, if there is a 128MB section with 126MB of memory followed by a 2MB hole, don't add that memory to Linux with CONFIG_ZONE_DEVICE. Might turn some memory unusable, but well, it would be the price to pay for simplicity. Being able to hotadd PMEM is more important than using each and every last MB of memory.

e) decrease the section size and drop sub-section hotadd support. As sub-section hotadd is 2MB and MAX_ORDER - 1 corresponds 4MB, this is mostly impossible. Worse on aarch64 with 64k base pages - 1024MB sections and MAX_ORDER - 1 corresponds 512MB. I think this is not feasible.

We might tackle 2. by making sure that get_dev_pagemap() only succeeds after memmap_init_zone_device() finished. As I said, not sure if that is already done.

Quoted 4 lines

Also is there any reference explaining what those struct pages are and why we cannot initialize them? I am sorry if this has been explained to me but I really cannot find that in my notes anywhere. Most pmem pages should be initialized via memmap_init_zone_device, right?

Long story short, IMHO we have to

a) fix pfn_to_online_page() to only succeed for !ZONE_DEVICE memory.
b) fix get_dev_pagemap() to only succeed if memmap_init_zone_device() completed. (again, unless this already works as desired)

Dealing with races (memory_offlining() racing with pfn_to_online_page()) is another story and stuff for the future. Another level of complexity :)

Quoted 4 lines

I am asking mostly because we are starting to see these issues in production and while the only visible problem so far is a crash when reading sysfs (removable file) I am worried we are just lucky no other pfn walker stumble over this.

Which exact issue are you seeing? The one tackled by "[PATCH v1] drivers/base/memory.c: indicate all memory blocks as removable" ?

Signature · 3 lines
Thanks,

David / dhildenb
#2 lore raw
Quoted 18 lines · David Hildenbrand
Quoted 9 lines · Michal Hocko

Hi,
back in March [1] you have recommended 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") to be backported to stable trees and that has led to a more general discussion about the current state of pfn walkers wrt. uninitialized pmem struct pages. We haven't concluded any specific solution for that except for a general sentiment that pfn_to_online_page should be able to catch those. I might have missed any follow ups on that but I do not think we have landed on any actual solution in the end. Have I just missed any followups?

Thanks for raising this issue. It's still on my list of "broken and non-trivial to fix" things.

So, as far as I recall, we still have the following two issues remaining:

1. pfn_to_online_page() false positives

Quoted 8 lines

The semantics of pfn_to_online_page() were broken with sub-section hot-add in corner cases.

If we have ZONE_DEVICE hot-added memory that overlaps in a section with boot memory, this memory section will contain parts ZONE_DEVICE memory and parts !ZONE_DEVICE memory. This can happen in sub-section granularity (2MB IIRC). pfn_to_online_page() will succeed on ZONE_DEVICE memory parts as the whole section is marked as online. Bad.

OK, I was not aware of this problem. Anyway, those pages should be still allocated and their state should retain their last state. I would have to double check but this shouldn't be harmfull. Or what would be an actual problem?

Quoted 13 lines

One instance where this is still an issue is mm/memory-failure.c:memory_failure() and mm/memory-failure.c:soft_offline_page(). I thought for a while about "fixing" these, but to me it felt like fixing pfn_to_online_page() is actually the right approach.

But worse, before ZONE_DEVICE hot-add
1. The whole section memmap does already exist (early sections always have a full memmap for the whole section)
2. The whole section memmap is initialized (although eventually with dummy node/zone 0/0 for memory holes until that part is fixed) and might be accessed by pfn walkers.

Quoted 2 lines

So when hotadding ZONE_DEVICE we are modifying already existing and visible memmaps. Bad.

Could you elaborate please?

Quoted 6 lines

2. Deferred init of ZONE_DEVICE ranges

memmap_init_zone_device() runs after the ZONE_DEVICE zone was resized and outside the memhp lock. I did not follow if the use of get_dev_pagemap() actually makes sure that memmap_init_zone_device() in pagemap_range() actually completed. I don't think it does.

So a pfn walker can see an unitialized struct page for a while, right?

The problem that I have encountered is that some zone device pages are not initialized at all. That sounds like a different from those 2 above. I am having hard time to track what kind of pages those are and why we cannot initialized their zone/node and make them reserved at least.

Quoted 6 lines
Quoted 2 lines

Is anybody working on that?

I believe Dan mentioned somewhere that he wants to see a real instance of this producing a BUG before actually moving forward with a fix. I might be wrong.

We have seen reports about those uninitialized struct pages on our 5.3 based kernels. Backporting 53cdc1cb29e8 helped for the particular report but I still consider it a workaround rather than a fix. I do not have any reports for other pfn walkers but we might be just lucky and I will sleep better if I do not have rely on the luck.

[...]

I will think about your proposed solutions after I manage to get through my email backlog.

Thanks!

Signature · 2 lines
Michal Hocko
SUSE Labs
#3 lore raw
Quoted 46 lines · Michal Hocko
Quoted 18 lines · David Hildenbrand
Quoted 9 lines · Michal Hocko

Hi,
back in March [1] you have recommended 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") to be backported to stable trees and that has led to a more general discussion about the current state of pfn walkers wrt. uninitialized pmem struct pages. We haven't concluded any specific solution for that except for a general sentiment that pfn_to_online_page should be able to catch those. I might have missed any follow ups on that but I do not think we have landed on any actual solution in the end. Have I just missed any followups?

Thanks for raising this issue. It's still on my list of "broken and non-trivial to fix" things.

So, as far as I recall, we still have the following two issues remaining:

1. pfn_to_online_page() false positives

Quoted 8 lines

The semantics of pfn_to_online_page() were broken with sub-section hot-add in corner cases.

If we have ZONE_DEVICE hot-added memory that overlaps in a section with boot memory, this memory section will contain parts ZONE_DEVICE memory and parts !ZONE_DEVICE memory. This can happen in sub-section granularity (2MB IIRC). pfn_to_online_page() will succeed on ZONE_DEVICE memory parts as the whole section is marked as online. Bad.

OK, I was not aware of this problem. Anyway, those pages should be still allocated and their state should retain their last state. I would have to double check but this shouldn't be harmfull. Or what would be an actual problem?

Quoted 13 lines

One instance where this is still an issue is mm/memory-failure.c:memory_failure() and mm/memory-failure.c:soft_offline_page(). I thought for a while about "fixing" these, but to me it felt like fixing pfn_to_online_page() is actually the right approach.

But worse, before ZONE_DEVICE hot-add
1. The whole section memmap does already exist (early sections always have a full memmap for the whole section)
2. The whole section memmap is initialized (although eventually with dummy node/zone 0/0 for memory holes until that part is fixed) and might be accessed by pfn walkers.

Quoted 4 lines
Quoted 2 lines

So when hotadding ZONE_DEVICE we are modifying already existing and visible memmaps. Bad.

Could you elaborate please?

Simplistic example: Assume you have a VM with 64MB on x86-64.

We need exactly one memory section (-> one memory block device). We allocate the memmap for a full section - an "early section". So we have a memmap for 128MB, while 64MB are actually in use, the other 64MB is initialized (like a memory hole). pfn_to_online_page() would return a valid struct page for the whole section memmap.

The remaining 64MB can later be used for hot-adding ZONE_DEVICE memory, essentially re-initializing that part of the already-existing memmap.

See pfn_valid():

/*
* Traditionally early sections always returned pfn_valid() for
* the entire section-sized span.
*/
return early_section(ms) || pfn_section_valid(ms, pfn);

Depending on the memory layout of the system, a pfn walker might just be about to stumble over this range getting re-initialized.

Quoted 8 lines
Quoted 7 lines

2. Deferred init of ZONE_DEVICE ranges

memmap_init_zone_device() runs after the ZONE_DEVICE zone was resized and outside the memhp lock. I did not follow if the use of get_dev_pagemap() actually makes sure that memmap_init_zone_device() in pagemap_range() actually completed. I don't think it does.

Quoted 6 lines

So a pfn walker can see an unitialized struct page for a while, right?

The problem that I have encountered is that some zone device pages are not initialized at all. That sounds like a different from those 2 above. I am having hard time to track what kind of pages those are and why we cannot initialized their zone/node and make them reserved at least.

And you are sure that these are in fact ZONE_DEVICE pages? Not memory holes e.g., tackled by

commit 4b094b7851bf4bf551ad456195d3f26e1c03bd74

Author: David Hildenbrand <[email protected]>
Date:   Mon Feb 3 17:33:55 2020 -0800

mm/page_alloc.c: initialize memmap of unavailable memory directly

commit e822969cab48b786b64246aad1a3ba2a774f5d23

Author: David Hildenbrand <[email protected]>
Date:   Mon Feb 3 17:33:48 2020 -0800

mm/page_alloc.c: fix uninitialized memmaps on a partially populated
last section

(note there is currently an upstream discussion on improving this initialization, especially getting better node/zone information, mostly involving Andrea and Mike - but it only changes "how" these parts are initialized, not "if" or "when")

---

However, I do remember a discussion regarding "reserved altmap space" ZONE_DEVICE ranges, and whether to initialize them or leave them uninitialized. See comment in

commit 77e080e7680e1e615587352f70c87b9e98126d03

Author: Aneesh Kumar K.V <[email protected]>
Date:   Fri Oct 18 20:19:39 2019 -0700

mm/memunmap: don't access uninitialized memmap in memunmap_pages()

"With an altmap, the memmap falling into the reserved altmap space are not initialized and, therefore, contain a garbage NID and a garbage zone.".

I think the issue is that the ZONE_DEVICE pages that *host* the memmap of other pages might be left uninitialized.

Like pfn_to_page(VIRT_TO_PFN(pfn_to_page(zone_device_pfn))), which falls onto ZONE_DEVICE with an altmap, could be uninitialized. This is very similar to our Oscar's vmemmap-on-hotadded-memory approach, however, there we implicitly initialize the memmap of these pages just by the way the vmemmap is placed at the beginning of the memory block.

If altmap-reserved space is placed partially into an early section that is marked as online (issue 1. I described), we have the same issue as 1., just a little harder to fix :)

Quoted 8 lines
Quoted 7 lines
Quoted 3 lines

Is anybody working on that?

I believe Dan mentioned somewhere that he wants to see a real instance of this producing a BUG before actually moving forward with a fix. I might be wrong.

Quoted 5 lines

We have seen reports about those uninitialized struct pages on our 5.3 based kernels. Backporting 53cdc1cb29e8 helped for the particular report but I still consider it a workaround rather than a fix. I do not have any reports for other pfn walkers but we might be just lucky and I will sleep better if I do not have rely on the luck.

Yeah, I think we are now at 3 different but related problems.

Signature · 3 lines
Thanks,

David / dhildenb
#4 lore raw
Quoted 2 lines · David Hildenbrand
Quoted 1 line · Michal Hocko

On Mon 04-01-21 11:45:39, David Hildenbrand wrote:

[....]

Quoted 38 lines
Quoted 13 lines
Quoted 13 lines

One instance where this is still an issue is mm/memory-failure.c:memory_failure() and mm/memory-failure.c:soft_offline_page(). I thought for a while about "fixing" these, but to me it felt like fixing pfn_to_online_page() is actually the right approach.

But worse, before ZONE_DEVICE hot-add
1. The whole section memmap does already exist (early sections always have a full memmap for the whole section)
2. The whole section memmap is initialized (although eventually with dummy node/zone 0/0 for memory holes until that part is fixed) and might be accessed by pfn walkers.

Quoted 4 lines
Quoted 2 lines

So when hotadding ZONE_DEVICE we are modifying already existing and visible memmaps. Bad.

Could you elaborate please?

Simplistic example: Assume you have a VM with 64MB on x86-64.

We need exactly one memory section (-> one memory block device). We allocate the memmap for a full section - an "early section". So we have a memmap for 128MB, while 64MB are actually in use, the other 64MB is initialized (like a memory hole). pfn_to_online_page() would return a valid struct page for the whole section memmap.

The remaining 64MB can later be used for hot-adding ZONE_DEVICE memory, essentially re-initializing that part of the already-existing memmap.

See pfn_valid():

/*
* Traditionally early sections always returned pfn_valid() for
* the entire section-sized span.
*/
return early_section(ms) || pfn_section_valid(ms, pfn);

Quoted 2 lines

Depending on the memory layout of the system, a pfn walker might just be about to stumble over this range getting re-initialized.

Right. But as long as pfn walkers are not synchronized with the memory hotplug this is a general problem with any struct page. Whether it belongs to pmem or a regular memory, no?

Quoted 9 lines
Quoted 9 lines
Quoted 6 lines

2. Deferred init of ZONE_DEVICE ranges

memmap_init_zone_device() runs after the ZONE_DEVICE zone was resized and outside the memhp lock. I did not follow if the use of get_dev_pagemap() actually makes sure that memmap_init_zone_device() in pagemap_range() actually completed. I don't think it does.

So a pfn walker can see an unitialized struct page for a while, right?

Quoted 7 lines
Quoted 4 lines

The problem that I have encountered is that some zone device pages are not initialized at all. That sounds like a different from those 2 above. I am having hard time to track what kind of pages those are and why we cannot initialized their zone/node and make them reserved at least.

And you are sure that these are in fact ZONE_DEVICE pages? Not memory holes e.g., tackled by

Well, the physical address matches the pmem range so I believe this is the case.

[...]

Quoted 9 lines

However, I do remember a discussion regarding "reserved altmap space" ZONE_DEVICE ranges, and whether to initialize them or leave them uninitialized. See comment in

commit 77e080e7680e1e615587352f70c87b9e98126d03

Author: Aneesh Kumar K.V <[email protected]>
Date:   Fri Oct 18 20:19:39 2019 -0700

mm/memunmap: don't access uninitialized memmap in memunmap_pages()

yes, the reserved altmap space sounds like it might be it.

Quoted 12 lines

"With an altmap, the memmap falling into the reserved altmap space are not initialized and, therefore, contain a garbage NID and a garbage zone.".

I think the issue is that the ZONE_DEVICE pages that *host* the memmap of other pages might be left uninitialized.

Like pfn_to_page(VIRT_TO_PFN(pfn_to_page(zone_device_pfn))), which falls onto ZONE_DEVICE with an altmap, could be uninitialized. This is very similar to our Oscar's vmemmap-on-hotadded-memory approach, however, there we implicitly initialize the memmap of these pages just by the way the vmemmap is placed at the beginning of the memory block.

Quoted 3 lines

If altmap-reserved space is placed partially into an early section that is marked as online (issue 1. I described), we have the same issue as 1., just a little harder to fix :)

Would it be possible to iterate over the reserved space and initialize Node/zones at least?

Thanks!

Signature · 2 lines
Michal Hocko
SUSE Labs
#5 lore raw
Quoted 42 lines · Michal Hocko
Quoted 2 lines · David Hildenbrand
Quoted 1 line · Michal Hocko

On Mon 04-01-21 11:45:39, David Hildenbrand wrote:

[....]

Quoted 38 lines
Quoted 13 lines
Quoted 13 lines

One instance where this is still an issue is mm/memory-failure.c:memory_failure() and mm/memory-failure.c:soft_offline_page(). I thought for a while about "fixing" these, but to me it felt like fixing pfn_to_online_page() is actually the right approach.

But worse, before ZONE_DEVICE hot-add
1. The whole section memmap does already exist (early sections always have a full memmap for the whole section)
2. The whole section memmap is initialized (although eventually with dummy node/zone 0/0 for memory holes until that part is fixed) and might be accessed by pfn walkers.

Quoted 4 lines
Quoted 2 lines

So when hotadding ZONE_DEVICE we are modifying already existing and visible memmaps. Bad.

Could you elaborate please?

Simplistic example: Assume you have a VM with 64MB on x86-64.

We need exactly one memory section (-> one memory block device). We allocate the memmap for a full section - an "early section". So we have a memmap for 128MB, while 64MB are actually in use, the other 64MB is initialized (like a memory hole). pfn_to_online_page() would return a valid struct page for the whole section memmap.

The remaining 64MB can later be used for hot-adding ZONE_DEVICE memory, essentially re-initializing that part of the already-existing memmap.

See pfn_valid():

/*
* Traditionally early sections always returned pfn_valid() for
* the entire section-sized span.
*/
return early_section(ms) || pfn_section_valid(ms, pfn);

Quoted 6 lines
Quoted 2 lines

Depending on the memory layout of the system, a pfn walker might just be about to stumble over this range getting re-initialized.

Right. But as long as pfn walkers are not synchronized with the memory hotplug this is a general problem with any struct page. Whether it belongs to pmem or a regular memory, no?

Yes, however in this case even the memory hotplug lock does not help. But yes, related issues.

Quoted 33 lines
Quoted 10 lines
Quoted 10 lines
Quoted 7 lines

2. Deferred init of ZONE_DEVICE ranges

memmap_init_zone_device() runs after the ZONE_DEVICE zone was resized and outside the memhp lock. I did not follow if the use of get_dev_pagemap() actually makes sure that memmap_init_zone_device() in pagemap_range() actually completed. I don't think it does.

So a pfn walker can see an unitialized struct page for a while, right?

Quoted 7 lines
Quoted 4 lines

The problem that I have encountered is that some zone device pages are not initialized at all. That sounds like a different from those 2 above. I am having hard time to track what kind of pages those are and why we cannot initialized their zone/node and make them reserved at least.

And you are sure that these are in fact ZONE_DEVICE pages? Not memory holes e.g., tackled by

Well, the physical address matches the pmem range so I believe this is the case.

[...]

Quoted 9 lines

However, I do remember a discussion regarding "reserved altmap space" ZONE_DEVICE ranges, and whether to initialize them or leave them uninitialized. See comment in

commit 77e080e7680e1e615587352f70c87b9e98126d03

Author: Aneesh Kumar K.V <[email protected]>
Date:   Fri Oct 18 20:19:39 2019 -0700

mm/memunmap: don't access uninitialized memmap in memunmap_pages()

yes, the reserved altmap space sounds like it might be it.

[...]

Quoted 2 lines

Would it be possible to iterate over the reserved space and initialize Node/zones at least?

Right, I was confused by the terminology. We actually initialize the pages used for memory mapping in move_pfn_range_to_zone()->memmap_init_zone(). But we seem to exclude the "reserved space" - I think for good reason.

I think the issue is that this "reserved space" might actually get overridden by something else later, as it won't be used as a memmap, but just to store "anything".

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Signature · 3 lines
Thanks,

David / dhildenb
#6 lore raw
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

---
[1] it shares the same memory block with the regular memory as those are in the same 2G range.

Signature · 2 lines
Michal Hocko
SUSE Labs
#7 lore raw
Quoted 6 lines · Michal Hocko
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Quoted 7 lines

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

Okay, so we're not dealing with the "early section" mess I described, different story.

Due to [1], is_mem_section_removable() called pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not initialized.

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

If it's not self-hosted, initializing the relevant memmaps should work just fine I guess. Otherwise things get more complicated.

Signature · 3 lines
Thanks,

David / dhildenb
#8 lore raw
Quoted 22 lines · David Hildenbrand
Quoted 6 lines · Michal Hocko
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Quoted 7 lines

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

Okay, so we're not dealing with the "early section" mess I described, different story.

Due to [1], is_mem_section_removable() called pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not initialized.

Quoted 8 lines

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

If it's not self-hosted, initializing the relevant memmaps should work just fine I guess. Otherwise things get more complicated.

Oh, I forgot: pfn_to_online_page() should at least in your example make sure other pfn walkers are safe. It was just an issue of is_mem_section_removable().

Signature · 3 lines
Thanks,

David / dhildenb
#9 lore raw
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · David Hildenbrand
Quoted 6 lines · Michal Hocko
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Quoted 7 lines

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

Okay, so we're not dealing with the "early section" mess I described, different story.

Due to [1], is_mem_section_removable() called pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not initialized.

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

Quoted 6 lines
Quoted 2 lines

If it's not self-hosted, initializing the relevant memmaps should work just fine I guess. Otherwise things get more complicated.

Oh, I forgot: pfn_to_online_page() should at least in your example make sure other pfn walkers are safe. It was just an issue of is_mem_section_removable().

Hmm, I suspect you are right. I haven't put this together, thanks! The memory section is indeed marked offline so pfn_to_online_page would indeed bail out:
crash> p (0x6060000>>15)
$3 = 3084
crash> p mem_section[3084/128][3084 & 127]
$4 = {
section_mem_map = 18446736128020054019,
usage = 0xffff902dcf956680,
page_ext = 0x0,
pad = 0
}
crash> p 18446736128020054019 & (1UL<<2)
$5 = 0

That makes it considerably less of a problem than I thought!

Thanks David!

Signature · 2 lines
Michal Hocko
SUSE Labs
#10 lore raw
Quoted 52 lines · Michal Hocko
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · David Hildenbrand
Quoted 6 lines · Michal Hocko
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Quoted 7 lines

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

Okay, so we're not dealing with the "early section" mess I described, different story.

Due to [1], is_mem_section_removable() called pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not initialized.

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

Quoted 6 lines
Quoted 2 lines

If it's not self-hosted, initializing the relevant memmaps should work just fine I guess. Otherwise things get more complicated.

Oh, I forgot: pfn_to_online_page() should at least in your example make sure other pfn walkers are safe. It was just an issue of is_mem_section_removable().

Hmm, I suspect you are right. I haven't put this together, thanks! The memory section is indeed marked offline so pfn_to_online_page would indeed bail out:
crash> p (0x6060000>>15)
$3 = 3084
crash> p mem_section[3084/128][3084 & 127]
$4 = {
section_mem_map = 18446736128020054019,
usage = 0xffff902dcf956680,
page_ext = 0x0,
pad = 0
}
crash> p 18446736128020054019 & (1UL<<2)
$5 = 0

Quoted 1 line

That makes it considerably less of a problem than I thought!

Forgot to add that those who are running kernels without 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") for some reason can fix the crash by the following simple patch.

drivers/base/memory.c +7 -2
Index: linux-5.3-users_mhocko_SLE15-SP2_for-next/drivers/base/memory.c
===================================================================
@@ -152,9 +152,14 @@ static ssize_t removable_show(struct dev
 		goto out;
 
 	for (i = 0; i < sections_per_block; i++) {
-		if (!present_section_nr(mem->start_section_nr + i))
+		unsigned long nr = mem->start_section_nr + i;
+		if (!present_section_nr(nr))
 			continue;
-		pfn = section_nr_to_pfn(mem->start_section_nr + i);
+		if (!online_section_nr()) {
+			ret = 0;
+			break;
+		}
+		pfn = section_nr_to_pfn(nr);
 		ret &= is_mem_section_removable(pfn, PAGES_PER_SECTION);
 	}
 
Signature · 2 lines
Michal Hocko
SUSE Labs
#11 lore raw
Quoted 65 lines · Michal Hocko
Quoted 52 lines · Michal Hocko
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · David Hildenbrand
Quoted 6 lines · Michal Hocko
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Quoted 7 lines

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

Okay, so we're not dealing with the "early section" mess I described, different story.

Due to [1], is_mem_section_removable() called pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not initialized.

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

Quoted 6 lines
Quoted 2 lines

If it's not self-hosted, initializing the relevant memmaps should work just fine I guess. Otherwise things get more complicated.

Oh, I forgot: pfn_to_online_page() should at least in your example make sure other pfn walkers are safe. It was just an issue of is_mem_section_removable().

Hmm, I suspect you are right. I haven't put this together, thanks! The memory section is indeed marked offline so pfn_to_online_page would indeed bail out:
crash> p (0x6060000>>15)
$3 = 3084
crash> p mem_section[3084/128][3084 & 127]
$4 = {
section_mem_map = 18446736128020054019,
usage = 0xffff902dcf956680,
page_ext = 0x0,
pad = 0
}
crash> p 18446736128020054019 & (1UL<<2)
$5 = 0

Quoted 1 line

That makes it considerably less of a problem than I thought!

Forgot to add that those who are running kernels without 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") for some reason can fix the crash by the following simple patch.

Index: linux-5.3-users_mhocko_SLE15-SP2_for-next/drivers/base/memory.c
===================================================================

drivers/base/memory.c static ssize_t removable_show(struct dev · line 158

Quoted 9 lines
drivers/base/memory.c +3 -2
@@ -152,9 +152,14 @@ static ssize_t removable_show(struct dev
                goto out;
 
        for (i = 0; i < sections_per_block; i++) {
-               if (!present_section_nr(mem->start_section_nr + i))
+               unsigned long nr = mem->start_section_nr + i;
+               if (!present_section_nr(nr))
                        continue;
-               pfn = section_nr_to_pfn(mem->start_section_nr + i);
+               if (!online_section_nr()) {

I assume that's onlince_section_nr(nr) in the version that compiles?

This makes sense because the memory block size is larger than the section size. I suspect you have 1GB memory block size on this system, but since the System RAM and PMEM collide at a 512MB alignment in a memory block you end up walking the back end of the last 512MB of the System RAM memory block and run into the offline PMEM section.

So, I don't think it's pfn_to_online_page that necessarily needs to know how to disambiguate each page, it's things that walk sections and memory blocks and expects them to be consistent over the span.

#12 lore raw
Quoted 68 lines · Dan Williams
Quoted 67 lines · Michal Hocko
Quoted 52 lines · Michal Hocko
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · David Hildenbrand
Quoted 6 lines · Michal Hocko
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Quoted 7 lines

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

Okay, so we're not dealing with the "early section" mess I described, different story.

Due to [1], is_mem_section_removable() called pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not initialized.

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

Quoted 6 lines
Quoted 2 lines

If it's not self-hosted, initializing the relevant memmaps should work just fine I guess. Otherwise things get more complicated.

Oh, I forgot: pfn_to_online_page() should at least in your example make sure other pfn walkers are safe. It was just an issue of is_mem_section_removable().

Hmm, I suspect you are right. I haven't put this together, thanks! The memory section is indeed marked offline so pfn_to_online_page would indeed bail out:
crash> p (0x6060000>>15)
$3 = 3084
crash> p mem_section[3084/128][3084 & 127]
$4 = {
section_mem_map = 18446736128020054019,
usage = 0xffff902dcf956680,
page_ext = 0x0,
pad = 0
}
crash> p 18446736128020054019 & (1UL<<2)
$5 = 0

Quoted 1 line

That makes it considerably less of a problem than I thought!

Forgot to add that those who are running kernels without 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") for some reason can fix the crash by the following simple patch.

Index: linux-5.3-users_mhocko_SLE15-SP2_for-next/drivers/base/memory.c
===================================================================

drivers/base/memory.c +0 -0
@@ -152,9 +152,14 @@ static ssize_t removable_show(struct dev
                goto out;
 
Quoted 9 lines

drivers/base/memory.c static ssize_t removable_show(struct dev · line 158

Quoted 7 lines
drivers/base/memory.c +3 -2
        for (i = 0; i < sections_per_block; i++) {
-               if (!present_section_nr(mem->start_section_nr + i))
+               unsigned long nr = mem->start_section_nr + i;
+               if (!present_section_nr(nr))
                        continue;
-               pfn = section_nr_to_pfn(mem->start_section_nr + i);
+               if (!online_section_nr()) {

I assume that's onlince_section_nr(nr) in the version that compiles?

Yup.

Quoted 5 lines

This makes sense because the memory block size is larger than the section size. I suspect you have 1GB memory block size on this system, but since the System RAM and PMEM collide at a 512MB alignment in a memory block you end up walking the back end of the last 512MB of the System RAM memory block and run into the offline PMEM section.

Sections are 128MB and memory blocks are 2GB on this system.

Quoted 3 lines

So, I don't think it's pfn_to_online_page that necessarily needs to know how to disambiguate each page, it's things that walk sections and memory blocks and expects them to be consistent over the span.

Well, memory hotplug code is hard wired to sparse memory model so in this particular case asking about the section is ok. But pfn walkers shouldn't really care and only rely on pfn_to_online_page. But that will do the right thing here. So we are good as long as the section is marked properly. But this would become a problem as soon as the uninitialized pages where sharing the same memory section as David pointed out. pfn_to_online_page would then return something containing garbage. So we should still think of a way to either initialize all those pages or make sure pfn_to_online_page recognizes them. The former is preferred IMHO.

Signature · 2 lines
Michal Hocko
SUSE Labs
#13 lore raw
Quoted 94 lines · Michal Hocko
Quoted 68 lines · Dan Williams
Quoted 67 lines · Michal Hocko
Quoted 52 lines · Michal Hocko
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · David Hildenbrand
Quoted 6 lines · Michal Hocko
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Quoted 7 lines

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

Okay, so we're not dealing with the "early section" mess I described, different story.

Due to [1], is_mem_section_removable() called pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not initialized.

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

Quoted 6 lines
Quoted 2 lines

If it's not self-hosted, initializing the relevant memmaps should work just fine I guess. Otherwise things get more complicated.

Oh, I forgot: pfn_to_online_page() should at least in your example make sure other pfn walkers are safe. It was just an issue of is_mem_section_removable().

Hmm, I suspect you are right. I haven't put this together, thanks! The memory section is indeed marked offline so pfn_to_online_page would indeed bail out:
crash> p (0x6060000>>15)
$3 = 3084
crash> p mem_section[3084/128][3084 & 127]
$4 = {
section_mem_map = 18446736128020054019,
usage = 0xffff902dcf956680,
page_ext = 0x0,
pad = 0
}
crash> p 18446736128020054019 & (1UL<<2)
$5 = 0

Quoted 1 line

That makes it considerably less of a problem than I thought!

Forgot to add that those who are running kernels without 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") for some reason can fix the crash by the following simple patch.

Index: linux-5.3-users_mhocko_SLE15-SP2_for-next/drivers/base/memory.c
===================================================================

drivers/base/memory.c +0 -0
@@ -152,9 +152,14 @@ static ssize_t removable_show(struct dev
                goto out;
 
Quoted 9 lines

drivers/base/memory.c static ssize_t removable_show(struct dev · line 158

Quoted 7 lines
drivers/base/memory.c +3 -2
        for (i = 0; i < sections_per_block; i++) {
-               if (!present_section_nr(mem->start_section_nr + i))
+               unsigned long nr = mem->start_section_nr + i;
+               if (!present_section_nr(nr))
                        continue;
-               pfn = section_nr_to_pfn(mem->start_section_nr + i);
+               if (!online_section_nr()) {

I assume that's onlince_section_nr(nr) in the version that compiles?

Yup.

Quoted 5 lines

This makes sense because the memory block size is larger than the section size. I suspect you have 1GB memory block size on this system, but since the System RAM and PMEM collide at a 512MB alignment in a memory block you end up walking the back end of the last 512MB of the System RAM memory block and run into the offline PMEM section.

Sections are 128MB and memory blocks are 2GB on this system.

Quoted 3 lines

So, I don't think it's pfn_to_online_page that necessarily needs to know how to disambiguate each page, it's things that walk sections and memory blocks and expects them to be consistent over the span.

Quoted 9 lines

Well, memory hotplug code is hard wired to sparse memory model so in this particular case asking about the section is ok. But pfn walkers shouldn't really care and only rely on pfn_to_online_page. But that will do the right thing here. So we are good as long as the section is marked properly. But this would become a problem as soon as the uninitialized pages where sharing the same memory section as David pointed out. pfn_to_online_page would then return something containing garbage. So we should still think of a way to either initialize all those pages or make sure pfn_to_online_page recognizes them. The former is preferred IMHO.

The former would not have saved the crash in this case because pfn_to_online_page() is not used in v5.3:removable_show() that I can see, nor some of the other paths that might walk pfns and the wrong thing with ZONE_DEVICE.

However, I do think pfn_to_online_page() should be reliable, and I prefer to just brute force add a section flag to indicate whether the section might be ZONE_DEVICE polluted and fallback to the get_dev_pagemap() slow-path in that case.

...but it would still require hunting to find the places where pfn_to_online_page() is missing for assumptions like this crash which assumed memblock-online + section-present == section-online.

#14 lore raw
Quoted 105 lines · Dan Williams
Quoted 94 lines · Michal Hocko
Quoted 68 lines · Dan Williams
Quoted 67 lines · Michal Hocko
Quoted 52 lines · Michal Hocko
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · David Hildenbrand
Quoted 6 lines · Michal Hocko
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Quoted 7 lines

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

Okay, so we're not dealing with the "early section" mess I described, different story.

Due to [1], is_mem_section_removable() called pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not initialized.

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

Quoted 6 lines
Quoted 2 lines

If it's not self-hosted, initializing the relevant memmaps should work just fine I guess. Otherwise things get more complicated.

Oh, I forgot: pfn_to_online_page() should at least in your example make sure other pfn walkers are safe. It was just an issue of is_mem_section_removable().

Hmm, I suspect you are right. I haven't put this together, thanks! The memory section is indeed marked offline so pfn_to_online_page would indeed bail out:
crash> p (0x6060000>>15)
$3 = 3084
crash> p mem_section[3084/128][3084 & 127]
$4 = {
section_mem_map = 18446736128020054019,
usage = 0xffff902dcf956680,
page_ext = 0x0,
pad = 0
}
crash> p 18446736128020054019 & (1UL<<2)
$5 = 0

Quoted 1 line

That makes it considerably less of a problem than I thought!

Forgot to add that those who are running kernels without 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") for some reason can fix the crash by the following simple patch.

Index: linux-5.3-users_mhocko_SLE15-SP2_for-next/drivers/base/memory.c
===================================================================

drivers/base/memory.c +0 -0
@@ -152,9 +152,14 @@ static ssize_t removable_show(struct dev
                goto out;
 
Quoted 9 lines

drivers/base/memory.c static ssize_t removable_show(struct dev · line 158

Quoted 7 lines
drivers/base/memory.c +3 -2
        for (i = 0; i < sections_per_block; i++) {
-               if (!present_section_nr(mem->start_section_nr + i))
+               unsigned long nr = mem->start_section_nr + i;
+               if (!present_section_nr(nr))
                        continue;
-               pfn = section_nr_to_pfn(mem->start_section_nr + i);
+               if (!online_section_nr()) {

I assume that's onlince_section_nr(nr) in the version that compiles?

Yup.

Quoted 5 lines

This makes sense because the memory block size is larger than the section size. I suspect you have 1GB memory block size on this system, but since the System RAM and PMEM collide at a 512MB alignment in a memory block you end up walking the back end of the last 512MB of the System RAM memory block and run into the offline PMEM section.

Sections are 128MB and memory blocks are 2GB on this system.

Quoted 3 lines

So, I don't think it's pfn_to_online_page that necessarily needs to know how to disambiguate each page, it's things that walk sections and memory blocks and expects them to be consistent over the span.

Quoted 9 lines

Well, memory hotplug code is hard wired to sparse memory model so in this particular case asking about the section is ok. But pfn walkers shouldn't really care and only rely on pfn_to_online_page. But that will do the right thing here. So we are good as long as the section is marked properly. But this would become a problem as soon as the uninitialized pages where sharing the same memory section as David pointed out. pfn_to_online_page would then return something containing garbage. So we should still think of a way to either initialize all those pages or make sure pfn_to_online_page recognizes them. The former is preferred IMHO.

Quoted 4 lines

The former would not have saved the crash in this case because pfn_to_online_page() is not used in v5.3:removable_show() that I can see, nor some of the other paths that might walk pfns and the wrong thing with ZONE_DEVICE.

If the page was initialized properly, and by that I mean also have it reserved, then the old code would have properly reported is as not removable.

Quoted 4 lines

However, I do think pfn_to_online_page() should be reliable, and I prefer to just brute force add a section flag to indicate whether the section might be ZONE_DEVICE polluted and fallback to the get_dev_pagemap() slow-path in that case.

Do we have some spare room to hold that flag in a section?

Quoted 3 lines

...but it would still require hunting to find the places where pfn_to_online_page() is missing for assumptions like this crash which assumed memblock-online + section-present == section-online.

Yes, but most users should be using pfn_to_online_page already.

Signature · 2 lines
Michal Hocko
SUSE Labs
#15 lore raw
Quoted 120 lines · Michal Hocko
Quoted 105 lines · Dan Williams
Quoted 94 lines · Michal Hocko
Quoted 68 lines · Dan Williams
Quoted 67 lines · Michal Hocko
Quoted 52 lines · Michal Hocko
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · David Hildenbrand
Quoted 6 lines · Michal Hocko
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Quoted 7 lines

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

Okay, so we're not dealing with the "early section" mess I described, different story.

Due to [1], is_mem_section_removable() called pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not initialized.

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

Quoted 6 lines
Quoted 2 lines

If it's not self-hosted, initializing the relevant memmaps should work just fine I guess. Otherwise things get more complicated.

Oh, I forgot: pfn_to_online_page() should at least in your example make sure other pfn walkers are safe. It was just an issue of is_mem_section_removable().

Hmm, I suspect you are right. I haven't put this together, thanks! The memory section is indeed marked offline so pfn_to_online_page would indeed bail out:
crash> p (0x6060000>>15)
$3 = 3084
crash> p mem_section[3084/128][3084 & 127]
$4 = {
section_mem_map = 18446736128020054019,
usage = 0xffff902dcf956680,
page_ext = 0x0,
pad = 0
}
crash> p 18446736128020054019 & (1UL<<2)
$5 = 0

Quoted 1 line

That makes it considerably less of a problem than I thought!

Forgot to add that those who are running kernels without 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") for some reason can fix the crash by the following simple patch.

Index: linux-5.3-users_mhocko_SLE15-SP2_for-next/drivers/base/memory.c
===================================================================

drivers/base/memory.c +0 -0
@@ -152,9 +152,14 @@ static ssize_t removable_show(struct dev
                goto out;
 
Quoted 9 lines

drivers/base/memory.c static ssize_t removable_show(struct dev · line 158

Quoted 7 lines
drivers/base/memory.c +3 -2
        for (i = 0; i < sections_per_block; i++) {
-               if (!present_section_nr(mem->start_section_nr + i))
+               unsigned long nr = mem->start_section_nr + i;
+               if (!present_section_nr(nr))
                        continue;
-               pfn = section_nr_to_pfn(mem->start_section_nr + i);
+               if (!online_section_nr()) {

I assume that's onlince_section_nr(nr) in the version that compiles?

Yup.

Quoted 5 lines

This makes sense because the memory block size is larger than the section size. I suspect you have 1GB memory block size on this system, but since the System RAM and PMEM collide at a 512MB alignment in a memory block you end up walking the back end of the last 512MB of the System RAM memory block and run into the offline PMEM section.

Sections are 128MB and memory blocks are 2GB on this system.

Quoted 3 lines

So, I don't think it's pfn_to_online_page that necessarily needs to know how to disambiguate each page, it's things that walk sections and memory blocks and expects them to be consistent over the span.

Quoted 9 lines

Well, memory hotplug code is hard wired to sparse memory model so in this particular case asking about the section is ok. But pfn walkers shouldn't really care and only rely on pfn_to_online_page. But that will do the right thing here. So we are good as long as the section is marked properly. But this would become a problem as soon as the uninitialized pages where sharing the same memory section as David pointed out. pfn_to_online_page would then return something containing garbage. So we should still think of a way to either initialize all those pages or make sure pfn_to_online_page recognizes them. The former is preferred IMHO.

Quoted 4 lines

The former would not have saved the crash in this case because pfn_to_online_page() is not used in v5.3:removable_show() that I can see, nor some of the other paths that might walk pfns and the wrong thing with ZONE_DEVICE.

If the page was initialized properly, and by that I mean also have it reserved, then the old code would have properly reported is as not removable.

Quoted 4 lines

However, I do think pfn_to_online_page() should be reliable, and I prefer to just brute force add a section flag to indicate whether the section might be ZONE_DEVICE polluted and fallback to the get_dev_pagemap() slow-path in that case.

Quoted 8 lines

Do we have some spare room to hold that flag in a section?

Quoted 3 lines

...but it would still require hunting to find the places where pfn_to_online_page() is missing for assumptions like this crash which assumed memblock-online + section-present == section-online.

Yes, but most users should be using pfn_to_online_page already.

Quite honestly, let's not hack around this issue and just fix it properly - make pfn_to_online_page() only ever return an initialized, online (buddy) page, just as documented.

What speaks against not adding early sections in case they have a relevant hole at the end, besides wasting some MB? Relevant setups most probably just don't care. Print a warning.

Signature · 3 lines
Thanks,

David / dhildenb
#16 lore raw
Quoted 127 lines · David Hildenbrand
Quoted 126 lines · Michal Hocko
Quoted 105 lines · Dan Williams
Quoted 94 lines · Michal Hocko
Quoted 68 lines · Dan Williams
Quoted 67 lines · Michal Hocko
Quoted 52 lines · Michal Hocko
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · David Hildenbrand
Quoted 6 lines · Michal Hocko
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Quoted 7 lines

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

Okay, so we're not dealing with the "early section" mess I described, different story.

Due to [1], is_mem_section_removable() called pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not initialized.

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

Quoted 6 lines
Quoted 2 lines

If it's not self-hosted, initializing the relevant memmaps should work just fine I guess. Otherwise things get more complicated.

Oh, I forgot: pfn_to_online_page() should at least in your example make sure other pfn walkers are safe. It was just an issue of is_mem_section_removable().

Hmm, I suspect you are right. I haven't put this together, thanks! The memory section is indeed marked offline so pfn_to_online_page would indeed bail out:
crash> p (0x6060000>>15)
$3 = 3084
crash> p mem_section[3084/128][3084 & 127]
$4 = {
section_mem_map = 18446736128020054019,
usage = 0xffff902dcf956680,
page_ext = 0x0,
pad = 0
}
crash> p 18446736128020054019 & (1UL<<2)
$5 = 0

Quoted 1 line

That makes it considerably less of a problem than I thought!

Forgot to add that those who are running kernels without 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") for some reason can fix the crash by the following simple patch.

Index: linux-5.3-users_mhocko_SLE15-SP2_for-next/drivers/base/memory.c
===================================================================

drivers/base/memory.c +0 -0
@@ -152,9 +152,14 @@ static ssize_t removable_show(struct dev
                goto out;
 
Quoted 9 lines

drivers/base/memory.c static ssize_t removable_show(struct dev · line 158

Quoted 7 lines
drivers/base/memory.c +3 -2
        for (i = 0; i < sections_per_block; i++) {
-               if (!present_section_nr(mem->start_section_nr + i))
+               unsigned long nr = mem->start_section_nr + i;
+               if (!present_section_nr(nr))
                        continue;
-               pfn = section_nr_to_pfn(mem->start_section_nr + i);
+               if (!online_section_nr()) {

I assume that's onlince_section_nr(nr) in the version that compiles?

Yup.

Quoted 5 lines

This makes sense because the memory block size is larger than the section size. I suspect you have 1GB memory block size on this system, but since the System RAM and PMEM collide at a 512MB alignment in a memory block you end up walking the back end of the last 512MB of the System RAM memory block and run into the offline PMEM section.

Sections are 128MB and memory blocks are 2GB on this system.

Quoted 3 lines

So, I don't think it's pfn_to_online_page that necessarily needs to know how to disambiguate each page, it's things that walk sections and memory blocks and expects them to be consistent over the span.

Quoted 9 lines

Well, memory hotplug code is hard wired to sparse memory model so in this particular case asking about the section is ok. But pfn walkers shouldn't really care and only rely on pfn_to_online_page. But that will do the right thing here. So we are good as long as the section is marked properly. But this would become a problem as soon as the uninitialized pages where sharing the same memory section as David pointed out. pfn_to_online_page would then return something containing garbage. So we should still think of a way to either initialize all those pages or make sure pfn_to_online_page recognizes them. The former is preferred IMHO.

Quoted 4 lines

The former would not have saved the crash in this case because pfn_to_online_page() is not used in v5.3:removable_show() that I can see, nor some of the other paths that might walk pfns and the wrong thing with ZONE_DEVICE.

If the page was initialized properly, and by that I mean also have it reserved, then the old code would have properly reported is as not removable.

Quoted 4 lines

However, I do think pfn_to_online_page() should be reliable, and I prefer to just brute force add a section flag to indicate whether the section might be ZONE_DEVICE polluted and fallback to the get_dev_pagemap() slow-path in that case.

Do we have some spare room to hold that flag in a section?

Quoted 3 lines

...but it would still require hunting to find the places where pfn_to_online_page() is missing for assumptions like this crash which assumed memblock-online + section-present == section-online.

Quoted 6 lines
Quoted 2 lines

Yes, but most users should be using pfn_to_online_page already.

Quite honestly, let's not hack around this issue and just fix it properly - make pfn_to_online_page() only ever return an initialized, online (buddy) page, just as documented.

Just to make sure we are on the same page. You are agreeing with Dan that pfn_to_online_page should check for zone device pages? Ideally in a slow path.

Quoted 3 lines

What speaks against not adding early sections in case they have a relevant hole at the end, besides wasting some MB? Relevant setups most probably just don't care. Print a warning.

If we can avoid shared section reasonably then I do not object. Sacrificing few Megs of memory should be fine in most cases. It is not like reasonable systems would have many hybrid sections.

Signature · 2 lines
Michal Hocko
SUSE Labs
#17 lore raw
Quoted 131 lines · Michal Hocko
Quoted 130 lines · David Hildenbrand
Quoted 128 lines · Michal Hocko
Quoted 105 lines · Dan Williams
Quoted 94 lines · Michal Hocko
Quoted 68 lines · Dan Williams
Quoted 67 lines · Michal Hocko
Quoted 52 lines · Michal Hocko
Quoted 29 lines · David Hildenbrand
Quoted 28 lines · David Hildenbrand
Quoted 6 lines · Michal Hocko
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Quoted 7 lines

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

Okay, so we're not dealing with the "early section" mess I described, different story.

Due to [1], is_mem_section_removable() called pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not initialized.

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

Quoted 6 lines
Quoted 2 lines

If it's not self-hosted, initializing the relevant memmaps should work just fine I guess. Otherwise things get more complicated.

Oh, I forgot: pfn_to_online_page() should at least in your example make sure other pfn walkers are safe. It was just an issue of is_mem_section_removable().

Hmm, I suspect you are right. I haven't put this together, thanks! The memory section is indeed marked offline so pfn_to_online_page would indeed bail out:
crash> p (0x6060000>>15)
$3 = 3084
crash> p mem_section[3084/128][3084 & 127]
$4 = {
section_mem_map = 18446736128020054019,
usage = 0xffff902dcf956680,
page_ext = 0x0,
pad = 0
}
crash> p 18446736128020054019 & (1UL<<2)
$5 = 0

Quoted 1 line

That makes it considerably less of a problem than I thought!

Forgot to add that those who are running kernels without 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") for some reason can fix the crash by the following simple patch.

Index: linux-5.3-users_mhocko_SLE15-SP2_for-next/drivers/base/memory.c
===================================================================

drivers/base/memory.c +0 -0
@@ -152,9 +152,14 @@ static ssize_t removable_show(struct dev
                goto out;
 
Quoted 9 lines

drivers/base/memory.c static ssize_t removable_show(struct dev · line 158

Quoted 7 lines
drivers/base/memory.c +3 -2
        for (i = 0; i < sections_per_block; i++) {
-               if (!present_section_nr(mem->start_section_nr + i))
+               unsigned long nr = mem->start_section_nr + i;
+               if (!present_section_nr(nr))
                        continue;
-               pfn = section_nr_to_pfn(mem->start_section_nr + i);
+               if (!online_section_nr()) {

I assume that's onlince_section_nr(nr) in the version that compiles?

Yup.

Quoted 5 lines

This makes sense because the memory block size is larger than the section size. I suspect you have 1GB memory block size on this system, but since the System RAM and PMEM collide at a 512MB alignment in a memory block you end up walking the back end of the last 512MB of the System RAM memory block and run into the offline PMEM section.

Sections are 128MB and memory blocks are 2GB on this system.

Quoted 3 lines

So, I don't think it's pfn_to_online_page that necessarily needs to know how to disambiguate each page, it's things that walk sections and memory blocks and expects them to be consistent over the span.

Quoted 9 lines

Well, memory hotplug code is hard wired to sparse memory model so in this particular case asking about the section is ok. But pfn walkers shouldn't really care and only rely on pfn_to_online_page. But that will do the right thing here. So we are good as long as the section is marked properly. But this would become a problem as soon as the uninitialized pages where sharing the same memory section as David pointed out. pfn_to_online_page would then return something containing garbage. So we should still think of a way to either initialize all those pages or make sure pfn_to_online_page recognizes them. The former is preferred IMHO.

Quoted 4 lines

The former would not have saved the crash in this case because pfn_to_online_page() is not used in v5.3:removable_show() that I can see, nor some of the other paths that might walk pfns and the wrong thing with ZONE_DEVICE.

If the page was initialized properly, and by that I mean also have it reserved, then the old code would have properly reported is as not removable.

Quoted 4 lines

However, I do think pfn_to_online_page() should be reliable, and I prefer to just brute force add a section flag to indicate whether the section might be ZONE_DEVICE polluted and fallback to the get_dev_pagemap() slow-path in that case.

Do we have some spare room to hold that flag in a section?

Quoted 3 lines

...but it would still require hunting to find the places where pfn_to_online_page() is missing for assumptions like this crash which assumed memblock-online + section-present == section-online.

Yes, but most users should be using pfn_to_online_page already.

Quoted 7 lines
Quoted 3 lines

Quite honestly, let's not hack around this issue and just fix it properly - make pfn_to_online_page() only ever return an initialized, online (buddy) page, just as documented.

Just to make sure we are on the same page. You are agreeing with Dan that pfn_to_online_page should check for zone device pages? Ideally in a slow path.

The most important part for me is that pfn_to_online_page() behaves as documented. How that is implemented is a secondary concern. The easier, the better (e.g., just avoid the corner-case (!) issue we discovered completely).

Signature · 3 lines
Thanks,

David / dhildenb
#18 lore raw
In reply to David Hildenbrand
Quoted 15 lines · David Hildenbrand
Quoted 6 lines · Michal Hocko
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Quoted 7 lines

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

Quoted 6 lines

Okay, so we're not dealing with the "early section" mess I described, different story.

Due to [1], is_mem_section_removable() called pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not initialized.

Exactly!

Quoted 2 lines

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

Not sure what exactly you are asking for but crash says crash> kmem -p 6060000
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
fffff8c600181800 6060000 0 0 0 fffffc0000000

Quoted 2 lines

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

I am not really familiar with the pmem so I would need more assistance here. I've tried this (shot into the dark):
crash> struct page.pgmap fffff8c600181800

      pgmap = 0xfffff8c600181808
crash> struct -x dev_pagemap 0xfffff8c600181808
struct dev_pagemap {
  altmap = {
    base_pfn = 0xfffff8c600181808, 
    end_pfn = 0xfffff8c600181808, 
    reserve = 0x0, 
    free = 0x0, 
    align = 0x0, 
    alloc = 0xffffffff
  }, 
  res = {
    start = 0x0, 
    end = 0xfffffc0000000, 
    name = 0xfffff8c600181848 "H\030\030", 
    flags = 0xfffff8c600181848, 
    desc = 0x0, 
    parent = 0x0, 
    sibling = 0x0, 
    child = 0xffffffff
  }, 
  ref = 0x0, 
  internal_ref = {
    count = {
      counter = 0xfffffc0000000
    }, 
    percpu_count_ptr = 0xfffff8c600181888, 
    release = 0xfffff8c600181888, 
    confirm_switch = 0x0, 
    force_atomic = 0x0, 
    allow_reinit = 0x0, 
    rcu = {
      next = 0x0, 
      func = 0xffffffff
    }
  }, 
  done = {
    done = 0x0, 
    wait = {
      lock = {
        {
          rlock = {
            raw_lock = {
              {
                val = {
                  counter = 0xc0000000
                }, 
                {
                  locked = 0x0, 
                  pending = 0x0
                }, 
                {
                  locked_pending = 0x0, 
                  tail = 0xc000
                }
              }
            }
          }
        }
      }, 
      head = {
        next = 0xfffff8c6001818c8, 
        prev = 0xfffff8c6001818c8
      }
    }

},
dev = 0x0,
type = 0,
flags = 0x0,
ops = 0x0
}

Not sure whether this is of any use.

Quoted 2 lines

If it's not self-hosted, initializing the relevant memmaps should work just fine I guess. Otherwise things get more complicated.

Signature · 2 lines
Michal Hocko
SUSE Labs
#19 lore raw
Quoted 7 lines
Quoted 2 lines

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

Not sure what exactly you are asking for but crash says crash> kmem -p 6060000
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
fffff8c600181800 6060000 0 0 0 fffffc0000000

^ this looks like it was somewhat initialized. All flags zero, nid/zone set to -1 (wild guess) and thus the crash? weird

Quoted 8 lines
Quoted 3 lines

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

I am not really familiar with the pmem so I would need more assistance here. I've tried this (shot into the dark):
crash> struct page.pgmap fffff8c600181800

      pgmap = 0xfffff8c600181808

That's weird. If the memmap is at fffff8c600181800, why should the pgmap be at an offset of 8 bytes from there. The "pgmap" field is actually at an offset of 8 bytes within the memmap ...

Assuming the memmap is not actually ZONE_DEVICE, fffff8c600181800 really only contains garbage, including the pgmap pointer :(

Quoted 5 lines

crash> struct -x dev_pagemap 0xfffff8c600181808
struct dev_pagemap {

  altmap = {
    base_pfn = 0xfffff8c600181808, 
    end_pfn = 0xfffff8c600181808,

^ because this is very weird

Quoted 1 line
    reserve = 0x0, 

^ and this indicates nothing was actually reserved.

Signature · 3 lines
Thanks,

David / dhildenb
#20 lore raw
Quoted 10 lines · David Hildenbrand
Quoted 7 lines
Quoted 2 lines

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

Not sure what exactly you are asking for but crash says crash> kmem -p 6060000
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
fffff8c600181800 6060000 0 0 0 fffffc0000000

^ this looks like it was somewhat initialized. All flags zero, nid/zone set to -1 (wild guess) and thus the crash? weird

Yes that made me scratch my head as well.

Quoted 11 lines
Quoted 7 lines
Quoted 2 lines

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

I am not really familiar with the pmem so I would need more assistance here. I've tried this (shot into the dark):
crash> struct page.pgmap fffff8c600181800

      pgmap = 0xfffff8c600181808

That's weird. If the memmap is at fffff8c600181800, why should the pgmap be at an offset of 8 bytes from there. The "pgmap" field is actually at an offset of 8 bytes within the memmap ...

I haven't noticed this is an offset into struct page. This is indeed weird because the structure is much larger than struct page. But maybe it reuses storage of several struct pages in a row. Or maybe it is not pgmap but something else that shares the offset in the structure.

Signature · 2 lines
Michal Hocko
SUSE Labs
#21 lore raw
Quoted 9 lines · Michal Hocko
Quoted 8 lines · David Hildenbrand
Quoted 7 lines
Quoted 2 lines

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

Not sure what exactly you are asking for but crash says crash> kmem -p 6060000
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
fffff8c600181800 6060000 0 0 0 fffffc0000000

Quoted 4 lines
Quoted 2 lines

^ this looks like it was somewhat initialized. All flags zero, nid/zone set to -1 (wild guess) and thus the crash? weird

Yes that made me scratch my head as well.

My best guess would be that this is indeed part of the altmap reserve with a memmap on the altmap. Maybe these values are just leftovers on the PMEM device (e.g., from older kernels)?

In the kernel, I cannot find anything that would initialize node/zone to -1 only.

@Dan: Will get_dev_pagemap() return a valid dev_pagemap for pfns falling into the altmap reserved region? I think we should also initialize the memmap of the altmap reserved region somewhow.

Signature · 3 lines
Thanks,

David / dhildenb
#22 lore raw
In reply to Michal Hocko
Quoted 34 lines · Michal Hocko
Quoted 15 lines · David Hildenbrand
Quoted 6 lines · Michal Hocko
Quoted 1 line · David Hildenbrand

On 04.01.21 16:10, Michal Hocko wrote:

[...]

Quoted 2 lines

Do the physical addresses you see fall into the same section as boot memory? Or what's around these addresses?

Quoted 7 lines

Yes I am getting a garbage for the first struct page belonging to the pmem section [1]

[    0.020161] ACPI: SRAT: Node 0 PXM 0 [mem 0x100000000-0x603fffffff]
[    0.020163] ACPI: SRAT: Node 4 PXM 4 [mem 0x6060000000-0x11d5fffffff] non-volatile

The pfn without the initialized struct page is 0x6060000. This is a first pfn in a section.

Quoted 6 lines

Okay, so we're not dealing with the "early section" mess I described, different story.

Due to [1], is_mem_section_removable() called pfn_to_page(PHYS_PFN(0x6060000)). page_zone(page) made it crash, as not initialized.

Exactly!

Quoted 2 lines

Let's assume this is indeed a reserved pfn in the altmap. What's the actual address of the memmap?

Not sure what exactly you are asking for but crash says crash> kmem -p 6060000
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
fffff8c600181800 6060000 0 0 0 fffffc0000000

Quoted 7 lines
Quoted 2 lines

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

I am not really familiar with the pmem so I would need more assistance here. I've tried this (shot into the dark):
crash> struct page.pgmap fffff8c600181800

      pgmap = 0xfffff8c600181808

Does /proc/iomem show an active namespace in the range? You should be able to skip ahead to the first pfn in that namespace to find the first dev_pagemap. I would have expected pfn_to_online_page() to have saved you here. This address range is section aligned.

#23 lore raw
Quoted 1 line · Dan Williams

On Mon, Jan 4, 2021 at 7:59 AM Michal Hocko <[email protected]> wrote:

[...]

Quoted 8 lines
Quoted 8 lines

Not sure what exactly you are asking for but crash says crash> kmem -p 6060000
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
fffff8c600181800 6060000 0 0 0 fffffc0000000

Quoted 2 lines

I do wonder what hosts pfn_to_page(PHYS_PFN(0x6060000)) - is it actually part of the actual altmap (i.e. > 0x6060000) or maybe even self-hosted?

Quoted 6 lines
Quoted 4 lines

I am not really familiar with the pmem so I would need more assistance here. I've tried this (shot into the dark):
crash> struct page.pgmap fffff8c600181800

      pgmap = 0xfffff8c600181808

Does /proc/iomem show an active namespace in the range?

Any tips how I dig that out from the crash dump?

Quoted 4 lines

You should be
able to skip ahead to the first pfn in that namespace to find the first dev_pagemap. I would have expected pfn_to_online_page() to have saved you here. This address range is section aligned.

Well, the affected code in this case was

	end_pfn = min(start_pfn + nr_pages,
                        zone_end_pfn(page_zone(pfn_to_page(start_pfn))));

where start_pfn was the first pfn of a memory section. This code was completely unaware of zone device or dev_pagemap like most others pfn walkers. It just wanted to get bounds for the zone but it stumbled over uninitialized zone/node.

Signature · 2 lines
Michal Hocko
SUSE Labs
#24 lore raw
In reply to David Hildenbrand
Quoted 15 lines · David Hildenbrand
Quoted 9 lines · Michal Hocko

Hi,
back in March [1] you have recommended 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") to be backported to stable trees and that has led to a more general discussion about the current state of pfn walkers wrt. uninitialized pmem struct pages. We haven't concluded any specific solution for that except for a general sentiment that pfn_to_online_page should be able to catch those. I might have missed any follow ups on that but I do not think we have landed on any actual solution in the end. Have I just missed any followups?

Thanks for raising this issue. It's still on my list of "broken and non-trivial to fix" things.

Quoted 6 lines

So, as far as I recall, we still have the following two issues remaining:

1. pfn_to_online_page() false positives

The semantics of pfn_to_online_page() were broken with sub-section hot-add in corner cases.

The motivation for that backport was for pre-subsection kernels. When onlining pmem that collides with the same section as System-RAM we may have a situation like:

|--               SECTION                   -- |
|-- System RAM      --        PMEM          -- |
|-- pfn_valid()     --     PMEM metadata    -- |

So problem 0. is just System RAM + PMEM collisions independent of sub-section support.

Quoted 7 lines

If we have ZONE_DEVICE hot-added memory that overlaps in a section with boot memory, this memory section will contain parts ZONE_DEVICE memory and parts !ZONE_DEVICE memory. This can happen in sub-section granularity (2MB IIRC). pfn_to_online_page() will succeed on ZONE_DEVICE memory parts as the whole section is marked as online. Bad.

Quoted 5 lines

One instance where this is still an issue is mm/memory-failure.c:memory_failure() and mm/memory-failure.c:soft_offline_page(). I thought for a while about "fixing" these, but to me it felt like fixing pfn_to_online_page() is actually the right approach.

This is complicated by MEMORY_DEVICE_PRIVATE that I believe wants to say "yes" to pfn_to_online_page().

Quoted 10 lines

But worse, before ZONE_DEVICE hot-add
1. The whole section memmap does already exist (early sections always have a full memmap for the whole section)
2. The whole section memmap is initialized (although eventually with dummy node/zone 0/0 for memory holes until that part is fixed) and might be accessed by pfn walkers.

So when hotadding ZONE_DEVICE we are modifying already existing and visible memmaps. Bad.

Where does the rewrite of a dummy page entry matter in practice? It would certainly be exceedingly Bad if in-use 'struct page' instances we're rewritten. You're only alleging the former, correct?

Quoted 10 lines

2. Deferred init of ZONE_DEVICE ranges

memmap_init_zone_device() runs after the ZONE_DEVICE zone was resized and outside the memhp lock. I did not follow if the use of get_dev_pagemap() actually makes sure that memmap_init_zone_device() in pagemap_range() actually completed. I don't think it does.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

Is anybody working on that?

I believe Dan mentioned somewhere that he wants to see a real instance of this producing a BUG before actually moving forward with a fix. I might be wrong.

I think I'm missing an argument for the user-visible effects of the "Bad." statements above. I think soft_offline_page() is a candidate for a local fix because mm/memory-failure.c already has a significant amount of page-type specific knowledge. So teaching it "yes" for MEMORY_DEVICE_PRIVATE-ZONE_DEVICE and "no" for other ZONE_DEVICE seems ok to me.

Quoted 12 lines

We might tackle 1. by:

a) [worked-around] doing get_dev_pagemap() before pfn_to_online_page() - just plain ugly.

b) [worked-around] using a sub-section online-map and extending pfn_to_online_page(). IMHO ugly to fix this corner case.

c) [worked-around] extending pfn_to_online_page() by zone checks. IMHO racy.

Quoted 7 lines

d) fixed by not allowing ZONE_DEVICE and !ZONE_DEVICE within a single section. In the worst case, don't add partially present sections that have big holes in the beginning/end. Like, if there is a 128MB section with 126MB of memory followed by a 2MB hole, don't add that memory to Linux with CONFIG_ZONE_DEVICE. Might turn some memory unusable, but well, it would be the price to pay for simplicity. Being able to hotadd PMEM is more important than using each and every last MB of memory.

The collisions that are out there in the wild are 64MB System RAM followed by 64MB of PMEM. If you're suggesting reducing System RAM that collides with PMEM that's a consideration. It can't go the other way because there are deployed configurations that have persistent data there. Reducing System RAM runs into the problem of how early does the kernel know that it's bordering ZONE_DEVICE. It's not just PMEM, it's also EFI_MEMORY_SP (Linux Soft Reserved) memory.

Quoted 4 lines

e) decrease the section size and drop sub-section hotadd support. As sub-section hotadd is 2MB and MAX_ORDER - 1 corresponds 4MB, this is mostly impossible. Worse on aarch64 with 64k base pages - 1024MB sections and MAX_ORDER - 1 corresponds 512MB. I think this is not feasible.

f) Teach the places where pfn_to_online_page() gives the wrong answer how to handle ZONE_DEVICE. Like:

a78986aae9b2 KVM: MMU: Do not treat ZONE_DEVICE pages as being reserved

#25 lore raw
Quoted 1 line · Dan Williams

On Mon, Jan 4, 2021 at 2:45 AM David Hildenbrand <[email protected]> wrote:

[...]

Quoted 10 lines
Quoted 3 lines

I believe Dan mentioned somewhere that he wants to see a real instance of this producing a BUG before actually moving forward with a fix. I might be wrong.

I think I'm missing an argument for the user-visible effects of the "Bad." statements above. I think soft_offline_page() is a candidate for a local fix because mm/memory-failure.c already has a significant amount of page-type specific knowledge. So teaching it "yes" for MEMORY_DEVICE_PRIVATE-ZONE_DEVICE and "no" for other ZONE_DEVICE seems ok to me.

I believe we do not want to teach _every_ pfn walker about zone device pages. This would be quite error prone. Especially when a missig check could lead to a silently broken data or BUG_ON with debugging enabled (which is not the case for many production users). Or are we talking about different bugs here?

Signature · 2 lines
Michal Hocko
SUSE Labs
#26 lore raw
Quoted 14 lines · Michal Hocko
Quoted 1 line · Dan Williams

On Mon, Jan 4, 2021 at 2:45 AM David Hildenbrand <[email protected]> wrote:

[...]

Quoted 10 lines
Quoted 3 lines

I believe Dan mentioned somewhere that he wants to see a real instance of this producing a BUG before actually moving forward with a fix. I might be wrong.

I think I'm missing an argument for the user-visible effects of the "Bad." statements above. I think soft_offline_page() is a candidate for a local fix because mm/memory-failure.c already has a significant amount of page-type specific knowledge. So teaching it "yes" for MEMORY_DEVICE_PRIVATE-ZONE_DEVICE and "no" for other ZONE_DEVICE seems ok to me.

Quoted 5 lines

I believe we do not want to teach _every_ pfn walker about zone device pages. This would be quite error prone. Especially when a missig check could lead to a silently broken data or BUG_ON with debugging enabled (which is not the case for many production users). Or are we talking about different bugs here?

I'd like us to stick to the documentation, e.g., include/linux/mmzone.h

"
pfn_valid() is meant to be able to tell if a given PFN has valid memmap associated with it or not. This means that a struct page exists for this pfn. The caller cannot assume the page is fully initialized in general. Hotplugable pages might not have been onlined yet. pfn_to_online_page() will ensure the struct page is fully online and initialized. Special pages (e.g. ZONE_DEVICE) are never onlined and should be treated accordingly.
"

Signature · 3 lines
Thanks,

David / dhildenb
#27 lore raw
In reply to Dan Williams
Quoted 27 lines · Dan Williams
Quoted 15 lines · David Hildenbrand
Quoted 9 lines · Michal Hocko

Hi,
back in March [1] you have recommended 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") to be backported to stable trees and that has led to a more general discussion about the current state of pfn walkers wrt. uninitialized pmem struct pages. We haven't concluded any specific solution for that except for a general sentiment that pfn_to_online_page should be able to catch those. I might have missed any follow ups on that but I do not think we have landed on any actual solution in the end. Have I just missed any followups?

Thanks for raising this issue. It's still on my list of "broken and non-trivial to fix" things.

Quoted 6 lines

So, as far as I recall, we still have the following two issues remaining:

1. pfn_to_online_page() false positives

The semantics of pfn_to_online_page() were broken with sub-section hot-add in corner cases.

The motivation for that backport was for pre-subsection kernels. When onlining pmem that collides with the same section as System-RAM we may have a situation like:

Quoted 6 lines
|--               SECTION                   -- |
|-- System RAM      --        PMEM          -- |
|-- pfn_valid()     --     PMEM metadata    -- |

So problem 0. is just System RAM + PMEM collisions independent of sub-section support.

IIRC, you were not able to hot-add the PMEM device before sub-section support (which was bad and fixed by sub-section hot-add). How was this a problem before?

Quoted 25 lines
Quoted 8 lines

If we have ZONE_DEVICE hot-added memory that overlaps in a section with boot memory, this memory section will contain parts ZONE_DEVICE memory and parts !ZONE_DEVICE memory. This can happen in sub-section granularity (2MB IIRC). pfn_to_online_page() will succeed on ZONE_DEVICE memory parts as the whole section is marked as online. Bad.

Quoted 5 lines

One instance where this is still an issue is mm/memory-failure.c:memory_failure() and mm/memory-failure.c:soft_offline_page(). I thought for a while about "fixing" these, but to me it felt like fixing pfn_to_online_page() is actually the right approach.

This is complicated by MEMORY_DEVICE_PRIVATE that I believe wants to say "yes" to pfn_to_online_page().

Quoted 8 lines

But worse, before ZONE_DEVICE hot-add
1. The whole section memmap does already exist (early sections always have a full memmap for the whole section)
2. The whole section memmap is initialized (although eventually with dummy node/zone 0/0 for memory holes until that part is fixed) and might be accessed by pfn walkers.

Quoted 6 lines
Quoted 2 lines

So when hotadding ZONE_DEVICE we are modifying already existing and visible memmaps. Bad.

Where does the rewrite of a dummy page entry matter in practice? It would certainly be exceedingly Bad if in-use 'struct page' instances we're rewritten. You're only alleging the former, correct?

Yes, just another piece of the puzzle.

Quoted 15 lines
Quoted 8 lines

2. Deferred init of ZONE_DEVICE ranges

memmap_init_zone_device() runs after the ZONE_DEVICE zone was resized and outside the memhp lock. I did not follow if the use of get_dev_pagemap() actually makes sure that memmap_init_zone_device() in pagemap_range() actually completed. I don't think it does.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

Is anybody working on that?

I believe Dan mentioned somewhere that he wants to see a real instance of this producing a BUG before actually moving forward with a fix. I might be wrong.

Quoted 6 lines

I think I'm missing an argument for the user-visible effects of the "Bad." statements above. I think soft_offline_page() is a candidate for a local fix because mm/memory-failure.c already has a significant amount of page-type specific knowledge. So teaching it "yes" for MEMORY_DEVICE_PRIVATE-ZONE_DEVICE and "no" for other ZONE_DEVICE seems ok to me.

I am not completely against working around the issue we have in the code but nobody stumbled over them. IMHO it's just error prone code and handling we have here that will bite us in the long term. E.g., any pfn walker/code that sticks to the current documentation of pfn_to_online_page().

I am not sure it's the right thing to do for MEMORY_DEVICE_PRIVATE-ZONE_DEVICE, that requires more discussion.

Quoted 1 line
Quoted 1 line

We might tackle 1. by:

[...]

Quoted 8 lines
Quoted 7 lines

d) fixed by not allowing ZONE_DEVICE and !ZONE_DEVICE within a single section. In the worst case, don't add partially present sections that have big holes in the beginning/end. Like, if there is a 128MB section with 126MB of memory followed by a 2MB hole, don't add that memory to Linux with CONFIG_ZONE_DEVICE. Might turn some memory unusable, but well, it would be the price to pay for simplicity. Being able to hotadd PMEM is more important than using each and every last MB of memory.

Quoted 7 lines

The collisions that are out there in the wild are 64MB System RAM followed by 64MB of PMEM. If you're suggesting reducing System RAM that collides with PMEM that's a consideration. It can't go the other way because there are deployed configurations that have persistent data there. Reducing System RAM runs into the problem of how early does the kernel know that it's bordering ZONE_DEVICE. It's not just PMEM, it's also EFI_MEMORY_SP (Linux Soft Reserved) memory.

Yeah, obviously the first one. Being able to add+use PMEM is more important than using each and every last MB of main memory.

I wonder if we can just stop adding any system RAM like

[ Memory Section ]
[ RAM ] [ Hole ]

When there could be the possibility that the hole might actually be PMEM. (e.g., with CONFIG_ZONE_DEVICE and it being the last section in a sequence of sections, not just a tiny hole)

Signature · 3 lines
Thanks,

David / dhildenb
#28 lore raw
Quoted 33 lines · David Hildenbrand
Quoted 31 lines · Dan Williams
Quoted 15 lines · David Hildenbrand
Quoted 9 lines · Michal Hocko

Hi,
back in March [1] you have recommended 53cdc1cb29e8 ("drivers/base/memory.c: indicate all memory blocks as removable") to be backported to stable trees and that has led to a more general discussion about the current state of pfn walkers wrt. uninitialized pmem struct pages. We haven't concluded any specific solution for that except for a general sentiment that pfn_to_online_page should be able to catch those. I might have missed any follow ups on that but I do not think we have landed on any actual solution in the end. Have I just missed any followups?

Thanks for raising this issue. It's still on my list of "broken and non-trivial to fix" things.

Quoted 6 lines

So, as far as I recall, we still have the following two issues remaining:

1. pfn_to_online_page() false positives

The semantics of pfn_to_online_page() were broken with sub-section hot-add in corner cases.

The motivation for that backport was for pre-subsection kernels. When onlining pmem that collides with the same section as System-RAM we may have a situation like:

|--               SECTION                   -- |
|-- System RAM      --        PMEM          -- |
|-- pfn_valid()     --     PMEM metadata    -- |
Quoted 6 lines
Quoted 2 lines

So problem 0. is just System RAM + PMEM collisions independent of sub-section support.

IIRC, you were not able to hot-add the PMEM device before sub-section support (which was bad and fixed by sub-section hot-add). How was this a problem before?

The details are in the subsection changelog but the tl;dr is that I clumsily hacked around it by throwing away PMEM, but that breaks when PMEM shifts around in physical address space from one boot to the next. The moving around is on 64MB granularity, so subsection hotplug lets those shifts be covered and enables smaller granularity / alignment support for smaller things like p2pdma mappings.

Quoted 88 lines
Quoted 26 lines
Quoted 9 lines

If we have ZONE_DEVICE hot-added memory that overlaps in a section with boot memory, this memory section will contain parts ZONE_DEVICE memory and parts !ZONE_DEVICE memory. This can happen in sub-section granularity (2MB IIRC). pfn_to_online_page() will succeed on ZONE_DEVICE memory parts as the whole section is marked as online. Bad.

Quoted 5 lines

One instance where this is still an issue is mm/memory-failure.c:memory_failure() and mm/memory-failure.c:soft_offline_page(). I thought for a while about "fixing" these, but to me it felt like fixing pfn_to_online_page() is actually the right approach.

This is complicated by MEMORY_DEVICE_PRIVATE that I believe wants to say "yes" to pfn_to_online_page().

Quoted 8 lines

But worse, before ZONE_DEVICE hot-add
1. The whole section memmap does already exist (early sections always have a full memmap for the whole section)
2. The whole section memmap is initialized (although eventually with dummy node/zone 0/0 for memory holes until that part is fixed) and might be accessed by pfn walkers.

Quoted 6 lines
Quoted 2 lines

So when hotadding ZONE_DEVICE we are modifying already existing and visible memmaps. Bad.

Where does the rewrite of a dummy page entry matter in practice? It would certainly be exceedingly Bad if in-use 'struct page' instances we're rewritten. You're only alleging the former, correct?

Yes, just another piece of the puzzle.

Quoted 15 lines
Quoted 8 lines

2. Deferred init of ZONE_DEVICE ranges

memmap_init_zone_device() runs after the ZONE_DEVICE zone was resized and outside the memhp lock. I did not follow if the use of get_dev_pagemap() actually makes sure that memmap_init_zone_device() in pagemap_range() actually completed. I don't think it does.

Quoted 1 line
Quoted 6 lines
Quoted 2 lines

Is anybody working on that?

I believe Dan mentioned somewhere that he wants to see a real instance of this producing a BUG before actually moving forward with a fix. I might be wrong.

Quoted 6 lines

I think I'm missing an argument for the user-visible effects of the "Bad." statements above. I think soft_offline_page() is a candidate for a local fix because mm/memory-failure.c already has a significant amount of page-type specific knowledge. So teaching it "yes" for MEMORY_DEVICE_PRIVATE-ZONE_DEVICE and "no" for other ZONE_DEVICE seems ok to me.

I am not completely against working around the issue we have in the code but nobody stumbled over them. IMHO it's just error prone code and handling we have here that will bite us in the long term. E.g., any pfn walker/code that sticks to the current documentation of pfn_to_online_page().

I am not sure it's the right thing to do for MEMORY_DEVICE_PRIVATE-ZONE_DEVICE, that requires more discussion.

Quoted 1 line
Quoted 1 line

We might tackle 1. by:

[...]

Quoted 8 lines
Quoted 7 lines

d) fixed by not allowing ZONE_DEVICE and !ZONE_DEVICE within a single section. In the worst case, don't add partially present sections that have big holes in the beginning/end. Like, if there is a 128MB section with 126MB of memory followed by a 2MB hole, don't add that memory to Linux with CONFIG_ZONE_DEVICE. Might turn some memory unusable, but well, it would be the price to pay for simplicity. Being able to hotadd PMEM is more important than using each and every last MB of memory.

Quoted 7 lines

The collisions that are out there in the wild are 64MB System RAM followed by 64MB of PMEM. If you're suggesting reducing System RAM that collides with PMEM that's a consideration. It can't go the other way because there are deployed configurations that have persistent data there. Reducing System RAM runs into the problem of how early does the kernel know that it's bordering ZONE_DEVICE. It's not just PMEM, it's also EFI_MEMORY_SP (Linux Soft Reserved) memory.

Yeah, obviously the first one. Being able to add+use PMEM is more important than using each and every last MB of main memory.

Quoted 8 lines

I wonder if we can just stop adding any system RAM like

[ Memory Section ]
[ RAM ] [ Hole ]

When there could be the possibility that the hole might actually be PMEM. (e.g., with CONFIG_ZONE_DEVICE and it being the last section in a sequence of sections, not just a tiny hole)

I like the simplicity of it... I worry that the capacity loss regression is easy to notice by looking at the output of free(1) from one kernel to the next and someone screams.

#29 lore raw
Quoted 8 lines
Quoted 8 lines

Yeah, obviously the first one. Being able to add+use PMEM is more important than using each and every last MB of main memory.

I wonder if we can just stop adding any system RAM like

[ Memory Section ]
[ RAM ] [ Hole ]

Quoted 7 lines
Quoted 3 lines

When there could be the possibility that the hole might actually be PMEM. (e.g., with CONFIG_ZONE_DEVICE and it being the last section in a sequence of sections, not just a tiny hole)

I like the simplicity of it... I worry that the capacity loss regression is easy to notice by looking at the output of free(1) from one kernel to the next and someone screams.

Well, you can always make it configurable and then simply fail to add PMEM later if impossible (trying to sub-section hot-add into early section). It's in the hands of the sysadmin then ("max out system ram" vs. "support any PMEM device that could eventually be there at runtime"). Distros would go for the second.

I agree that it's not optimal, but sometimes simplicity has to win.

Signature · 3 lines
Thanks,

David / dhildenb
#30 lore raw
Quoted 17 lines · David Hildenbrand
Quoted 9 lines
Quoted 9 lines

Yeah, obviously the first one. Being able to add+use PMEM is more important than using each and every last MB of main memory.

I wonder if we can just stop adding any system RAM like

[ Memory Section ]
[ RAM ] [ Hole ]

Quoted 7 lines
Quoted 3 lines

When there could be the possibility that the hole might actually be PMEM. (e.g., with CONFIG_ZONE_DEVICE and it being the last section in a sequence of sections, not just a tiny hole)

I like the simplicity of it... I worry that the capacity loss regression is easy to notice by looking at the output of free(1) from one kernel to the next and someone screams.

Quoted 7 lines

Well, you can always make it configurable and then simply fail to add PMEM later if impossible (trying to sub-section hot-add into early section). It's in the hands of the sysadmin then ("max out system ram" vs. "support any PMEM device that could eventually be there at runtime"). Distros would go for the second.

I agree that it's not optimal, but sometimes simplicity has to win.

Here's where we left it last time, open to pfn_to_online_page hacks...

https://lore.kernel.org/linux-mm/CAPcyv4ivq=EPUePXiX2ErcVyF7+dV9Yv215Oue7X_Y2X_Jfw8Q@mail.gmail.com

I don't think a slow-path flag in the mem-section is too onerous, but I'll withhold judgement until I have the patch I'm thinking of in-hand. Let me give it a shot, you can always nack the final result.

#31 lore raw
Quoted 24 lines · Dan Williams
Quoted 23 lines · David Hildenbrand
Quoted 9 lines
Quoted 9 lines

Yeah, obviously the first one. Being able to add+use PMEM is more important than using each and every last MB of main memory.

I wonder if we can just stop adding any system RAM like

[ Memory Section ]
[ RAM ] [ Hole ]

Quoted 7 lines
Quoted 3 lines

When there could be the possibility that the hole might actually be PMEM. (e.g., with CONFIG_ZONE_DEVICE and it being the last section in a sequence of sections, not just a tiny hole)

I like the simplicity of it... I worry that the capacity loss regression is easy to notice by looking at the output of free(1) from one kernel to the next and someone screams.

Well, you can always make it configurable and then simply fail to add PMEM later if impossible (trying to sub-section hot-add into early section). It's in the hands of the sysadmin then ("max out system ram" vs. "support any PMEM device that could eventually be there at runtime"). Distros would go for the second.

Quoted 6 lines
Quoted 1 line

I agree that it's not optimal, but sometimes simplicity has to win.

Here's where we left it last time, open to pfn_to_online_page hacks...

https://lore.kernel.org/linux-mm/CAPcyv4ivq=EPUePXiX2ErcVyF7+dV9Yv215Oue7X_Y2X_Jfw8Q@mail.gmail.com

Yeah, I recall. That's why I favor simple approaches right now - less brain power to waste ;)

Quoted 3 lines

I don't think a slow-path flag in the mem-section is too onerous, but I'll withhold judgement until I have the patch I'm thinking of in-hand. Let me give it a shot, you can always nack the final result.

Sure!

Signature · 3 lines
Thanks,

David / dhildenb