collision between ZONE_MOVABLE and memblock allocations

Hello,

I've been trying to use the 'movablecore=' kernel command line option to create a ZONE_MOVABLE memory zone on my x86_64 systems, and have noticed that offlining the resulting ZONE_MOVABLE area consistently fails in my setups because that zone contains unmovable pages. My testing has been in a x86_64 QEMU VM with a single NUMA node and 4G, 8G or 16G of memory, all of which fail 100% of the time.

Digging into it a bit, these unmovable pages are Reserved pages which were allocated in early boot as part of the memblock allocator. Many of these allocations are for data structures for the SPARSEMEM memory model, including 'struct mem_section' objects. These memblock allocations can be tracked by setting the 'memblock=debug' kernel command line parameter, and are marked as reserved in:

memmap_init_reserved_pages()
reserve_bootmem_region()

With the command line params 'movablecore=256M memblock=debug' and a v6.5.0-rc2 kernel I get the following on my 4G system:

  # lsmem --split ZONES --output-all
  RANGE                                  SIZE  STATE REMOVABLE BLOCK NODE   ZONES
  0x0000000000000000-0x0000000007ffffff  128M online       yes     0    0    None
  0x0000000008000000-0x00000000bfffffff  2.9G online       yes  1-23    0   DMA32
  0x0000000100000000-0x000000012fffffff  768M online       yes 32-37    0  Normal
  0x0000000130000000-0x000000013fffffff  256M online       yes 38-39    0 Movable

Memory block size: 128M
Total online memory: 4G
Total offline memory: 0B

And when I try to offline memory block 39, I get:

  # echo 0 > /sys/devices/system/memory/memory39/online
  bash: echo: write error: Device or resource busy

with dmesg saying:

  [   57.439849] page:0000000076a3e320 refcount:1 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x13ff00
  [   57.444073] flags: 0x1fffffc0001000(reserved|node=0|zone=3|lastcpupid=0x1fffff)
  [   57.447301] page_type: 0xffffffff()
  [   57.448754] raw: 001fffffc0001000 ffffdd6384ffc008 ffffdd6384ffc008 0000000000000000
  [   57.450383] raw: 0000000000000000 0000000000000000 00000001ffffffff 0000000000000000
  [   57.452011] page dumped because: unmovable page

Looking back at the memblock allocations, I can see that the physical address for pfn:0x13ff00 was used in a memblock allocation:

  [    0.395180] memblock_reserve: [0x000000013ff00000-0x000000013ffbffff] memblock_alloc_range_nid+0xe0/0x150

The full dmesg output can be found here: https://pastebin.com/cNztqa4u

The 'movablecore=' command line parameter is handled in 'find_zone_movable_pfns_for_nodes()', which decides where ZONE_MOVABLE should start and end. Currently ZONE_MOVABLE is always located at the end of a NUMA node.

The issue is that the memblock allocator and the processing of the movablecore= command line parameter don't know about one another, and in my x86_64 testing they both always use memory at the end of the NUMA node and have collisions.

From several comments in the code I believe that this is a known issue:

https://elixir.bootlin.com/linux/v6.5-rc2/source/mm/page_isolation.c#L59
/*
* Both, bootmem allocations and memory holes are marked
* PG_reserved and are unmovable. We can even have unmovable
* allocations inside ZONE_MOVABLE, for example when
* specifying "movablecore".
*/

https://elixir.bootlin.com/linux/v6.5-rc2/source/include/linux/mmzone.h#L765
* 2. memblock allocations: kernelcore/movablecore setups might create
* situations where ZONE_MOVABLE contains unmovable allocations
* after boot. Memory offlining and allocations fail early.

We check for these unmovable pages by scanning for 'PageReserved()' in the area we are trying to offline, which happens in has_unmovable_pages().

Interestingly, the boot timing works out like this:

1. Allocate memblock areas to set up the SPARSEMEM model.

  [    0.369990] Call Trace:
  [    0.370404]  <TASK>
  [    0.370759]  ? dump_stack_lvl+0x43/0x60
  [    0.371410]  ? sparse_init_nid+0x2dc/0x560
  [    0.372116]  ? sparse_init+0x346/0x450
  [    0.372755]  ? paging_init+0xa/0x20
  [    0.373349]  ? setup_arch+0xa6a/0xfc0
  [    0.373970]  ? slab_is_available+0x5/0x20
  [    0.374651]  ? start_kernel+0x5e/0x770
  [    0.375290]  ? x86_64_start_reservations+0x14/0x30
  [    0.376109]  ? x86_64_start_kernel+0x71/0x80
  [    0.376835]  ? secondary_startup_64_no_verify+0x167/0x16b
  [    0.377755]  </TASK>

2. Process movablecore= kernel command line parameter and set up memory zones

  [    0.489382] Call Trace:
  [    0.489818]  <TASK>
  [    0.490187]  ? dump_stack_lvl+0x43/0x60
  [    0.490873]  ? free_area_init+0x115/0xc80
  [    0.491588]  ? __printk_cpu_sync_put+0x5/0x30
  [    0.492354]  ? dump_stack_lvl+0x48/0x60
  [    0.493002]  ? sparse_init_nid+0x2dc/0x560
  [    0.493697]  ? zone_sizes_init+0x60/0x80
  [    0.494361]  ? setup_arch+0xa6a/0xfc0
  [    0.494981]  ? slab_is_available+0x5/0x20
  [    0.495674]  ? start_kernel+0x5e/0x770
  [    0.496312]  ? x86_64_start_reservations+0x14/0x30
  [    0.497123]  ? x86_64_start_kernel+0x71/0x80
  [    0.497847]  ? secondary_startup_64_no_verify+0x167/0x16b
  [    0.498768]  </TASK>

3. Mark memblock areas as Reserved.

  [    0.761136] Call Trace:
  [    0.761534]  <TASK>
  [    0.761876]  dump_stack_lvl+0x43/0x60
  [    0.762474]  reserve_bootmem_region+0x1e/0x170
  [    0.763201]  memblock_free_all+0xe3/0x250
  [    0.763862]  ? swiotlb_init_io_tlb_mem.constprop.0+0x11a/0x130
  [    0.764812]  ? swiotlb_init_remap+0x195/0x2c0
  [    0.765519]  mem_init+0x19/0x1b0
  [    0.766047]  mm_core_init+0x9c/0x3d0
  [    0.766630]  start_kernel+0x264/0x770
  [    0.767229]  x86_64_start_reservations+0x14/0x30
  [    0.767987]  x86_64_start_kernel+0x71/0x80
  [    0.768666]  secondary_startup_64_no_verify+0x167/0x16b
  [    0.769534]  </TASK>

So, during ZONE_MOVABLE setup we currently can't do the same has_unmovable_pages() scan looking for PageReserved() to check for overlap because the pages have not yet been marked as Reserved.

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

Here are the options I currently see for resolution:

1. Change the way ZONE_MOVABLE memory is allocated so that it is allocated from the beginning of the NUMA node instead of the end. This should fix my use case, but again is prone to breakage in other configurations (# of NUMA nodes, other architectures) where ZONE_MOVABLE and memblock allocations might overlap. I think that this should be relatively straightforward and low risk, though.

2. Make the code which processes the movablecore= command line option aware of the memblock allocations, and have it choose a region for ZONE_MOVABLE which does not have these allocations. This might be done by checking for PageReserved() as we do with offlining memory, though that will take some boot time reordering, or we'll have to figure out the overlap in another way. This may also result in us having two ZONE_NORMAL zones for a given NUMA node, with a ZONE_MOVABLE section in between them. I'm not sure if this is allowed? If we can get it working, this seems like the most correct solution to me, but also the most difficult and risky because it involves significant changes in the code for memory setup at early boot.

Am I missing anything are there other solutions we should consider, or do you have an opinion on which solution we should pursue?

Thanks,
- Ross

3. Switch memblock to use bottom up allocations. Historically memblock allocated memory from the top to avoid corrupting the kernel image and to avoid exhausting precious ZONE_DMA. I believe we can use bottom-up allocations with lower limit of memblock allocations set to 16M.

With the hack below no memblock allocations will end up in ZONE_MOVABLE:

Quoted 6 lines · Mike Rapoport

3. Switch memblock to use bottom up allocations. Historically memblock allocated memory from the top to avoid corrupting the kernel image and to avoid exhausting precious ZONE_DMA. I believe we can use bottom-up allocations with lower limit of memblock allocations set to 16M.

With the hack below no memblock allocations will end up in ZONE_MOVABLE:

Yep, I've confirmed that for my use cases at least this does the trick, thank you! I had thought about moving the memblock allocations, but had no idea it was (basically) already supported and thought it'd be much riskier than just adjusting where ZONE_MOVABLE lived.

Is there a reason for this to not be a real option for users, maybe per a kernel config knob or something? I'm happy to explore other options in this thread, but this is doing the trick so far.

Quoted 8 lines · Ross Zwisler

Quoted 6 lines · Mike Rapoport

3. Switch memblock to use bottom up allocations. Historically memblock allocated memory from the top to avoid corrupting the kernel image and to avoid exhausting precious ZONE_DMA. I believe we can use bottom-up allocations with lower limit of memblock allocations set to 16M.

With the hack below no memblock allocations will end up in ZONE_MOVABLE:

Quoted 8 lines

Yep, I've confirmed that for my use cases at least this does the trick, thank you! I had thought about moving the memblock allocations, but had no idea it was (basically) already supported and thought it'd be much riskier than just adjusting where ZONE_MOVABLE lived.

Is there a reason for this to not be a real option for users, maybe per a kernel config knob or something? I'm happy to explore other options in this thread, but this is doing the trick so far.

I think we can make x86 always use bottom up.

To do this properly we'd need to set lower limit for memblock allocations to MAX_DMA32_PFN and allow fallback below it so that early allocations won't eat memory from ZONE_DMA32.

Aside from x86 boot being fragile in general I don't see why this wouldn't work.

Quoted 24 lines · Mike Rapoport

Quoted 8 lines · Ross Zwisler

Quoted 6 lines · Mike Rapoport

3. Switch memblock to use bottom up allocations. Historically memblock allocated memory from the top to avoid corrupting the kernel image and to avoid exhausting precious ZONE_DMA. I believe we can use bottom-up allocations with lower limit of memblock allocations set to 16M.

With the hack below no memblock allocations will end up in ZONE_MOVABLE:

Quoted 8 lines

Yep, I've confirmed that for my use cases at least this does the trick, thank you! I had thought about moving the memblock allocations, but had no idea it was (basically) already supported and thought it'd be much riskier than just adjusting where ZONE_MOVABLE lived.

Is there a reason for this to not be a real option for users, maybe per a kernel config knob or something? I'm happy to explore other options in this thread, but this is doing the trick so far.

I think we can make x86 always use bottom up.

To do this properly we'd need to set lower limit for memblock allocations to MAX_DMA32_PFN and allow fallback below it so that early allocations won't eat memory from ZONE_DMA32.

Quoted 2 lines

Aside from x86 boot being fragile in general I don't see why this wouldn't work.

This would add a very subtle depency of a functionality on the specific boot allocator behavior and that is bad for long term maintenance.

Signature · 2 lines

Michal Hocko
SUSE Labs

Quoted 25 lines · Michal Hocko

Quoted 24 lines · Mike Rapoport

Quoted 8 lines · Ross Zwisler

Quoted 6 lines · Mike Rapoport

3. Switch memblock to use bottom up allocations. Historically memblock allocated memory from the top to avoid corrupting the kernel image and to avoid exhausting precious ZONE_DMA. I believe we can use bottom-up allocations with lower limit of memblock allocations set to 16M.

With the hack below no memblock allocations will end up in ZONE_MOVABLE:

Quoted 8 lines

Yep, I've confirmed that for my use cases at least this does the trick, thank you! I had thought about moving the memblock allocations, but had no idea it was (basically) already supported and thought it'd be much riskier than just adjusting where ZONE_MOVABLE lived.

Is there a reason for this to not be a real option for users, maybe per a kernel config knob or something? I'm happy to explore other options in this thread, but this is doing the trick so far.

I think we can make x86 always use bottom up.

To do this properly we'd need to set lower limit for memblock allocations to MAX_DMA32_PFN and allow fallback below it so that early allocations won't eat memory from ZONE_DMA32.

Quoted 5 lines

Quoted 2 lines

Aside from x86 boot being fragile in general I don't see why this wouldn't work.

This would add a very subtle depency of a functionality on the specific boot allocator behavior and that is bad for long term maintenance.

What do you mean by "specific boot allocator behavior"?

Quoted 29 lines · Mike Rapoport

Quoted 28 lines · Michal Hocko

Quoted 26 lines · Mike Rapoport

Quoted 8 lines · Ross Zwisler

Quoted 6 lines · Mike Rapoport

3. Switch memblock to use bottom up allocations. Historically memblock allocated memory from the top to avoid corrupting the kernel image and to avoid exhausting precious ZONE_DMA. I believe we can use bottom-up allocations with lower limit of memblock allocations set to 16M.

With the hack below no memblock allocations will end up in ZONE_MOVABLE:

Quoted 8 lines

Yep, I've confirmed that for my use cases at least this does the trick, thank you! I had thought about moving the memblock allocations, but had no idea it was (basically) already supported and thought it'd be much riskier than just adjusting where ZONE_MOVABLE lived.

Is there a reason for this to not be a real option for users, maybe per a kernel config knob or something? I'm happy to explore other options in this thread, but this is doing the trick so far.

I think we can make x86 always use bottom up.

To do this properly we'd need to set lower limit for memblock allocations to MAX_DMA32_PFN and allow fallback below it so that early allocations won't eat memory from ZONE_DMA32.

Aside from x86 boot being fragile in general I don't see why this wouldn't work.

Quoted 4 lines

Quoted 2 lines

This would add a very subtle depency of a functionality on the specific boot allocator behavior and that is bad for long term maintenance.

What do you mean by "specific boot allocator behavior"?

I mean that the expectation that the boot allocator starts from low addresses and functionality depending on that is too fragile. This has already caused some problems in the past IIRC.

Signature · 2 lines

Michal Hocko
SUSE Labs

Quoted 30 lines · Michal Hocko

Quoted 29 lines · Mike Rapoport

Quoted 28 lines · Michal Hocko

Quoted 26 lines · Mike Rapoport

Quoted 8 lines · Ross Zwisler

Quoted 6 lines · Mike Rapoport

3. Switch memblock to use bottom up allocations. Historically memblock allocated memory from the top to avoid corrupting the kernel image and to avoid exhausting precious ZONE_DMA. I believe we can use bottom-up allocations with lower limit of memblock allocations set to 16M.

With the hack below no memblock allocations will end up in ZONE_MOVABLE:

Quoted 8 lines

Yep, I've confirmed that for my use cases at least this does the trick, thank you! I had thought about moving the memblock allocations, but had no idea it was (basically) already supported and thought it'd be much riskier than just adjusting where ZONE_MOVABLE lived.

Is there a reason for this to not be a real option for users, maybe per a kernel config knob or something? I'm happy to explore other options in this thread, but this is doing the trick so far.

I think we can make x86 always use bottom up.

To do this properly we'd need to set lower limit for memblock allocations to MAX_DMA32_PFN and allow fallback below it so that early allocations won't eat memory from ZONE_DMA32.

Aside from x86 boot being fragile in general I don't see why this wouldn't work.

Quoted 8 lines

Quoted 4 lines

Quoted 2 lines

This would add a very subtle depency of a functionality on the specific boot allocator behavior and that is bad for long term maintenance.

What do you mean by "specific boot allocator behavior"?

I mean that the expectation that the boot allocator starts from low addresses and functionality depending on that is too fragile. This has already caused some problems in the past IIRC.

Well, any change in x86 boot sequence may cause all sorts of problems :)

We do some of the boot time allocations from low addresses when movable_node is enabled and that is entirely implicit and buried deep inside the code.

What I'm suggesting is to switch the allocations to bottom-up once and for all with explicitly set lower limit and a defined semantics for a fallback.

This might cause some bumps in the beginning, but I don't expect it to be a maintenance problem in the long run.

And it will free higher memory from early allocations for all usecases, not just this one.

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

Here are the options I currently see for resolution:

1. Change the way ZONE_MOVABLE memory is allocated so that it is allocated from the beginning of the NUMA node instead of the end. This should fix my use case, but again is prone to breakage in other configurations (# of NUMA nodes, other architectures) where ZONE_MOVABLE and memblock allocations might overlap. I think that this should be relatively straightforward and low risk, though.

2. Make the code which processes the movablecore= command line option aware of the memblock allocations, and have it choose a region for ZONE_MOVABLE which does not have these allocations. This might be done by checking for PageReserved() as we do with offlining memory, though that will take some boot time reordering, or we'll have to figure out the overlap in another way. This may also result in us having two ZONE_NORMAL zones for a given NUMA node, with a ZONE_MOVABLE section in between them. I'm not sure if this is allowed?

If
we can get it working, this seems like the most correct solution to me, but also the most difficult and risky because it involves significant changes in the code for memory setup at early boot.

Am I missing anything are there other solutions we should consider, or do you have an opinion on which solution we should pursue?

On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
[...]

Quoted 4 lines

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

movablecore is kinda hack and I would be more inclined to get rid of it rather than build more into it. Could you be more specific about your use case?

Quoted 8 lines

Here are the options I currently see for resolution:

1. Change the way ZONE_MOVABLE memory is allocated so that it is allocated from the beginning of the NUMA node instead of the end. This should fix my use case, but again is prone to breakage in other configurations (# of NUMA nodes, other architectures) where ZONE_MOVABLE and memblock allocations might overlap. I think that this should be relatively straightforward and low risk, though.

Quoted 7 lines

2. Make the code which processes the movablecore= command line option aware of the memblock allocations, and have it choose a region for ZONE_MOVABLE which does not have these allocations. This might be done by checking for PageReserved() as we do with offlining memory, though that will take some boot time reordering, or we'll have to figure out the overlap in another way. This may also result in us having two ZONE_NORMAL zones for a given NUMA node, with a ZONE_MOVABLE section in between them. I'm not sure if this is allowed?

Yes, this is no problem. Zones are allowed to be sparse.

Quoted 7 lines

If
we can get it working, this seems like the most correct solution to me, but also the most difficult and risky because it involves significant changes in the code for memory setup at early boot.

Am I missing anything are there other solutions we should consider, or do you have an opinion on which solution we should pursue?

If this really needs to be addressed than 2) is certainly a more robust approach.

Signature · 2 lines

Michal Hocko
SUSE Labs

Quoted 27 lines · Michal Hocko

On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
[...]

Quoted 4 lines

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

movablecore is kinda hack and I would be more inclined to get rid of it rather than build more into it. Could you be more specific about your use case?

Quoted 8 lines

Here are the options I currently see for resolution:

1. Change the way ZONE_MOVABLE memory is allocated so that it is allocated from the beginning of the NUMA node instead of the end. This should fix my use case, but again is prone to breakage in other configurations (# of NUMA nodes, other architectures) where ZONE_MOVABLE and memblock allocations might overlap. I think that this should be relatively straightforward and low risk, though.

Quoted 7 lines

2. Make the code which processes the movablecore= command line option aware of the memblock allocations, and have it choose a region for ZONE_MOVABLE which does not have these allocations. This might be done by checking for PageReserved() as we do with offlining memory, though that will take some boot time reordering, or we'll have to figure out the overlap in another way. This may also result in us having two ZONE_NORMAL zones for a given NUMA node, with a ZONE_MOVABLE section in between them. I'm not sure if this is allowed?

Quoted 1 line

Yes, this is no problem. Zones are allowed to be sparse.

The current initialization order is roughly

* very early initialization with some memblock allocations
* determine zone locations and sizes
* initialize memory map
- memblock_alloc(lots of memory)
* lots of unrelated initializations that may allocate memory
* release free pages from memblock to the buddy allocator

With 2) we can make sure the memory map and early allocations won't be in the ZONE_MOVABLE, but we'll still may have reserved pages there.

Quoted 39 lines · Mike Rapoport

Quoted 27 lines · Michal Hocko

On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
[...]

Quoted 4 lines

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

movablecore is kinda hack and I would be more inclined to get rid of it rather than build more into it. Could you be more specific about your use case?

Quoted 8 lines

Here are the options I currently see for resolution:

1. Change the way ZONE_MOVABLE memory is allocated so that it is allocated from the beginning of the NUMA node instead of the end. This should fix my use case, but again is prone to breakage in other configurations (# of NUMA nodes, other architectures) where ZONE_MOVABLE and memblock allocations might overlap. I think that this should be relatively straightforward and low risk, though.

Quoted 7 lines

2. Make the code which processes the movablecore= command line option aware of the memblock allocations, and have it choose a region for ZONE_MOVABLE which does not have these allocations. This might be done by checking for PageReserved() as we do with offlining memory, though that will take some boot time reordering, or we'll have to figure out the overlap in another way. This may also result in us having two ZONE_NORMAL zones for a given NUMA node, with a ZONE_MOVABLE section in between them. I'm not sure if this is allowed?

Quoted 1 line

Yes, this is no problem. Zones are allowed to be sparse.

The current initialization order is roughly

* very early initialization with some memblock allocations
* determine zone locations and sizes
* initialize memory map
- memblock_alloc(lots of memory)
* lots of unrelated initializations that may allocate memory
* release free pages from memblock to the buddy allocator

Quoted 2 lines

With 2) we can make sure the memory map and early allocations won't be in the ZONE_MOVABLE, but we'll still may have reserved pages there.

Yes this will always be fragile. If the spefic placement of the movable memory is not important and the only thing that matters is the size and numa locality then an easier to maintain solution would be to simply offline enough memory blocks very early in the userspace bring up and online it back as movable. If offlining fails just try another memblock. This doesn't require any kernel code change.

Quoted 43 lines · Michal Hocko

Quoted 39 lines · Mike Rapoport

Quoted 27 lines · Michal Hocko

On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
[...]

Quoted 4 lines

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

movablecore is kinda hack and I would be more inclined to get rid of it rather than build more into it. Could you be more specific about your use case?

Quoted 8 lines

Here are the options I currently see for resolution:

1. Change the way ZONE_MOVABLE memory is allocated so that it is allocated from the beginning of the NUMA node instead of the end. This should fix my use case, but again is prone to breakage in other configurations (# of NUMA nodes, other architectures) where ZONE_MOVABLE and memblock allocations might overlap. I think that this should be relatively straightforward and low risk, though.

Quoted 7 lines

2. Make the code which processes the movablecore= command line option aware of the memblock allocations, and have it choose a region for ZONE_MOVABLE which does not have these allocations. This might be done by checking for PageReserved() as we do with offlining memory, though that will take some boot time reordering, or we'll have to figure out the overlap in another way. This may also result in us having two ZONE_NORMAL zones for a given NUMA node, with a ZONE_MOVABLE section in between them. I'm not sure if this is allowed?

Quoted 1 line

Yes, this is no problem. Zones are allowed to be sparse.

The current initialization order is roughly

* very early initialization with some memblock allocations
* determine zone locations and sizes
* initialize memory map
- memblock_alloc(lots of memory)
* lots of unrelated initializations that may allocate memory
* release free pages from memblock to the buddy allocator

Quoted 2 lines

With 2) we can make sure the memory map and early allocations won't be in the ZONE_MOVABLE, but we'll still may have reserved pages there.

Quoted 6 lines

Yes this will always be fragile. If the spefic placement of the movable memory is not important and the only thing that matters is the size and numa locality then an easier to maintain solution would be to simply offline enough memory blocks very early in the userspace bring up and online it back as movable. If offlining fails just try another memblock. This doesn't require any kernel code change.

As an alternative, we might use the "memmap=nn[KMG]!ss[KMG]" [1] parameter to mark some memory as protected.

That memory can then be configured as devdax device and online to ZONE_MOVABLE (dev/dax).

[1] https://docs.pmem.io/persistent-memory/getting-started-guide/creating-development-environments/linux-environments/linux-memmap

Quoted 7 lines

As an alternative, we might use the "memmap=nn[KMG]!ss[KMG]" [1] parameter to mark some memory as protected.

That memory can then be configured as devdax device and online to ZONE_MOVABLE (dev/dax).

[1] https://docs.pmem.io/persistent-memory/getting-started-guide/creating-development-environments/linux-environments/linux-memmap

I've previously been reconfiguring devdax memory like this:

   ndctl create-namespace --reconfig=namespace0.0 -m devdax -f
   daxctl reconfigure-device --mode=system-ram dax0.0

Is this how you've been doing it, or is there something else I should consider?

I just sent mail to Michal outlining my use case, hopefully it makes sense.

I had thought about using 'memmap=' in the first kernel and the worry was that I'd have to support many different machines with different memory configurations, and have to hard-code memory offsets and lengths for the various memmap= kernel command line parameters.

If I can make ZONE_MOVABLE
work that's preferable because the kernel will choose the correct usermem-only region for me, and then I can just use that region for the crash kernel and 3rd kernel boots.

On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
[...]

Quoted 4 lines

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

movablecore is kinda hack and I would be more inclined to get rid of it rather than build more into it. Could you be more specific about your use case?

If the spefic placement of the movable memory is not important and the only thing that matters is the size and numa locality then an easier to maintain solution would be to simply offline enough memory blocks very early in the userspace bring up and online it back as movable. If offlining fails just try another memblock. This doesn't require any kernel code change.

Quoted 10 lines · Michal Hocko

On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
[...]

Quoted 4 lines

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

movablecore is kinda hack and I would be more inclined to get rid of it rather than build more into it. Could you be more specific about your use case?

The problem that I'm trying to solve is that I'd like to be able to get kernel core dumps off machines (chromebooks) so that we can debug crashes. Because the memory used by the crash kernel ("crashkernel=" kernel command line option) is consumed the entire time the machine is booted, there is a strong motivation to keep the crash kernel as small and as simple as possible. To this end I'm trying to get away without SSD drivers, not having to worry about encryption on the SSDs, etc.

So, the rough plan right now is:

1) During boot set aside some memory that won't contain kernel allocations. I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways.

We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE region (or whatever non-kernel region) will be set aside as PMEM in the crash kernel. This is done with the memmap=nn[KMG]!ss[KMG] kernel command line parameter passed to the crash kernel.

So, in my sample 4G VM system, I see:

  # lsmem --split ZONES --output-all
  RANGE                                  SIZE  STATE REMOVABLE BLOCK NODE   ZONES
  0x0000000000000000-0x0000000007ffffff  128M online       yes     0    0    None
  0x0000000008000000-0x00000000bfffffff  2.9G online       yes  1-23    0   DMA32
  0x0000000100000000-0x000000012fffffff  768M online       yes 32-37    0  Normal
  0x0000000130000000-0x000000013fffffff  256M online       yes 38-39    0 Movable

Memory block size: 128M
Total online memory: 4G
Total offline memory: 0B

so I'll pass "memmap=256M!0x130000000" to the crash kernel.

2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set aside only contains user data, which we don't want to store anyway. We make a filesystem in there, and create a kernel crash dump using 'makedumpfile':

mkfs.ext4 /dev/pmem0
mount /dev/pmem0 /mnt
makedumpfile -c -d 31 /proc/vmcore /mnt/kdump

We then set up the next full kernel boot to also have this same PMEM region, using the same memmap kernel parameter. We reboot back into a full kernel.

3) The next full kernel will be a normal boot with a full networking stack, SSD drivers, disk encryption, etc. We mount up our PMEM filesystem, pull out the kdump and either store it somewhere persistent or upload it somewhere. We can then unmount the PMEM and reconfigure it back to system ram so that the live system isn't missing memory.

  ndctl create-namespace --reconfig=namespace0.0 -m devdax -f
  daxctl reconfigure-device --mode=system-ram dax0.0

This is the flow I'm trying to support, and have mostly working in a VM, except up until now makedumpfile would crash because all the memblock structures it needed were in the PMEM area that I had just wiped out by making a new filesystem. :)

Do you see any blockers that would make this infeasible?

For the non-kernel memory, is the ZONE_MOVABLE path that I'm currently pursuing the best option, or would we be better off with your suggestion elsewhere in this thread:

Quoted 5 lines

If the spefic placement of the movable memory is not important and the only thing that matters is the size and numa locality then an easier to maintain solution would be to simply offline enough memory blocks very early in the userspace bring up and online it back as movable. If offlining fails just try another memblock. This doesn't require any kernel code change.

If this 2nd way is preferred, can you point me to how I can offline the memory blocks & then get them back later in boot?

Quoted 10 lines · Michal Hocko

On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
[...]

Quoted 4 lines

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

movablecore is kinda hack and I would be more inclined to get rid of it rather than build more into it. Could you be more specific about your use case?

The problem that I'm trying to solve is that I'd like to be able to get kernel core dumps off machines (chromebooks) so that we can debug crashes. Because the memory used by the crash kernel ("crashkernel=" kernel command line option) is consumed the entire time the machine is booted, there is a strong motivation to keep the crash kernel as small and as simple as possible. To this end I'm trying to get away without SSD drivers, not having to worry about encryption on the SSDs, etc.

So, the rough plan right now is:

1) During boot set aside some memory that won't contain kernel allocations. I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways.

We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE region (or whatever non-kernel region) will be set aside as PMEM in the crash kernel. This is done with the memmap=nn[KMG]!ss[KMG] kernel command line parameter passed to the crash kernel.

So, in my sample 4G VM system, I see:

  # lsmem --split ZONES --output-all
  RANGE                                  SIZE  STATE REMOVABLE BLOCK NODE   ZONES
  0x0000000000000000-0x0000000007ffffff  128M online       yes     0    0    None
  0x0000000008000000-0x00000000bfffffff  2.9G online       yes  1-23    0   DMA32
  0x0000000100000000-0x000000012fffffff  768M online       yes 32-37    0  Normal
  0x0000000130000000-0x000000013fffffff  256M online       yes 38-39    0 Movable

Memory block size: 128M
Total online memory: 4G
Total offline memory: 0B

so I'll pass "memmap=256M!0x130000000" to the crash kernel.

2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set aside only contains user data, which we don't want to store anyway. We make a filesystem in there, and create a kernel crash dump using 'makedumpfile':

mkfs.ext4 /dev/pmem0
mount /dev/pmem0 /mnt
makedumpfile -c -d 31 /proc/vmcore /mnt/kdump

We then set up the next full kernel boot to also have this same PMEM region, using the same memmap kernel parameter. We reboot back into a full kernel.

Quoted 53 lines · Ross Zwisler

Quoted 10 lines · Michal Hocko

On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
[...]

Quoted 4 lines

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

movablecore is kinda hack and I would be more inclined to get rid of it rather than build more into it. Could you be more specific about your use case?

The problem that I'm trying to solve is that I'd like to be able to get kernel core dumps off machines (chromebooks) so that we can debug crashes. Because the memory used by the crash kernel ("crashkernel=" kernel command line option) is consumed the entire time the machine is booted, there is a strong motivation to keep the crash kernel as small and as simple as possible. To this end I'm trying to get away without SSD drivers, not having to worry about encryption on the SSDs, etc.

So, the rough plan right now is:

1) During boot set aside some memory that won't contain kernel allocations. I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways.

We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE region (or whatever non-kernel region) will be set aside as PMEM in the crash kernel. This is done with the memmap=nn[KMG]!ss[KMG] kernel command line parameter passed to the crash kernel.

So, in my sample 4G VM system, I see:

  # lsmem --split ZONES --output-all
  RANGE                                  SIZE  STATE REMOVABLE BLOCK NODE   ZONES
  0x0000000000000000-0x0000000007ffffff  128M online       yes     0    0    None
  0x0000000008000000-0x00000000bfffffff  2.9G online       yes  1-23    0   DMA32
  0x0000000100000000-0x000000012fffffff  768M online       yes 32-37    0  Normal
  0x0000000130000000-0x000000013fffffff  256M online       yes 38-39    0 Movable

Memory block size: 128M
Total online memory: 4G
Total offline memory: 0B

so I'll pass "memmap=256M!0x130000000" to the crash kernel.

2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set aside only contains user data, which we don't want to store anyway. We make a filesystem in there, and create a kernel crash dump using 'makedumpfile':

mkfs.ext4 /dev/pmem0
mount /dev/pmem0 /mnt
makedumpfile -c -d 31 /proc/vmcore /mnt/kdump

Quoted 2 lines

We then set up the next full kernel boot to also have this same PMEM region, using the same memmap kernel parameter. We reboot back into a full kernel.

Btw. How do you ensure that the address range doesn't get reinitialized by POST? Do you rely on kexec boot here?

Quoted 10 lines · Michal Hocko

On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
[...]

Quoted 4 lines

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

movablecore is kinda hack and I would be more inclined to get rid of it rather than build more into it. Could you be more specific about your use case?

The problem that I'm trying to solve is that I'd like to be able to get kernel core dumps off machines (chromebooks) so that we can debug crashes. Because the memory used by the crash kernel ("crashkernel=" kernel command line option) is consumed the entire time the machine is booted, there is a strong motivation to keep the crash kernel as small and as simple as possible. To this end I'm trying to get away without SSD drivers, not having to worry about encryption on the SSDs, etc.

So, the rough plan right now is:

Quoted 1 line

1) During boot set aside some memory that won't contain kernel

I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways.

We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE region (or whatever non-kernel region) will be set aside as PMEM in the crash kernel. This is done with the memmap=nn[KMG]!ss[KMG] kernel command line parameter passed to the crash kernel.

So, in my sample 4G VM system, I see:

   # lsmem --split ZONES --output-all
   RANGE                                  SIZE  STATE REMOVABLE BLOCK NODE   ZONES
   0x0000000000000000-0x0000000007ffffff  128M online       yes     0    0    None
   0x0000000008000000-0x00000000bfffffff  2.9G online       yes  1-23    0   DMA32
   0x0000000100000000-0x000000012fffffff  768M online       yes 32-37    0  Normal
   0x0000000130000000-0x000000013fffffff  256M online       yes 38-39    0 Movable

Memory block size: 128M
Total online memory: 4G
Total offline memory: 0B

so I'll pass "memmap=256M!0x130000000" to the crash kernel.

2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set aside only contains user data, which we don't want to store anyway.

We make a
filesystem in there, and create a kernel crash dump using 'makedumpfile':

mkfs.ext4 /dev/pmem0
mount /dev/pmem0 /mnt
makedumpfile -c -d 31 /proc/vmcore /mnt/kdump

We then set up the next full kernel boot to also have this same PMEM region, using the same memmap kernel parameter. We reboot back into a full kernel.

3) The next full kernel will be a normal boot with a full networking stack, SSD drivers, disk encryption, etc. We mount up our PMEM filesystem, pull out the kdump and either store it somewhere persistent or upload it somewhere. We can then unmount the PMEM and reconfigure it back to system ram so that the live system isn't missing memory.

   ndctl create-namespace --reconfig=namespace0.0 -m devdax -f
   daxctl reconfigure-device --mode=system-ram dax0.0

This is the flow I'm trying to support, and have mostly working in a VM, except up until now makedumpfile would crash because all the memblock structures it needed were in the PMEM area that I had just wiped out by making a new filesystem. :)

Quoted 12 lines · Ross Zwisler

Quoted 10 lines · Michal Hocko

On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
[...]

Quoted 4 lines

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

movablecore is kinda hack and I would be more inclined to get rid of it rather than build more into it. Could you be more specific about your use case?

Quoted 7 lines

The problem that I'm trying to solve is that I'd like to be able to get kernel core dumps off machines (chromebooks) so that we can debug crashes. Because the memory used by the crash kernel ("crashkernel=" kernel command line option) is consumed the entire time the machine is booted, there is a strong motivation to keep the crash kernel as small and as simple as possible. To this end I'm trying to get away without SSD drivers, not having to worry about encryption on the SSDs, etc.

Okay, so you intend to keep the crashkernel area as small as possible.

Quoted 3 lines

So, the rough plan right now is:

Quoted 1 line

1) During boot set aside some memory that won't contain kernel

allocations.

Quoted 16 lines

I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways.

We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE region (or whatever non-kernel region) will be set aside as PMEM in the crash kernel. This is done with the memmap=nn[KMG]!ss[KMG] kernel command line parameter passed to the crash kernel.

So, in my sample 4G VM system, I see:

    # lsmem --split ZONES --output-all
    RANGE                                  SIZE  STATE REMOVABLE BLOCK NODE   ZONES
    0x0000000000000000-0x0000000007ffffff  128M online       yes     0    0    None
    0x0000000008000000-0x00000000bfffffff  2.9G online       yes  1-23    0   DMA32
    0x0000000100000000-0x000000012fffffff  768M online       yes 32-37    0  Normal
    0x0000000130000000-0x000000013fffffff  256M online       yes 38-39    0 Movable

Quoted 8 lines

Memory block size: 128M
Total online memory: 4G
Total offline memory: 0B

so I'll pass "memmap=256M!0x130000000" to the crash kernel.

2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set aside only contains user data, which we don't want to store anyway.

I raised that in different context already, but such assumptions are not 100% future proof IMHO. For example, we might at one point be able to make user page tables movable and place them on there.

But yes, most kernel data structures (which you care about) will probably never be movable and never end up on these regions.

Quoted 16 lines

We make a
filesystem in there, and create a kernel crash dump using 'makedumpfile':

mkfs.ext4 /dev/pmem0
mount /dev/pmem0 /mnt
makedumpfile -c -d 31 /proc/vmcore /mnt/kdump

We then set up the next full kernel boot to also have this same PMEM region, using the same memmap kernel parameter. We reboot back into a full kernel.

3) The next full kernel will be a normal boot with a full networking stack, SSD drivers, disk encryption, etc. We mount up our PMEM filesystem, pull out the kdump and either store it somewhere persistent or upload it somewhere. We can then unmount the PMEM and reconfigure it back to system ram so that the live system isn't missing memory.

Quoted 7 lines

    ndctl create-namespace --reconfig=namespace0.0 -m devdax -f
    daxctl reconfigure-device --mode=system-ram dax0.0

This is the flow I'm trying to support, and have mostly working in a VM, except up until now makedumpfile would crash because all the memblock structures it needed were in the PMEM area that I had just wiped out by making a new filesystem. :)

Thinking out loud (and remembering that some architectures relocate the crashkernel during kexec, if I am not wrong), maybe the following would also work and make your setup eventually easier:

1) Don't reserve a crashkernel area in the traditional way, instead reserve that area using CMA. It can be used for MOVABLE allocations.

2) Let kexec load the crashkernel+initrd into ordinary memory only (consuming as much as you would need there).

Quoted 12 lines · Ross Zwisler

Quoted 10 lines · Michal Hocko

On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
[...]

Quoted 4 lines

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

movablecore is kinda hack and I would be more inclined to get rid of it rather than build more into it. Could you be more specific about your use case?

Quoted 7 lines

The problem that I'm trying to solve is that I'd like to be able to get kernel core dumps off machines (chromebooks) so that we can debug crashes. Because the memory used by the crash kernel ("crashkernel=" kernel command line option) is consumed the entire time the machine is booted, there is a strong motivation to keep the crash kernel as small and as simple as possible. To this end I'm trying to get away without SSD drivers, not having to worry about encryption on the SSDs, etc.

Okay, so you intend to keep the crashkernel area as small as possible.

Quoted 3 lines

So, the rough plan right now is:

Quoted 1 line

1) During boot set aside some memory that won't contain kernel

allocations.

Quoted 19 lines

I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways.

We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE region (or whatever non-kernel region) will be set aside as PMEM in the crash kernel. This is done with the memmap=nn[KMG]!ss[KMG] kernel command line parameter passed to the crash kernel.

So, in my sample 4G VM system, I see:

   # lsmem --split ZONES --output-all
   RANGE                                  SIZE  STATE REMOVABLE BLOCK NODE   ZONES
   0x0000000000000000-0x0000000007ffffff  128M online       yes     0    0    None
   0x0000000008000000-0x00000000bfffffff  2.9G online       yes  1-23    0   DMA32
   0x0000000100000000-0x000000012fffffff  768M online       yes 32-37    0  Normal
   0x0000000130000000-0x000000013fffffff  256M online       yes 38-39    0 Movable
   Memory block size:       128M
   Total online memory:       4G
   Total offline memory:      0B

Quoted 4 lines

so I'll pass "memmap=256M!0x130000000" to the crash kernel.

2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set aside only contains user data, which we don't want to store anyway.

I raised that in different context already, but such assumptions are not 100% future proof IMHO. For example, we might at one point be able to make user page tables movable and place them on there.

But yes, most kernel data structures (which you care about) will probably never be movable and never end up on these regions.

Quoted 16 lines

We make a
filesystem in there, and create a kernel crash dump using 'makedumpfile':

mkfs.ext4 /dev/pmem0
mount /dev/pmem0 /mnt
makedumpfile -c -d 31 /proc/vmcore /mnt/kdump

We then set up the next full kernel boot to also have this same PMEM region, using the same memmap kernel parameter. We reboot back into a full kernel.

3) The next full kernel will be a normal boot with a full networking stack, SSD drivers, disk encryption, etc. We mount up our PMEM filesystem, pull out the kdump and either store it somewhere persistent or upload it somewhere. We can then unmount the PMEM and reconfigure it back to system ram so that the live system isn't missing memory.

Quoted 7 lines

   ndctl create-namespace --reconfig=namespace0.0 -m devdax -f
   daxctl reconfigure-device --mode=system-ram dax0.0

This is the flow I'm trying to support, and have mostly working in a VM, except up until now makedumpfile would crash because all the memblock structures it needed were in the PMEM area that I had just wiped out by making a new filesystem. :)

Thinking out loud (and remembering that some architectures relocate the crashkernel during kexec, if I am not wrong), maybe the following would also work and make your setup eventually easier:

1) Don't reserve a crashkernel area in the traditional way, instead reserve that area using CMA. It can be used for MOVABLE allocations.

2) Let kexec load the crashkernel+initrd into ordinary memory only (consuming as much as you would need there).

3) On kexec, relocate the crashkernel+initrd into the CMA area (overwriting any movable data in there)

4) In makedumpfile, don't dump any memory that falls into the crashkernel area. It might already have been overwritten by the second kernel

Quoted 93 lines · David Hildenbrand

Quoted 12 lines · Ross Zwisler

Quoted 10 lines · Michal Hocko

On Tue 18-07-23 16:01:06, Ross Zwisler wrote:
[...]

Quoted 4 lines

I do think that we need to fix this collision between ZONE_MOVABLE and memmap allocations, because this issue essentially makes the movablecore= kernel command line parameter useless in many cases, as the ZONE_MOVABLE region it creates will often actually be unmovable.

movablecore is kinda hack and I would be more inclined to get rid of it rather than build more into it. Could you be more specific about your use case?

Quoted 7 lines

The problem that I'm trying to solve is that I'd like to be able to get kernel core dumps off machines (chromebooks) so that we can debug crashes. Because the memory used by the crash kernel ("crashkernel=" kernel command line option) is consumed the entire time the machine is booted, there is a strong motivation to keep the crash kernel as small and as simple as possible. To this end I'm trying to get away without SSD drivers, not having to worry about encryption on the SSDs, etc.

Okay, so you intend to keep the crashkernel area as small as possible.

Quoted 3 lines

So, the rough plan right now is:

Quoted 1 line

1) During boot set aside some memory that won't contain kernel

allocations.

Quoted 19 lines

I'm trying to do this now with ZONE_MOVABLE, but I'm open to better ways.

We set aside memory for a crash kernel & arm it so that the ZONE_MOVABLE region (or whatever non-kernel region) will be set aside as PMEM in the crash kernel. This is done with the memmap=nn[KMG]!ss[KMG] kernel command line parameter passed to the crash kernel.

So, in my sample 4G VM system, I see:

    # lsmem --split ZONES --output-all
    RANGE                                  SIZE  STATE REMOVABLE BLOCK NODE   ZONES
    0x0000000000000000-0x0000000007ffffff  128M online       yes     0    0    None
    0x0000000008000000-0x00000000bfffffff  2.9G online       yes  1-23    0   DMA32
    0x0000000100000000-0x000000012fffffff  768M online       yes 32-37    0  Normal
    0x0000000130000000-0x000000013fffffff  256M online       yes 38-39    0 Movable
    Memory block size:       128M
    Total online memory:       4G
    Total offline memory:      0B

Quoted 4 lines

so I'll pass "memmap=256M!0x130000000" to the crash kernel.

2) When we hit a kernel crash, we know (hope?) that the PMEM region we've set aside only contains user data, which we don't want to store anyway.

I raised that in different context already, but such assumptions are not 100% future proof IMHO. For example, we might at one point be able to make user page tables movable and place them on there.

But yes, most kernel data structures (which you care about) will probably never be movable and never end up on these regions.

Quoted 16 lines

We make a
filesystem in there, and create a kernel crash dump using 'makedumpfile':

mkfs.ext4 /dev/pmem0
mount /dev/pmem0 /mnt
makedumpfile -c -d 31 /proc/vmcore /mnt/kdump

We then set up the next full kernel boot to also have this same PMEM region, using the same memmap kernel parameter. We reboot back into a full kernel.

3) The next full kernel will be a normal boot with a full networking stack, SSD drivers, disk encryption, etc. We mount up our PMEM filesystem, pull out the kdump and either store it somewhere persistent or upload it somewhere. We can then unmount the PMEM and reconfigure it back to system ram so that the live system isn't missing memory.

Quoted 7 lines

    ndctl create-namespace --reconfig=namespace0.0 -m devdax -f
    daxctl reconfigure-device --mode=system-ram dax0.0

This is the flow I'm trying to support, and have mostly working in a VM, except up until now makedumpfile would crash because all the memblock structures it needed were in the PMEM area that I had just wiped out by making a new filesystem. :)

Thinking out loud (and remembering that some architectures relocate the crashkernel during kexec, if I am not wrong), maybe the following would also work and make your setup eventually easier:

1) Don't reserve a crashkernel area in the traditional way, instead reserve that area using CMA. It can be used for MOVABLE allocations.

2) Let kexec load the crashkernel+initrd into ordinary memory only (consuming as much as you would need there).

Quoted 5 lines

3) On kexec, relocate the crashkernel+initrd into the CMA area (overwriting any movable data in there)

4) In makedumpfile, don't dump any memory that falls into the crashkernel area. It might already have been overwritten by the second kernel

This is more or less what Jiri is looking into.