powerpc/kernel: Enables memory hot-remove after reboot on pseries guests
PATCH v3 1/1 powerpc/kernel: Enables memory hot-remove after reboot on pseries guests
While providing guests, it's desirable to resize it's memory on demand.
By now, it's possible to do so by creating a guest with a small base memory, hot-plugging all the rest, and using 'movable_node' kernel command-line parameter, which puts all hot-plugged memory in ZONE_MOVABLE, allowing it to be removed whenever needed.
But there is an issue regarding guest reboot:
If memory is hot-plugged, and then the guest is rebooted, all hot-plugged memory goes to ZONE_NORMAL, which offers no guaranteed hot-removal. It usually prevents this memory to be hot-removed from the guest.
It's possible to use device-tree information to fix that behavior, as it stores flags for LMB ranges on ibm,dynamic-memory-vN.
It involves marking each memblock with the correct flags as hotpluggable memory, which mm/memblock.c puts in ZONE_MOVABLE during boot if 'movable_node' is passed.
For carrying such information, the new flag DRCONF_MEM_HOTREMOVABLE was proposed and accepted into Power Architecture documentation. This flag should be:
- true (b=1) if the hypervisor may want to hot-remove it later, and
- false (b=0) if it does not care.
During boot, guest kernel reads the device-tree, early_init_drmem_lmb() is called for every added LMBs. Here, checking for this new flag and marking memblocks as hotplugable memory is enough to get the desirable behavior.
This should cause no change if 'movable_node' parameter is not passed in kernel command-line.
---
Changes since v2:
- New flag name changed from DRCONF_MEM_HOTPLUGGED to DRCONF_MEM_HOTREMOVABLE
Changes since v1:
- Adds new flag, so PowerVM is compatible with the change.
- Fixes mistakes in code
--- arch/powerpc/include/asm/drmem.h | 1 + arch/powerpc/kernel/prom.c | 9 +++++++-- 2 files changed, 8 insertions(+), 2 deletions(-)
arch/powerpc/include/asm/drmem.h +1 -0
@@ -65,6 +65,7 @@ struct of_drconf_cell_v2 {#define DRCONF_MEM_ASSIGNED 0x00000008#define DRCONF_MEM_AI_INVALID 0x00000040#define DRCONF_MEM_RESERVED 0x00000080 +#define DRCONF_MEM_HOTREMOVABLE 0x00000100static inline u32 drmem_lmb_size (void ) {
arch/powerpc/kernel/prom.c +7 -2
@@ -515,9 +515,14 @@ static void __init early_init_drmem_lmb(struct drmem_lmb *lmb, size = 0x80000000ul - base; } +if (!validate_mem_limit (base, &size)) +continue ; +DBG ("Adding: %llx -> %llx\n" , base, size); -if (validate_mem_limit (base, &size)) -memblock_add (base, size); +memblock_add (base, size); + +if (lmb->flags &DRCONF_MEM_HOTREMOVABLE ) +memblock_mark_hotplug (base, size); }while (--rngs); }#endif /* CONFIG_PPC_PSERIES */
Signature · 1 line
2.25.1
Quoted 13 lines · Leonardo Bras
While providing guests, it's desirable to resize it's memory on demand.
By now, it's possible to do so by creating a guest with a small base memory, hot-plugging all the rest, and using '
movable_node' kernel command-line parameter, which puts all hot-plugged memory inZONE_MOVABLE, allowing it to be removed whenever needed.But there is an issue regarding guest reboot:
If memory is hot-plugged, and then the guest is rebooted, all hot-plugged memory goes toZONE_NORMAL, which offers no guaranteed hot-removal. It usually prevents this memory to be hot-removed from the guest.
Quoted 5 lines
It's possible to use device-tree information to fix that behavior, as it stores flags for LMB ranges on ibm,dynamic-memory-vN.
It involves marking each memblock with the correct flags as hotpluggable memory, whichmm/memblock.cputs inZONE_MOVABLEduring boot if 'movable_node' is passed.
I don't really understand why the flag is needed at all. According to PAPR any memory provided by dynamic reconfiguration can be hot-removed so why aren't we treating all DR memory as hot removable? The only memory guaranteed to be there 100% of the time is what's in the /memory@0 node since that's supposed to cover the real mode area.
Hello Oliver, thank you for the feedback.
Comments inline:
Quoted 6 lines · Oliver O'Halloran
I don't really understand why the flag is needed at all. According to PAPR any memory provided by dynamic reconfiguration can be hot-removed so why aren't we treating all DR memory as hot removable? The only memory guaranteed to be there 100% of the time is what's in the /memory@0 node since that's supposed to cover the real mode area.
All LMBs are listed in DR memory, even the base memory.
The v1 of the patch would work this way, as qemu would configure it's DR memory with (DRC_INVALID | RESERVED) flags and the hot-added memory with (ASSIGNED) flag. Looking for assigned flag would be enough.
But as of today, PowerVM doesn't seem to work that way.
When you boot a PowerVM virtual machine with Linux, all memory is added with the same flags (ASSIGNED).
To create a solution that doesn't break PowerVM, this new flag was made necessary.
Best regards,
Leonardo Bras
Quoted 18 lines · Leonardo Bras
Hello Oliver, thank you for the feedback.
Comments inline:Quoted 6 lines · Oliver O'Halloran
I don't really understand why the flag is needed at all. According to PAPR any memory provided by dynamic reconfiguration can be hot-removed so why aren't we treating all DR memory as hot removable? The only memory guaranteed to be there 100% of the time is what's in the /memory@0 node since that's supposed to cover the real mode area.
All LMBs are listed in DR memory, even the base memory.
The v1 of the patch would work this way, as qemu would configure it's DR memory with (
DRC_INVALID| RESERVED) flags and the hot-added memory with (ASSIGNED) flag. Looking for assigned flag would be enough.
Quoted 6 lines
But as of today, PowerVM doesn't seem to work that way.
When you boot a PowerVM virtual machine with Linux, all memory is added with the same flags (ASSIGNED).To create a solution that doesn't break PowerVM, this new flag was made necessary.
I'm still not convinced it's necessary. Why not check memory@0 and use the size as a clip level? Any memory above that level gets marked as hotpluggable and anything below doesn't. Seems to me that would work on all current platforms, so what am I missing here?
Quoted 3 lines
Best regards,
Leonardo Bras
Quoted 21 lines · Oliver O'Halloran
Quoted 20 lines · Leonardo Bras
Hello Oliver, thank you for the feedback.
Comments inline:Quoted 5 lines · Oliver O'Halloran
I don't really understand why the flag is needed at all. According to PAPR any memory provided by dynamic reconfiguration can be hot-removed so why aren't we treating all DR memory as hot removable? The only memory guaranteed to be there 100% of the time is what's in the /memory@0 node since that's supposed to cover the real mode area.
All LMBs are listed in DR memory, even the base memory.
The v1 of the patch would work this way, as qemu would configure it's DR memory with (
DRC_INVALID| RESERVED) flags and the hot-added memory with (ASSIGNED) flag. Looking for assigned flag would be enough.But as of today, PowerVM doesn't seem to work that way.
When you boot a PowerVM virtual machine with Linux, all memory is added with the same flags (ASSIGNED).
Quoted 8 lines
Quoted 2 lines
To create a solution that doesn't break PowerVM, this new flag was made necessary.
I'm still not convinced it's necessary. Why not check memory@0 and use the size as a clip level? Any memory above that level gets marked as hotpluggable and anything below doesn't. Seems to me that would work on all current platforms, so what am I missing here?
Humm, wouldn't that assume that all unmovable memory should be at the first LMBs?
If we use the recently approved flag, there would be no such limitation, and there would be possible to do some other things, like hot-adding more unmovable memory to kernel usage.
What do you think?
Best regards,
Quoted 32 lines · Leonardo Bras
While providing guests, it's desirable to resize it's memory on demand.
By now, it's possible to do so by creating a guest with a small base memory, hot-plugging all the rest, and using '
movable_node' kernel command-line parameter, which puts all hot-plugged memory inZONE_MOVABLE, allowing it to be removed whenever needed.But there is an issue regarding guest reboot:
If memory is hot-plugged, and then the guest is rebooted, all hot-plugged memory goes toZONE_NORMAL, which offers no guaranteed hot-removal. It usually prevents this memory to be hot-removed from the guest.It's possible to use device-tree information to fix that behavior, as it stores flags for LMB ranges on ibm,dynamic-memory-vN.
It involves marking each memblock with the correct flags as hotpluggable memory, whichmm/memblock.cputs inZONE_MOVABLEduring boot if 'movable_node' is passed.For carrying such information, the new flag
DRCONF_MEM_HOTREMOVABLEwas proposed and accepted into Power Architecture documentation. This flag should be:
- true (b=1) if the hypervisor may want to hot-remove it later, and
- false (b=0) if it does not care.During boot, guest kernel reads the device-tree,
early_init_drmem_lmb()is called for every added LMBs. Here, checking for this new flag and marking memblocks as hotplugable memory is enough to get the desirable behavior.This should cause no change if '
movable_node' parameter is not passed in kernel command-line.
Quoted 8 lines
Signed-off-by Leonardo Bras <[email protected]>Reviewed-by Bharata B Rao <[email protected]>---
Changes since v2:
- New flag name changed fromDRCONF_MEM_HOTPLUGGEDtoDRCONF_MEM_HOTREMOVABLE
The patch would be more complete with the following change that ensures that DRCONF_MEM_HOTREMOVABLE flag is set for non-boot-time hotplugged memory too. This will ensure that ibm,dynamic-memory-vN property reflects the right flags value for memory that gets hotplugged post boot.
arch/powerpc/platforms/pseries/hotplug-memory.c +4 -2
@@ -395,7 +395,8 @@ static int dlpar_remove_lmb(struct drmem_lmb *lmb)invalidate_lmb_associativity_index (lmb);lmb_clear_nid (lmb); - lmb->flags &= ~DRCONF_MEM_ASSIGNED ; + lmb->flags &= ~(DRCONF_MEM_ASSIGNED | +DRCONF_MEM_HOTREMOVABLE );return 0 ; }
@@ -678,7 +679,8 @@ static int dlpar_add_lmb(struct drmem_lmb *lmb)invalidate_lmb_associativity_index (lmb);lmb_clear_nid (lmb); }else { - lmb->flags |=DRCONF_MEM_ASSIGNED ; + lmb->flags |= (DRCONF_MEM_ASSIGNED | +DRCONF_MEM_HOTREMOVABLE ); }return rc;
Regards,
Bharata.
Hello Bharata,
Quoted 6 lines · Bharata B Rao
The patch would be more complete with the following change that ensures that
DRCONF_MEM_HOTREMOVABLEflag is set for non-boot-time hotplugged memory too. This will ensure that ibm,dynamic-memory-vN property reflects the right flags value for memory that gets hotplugged post boot.
You just sent that on a separated patchset, so I think it's dealt with. Do you have any other comments on the present patch?
Best regards,
Quoted 10 lines · Leonardo Bras
Hello Bharata,
Quoted 6 lines · Bharata B Rao
The patch would be more complete with the following change that ensures that
DRCONF_MEM_HOTREMOVABLEflag is set for non-boot-time hotplugged memory too. This will ensure that ibm,dynamic-memory-vN property reflects the right flags value for memory that gets hotplugged post boot.
Quoted 2 lines
You just sent that on a separated patchset, so I think it's dealt with. Do you have any other comments on the present patch?
None, thanks.
Regards,
Bharata.
Hello Michael,
Would it be ok to add this patch for 5.7 ? Or too late?
Regards,
Quoted 19 lines · Bharata B Rao
Quoted 10 lines · Leonardo Bras
Hello Bharata,
Quoted 6 lines · Bharata B Rao
The patch would be more complete with the following change that ensures that
DRCONF_MEM_HOTREMOVABLEflag is set for non-boot-time hotplugged memory too. This will ensure that ibm,dynamic-memory-vN property reflects the right flags value for memory that gets hotplugged post boot.Quoted 2 lines
You just sent that on a separated patchset, so I think it's dealt with. Do you have any other comments on the present patch?
None, thanks.
Regards,
Bharata.
Michael Ellerman
AppliedQuoted 8 lines · Leonardo Bras
While providing guests, it's desirable to resize it's memory on demand.
By now, it's possible to do so by creating a guest with a small base memory, hot-plugging all the rest, and using '
movable_node' kernel command-line parameter, which puts all hot-plugged memory inZONE_MOVABLE, allowing it to be removed whenever needed.[...]
Applied to powerpc/next.
[1/1] powerpc/kernel: Enables memory hot-remove after reboot on pseries guests
https://git.kernel.org/powerpc/c/b6eca183e23e7a6625a0d2cdb806b7cd1abcd2d2
cheers