libnvdimm/e820: Add a new parameter to configure many regions per e820 entry
Subsystems (2)
PATCH v2 1/1 libnvdimm/e820: Add a new parameter to configure many regions per e820 entry
Currently, the user has to specify each memory region to be used with nvdimm via the memmap parameter. Due to the character limit of the command line, this makes it impossible to have a lot of pmem devices. This new parameter solves this issue by allowing users to divide one e820 entry into many nvdimm regions.
This change is needed for the hypervisor live update. VMs' memory will be backed by those emulated pmem devices. To support various VM shapes I want to create devdax devices at 1GB granularity similar to hugetlb.
It's also possible to expand this parameter in the future, e.g. to specify the type of the device (fsdax/devdax).
---
v2: Fixed a crash when pmem parameter is omitted.
--- .../admin-guide/kernel-parameters.txt | 7 + drivers/nvdimm/e820.c | 149 +++++++++++++++++- 2 files changed, 153 insertions(+), 3 deletions(-)
Documentation/admin-guide/kernel-parameters.txt +7 -0
@@ -3849,6 +3849,13 @@ n2= [NET] SDL Inc. RISCom/N2 synchronous serial card + nd_e820.pmem=ss[KMG],nn[KMG] + Divide one e820 entry specified by memmap=x!ss + (that is starting at ss) into pmem devices of size nn. + There can be only one pmem parameter per one e820 + entry. The size of the e820 entry has to be divisible + by the device size. + netdev= [NET] Network devices parameters Format: <irq>,<io>,<mem_start>,<mem_end>,<name> Note that mem_start is often overloaded to mean
drivers/nvdimm/e820.c +146 -3
182 lines not on this page · raw message
Signature · 1 line
2.49.0.777.g153de2bbd5-goog
Quoted 9 lines · Michal Clapinski
Currently, the user has to specify each memory region to be used with nvdimm via the memmap parameter. Due to the character limit of the command line, this makes it impossible to have a lot of pmem devices. This new parameter solves this issue by allowing users to divide one e820 entry into many nvdimm regions.
This change is needed for the hypervisor live update. VMs' memory will be backed by those emulated pmem devices. To support various VM shapes I want to create devdax devices at 1GB granularity similar to hugetlb.
Why is it not sufficient to create a region out of a single memmap range and create multiple 1G dax devices within that single range?
Ira
Quoted 5 lines
It's also possible to expand this parameter in the future, e.g. to specify the type of the device (fsdax/devdax).
Signed-off-by Michal Clapinski <[email protected]>
[snip]
Quoted 8 lines · Ira Weiny
Quoted 6 lines · Michal Clapinski
Currently, the user has to specify each memory region to be used with nvdimm via the memmap parameter. Due to the character limit of the command line, this makes it impossible to have a lot of pmem devices. This new parameter solves this issue by allowing users to divide one e820 entry into many nvdimm regions.
Quoted 6 lines
Quoted 3 lines
This change is needed for the hypervisor live update. VMs' memory will be backed by those emulated pmem devices. To support various VM shapes I want to create devdax devices at 1GB granularity similar to hugetlb.
Why is it not sufficient to create a region out of a single memmap range and create multiple 1G dax devices within that single range?
This method implies using the ndctl tool to create regions and convert them to dax devices from userspace. This does not work for our use case. We must have these 1 GB regions available during boot because we do not want to lose memory for a devdax label. I.e., if fsdax is created during boot (i.e. default pmem format), it does not have a label. However, if it is created from userspace, we create a label with partition properties, UUID, etc. Here, we need to use kernel parameters to specify the properties of the pmem devices during boot so they can persist across reboots without losing any memory to labels.
Pasha
Quoted 16 lines · Pasha Tatashin
Quoted 8 lines · Ira Weiny
Quoted 6 lines · Michal Clapinski
Currently, the user has to specify each memory region to be used with nvdimm via the memmap parameter. Due to the character limit of the command line, this makes it impossible to have a lot of pmem devices. This new parameter solves this issue by allowing users to divide one e820 entry into many nvdimm regions.
Quoted 6 lines
Quoted 3 lines
This change is needed for the hypervisor live update. VMs' memory will be backed by those emulated pmem devices. To support various VM shapes I want to create devdax devices at 1GB granularity similar to hugetlb.
Why is it not sufficient to create a region out of a single memmap range and create multiple 1G dax devices within that single range?
Quoted 7 lines
This method implies using the ndctl tool to create regions and convert them to dax devices from userspace. This does not work for our use case. We must have these 1 GB regions available during boot because we do not want to lose memory for a devdax label. I.e., if fsdax is created during boot (i.e. default pmem format), it does not have a label. However, if it is created from userspace, we create a label with partition properties, UUID, etc. Here, we need to use kernel
Doesn't ndctl refuse to alter namespaces on "legacy" (i.e. memmap=) regions?
Quoted 5 lines
parameters to specify the properties of the pmem devices during boot so they can persist across reboots without losing any memory to labels.
Pasha
Signature · 2 lines
Sincerely yours, Mike.
Quoted 26 lines · Mike Rapoport
Quoted 16 lines · Pasha Tatashin
Quoted 8 lines · Ira Weiny
Quoted 6 lines · Michal Clapinski
Currently, the user has to specify each memory region to be used with nvdimm via the memmap parameter. Due to the character limit of the command line, this makes it impossible to have a lot of pmem devices. This new parameter solves this issue by allowing users to divide one e820 entry into many nvdimm regions.
Quoted 6 lines
Quoted 3 lines
This change is needed for the hypervisor live update. VMs' memory will be backed by those emulated pmem devices. To support various VM shapes I want to create devdax devices at 1GB granularity similar to hugetlb.
Why is it not sufficient to create a region out of a single memmap range and create multiple 1G dax devices within that single range?
Quoted 7 lines
This method implies using the ndctl tool to create regions and convert them to dax devices from userspace. This does not work for our use case. We must have these 1 GB regions available during boot because we do not want to lose memory for a devdax label. I.e., if fsdax is created during boot (i.e. default pmem format), it does not have a label. However, if it is created from userspace, we create a label with partition properties, UUID, etc. Here, we need to use kernel
Quoted 2 lines
Doesn't ndctl refuse to alter namespaces on "legacy" (i.e. memmap=) regions?
Hi Mike
ndctl works with legacy namespaces just fine. We can convert them to devdax/fsdax/raw pmem, create remove label, etc.
Pasha
Quoted 10 lines
Quoted 6 lines
parameters to specify the properties of the pmem devices during boot so they can persist across reboots without losing any memory to labels.
Pasha
--
Sincerely yours,
Mike.
Quoted 9 lines · Michal Clapinski
Currently, the user has to specify each memory region to be used with nvdimm via the memmap parameter. Due to the character limit of the command line, this makes it impossible to have a lot of pmem devices. This new parameter solves this issue by allowing users to divide one e820 entry into many nvdimm regions.
This change is needed for the hypervisor live update. VMs' memory will be backed by those emulated pmem devices. To support various VM shapes I want to create devdax devices at 1GB granularity similar to hugetlb.
This looks fairly straightforward, but if this moves forward I would explicitly call the parameter something like "split" instead of "pmem" to align it better with its usage.
However, while this is expedient I wonder if you would be better served with ACPI table injection to get more control and configuration options...
Quoted 2 lines
It's also possible to expand this parameter in the future, e.g. to specify the type of the device (fsdax/devdax).
...for example, if you injected or customized your BIOS to supply an ACPI NFIT table you could get to deeper degrees of customization without wrestling with command lines. Supply an ACPI NFIT that carves up a large memory-type range into an aribtrary number of regions. In the NFIT there is a natural place to specify whether the range gets sent to PMEM. See call to nvdimm_pmem_region_create() near NFIT_SPA_PM in acpi_nfit_register_region()", and "simply" pick a new guid to signify direct routing to device-dax. I say simply, but that implies new ACPI NFIT driver plumbing for the new mode.
Another overlooked detail about NFIT is that there is an opportunity to determine cases where the platform might have changed the physical address map from one boot to the next. In other words, I cringe at the fragility of memmap=, but I understand that it has the benefit of being simple. See the "nd_set cookie" concept in acpi_nfit_init_interleave_set().
Quoted 33 lines · Dan Williams
Quoted 9 lines · Michal Clapinski
Currently, the user has to specify each memory region to be used with nvdimm via the memmap parameter. Due to the character limit of the command line, this makes it impossible to have a lot of pmem devices. This new parameter solves this issue by allowing users to divide one e820 entry into many nvdimm regions.
This change is needed for the hypervisor live update. VMs' memory will be backed by those emulated pmem devices. To support various VM shapes I want to create devdax devices at 1GB granularity similar to hugetlb.
This looks fairly straightforward, but if this moves forward I would explicitly call the parameter something like "split" instead of "pmem" to align it better with its usage.
However, while this is expedient I wonder if you would be better served with ACPI table injection to get more control and configuration options...
Quoted 2 lines
It's also possible to expand this parameter in the future, e.g. to specify the type of the device (fsdax/devdax).
...for example, if you injected or customized your BIOS to supply an ACPI NFIT table you could get to deeper degrees of customization without wrestling with command lines. Supply an ACPI NFIT that carves up a large memory-type range into an aribtrary number of regions. In the NFIT there is a natural place to specify whether the range gets sent to PMEM. See call to
nvdimm_pmem_region_create()nearNFIT_SPA_PMinacpi_nfit_register_region()", and "simply" pick a new guid to signify direct routing to device-dax. I say simply, but that implies new ACPI NFIT driver plumbing for the new mode.
Quoted 6 lines
Another overlooked detail about NFIT is that there is an opportunity to determine cases where the platform might have changed the physical address map from one boot to the next. In other words, I cringe at the fragility of memmap=, but I understand that it has the benefit of being simple. See the "
nd_setcookie" concept inacpi_nfit_init_interleave_set().
I also dislike the potential fragility of the memmap= parameter; however, in our environment, kernel parameters are specifically crafted for target machine configurations and supplied separately from the kernel binary, giving us good control.
Regarding the ACPI NFIT suggestion: Our use case involves reusing the same physical machines (with unchanged firmware) for various configurations (similar to loaning them out). An advantage for us is that switching the machine's role only requires changing the kernel parameters. The ACPI approach, potentially requiring firmware changes, would break this dynamic reconfiguration.
As I understand, using ACPI injection instead of firmware change doesn't eliminate fragility concerns either. We would still need to carefully reserve the specific physical range for a particular machine configuration, and it also adds a dependency on managing and packaging an external NFIT injection file and process. We have a process for kernel parameters but doing this externally would complicate things for us.
Also, I might be missing something, but I haven't found a standard way to automatically create devdax devices using NFIT injection. Our current plan is to expand the proposed kernel parameter. We are working on making it default to creating either fsdax or devdax type regions, without requiring explicit labels, and ensuring these regions remain stable across kexec as long as the kernel parameter itself doesn't change (in a way kernel parameters take the role of the labels).
Pasha
Quoted 53 lines · Pasha Tatashin
Quoted 33 lines · Dan Williams
Quoted 9 lines · Michal Clapinski
Currently, the user has to specify each memory region to be used with nvdimm via the memmap parameter. Due to the character limit of the command line, this makes it impossible to have a lot of pmem devices. This new parameter solves this issue by allowing users to divide one e820 entry into many nvdimm regions.
This change is needed for the hypervisor live update. VMs' memory will be backed by those emulated pmem devices. To support various VM shapes I want to create devdax devices at 1GB granularity similar to hugetlb.
This looks fairly straightforward, but if this moves forward I would explicitly call the parameter something like "split" instead of "pmem" to align it better with its usage.
However, while this is expedient I wonder if you would be better served with ACPI table injection to get more control and configuration options...
Quoted 2 lines
It's also possible to expand this parameter in the future, e.g. to specify the type of the device (fsdax/devdax).
...for example, if you injected or customized your BIOS to supply an ACPI NFIT table you could get to deeper degrees of customization without wrestling with command lines. Supply an ACPI NFIT that carves up a large memory-type range into an aribtrary number of regions. In the NFIT there is a natural place to specify whether the range gets sent to PMEM. See call to
nvdimm_pmem_region_create()nearNFIT_SPA_PMinacpi_nfit_register_region()", and "simply" pick a new guid to signify direct routing to device-dax. I say simply, but that implies new ACPI NFIT driver plumbing for the new mode.Quoted 6 lines
Another overlooked detail about NFIT is that there is an opportunity to determine cases where the platform might have changed the physical address map from one boot to the next. In other words, I cringe at the fragility of memmap=, but I understand that it has the benefit of being simple. See the "
nd_setcookie" concept inacpi_nfit_init_interleave_set().I also dislike the potential fragility of the memmap= parameter; however, in our environment, kernel parameters are specifically crafted for target machine configurations and supplied separately from the kernel binary, giving us good control.
Regarding the ACPI NFIT suggestion: Our use case involves reusing the same physical machines (with unchanged firmware) for various configurations (similar to loaning them out). An advantage for us is that switching the machine's role only requires changing the kernel parameters. The ACPI approach, potentially requiring firmware changes, would break this dynamic reconfiguration.
Quoted 7 lines
As I understand, using ACPI injection instead of firmware change doesn't eliminate fragility concerns either. We would still need to carefully reserve the specific physical range for a particular machine configuration, and it also adds a dependency on managing and packaging an external NFIT injection file and process. We have a process for kernel parameters but doing this externally would complicate things for us.
Lets unpack a few things. My assumption is that ACPI table injection deployment is similar in complexity to kernel parameters because it is data appended to an initrd. So if a deployment flow can:
echo $parameters >> $boot_config
...it can instead:
cat $base_initrd $nfit > $amended_initrd
As for the fragility I do agree that without platform firmware changes (base system NFIT) then it would be difficult to detect that the platform is booting in an unexpected physical memory layout.
So memmap= would be used to mark the memory as Reserved and then the injected NFIT carves it up and optionally routes it to pmem or devdax.
The aspect I have not tried though is injecting an ACPI0012 device if the platform does not already have one...
I think it is solvable and avoids continuing to stress the kernel command line interface where ACPI can takeover. At a minimum confirm whether amending initrds is a non-starter in your environment.
Quoted 2 lines
Also, I might be missing something, but I haven't found a standard way to automatically create devdax devices using NFIT injection. Our
Yes, this is not there today, but would fit cleanly as a new Linux specific "Address Range Type GUID".
Quoted 6 lines
current plan is to expand the proposed kernel parameter. We are working on making it default to creating either fsdax or devdax type regions, without requiring explicit labels, and ensuring these regions remain stable across kexec as long as the kernel parameter itself doesn't change (in a way kernel parameters take the role of the labels).
Yes, this should all work without labels.
Dan Williams wrote:
[..]
Quoted 3 lines
I think it is solvable and avoids continuing to stress the kernel command line interface where ACPI can takeover. At a minimum confirm whether amending initrds is a non-starter in your environment.
When I say "it is solvable" I am specifically referring to tweaks to the NFIT driver to get it to operate without an ACPI0012 device or otherwise tweak the table injection code to automate adding an ACPI0012 device when an ACPI NFIT is injected.