Skip to contentSkip to conversation

Disable auto_movable_ratio for selfhosted memmap

RFC Oscar Salvador 4 participants 24 messages Jul 28, 2025 – Jul 29, 2025 Memory management

lore
Share & tools
lore
#0 lore raw

Hi,

Currently, we have several mechanisms to pick a zone for the new memory we are onlining.
Eventually, we will land on zone_for_pfn_range() which will pick the zone.

Two of these mechanisms are 'movable_node' and 'auto-movable' policy. The former will put every single hotpluggled memory in ZONE_MOVABLE (unless we can keep zones contiguous by not doing so), while the latter will put it in ZONA_MOVABLE IFF we are within the established ratio MOVABLE:KERNEL.

It seems, the later doesn't play well with CXL memory where CXL cards hold really large amounts of memory, making the ratio fail, and since CXL cards must be removed as a unit, it can't be done if any memory block fell within !ZONE_MOVABLE zone.

One way to tackle this would be update the ratio every time a new CXL card gets inserted, but this seems suboptimal.
Another way is that since CXL memory works with selfhosted memmap, we could relax the check when 'auto-movable' and only look at the ratio if we aren't working with selfhosted memmap.

Something like the following (acthung: it's just a PoC) Comments? Ideas?

diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index 5c6c1d6bb59f..ff87cfb3881a 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -234,7 +234,7 @@ static int memory_block_online(struct memory_block *mem)
return -EHWPOISON;

zone = zone_for_pfn_range(mem->online_type, mem->nid, mem->group,
- start_pfn, nr_pages);
+ start_pfn, nr_pages, mem->altmap);

/*
* Although vmemmap pages have a different lifecycle than the pages
@@ -473,11 +473,11 @@ static ssize_t phys_device_show(struct device *dev,
static int print_allowed_zone(char *buf, int len, int nid,
struct memory_group *group, unsigned long start_pfn, unsigned long nr_pages,
- int online_type, struct zone *default_zone)
+ int online_type, struct zone *default_zone, struct vmem_altmap *altmap)
{
struct zone *zone;

- zone = zone_for_pfn_range(online_type, nid, group, start_pfn, nr_pages);
+ zone = zone_for_pfn_range(online_type, nid, group, start_pfn, nr_pages, altmap);
if (zone == default_zone)
return 0;

@@ -509,13 +509,13 @@ static ssize_t valid_zones_show(struct device *dev,
}

default_zone = zone_for_pfn_range(MMOP_ONLINE, nid, group,
- start_pfn, nr_pages);
+ start_pfn, nr_pages, mem->altmap);

len = sysfs_emit(buf, "%s", default_zone->name);
len += print_allowed_zone(buf, len, nid, group, start_pfn, nr_pages,
- MMOP_ONLINE_KERNEL, default_zone);
+ MMOP_ONLINE_KERNEL, default_zone, mem->altmap);
len += print_allowed_zone(buf, len, nid, group, start_pfn, nr_pages,
- MMOP_ONLINE_MOVABLE, default_zone);
+ MMOP_ONLINE_MOVABLE, default_zone, mem->altmap);
len += sysfs_emit_at(buf, len, "\n");
return len;
}
diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h
index 23f038a16231..89f7b9c5d995 100644
--- a/include/linux/memory_hotplug.h
+++ b/include/linux/memory_hotplug.h
@@ -328,7 +328,7 @@ extern struct page *sparse_decode_mem_map(unsigned long coded_mem_map,
unsigned long pnum);
extern struct zone *zone_for_pfn_range(int online_type, int nid,
struct memory_group *group, unsigned long start_pfn,
- unsigned long nr_pages);
+ unsigned long nr_pages, struct vmem_altmap *altmap);
extern int arch_create_linear_mapping(int nid, u64 start, u64 size,
struct mhp_params *params);
void arch_remove_linear_mapping(u64 start, u64 size);
diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c
index 69a636e20f7b..6c6600a9c839 100644
--- a/mm/memory_hotplug.c
+++ b/mm/memory_hotplug.c
@@ -1048,7 +1048,7 @@ static inline struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn

struct zone *zone_for_pfn_range(int online_type, int nid,
struct memory_group *group, unsigned long start_pfn,
- unsigned long nr_pages)
+ unsigned long nr_pages, struct vmem_altmap *altmap)
{
if (online_type == MMOP_ONLINE_KERNEL)
return default_kernel_zone_for_pfn(nid, start_pfn, nr_pages);
@@ -1056,6 +1056,10 @@ struct zone *zone_for_pfn_range(int online_type, int nid,
if (online_type == MMOP_ONLINE_MOVABLE)
return &NODE_DATA(nid)->node_zones[ZONE_MOVABLE];

+ /* Selfhosted memmap, skip ratio check */
+ if (online_policy == ONLINE_POLICY_AUTO_MOVABLE && altmap)
+ return &NODE_DATA(nid)->node_zones[ZONE_MOVABLE];
+
if (online_policy == ONLINE_POLICY_AUTO_MOVABLE)
return auto_movable_zone_for_pfn(nid, group, start_pfn, nr_pages);

Signature · 2 lines
Oscar Salvador
SUSE Labs
#1 lore raw
Quoted 1 line · Oscar Salvador

Hi,

Hi,

Quoted 10 lines

Currently, we have several mechanisms to pick a zone for the new memory we are onlining.
Eventually, we will land on zone_for_pfn_range() which will pick the zone.

Two of these mechanisms are 'movable_node' and 'auto-movable' policy. The former will put every single hotpluggled memory in ZONE_MOVABLE (unless we can keep zones contiguous by not doing so), while the latter will put it in ZONA_MOVABLE IFF we are within the established ratio MOVABLE:KERNEL.

It's more complicated, because we have the concept of memory groups.

Dynamic memory groups allow for a mixture of MOVABLE vs. NORMAL within the group, static memory groups want a single type.

Hotplugging a large DIMM would online it either as MOVABLE or NORMAL. Similarly with CXL.

Quoted 5 lines

It seems, the later doesn't play well with CXL memory where CXL cards hold really large amounts of memory, making the ratio fail, and since CXL cards must be removed as a unit, it can't be done if any memory block fell within !ZONE_MOVABLE zone.

So, user space configured a ratio and the kernel does exactly that: obey the ratio.

Quoted 6 lines

One way to tackle this would be update the ratio every time a new CXL card gets inserted, but this seems suboptimal.
Another way is that since CXL memory works with selfhosted memmap, we could relax the check when 'auto-movable' and only look at the ratio if we aren't working with selfhosted memmap.

The memmap is only a small piece of unmovable data we require late at runtime (a bigger factor is user space page tables actually mapping that memory). The zone ratio we have configured in the kernel dates back to the highmem times, where such ratios were considered safe. Maybe there are better defaults for the ratios today, but it really depends on the workload.

One could find ways of subtracting the selfhosted part, to account it differently in the kernel, but the memmap is not the only consumer that affects the ratio.

I mean, the memmap is roughly 1.6%, I don't think that really makes a difference for you, does it? Can you share some real-life examples?

I have a colleague working on one of my old prototypes (memoryhotplugd) for replacing udev rules.

The idea there is, to detect that CXL memory is getting hotplugged and keep it offline. Because user space hotplugging that memory (daxctl) will explicitly online it to the proper zone.

Things like virtio-mem, DIMMs etc can happily use the auto-movable behavior. But the auto-movable behavior doesn't quite make sense if (a) you want everything movable and (b) daxctl already expects to online the memory itself, usually to ZONE_MOVABLE.

So I think this is mostly a user-space problem to solve.

Signature · 3 lines
Cheers,

David / dhildenb
#2 lore raw
Quoted 2 lines · David Hildenbrand
Quoted 1 line · Oscar Salvador

Hi,

[ .. ]

Quoted 9 lines
Quoted 6 lines

One way to tackle this would be update the ratio every time a new CXL card gets inserted, but this seems suboptimal.
Another way is that since CXL memory works with selfhosted memmap, we could relax the check when 'auto-movable' and only look at the ratio if we aren't working with selfhosted memmap.

The memmap is only a small piece of unmovable data we require late at runtime (a bigger factor is user space page tables actually mapping that memory). The zone ratio we have configured in the kernel dates back to the highmem times, where such ratios were considered safe. Maybe there are better defaults for the ratios today, but it really depends on the workload.

Point is, the ratio is accounted for the _entire_ memory. Which means that you have to _know_ how much memory you are going to plug in prior to plugging that in.
So to make that correct one would need to update the ratio prior to plug in one module, check if that succeeded, update the ratio, plug in the next module, check that, etc.

Really?

Quoted 5 lines

One could find ways of subtracting the selfhosted part, to account it differently in the kernel, but the memmap is not the only consumer that affects the ratio.

I mean, the memmap is roughly 1.6%, I don't think that really makes a difference for you, does it? Can you share some real-life examples?

Quoted 8 lines

I have a colleague working on one of my old prototypes (memoryhotplugd) for replacing udev rules.

The idea there is, to detect that CXL memory is getting hotplugged and keep it offline. Because user space hotplugging that memory (daxctl) will explicitly online it to the proper zone.

Things like virtio-mem, DIMMs etc can happily use the auto-movable behavior. But the auto-movable behavior doesn't quite make sense if (a) you want everything movable and (b) daxctl already expects to online the memory itself, usually to ZONE_MOVABLE.

So I think this is mostly a user-space problem to solve.

Hmm.
Yes, and no.

While CXL memory is hotpluggable (it's a PCI device, after all), it won't be hotplugged on a regular basis.
So the current use-case I'm aware of is that the system will be configured once, and then it will be expected to come up in the very same state after reboot.
As such a daemon is a bit of an overkill, as the number of events it would need to listen to is in the very low single-digit range.

Udev rules would work fine, though (in fact, we already have one ...) so I'd be happy to keep CXL memory offline after boot / hotplug and let udev / daxctl handle things.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#3 lore raw
Quoted 19 lines · Hannes Reinecke
Quoted 2 lines · David Hildenbrand
Quoted 1 line · Oscar Salvador

Hi,

[ .. ]

Quoted 8 lines
Quoted 7 lines

One way to tackle this would be update the ratio every time a new CXL card gets inserted, but this seems suboptimal.
Another way is that since CXL memory works with selfhosted memmap, we could relax
the check when 'auto-movable' and only look at the ratio if we aren't working with selfhosted memmap.

Quoted 7 lines

The memmap is only a small piece of unmovable data we require late at runtime (a bigger factor is user space page tables actually mapping that memory). The zone ratio we have configured in the kernel dates back to the highmem times, where such ratios were considered safe. Maybe there are better defaults for the ratios today, but it really depends on the workload.

Quoted 5 lines

Point is, the ratio is accounted for the _entire_ memory.
Which means that you have to _know_ how much memory you are going to
plug in prior to plugging that in.
So to make that correct one would need to update the ratio prior to> plug in one module, check if that succeeded, update the ratio, plug in the next module, check that, etc.

I am confused. We know how big a DIMM is at the time we plug it. I assume you talk about CXL?

Can you describe how that workflow would look like with tools like daxctl?

(what is a "module"? A DIMM?)

Quoted 21 lines
Quoted 21 lines

One could find ways of subtracting the selfhosted part, to account it differently in the kernel, but the memmap is not the only consumer that affects the ratio.

I mean, the memmap is roughly 1.6%, I don't think that really makes a difference for you, does it? Can you share some real-life examples?

I have a colleague working on one of my old prototypes (memoryhotplugd) for replacing udev rules.

The idea there is, to detect that CXL memory is getting hotplugged and keep it offline. Because user space hotplugging that memory (daxctl) will explicitly online it to the proper zone.

Things like virtio-mem, DIMMs etc can happily use the auto-movable behavior. But the auto-movable behavior doesn't quite make sense if (a) you want everything movable and (b) daxctl already expects to online the memory itself, usually to ZONE_MOVABLE.

Quoted 7 lines
Quoted 2 lines

So I think this is mostly a user-space problem to solve.

Hmm.
Yes, and no.

While CXL memory is hotpluggable (it's a PCI device, after all), it won't be hotplugged on a regular basis.

I've been told that with dynamic memory pooling it is supposed to get much more dynamic.

Quoted 5 lines

So the current use-case I'm aware of is that the system will be configured once, and then it will be expected to come up in the very same state after reboot.
As such a daemon is a bit of an overkill, as the number of events it would need to listen to is in the very low single-digit range.

I am mostly concerned with all the use cases that existed before CXL (in particular, virtio-mem, standby memory on s390x, DIMMs) where you see memory hotplug way more frequently and also would want to deal with things such as memory onlining failing in some environments more gracefully (e.g., retry).

What I realized is that
(1) udev rules are not a good for all use cases
(2) auto-onlining in the kernel is not good fit for all use cases

The goal of the daemon will be to configure auto-onlining in the kernel where possible (e.g., only virtio-mem, only CXL), but fallback to manual onlining in case mixtures might be possible (CXL and virtio-mem etc). I expect the latter to be rare, but sometimes we can't make a fully reliable decision of what might get hotplugged in the future ...

Signature · 3 lines
Cheers,

David / dhildenb
#4 lore raw
In reply to Oscar Salvador
Quoted 12 lines · Oscar Salvador

Hi,

Currently, we have several mechanisms to pick a zone for the new memory we are onlining.
Eventually, we will land on zone_for_pfn_range() which will pick the zone.

Two of these mechanisms are 'movable_node' and 'auto-movable' policy. The former will put every single hotpluggled memory in ZONE_MOVABLE (unless we can keep zones contiguous by not doing so), while the latter will put it in ZONA_MOVABLE IFF we are within the established ratio MOVABLE:KERNEL.

Quoted 4 lines

It seems, the later doesn't play well with CXL memory where CXL cards hold really large amounts of memory, making the ratio fail, and since CXL cards must be removed as a unit, it can't be done if any memory block fell within !ZONE_MOVABLE zone.

I suspect this is just an example of how our existing memory hotplug interface based on memory blocks is just suoptimal and it doesn't fit new usecases. We should start thinking about how a new v2 api should look like. I am not sure how that should look like but I believe we should be able to express a "device" as whole rather than having a very loosely bound generic memblocks. Anyway this is likely for a longer discussion and a long term plan rather than addressing this particular issue.

Quoted 2 lines

One way to tackle this would be update the ratio every time a new CXL card gets inserted, but this seems suboptimal.

I do not think this is a usable interface.

Quoted 3 lines

Another way is that since CXL memory works with selfhosted memmap, we could relax the check when 'auto-movable' and only look at the ratio if we aren't working with selfhosted memmap.

This is likely the only choice we have with the current interface. We either need a way to disable the ratio altogether or make it more automagic and treat self hosted memory differently because that memory doesn't eat up ZONE_NORMAL memory and therefore cannot deplete it for ZONE_MOVABLE.

Lowmem (ZONE_NORMAL) oom problems are still possible but kinda unavoidable no matter what the hotplug interface is as the CXL usecase really needs its memory to be movable to operate it as desired AFAIU.

Quoted 10 lines

Something like the following (acthung: it's just a PoC) Comments? Ideas?

diff --git a/drivers/base/memory.c b/drivers/base/memory.c
index 5c6c1d6bb59f..ff87cfb3881a 100644
--- a/drivers/base/memory.c
+++ b/drivers/base/memory.c
@@ -234,7 +234,7 @@ static int memory_block_online(struct memory_block *mem)
return -EHWPOISON;

Quoted 3 lines

zone = zone_for_pfn_range(mem->online_type, mem->nid, mem->group,
- start_pfn, nr_pages);
+ start_pfn, nr_pages, mem->altmap);

Shouldn't this be a more descriptive (enum like) argument? ONLINE_MOVABLE, ONLINE_AUTO etc..

Signature · 2 lines
Michal Hocko
SUSE Labs
#5 lore raw
Quoted 18 lines · Michal Hocko
Quoted 12 lines · Oscar Salvador

Hi,

Currently, we have several mechanisms to pick a zone for the new memory we are onlining.
Eventually, we will land on zone_for_pfn_range() which will pick the zone.

Two of these mechanisms are 'movable_node' and 'auto-movable' policy. The former will put every single hotpluggled memory in ZONE_MOVABLE (unless we can keep zones contiguous by not doing so), while the latter will put it in ZONA_MOVABLE IFF we are within the established ratio MOVABLE:KERNEL.

Quoted 4 lines

It seems, the later doesn't play well with CXL memory where CXL cards hold really large amounts of memory, making the ratio fail, and since CXL cards must be removed as a unit, it can't be done if any memory block fell within !ZONE_MOVABLE zone.

Quoted 8 lines

I suspect this is just an example of how our existing memory hotplug interface based on memory blocks is just suoptimal and it doesn't fit new usecases. We should start thinking about how a new v2 api should look like. I am not sure how that should look like but I believe we should be able to express a "device" as whole rather than having a very loosely bound generic memblocks. Anyway this is likely for a longer discussion and a long term plan rather than addressing this particular issue.

We have that concept with memory groups in the kernel already.

In dax/kmem we register a static memory group. It will be considered one union.

Quoted 10 lines
Quoted 3 lines

One way to tackle this would be update the ratio every time a new CXL card gets inserted, but this seems suboptimal.

I do not think this is a usable interface.

Quoted 3 lines

Another way is that since CXL memory works with selfhosted memmap, we could relax the check when 'auto-movable' and only look at the ratio if we aren't working with selfhosted memmap.

Quoted 5 lines

This is likely the only choice we have with the current interface. We either need a way to disable the ratio altogether or make it more automagic and treat self hosted memory differently because that memory doesn't eat up ZONE_NORMAL memory and therefore cannot deplete it for ZONE_MOVABLE.

daxctl wants to online memory itself. We want to keep that memory offline from a kernel perspective and let daxctl handle it in this case.

We have that problem in RHEL where we currently require user space to disable udev rules so daxctl "can win".

Signature · 3 lines
Cheers,

David / dhildenb
#6 lore raw
Quoted 28 lines · David Hildenbrand
Quoted 18 lines · Michal Hocko
Quoted 12 lines · Oscar Salvador

Hi,

Currently, we have several mechanisms to pick a zone for the new memory we are onlining.
Eventually, we will land on zone_for_pfn_range() which will pick the zone.

Two of these mechanisms are 'movable_node' and 'auto-movable' policy. The former will put every single hotpluggled memory in ZONE_MOVABLE (unless we can keep zones contiguous by not doing so), while the latter will put it in ZONA_MOVABLE IFF we are within the established ratio MOVABLE:KERNEL.

Quoted 4 lines

It seems, the later doesn't play well with CXL memory where CXL cards hold really large amounts of memory, making the ratio fail, and since CXL cards must be removed as a unit, it can't be done if any memory block fell within !ZONE_MOVABLE zone.

Quoted 8 lines

I suspect this is just an example of how our existing memory hotplug interface based on memory blocks is just suoptimal and it doesn't fit new usecases. We should start thinking about how a new v2 api should look like. I am not sure how that should look like but I believe we should be able to express a "device" as whole rather than having a very loosely bound generic memblocks. Anyway this is likely for a longer discussion and a long term plan rather than addressing this particular issue.

Quoted 1 line

We have that concept with memory groups in the kernel already.

I must have missed that. I will have a look, thanks! Do we have any documentation for that? Memory group is an overloaded term in the kernel.

Quoted 2 lines

In dax/kmem we register a static memory group. It will be considered one union.

But we still do export those memory blocks and let udev or whoever act on those right? If that is the case then ....

[...]

Quoted 5 lines

daxctl wants to online memory itself. We want to keep that memory offline from a kernel perspective and let daxctl handle it in this case.

We have that problem in RHEL where we currently require user space to disable udev rules so daxctl "can win".

... this is the result. Those shouldn't really race. If udev is suppose to see the device then only in its entirity so regular memory block based onlining rules shouldn't even see that memory. Or am I completely missing the picture?

Signature · 2 lines
Michal Hocko
SUSE Labs
#7 lore raw
Quoted 29 lines · Michal Hocko
Quoted 28 lines · David Hildenbrand
Quoted 26 lines · Michal Hocko
Quoted 12 lines · Oscar Salvador

Hi,

Currently, we have several mechanisms to pick a zone for the new memory we are onlining.
Eventually, we will land on zone_for_pfn_range() which will pick the zone.

Two of these mechanisms are 'movable_node' and 'auto-movable' policy. The former will put every single hotpluggled memory in ZONE_MOVABLE (unless we can keep zones contiguous by not doing so), while the latter will put it in ZONA_MOVABLE IFF we are within the established ratio MOVABLE:KERNEL.

Quoted 4 lines

It seems, the later doesn't play well with CXL memory where CXL cards hold really large amounts of memory, making the ratio fail, and since CXL cards must be removed as a unit, it can't be done if any memory block fell within !ZONE_MOVABLE zone.

I suspect this is just an example of how our existing memory hotplug interface based on memory blocks is just suoptimal and it doesn't fit new usecases. We should start thinking about how a new v2 api should look like. I am not sure how that should look like but I believe we should be able to express a "device" as whole rather than having a very loosely bound generic memblocks. Anyway this is likely for a longer discussion and a long term plan rather than addressing this particular issue.

Quoted 5 lines
Quoted 1 line

We have that concept with memory groups in the kernel already.

I must have missed that. I will have a look, thanks! Do we have any documentation for that? Memory group is an overloaded term in the kernel.

It's an internal concept so far, the grouping is not exposed to user space.

We have kerneldoc for e.g., "struct memory_group". E.g., from there

"A memory group logically groups memory blocks; each memory block belongs to at most one memory group. A memory group corresponds to a memory device, such as a DIMM or a NUMA node, which spans multiple memory blocks and might even span multiple non-contiguous physical memory ranges."

Quoted 6 lines
Quoted 3 lines

In dax/kmem we register a static memory group. It will be considered one union.

But we still do export those memory blocks and let udev or whoever act on those right? If that is the case then ....

Yes.

Quoted 6 lines

[...]

Quoted 3 lines

daxctl wants to online memory itself. We want to keep that memory offline from a kernel perspective and let daxctl handle it in this case.

Quoted 7 lines
Quoted 2 lines

We have that problem in RHEL where we currently require user space to disable udev rules so daxctl "can win".

... this is the result. Those shouldn't really race. If udev is suppose to see the device then only in its entirity so regular memory block based onlining rules shouldn't even see that memory. Or am I completely missing the picture?

We can't break user space, which relies on individual memory blocks.

So udev or $whatever will right now see individual memory blocks. We could export the group id to user space if that is of any help, but at least for daxctl purposes, it will be sufficient to identify "oh, this was added by dax/kmem" (which we can obtain from /proc/iomem) and say "okay, I'll let user-space deal with it."

Having the whole thing exposed as a unit is not really solving any problems unless I am missing something important.

Signature · 3 lines
Cheers,

David / dhildenb
#8 lore raw
Quoted 65 lines · David Hildenbrand
Quoted 29 lines · Michal Hocko
Quoted 28 lines · David Hildenbrand
Quoted 26 lines · Michal Hocko
Quoted 12 lines · Oscar Salvador

Hi,

Currently, we have several mechanisms to pick a zone for the new memory we are onlining.
Eventually, we will land on zone_for_pfn_range() which will pick the zone.

Two of these mechanisms are 'movable_node' and 'auto-movable' policy. The former will put every single hotpluggled memory in ZONE_MOVABLE (unless we can keep zones contiguous by not doing so), while the latter will put it in ZONA_MOVABLE IFF we are within the established ratio MOVABLE:KERNEL.

Quoted 4 lines

It seems, the later doesn't play well with CXL memory where CXL cards hold really large amounts of memory, making the ratio fail, and since CXL cards must be removed as a unit, it can't be done if any memory block fell within !ZONE_MOVABLE zone.

I suspect this is just an example of how our existing memory hotplug interface based on memory blocks is just suoptimal and it doesn't fit new usecases. We should start thinking about how a new v2 api should look like. I am not sure how that should look like but I believe we should be able to express a "device" as whole rather than having a very loosely bound generic memblocks. Anyway this is likely for a longer discussion and a long term plan rather than addressing this particular issue.

Quoted 5 lines
Quoted 1 line

We have that concept with memory groups in the kernel already.

I must have missed that. I will have a look, thanks! Do we have any documentation for that? Memory group is an overloaded term in the kernel.

It's an internal concept so far, the grouping is not exposed to user space.

We have kerneldoc for e.g., "struct memory_group". E.g., from there

"A memory group logically groups memory blocks; each memory block belongs to at most one memory group. A memory group corresponds to a memory device, such as a DIMM or a NUMA node, which spans multiple memory blocks and might even span multiple non-contiguous physical memory ranges."

Quoted 6 lines
Quoted 3 lines

In dax/kmem we register a static memory group. It will be considered one union.

But we still do export those memory blocks and let udev or whoever act on those right? If that is the case then ....

Yes.

Quoted 6 lines

[...]

Quoted 3 lines

daxctl wants to online memory itself. We want to keep that memory offline from a kernel perspective and let daxctl handle it in this case.

Quoted 7 lines
Quoted 2 lines

We have that problem in RHEL where we currently require user space to disable udev rules so daxctl "can win".

... this is the result. Those shouldn't really race. If udev is suppose to see the device then only in its entirity so regular memory block based onlining rules shouldn't even see that memory. Or am I completely missing the picture?

Quoted 6 lines

We can't break user space, which relies on individual memory blocks.

So udev or $whatever will right now see individual memory blocks. We could export the group id to user space if that is of any help, but at least for daxctl purposes, it will be sufficient to identify "oh, this was added by dax/kmem" (which we can obtain from /proc/iomem) and say "okay, I'll let user-space deal with it."

Having the whole thing exposed as a unit is not really solving any problems unless I am missing something important.

Basically it boils down to:
Who should be responsible for onlining the memory?

As it stands we have two methods:
- user-space as per sysfs attributes
- kernel policy

And to make matters worse, we have two competing user-space programs:
- udev
- daxctl
neither of which is (or can be made) aware of each other. This leads to races and/or inconsistencies.

As we've seen the current kernel policy (cf the 'ratio' discussion) doesn't really fit how users expect CXL to work, so one is tempted to not having the kernel to do the onlining. But then the user is caught in the udev vs daxctl race, requiring awkward cludges on either side.

Can't we make daxctl aware of udev? IE updating daxctl call out to udev and just wait for udev to complete its thing?
At worst we're running into a timeout if some udev rules are garbage, but daxctl will be able to see the final state and we would avoid the need for modifying and/or moving udev rules.
(Which, incidentally, is required on SLES, too :-)

Discussion point for LPC?

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#9 lore raw
Quoted 6 lines · Hannes Reinecke

On 7/28/25 11:10, David Hildenbrand wrote:
And to make matters worse, we have two competing user-space programs:
- udev
- daxctl
neither of which is (or can be made) aware of each other. This leads to races and/or inconsistencies.

Would it help if generic udev memory hotplug rule exclude anything that is dax backed? Is there a way to check for that? Sorry if this is a stupid question.

To me it sounds like daxctl should be the one to online the memory excluseively and udev should just care about regular memory.

Signature · 2 lines
Michal Hocko
SUSE Labs
#10 lore raw
Quoted 11 lines · Michal Hocko
Quoted 6 lines · Hannes Reinecke

On 7/28/25 11:10, David Hildenbrand wrote:
And to make matters worse, we have two competing user-space programs:
- udev
- daxctl
neither of which is (or can be made) aware of each other. This leads to races and/or inconsistencies.

Would it help if generic udev memory hotplug rule exclude anything that is dax backed? Is there a way to check for that? Sorry if this is a stupid question.

Parsing /proc/iomem, it's indicated as "System RAM (kmem)".

Signature · 3 lines
Cheers,

David / dhildenb
#11 lore raw
Quoted 9 lines · David Hildenbrand
Quoted 8 lines · Michal Hocko
Quoted 6 lines · Hannes Reinecke

On 7/28/25 11:10, David Hildenbrand wrote:
And to make matters worse, we have two competing user-space programs:
- udev
- daxctl
neither of which is (or can be made) aware of each other. This leads to races and/or inconsistencies.

Quoted 5 lines
Quoted 3 lines

Would it help if generic udev memory hotplug rule exclude anything that is dax backed? Is there a way to check for that? Sorry if this is a stupid question.

Parsing /proc/iomem, it's indicated as "System RAM (kmem)".

I would rather do it the other way round, and make daxctl aware of udev. In the end, even 'daxctl' uses the sysfs interface to online memory, which really is the territory of udev and can easily be done via udev rules (for static configuration).

Note, we do a similar thing on s/390; the configuration tool there just spits out udev rules.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#12 lore raw
Quoted 13 lines · Hannes Reinecke
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Michal Hocko
Quoted 6 lines · Hannes Reinecke

On 7/28/25 11:10, David Hildenbrand wrote:
And to make matters worse, we have two competing user-space programs:
- udev
- daxctl
neither of which is (or can be made) aware of each other. This leads to races and/or inconsistencies.

Would it help if generic udev memory hotplug rule exclude anything that is dax backed? Is there a way to check for that? Sorry if this is a stupid question.

Quoted 6 lines
Quoted 2 lines

Parsing /proc/iomem, it's indicated as "System RAM (kmem)".

I would rather do it the other way round, and make daxctl aware of udev. In the end, even 'daxctl' uses the sysfs interface to online memory, which really is the territory of udev and can easily be done via udev rules (for static configuration).

udev doesn't really have any context what user space wants to do with the memory and therefore how to online it. Therefore we have (arguably) ugly hacks like auto onlining and movable_ration etc. daxctl can take information from the admin directly and therfore it can do what is needed without further hacks.

Quoted 2 lines

Note, we do a similar thing on s/390; the configuration tool there just spits out udev rules.

Those were easy times when you just need to online memory without any more requirements where it should land.

Signature · 2 lines
Michal Hocko
SUSE Labs
#13 lore raw
Quoted 21 lines · Michal Hocko
Quoted 13 lines · Hannes Reinecke
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Michal Hocko
Quoted 6 lines · Hannes Reinecke

On 7/28/25 11:10, David Hildenbrand wrote:
And to make matters worse, we have two competing user-space programs:
- udev
- daxctl
neither of which is (or can be made) aware of each other. This leads to races and/or inconsistencies.

Would it help if generic udev memory hotplug rule exclude anything that is dax backed? Is there a way to check for that? Sorry if this is a stupid question.

Quoted 6 lines
Quoted 2 lines

Parsing /proc/iomem, it's indicated as "System RAM (kmem)".

I would rather do it the other way round, and make daxctl aware of udev. In the end, even 'daxctl' uses the sysfs interface to online memory, which really is the territory of udev and can easily be done via udev rules (for static configuration).

Quoted 5 lines

udev doesn't really have any context what user space wants to do with the memory and therefore how to online it. Therefore we have (arguably) ugly hacks like auto onlining and movable_ration etc. daxctl can take information from the admin directly and therfore it can do what is needed without further hacks.

Really the only difference between daxctl and everything else is the way the memory is added.

daxctl triggers hotplug of memory synchronously, everything else is asynchronous.

On most systems, the admin (the same one that triggers onlining) could just set the auto-onlining policy accordingly instead of manually onlining memory blocks from user space.

Quoted 6 lines
Quoted 3 lines

Note, we do a similar thing on s/390; the configuration tool there just spits out udev rules.

Those were easy times when you just need to online memory without any more requirements where it should land.

Again, I don't think udev is the future for that.

What I think we (Red Hat) want is a better and easier way to configure the kernel policy.

If you want to control onlining manually, then disable the auto-online policy.

Signature · 3 lines
Cheers,

David / dhildenb
#14 lore raw
In reply to Michal Hocko
Quoted 21 lines · Michal Hocko
Quoted 13 lines · Hannes Reinecke
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Michal Hocko
Quoted 6 lines · Hannes Reinecke

On 7/28/25 11:10, David Hildenbrand wrote:
And to make matters worse, we have two competing user-space programs:
- udev
- daxctl
neither of which is (or can be made) aware of each other. This leads to races and/or inconsistencies.

Would it help if generic udev memory hotplug rule exclude anything that is dax backed? Is there a way to check for that? Sorry if this is a stupid question.

Quoted 6 lines
Quoted 2 lines

Parsing /proc/iomem, it's indicated as "System RAM (kmem)".

I would rather do it the other way round, and make daxctl aware of udev. In the end, even 'daxctl' uses the sysfs interface to online memory, which really is the territory of udev and can easily be done via udev rules (for static configuration).

Quoted 6 lines

udev doesn't really have any context what user space wants to do with the memory and therefore how to online it. Therefore we have (arguably) ugly hacks like auto onlining and movable_ration etc. daxctl can take information from the admin directly and therfore it can do what is needed without further hacks.

Huh?
I thought udev was _all_ about userspace preferences...
We can easily have udev rules onlining memory with whatever policy the user want; the whole point of udev rules is that they are dynamic and can include policy decisions.

Quoted 5 lines
Quoted 2 lines

Note, we do a similar thing on s/390; the configuration tool there just spits out udev rules.

Those were easy times when you just need to online memory without any more requirements where it should land.

Sorry, I don't get that.
udev rules can easily parse any user-space policy, and you can have a policy as detailed as you want.
And each installation can have its own udev rules.
Why wouldn't that work?

(Excluding main memory, obviously. We need memory to execute userspace processes after all).

I do think we're talking past each other...

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#15 lore raw
Quoted 28 lines · Hannes Reinecke
Quoted 21 lines · Michal Hocko
Quoted 13 lines · Hannes Reinecke
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Michal Hocko
Quoted 6 lines · Hannes Reinecke

On 7/28/25 11:10, David Hildenbrand wrote:
And to make matters worse, we have two competing user-space programs:
- udev
- daxctl
neither of which is (or can be made) aware of each other. This leads to races and/or inconsistencies.

Would it help if generic udev memory hotplug rule exclude anything that is dax backed? Is there a way to check for that? Sorry if this is a stupid question.

Quoted 6 lines
Quoted 2 lines

Parsing /proc/iomem, it's indicated as "System RAM (kmem)".

I would rather do it the other way round, and make daxctl aware of udev. In the end, even 'daxctl' uses the sysfs interface to online memory, which really is the territory of udev and can easily be done via udev rules (for static configuration).

Quoted 6 lines

udev doesn't really have any context what user space wants to do with the memory and therefore how to online it. Therefore we have (arguably) ugly hacks like auto onlining and movable_ration etc. daxctl can take information from the admin directly and therfore it can do what is needed without further hacks.

Quoted 5 lines

Huh?
I thought udev was _all_ about userspace preferences...
We can easily have udev rules onlining memory with whatever policy the user want; the whole point of udev rules is that they are dynamic and can include policy decisions.

My experience with memory hotplug and udev doesn't match that. Udev sees memory blocks showing up rather than understanding any concept of what is the memory behind that. So any actual policy is rather hard to define. You would need to backtrack what kind of memory blocks you are seeing and what the initiator could have intended with them.

While this could work reasonably for regular RAM appearing to your system asynchronously (e.g. physical memory plugged in or virtual system getting more memory) when you always want to online it in a certain way I suspect this falls short for synchronous daxctl like usecase where you know what to do with that memory and you can operate on sysfs directly. Udev just makes the life much more complicated for the later IMO.

Signature · 2 lines
Michal Hocko
SUSE Labs
#16 lore raw
Quoted 41 lines · Michal Hocko
Quoted 28 lines · Hannes Reinecke
Quoted 21 lines · Michal Hocko
Quoted 13 lines · Hannes Reinecke
Quoted 12 lines · David Hildenbrand
Quoted 11 lines · Michal Hocko
Quoted 6 lines · Hannes Reinecke

On 7/28/25 11:10, David Hildenbrand wrote:
And to make matters worse, we have two competing user-space programs:
- udev
- daxctl
neither of which is (or can be made) aware of each other. This leads to races and/or inconsistencies.

Would it help if generic udev memory hotplug rule exclude anything that is dax backed? Is there a way to check for that? Sorry if this is a stupid question.

Quoted 6 lines
Quoted 2 lines

Parsing /proc/iomem, it's indicated as "System RAM (kmem)".

I would rather do it the other way round, and make daxctl aware of udev. In the end, even 'daxctl' uses the sysfs interface to online memory, which really is the territory of udev and can easily be done via udev rules (for static configuration).

Quoted 6 lines

udev doesn't really have any context what user space wants to do with the memory and therefore how to online it. Therefore we have (arguably) ugly hacks like auto onlining and movable_ration etc. daxctl can take information from the admin directly and therfore it can do what is needed without further hacks.

Quoted 5 lines

Huh?
I thought udev was _all_ about userspace preferences...
We can easily have udev rules onlining memory with whatever policy the user want; the whole point of udev rules is that they are dynamic and can include policy decisions.

My experience with memory hotplug and udev doesn't match that. Udev sees memory blocks showing up rather than understanding any concept of what is the memory behind that. So any actual policy is rather hard to define. You would need to backtrack what kind of memory blocks you are seeing and what the initiator could have intended with them.

Quoted 6 lines

While this could work reasonably for regular RAM appearing to your system asynchronously (e.g. physical memory plugged in or virtual system getting more memory) when you always want to online it in a certain way I suspect this falls short for synchronous daxctl like usecase where you know what to do with that memory and you can operate on sysfs directly. Udev just makes the life much more complicated for the later IMO.

Not disagreeing with that.
Problem is that memory hotplug is tied to sysfs (and, with that, to udev) such that currently there is no way of _not_ sending uevents (and, consequently, udev interfering).

We could (for any of the 'auto' modes) disable uevent generation via the dev_set_uevent_suppress() thingie.
Or we could teach daxctl to wait for uevents after an action has been triggered.

But the current situation is really daft, requiring the user to move away udev rules in specific situations.

Cheers,

Hannes

Signature · 4 lines
Dr. Hannes Reinecke                  Kernel Storage Architect
[email protected]                                +49 911 74053 688
SUSE Software Solutions GmbH, Frankenstr. 146, 90461 Nürnberg
HRB 36809 (AG Nürnberg), GF: I. Totev, A. McDonald, W. Knoblich
#17 lore raw
In reply to Hannes Reinecke
Quoted 104 lines · Hannes Reinecke
Quoted 77 lines · David Hildenbrand
Quoted 34 lines · Michal Hocko
Quoted 33 lines · David Hildenbrand
Quoted 31 lines · Michal Hocko
Quoted 15 lines · Oscar Salvador

Hi,

Currently, we have several mechanisms to pick a zone for the new memory we are
onlining.
Eventually, we will land on zone_for_pfn_range() which will pick the zone.

Two of these mechanisms are 'movable_node' and 'auto-movable' policy. The former will put every single hotpluggled memory in ZONE_MOVABLE (unless we can keep zones contiguous by not doing so), while the latter
will put it in ZONA_MOVABLE IFF we are within the established ratio MOVABLE:KERNEL.

Quoted 6 lines

It seems, the later doesn't play well with CXL memory where CXL cards hold really
large amounts of memory, making the ratio fail, and since CXL cards must be removed
as a unit, it can't be done if any memory block fell within !ZONE_MOVABLE zone.

I suspect this is just an example of how our existing memory hotplug interface based on memory blocks is just suoptimal and it doesn't fit new usecases. We should start thinking about how a new v2 api should look like. I am not sure how that should look like but I believe we should be able to express a "device" as whole rather than having a very loosely bound generic memblocks. Anyway this is likely for a longer discussion and a long term plan rather than addressing this particular issue.

Quoted 5 lines
Quoted 1 line

We have that concept with memory groups in the kernel already.

I must have missed that. I will have a look, thanks! Do we have any documentation for that? Memory group is an overloaded term in the kernel.

It's an internal concept so far, the grouping is not exposed to user space.

We have kerneldoc for e.g., "struct memory_group". E.g., from there

"A memory group logically groups memory blocks; each memory block belongs to at most one memory group. A memory group corresponds to a memory device, such as a DIMM or a NUMA node, which spans multiple memory blocks and might even span multiple non-contiguous physical memory ranges."

Quoted 6 lines
Quoted 3 lines

In dax/kmem we register a static memory group. It will be considered one union.

But we still do export those memory blocks and let udev or whoever act on those right? If that is the case then ....

Yes.

Quoted 7 lines

[...]

Quoted 4 lines

daxctl wants to online memory itself. We want to keep that memory offline
from a kernel perspective and let daxctl handle it in this case.

Quoted 7 lines
Quoted 2 lines

We have that problem in RHEL where we currently require user space to disable udev rules so daxctl "can win".

... this is the result. Those shouldn't really race. If udev is suppose to see the device then only in its entirity so regular memory block based onlining rules shouldn't even see that memory. Or am I completely missing the picture?

We can't break user space, which relies on individual memory blocks.

Quoted 9 lines

So udev or $whatever will right now see individual memory blocks. We could export the group id to user space if that is of any help, but at least for daxctl purposes, it will be sufficient to identify "oh, this was added by dax/kmem" (which we can obtain from /proc/iomem) and say "okay, I'll let user-space deal with it."

Having the whole thing exposed as a unit is not really solving any problems unless I am missing something important.

Basically it boils down to:
Who should be responsible for onlining the memory?

As it stands we have two methods:
- user-space as per sysfs attributes
- kernel policy

And to make matters worse, we have two competing user-space programs:
- udev
- daxctl
neither of which is (or can be made) aware of each other. This leads to races and/or inconsistencies.

As we've seen the current kernel policy (cf the 'ratio' discussion) doesn't really fit how users expect CXL to work, so one is tempted to not having the kernel to do the onlining. But then the user is caught in the udev vs daxctl race, requiring awkward cludges on either side.

Quoted 6 lines
Quoted 1 line

Can't we make daxctl aware of udev? IE updating daxctl call out to

udev and just wait for udev to complete its thing?
At worst we're running into a timeout if some udev rules are garbage, but daxctl will be able to see the final state and we would avoid the need for modifying and/or moving udev rules.
(Which, incidentally, is required on SLES, too :-)

I will try moving away from udev for memory onlining completely in RHEL -- let's see if I will succeed ;) .

We really want to make use of auto-onlining in the kernel where possible, and do it manually in user space only in a handful of cases (e.g., CXL, standby memory on s390x). Configuring auto-onlining is the nasty bit that still needs to be done manually by the admin, and that's really the nasty bit.

Quoted 2 lines

Discussion point for LPC?

Yes, probably.

Signature · 3 lines
Cheers,

David / dhildenb
#18 lore raw
In reply to David Hildenbrand
Quoted 2 lines · David Hildenbrand
Quoted 1 line · Michal Hocko

On Mon 28-07-25 10:53:08, David Hildenbrand wrote:

[...]

Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

daxctl wants to online memory itself. We want to keep that memory offline from a kernel perspective and let daxctl handle it in this case.

We have that problem in RHEL where we currently require user space to disable udev rules so daxctl "can win".

Quoted 6 lines
Quoted 4 lines

... this is the result. Those shouldn't really race. If udev is suppose to see the device then only in its entirity so regular memory block based onlining rules shouldn't even see that memory. Or am I completely missing the picture?

We can't break user space, which relies on individual memory blocks.

We do have userspace which onlines specific memory blocks and we cannot break that. But do we have any userspace that wants to online CXL like memory (or in general dax like memory) that would need to operate on those memory blocks with that kind of granularity?

In other words what would break if we didn't expose CXL memory through memory blocks in sysfs?

Quoted 8 lines

So udev or $whatever will right now see individual memory blocks. We could export the group id to user space if that is of any help, but at least for daxctl purposes, it will be sufficient to identify "oh, this was added by dax/kmem" (which we can obtain from /proc/iomem) and say "okay, I'll let user-space deal with it."

Having the whole thing exposed as a unit is not really solving any problems unless I am missing something important.

If we need to handle that thing as whole we should have an interface that allows for that. Per block breakdown doesn't really help anything. It just makes the whole problem much more complex.

Signature · 2 lines
Michal Hocko
SUSE Labs
#19 lore raw
Quoted 15 lines · Michal Hocko
Quoted 2 lines · David Hildenbrand
Quoted 1 line · Michal Hocko

On Mon 28-07-25 10:53:08, David Hildenbrand wrote:

[...]

Quoted 11 lines
Quoted 10 lines
Quoted 5 lines

daxctl wants to online memory itself. We want to keep that memory offline from a kernel perspective and let daxctl handle it in this case.

We have that problem in RHEL where we currently require user space to disable udev rules so daxctl "can win".

... this is the result. Those shouldn't really race. If udev is suppose to see the device then only in its entirity so regular memory block based onlining rules shouldn't even see that memory. Or am I completely missing the picture?

Quoted 6 lines
Quoted 1 line

We can't break user space, which relies on individual memory blocks.

We do have userspace which onlines specific memory blocks and we cannot break that. But do we have any userspace that wants to online CXL like memory (or in general dax like memory) that would need to operate on those memory blocks with that kind of granularity?

I'm afraid that ship has sailed.

Quoted 3 lines

In other words what would break if we didn't expose CXL memory through memory blocks in sysfs?

I think the whole libdaxctl handling for onlining memory is based on that.

Signature · 3 lines
Cheers,

David / dhildenb
#20 lore raw
Quoted 23 lines · David Hildenbrand
Quoted 15 lines · Michal Hocko
Quoted 2 lines · David Hildenbrand
Quoted 1 line · Michal Hocko

On Mon 28-07-25 10:53:08, David Hildenbrand wrote:

[...]

Quoted 11 lines
Quoted 10 lines
Quoted 5 lines

daxctl wants to online memory itself. We want to keep that memory offline from a kernel perspective and let daxctl handle it in this case.

We have that problem in RHEL where we currently require user space to disable udev rules so daxctl "can win".

... this is the result. Those shouldn't really race. If udev is suppose to see the device then only in its entirity so regular memory block based onlining rules shouldn't even see that memory. Or am I completely missing the picture?

Quoted 6 lines
Quoted 1 line

We can't break user space, which relies on individual memory blocks.

We do have userspace which onlines specific memory blocks and we cannot break that. But do we have any userspace that wants to online CXL like memory (or in general dax like memory) that would need to operate on those memory blocks with that kind of granularity?

Quoted 8 lines

I'm afraid that ship has sailed.

Quoted 3 lines

In other words what would break if we didn't expose CXL memory through memory blocks in sysfs?

I think the whole libdaxctl handling for onlining memory is based on that.

Sorry, forgot to add a pointer:

https://github.com/pmem/ndctl/blob/main/daxctl/lib/libdaxctl.c

Signature · 3 lines
Cheers,

David / dhildenb
#22 lore raw
Quoted 10 lines · Michal Hocko

On Mon 28-07-25 14:27:29, David Hildenbrand wrote:
[...]

Quoted 6 lines
Quoted 2 lines

I think the whole libdaxctl handling for onlining memory is based on that.

Sorry, forgot to add a pointer:

https://github.com/pmem/ndctl/blob/main/daxctl/lib/libdaxctl.c

Thanks for the pointer! I will have a look.

In particular daxctl_memory_op(), used to implement stuff like

daxctl online-memory
daxctl offline-memory
daxctl reconfigure-device (--no-online, --no-movable)

I am not sure if they also offline memory automatically before disabling a device.

But in essence, they want to to everything automatically as possible.

Signature · 3 lines
Cheers,

David / dhildenb
#23 lore raw
In reply to David Hildenbrand
Quoted 1 line · David Hildenbrand

On 28.07.25 14:17, Michal Hocko wrote:

[...]

Quoted 4 lines
Quoted 2 lines

In other words what would break if we didn't expose CXL memory through memory blocks in sysfs?

I think the whole libdaxctl handling for onlining memory is based on that.

I am not familiar with libdaxctl so bear with me. What exactly from the existing sysfs interface does it need?

Signature · 2 lines
Michal Hocko
SUSE Labs