mm/memory_hotplug: Make unpopulated zones PCP structures unreachable during hot remove
Subsystems (4)
PATCH 1/1 mm/memory_hotplug: Make unpopulated zones PCP structures unreachable during hot remove
zone_pcp_reset allegedly protects against a race with drain_pages using local_irq_save but this is bogus. local_irq_save only operates on the local CPU. If memory hotplug is running on CPU A and drain_pages is running on CPU B, disabling IRQs on CPU A does not affect CPU B and offers no protection.
This patch reorders memory hotremove such that the PCP structures relevant to the zone are no longer reachable by the time the structures are freed. With this reordering, no protection is required to prevent a use-after-free and the IRQs can be left enabled. zone_pcp_reset is renamed to zone_pcp_destroy to make it clear that the per-cpu structures are deleted when the function returns.
--- mm/internal.h | 2 +- mm/memory_hotplug.c | 10 +++++++--- mm/page_alloc.c | 22 ++++++++++++++++------ 3 files changed, 24 insertions(+), 10 deletions(-)
mm/internal.h +1 -1
@@ -203,7 +203,7 @@ extern void free_unref_page(struct page *page);extern void free_unref_page_list (struct list_head *list);extern void zone_pcp_update (struct zone *zone); -extern void zone_pcp_reset (struct zone *zone); +extern void zone_pcp_destroy (struct zone *zone);extern void zone_pcp_disable (struct zone *zone);extern void zone_pcp_enable (struct zone *zone);
mm/memory_hotplug.c +7 -3
@@ -1687,12 +1687,16 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages) zone->nr_isolate_pageblock -= nr_pages / pageblock_nr_pages;spin_unlock_irqrestore (&zone->lock, flags); -zone_pcp_enable (zone); -/* removal success */ adjust_managed_page_count (pfn_to_page (start_pfn), -nr_pages); zone->present_pages -= nr_pages; +/* +* Restore PCP after managed pages has been updated. Unpopulated +* zones PCP structures will remain unusable. +*/ +zone_pcp_enable (zone); +pgdat_resize_lock (zone->zone_pgdat, &flags); zone->zone_pgdat->node_present_pages -= nr_pages;pgdat_resize_unlock (zone->zone_pgdat, &flags);
@@ -1700,8 +1704,8 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages)init_per_zone_wmark_min ();if (!populated_zone (zone)) { -zone_pcp_reset (zone);build_all_zonelists (NULL ); +zone_pcp_destroy (zone); }else zone_pcp_update (zone);
mm/page_alloc.c +16 -6
@@ -8946,18 +8946,29 @@ void zone_pcp_disable(struct zone *zone)void zone_pcp_enable (struct zone *zone) { -__zone_set_pageset_high_and_batch (zone, zone->pageset_high, zone->pageset_batch); +/* +* If the zone is populated, restore the high and batch counts. +* If unpopulated, leave the high and batch count as 0 and 1 +* respectively as done by zone_pcp_disable. The per-cpu +* structures will later be freed by zone_pcp_destroy. +*/ +if (populated_zone (zone)) +__zone_set_pageset_high_and_batch (zone, zone->pageset_high, zone->pageset_batch); +mutex_unlock (&pcp_batch_high_lock); } -void zone_pcp_reset (struct zone *zone) +/* +* Called when a zone has been hot-removed. At this point, the PCP has been +* drained, disabled and the zone is removed from the zonelists so the +* structures are no longer in use. PCP was disabled/drained by +* zone_pcp_disable. This function will drain any remaining vmstat deltas. +*/ +void zone_pcp_destroy (struct zone *zone) { -unsigned long flags;int cpu;struct per_cpu_pageset *pset; -/* avoid races with drain_pages() */ -local_irq_save (flags);if (zone->pageset != &boot_pageset) {for_each_online_cpu (cpu) { pset =per_cpu_ptr (zone->pageset, cpu);
@@ -8966,7 +8977,6 @@ void zone_pcp_reset(struct zone *zone)free_percpu (zone->pageset); zone->pageset = &boot_pageset; } -local_irq_restore (flags); }#ifdef CONFIG_MEMORY_HOTREMOVE
Michal Hocko
reviewerQuoted 5 lines · Mel Gorman
zone_pcp_resetallegedly protects against a race withdrain_pagesusinglocal_irq_savebut this is bogus.local_irq_saveonly operates on the local CPU. If memory hotplug is running on CPU A anddrain_pagesis running on CPU B, disabling IRQs on CPU A does not affect CPU B and offers no protection.
Yes, the synchronization aspect is bogus indeed.
Quoted 6 lines
This patch reorders memory hotremove such that the PCP structures relevant to the zone are no longer reachable by the time the structures are freed. With this reordering, no protection is required to prevent a use-after-free and the IRQs can be left enabled.
zone_pcp_resetis renamed tozone_pcp_destroyto make it clear that the per-cpu structures are deleted when the function returns.
Wouldn't it be much easier to simply not destroy/reset pcp of an empty zone at all? The whole point of this exercise seems to be described in 340175b7d14d5. setup_zone_pageset can check for an already allocated pcp and simply reinitialize it.
Signature · 2 lines
Michal Hocko SUSE Labs
Michal Hocko
reviewerQuoted 16 lines · Michal Hocko
Quoted 5 lines · Mel Gorman
zone_pcp_resetallegedly protects against a race withdrain_pagesusinglocal_irq_savebut this is bogus.local_irq_saveonly operates on the local CPU. If memory hotplug is running on CPU A anddrain_pagesis running on CPU B, disabling IRQs on CPU A does not affect CPU B and offers no protection.Yes, the synchronization aspect is bogus indeed.
Quoted 6 lines
This patch reorders memory hotremove such that the PCP structures relevant to the zone are no longer reachable by the time the structures are freed. With this reordering, no protection is required to prevent a use-after-free and the IRQs can be left enabled.
zone_pcp_resetis renamed tozone_pcp_destroyto make it clear that the per-cpu structures are deleted when the function returns.
Quoted 4 lines
Wouldn't it be much easier to simply not destroy/reset pcp of an empty zone at all? The whole point of this exercise seems to be described in
340175b7d14d5.setup_zone_pagesetcan check for an already allocated pcp and simply reinitialize it.
I meant this
mm/memory_hotplug.c +0 -1
@@ -1810,7 +1818,6 @@ int __ref offline_pages(unsigned long start_pfn, unsigned long nr_pages)init_per_zone_wmark_min ();if (!populated_zone (zone)) { -zone_pcp_reset (zone);build_all_zonelists (NULL ); }else zone_pcp_update (zone);
mm/page_alloc.c +7 -20
@@ -6496,7 +6496,13 @@ void __meminit setup_zone_pageset(struct zone *zone)struct per_cpu_pageset *p;int cpu; - zone->pageset = alloc_percpu(struct per_cpu_pageset); +/* +* zone could have gone completely offline during memory hotplug +* when the pgdat is left behind for simplicity. On a next onlining +* we do not need to reallocate pcp state. +*/ +if (!zone->pageset) + zone->pageset = alloc_percpu(struct per_cpu_pageset);for_each_possible_cpu (cpu) { p =per_cpu_ptr (zone->pageset, cpu);pageset_init (p);
@@ -8803,25 +8809,6 @@ void zone_pcp_enable(struct zone *zone)mutex_unlock (&pcp_batch_high_lock); } -void zone_pcp_reset (struct zone *zone) -{ -unsigned long flags; -int cpu; -struct per_cpu_pageset *pset; - -/* avoid races with drain_pages() */ -local_irq_save (flags); -if (zone->pageset != &boot_pageset) { -for_each_online_cpu (cpu) { - pset =per_cpu_ptr (zone->pageset, cpu); -drain_zonestat (zone, pset); - } -free_percpu (zone->pageset); - zone->pageset = &boot_pageset; - } -local_irq_restore (flags); -} -#ifdef CONFIG_MEMORY_HOTREMOVE /* * All pages in the range must be in a single zone, must not contain holes,
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 17 lines · Michal Hocko
Quoted 16 lines · Michal Hocko
Quoted 5 lines · Mel Gorman
zone_pcp_resetallegedly protects against a race withdrain_pagesusinglocal_irq_savebut this is bogus.local_irq_saveonly operates on the local CPU. If memory hotplug is running on CPU A anddrain_pagesis running on CPU B, disabling IRQs on CPU A does not affect CPU B and offers no protection.Yes, the synchronization aspect is bogus indeed.
Quoted 6 lines
This patch reorders memory hotremove such that the PCP structures relevant to the zone are no longer reachable by the time the structures are freed. With this reordering, no protection is required to prevent a use-after-free and the IRQs can be left enabled.
zone_pcp_resetis renamed tozone_pcp_destroyto make it clear that the per-cpu structures are deleted when the function returns.
Quoted 7 lines
Quoted 4 lines
Wouldn't it be much easier to simply not destroy/reset pcp of an empty zone at all? The whole point of this exercise seems to be described in
340175b7d14d5.setup_zone_pagesetcan check for an already allocated pcp and simply reinitialize it.I meant this
It might be simplier but if the intention is to free as much memory as possible during hot-remove, it seems wasteful to leave the per-cpu structures behind if we do not have to. If a problem with my patch can be spotted then I'm happy to go with an alternative fix but there are two minor issues with your proposed fix.
Quoted 6 lines
mm/page_alloc.c +0
-0@@ -6496,7 +6496,13 @@ void __meminit setup_zone_pageset(struct zone *zone)struct per_cpu_pageset *p;
mm/page_alloc.c void __meminit setup_zone_pageset(struct zone *zone) · line 6505
Quoted 10 lines
mm/page_alloc.c +7
-1int cpu; - zone->pageset = alloc_percpu(struct per_cpu_pageset); +/* +* zone could have gone completely offline during memory hotplug +* when the pgdat is left behind for simplicity. On a next onlining +* we do not need to reallocate pcp state. +*/ +if (!zone->pageset) + zone->pageset = alloc_percpu(struct per_cpu_pageset);
Should be "if (zone->pageset != &boot_pageset)" ?
Quoted 16 lines
mm/page_alloc.c +0
-9for_each_possible_cpu (cpu) { p =per_cpu_ptr (zone->pageset, cpu);pageset_init (p);@@ -8803,25 +8809,6 @@ void zone_pcp_enable(struct zone *zone)mutex_unlock (&pcp_batch_high_lock); } -void zone_pcp_reset (struct zone *zone) -{ -unsigned long flags; -int cpu; -struct per_cpu_pageset *pset; - -/* avoid races with drain_pages() */ -local_irq_save (flags); -if (zone->pageset != &boot_pageset) {
mm/page_alloc.c void zone_pcp_enable(struct zone *zone) · line 8824
Quoted 10 lines
mm/page_alloc.c +0
-10-for_each_online_cpu (cpu) { - pset =per_cpu_ptr (zone->pageset, cpu); -drain_zonestat (zone, pset); - } -free_percpu (zone->pageset); - zone->pageset = &boot_pageset; - } -local_irq_restore (flags); -} -
zone_pcp_reset still needs to exist to drain the remaining vmstats or it'll break 5a883813845a ("memory-hotplug: fix zone stat mismatch").
Signature · 2 lines
Mel Gorman SUSE Labs
Michal Hocko
reviewerQuoted 23 lines · Mel Gorman
Quoted 22 lines · Michal Hocko
Quoted 20 lines · Michal Hocko
Quoted 5 lines · Mel Gorman
zone_pcp_resetallegedly protects against a race withdrain_pagesusinglocal_irq_savebut this is bogus.local_irq_saveonly operates on the local CPU. If memory hotplug is running on CPU A anddrain_pagesis running on CPU B, disabling IRQs on CPU A does not affect CPU B and offers no protection.Yes, the synchronization aspect is bogus indeed.
Quoted 6 lines
This patch reorders memory hotremove such that the PCP structures relevant to the zone are no longer reachable by the time the structures are freed. With this reordering, no protection is required to prevent a use-after-free and the IRQs can be left enabled.
zone_pcp_resetis renamed tozone_pcp_destroyto make it clear that the per-cpu structures are deleted when the function returns.Wouldn't it be much easier to simply not destroy/reset pcp of an empty zone at all? The whole point of this exercise seems to be described in
340175b7d14d5.setup_zone_pagesetcan check for an already allocated pcp and simply reinitialize it.
Quoted 6 lines
Quoted 2 lines
I meant this
It might be simplier but if the intention is to free as much memory as possible during hot-remove, it seems wasteful to leave the per-cpu structures behind if we do not have to.
We do leave the whole pgdat behind. I do not think pagesets really do matter.
Quoted 3 lines
If a problem with my patch can
be spotted then I'm happy to go with an alternative fix but there are two minor issues with your proposed fix.
I will not insist but this code has proven to bitrot and I just find it much simpler to drop it altogether rather than conserve it in some form. Not something I would insist though.
Quoted 8 lines
Quoted 8 lines
mm/page_alloc.c +0
-0@@ -6496,7 +6496,13 @@ void __meminit setup_zone_pageset(struct zone *zone)struct per_cpu_pageset *p;int cpu;
Quoted 10 lines
mm/page_alloc.c void __meminit setup_zone_pageset(struct zone *zone) · line 6505
Quoted 8 lines
mm/page_alloc.c +7
-1- zone->pageset = alloc_percpu(struct per_cpu_pageset); +/* +* zone could have gone completely offline during memory hotplug +* when the pgdat is left behind for simplicity. On a next onlining +* we do not need to reallocate pcp state. +*/ +if (!zone->pageset) + zone->pageset = alloc_percpu(struct per_cpu_pageset);Should be "if (
zone->pageset!=&boot_pageset)" ?
Memory hotplug really wants the NULL
check. it doesn't use boot_pageset (if we drop rest to boot_pageset). But you are right that the boot time initialization first sets boot_pageset (zone_pcp_init) and initializes real pagesets later (setup_per_cpu_pageset). But this can be handled at the memory hotplug layer I believe
mm/memory_hotplug.c +2 -1
@@ -883,7 +883,8 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages,*/ if (!populated_zone (zone)) { need_zonelists_rebuild =1 ; -setup_zone_pageset (zone); +if (!zone->pageset) +setup_zone_pageset (zone); }
Quoted 22 lines
Quoted 22 lines
mm/page_alloc.c +0
-15for_each_possible_cpu (cpu) { p =per_cpu_ptr (zone->pageset, cpu);pageset_init (p);@@ -8803,25 +8809,6 @@ void zone_pcp_enable(struct zone *zone)mutex_unlock (&pcp_batch_high_lock); } -void zone_pcp_reset (struct zone *zone) -{ -unsigned long flags; -int cpu; -struct per_cpu_pageset *pset; - -/* avoid races with drain_pages() */ -local_irq_save (flags); -if (zone->pageset != &boot_pageset) { -for_each_online_cpu (cpu) { - pset =per_cpu_ptr (zone->pageset, cpu); -drain_zonestat (zone, pset); - } -free_percpu (zone->pageset); - zone->pageset = &boot_pageset;
Quoted 8 lines
mm/page_alloc.c void zone_pcp_enable(struct zone *zone) · line 8824
Quoted 4 lines
mm/page_alloc.c +0
-4- } -local_irq_restore (flags); -} -
zone_pcp_resetstill needs to exist to drain the remaining vmstats or it'll break5a883813845a("memory-hotplug: fix zone stat mismatch").
Are you sure we are reseting vmstats in the hotremove. I do not see anything like that. Maybe this was needed at the time. I will double check.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 17 lines · Michal Hocko
Quoted 17 lines
Quoted 16 lines
Quoted 9 lines
Yes, the synchronization aspect is bogus indeed.
Quoted 6 lines
This patch reorders memory hotremove such that the PCP structures relevant to the zone are no longer reachable by the time the structures are freed. With this reordering, no protection is required to prevent a use-after-free and the IRQs can be left enabled.
zone_pcp_resetis renamed tozone_pcp_destroyto make it clear that the per-cpu structures are deleted when the function returns.Quoted 4 lines
Wouldn't it be much easier to simply not destroy/reset pcp of an empty zone at all? The whole point of this exercise seems to be described in
340175b7d14d5.setup_zone_pagesetcan check for an already allocated pcp and simply reinitialize it.I meant this
Quoted 7 lines
Quoted 3 lines
It might be simplier but if the intention is to free as much memory as possible during hot-remove, it seems wasteful to leave the per-cpu structures behind if we do not have to.
We do leave the whole pgdat behind. I do not think pagesets really do matter.
Probably not given that zone and node hotplug is fragile to say the least with a low success rate. Still, it seems wasteful if we can preserve the freeing part and the alternative fix is more subtle than it appears.
Quoted 39 lines
Quoted 3 lines
If a problem with my patch can
be spotted then I'm happy to go with an alternative fix but there are two minor issues with your proposed fix.I will not insist but this code has proven to bitrot and I just find it much simpler to drop it altogether rather than conserve it in some form. Not something I would insist though.
Quoted 8 lines
Quoted 8 lines
mm/page_alloc.c +0
-0@@ -6496,7 +6496,13 @@ void __meminit setup_zone_pageset(struct zone *zone)struct per_cpu_pageset *p;int cpu;Quoted 10 lines
mm/page_alloc.c void __meminit setup_zone_pageset(struct zone *zone) · line 6505
Quoted 8 lines
mm/page_alloc.c +7
-1- zone->pageset = alloc_percpu(struct per_cpu_pageset); +/* +* zone could have gone completely offline during memory hotplug +* when the pgdat is left behind for simplicity. On a next onlining +* we do not need to reallocate pcp state. +*/ +if (!zone->pageset) + zone->pageset = alloc_percpu(struct per_cpu_pageset);Should be "if (
zone->pageset!=&boot_pageset)" ?Memory hotplug really wants the NULL
check. it doesn't useboot_pageset(if we drop rest toboot_pageset). But you are right that the boot time initialization first setsboot_pageset(zone_pcp_init) and initializes real pagesets later (setup_per_cpu_pageset). But this can be handled at the memory hotplug layer I believemm/memory_hotplug.c +0
-0
mm/memory_hotplug.c int __ref online_pages(unsigned long pfn, unsigned long nr_pages, · line 889
Quoted 8 lines
mm/memory_hotplug.c +2
-1@@ -883,7 +883,8 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages,*/ if (!populated_zone (zone)) { need_zonelists_rebuild =1 ; -setup_zone_pageset (zone); +if (!zone->pageset) +setup_zone_pageset (zone); }
I think this will be fragile because it makes assumptions on how boot works -- specifically that zone->pageset is left alone for unpopulated zones. This will only work for zones in unpopulated *nodes* during boot because of this sequence
free_area_init_core(pgdat)
for (j = 0; j < MAX_NR_ZONES; j++) {
...
zone_init_internals(zone)
}
-> zone_init_internals(zone)
-> zone_pcp_init(zone)
zone->pageset = &boot_pageset;I think a NULL check will mean that hotplugging a previously unpopulated zone will run the risk of using boot_pageset. That will likely explode because IRQs disabled does not protect boot_pageset, it only works early in boot or during the initial hotplug when it's not visible yet.
Quoted 31 lines
mm/memory_hotplug.c +0
-0> > for_each_possible_cpu(cpu) {Quoted 21 lines
Quoted 21 lines
mm/page_alloc.c +0
-15p =per_cpu_ptr (zone->pageset, cpu);pageset_init (p);@@ -8803,25 +8809,6 @@ void zone_pcp_enable(struct zone *zone)mutex_unlock (&pcp_batch_high_lock); } -void zone_pcp_reset (struct zone *zone) -{ -unsigned long flags; -int cpu; -struct per_cpu_pageset *pset; - -/* avoid races with drain_pages() */ -local_irq_save (flags); -if (zone->pageset != &boot_pageset) { -for_each_online_cpu (cpu) { - pset =per_cpu_ptr (zone->pageset, cpu); -drain_zonestat (zone, pset); - } -free_percpu (zone->pageset); - zone->pageset = &boot_pageset;Quoted 8 lines
mm/page_alloc.c void zone_pcp_enable(struct zone *zone) · line 8824
Quoted 4 lines
mm/page_alloc.c +0
-4- } -local_irq_restore (flags); -} -
zone_pcp_resetstill needs to exist to drain the remaining vmstats or it'll break5a883813845a("memory-hotplug: fix zone stat mismatch").
Quoted 3 lines
Are you sure we are reseting vmstats in the hotremove. I do not see anything like that. Maybe this was needed at the time. I will double check.
zone_pcp_reset calls drain_zonestat to apply the per-cpu vmstat deltas to the atomic per-zone and global stats.
If anything, the minimal "fix" is to simply delete IRQ disable/enable on the grounds that IRQs protect nothing and assume the existing hotplug paths guarantees the PCP cannot be used after zone_pcp_enable(). That should be the case already because all the pages have been freed and there is nothing to even put into the PCPs but I worried that the PCP structure itself might still be reachable even if it's useless which is why I freed the structure once they could not be reached via zonelists.
Signature · 2 lines
Mel Gorman SUSE Labs
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
zone_pcp_resetstill needs to exist to drain the remaining vmstats or it'll break5a883813845a("memory-hotplug: fix zone stat mismatch").Are you sure we are reseting vmstats in the hotremove. I do not see anything like that. Maybe this was needed at the time. I will double check.
Quoted 6 lines
zone_pcp_resetcallsdrain_zonestatto apply the per-cpu vmstat deltas to the atomic per-zone and global stats.If anything, the minimal "fix" is to simply delete IRQ disable/enable on the grounds that IRQs protect nothing and assume the existing hotplug paths guarantees the PCP cannot be used after
zone_pcp_enable(). That
^ that sounds sane to me
Quoted 5 lines
should be the case already because all the pages have been freed and there is nothing to even put into the PCPs but I worried that the PCP structure itself might still be reachable even if it's useless which is why I freed the structure once they could not be reached via zonelists.
Signature · 3 lines
Thanks, David / dhildenb
Michal Hocko
reviewerOn Fri 09-04-21 16:12:59, Mel Gorman wrote:
[...]
Quoted 7 lines
If anything, the minimal "fix" is to simply delete IRQ disable/enable on the grounds that IRQs protect nothing and assume the existing hotplug paths guarantees the PCP cannot be used after
zone_pcp_enable(). That should be the case already because all the pages have been freed and there is nothing to even put into the PCPs but I worried that the PCP structure itself might still be reachable even if it's useless which is why I freed the structure once they could not be reached via zonelists.
OK. Let's do that for now and I will put a follow up on my todo list.
Thanks!
Signature · 2 lines
Michal Hocko SUSE Labs