Skip to contentSkip to conversation

mm/memory_hotplug: Shrink zones before removing memory

PATCH Applied David Hildenbrand 9 participants 75 messages Oct 6, 2019 – Feb 5, 2020

Subsystems (10)
Memory management x86 PowerPC arm64 +6 more subsystems
lore
Share & tools
lore
#0 lore raw

v6 cover mm/memory_hotplug: Shrink zones before removing memory

This series fixes the access of uninitialized memmaps when shrinking zones/nodes and when removing memory. Also, it contains all fixes for crashes that can be triggered when removing certain namespace using memunmap_pages() - ZONE_DEVICE, reported by Aneesh.

We stop trying to shrink ZONE_DEVICE, as it's buggy, fixing it would be more involved (we don't have SECTION_IS_ONLINE as an indicator), and shrinking is only of limited use (set_zone_contiguous() cannot detect the ZONE_DEVICE as contiguous).

We continue shrinking !ZONE_DEVICE zones, however, I reduced the amount of code to a minimum. Shrinking is especially necessary to keep zone->contiguous set where possible, especially, on memory unplug of DIMMs at zone boundaries.

--------------------------------------------------------------------------

Zones are now properly shrunk when offlining memory blocks or when onlining failed. This allows to properly shrink zones on memory unplug even if the separate memory blocks of a DIMM were onlined to different zones or re-onlined to a different zone after offlining.

Example:

:/# cat /proc/zoneinfo
Node 1, zone  Movable
        spanned  0
        present  0
        managed  0
:/# echo "online_movable" > /sys/devices/system/memory/memory41/state
:/# echo "online_movable" > /sys/devices/system/memory/memory43/state
:/# cat /proc/zoneinfo
Node 1, zone  Movable
        spanned  98304
        present  65536
        managed  65536
:/# echo 0 > /sys/devices/system/memory/memory43/online
:/# cat /proc/zoneinfo
Node 1, zone  Movable
        spanned  32768
        present  32768
        managed  32768
:/# echo 0 > /sys/devices/system/memory/memory41/online
:/# cat /proc/zoneinfo
Node 1, zone  Movable
        spanned  0
        present  0
        managed  0

--------------------------------------------------------------------------

I tested this with DIMMs on x86. I didn't test the ZONE_DEVICE part.

v4 -> v6:
- "mm/memunmap: Don't access uninitialized memmap in memunmap_pages()" -- Minimize code changes, rephrase subject and description
- "mm/memory_hotplug: Don't access uninitialized memmaps in shrink_zone_span()" -- Add ifdef to make it compile without ZONE_DEVICE

v4 -> v5:
- "mm/memory_hotplug: Don't access uninitialized memmaps in shrink_zone_span()" -- Add more details why ZONE_DEVICE is special
- Include two patches from Aneesh
-- "mm/memunmap: Use the correct start and end pfn when removing pages
from zone"
-- "mm/memmap_init: Update variable name in memmap_init_zone"

v3 -> v4:
- Drop "mm/memremap: Get rid of memmap_init_zone_device()"
-- As Alexander noticed, it was messy either way
- Drop "mm/memory_hotplug: Exit early in __remove_pages() on BUGs"
- Drop "mm: Exit early in set_zone_contiguous() if already contiguous"
- Drop "mm/memory_hotplug: Optimize zone shrinking code when checking for holes"
- Merged "mm/memory_hotplug: Remove pages from a zone before removing memory" and "mm/memory_hotplug: Remove zone parameter from __remove_pages()" into "mm/memory_hotplug: Shrink zones when offlining memory"
- Added "mm/memory_hotplug: Poison memmap in remove_pfn_range_from_zone()"
- Stop shrinking ZONE_DEVICE
- Reshuffle patches, moving all fixes to the front. Add Fixes: tags.
- Change subject/description of various patches
- Minor changes (too many to mention)

Cc Aneesh Kumar K.V <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Dan Williams <[email protected]>
Cc Michal Hocko <[email protected]>

Aneesh Kumar K.V (2):
mm/memunmap: Don't access uninitialized memmap in memunmap_pages() mm/memmap_init: Update variable name in memmap_init_zone

David Hildenbrand (8):
mm/memory_hotplug: Don't access uninitialized memmaps in
shrink_pgdat_span()
mm/memory_hotplug: Don't access uninitialized memmaps in
shrink_zone_span()
mm/memory_hotplug: Shrink zones when offlining memory mm/memory_hotplug: Poison memmap in remove_pfn_range_from_zone() mm/memory_hotplug: We always have a zone in
find_(smallest|biggest)_section_pfn
mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span() mm/memory_hotplug: Drop local variables in shrink_zone_span() mm/memory_hotplug: Cleanup __remove_pages()

 arch/arm64/mm/mmu.c            |   4 +-
 arch/ia64/mm/init.c            |   4 +-
 arch/powerpc/mm/mem.c          |   3 +-
 arch/s390/mm/init.c            |   4 +-
 arch/sh/mm/init.c              |   4 +-
 arch/x86/mm/init_32.c          |   4 +-
 arch/x86/mm/init_64.c          |   4 +-
 include/linux/memory_hotplug.h |   7 +-
 mm/memory_hotplug.c            | 186 ++++++++++++---------------------
 mm/memremap.c                  |  13 ++-
 mm/page_alloc.c                |   8 +-
 11 files changed, 90 insertions(+), 151 deletions(-)
Signature · 1 line
2.21.0
#1 lore raw

PATCH v6 1/10 mm/memunmap: Don't access uninitialized memmap in memunmap_pages()

From: "Aneesh Kumar K.V" <[email protected]>

With an altmap, the memmap falling into the reserved altmap space are not initialized and, therefore, contain a garbage NID and a garbage zone. Make sure to read the NID/zone from a memmap that was initialzed.

This fixes a kernel crash that is observed when destroying a namespace:

[   81.356173] kernel BUG at include/linux/mm.h:1107!
cpu 0x1: Vector: 700 (Program Check) at [c000000274087890]
    pc: c0000000004b9728: memunmap_pages+0x238/0x340
    lr: c0000000004b9724: memunmap_pages+0x234/0x340
...
    pid   = 3669, comm = ndctl

kernel BUG at include/linux/mm.h:1107!
[c000000274087ba0] c0000000009e3500 devm_action_release+0x30/0x50 [c000000274087bc0] c0000000009e4758 release_nodes+0x268/0x2d0 [c000000274087c30] c0000000009dd144 device_release_driver_internal+0x174/0x240 [c000000274087c70] c0000000009d9dfc unbind_store+0x13c/0x190 [c000000274087cb0] c0000000009d8a24 drv_attr_store+0x44/0x60 [c000000274087cd0] c0000000005a7470 sysfs_kf_write+0x70/0xa0 [c000000274087d10] c0000000005a5cac kernfs_fop_write+0x1ac/0x290 [c000000274087d60] c0000000004be45c __vfs_write+0x3c/0x70 [c000000274087d80] c0000000004c26e4 vfs_write+0xe4/0x200 [c000000274087dd0] c0000000004c2a6c ksys_write+0x7c/0x140 [c000000274087e20] c00000000000bbd0 system_call+0x5c/0x68

Cc Dan Williams <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Ira Weiny <[email protected]>
[ minimze code changes, rephrase description ]
---
 mm/memremap.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
mm/memremap.c +7 -4
index 557e53c6fb46..8c2fb44c3b4d 100644
@@ -123,6 +123,7 @@ static void dev_pagemap_cleanup(struct dev_pagemap *pgmap)
 void memunmap_pages(struct dev_pagemap *pgmap)
 {
 	struct resource *res = &pgmap->res;
+	struct page *first_page;
 	unsigned long pfn;
 	int nid;
 
@@ -131,14 +132,16 @@ void memunmap_pages(struct dev_pagemap *pgmap)
 		put_page(pfn_to_page(pfn));
 	dev_pagemap_cleanup(pgmap);
 
+	/* make sure to access a memmap that was actually initialized */
+	first_page = pfn_to_page(pfn_first(pgmap));
+
 	/* pages are dead and unused, undo the arch mapping */
-	nid = page_to_nid(pfn_to_page(PHYS_PFN(res->start)));
+	nid = page_to_nid(first_page);
 
 	mem_hotplug_begin();
 	if (pgmap->type = MEMORY_DEVICE_PRIVATE) {
-		pfn = PHYS_PFN(res->start);
-		__remove_pages(page_zone(pfn_to_page(pfn)), pfn,
-				 PHYS_PFN(resource_size(res)), NULL);
+		__remove_pages(page_zone(first_page), PHYS_PFN(res->start),
+			       PHYS_PFN(resource_size(res)), NULL);
 	} else {
 		arch_remove_memory(nid, res->start, resource_size(res),
 				pgmap_altmap(pgmap));
Signature · 1 line
2.21.0
#2 lore raw

Re: [PATCH v6 01/10] mm/memunmap: Don't access uninitialized memmap in memunmap_pages()

Hello David,

patch 05/10 is missing in the patch series.

Quoted 52 lines · David Hildenbrand

From: "Aneesh Kumar K.V" <[email protected]>

With an altmap, the memmap falling into the reserved altmap space are not initialized and, therefore, contain a garbage NID and a garbage zone. Make sure to read the NID/zone from a memmap that was initialzed.

This fixes a kernel crash that is observed when destroying a namespace:

[   81.356173] kernel BUG at include/linux/mm.h:1107!
cpu 0x1: Vector: 700 (Program Check) at [c000000274087890]
    pc: c0000000004b9728: memunmap_pages+0x238/0x340
    lr: c0000000004b9724: memunmap_pages+0x234/0x340
...
    pid   = 3669, comm = ndctl

kernel BUG at include/linux/mm.h:1107!
[c000000274087ba0] c0000000009e3500 devm_action_release+0x30/0x50 [c000000274087bc0] c0000000009e4758 release_nodes+0x268/0x2d0 [c000000274087c30] c0000000009dd144 device_release_driver_internal+0x174/0x240 [c000000274087c70] c0000000009d9dfc unbind_store+0x13c/0x190 [c000000274087cb0] c0000000009d8a24 drv_attr_store+0x44/0x60 [c000000274087cd0] c0000000005a7470 sysfs_kf_write+0x70/0xa0 [c000000274087d10] c0000000005a5cac kernfs_fop_write+0x1ac/0x290 [c000000274087d60] c0000000004be45c __vfs_write+0x3c/0x70 [c000000274087d80] c0000000004c26e4 vfs_write+0xe4/0x200 [c000000274087dd0] c0000000004c2a6c ksys_write+0x7c/0x140 [c000000274087e20] c00000000000bbd0 system_call+0x5c/0x68

Cc Dan Williams <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Ira Weiny <[email protected]>
[ minimze code changes, rephrase description ]
---
 mm/memremap.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
mm/memremap.c +1 -0
index 557e53c6fb46..8c2fb44c3b4d 100644
@@ -123,6 +123,7 @@ static void dev_pagemap_cleanup(struct dev_pagemap *pgmap)
 void memunmap_pages(struct dev_pagemap *pgmap)
 {
 	struct resource *res = &pgmap->res;
+	struct page *first_page;
 	unsigned long pfn;
 	int nid;
 

mm/memremap.c void memunmap_pages(struct dev_pagemap *pgmap) · line 139

Quoted 9 lines
mm/memremap.c +4 -1
@@ -131,14 +132,16 @@ void memunmap_pages(struct dev_pagemap *pgmap)
 		put_page(pfn_to_page(pfn));
 	dev_pagemap_cleanup(pgmap);
 
+	/* make sure to access a memmap that was actually initialized */
+	first_page = pfn_to_page(pfn_first(pgmap));
+
 	/* pages are dead and unused, undo the arch mapping */
-	nid = page_to_nid(pfn_to_page(PHYS_PFN(res->start)));
+	nid = page_to_nid(first_page);

Why we need 'nid = page_to_nid(first_page)' we didnt use it anymore in this function ?

Quoted 14 lines
mm/memremap.c +2 -3
 	mem_hotplug_begin();
 	if (pgmap->type = MEMORY_DEVICE_PRIVATE) {
-		pfn = PHYS_PFN(res->start);
-		__remove_pages(page_zone(pfn_to_page(pfn)), pfn,
-				 PHYS_PFN(resource_size(res)), NULL);
+		__remove_pages(page_zone(first_page), PHYS_PFN(res->start),
+			       PHYS_PFN(resource_size(res)), NULL);
 	} else {
 		arch_remove_memory(nid, res->start, resource_size(res),
 				pgmap_altmap(pgmap));
Signature · 1 line
2.21.0

Best regards
Damian

#3 lore raw

Re: [PATCH v6 01/10] mm/memunmap: Don't access uninitialized memmap in memunmap_pages()

Quoted 4 lines · Damian Tometzki

Hello David,

patch 05/10 is missing in the patch series.

Hi Damian,

not really. Could be that lkml is slow today. E.g., check

https://marc.info/?l=linux-mm&m7035222620403&w=2

and especially

https://marc.info/?l=linux-mm&m7035225120440&w=2

All mails popped up on the mm list.

Quoted 57 lines · David Hildenbrand
Quoted 55 lines

From: "Aneesh Kumar K.V" <[email protected]>

With an altmap, the memmap falling into the reserved altmap space are not initialized and, therefore, contain a garbage NID and a garbage zone. Make sure to read the NID/zone from a memmap that was initialzed.

This fixes a kernel crash that is observed when destroying a namespace:

[   81.356173] kernel BUG at include/linux/mm.h:1107!
cpu 0x1: Vector: 700 (Program Check) at [c000000274087890]
    pc: c0000000004b9728: memunmap_pages+0x238/0x340
    lr: c0000000004b9724: memunmap_pages+0x234/0x340
...
    pid   = 3669, comm = ndctl

kernel BUG at include/linux/mm.h:1107!
[c000000274087ba0] c0000000009e3500 devm_action_release+0x30/0x50 [c000000274087bc0] c0000000009e4758 release_nodes+0x268/0x2d0 [c000000274087c30] c0000000009dd144 device_release_driver_internal+0x174/0x240 [c000000274087c70] c0000000009d9dfc unbind_store+0x13c/0x190 [c000000274087cb0] c0000000009d8a24 drv_attr_store+0x44/0x60 [c000000274087cd0] c0000000005a7470 sysfs_kf_write+0x70/0xa0 [c000000274087d10] c0000000005a5cac kernfs_fop_write+0x1ac/0x290 [c000000274087d60] c0000000004be45c __vfs_write+0x3c/0x70 [c000000274087d80] c0000000004c26e4 vfs_write+0xe4/0x200 [c000000274087dd0] c0000000004c2a6c ksys_write+0x7c/0x140 [c000000274087e20] c00000000000bbd0 system_call+0x5c/0x68

Cc Dan Williams <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Ira Weiny <[email protected]>
[ minimze code changes, rephrase description ]
---
 mm/memremap.c | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)
mm/memremap.c +1 -0
index 557e53c6fb46..8c2fb44c3b4d 100644
@@ -123,6 +123,7 @@ static void dev_pagemap_cleanup(struct dev_pagemap *pgmap)
 void memunmap_pages(struct dev_pagemap *pgmap)
 {
 	struct resource *res = &pgmap->res;
+	struct page *first_page;
 	unsigned long pfn;
 	int nid;
 
@@ -131,14 +132,16 @@ void memunmap_pages(struct dev_pagemap *pgmap)
 		put_page(pfn_to_page(pfn));
 	dev_pagemap_cleanup(pgmap);
 
Quoted 8 lines

mm/memremap.c void memunmap_pages(struct dev_pagemap *pgmap) · line 139

Quoted 6 lines
mm/memremap.c +4 -1
+	/* make sure to access a memmap that was actually initialized */
+	first_page = pfn_to_page(pfn_first(pgmap));
+
 	/* pages are dead and unused, undo the arch mapping */
-	nid = page_to_nid(pfn_to_page(PHYS_PFN(res->start)));
+	nid = page_to_nid(first_page);

Why we need 'nid = page_to_nid(first_page)' we didnt use it anymore in this function ?

Please see ...

Quoted 11 lines

mm/memremap.c void memunmap_pages(struct dev_pagemap *pgmap) · line 146

Quoted 11 lines
mm/memremap.c +2 -3
 	mem_hotplug_begin();
 	if (pgmap->type = MEMORY_DEVICE_PRIVATE) {
-		pfn = PHYS_PFN(res->start);
-		__remove_pages(page_zone(pfn_to_page(pfn)), pfn,
-				 PHYS_PFN(resource_size(res)), NULL);
+		__remove_pages(page_zone(first_page), PHYS_PFN(res->start),
+			       PHYS_PFN(resource_size(res)), NULL);
 	} else {
 		arch_remove_memory(nid, res->start, resource_size(res),

^ here

:)

Quoted 1 line
Quoted 1 line
mm/memremap.c +0 -0
 				pgmap_altmap(pgmap));
Signature · 3 lines
Thanks,

David / dhildenb
#4 lore raw

Re: [PATCH v6 01/10] mm/memunmap: Don't access uninitialized memmap in memunmap_pages()

In reply to David Hildenbrand
Quoted 63 lines · David Hildenbrand

From: "Aneesh Kumar K.V" <[email protected]>

With an altmap, the memmap falling into the reserved altmap space are not initialized and, therefore, contain a garbage NID and a garbage zone. Make sure to read the NID/zone from a memmap that was initialzed.

This fixes a kernel crash that is observed when destroying a namespace:

[   81.356173] kernel BUG at include/linux/mm.h:1107!
cpu 0x1: Vector: 700 (Program Check) at [c000000274087890]
     pc: c0000000004b9728: memunmap_pages+0x238/0x340
     lr: c0000000004b9724: memunmap_pages+0x234/0x340
...
     pid   = 3669, comm = ndctl

kernel BUG at include/linux/mm.h:1107!
[c000000274087ba0] c0000000009e3500 devm_action_release+0x30/0x50 [c000000274087bc0] c0000000009e4758 release_nodes+0x268/0x2d0 [c000000274087c30] c0000000009dd144 device_release_driver_internal+0x174/0x240 [c000000274087c70] c0000000009d9dfc unbind_store+0x13c/0x190 [c000000274087cb0] c0000000009d8a24 drv_attr_store+0x44/0x60 [c000000274087cd0] c0000000005a7470 sysfs_kf_write+0x70/0xa0 [c000000274087d10] c0000000005a5cac kernfs_fop_write+0x1ac/0x290 [c000000274087d60] c0000000004be45c __vfs_write+0x3c/0x70 [c000000274087d80] c0000000004c26e4 vfs_write+0xe4/0x200 [c000000274087dd0] c0000000004c2a6c ksys_write+0x7c/0x140 [c000000274087e20] c00000000000bbd0 system_call+0x5c/0x68

Cc Dan Williams <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Ira Weiny <[email protected]>
[ minimze code changes, rephrase description ]
---
  mm/memremap.c | 11 +++++++----
  1 file changed, 7 insertions(+), 4 deletions(-)
mm/memremap.c +5 -1
index 557e53c6fb46..8c2fb44c3b4d 100644
@@ -123,6 +123,7 @@ static void dev_pagemap_cleanup(struct dev_pagemap *pgmap)
  void memunmap_pages(struct dev_pagemap *pgmap)
  {
  	struct resource *res = &pgmap->res;
+	struct page *first_page;
  	unsigned long pfn;
  	int nid;
 
@@ -131,14 +132,16 @@ void memunmap_pages(struct dev_pagemap *pgmap)
  		put_page(pfn_to_page(pfn));
  	dev_pagemap_cleanup(pgmap);
 
+	/* make sure to access a memmap that was actually initialized */
+	first_page = pfn_to_page(pfn_first(pgmap));
+
  	/* pages are dead and unused, undo the arch mapping */
-	nid = page_to_nid(pfn_to_page(PHYS_PFN(res->start)));
+	nid = page_to_nid(first_page);
 

mm/memremap.c void memunmap_pages(struct dev_pagemap *pgmap) · line 147

Quoted 10 lines
mm/memremap.c +2 -3
  	mem_hotplug_begin();
  	if (pgmap->type = MEMORY_DEVICE_PRIVATE) {
-		pfn = PHYS_PFN(res->start);
-		__remove_pages(page_zone(pfn_to_page(pfn)), pfn,
-				 PHYS_PFN(resource_size(res)), NULL);
+		__remove_pages(page_zone(first_page), PHYS_PFN(res->start),
+			       PHYS_PFN(resource_size(res)), NULL);
  	} else {
  		arch_remove_memory(nid, res->start, resource_size(res),
  				pgmap_altmap(pgmap));

@Andrew, can you add

Fixes 2c2a5af6fed2 ("mm, memory_hotplug: add nid parameter to

arch_remove_memory")

(which basically introduced the nid = page_to_nid(first_page))

The "page_zone(pfn_to_page(pfn)" was introduced by 69324b8f4833 ("mm,
devm_memremap_pages: add MEMORY_DEVICE_PRIVATE support"), however, I think we will never have driver reserved memory with MEMORY_DEVICE_PRIVATE (no altmap AFAIKS).

Also, I think

makes sense.

Signature · 3 lines
Thanks,

David / dhildenb
#5 lore raw

PATCH v6 2/10 mm/memmap_init: Update variable name in memmap_init_zone

In reply to David Hildenbrand

From: "Aneesh Kumar K.V" <[email protected]>

The third argument is actually number of pages. Changes the variable name from size to nr_pages to indicate this better.

No functional change in this patch.

Cc Andrew Morton <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Vlastimil Babka <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Mel Gorman <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Dan Williams <[email protected]>
Cc Alexander Duyck <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Alexander Potapenko <[email protected]>
---
 mm/page_alloc.c | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)
mm/page_alloc.c +4 -4
index 15c2050c629b..b0b2d5464000 100644
@@ -5936,10 +5936,10 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,
 #ifdef CONFIG_ZONE_DEVICE
 void __ref memmap_init_zone_device(struct zone *zone,
 				   unsigned long start_pfn,
-				   unsigned long size,
+				   unsigned long nr_pages,
 				   struct dev_pagemap *pgmap)
 {
-	unsigned long pfn, end_pfn = start_pfn + size;
+	unsigned long pfn, end_pfn = start_pfn + nr_pages;
 	struct pglist_data *pgdat = zone->zone_pgdat;
 	struct vmem_altmap *altmap = pgmap_altmap(pgmap);
 	unsigned long zone_idx = zone_idx(zone);
@@ -5956,7 +5956,7 @@ void __ref memmap_init_zone_device(struct zone *zone,
 	 */
 	if (altmap) {
 		start_pfn = altmap->base_pfn + vmem_altmap_offset(altmap);
-		size = end_pfn - start_pfn;
+		nr_pages = end_pfn - start_pfn;
 	}
 
 	for (pfn = start_pfn; pfn < end_pfn; pfn++) {
@@ -6003,7 +6003,7 @@ void __ref memmap_init_zone_device(struct zone *zone,
 	}
 
 	pr_info("%s initialised %lu pages in %ums\n", __func__,
-		size, jiffies_to_msecs(jiffies - start));
+		nr_pages, jiffies_to_msecs(jiffies - start));
 }
 
 #endif
Signature · 1 line
2.21.0
#6 lore raw

PATCH v6 3/10 mm/memory_hotplug: Don't access uninitialized memmaps in shrink_pgdat_span()

In reply to David Hildenbrand

We might use the nid of memmaps that were never initialized. For example, if the memmap was poisoned, we will crash the kernel in pfn_to_nid() right now. Let's use the calculated boundaries of the separate zones instead. This now also avoids having to iterate over a whole bunch of subsections again, after shrinking one zone.

Before commit d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug"), the memmap was initialized to 0 and the node was set to the right value. After that commit, the node might be garbage.

We'll have to fix shrink_zone_span() next.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
Fixes d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug")
---
 mm/memory_hotplug.c | 72 ++++++++++-----------------------------------
 1 file changed, 15 insertions(+), 57 deletions(-)
mm/memory_hotplug.c +15 -57
index 680b4b3e57d9..86b4dc18e831 100644
@@ -436,67 +436,25 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 	zone_span_writeunlock(zone);
 }
 
-static void shrink_pgdat_span(struct pglist_data *pgdat,
-			      unsigned long start_pfn, unsigned long end_pfn)
+static void update_pgdat_span(struct pglist_data *pgdat)
 {
-	unsigned long pgdat_start_pfn = pgdat->node_start_pfn;
-	unsigned long p = pgdat_end_pfn(pgdat); /* pgdat_end_pfn namespace clash */
-	unsigned long pgdat_end_pfn = p;
-	unsigned long pfn;
-	int nid = pgdat->node_id;
-
-	if (pgdat_start_pfn = start_pfn) {
-		/*
-		 * If the section is smallest section in the pgdat, it need
-		 * shrink pgdat->node_start_pfn and pgdat->node_spanned_pages.
-		 * In this case, we find second smallest valid mem_section
-		 * for shrinking zone.
-		 */
-		pfn = find_smallest_section_pfn(nid, NULL, end_pfn,
-						pgdat_end_pfn);
-		if (pfn) {
-			pgdat->node_start_pfn = pfn;
-			pgdat->node_spanned_pages = pgdat_end_pfn - pfn;
-		}
-	} else if (pgdat_end_pfn = end_pfn) {
-		/*
-		 * If the section is biggest section in the pgdat, it need
-		 * shrink pgdat->node_spanned_pages.
-		 * In this case, we find second biggest valid mem_section for
-		 * shrinking zone.
-		 */
-		pfn = find_biggest_section_pfn(nid, NULL, pgdat_start_pfn,
-					       start_pfn);
-		if (pfn)
-			pgdat->node_spanned_pages = pfn - pgdat_start_pfn + 1;
-	}
-
-	/*
-	 * If the section is not biggest or smallest mem_section in the pgdat,
-	 * it only creates a hole in the pgdat. So in this case, we need not
-	 * change the pgdat.
-	 * But perhaps, the pgdat has only hole data. Thus it check the pgdat
-	 * has only hole or not.
-	 */
-	pfn = pgdat_start_pfn;
-	for (; pfn < pgdat_end_pfn; pfn += PAGES_PER_SUBSECTION) {
-		if (unlikely(!pfn_valid(pfn)))
-			continue;
-
-		if (pfn_to_nid(pfn) != nid)
-			continue;
+	unsigned long node_start_pfn = 0, node_end_pfn = 0;
+	struct zone *zone;
 
-		/* Skip range to be removed */
-		if (pfn >= start_pfn && pfn < end_pfn)
-			continue;
+	for (zone = pgdat->node_zones;
+	     zone < pgdat->node_zones + MAX_NR_ZONES; zone++) {
+		unsigned long zone_end_pfn = zone->zone_start_pfn +
+					     zone->spanned_pages;
 
-		/* If we find valid section, we have nothing to do */
-		return;
+		/* No need to lock the zones, they can't change. */
+		if (zone_end_pfn > node_end_pfn)
+			node_end_pfn = zone_end_pfn;
+		if (zone->zone_start_pfn < node_start_pfn)
+			node_start_pfn = zone->zone_start_pfn;
 	}
 
-	/* The pgdat has no valid section */
-	pgdat->node_start_pfn = 0;
-	pgdat->node_spanned_pages = 0;
+	pgdat->node_start_pfn = node_start_pfn;
+	pgdat->node_spanned_pages = node_end_pfn - node_start_pfn;
 }
 
 static void __remove_zone(struct zone *zone, unsigned long start_pfn,
@@ -507,7 +465,7 @@ static void __remove_zone(struct zone *zone, unsigned long start_pfn,
 
 	pgdat_resize_lock(zone->zone_pgdat, &flags);
 	shrink_zone_span(zone, start_pfn, start_pfn + nr_pages);
-	shrink_pgdat_span(pgdat, start_pfn, start_pfn + nr_pages);
+	update_pgdat_span(pgdat);
 	pgdat_resize_unlock(zone->zone_pgdat, &flags);
 }
 
Signature · 1 line
2.21.0
#7 lore raw

Re: [PATCH v6 03/10] mm/memory_hotplug: Don't access uninitialized memmaps in shrink_pgdat_span()

Quoted 12 lines · David Hildenbrand

We might use the nid of memmaps that were never initialized. For example, if the memmap was poisoned, we will crash the kernel in pfn_to_nid() right now. Let's use the calculated boundaries of the separate zones instead. This now also avoids having to iterate over a whole bunch of subsections again, after shrinking one zone.

Before commit d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug"), the memmap was initialized to 0 and the node was set to the right value. After that commit, the node might be garbage.

We'll have to fix shrink_zone_span() next.

Quoted 8 lines
Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
Fixes d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug")

@Andrew, can you convert that to

Fixes f1dd2cd13c4b ("mm, memory_hotplug: do not associate hotadded memory to zones until online") # visible after d0dc12e86b319

and add

Cc [email protected] # v4.13+
Signature · 3 lines
Thanks,

David / dhildenb
#8 lore raw

PATCH v6 4/10 mm/memory_hotplug: Don't access uninitialized memmaps in shrink_zone_span()

In reply to David Hildenbrand

Let's limit shrinking to !ZONE_DEVICE so we can fix the current code. We should never try to touch the memmap of offline sections where we could have uninitialized memmaps and could trigger BUGs when calling page_to_nid() on poisoned pages.

There is no reliable way to distinguish an uninitialized memmap from an initialized memmap that belongs to ZONE_DEVICE, as we don't have anything like SECTION_IS_ONLINE we can use similar to pfn_to_online_section() for !ZONE_DEVICE memory. E.g., set_zone_contiguous() similarly relies on pfn_to_online_section() and will therefore never set a ZONE_DEVICE zone consecutive. Stopping to shrink the ZONE_DEVICE therefore results in no observable changes, besides /proc/zoneinfo indicating different boundaries - something we can totally live with.

Before commit d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug"), the memmap was initialized with 0 and the node with the right value. So the zone might be wrong but not garbage. After that commit, both the zone and the node will be garbage when touching uninitialized memmaps.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Fixes d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug")
---
 mm/memory_hotplug.c | 16 +++++++++++++---
 1 file changed, 13 insertions(+), 3 deletions(-)
mm/memory_hotplug.c +13 -3
index 86b4dc18e831..f96608d24f6a 100644
@@ -331,7 +331,7 @@ static unsigned long find_smallest_section_pfn(int nid, struct zone *zone,
 				     unsigned long end_pfn)
 {
 	for (; start_pfn < end_pfn; start_pfn += PAGES_PER_SUBSECTION) {
-		if (unlikely(!pfn_valid(start_pfn)))
+		if (unlikely(!pfn_to_online_page(start_pfn)))
 			continue;
 
 		if (unlikely(pfn_to_nid(start_pfn) != nid))
@@ -356,7 +356,7 @@ static unsigned long find_biggest_section_pfn(int nid, struct zone *zone,
 	/* pfn is the end pfn of a memory section. */
 	pfn = end_pfn - 1;
 	for (; pfn >= start_pfn; pfn -= PAGES_PER_SUBSECTION) {
-		if (unlikely(!pfn_valid(pfn)))
+		if (unlikely(!pfn_to_online_page(pfn)))
 			continue;
 
 		if (unlikely(pfn_to_nid(pfn) != nid))
@@ -415,7 +415,7 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 	 */
 	pfn = zone_start_pfn;
 	for (; pfn < zone_end_pfn; pfn += PAGES_PER_SUBSECTION) {
-		if (unlikely(!pfn_valid(pfn)))
+		if (unlikely(!pfn_to_online_page(pfn)))
 			continue;
 
 		if (page_zone(pfn_to_page(pfn)) != zone)
@@ -463,6 +463,16 @@ static void __remove_zone(struct zone *zone, unsigned long start_pfn,
 	struct pglist_data *pgdat = zone->zone_pgdat;
 	unsigned long flags;
 
+#ifdef CONFIG_ZONE_DEVICE
+	/*
+	 * Zone shrinking code cannot properly deal with ZONE_DEVICE. So
+	 * we will not try to shrink the zones - which is okay as
+	 * set_zone_contiguous() cannot deal with ZONE_DEVICE either way.
+	 */
+	if (zone_idx(zone) = ZONE_DEVICE)
+		return;
+#endif
+
 	pgdat_resize_lock(zone->zone_pgdat, &flags);
 	shrink_zone_span(zone, start_pfn, start_pfn + nr_pages);
 	update_pgdat_span(pgdat);
Signature · 1 line
2.21.0
#9 lore raw

Re: [PATCH v6 04/10] mm/memory_hotplug: Don't access uninitialized memmaps in shrink_zone_span()

Quoted 21 lines · David Hildenbrand

Let's limit shrinking to !ZONE_DEVICE so we can fix the current code. We should never try to touch the memmap of offline sections where we could have uninitialized memmaps and could trigger BUGs when calling page_to_nid() on poisoned pages.

There is no reliable way to distinguish an uninitialized memmap from an initialized memmap that belongs to ZONE_DEVICE, as we don't have anything like SECTION_IS_ONLINE we can use similar to pfn_to_online_section() for !ZONE_DEVICE memory. E.g., set_zone_contiguous() similarly relies on pfn_to_online_section() and will therefore never set a ZONE_DEVICE zone consecutive. Stopping to shrink the ZONE_DEVICE therefore results in no observable changes, besides /proc/zoneinfo indicating different boundaries - something we can totally live with.

Before commit d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug"), the memmap was initialized with 0 and the node with the right value. So the zone might be wrong but not garbage. After that commit, both the zone and the node will be garbage when touching uninitialized memmaps.

Quoted 7 lines
Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Fixes d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug")

@Andrew, can you convert that to

Fixes f1dd2cd13c4b ("mm, memory_hotplug: do not associate hotadded

memory to zones until online") # visible after d0dc12e86b319

and add

Cc [email protected] # v4.13+
Signature · 3 lines
Thanks,

David / dhildenb
maintainer
#10 lore raw

Re: [PATCH v6 04/10] mm/memory_hotplug: Don't access uninitialized memmaps in shrink_zone_span()

Quoted 10 lines · On Mon, 14 Oct 2019 11:32:13 +0200 David Hildenbrand <[email protected]> wrote:
Quoted 1 line
Fixes d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug")

@Andrew, can you convert that to

Fixes f1dd2cd13c4b ("mm, memory_hotplug: do not associate hotadded

memory to zones until online") # visible after d0dc12e86b319

and add

Cc [email protected] # v4.13+

Done, thanks.

#11 lore raw

Re: [PATCH v6 04/10] mm/memory_hotplug: Don't access uninitialized memmaps in shrink_zone_span()

Quoted 6 lines · Andrew Morton
Quoted 4 lines · On Mon, 14 Oct 2019 11:32:13 +0200 David Hildenbrand <[email protected]> wrote:
Quoted 1 line
Fixes d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug")

@Andrew, can you convert that to

Quoted 9 lines
Quoted 6 lines
Fixes f1dd2cd13c4b ("mm, memory_hotplug: do not associate hotadded

memory to zones until online") # visible after d0dc12e86b319

and add

Cc [email protected] # v4.13+

Done, thanks.

Just a note that Toshiki reported a BUG (race between delayed initialization of ZONE_DEVICE memmaps without holding the memory hotplug lock and concurrent zone shrinking).

https://lkml.org/lkml/2019/11/14/1040

"Iteration of create and destroy namespace causes the panic as below:

[   41.207694] kernel BUG at mm/page_alloc.c:535!
[   41.208109] invalid opcode: 0000 [#1] SMP PTI
[   41.208508] CPU: 7 PID: 2766 Comm: ndctl Not tainted 5.4.0-rc4 #6
[   41.209064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
[   41.210175] RIP: 0010:set_pfnblock_flags_mask+0x95/0xf0
[   41.210643] Code: 04 41 83 e2 3c 48 8d 04 a8 48 c1 e0 07 48 03 04 dd e0 59 55 bb 48 8b 58 68 48 39 da 73 0e 48 c7 c6 70 ac 11 bb e8 1b b2 fd ff <0f> 0b 48 03 58 78 48 39 da 73 e9 49 01 ca b9 3f 00 00 00 4f 8d 0c
[   41.212354] RSP: 0018:ffffac0d41557c80 EFLAGS: 00010246
[   41.212821] RAX: 000000000000004a RBX: 0000000000244a00 RCX: 0000000000000000
[   41.213459] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffbb1197dc
[   41.214100] RBP: 000000000000000c R08: 0000000000000439 R09: 0000000000000059
[   41.214736] R10: 0000000000000000 R11: ffffac0d41557b08 R12: ffff8be475ea72b0
[   41.215376] R13: 000000000000fa00 R14: 0000000000250000 R15: 00000000fffc0bb5
[   41.216008] FS:  00007f30862ab600(0000) GS:ffff8be57bc40000(0000) knlGS:0000000000000000
[   41.216771] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   41.217299] CR2: 000055e824d0d508 CR3: 0000000231dac000 CR4: 00000000000006e0
[   41.217934] Call Trace:
[   41.218225]  memmap_init_zone_device+0x165/0x17c
[   41.218642]  memremap_pages+0x4c1/0x540
[   41.218989]  devm_memremap_pages+0x1d/0x60
[   41.219367]  pmem_attach_disk+0x16b/0x600 [nd_pmem]
[   41.219804]  ? devm_nsio_enable+0xb8/0xe0
[   41.220172]  nvdimm_bus_probe+0x69/0x1c0
[   41.220526]  really_probe+0x1c2/0x3e0
[   41.220856]  driver_probe_device+0xb4/0x100
[   41.221238]  device_driver_attach+0x4f/0x60
[   41.221611]  bind_store+0xc9/0x110
[   41.221919]  kernfs_fop_write+0x116/0x190
[   41.222326]  vfs_write+0xa5/0x1a0
[   41.222626]  ksys_write+0x59/0xd0
[   41.222927]  do_syscall_64+0x5b/0x180
[   41.223264]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   41.223714] RIP: 0033:0x7f30865d0ed8
[   41.224037] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 45 78 0d 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
[   41.225920] RSP: 002b:00007fffe5d30a78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[   41.226608] RAX: ffffffffffffffda RBX: 000055e824d07f40 RCX: 00007f30865d0ed8
[   41.227242] RDX: 0000000000000007 RSI: 000055e824d07f40 RDI: 0000000000000004
[   41.227870] RBP: 0000000000000007 R08: 0000000000000007 R09: 0000000000000006
[   41.228753] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
[   41.229419] R13: 00007f30862ab528 R14: 0000000000000001 R15: 000055e824d07f40

While creating a namespace and initializing memmap, if you destroy the namespace and shrink the zone, it will initialize the memmap outside the zone and trigger VM_BUG_ON_PAGE(!zone_spans_pfn(page_zone(page), pfn), page) in set_pfnblock_flags_mask()."

This BUG is also mitigated by this commit, where we for now stop to shrink the ZONE_DEVICE zone until we can do it in a safe and clean way.

Signature · 3 lines
Thanks,

David / dhildenb
maintainer
#12 lore raw

Re: [PATCH v6 04/10] mm/memory_hotplug: Don't access uninitialized memmaps in shrink_zone_span()

Quoted 71 lines · On Tue, 19 Nov 2019 15:16:22 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 4 lines · On Mon, 14 Oct 2019 11:32:13 +0200 David Hildenbrand <[email protected]> wrote:
Quoted 1 line
Fixes d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug")

@Andrew, can you convert that to

Quoted 9 lines
Quoted 6 lines
Fixes f1dd2cd13c4b ("mm, memory_hotplug: do not associate hotadded

memory to zones until online") # visible after d0dc12e86b319

and add

Cc [email protected] # v4.13+

Done, thanks.

Just a note that Toshiki reported a BUG (race between delayed initialization of ZONE_DEVICE memmaps without holding the memory hotplug lock and concurrent zone shrinking).

https://lkml.org/lkml/2019/11/14/1040

"Iteration of create and destroy namespace causes the panic as below:

[   41.207694] kernel BUG at mm/page_alloc.c:535!
[   41.208109] invalid opcode: 0000 [#1] SMP PTI
[   41.208508] CPU: 7 PID: 2766 Comm: ndctl Not tainted 5.4.0-rc4 #6
[   41.209064] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
[   41.210175] RIP: 0010:set_pfnblock_flags_mask+0x95/0xf0
[   41.210643] Code: 04 41 83 e2 3c 48 8d 04 a8 48 c1 e0 07 48 03 04 dd e0 59 55 bb 48 8b 58 68 48 39 da 73 0e 48 c7 c6 70 ac 11 bb e8 1b b2 fd ff <0f> 0b 48 03 58 78 48 39 da 73 e9 49 01 ca b9 3f 00 00 00 4f 8d 0c
[   41.212354] RSP: 0018:ffffac0d41557c80 EFLAGS: 00010246
[   41.212821] RAX: 000000000000004a RBX: 0000000000244a00 RCX: 0000000000000000
[   41.213459] RDX: 0000000000000000 RSI: 0000000000000000 RDI: ffffffffbb1197dc
[   41.214100] RBP: 000000000000000c R08: 0000000000000439 R09: 0000000000000059
[   41.214736] R10: 0000000000000000 R11: ffffac0d41557b08 R12: ffff8be475ea72b0
[   41.215376] R13: 000000000000fa00 R14: 0000000000250000 R15: 00000000fffc0bb5
[   41.216008] FS:  00007f30862ab600(0000) GS:ffff8be57bc40000(0000) knlGS:0000000000000000
[   41.216771] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   41.217299] CR2: 000055e824d0d508 CR3: 0000000231dac000 CR4: 00000000000006e0
[   41.217934] Call Trace:
[   41.218225]  memmap_init_zone_device+0x165/0x17c
[   41.218642]  memremap_pages+0x4c1/0x540
[   41.218989]  devm_memremap_pages+0x1d/0x60
[   41.219367]  pmem_attach_disk+0x16b/0x600 [nd_pmem]
[   41.219804]  ? devm_nsio_enable+0xb8/0xe0
[   41.220172]  nvdimm_bus_probe+0x69/0x1c0
[   41.220526]  really_probe+0x1c2/0x3e0
[   41.220856]  driver_probe_device+0xb4/0x100
[   41.221238]  device_driver_attach+0x4f/0x60
[   41.221611]  bind_store+0xc9/0x110
[   41.221919]  kernfs_fop_write+0x116/0x190
[   41.222326]  vfs_write+0xa5/0x1a0
[   41.222626]  ksys_write+0x59/0xd0
[   41.222927]  do_syscall_64+0x5b/0x180
[   41.223264]  entry_SYSCALL_64_after_hwframe+0x44/0xa9
[   41.223714] RIP: 0033:0x7f30865d0ed8
[   41.224037] Code: 89 02 48 c7 c0 ff ff ff ff eb b3 0f 1f 80 00 00 00 00 f3 0f 1e fa 48 8d 05 45 78 0d 00 8b 00 85 c0 75 17 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 58 c3 0f 1f 80 00 00 00 00 41 54 49 89 d4 55
[   41.225920] RSP: 002b:00007fffe5d30a78 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
[   41.226608] RAX: ffffffffffffffda RBX: 000055e824d07f40 RCX: 00007f30865d0ed8
[   41.227242] RDX: 0000000000000007 RSI: 000055e824d07f40 RDI: 0000000000000004
[   41.227870] RBP: 0000000000000007 R08: 0000000000000007 R09: 0000000000000006
[   41.228753] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000004
[   41.229419] R13: 00007f30862ab528 R14: 0000000000000001 R15: 000055e824d07f40

While creating a namespace and initializing memmap, if you destroy the namespace and shrink the zone, it will initialize the memmap outside the zone and trigger VM_BUG_ON_PAGE(!zone_spans_pfn(page_zone(page), pfn), page) in set_pfnblock_flags_mask()."

Quoted 4 lines

This BUG is also mitigated by this commit, where we for now stop to shrink the ZONE_DEVICE zone until we can do it in a safe and clean way.

OK, thanks. I updated the changelog, added Reported-by:Toshiki and shall squeeze this fix into 5.4.

#13 lore raw

PATCH v6 5/10 mm/memory_hotplug: Shrink zones when offlining memory

In reply to David Hildenbrand

We currently try to shrink a single zone when removing memory. We use the zone of the first page of the memory we are removing. If that memmap was never initialized (e.g., memory was never onlined), we will read garbage and can trigger kernel BUGs (due to a stale pointer):

:/# [   23.912993] BUG: unable to handle page fault for address: 000000000000353d
[   23.914219] #PF: supervisor write access in kernel mode
[   23.915199] #PF: error_code(0x0002) - not-present page
[   23.916160] PGD 0 P4D 0
[   23.916627] Oops: 0002 [#1] SMP PTI
[   23.917256] CPU: 1 PID: 7 Comm: kworker/u8:0 Not tainted 5.3.0-rc5-next-20190820+ #317
[   23.918900] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.4
[   23.921194] Workqueue: kacpi_hotplug acpi_hotplug_work_fn
[   23.922249] RIP: 0010:clear_zone_contiguous+0x5/0x10
[   23.923173] Code: 48 89 c6 48 89 c3 e8 2a fe ff ff 48 85 c0 75 cf 5b 5d c3 c6 85 fd 05 00 00 01 5b 5d c3 0f 1f 840
[   23.926876] RSP: 0018:ffffad2400043c98 EFLAGS: 00010246
[   23.927928] RAX: 0000000000000000 RBX: 0000000200000000 RCX: 0000000000000000
[   23.929458] RDX: 0000000000200000 RSI: 0000000000140000 RDI: 0000000000002f40
[   23.930899] RBP: 0000000140000000 R08: 0000000000000000 R09: 0000000000000001
[   23.932362] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000140000
[   23.933603] R13: 0000000000140000 R14: 0000000000002f40 R15: ffff9e3e7aff3680
[   23.934913] FS:  0000000000000000(0000) GS:ffff9e3e7bb00000(0000) knlGS:0000000000000000
[   23.936294] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   23.937481] CR2: 000000000000353d CR3: 0000000058610000 CR4: 00000000000006e0
[   23.938687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   23.939889] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   23.941168] Call Trace:
[   23.941580]  __remove_pages+0x4b/0x640
[   23.942303]  ? mark_held_locks+0x49/0x70
[   23.943149]  arch_remove_memory+0x63/0x8d
[   23.943921]  try_remove_memory+0xdb/0x130
[   23.944766]  ? walk_memory_blocks+0x7f/0x9e
[   23.945616]  __remove_memory+0xa/0x11
[   23.946274]  acpi_memory_device_remove+0x70/0x100
[   23.947308]  acpi_bus_trim+0x55/0x90
[   23.947914]  acpi_device_hotplug+0x227/0x3a0
[   23.948714]  acpi_hotplug_work_fn+0x1a/0x30
[   23.949433]  process_one_work+0x221/0x550
[   23.950190]  worker_thread+0x50/0x3b0
[   23.950993]  kthread+0x105/0x140
[   23.951644]  ? process_one_work+0x550/0x550
[   23.952508]  ? kthread_park+0x80/0x80
[   23.953367]  ret_from_fork+0x3a/0x50
[   23.954025] Modules linked in:
[   23.954613] CR2: 000000000000353d
[   23.955248] ---[ end trace 93d982b1fb3e1a69 ]---

Instead, shrink the zones when offlining memory or when onlining failed. Introduce and use remove_pfn_range_from_zone(() for that. We now properly shrink the zones, even if we have DIMMs whereby
- Some memory blocks fall into no zone (never onlined)
- Some memory blocks fall into multiple zones (offlined+re-onlined)
- Multiple memory blocks that fall into different zones

Drop the zone parameter (with a potential dubious value) from __remove_pages() and __remove_section().

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Tony Luck <[email protected]>
Cc Fenghua Yu <[email protected]>
Cc Benjamin Herrenschmidt <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Vasily Gorbik <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Yoshinori Sato <[email protected]>
Cc Rich Felker <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Steve Capper <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Anshuman Khandual <[email protected]>
Cc Yu Zhao <[email protected]>
Cc Jun Yao <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc "Matthew Wilcox (Oracle)" <[email protected]>
Cc Christophe Leroy <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Gerald Schaefer <[email protected]>
Cc Halil Pasic <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Masahiro Yamada <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
Cc Qian Cai <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Ira Weiny <[email protected]>
Fixes d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug")
---
 arch/arm64/mm/mmu.c            |  4 +---
 arch/ia64/mm/init.c            |  4 +---
 arch/powerpc/mm/mem.c          |  3 +--
 arch/s390/mm/init.c            |  4 +---
 arch/sh/mm/init.c              |  4 +---
 arch/x86/mm/init_32.c          |  4 +---
 arch/x86/mm/init_64.c          |  4 +---
 include/linux/memory_hotplug.h |  7 +++++--
 mm/memory_hotplug.c            | 31 ++++++++++++++++---------------
 mm/memremap.c                  |  2 +-
 10 files changed, 29 insertions(+), 38 deletions(-)
arch/arm64/mm/mmu.c +1 -3
index 60c929f3683b..d10247fab0fd 100644
@@ -1069,7 +1069,6 @@ void arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
 	/*
 	 * FIXME: Cleanup page tables (also in arch_add_memory() in case
@@ -1078,7 +1077,6 @@ void arch_remove_memory(int nid, u64 start, u64 size,
 	 * unplug. ARCH_ENABLE_MEMORY_HOTREMOVE must not be
 	 * unlocked yet.
 	 */
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 }
 #endif
arch/ia64/mm/init.c +1 -3
index bf9df2625bc8..a6dd80a2c939 100644
@@ -689,9 +689,7 @@ void arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 }
 #endif
arch/powerpc/mm/mem.c +1 -2
index be941d382c8d..97e5922cb52e 100644
@@ -130,10 +130,9 @@ void __ref arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct page *page = pfn_to_page(start_pfn) + vmem_altmap_offset(altmap);
 	int ret;
 
-	__remove_pages(page_zone(page), start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 
 	/* Remove htab bolted mappings for this section of memory */
 	start = (unsigned long)__va(start);
arch/s390/mm/init.c +1 -3
index a124f19f7b3c..c1d96e588152 100644
@@ -291,10 +291,8 @@ void arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 	vmem_remove_mapping(start, size);
 }
 #endif /* CONFIG_MEMORY_HOTPLUG */
arch/sh/mm/init.c +1 -3
index dfdbaa50946e..d1b1ff2be17a 100644
@@ -434,9 +434,7 @@ void arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = PFN_DOWN(start);
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 }
 #endif /* CONFIG_MEMORY_HOTPLUG */
arch/x86/mm/init_32.c +1 -3
index 930edeb41ec3..0a74407ef92e 100644
@@ -865,10 +865,8 @@ void arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 }
 #endif
 
arch/x86/mm/init_64.c +1 -3
index a6b5c653727b..b8541d77452c 100644
@@ -1212,10 +1212,8 @@ void __ref arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct page *page = pfn_to_page(start_pfn) + vmem_altmap_offset(altmap);
-	struct zone *zone = page_zone(page);
 
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 	kernel_physical_mapping_remove(start, start + size);
 }
 #endif /* CONFIG_MEMORY_HOTPLUG */
include/linux/memory_hotplug.h +5 -2
index bc477e98a310..517b70943732 100644
@@ -126,8 +126,8 @@ static inline bool movable_node_is_enabled(void)
 
 extern void arch_remove_memory(int nid, u64 start, u64 size,
 			       struct vmem_altmap *altmap);
-extern void __remove_pages(struct zone *zone, unsigned long start_pfn,
-			   unsigned long nr_pages, struct vmem_altmap *altmap);
+extern void __remove_pages(unsigned long start_pfn, unsigned long nr_pages,
+			   struct vmem_altmap *altmap);
 
 /* reasonably generic interface to expand the physical pages */
 extern int __add_pages(int nid, unsigned long start_pfn, unsigned long nr_pages,
@@ -346,6 +346,9 @@ extern int add_memory(int nid, u64 start, u64 size);
 extern int add_memory_resource(int nid, struct resource *resource);
 extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,
 		unsigned long nr_pages, struct vmem_altmap *altmap);
+extern void remove_pfn_range_from_zone(struct zone *zone,
+				       unsigned long start_pfn,
+				       unsigned long nr_pages);
 extern bool is_memblock_offlined(struct memory_block *mem);
 extern int sparse_add_section(int nid, unsigned long pfn,
 		unsigned long nr_pages, struct vmem_altmap *altmap);
mm/memory_hotplug.c +16 -15
index f96608d24f6a..5b003ffa5dc9 100644
@@ -457,8 +457,9 @@ static void update_pgdat_span(struct pglist_data *pgdat)
 	pgdat->node_spanned_pages = node_end_pfn - node_start_pfn;
 }
 
-static void __remove_zone(struct zone *zone, unsigned long start_pfn,
-		unsigned long nr_pages)
+void __ref remove_pfn_range_from_zone(struct zone *zone,
+				      unsigned long start_pfn,
+				      unsigned long nr_pages)
 {
 	struct pglist_data *pgdat = zone->zone_pgdat;
 	unsigned long flags;
@@ -473,28 +474,30 @@ static void __remove_zone(struct zone *zone, unsigned long start_pfn,
 		return;
 #endif
 
+	clear_zone_contiguous(zone);
+
 	pgdat_resize_lock(zone->zone_pgdat, &flags);
 	shrink_zone_span(zone, start_pfn, start_pfn + nr_pages);
 	update_pgdat_span(pgdat);
 	pgdat_resize_unlock(zone->zone_pgdat, &flags);
+
+	set_zone_contiguous(zone);
 }
 
-static void __remove_section(struct zone *zone, unsigned long pfn,
-		unsigned long nr_pages, unsigned long map_offset,
-		struct vmem_altmap *altmap)
+static void __remove_section(unsigned long pfn, unsigned long nr_pages,
+			     unsigned long map_offset,
+			     struct vmem_altmap *altmap)
 {
 	struct mem_section *ms = __nr_to_section(pfn_to_section_nr(pfn));
 
 	if (WARN_ON_ONCE(!valid_section(ms)))
 		return;
 
-	__remove_zone(zone, pfn, nr_pages);
 	sparse_remove_section(ms, pfn, nr_pages, map_offset, altmap);
 }
 
 /**
- * __remove_pages() - remove sections of pages from a zone
- * @zone: zone from which pages need to be removed
+ * __remove_pages() - remove sections of pages
  * @pfn: starting pageframe (must be aligned to start of a section)
  * @nr_pages: number of pages to remove (must be multiple of section size)
  * @altmap: alternative device page map or %NULL if default memmap is used
@@ -504,16 +507,14 @@ static void __remove_section(struct zone *zone, unsigned long pfn,
  * sure that pages are marked reserved and zones are adjust properly by
  * calling offline_pages().
  */
-void __remove_pages(struct zone *zone, unsigned long pfn,
-		    unsigned long nr_pages, struct vmem_altmap *altmap)
+void __remove_pages(unsigned long pfn, unsigned long nr_pages,
+		    struct vmem_altmap *altmap)
 {
 	unsigned long map_offset = 0;
 	unsigned long nr, start_sec, end_sec;
 
 	map_offset = vmem_altmap_offset(altmap);
 
-	clear_zone_contiguous(zone);
-
 	if (check_pfn_span(pfn, nr_pages, "remove"))
 		return;
 
@@ -525,13 +526,11 @@ void __remove_pages(struct zone *zone, unsigned long pfn,
 		cond_resched();
 		pfns = min(nr_pages, PAGES_PER_SECTION
 				- (pfn & ~PAGE_SECTION_MASK));
-		__remove_section(zone, pfn, pfns, map_offset, altmap);
+		__remove_section(pfn, pfns, map_offset, altmap);
 		pfn += pfns;
 		nr_pages -= pfns;
 		map_offset = 0;
 	}
-
-	set_zone_contiguous(zone);
 }
 
 int set_online_page_callback(online_page_callback_t callback)
@@ -859,6 +858,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
 		 (unsigned long long) pfn << PAGE_SHIFT,
 		 (((unsigned long long) pfn + nr_pages) << PAGE_SHIFT) - 1);
 	memory_notify(MEM_CANCEL_ONLINE, &arg);
+	remove_pfn_range_from_zone(zone, pfn, nr_pages);
 	mem_hotplug_done();
 	return ret;
 }
@@ -1605,6 +1605,7 @@ static int __ref __offline_pages(unsigned long start_pfn,
 	writeback_set_ratelimit();
 
 	memory_notify(MEM_OFFLINE, &arg);
+	remove_pfn_range_from_zone(zone, start_pfn, nr_pages);
 	mem_hotplug_done();
 	return 0;
 
mm/memremap.c +1 -1
index 8c2fb44c3b4d..70263e6f093e 100644
@@ -140,7 +140,7 @@ void memunmap_pages(struct dev_pagemap *pgmap)
 
 	mem_hotplug_begin();
 	if (pgmap->type == MEMORY_DEVICE_PRIVATE) {
-		__remove_pages(page_zone(first_page), PHYS_PFN(res->start),
+		__remove_pages(PHYS_PFN(res->start),
 			       PHYS_PFN(resource_size(res)), NULL);
 	} else {
 		arch_remove_memory(nid, res->start, resource_size(res),
Signature · 1 line
2.21.0
#14 lore raw

Re: [PATCH v6 05/10] mm/memory_hotplug: Shrink zones when offlining memory

Quoted 100 lines · David Hildenbrand

We currently try to shrink a single zone when removing memory. We use the zone of the first page of the memory we are removing. If that memmap was never initialized (e.g., memory was never onlined), we will read garbage and can trigger kernel BUGs (due to a stale pointer):

:/# [   23.912993] BUG: unable to handle page fault for address: 000000000000353d
[   23.914219] #PF: supervisor write access in kernel mode
[   23.915199] #PF: error_code(0x0002) - not-present page
[   23.916160] PGD 0 P4D 0
[   23.916627] Oops: 0002 [#1] SMP PTI
[   23.917256] CPU: 1 PID: 7 Comm: kworker/u8:0 Not tainted 5.3.0-rc5-next-20190820+ #317
[   23.918900] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.4
[   23.921194] Workqueue: kacpi_hotplug acpi_hotplug_work_fn
[   23.922249] RIP: 0010:clear_zone_contiguous+0x5/0x10
[   23.923173] Code: 48 89 c6 48 89 c3 e8 2a fe ff ff 48 85 c0 75 cf 5b 5d c3 c6 85 fd 05 00 00 01 5b 5d c3 0f 1f 840
[   23.926876] RSP: 0018:ffffad2400043c98 EFLAGS: 00010246
[   23.927928] RAX: 0000000000000000 RBX: 0000000200000000 RCX: 0000000000000000
[   23.929458] RDX: 0000000000200000 RSI: 0000000000140000 RDI: 0000000000002f40
[   23.930899] RBP: 0000000140000000 R08: 0000000000000000 R09: 0000000000000001
[   23.932362] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000140000
[   23.933603] R13: 0000000000140000 R14: 0000000000002f40 R15: ffff9e3e7aff3680
[   23.934913] FS:  0000000000000000(0000) GS:ffff9e3e7bb00000(0000) knlGS:0000000000000000
[   23.936294] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   23.937481] CR2: 000000000000353d CR3: 0000000058610000 CR4: 00000000000006e0
[   23.938687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   23.939889] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   23.941168] Call Trace:
[   23.941580]  __remove_pages+0x4b/0x640
[   23.942303]  ? mark_held_locks+0x49/0x70
[   23.943149]  arch_remove_memory+0x63/0x8d
[   23.943921]  try_remove_memory+0xdb/0x130
[   23.944766]  ? walk_memory_blocks+0x7f/0x9e
[   23.945616]  __remove_memory+0xa/0x11
[   23.946274]  acpi_memory_device_remove+0x70/0x100
[   23.947308]  acpi_bus_trim+0x55/0x90
[   23.947914]  acpi_device_hotplug+0x227/0x3a0
[   23.948714]  acpi_hotplug_work_fn+0x1a/0x30
[   23.949433]  process_one_work+0x221/0x550
[   23.950190]  worker_thread+0x50/0x3b0
[   23.950993]  kthread+0x105/0x140
[   23.951644]  ? process_one_work+0x550/0x550
[   23.952508]  ? kthread_park+0x80/0x80
[   23.953367]  ret_from_fork+0x3a/0x50
[   23.954025] Modules linked in:
[   23.954613] CR2: 000000000000353d
[   23.955248] ---[ end trace 93d982b1fb3e1a69 ]---

Instead, shrink the zones when offlining memory or when onlining failed. Introduce and use remove_pfn_range_from_zone(() for that. We now properly shrink the zones, even if we have DIMMs whereby
- Some memory blocks fall into no zone (never onlined)
- Some memory blocks fall into multiple zones (offlined+re-onlined)
- Multiple memory blocks that fall into different zones

Drop the zone parameter (with a potential dubious value) from __remove_pages() and __remove_section().

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Tony Luck <[email protected]>
Cc Fenghua Yu <[email protected]>
Cc Benjamin Herrenschmidt <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Vasily Gorbik <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Yoshinori Sato <[email protected]>
Cc Rich Felker <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Steve Capper <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Anshuman Khandual <[email protected]>
Cc Yu Zhao <[email protected]>
Cc Jun Yao <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc "Matthew Wilcox (Oracle)" <[email protected]>
Cc Christophe Leroy <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Gerald Schaefer <[email protected]>
Cc Halil Pasic <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Masahiro Yamada <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
Cc Qian Cai <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Quoted 8 lines
Cc Logan Gunthorpe <[email protected]>
Cc Ira Weiny <[email protected]>
Fixes d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug")

@Andrew, can you convert that to

Fixes f1dd2cd13c4b ("mm, memory_hotplug: do not associate hotadded memory to zones until online") # visible after d0dc12e86b319

While adding cc'ing [email protected] # v4.13+ would be nice, I doubt it will be easily possible to backport, as we are missing some prereq patches (e.g., from Oscar like 2c2a5af6fed2 ("mm,
memory_hotplug: add nid parameter to arch_remove_memory")). But, it could be done with some work.

I think "Cc: [email protected] # v5.0+" could be done more easily. Maybe it's okay to not cc:stable this one. We usually online all memory (except s390x), however, s390x does not remove that memory ever. Devmem with driver reserved memory would be, however, worth backporting this.

Thoughts?

Signature · 3 lines
Thanks,

David / dhildenb
maintainer
#15 lore raw

Re: [PATCH v6 05/10] mm/memory_hotplug: Shrink zones when offlining memory

Quoted 5 lines · On Mon, 14 Oct 2019 11:39:13 +0200 David Hildenbrand <[email protected]> wrote:
Quoted 1 line
Fixes d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug")

@Andrew, can you convert that to

Fixes f1dd2cd13c4b ("mm, memory_hotplug: do not associate hotadded memory to zones until online") # visible after d0dc12e86b319

Done.

Quoted 11 lines

While adding cc'ing [email protected] # v4.13+ would be nice, I doubt it will be easily possible to backport, as we are missing some prereq patches (e.g., from Oscar like 2c2a5af6fed2 ("mm,
memory_hotplug: add nid parameter to arch_remove_memory")). But, it could be done with some work.

I think "Cc: [email protected] # v5.0+" could be done more easily. Maybe it's okay to not cc:stable this one. We usually online all memory (except s390x), however, s390x does not remove that memory ever. Devmem with driver reserved memory would be, however, worth backporting this.

I added

Cc <[email protected]> [5.0+]
#16 lore raw

Re: [PATCH v6 05/10] mm/memory_hotplug: Shrink zones when offlining memory

In reply to David Hildenbrand
Quoted 371 lines · David Hildenbrand

We currently try to shrink a single zone when removing memory. We use the zone of the first page of the memory we are removing. If that memmap was never initialized (e.g., memory was never onlined), we will read garbage and can trigger kernel BUGs (due to a stale pointer):

:/# [   23.912993] BUG: unable to handle page fault for address: 000000000000353d
[   23.914219] #PF: supervisor write access in kernel mode
[   23.915199] #PF: error_code(0x0002) - not-present page
[   23.916160] PGD 0 P4D 0
[   23.916627] Oops: 0002 [#1] SMP PTI
[   23.917256] CPU: 1 PID: 7 Comm: kworker/u8:0 Not tainted 5.3.0-rc5-next-20190820+ #317
[   23.918900] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.12.1-0-ga5cab58e9a3f-prebuilt.qemu.4
[   23.921194] Workqueue: kacpi_hotplug acpi_hotplug_work_fn
[   23.922249] RIP: 0010:clear_zone_contiguous+0x5/0x10
[   23.923173] Code: 48 89 c6 48 89 c3 e8 2a fe ff ff 48 85 c0 75 cf 5b 5d c3 c6 85 fd 05 00 00 01 5b 5d c3 0f 1f 840
[   23.926876] RSP: 0018:ffffad2400043c98 EFLAGS: 00010246
[   23.927928] RAX: 0000000000000000 RBX: 0000000200000000 RCX: 0000000000000000
[   23.929458] RDX: 0000000000200000 RSI: 0000000000140000 RDI: 0000000000002f40
[   23.930899] RBP: 0000000140000000 R08: 0000000000000000 R09: 0000000000000001
[   23.932362] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000140000
[   23.933603] R13: 0000000000140000 R14: 0000000000002f40 R15: ffff9e3e7aff3680
[   23.934913] FS:  0000000000000000(0000) GS:ffff9e3e7bb00000(0000) knlGS:0000000000000000
[   23.936294] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[   23.937481] CR2: 000000000000353d CR3: 0000000058610000 CR4: 00000000000006e0
[   23.938687] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[   23.939889] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[   23.941168] Call Trace:
[   23.941580]  __remove_pages+0x4b/0x640
[   23.942303]  ? mark_held_locks+0x49/0x70
[   23.943149]  arch_remove_memory+0x63/0x8d
[   23.943921]  try_remove_memory+0xdb/0x130
[   23.944766]  ? walk_memory_blocks+0x7f/0x9e
[   23.945616]  __remove_memory+0xa/0x11
[   23.946274]  acpi_memory_device_remove+0x70/0x100
[   23.947308]  acpi_bus_trim+0x55/0x90
[   23.947914]  acpi_device_hotplug+0x227/0x3a0
[   23.948714]  acpi_hotplug_work_fn+0x1a/0x30
[   23.949433]  process_one_work+0x221/0x550
[   23.950190]  worker_thread+0x50/0x3b0
[   23.950993]  kthread+0x105/0x140
[   23.951644]  ? process_one_work+0x550/0x550
[   23.952508]  ? kthread_park+0x80/0x80
[   23.953367]  ret_from_fork+0x3a/0x50
[   23.954025] Modules linked in:
[   23.954613] CR2: 000000000000353d
[   23.955248] ---[ end trace 93d982b1fb3e1a69 ]---

Instead, shrink the zones when offlining memory or when onlining failed. Introduce and use remove_pfn_range_from_zone(() for that. We now properly shrink the zones, even if we have DIMMs whereby
- Some memory blocks fall into no zone (never onlined)
- Some memory blocks fall into multiple zones (offlined+re-onlined)
- Multiple memory blocks that fall into different zones

Drop the zone parameter (with a potential dubious value) from __remove_pages() and __remove_section().

Cc Catalin Marinas <[email protected]>
Cc Will Deacon <[email protected]>
Cc Tony Luck <[email protected]>
Cc Fenghua Yu <[email protected]>
Cc Benjamin Herrenschmidt <[email protected]>
Cc Paul Mackerras <[email protected]>
Cc Michael Ellerman <[email protected]>
Cc Heiko Carstens <[email protected]>
Cc Vasily Gorbik <[email protected]>
Cc Christian Borntraeger <[email protected]>
Cc Yoshinori Sato <[email protected]>
Cc Rich Felker <[email protected]>
Cc Dave Hansen <[email protected]>
Cc Andy Lutomirski <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Thomas Gleixner <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Borislav Petkov <[email protected]>
Cc "H. Peter Anvin" <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Mark Rutland <[email protected]>
Cc Steve Capper <[email protected]>
Cc Mike Rapoport <[email protected]>
Cc Anshuman Khandual <[email protected]>
Cc Yu Zhao <[email protected]>
Cc Jun Yao <[email protected]>
Cc Robin Murphy <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc "Matthew Wilcox (Oracle)" <[email protected]>
Cc Christophe Leroy <[email protected]>
Cc "Aneesh Kumar K.V" <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Gerald Schaefer <[email protected]>
Cc Halil Pasic <[email protected]>
Cc Tom Lendacky <[email protected]>
Cc Greg Kroah-Hartman <[email protected]>
Cc Masahiro Yamada <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
Cc Qian Cai <[email protected]>
Cc Jason Gunthorpe <[email protected]>
Cc Logan Gunthorpe <[email protected]>
Cc Ira Weiny <[email protected]>
Fixes d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug")
---
  arch/arm64/mm/mmu.c            |  4 +---
  arch/ia64/mm/init.c            |  4 +---
  arch/powerpc/mm/mem.c          |  3 +--
  arch/s390/mm/init.c            |  4 +---
  arch/sh/mm/init.c              |  4 +---
  arch/x86/mm/init_32.c          |  4 +---
  arch/x86/mm/init_64.c          |  4 +---
  include/linux/memory_hotplug.h |  7 +++++--
  mm/memory_hotplug.c            | 31 ++++++++++++++++---------------
  mm/memremap.c                  |  2 +-
  10 files changed, 29 insertions(+), 38 deletions(-)
arch/arm64/mm/mmu.c +1 -3
index 60c929f3683b..d10247fab0fd 100644
@@ -1069,7 +1069,6 @@ void arch_remove_memory(int nid, u64 start, u64 size,
  {
  	unsigned long start_pfn = start >> PAGE_SHIFT;
  	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
  	/*
  	 * FIXME: Cleanup page tables (also in arch_add_memory() in case
@@ -1078,7 +1077,6 @@ void arch_remove_memory(int nid, u64 start, u64 size,
  	 * unplug. ARCH_ENABLE_MEMORY_HOTREMOVE must not be
  	 * unlocked yet.
  	 */
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
  }
  #endif
arch/ia64/mm/init.c +1 -3
index bf9df2625bc8..a6dd80a2c939 100644
@@ -689,9 +689,7 @@ void arch_remove_memory(int nid, u64 start, u64 size,
  {
  	unsigned long start_pfn = start >> PAGE_SHIFT;
  	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
  }
  #endif
arch/powerpc/mm/mem.c +1 -2
index be941d382c8d..97e5922cb52e 100644
@@ -130,10 +130,9 @@ void __ref arch_remove_memory(int nid, u64 start, u64 size,
  {
  	unsigned long start_pfn = start >> PAGE_SHIFT;
  	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct page *page = pfn_to_page(start_pfn) + vmem_altmap_offset(altmap);
  	int ret;
 
-	__remove_pages(page_zone(page), start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 
  	/* Remove htab bolted mappings for this section of memory */
  	start = (unsigned long)__va(start);
arch/s390/mm/init.c +1 -3
index a124f19f7b3c..c1d96e588152 100644
@@ -291,10 +291,8 @@ void arch_remove_memory(int nid, u64 start, u64 size,
  {
  	unsigned long start_pfn = start >> PAGE_SHIFT;
  	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
  	vmem_remove_mapping(start, size);
  }
  #endif /* CONFIG_MEMORY_HOTPLUG */
arch/sh/mm/init.c +1 -3
index dfdbaa50946e..d1b1ff2be17a 100644
@@ -434,9 +434,7 @@ void arch_remove_memory(int nid, u64 start, u64 size,
  {
  	unsigned long start_pfn = PFN_DOWN(start);
  	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
  }
  #endif /* CONFIG_MEMORY_HOTPLUG */
arch/x86/mm/init_32.c +1 -3
index 930edeb41ec3..0a74407ef92e 100644
@@ -865,10 +865,8 @@ void arch_remove_memory(int nid, u64 start, u64 size,
  {
  	unsigned long start_pfn = start >> PAGE_SHIFT;
  	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
  }
  #endif
 
arch/x86/mm/init_64.c +1 -3
index a6b5c653727b..b8541d77452c 100644
@@ -1212,10 +1212,8 @@ void __ref arch_remove_memory(int nid, u64 start, u64 size,
  {
  	unsigned long start_pfn = start >> PAGE_SHIFT;
  	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct page *page = pfn_to_page(start_pfn) + vmem_altmap_offset(altmap);
-	struct zone *zone = page_zone(page);
 
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
  	kernel_physical_mapping_remove(start, start + size);
  }
  #endif /* CONFIG_MEMORY_HOTPLUG */
include/linux/memory_hotplug.h +5 -2
index bc477e98a310..517b70943732 100644
@@ -126,8 +126,8 @@ static inline bool movable_node_is_enabled(void)
 
  extern void arch_remove_memory(int nid, u64 start, u64 size,
  			       struct vmem_altmap *altmap);
-extern void __remove_pages(struct zone *zone, unsigned long start_pfn,
-			   unsigned long nr_pages, struct vmem_altmap *altmap);
+extern void __remove_pages(unsigned long start_pfn, unsigned long nr_pages,
+			   struct vmem_altmap *altmap);
 
  /* reasonably generic interface to expand the physical pages */
  extern int __add_pages(int nid, unsigned long start_pfn, unsigned long nr_pages,
@@ -346,6 +346,9 @@ extern int add_memory(int nid, u64 start, u64 size);
  extern int add_memory_resource(int nid, struct resource *resource);
  extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,
  		unsigned long nr_pages, struct vmem_altmap *altmap);
+extern void remove_pfn_range_from_zone(struct zone *zone,
+				       unsigned long start_pfn,
+				       unsigned long nr_pages);
  extern bool is_memblock_offlined(struct memory_block *mem);
  extern int sparse_add_section(int nid, unsigned long pfn,
  		unsigned long nr_pages, struct vmem_altmap *altmap);
mm/memory_hotplug.c +16 -15
index f96608d24f6a..5b003ffa5dc9 100644
@@ -457,8 +457,9 @@ static void update_pgdat_span(struct pglist_data *pgdat)
  	pgdat->node_spanned_pages = node_end_pfn - node_start_pfn;
  }
 
-static void __remove_zone(struct zone *zone, unsigned long start_pfn,
-		unsigned long nr_pages)
+void __ref remove_pfn_range_from_zone(struct zone *zone,
+				      unsigned long start_pfn,
+				      unsigned long nr_pages)
  {
  	struct pglist_data *pgdat = zone->zone_pgdat;
  	unsigned long flags;
@@ -473,28 +474,30 @@ static void __remove_zone(struct zone *zone, unsigned long start_pfn,
  		return;
  #endif
 
+	clear_zone_contiguous(zone);
+
  	pgdat_resize_lock(zone->zone_pgdat, &flags);
  	shrink_zone_span(zone, start_pfn, start_pfn + nr_pages);
  	update_pgdat_span(pgdat);
  	pgdat_resize_unlock(zone->zone_pgdat, &flags);
+
+	set_zone_contiguous(zone);
  }
 
-static void __remove_section(struct zone *zone, unsigned long pfn,
-		unsigned long nr_pages, unsigned long map_offset,
-		struct vmem_altmap *altmap)
+static void __remove_section(unsigned long pfn, unsigned long nr_pages,
+			     unsigned long map_offset,
+			     struct vmem_altmap *altmap)
  {
  	struct mem_section *ms = __nr_to_section(pfn_to_section_nr(pfn));
 
  	if (WARN_ON_ONCE(!valid_section(ms)))
  		return;
 
-	__remove_zone(zone, pfn, nr_pages);
  	sparse_remove_section(ms, pfn, nr_pages, map_offset, altmap);
  }
 
  /**
- * __remove_pages() - remove sections of pages from a zone
- * @zone: zone from which pages need to be removed
+ * __remove_pages() - remove sections of pages
   * @pfn: starting pageframe (must be aligned to start of a section)
   * @nr_pages: number of pages to remove (must be multiple of section size)
   * @altmap: alternative device page map or %NULL if default memmap is used
@@ -504,16 +507,14 @@ static void __remove_section(struct zone *zone, unsigned long pfn,
   * sure that pages are marked reserved and zones are adjust properly by
   * calling offline_pages().
   */
-void __remove_pages(struct zone *zone, unsigned long pfn,
-		    unsigned long nr_pages, struct vmem_altmap *altmap)
+void __remove_pages(unsigned long pfn, unsigned long nr_pages,
+		    struct vmem_altmap *altmap)
  {
  	unsigned long map_offset = 0;
  	unsigned long nr, start_sec, end_sec;
 
  	map_offset = vmem_altmap_offset(altmap);
 
-	clear_zone_contiguous(zone);
-
  	if (check_pfn_span(pfn, nr_pages, "remove"))
  		return;
 
@@ -525,13 +526,11 @@ void __remove_pages(struct zone *zone, unsigned long pfn,
  		cond_resched();
  		pfns = min(nr_pages, PAGES_PER_SECTION
  				- (pfn & ~PAGE_SECTION_MASK));
-		__remove_section(zone, pfn, pfns, map_offset, altmap);
+		__remove_section(pfn, pfns, map_offset, altmap);
  		pfn += pfns;
  		nr_pages -= pfns;
  		map_offset = 0;
  	}
-
-	set_zone_contiguous(zone);
  }
 
  int set_online_page_callback(online_page_callback_t callback)
@@ -859,6 +858,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
  		 (unsigned long long) pfn << PAGE_SHIFT,
  		 (((unsigned long long) pfn + nr_pages) << PAGE_SHIFT) - 1);
  	memory_notify(MEM_CANCEL_ONLINE, &arg);
+	remove_pfn_range_from_zone(zone, pfn, nr_pages);
  	mem_hotplug_done();
  	return ret;
  }
@@ -1605,6 +1605,7 @@ static int __ref __offline_pages(unsigned long start_pfn,
  	writeback_set_ratelimit();
 
  	memory_notify(MEM_OFFLINE, &arg);
+	remove_pfn_range_from_zone(zone, start_pfn, nr_pages);
  	mem_hotplug_done();
  	return 0;
 
mm/memremap.c +0 -0
index 8c2fb44c3b4d..70263e6f093e 100644

mm/memremap.c void memunmap_pages(struct dev_pagemap *pgmap) · line 146

Quoted 8 lines
mm/memremap.c +1 -1
@@ -140,7 +140,7 @@ void memunmap_pages(struct dev_pagemap *pgmap)
 
  	mem_hotplug_begin();
  	if (pgmap->type == MEMORY_DEVICE_PRIVATE) {
-		__remove_pages(page_zone(first_page), PHYS_PFN(res->start),
+		__remove_pages(PHYS_PFN(res->start),
  			       PHYS_PFN(resource_size(res)), NULL);
  	} else {
  		arch_remove_memory(nid, res->start, resource_size(res),

I think I just found an issue with try_offline_node(). try_offline_node() is pretty much broken already (touches garbage memmaps and will not considers mixed NIDs within sections), however, relies on the node span to look for memory sections to probe. So it seems to rely on the nodes getting shrunk when removing memory, not when offlining.

As we shrink the node span when offlining now and not when removing, this can go wrong once we offline the last memory block of the node and offline the last CPU. We could still have memory around that we could re-online, however, the node would already be offline. Unlikely, but possible.

Note that the same is also broken without this patch in case memory is never onlined. The "pfn_to_nid(pfn) != nid" can easily succeed on the garbage memmap, resulting in no memory being detected as belonging to the node. Also, resize_pgdat_range() is called when onlining memory, not when adding it. :/ Oh this is so broken :)

The right fix is probably to walk over all memory blocks that could exist and test if they belong to the nid (if offline, check the block->nid, if online check all pageblocks). A fix we can then move in front of this patch.

Will look into this this week.

Signature · 3 lines
Thanks,

David / dhildenb
maintainer
#17 lore raw

Re: [PATCH v6 05/10] mm/memory_hotplug: Shrink zones when offlining memory

Quoted 19 lines · On Sun, 27 Oct 2019 23:45:52 +0100 David Hildenbrand <[email protected]> wrote:

I think I just found an issue with try_offline_node(). try_offline_node() is pretty much broken already (touches garbage memmaps and will not considers mixed NIDs within sections), however, relies on the node span to look for memory sections to probe. So it seems to rely on the nodes getting shrunk when removing memory, not when offlining.

As we shrink the node span when offlining now and not when removing, this can go wrong once we offline the last memory block of the node and offline the last CPU. We could still have memory around that we could re-online, however, the node would already be offline. Unlikely, but possible.

Note that the same is also broken without this patch in case memory is never onlined. The "pfn_to_nid(pfn) != nid" can easily succeed on the garbage memmap, resulting in no memory being detected as belonging to the node. Also, resize_pgdat_range() is called when onlining memory, not when adding it. :/ Oh this is so broken :)

Quoted 6 lines

The right fix is probably to walk over all memory blocks that could exist and test if they belong to the nid (if offline, check the block->nid, if online check all pageblocks). A fix we can then move in front of this patch.

Will look into this this week.

And this series shows almost no sign of having been reviewed. I'll hold it over for 5.6.

#18 lore raw

Re: [PATCH v6 05/10] mm/memory_hotplug: Shrink zones when offlining memory

Quoted 30 lines

Am 01.12.2019 um 00:22 schrieb Andrew Morton <[email protected]>:

Quoted 24 lines · On Sun, 27 Oct 2019 23:45:52 +0100 David Hildenbrand <[email protected]> wrote:

I think I just found an issue with try_offline_node(). try_offline_node() is pretty much broken already (touches garbage memmaps and will not considers mixed NIDs within sections), however, relies on the node span to look for memory sections to probe. So it seems to rely on the nodes getting shrunk when removing memory, not when offlining.

As we shrink the node span when offlining now and not when removing, this can go wrong once we offline the last memory block of the node and offline the last CPU. We could still have memory around that we could re-online, however, the node would already be offline. Unlikely, but possible.

Note that the same is also broken without this patch in case memory is never onlined. The "pfn_to_nid(pfn) != nid" can easily succeed on the garbage memmap, resulting in no memory being detected as belonging to the node. Also, resize_pgdat_range() is called when onlining memory, not when adding it. :/ Oh this is so broken :)

The right fix is probably to walk over all memory blocks that could exist and test if they belong to the nid (if offline, check the block->nid, if online check all pageblocks). A fix we can then move in front of this patch.

Quoted 5 lines
Quoted 1 line

Will look into this this week.

And this series shows almost no sign of having been reviewed. I'll hold it over for 5.6.

Makes sense, can‘t do anything about it. Btw, this one is the last stable patch to fix access of uninitialized memmaps that is not upstream yet... so it has to remain broken for some longer.

#19 lore raw

Re: [PATCH v6 05/10] mm/memory_hotplug: Shrink zones when offlining memory

In reply to Andrew Morton
Quoted 26 lines · Andrew Morton
Quoted 24 lines · On Sun, 27 Oct 2019 23:45:52 +0100 David Hildenbrand <[email protected]> wrote:

I think I just found an issue with try_offline_node(). try_offline_node() is pretty much broken already (touches garbage memmaps and will not considers mixed NIDs within sections), however, relies on the node span to look for memory sections to probe. So it seems to rely on the nodes getting shrunk when removing memory, not when offlining.

As we shrink the node span when offlining now and not when removing, this can go wrong once we offline the last memory block of the node and offline the last CPU. We could still have memory around that we could re-online, however, the node would already be offline. Unlikely, but possible.

Note that the same is also broken without this patch in case memory is never onlined. The "pfn_to_nid(pfn) != nid" can easily succeed on the garbage memmap, resulting in no memory being detected as belonging to the node. Also, resize_pgdat_range() is called when onlining memory, not when adding it. :/ Oh this is so broken :)

The right fix is probably to walk over all memory blocks that could exist and test if they belong to the nid (if offline, check the block->nid, if online check all pageblocks). A fix we can then move in front of this patch.

Quoted 5 lines
Quoted 1 line

Will look into this this week.

And this series shows almost no sign of having been reviewed. I'll hold it over for 5.6.

Hi Andrew, any chance we can get the (now at least reviewed - thx Oscar) fix in patch #5 into 5.5? (I want to do the final stable backports for the uninitialized memmap stuff)

Signature · 3 lines
Thanks,

David / dhildenb
maintainer
#20 lore raw

Re: [PATCH v6 05/10] mm/memory_hotplug: Shrink zones when offlining memory

Quoted 29 lines · On Wed, 18 Dec 2019 18:08:04 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 28 lines · Andrew Morton
Quoted 25 lines · On Sun, 27 Oct 2019 23:45:52 +0100 David Hildenbrand <[email protected]> wrote:

I think I just found an issue with try_offline_node(). try_offline_node() is pretty much broken already (touches garbage memmaps and will not considers mixed NIDs within sections), however, relies on the node span to look for memory sections to probe. So it seems to rely on the nodes getting shrunk when removing memory, not when offlining.

As we shrink the node span when offlining now and not when removing, this can go wrong once we offline the last memory block of the node and offline the last CPU. We could still have memory around that we could re-online, however, the node would already be offline. Unlikely, but possible.

Note that the same is also broken without this patch in case memory is never onlined. The "pfn_to_nid(pfn) != nid" can easily succeed on the garbage memmap, resulting in no memory being detected as belonging to the node. Also, resize_pgdat_range() is called when onlining memory, not when adding it. :/ Oh this is so broken :)

The right fix is probably to walk over all memory blocks that could exist and test if they belong to the nid (if offline, check the block->nid, if online check all pageblocks). A fix we can then move in front of this patch.

Will look into this this week.

Quoted 7 lines
Quoted 3 lines

And this series shows almost no sign of having been reviewed. I'll hold it over for 5.6.

Hi Andrew, any chance we can get the (now at least reviewed - thx Oscar) fix in patch #5 into 5.5? (I want to do the final stable backports for the uninitialized memmap stuff)

Sure, I queued it for the next batch of 5.5 fixes.

maintainer Reviewed-by on v6 5/10
#21 lore raw

Re: [PATCH v6 05/10] mm/memory_hotplug: Shrink zones when offlining memory

In reply to David Hildenbrand
Quoted 2 lines · David Hildenbrand
Fixes d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug")

I did not see anything wrong with the taken approach, and makes sense to me. The only thing that puzzles me is we seem to not balance spanned_pages for ZONE_DEVICE anymore.
memremap_pages() increments them via move_pfn_range_to_zone, but we skip ZONE_DEVICE in remove_pfn_range_from_zone.

That is not really related to this patch, so I might be missing something, but it caught my eye while reviewing this.

Anyway, for this one:

off-topic: I __think__ we really need to trim the CC list.

Quoted 272 lines
---
 arch/arm64/mm/mmu.c            |  4 +---
 arch/ia64/mm/init.c            |  4 +---
 arch/powerpc/mm/mem.c          |  3 +--
 arch/s390/mm/init.c            |  4 +---
 arch/sh/mm/init.c              |  4 +---
 arch/x86/mm/init_32.c          |  4 +---
 arch/x86/mm/init_64.c          |  4 +---
 include/linux/memory_hotplug.h |  7 +++++--
 mm/memory_hotplug.c            | 31 ++++++++++++++++---------------
 mm/memremap.c                  |  2 +-
 10 files changed, 29 insertions(+), 38 deletions(-)
arch/arm64/mm/mmu.c +1 -3
index 60c929f3683b..d10247fab0fd 100644
@@ -1069,7 +1069,6 @@ void arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
 	/*
 	 * FIXME: Cleanup page tables (also in arch_add_memory() in case
@@ -1078,7 +1077,6 @@ void arch_remove_memory(int nid, u64 start, u64 size,
 	 * unplug. ARCH_ENABLE_MEMORY_HOTREMOVE must not be
 	 * unlocked yet.
 	 */
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 }
 #endif
arch/ia64/mm/init.c +1 -3
index bf9df2625bc8..a6dd80a2c939 100644
@@ -689,9 +689,7 @@ void arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 }
 #endif
arch/powerpc/mm/mem.c +1 -2
index be941d382c8d..97e5922cb52e 100644
@@ -130,10 +130,9 @@ void __ref arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct page *page = pfn_to_page(start_pfn) + vmem_altmap_offset(altmap);
 	int ret;
 
-	__remove_pages(page_zone(page), start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 
 	/* Remove htab bolted mappings for this section of memory */
 	start = (unsigned long)__va(start);
arch/s390/mm/init.c +1 -3
index a124f19f7b3c..c1d96e588152 100644
@@ -291,10 +291,8 @@ void arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 	vmem_remove_mapping(start, size);
 }
 #endif /* CONFIG_MEMORY_HOTPLUG */
arch/sh/mm/init.c +1 -3
index dfdbaa50946e..d1b1ff2be17a 100644
@@ -434,9 +434,7 @@ void arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = PFN_DOWN(start);
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 }
 #endif /* CONFIG_MEMORY_HOTPLUG */
arch/x86/mm/init_32.c +1 -3
index 930edeb41ec3..0a74407ef92e 100644
@@ -865,10 +865,8 @@ void arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct zone *zone;
 
-	zone = page_zone(pfn_to_page(start_pfn));
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 }
 #endif
 
arch/x86/mm/init_64.c +1 -3
index a6b5c653727b..b8541d77452c 100644
@@ -1212,10 +1212,8 @@ void __ref arch_remove_memory(int nid, u64 start, u64 size,
 {
 	unsigned long start_pfn = start >> PAGE_SHIFT;
 	unsigned long nr_pages = size >> PAGE_SHIFT;
-	struct page *page = pfn_to_page(start_pfn) + vmem_altmap_offset(altmap);
-	struct zone *zone = page_zone(page);
 
-	__remove_pages(zone, start_pfn, nr_pages, altmap);
+	__remove_pages(start_pfn, nr_pages, altmap);
 	kernel_physical_mapping_remove(start, start + size);
 }
 #endif /* CONFIG_MEMORY_HOTPLUG */
include/linux/memory_hotplug.h +5 -2
index bc477e98a310..517b70943732 100644
@@ -126,8 +126,8 @@ static inline bool movable_node_is_enabled(void)
 
 extern void arch_remove_memory(int nid, u64 start, u64 size,
 			       struct vmem_altmap *altmap);
-extern void __remove_pages(struct zone *zone, unsigned long start_pfn,
-			   unsigned long nr_pages, struct vmem_altmap *altmap);
+extern void __remove_pages(unsigned long start_pfn, unsigned long nr_pages,
+			   struct vmem_altmap *altmap);
 
 /* reasonably generic interface to expand the physical pages */
 extern int __add_pages(int nid, unsigned long start_pfn, unsigned long nr_pages,
@@ -346,6 +346,9 @@ extern int add_memory(int nid, u64 start, u64 size);
 extern int add_memory_resource(int nid, struct resource *resource);
 extern void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,
 		unsigned long nr_pages, struct vmem_altmap *altmap);
+extern void remove_pfn_range_from_zone(struct zone *zone,
+				       unsigned long start_pfn,
+				       unsigned long nr_pages);
 extern bool is_memblock_offlined(struct memory_block *mem);
 extern int sparse_add_section(int nid, unsigned long pfn,
 		unsigned long nr_pages, struct vmem_altmap *altmap);
mm/memory_hotplug.c +16 -15
index f96608d24f6a..5b003ffa5dc9 100644
@@ -457,8 +457,9 @@ static void update_pgdat_span(struct pglist_data *pgdat)
 	pgdat->node_spanned_pages = node_end_pfn - node_start_pfn;
 }
 
-static void __remove_zone(struct zone *zone, unsigned long start_pfn,
-		unsigned long nr_pages)
+void __ref remove_pfn_range_from_zone(struct zone *zone,
+				      unsigned long start_pfn,
+				      unsigned long nr_pages)
 {
 	struct pglist_data *pgdat = zone->zone_pgdat;
 	unsigned long flags;
@@ -473,28 +474,30 @@ static void __remove_zone(struct zone *zone, unsigned long start_pfn,
 		return;
 #endif
 
+	clear_zone_contiguous(zone);
+
 	pgdat_resize_lock(zone->zone_pgdat, &flags);
 	shrink_zone_span(zone, start_pfn, start_pfn + nr_pages);
 	update_pgdat_span(pgdat);
 	pgdat_resize_unlock(zone->zone_pgdat, &flags);
+
+	set_zone_contiguous(zone);
 }
 
-static void __remove_section(struct zone *zone, unsigned long pfn,
-		unsigned long nr_pages, unsigned long map_offset,
-		struct vmem_altmap *altmap)
+static void __remove_section(unsigned long pfn, unsigned long nr_pages,
+			     unsigned long map_offset,
+			     struct vmem_altmap *altmap)
 {
 	struct mem_section *ms = __nr_to_section(pfn_to_section_nr(pfn));
 
 	if (WARN_ON_ONCE(!valid_section(ms)))
 		return;
 
-	__remove_zone(zone, pfn, nr_pages);
 	sparse_remove_section(ms, pfn, nr_pages, map_offset, altmap);
 }
 
 /**
- * __remove_pages() - remove sections of pages from a zone
- * @zone: zone from which pages need to be removed
+ * __remove_pages() - remove sections of pages
  * @pfn: starting pageframe (must be aligned to start of a section)
  * @nr_pages: number of pages to remove (must be multiple of section size)
  * @altmap: alternative device page map or %NULL if default memmap is used
@@ -504,16 +507,14 @@ static void __remove_section(struct zone *zone, unsigned long pfn,
  * sure that pages are marked reserved and zones are adjust properly by
  * calling offline_pages().
  */
-void __remove_pages(struct zone *zone, unsigned long pfn,
-		    unsigned long nr_pages, struct vmem_altmap *altmap)
+void __remove_pages(unsigned long pfn, unsigned long nr_pages,
+		    struct vmem_altmap *altmap)
 {
 	unsigned long map_offset = 0;
 	unsigned long nr, start_sec, end_sec;
 
 	map_offset = vmem_altmap_offset(altmap);
 
-	clear_zone_contiguous(zone);
-
 	if (check_pfn_span(pfn, nr_pages, "remove"))
 		return;
 
@@ -525,13 +526,11 @@ void __remove_pages(struct zone *zone, unsigned long pfn,
 		cond_resched();
 		pfns = min(nr_pages, PAGES_PER_SECTION
 				- (pfn & ~PAGE_SECTION_MASK));
-		__remove_section(zone, pfn, pfns, map_offset, altmap);
+		__remove_section(pfn, pfns, map_offset, altmap);
 		pfn += pfns;
 		nr_pages -= pfns;
 		map_offset = 0;
 	}
-
-	set_zone_contiguous(zone);
 }
 
 int set_online_page_callback(online_page_callback_t callback)
@@ -859,6 +858,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ
 		 (unsigned long long) pfn << PAGE_SHIFT,
 		 (((unsigned long long) pfn + nr_pages) << PAGE_SHIFT) - 1);
 	memory_notify(MEM_CANCEL_ONLINE, &arg);
+	remove_pfn_range_from_zone(zone, pfn, nr_pages);
 	mem_hotplug_done();
 	return ret;
 }
@@ -1605,6 +1605,7 @@ static int __ref __offline_pages(unsigned long start_pfn,
 	writeback_set_ratelimit();
 
 	memory_notify(MEM_OFFLINE, &arg);
+	remove_pfn_range_from_zone(zone, start_pfn, nr_pages);
 	mem_hotplug_done();
 	return 0;
 
mm/memremap.c +1 -1
index 8c2fb44c3b4d..70263e6f093e 100644
@@ -140,7 +140,7 @@ void memunmap_pages(struct dev_pagemap *pgmap)
 
 	mem_hotplug_begin();
 	if (pgmap->type == MEMORY_DEVICE_PRIVATE) {
-		__remove_pages(page_zone(first_page), PHYS_PFN(res->start),
+		__remove_pages(PHYS_PFN(res->start),
 			       PHYS_PFN(resource_size(res)), NULL);
 	} else {
 		arch_remove_memory(nid, res->start, resource_size(res),
Signature · 1 line
2.21.0
Signature · 2 lines
Oscar Salvador
SUSE L3
#22 lore raw

Re: [PATCH v6 05/10] mm/memory_hotplug: Shrink zones when offlining memory

Quoted 9 lines · Oscar Salvador
Quoted 2 lines · David Hildenbrand
Fixes d0dc12e86b31 ("mm/memory_hotplug: optimize memory hotplug")

I did not see anything wrong with the taken approach, and makes sense to me. The only thing that puzzles me is we seem to not balance spanned_pages for ZONE_DEVICE anymore.
memremap_pages() increments them via move_pfn_range_to_zone, but we skip ZONE_DEVICE in remove_pfn_range_from_zone.

Yes, documented e.g., in

commit 7ce700bf11b5e2cb84e4352bbdf2123a7a239c84

Author: David Hildenbrand <[email protected]>
Date:   Thu Nov 21 17:53:56 2019 -0800

mm/memory_hotplug: don't access uninitialized memmaps in
shrink_zone_span()

Needs some more thought - but is definitely not urgent (well, now it's at least no longer completely broken).

Quoted 8 lines

That is not really related to this patch, so I might be missing something, but it caught my eye while reviewing this.

Anyway, for this one:

Thanks!

Quoted 2 lines

off-topic: I __think__ we really need to trim the CC list.

Yes we should :) - done.

Signature · 3 lines
Thanks,

David / dhildenb
#23 lore raw

PATCH v6 6/10 mm/memory_hotplug: Poison memmap in remove_pfn_range_from_zone()

In reply to David Hildenbrand

Let's poison the pages similar to when adding new memory in sparse_add_section(). Also call remove_pfn_range_from_zone() from memunmap_pages(), so we can poison the memmap from there as well.

While at it, calculate the pfn in memunmap_pages() only once.

Cc Andrew Morton <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
---
 mm/memory_hotplug.c | 3 +++
 mm/memremap.c       | 2 ++
 2 files changed, 5 insertions(+)
mm/memory_hotplug.c +3 -0
index 5b003ffa5dc9..bf5173e7913d 100644
@@ -464,6 +464,9 @@ void __ref remove_pfn_range_from_zone(struct zone *zone,
 	struct pglist_data *pgdat = zone->zone_pgdat;
 	unsigned long flags;
 
+	/* Poison struct pages because they are now uninitialized again. */
+	page_init_poison(pfn_to_page(start_pfn), sizeof(struct page) * nr_pages);
+
 #ifdef CONFIG_ZONE_DEVICE
 	/*
 	 * Zone shrinking code cannot properly deal with ZONE_DEVICE. So
mm/memremap.c +2 -0
index 70263e6f093e..7fed8bd32a18 100644
@@ -139,6 +139,8 @@ void memunmap_pages(struct dev_pagemap *pgmap)
 	nid = page_to_nid(first_page);
 
 	mem_hotplug_begin();
+	remove_pfn_range_from_zone(page_zone(first_page), PHYS_PFN(res->start),
+				   PHYS_PFN(resource_size(res)));
 	if (pgmap->type = MEMORY_DEVICE_PRIVATE) {
 		__remove_pages(PHYS_PFN(res->start),
 			       PHYS_PFN(resource_size(res)), NULL);
Signature · 1 line
2.21.0
#24 lore raw

Re: [PATCH v6 06/10] mm/memory_hotplug: Poison memmap in remove_pfn_range_from_zone()

Quoted 5 lines · David Hildenbrand

Let's poison the pages similar to when adding new memory in sparse_add_section(). Also call remove_pfn_range_from_zone() from memunmap_pages(), so we can poison the memmap from there as well.

While at it, calculate the pfn in memunmap_pages() only once.

FWIW, this comment is stale and could be dropped :)

Signature · 3 lines
Thanks,

David / dhildenb
maintainer Reviewed-by on v6 6/10
#25 lore raw

Re: [PATCH v6 06/10] mm/memory_hotplug: Poison memmap in remove_pfn_range_from_zone()

In reply to David Hildenbrand
Quoted 6 lines · David Hildenbrand

Let's poison the pages similar to when adding new memory in sparse_add_section(). Also call remove_pfn_range_from_zone() from memunmap_pages(), so we can poison the memmap from there as well.

While at it, calculate the pfn in memunmap_pages() only once.

Quoted 7 lines
Cc Andrew Morton <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>

Looks good to me, it is fine as long as we do not access those pages later on, and if my eyes did not lie to me, we have to proper checks (pfn_to_online_page) in place to avoid that, so:

Signature · 2 lines
Oscar Salvador
SUSE L3
#26 lore raw

PATCH v6 7/10 mm/memory_hotplug: We always have a zone in find_(smallest|biggest)_section_pfn

In reply to David Hildenbrand

With shrink_pgdat_span() out of the way, we now always have a valid zone.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
---
 mm/memory_hotplug.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/memory_hotplug.c +2 -2
index bf5173e7913d..f294918f7211 100644
@@ -337,7 +337,7 @@ static unsigned long find_smallest_section_pfn(int nid, struct zone *zone,
 		if (unlikely(pfn_to_nid(start_pfn) != nid))
 			continue;
 
-		if (zone && zone != page_zone(pfn_to_page(start_pfn)))
+		if (zone != page_zone(pfn_to_page(start_pfn)))
 			continue;
 
 		return start_pfn;
@@ -362,7 +362,7 @@ static unsigned long find_biggest_section_pfn(int nid, struct zone *zone,
 		if (unlikely(pfn_to_nid(pfn) != nid))
 			continue;
 
-		if (zone && zone != page_zone(pfn_to_page(pfn)))
+		if (zone != page_zone(pfn_to_page(pfn)))
 			continue;
 
 		return pfn;
Signature · 1 line
2.21.0
maintainer Reviewed-by on v6 7/10
#27 lore raw

Re: [PATCH v6 07/10] mm/memory_hotplug: We always have a zone in find_(smallest|biggest)_section_pfn

Quoted 11 lines · David Hildenbrand

With shrink_pgdat_span() out of the way, we now always have a valid zone.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
Quoted 29 lines
---
 mm/memory_hotplug.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
mm/memory_hotplug.c +2 -2
index bf5173e7913d..f294918f7211 100644
@@ -337,7 +337,7 @@ static unsigned long find_smallest_section_pfn(int nid, struct zone *zone,
 		if (unlikely(pfn_to_nid(start_pfn) != nid))
 			continue;
 
-		if (zone && zone != page_zone(pfn_to_page(start_pfn)))
+		if (zone != page_zone(pfn_to_page(start_pfn)))
 			continue;
 
 		return start_pfn;
@@ -362,7 +362,7 @@ static unsigned long find_biggest_section_pfn(int nid, struct zone *zone,
 		if (unlikely(pfn_to_nid(pfn) != nid))
 			continue;
 
-		if (zone && zone != page_zone(pfn_to_page(pfn)))
+		if (zone != page_zone(pfn_to_page(pfn)))
 			continue;
 
 		return pfn;
Signature · 1 line
2.21.0
Signature · 2 lines
Oscar Salvador
SUSE L3
Reviewed-by on v6 7/10
#28 lore raw

Re: [PATCH v6 07/10] mm/memory_hotplug: We always have a zone in find_(smallest|biggest)_section_pfn

In reply to David Hildenbrand
Quoted 11 lines · David Hildenbrand

With shrink_pgdat_span() out of the way, we now always have a valid zone.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
Signature · 2 lines
Wei Yang
Help you, Help me
#29 lore raw

Re: [PATCH v6 07/10] mm/memory_hotplug: We always have a zone in find_(smallest|biggest)_section_pfn

Quoted 13 lines · Wei Yang
Quoted 11 lines · David Hildenbrand

With shrink_pgdat_span() out of the way, we now always have a valid zone.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
Quoted 1 line

Just FYI, the patches are now upstream, so the rb's can no longer be applied. (but we can send fixes if we find that something is broken ;) ). Thanks!

Signature · 3 lines
Thanks,

David / dhildenb
#30 lore raw

Re: [PATCH v6 07/10] mm/memory_hotplug: We always have a zone in find_(smallest|biggest)_section_pfn

Quoted 14 lines · David Hildenbrand
Quoted 13 lines · Wei Yang
Quoted 11 lines · David Hildenbrand

With shrink_pgdat_span() out of the way, we now always have a valid zone.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
Quoted 6 lines
Quoted 1 line

Just FYI, the patches are now upstream, so the rb's can no longer be applied. (but we can send fixes if we find that something is broken ;) ). Thanks!

Thanks for reminding. :-)

Quoted 4 lines
Signature · 3 lines
Thanks,

David / dhildenb
Signature · 2 lines
Wei Yang
Help you, Help me
#31 lore raw

PATCH v6 8/10 mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

In reply to David Hildenbrand

If we have holes, the holes will automatically get detected and removed once we remove the next bigger/smaller section. The extra checks can go.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
---
 mm/memory_hotplug.c | 34 +++++++---------------------------
 1 file changed, 7 insertions(+), 27 deletions(-)
mm/memory_hotplug.c +7 -27
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn = end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 					       start_pfn);
 		if (pfn)
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
+		}
 	}
-
-	/*
-	 * The section is not biggest or smallest mem_section in the zone, it
-	 * only creates a hole in the zone. So in this case, we need not
-	 * change the zone. But perhaps, the zone has only hole data. Thus
-	 * it check the zone has only hole or not.
-	 */
-	pfn = zone_start_pfn;
-	for (; pfn < zone_end_pfn; pfn += PAGES_PER_SUBSECTION) {
-		if (unlikely(!pfn_to_online_page(pfn)))
-			continue;
-
-		if (page_zone(pfn_to_page(pfn)) != zone)
-			continue;
-
-		/* Skip range to be removed */
-		if (pfn >= start_pfn && pfn < end_pfn)
-			continue;
-
-		/* If we find valid section, we have nothing to do */
-		zone_span_writeunlock(zone);
-		return;
-	}
-
-	/* The zone has no valid section */
-	zone->zone_start_pfn = 0;
-	zone->spanned_pages = 0;
 	zone_span_writeunlock(zone);
 }
 
Signature · 1 line
2.21.0
maintainer Reviewed-by on v6 8/10
#32 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 4 lines · David Hildenbrand

If we have holes, the holes will automatically get detected and removed once we remove the next bigger/smaller section. The extra checks can go.

Quoted 8 lines
Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>

Heh, I have been here before.
I have to confess that when I wrote my version of this I was not really 100% about removing it, because hotplug was a sort of a "catchall" for all sort of weird and corner-cases configurations, but thinking more about it, I cannot think of any situation that would make this blow up.

Quoted 60 lines
---
 mm/memory_hotplug.c | 34 +++++++---------------------------
 1 file changed, 7 insertions(+), 27 deletions(-)
mm/memory_hotplug.c +7 -27
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn = end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 					       start_pfn);
 		if (pfn)
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
+		}
 	}
-
-	/*
-	 * The section is not biggest or smallest mem_section in the zone, it
-	 * only creates a hole in the zone. So in this case, we need not
-	 * change the zone. But perhaps, the zone has only hole data. Thus
-	 * it check the zone has only hole or not.
-	 */
-	pfn = zone_start_pfn;
-	for (; pfn < zone_end_pfn; pfn += PAGES_PER_SUBSECTION) {
-		if (unlikely(!pfn_to_online_page(pfn)))
-			continue;
-
-		if (page_zone(pfn_to_page(pfn)) != zone)
-			continue;
-
-		/* Skip range to be removed */
-		if (pfn >= start_pfn && pfn < end_pfn)
-			continue;
-
-		/* If we find valid section, we have nothing to do */
-		zone_span_writeunlock(zone);
-		return;
-	}
-
-	/* The zone has no valid section */
-	zone->zone_start_pfn = 0;
-	zone->spanned_pages = 0;
 	zone_span_writeunlock(zone);
 }
 
Signature · 1 line
2.21.0
Signature · 2 lines
Oscar Salvador
SUSE L3
#33 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 14 lines · Oscar Salvador
Quoted 4 lines · David Hildenbrand

If we have holes, the holes will automatically get detected and removed once we remove the next bigger/smaller section. The extra checks can go.

Quoted 8 lines
Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
Quoted 7 lines

Heh, I have been here before.
I have to confess that when I wrote my version of this I was not really 100% about removing it, because hotplug was a sort of a "catchall" for all sort of weird and corner-cases configurations, but thinking more about it, I cannot think of any situation that would make this blow up.

Thanks for your review Oscar!

Signature · 3 lines
Thanks,

David / dhildenb
#34 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

In reply to David Hildenbrand
Quoted 31 lines · David Hildenbrand

If we have holes, the holes will automatically get detected and removed once we remove the next bigger/smaller section. The extra checks can go.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
---
 mm/memory_hotplug.c | 34 +++++++---------------------------
 1 file changed, 7 insertions(+), 27 deletions(-)
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn = end_pfn) {
 		/*

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 413

Quoted 6 lines
mm/memory_hotplug.c +3 -0
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 					       start_pfn);
 		if (pfn)
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;

Thinking in which case (zone_start_pfn != start_pfn) and it comes here.

Quoted 36 lines
mm/memory_hotplug.c +1 -27
+		}
 	}
-
-	/*
-	 * The section is not biggest or smallest mem_section in the zone, it
-	 * only creates a hole in the zone. So in this case, we need not
-	 * change the zone. But perhaps, the zone has only hole data. Thus
-	 * it check the zone has only hole or not.
-	 */
-	pfn = zone_start_pfn;
-	for (; pfn < zone_end_pfn; pfn += PAGES_PER_SUBSECTION) {
-		if (unlikely(!pfn_to_online_page(pfn)))
-			continue;
-
-		if (page_zone(pfn_to_page(pfn)) != zone)
-			continue;
-
-		/* Skip range to be removed */
-		if (pfn >= start_pfn && pfn < end_pfn)
-			continue;
-
-		/* If we find valid section, we have nothing to do */
-		zone_span_writeunlock(zone);
-		return;
-	}
-
-	/* The zone has no valid section */
-	zone->zone_start_pfn = 0;
-	zone->spanned_pages = 0;
 	zone_span_writeunlock(zone);
 }
 
Signature · 1 line
2.21.0
#35 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 32 lines · Baoquan He
Quoted 31 lines · David Hildenbrand

If we have holes, the holes will automatically get detected and removed once we remove the next bigger/smaller section. The extra checks can go.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
---
 mm/memory_hotplug.c | 34 +++++++---------------------------
 1 file changed, 7 insertions(+), 27 deletions(-)
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn = end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
Quoted 8 lines

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 413

Quoted 6 lines
mm/memory_hotplug.c +3 -0
 					       start_pfn);
 		if (pfn)
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;

Thinking in which case (zone_start_pfn != start_pfn) and it comes here.

Could only happen in case the zone_start_pfn would have been "out of the zone already". If you ask me: unlikely :)

This change at least maintains the same result as before (where the all-holes check would have caught it).

Signature · 3 lines
Thanks,

David / dhildenb
#36 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 42 lines · David Hildenbrand
Quoted 32 lines · Baoquan He
Quoted 31 lines · David Hildenbrand

If we have holes, the holes will automatically get detected and removed once we remove the next bigger/smaller section. The extra checks can go.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
---
 mm/memory_hotplug.c | 34 +++++++---------------------------
 1 file changed, 7 insertions(+), 27 deletions(-)
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn = end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
Quoted 8 lines

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 413

Quoted 6 lines
mm/memory_hotplug.c +3 -0
 					       start_pfn);
 		if (pfn)
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;

Thinking in which case (zone_start_pfn != start_pfn) and it comes here.

Quoted 2 lines

Could only happen in case the zone_start_pfn would have been "out of the zone already". If you ask me: unlikely :)

Yeah, I also think it's unlikely to come here.

The 'if (zone_start_pfn = start_pfn)' checking also covers the case (zone_start_pfn = start_pfn && zone_end_pfn = end_pfn). So this zone_start_pfn/spanned_pages resetting can be removed to avoid confusion.

#37 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 46 lines · Baoquan He
Quoted 42 lines · David Hildenbrand
Quoted 32 lines · Baoquan He
Quoted 31 lines · David Hildenbrand

If we have holes, the holes will automatically get detected and removed once we remove the next bigger/smaller section. The extra checks can go.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
---
 mm/memory_hotplug.c | 34 +++++++---------------------------
 1 file changed, 7 insertions(+), 27 deletions(-)
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn == end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
Quoted 8 lines

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 413

Quoted 6 lines
mm/memory_hotplug.c +3 -0
 					       start_pfn);
 		if (pfn)
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;

Thinking in which case (zone_start_pfn != start_pfn) and it comes here.

Quoted 2 lines

Could only happen in case the zone_start_pfn would have been "out of the zone already". If you ask me: unlikely :)

Quoted 6 lines

Yeah, I also think it's unlikely to come here.

The 'if (zone_start_pfn == start_pfn)' checking also covers the case (zone_start_pfn == start_pfn && zone_end_pfn == end_pfn). So this zone_start_pfn/spanned_pages resetting can be removed to avoid confusion.

At least I would find it more confusing without it (or want a comment explaining why this does not have to be handled and why the !pfn case is not possible).

Anyhow, that patch is already upstream and I don't consider this high priority. Thanks :)

Signature · 3 lines
Thanks,

David / dhildenb
#38 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 49 lines · David Hildenbrand
Quoted 48 lines · Baoquan He
Quoted 42 lines · David Hildenbrand
Quoted 32 lines · Baoquan He
Quoted 31 lines · David Hildenbrand

If we have holes, the holes will automatically get detected and removed once we remove the next bigger/smaller section. The extra checks can go.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
---
 mm/memory_hotplug.c | 34 +++++++---------------------------
 1 file changed, 7 insertions(+), 27 deletions(-)
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn = end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
Quoted 8 lines

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 413

Quoted 6 lines
mm/memory_hotplug.c +3 -0
 					       start_pfn);
 		if (pfn)
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;

Thinking in which case (zone_start_pfn != start_pfn) and it comes here.

Quoted 2 lines

Could only happen in case the zone_start_pfn would have been "out of the zone already". If you ask me: unlikely :)

Yeah, I also think it's unlikely to come here.

Quoted 8 lines
Quoted 4 lines

The 'if (zone_start_pfn = start_pfn)' checking also covers the case (zone_start_pfn = start_pfn && zone_end_pfn = end_pfn). So this zone_start_pfn/spanned_pages resetting can be removed to avoid confusion.

At least I would find it more confusing without it (or want a comment explaining why this does not have to be handled and why the !pfn case is not possible).

I don't get why being w/o it will be more confusing, but it's OK since it doesn't impact anything.

Quoted 3 lines

Anyhow, that patch is already upstream and I don't consider this high priority. Thanks :)

Yeah, noticed you told Wei the status in another patch thread, I am fine with it, just leave it to you to decide. Thanks.

#39 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 55 lines · Baoquan He
Quoted 54 lines · David Hildenbrand
Quoted 52 lines · Baoquan He
Quoted 42 lines · David Hildenbrand
Quoted 32 lines · Baoquan He
Quoted 31 lines · David Hildenbrand

If we have holes, the holes will automatically get detected and removed once we remove the next bigger/smaller section. The extra checks can go.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
---
 mm/memory_hotplug.c | 34 +++++++---------------------------
 1 file changed, 7 insertions(+), 27 deletions(-)
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn = end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
Quoted 8 lines

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 413

Quoted 6 lines
mm/memory_hotplug.c +3 -0
 					       start_pfn);
 		if (pfn)
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;

Thinking in which case (zone_start_pfn != start_pfn) and it comes here.

Quoted 2 lines

Could only happen in case the zone_start_pfn would have been "out of the zone already". If you ask me: unlikely :)

Yeah, I also think it's unlikely to come here.

The 'if (zone_start_pfn = start_pfn)' checking also covers the case (zone_start_pfn = start_pfn && zone_end_pfn = end_pfn). So this zone_start_pfn/spanned_pages resetting can be removed to avoid confusion.

Quoted 6 lines
Quoted 3 lines

At least I would find it more confusing without it (or want a comment explaining why this does not have to be handled and why the !pfn case is not possible).

I don't get why being w/o it will be more confusing, but it's OK since it doesn't impact anything.

Because we could actually BUG_ON(!pfn) here, right? Only having a "if (pfn)" leaves the reader wondering "why is the other case not handled".

Quoted 7 lines
Quoted 4 lines

Anyhow, that patch is already upstream and I don't consider this high priority. Thanks :)

Yeah, noticed you told Wei the status in another patch thread, I am fine with it, just leave it to you to decide. Thanks.

I am fairly busy right now. Can you send a patch (double-checking and making this eventually unconditional?). Thanks!

Signature · 3 lines
Thanks,

David / dhildenb
#40 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 68 lines · David Hildenbrand
Quoted 55 lines · Baoquan He
Quoted 54 lines · David Hildenbrand
Quoted 52 lines · Baoquan He
Quoted 42 lines · David Hildenbrand
Quoted 32 lines · Baoquan He
Quoted 31 lines · David Hildenbrand

If we have holes, the holes will automatically get detected and removed once we remove the next bigger/smaller section. The extra checks can go.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
---
 mm/memory_hotplug.c | 34 +++++++---------------------------
 1 file changed, 7 insertions(+), 27 deletions(-)
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn = end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
Quoted 8 lines

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 413

Quoted 6 lines
mm/memory_hotplug.c +3 -0
 					       start_pfn);
 		if (pfn)
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;

Thinking in which case (zone_start_pfn != start_pfn) and it comes here.

Quoted 2 lines

Could only happen in case the zone_start_pfn would have been "out of the zone already". If you ask me: unlikely :)

Yeah, I also think it's unlikely to come here.

The 'if (zone_start_pfn = start_pfn)' checking also covers the case (zone_start_pfn = start_pfn && zone_end_pfn = end_pfn). So this zone_start_pfn/spanned_pages resetting can be removed to avoid confusion.

Quoted 6 lines
Quoted 3 lines

At least I would find it more confusing without it (or want a comment explaining why this does not have to be handled and why the !pfn case is not possible).

I don't get why being w/o it will be more confusing, but it's OK since it doesn't impact anything.

Because we could actually BUG_ON(!pfn) here, right? Only having a "if (pfn)" leaves the reader wondering "why is the other case not handled".

Quoted 2 lines
Quoted 2 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Anyhow, that patch is already upstream and I don't consider this high priority. Thanks :)

Yeah, noticed you told Wei the status in another patch thread, I am fine with it, just leave it to you to decide. Thanks.

I am fairly busy right now. Can you send a patch (double-checking and making this eventually unconditional?). Thanks!

Understood, sorry about the noise, David. I will think about this.

#41 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Anyhow, that patch is already upstream and I don't consider this high priority. Thanks :)

Yeah, noticed you told Wei the status in another patch thread, I am fine with it, just leave it to you to decide. Thanks.

I am fairly busy right now. Can you send a patch (double-checking and making this eventually unconditional?). Thanks!

Understood, sorry about the noise, David. I will think about this.

No need to excuse, really, I'm very happy about review feedback!

The review of this series happened fairly late. Bad, because it's not perfect, but good, because no serious stuff was found (so far :) ). If you also don't have time to look into this, I can put it onto my todo list, just let me know.

Thanks!

Signature · 3 lines
Thanks,

David / dhildenb
#42 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 6 lines · David Hildenbrand
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines

Anyhow, that patch is already upstream and I don't consider this high priority. Thanks :)

Yeah, noticed you told Wei the status in another patch thread, I am fine with it, just leave it to you to decide. Thanks.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I am fairly busy right now. Can you send a patch (double-checking and making this eventually unconditional?). Thanks!

Understood, sorry about the noise, David. I will think about this.

No need to excuse, really, I'm very happy about review feedback!

Glad to hear it, thanks.

Quoted 4 lines

The review of this series happened fairly late. Bad, because it's not perfect, but good, because no serious stuff was found (so far :) ). If you also don't have time to look into this, I can put it onto my todo list, just let me know.

Both is OK to me, as long as thing is clear to us. I will discuss with Wei Yang for now. You can post patch anytime if you make one.

#43 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

In reply to David Hildenbrand
Quoted 31 lines · David Hildenbrand

If we have holes, the holes will automatically get detected and removed once we remove the next bigger/smaller section. The extra checks can go.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>

---
mm/memory_hotplug.c | 34 +++++++---------------------------

1 file changed, 7 insertions(+), 27 deletions(-)
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn == end_pfn) {
 		/*

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 415

Quoted 8 lines
mm/memory_hotplug.c +4 -0
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 					       start_pfn);
 		if (pfn)
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
+		}
 	}

If it is me, I would like to take out these two similar logic out.

For example:

	if () {
	} else if () {
	} else {
		goto out;
	}


	/* The zone has no valid section */
	if (!pfn) {
			zone->zone_start_pfn = 0;
			zone->spanned_pages = 0;
	}

out:

	zone_span_writeunlock(zone);

Well, this is just my personal taste :-)

Quoted 32 lines
mm/memory_hotplug.c +0 -27
-
-	/*
-	 * The section is not biggest or smallest mem_section in the zone, it
-	 * only creates a hole in the zone. So in this case, we need not
-	 * change the zone. But perhaps, the zone has only hole data. Thus
-	 * it check the zone has only hole or not.
-	 */
-	pfn = zone_start_pfn;
-	for (; pfn < zone_end_pfn; pfn += PAGES_PER_SUBSECTION) {
-		if (unlikely(!pfn_to_online_page(pfn)))
-			continue;
-
-		if (page_zone(pfn_to_page(pfn)) != zone)
-			continue;
-
-		/* Skip range to be removed */
-		if (pfn >= start_pfn && pfn < end_pfn)
-			continue;
-
-		/* If we find valid section, we have nothing to do */
-		zone_span_writeunlock(zone);
-		return;
-	}
-
-	/* The zone has no valid section */
-	zone->zone_start_pfn = 0;
-	zone->spanned_pages = 0;
 	zone_span_writeunlock(zone);
 }
 
Signature · 1 line
2.21.0
Signature · 2 lines
Wei Yang
Help you, Help me
#44 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Hi Wei Yang,

Quoted 17 lines · Wei Yang
Quoted 17 lines
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn = end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 					       start_pfn);
 		if (pfn)
Quoted 8 lines

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 415

Quoted 6 lines
mm/memory_hotplug.c +4 -0
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
+		}
 	}

If it is me, I would like to take out these two similar logic out.

I also like this style.

Quoted 7 lines

For example:

	if () {
	} else if () {
	} else {
		goto out;

Here the last else is unnecessary, right?

Quoted 3 lines
	}

Like this, I believe both David and I will be satisfactory. Even though I still think his 2nd resetting is not needed :-)

Quoted 48 lines
	/* The zone has no valid section */
	if (!pfn) {
			zone->zone_start_pfn = 0;
			zone->spanned_pages = 0;
	}

out:

	zone_span_writeunlock(zone);

Well, this is just my personal taste :-)

Quoted 32 lines
mm/memory_hotplug.c +0 -27
-
-	/*
-	 * The section is not biggest or smallest mem_section in the zone, it
-	 * only creates a hole in the zone. So in this case, we need not
-	 * change the zone. But perhaps, the zone has only hole data. Thus
-	 * it check the zone has only hole or not.
-	 */
-	pfn = zone_start_pfn;
-	for (; pfn < zone_end_pfn; pfn += PAGES_PER_SUBSECTION) {
-		if (unlikely(!pfn_to_online_page(pfn)))
-			continue;
-
-		if (page_zone(pfn_to_page(pfn)) != zone)
-			continue;
-
-		/* Skip range to be removed */
-		if (pfn >= start_pfn && pfn < end_pfn)
-			continue;
-
-		/* If we find valid section, we have nothing to do */
-		zone_span_writeunlock(zone);
-		return;
-	}
-
-	/* The zone has no valid section */
-	zone->zone_start_pfn = 0;
-	zone->spanned_pages = 0;
 	zone_span_writeunlock(zone);
 }
 
Signature · 1 line
2.21.0
Signature · 2 lines
Wei Yang
Help you, Help me
#45 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 31 lines · Baoquan He

Hi Wei Yang,

Quoted 17 lines · Wei Yang
Quoted 17 lines
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn == end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 					       start_pfn);
 		if (pfn)
Quoted 8 lines

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 415

Quoted 6 lines
mm/memory_hotplug.c +4 -0
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
+		}
 	}

If it is me, I would like to take out these two similar logic out.

I also like this style.

Quoted 1 line
Quoted 8 lines
Quoted 6 lines

For example:

	if () {
	} else if () {
	} else {
		goto out;

Here the last else is unnecessary, right?

I am afraid not.

If the range is not the first or last, we would leave pfn not initialized.

Signature · 2 lines
Wei Yang
Help you, Help me
#46 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 38 lines · Wei Yang
Quoted 37 lines · Baoquan He

Hi Wei Yang,

Quoted 17 lines · Wei Yang
Quoted 17 lines
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn = end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 					       start_pfn);
 		if (pfn)
Quoted 8 lines

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 415

Quoted 6 lines
mm/memory_hotplug.c +4 -0
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
+		}
 	}

If it is me, I would like to take out these two similar logic out.

I also like this style.

Quoted 7 lines

For example:

	if () {
	} else if () {
	} else {
		goto out;
Quoted 6 lines
Quoted 2 lines

Here the last else is unnecessary, right?

I am afraid not.

If the range is not the first or last, we would leave pfn not initialized.

Ah, you are right. I forgot that one. Then pfn can be assigned the zone_start_pfn as the old code. Then the following logic is the same as the original code, find_smallest_section_pfn()/find_biggest_section_pfn() have done the iteration the old for loop was doing.

	unsigned long pfn = zone_start_pfn;	
	if () {
	} else if () {
	} 

	/* The zone has no valid section */
	if (!pfn) {
        	zone->zone_start_pfn = 0;
        	zone->spanned_pages = 0;
	}
#47 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 51 lines · Baoquan He
Quoted 38 lines · Wei Yang
Quoted 37 lines · Baoquan He

Hi Wei Yang,

Quoted 17 lines · Wei Yang
Quoted 17 lines
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn == end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 					       start_pfn);
 		if (pfn)
Quoted 8 lines

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 415

Quoted 6 lines
mm/memory_hotplug.c +4 -0
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
+		}
 	}

If it is me, I would like to take out these two similar logic out.

I also like this style.

Quoted 7 lines

For example:

	if () {
	} else if () {
	} else {
		goto out;
Quoted 6 lines
Quoted 2 lines

Here the last else is unnecessary, right?

I am afraid not.

If the range is not the first or last, we would leave pfn not initialized.

Ah, you are right. I forgot that one. Then pfn can be assigned the zone_start_pfn as the old code. Then the following logic is the same as the original code, find_smallest_section_pfn()/find_biggest_section_pfn() have done the iteration the old for loop was doing.

Quoted 10 lines
	unsigned long pfn = zone_start_pfn;	
	if () {
	} else if () {
	} 

	/* The zone has no valid section */
	if (!pfn) {
       	zone->zone_start_pfn = 0;
       	zone->spanned_pages = 0;
	}

This one look better :-)

Thanks

Signature · 2 lines
Wei Yang
Help you, Help me
#48 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 57 lines · Wei Yang
Quoted 56 lines · Baoquan He
Quoted 38 lines · Wei Yang
Quoted 37 lines · Baoquan He

Hi Wei Yang,

Quoted 17 lines · Wei Yang
Quoted 17 lines
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn = end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 					       start_pfn);
 		if (pfn)
Quoted 8 lines

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 415

Quoted 6 lines
mm/memory_hotplug.c +4 -0
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
+		}
 	}

If it is me, I would like to take out these two similar logic out.

I also like this style.

Quoted 7 lines

For example:

	if () {
	} else if () {
	} else {
		goto out;
Quoted 6 lines
Quoted 2 lines

Here the last else is unnecessary, right?

I am afraid not.

If the range is not the first or last, we would leave pfn not initialized.

Ah, you are right. I forgot that one. Then pfn can be assigned the zone_start_pfn as the old code. Then the following logic is the same as the original code, find_smallest_section_pfn()/find_biggest_section_pfn() have done the iteration the old for loop was doing.

	unsigned long pfn = zone_start_pfn;	
	if () {
	} else if () {
	} 
Quoted 7 lines
Quoted 5 lines
	/* The zone has no valid section */
	if (!pfn) {
       	zone->zone_start_pfn = 0;
       	zone->spanned_pages = 0;
	}

This one look better :-)

Thanks for your confirmation, I will make one patch like this and post.

#49 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 64 lines · Baoquan He
Quoted 63 lines · Wei Yang
Quoted 61 lines · Baoquan He
Quoted 38 lines · Wei Yang
Quoted 37 lines · Baoquan He

Hi Wei Yang,

Quoted 17 lines · Wei Yang
Quoted 17 lines
mm/memory_hotplug.c +3 -0
index f294918f7211..8dafa1ba8d9f 100644
@@ -393,6 +393,9 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		if (pfn) {
 			zone->zone_start_pfn = pfn;
 			zone->spanned_pages = zone_end_pfn - pfn;
+		} else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
 		}
 	} else if (zone_end_pfn == end_pfn) {
 		/*
@@ -405,34 +408,11 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 					       start_pfn);
 		if (pfn)
Quoted 8 lines

mm/memory_hotplug.c static void shrink_zone_span(struct zone *zone, unsigned long start_pfn, · line 415

Quoted 6 lines
mm/memory_hotplug.c +4 -0
 			zone->spanned_pages = pfn - zone_start_pfn + 1;
+		else {
+			zone->zone_start_pfn = 0;
+			zone->spanned_pages = 0;
+		}
 	}

If it is me, I would like to take out these two similar logic out.

I also like this style.

Quoted 7 lines

For example:

	if () {
	} else if () {
	} else {
		goto out;
Quoted 6 lines
Quoted 2 lines

Here the last else is unnecessary, right?

I am afraid not.

If the range is not the first or last, we would leave pfn not initialized.

Ah, you are right. I forgot that one. Then pfn can be assigned the zone_start_pfn as the old code. Then the following logic is the same as the original code, find_smallest_section_pfn()/find_biggest_section_pfn() have done the iteration the old for loop was doing.

	unsigned long pfn = zone_start_pfn;	
	if () {
	} else if () {
	} 

	/* The zone has no valid section */
	if (!pfn) {
       	zone->zone_start_pfn = 0;
       	zone->spanned_pages = 0;
	}
Quoted 3 lines
Quoted 1 line

This one look better :-)

Thanks for your confirmation, I will make one patch like this and post.

Sure :-)

Signature · 2 lines
Wei Yang
Help you, Help me
#50 lore raw

RE: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

In reply to Wei Yang

From: Wei Yang

Quoted 1 line

Sent: 05 February 2020 09:59

...

Quoted 9 lines

If it is me, I would like to take out these two similar logic out.

For example:

	if () {
	} else if () {
	} else {
		goto out;
	}

I'm pretty sure the kernel layout rules disallow 'else if'. It is also pretty horrid unless the conditionals are all related (so it is almost a switch statement).

David

-
Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales)

#51 lore raw

Re: [PATCH v6 08/10] mm/memory_hotplug: Don't check for "all holes" in shrink_zone_span()

Quoted 7 lines · David Laight

From: Wei Yang

Quoted 1 line

Sent: 05 February 2020 09:59

...

Quoted 4 lines

If it is me, I would like to take out these two similar logic out.

For example:

Quoted 7 lines
Quoted 5 lines
	if () {
	} else if () {
	} else {
		goto out;
	}

I'm pretty sure the kernel layout rules disallow 'else if'.

Huh?

git grep "else if" | wc -l
49336

I'm afraid I don't get what you mean :)

Signature · 3 lines
Thanks,

David / dhildenb
#52 lore raw

PATCH v6 9/10 mm/memory_hotplug: Drop local variables in shrink_zone_span()

In reply to David Hildenbrand

Get rid of the unnecessary local variables.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
---
 mm/memory_hotplug.c | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)
mm/memory_hotplug.c +6 -9
index 8dafa1ba8d9f..843481bd507d 100644
@@ -374,14 +374,11 @@ static unsigned long find_biggest_section_pfn(int nid, struct zone *zone,
 static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 			     unsigned long end_pfn)
 {
-	unsigned long zone_start_pfn = zone->zone_start_pfn;
-	unsigned long z = zone_end_pfn(zone); /* zone_end_pfn namespace clash */
-	unsigned long zone_end_pfn = z;
 	unsigned long pfn;
 	int nid = zone_to_nid(zone);
 
 	zone_span_writelock(zone);
-	if (zone_start_pfn = start_pfn) {
+	if (zone->zone_start_pfn = start_pfn) {
 		/*
 		 * If the section is smallest section in the zone, it need
 		 * shrink zone->zone_start_pfn and zone->zone_spanned_pages.
@@ -389,25 +386,25 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		 * for shrinking zone.
 		 */
 		pfn = find_smallest_section_pfn(nid, zone, end_pfn,
-						zone_end_pfn);
+						zone_end_pfn(zone));
 		if (pfn) {
+			zone->spanned_pages = zone_end_pfn(zone) - pfn;
 			zone->zone_start_pfn = pfn;
-			zone->spanned_pages = zone_end_pfn - pfn;
 		} else {
 			zone->zone_start_pfn = 0;
 			zone->spanned_pages = 0;
 		}
-	} else if (zone_end_pfn = end_pfn) {
+	} else if (zone_end_pfn(zone) = end_pfn) {
 		/*
 		 * If the section is biggest section in the zone, it need
 		 * shrink zone->spanned_pages.
 		 * In this case, we find second biggest valid mem_section for
 		 * shrinking zone.
 		 */
-		pfn = find_biggest_section_pfn(nid, zone, zone_start_pfn,
+		pfn = find_biggest_section_pfn(nid, zone, zone->zone_start_pfn,
 					       start_pfn);
 		if (pfn)
-			zone->spanned_pages = pfn - zone_start_pfn + 1;
+			zone->spanned_pages = pfn - zone->zone_start_pfn + 1;
 		else {
 			zone->zone_start_pfn = 0;
 			zone->spanned_pages = 0;
Signature · 1 line
2.21.0
maintainer Reviewed-by on v6 9/10
#53 lore raw

Re: [PATCH v6 09/10] mm/memory_hotplug: Drop local variables in shrink_zone_span()

Quoted 19 lines · David Hildenbrand

Get rid of the unnecessary local variables.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
---
 mm/memory_hotplug.c | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)
mm/memory_hotplug.c +0 -0
index 8dafa1ba8d9f..843481bd507d 100644

mm/memory_hotplug.c static unsigned long find_biggest_section_pfn(int nid, struct zone *zone, · line 378

Quoted 8 lines
mm/memory_hotplug.c +0 -3
@@ -374,14 +374,11 @@ static unsigned long find_biggest_section_pfn(int nid, struct zone *zone,
 static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 			     unsigned long end_pfn)
 {
-	unsigned long zone_start_pfn = zone->zone_start_pfn;
-	unsigned long z = zone_end_pfn(zone); /* zone_end_pfn namespace clash */
-	unsigned long zone_end_pfn = z;
 	unsigned long pfn;
 	int nid = zone_to_nid(zone);

We could also remove the nid, right?
AFAICS, the nid is only used in find_{smallest/biggest}_section_pfn so we could place there as well.

Anyway, nothing to nit-pick about:

Quoted 41 lines
mm/memory_hotplug.c +6 -6
 	zone_span_writelock(zone);
-	if (zone_start_pfn = start_pfn) {
+	if (zone->zone_start_pfn = start_pfn) {
 		/*
 		 * If the section is smallest section in the zone, it need
 		 * shrink zone->zone_start_pfn and zone->zone_spanned_pages.
@@ -389,25 +386,25 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		 * for shrinking zone.
 		 */
 		pfn = find_smallest_section_pfn(nid, zone, end_pfn,
-						zone_end_pfn);
+						zone_end_pfn(zone));
 		if (pfn) {
+			zone->spanned_pages = zone_end_pfn(zone) - pfn;
 			zone->zone_start_pfn = pfn;
-			zone->spanned_pages = zone_end_pfn - pfn;
 		} else {
 			zone->zone_start_pfn = 0;
 			zone->spanned_pages = 0;
 		}
-	} else if (zone_end_pfn = end_pfn) {
+	} else if (zone_end_pfn(zone) = end_pfn) {
 		/*
 		 * If the section is biggest section in the zone, it need
 		 * shrink zone->spanned_pages.
 		 * In this case, we find second biggest valid mem_section for
 		 * shrinking zone.
 		 */
-		pfn = find_biggest_section_pfn(nid, zone, zone_start_pfn,
+		pfn = find_biggest_section_pfn(nid, zone, zone->zone_start_pfn,
 					       start_pfn);
 		if (pfn)
-			zone->spanned_pages = pfn - zone_start_pfn + 1;
+			zone->spanned_pages = pfn - zone->zone_start_pfn + 1;
 		else {
 			zone->zone_start_pfn = 0;
 			zone->spanned_pages = 0;
Signature · 1 line
2.21.0
Signature · 2 lines
Oscar Salvador
SUSE L3
#54 lore raw

Re: [PATCH v6 09/10] mm/memory_hotplug: Drop local variables in shrink_zone_span()

Quoted 24 lines · Oscar Salvador
Quoted 23 lines · David Hildenbrand

Get rid of the unnecessary local variables.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
---
 mm/memory_hotplug.c | 15 ++++++---------
 1 file changed, 6 insertions(+), 9 deletions(-)
mm/memory_hotplug.c +0 -1
index 8dafa1ba8d9f..843481bd507d 100644
@@ -374,14 +374,11 @@ static unsigned long find_biggest_section_pfn(int nid, struct zone *zone,
 static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 			     unsigned long end_pfn)
 {
-	unsigned long zone_start_pfn = zone->zone_start_pfn;
Quoted 8 lines

mm/memory_hotplug.c static unsigned long find_biggest_section_pfn(int nid, struct zone *zone, · line 378

Quoted 4 lines
mm/memory_hotplug.c +0 -2
-	unsigned long z = zone_end_pfn(zone); /* zone_end_pfn namespace clash */
-	unsigned long zone_end_pfn = z;
 	unsigned long pfn;
 	int nid = zone_to_nid(zone);

We could also remove the nid, right?
AFAICS, the nid is only used in find_{smallest/biggest}_section_pfn so we could place there as well.

I remember sending a patch on this (which was acked, but not picked up yet)...

oh, there it is :)

https://lore.kernel.org/linux-mm/[email protected]/

Thanks!

Signature · 3 lines
Thanks,

David / dhildenb
Reviewed-by on v6 9/10
#55 lore raw

Re: [PATCH v6 09/10] mm/memory_hotplug: Drop local variables in shrink_zone_span()

In reply to David Hildenbrand
Quoted 10 lines · David Hildenbrand

Get rid of the unnecessary local variables.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc David Hildenbrand <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>

Looks reasonable.

Quoted 57 lines

---
mm/memory_hotplug.c | 15 ++++++---------

1 file changed, 6 insertions(+), 9 deletions(-)
mm/memory_hotplug.c +6 -9
index 8dafa1ba8d9f..843481bd507d 100644
@@ -374,14 +374,11 @@ static unsigned long find_biggest_section_pfn(int nid, struct zone *zone,
 static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 			     unsigned long end_pfn)
 {
-	unsigned long zone_start_pfn = zone->zone_start_pfn;
-	unsigned long z = zone_end_pfn(zone); /* zone_end_pfn namespace clash */
-	unsigned long zone_end_pfn = z;
 	unsigned long pfn;
 	int nid = zone_to_nid(zone);
 
 	zone_span_writelock(zone);
-	if (zone_start_pfn == start_pfn) {
+	if (zone->zone_start_pfn == start_pfn) {
 		/*
 		 * If the section is smallest section in the zone, it need
 		 * shrink zone->zone_start_pfn and zone->zone_spanned_pages.
@@ -389,25 +386,25 @@ static void shrink_zone_span(struct zone *zone, unsigned long start_pfn,
 		 * for shrinking zone.
 		 */
 		pfn = find_smallest_section_pfn(nid, zone, end_pfn,
-						zone_end_pfn);
+						zone_end_pfn(zone));
 		if (pfn) {
+			zone->spanned_pages = zone_end_pfn(zone) - pfn;
 			zone->zone_start_pfn = pfn;
-			zone->spanned_pages = zone_end_pfn - pfn;
 		} else {
 			zone->zone_start_pfn = 0;
 			zone->spanned_pages = 0;
 		}
-	} else if (zone_end_pfn == end_pfn) {
+	} else if (zone_end_pfn(zone) == end_pfn) {
 		/*
 		 * If the section is biggest section in the zone, it need
 		 * shrink zone->spanned_pages.
 		 * In this case, we find second biggest valid mem_section for
 		 * shrinking zone.
 		 */
-		pfn = find_biggest_section_pfn(nid, zone, zone_start_pfn,
+		pfn = find_biggest_section_pfn(nid, zone, zone->zone_start_pfn,
 					       start_pfn);
 		if (pfn)
-			zone->spanned_pages = pfn - zone_start_pfn + 1;
+			zone->spanned_pages = pfn - zone->zone_start_pfn + 1;
 		else {
 			zone->zone_start_pfn = 0;
 			zone->spanned_pages = 0;
Signature · 1 line
2.21.0
Signature · 2 lines
Wei Yang
Help you, Help me
#56 lore raw

PATCH v6 10/10 mm/memory_hotplug: Cleanup __remove_pages()

In reply to David Hildenbrand

Let's drop the basically unused section stuff and simplify.

Also, let's use a shorter variant to calculate the number of pages to the next section boundary.

Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
---
 mm/memory_hotplug.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)
mm/memory_hotplug.c +6 -11
index 843481bd507d..2275240cfa10 100644
@@ -490,25 +490,20 @@ static void __remove_section(unsigned long pfn, unsigned long nr_pages,
 void __remove_pages(unsigned long pfn, unsigned long nr_pages,
 		    struct vmem_altmap *altmap)
 {
+	const unsigned long end_pfn = pfn + nr_pages;
+	unsigned long cur_nr_pages;
 	unsigned long map_offset = 0;
-	unsigned long nr, start_sec, end_sec;
 
 	map_offset = vmem_altmap_offset(altmap);
 
 	if (check_pfn_span(pfn, nr_pages, "remove"))
 		return;
 
-	start_sec = pfn_to_section_nr(pfn);
-	end_sec = pfn_to_section_nr(pfn + nr_pages - 1);
-	for (nr = start_sec; nr <= end_sec; nr++) {
-		unsigned long pfns;
-
+	for (; pfn < end_pfn; pfn += cur_nr_pages) {
 		cond_resched();
-		pfns = min(nr_pages, PAGES_PER_SECTION
-				- (pfn & ~PAGE_SECTION_MASK));
-		__remove_section(pfn, pfns, map_offset, altmap);
-		pfn += pfns;
-		nr_pages -= pfns;
+		/* Select all remaining pages up to the next section boundary */
+		cur_nr_pages = min(end_pfn - pfn, -(pfn | PAGE_SECTION_MASK));
+		__remove_section(pfn, cur_nr_pages, map_offset, altmap);
 		map_offset = 0;
 	}
 }
Signature · 1 line
2.21.0
maintainer Reviewed-by on v6 10/10
#57 lore raw

Re: [PATCH v6 10/10] mm/memory_hotplug: Cleanup __remove_pages()

Quoted 5 lines · David Hildenbrand

Let's drop the basically unused section stuff and simplify.

Also, let's use a shorter variant to calculate the number of pages to the next section boundary.

Quoted 7 lines
Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>

I have to confess that it took me while to wrap around my head with the new min() change, but looks ok:

Quoted 44 lines
---
 mm/memory_hotplug.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)
mm/memory_hotplug.c +6 -11
index 843481bd507d..2275240cfa10 100644
@@ -490,25 +490,20 @@ static void __remove_section(unsigned long pfn, unsigned long nr_pages,
 void __remove_pages(unsigned long pfn, unsigned long nr_pages,
 		    struct vmem_altmap *altmap)
 {
+	const unsigned long end_pfn = pfn + nr_pages;
+	unsigned long cur_nr_pages;
 	unsigned long map_offset = 0;
-	unsigned long nr, start_sec, end_sec;
 
 	map_offset = vmem_altmap_offset(altmap);
 
 	if (check_pfn_span(pfn, nr_pages, "remove"))
 		return;
 
-	start_sec = pfn_to_section_nr(pfn);
-	end_sec = pfn_to_section_nr(pfn + nr_pages - 1);
-	for (nr = start_sec; nr <= end_sec; nr++) {
-		unsigned long pfns;
-
+	for (; pfn < end_pfn; pfn += cur_nr_pages) {
 		cond_resched();
-		pfns = min(nr_pages, PAGES_PER_SECTION
-				- (pfn & ~PAGE_SECTION_MASK));
-		__remove_section(pfn, pfns, map_offset, altmap);
-		pfn += pfns;
-		nr_pages -= pfns;
+		/* Select all remaining pages up to the next section boundary */
+		cur_nr_pages = min(end_pfn - pfn, -(pfn | PAGE_SECTION_MASK));
+		__remove_section(pfn, cur_nr_pages, map_offset, altmap);
 		map_offset = 0;
 	}
 }
Signature · 1 line
2.21.0
Signature · 2 lines
Oscar Salvador
SUSE L3
#58 lore raw

Re: [PATCH v6 10/10] mm/memory_hotplug: Cleanup __remove_pages()

Quoted 14 lines · Oscar Salvador
Quoted 5 lines · David Hildenbrand

Let's drop the basically unused section stuff and simplify.

Also, let's use a shorter variant to calculate the number of pages to the next section boundary.

Quoted 7 lines
Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>
Quoted 2 lines

I have to confess that it took me while to wrap around my head with the new min() change, but looks ok:

It's a pattern commonly used in compilers and emulators to calculate the number of bytes to the next block/alignment. (we're missing a macro (like we have ALIGN_UP/IS_ALIGNED) for that - but it's hard to come up with a good name (e.g., SIZE_TO_NEXT_ALIGN) .

Signature · 3 lines
Thanks,

David / dhildenb
#59 lore raw

Re: [PATCH v6 10/10] mm/memory_hotplug: Cleanup __remove_pages()

Quoted 8 lines · David Hildenbrand
Quoted 2 lines · Oscar Salvador

I have to confess that it took me while to wrap around my head with the new min() change, but looks ok:

It's a pattern commonly used in compilers and emulators to calculate the number of bytes to the next block/alignment. (we're missing a macro (like we have ALIGN_UP/IS_ALIGNED) for that - but it's hard to come up with a good name (e.g., SIZE_TO_NEXT_ALIGN) .

You can just write the easy to understand

... ALIGN_UP(x) - x ...

which is better *without* having a separate name. Does that not generate good machine code for you?

Segher

#60 lore raw

Re: [PATCH v6 10/10] mm/memory_hotplug: Cleanup __remove_pages()

Quoted 5 lines · Segher Boessenkool
Quoted 4 lines · David Hildenbrand
Quoted 2 lines · Oscar Salvador

I have to confess that it took me while to wrap around my head with the new min() change, but looks ok:

Quoted 8 lines
Quoted 4 lines

It's a pattern commonly used in compilers and emulators to calculate the number of bytes to the next block/alignment. (we're missing a macro (like we have ALIGN_UP/IS_ALIGNED) for that - but it's hard to come up with a good name (e.g., SIZE_TO_NEXT_ALIGN) .

You can just write the easy to understand

... ALIGN_UP(x) - x ...

you mean

ALIGN_UP(x, PAGES_PER_SECTION) - x

but ...

Quoted 3 lines

which is better *without* having a separate name. Does that not generate good machine code for you?

1. There is no ALIGN_UP. "SECTION_ALIGN_UP(x) - x" would be possible
2. It would be wrong if x is already aligned.

e.g., let's use 4096 for simplicity as we all know that value by heart (for both x and the block size).

a) -(4096 | -4096) -> 4096

b) #define ALIGN_UP(x, a) ((x + a - 1) & -(a))

ALIGN_UP(4096, 4096) - 4096 -> 0

Not as easy as it seems ...

Signature · 3 lines
Thanks,

David / dhildenb
#61 lore raw

Re: [PATCH v6 10/10] mm/memory_hotplug: Cleanup __remove_pages()

Quoted 13 lines · David Hildenbrand
Quoted 9 lines · Segher Boessenkool
Quoted 4 lines · David Hildenbrand

It's a pattern commonly used in compilers and emulators to calculate the number of bytes to the next block/alignment. (we're missing a macro (like we have ALIGN_UP/IS_ALIGNED) for that - but it's hard to come up with a good name (e.g., SIZE_TO_NEXT_ALIGN) .

You can just write the easy to understand

... ALIGN_UP(x) - x ...

you mean

Quoted 8 lines

ALIGN_UP(x, PAGES_PER_SECTION) - x

but ...

Quoted 2 lines

which is better *without* having a separate name. Does that not generate good machine code for you?

1. There is no ALIGN_UP. "SECTION_ALIGN_UP(x) - x" would be possible

Erm, you started it ;-)

Quoted 5 lines

2. It would be wrong if x is already aligned.

e.g., let's use 4096 for simplicity as we all know that value by heart (for both x and the block size).

Quoted 7 lines

a) -(4096 | -4096) -> 4096

b) #define ALIGN_UP(x, a) ((x + a - 1) & -(a))

ALIGN_UP(4096, 4096) - 4096 -> 0

Not as easy as it seems ...

If you always want to return a number >= 1, it it simply
ALIGN_UP(x + 1) - x
(and replace 1 by any other minimum size required). This *also* is easy to read, without having to have any details (and quirks :-/ ) of those utility functions memorised.

Segher

#62 lore raw

Re: [PATCH v6 10/10] mm/memory_hotplug: Cleanup __remove_pages()

Quoted 16 lines · Segher Boessenkool
Quoted 15 lines · David Hildenbrand
Quoted 9 lines · Segher Boessenkool
Quoted 4 lines · David Hildenbrand

It's a pattern commonly used in compilers and emulators to calculate the number of bytes to the next block/alignment. (we're missing a macro (like we have ALIGN_UP/IS_ALIGNED) for that - but it's hard to come up with a good name (e.g., SIZE_TO_NEXT_ALIGN) .

You can just write the easy to understand

... ALIGN_UP(x) - x ...

you mean

ALIGN_UP(x, PAGES_PER_SECTION) - x

Quoted 8 lines
Quoted 6 lines

but ...

Quoted 2 lines

which is better *without* having a separate name. Does that not generate good machine code for you?

1. There is no ALIGN_UP. "SECTION_ALIGN_UP(x) - x" would be possible

Erm, you started it ;-)

Yeah, I was thinking in the wrong code base :)

Quoted 8 lines
Quoted 8 lines

2. It would be wrong if x is already aligned.

e.g., let's use 4096 for simplicity as we all know that value by heart (for both x and the block size).

a) -(4096 | -4096) -> 4096

Quoted 8 lines
Quoted 5 lines

b) #define ALIGN_UP(x, a) ((x + a - 1) & -(a))

ALIGN_UP(4096, 4096) - 4096 -> 0

Not as easy as it seems ...

If you always want to return a number >= 1, it it simply
ALIGN_UP(x + 1) - x

I'm sorry to have to correct you again for some corner cases:

ALIGN_UP(1, 4096) - 4096 = 0

Again, not as easy as it seems ...

Signature · 3 lines
Thanks,

David / dhildenb
#63 lore raw

Re: [PATCH v6 10/10] mm/memory_hotplug: Cleanup __remove_pages()

Quoted 6 lines

I'm sorry to have to correct you again for some corner cases:

ALIGN_UP(1, 4096) - 4096 = 0

Again, not as easy as it seems ...

Eh, wait, I'm messing up things. Will double check :)

Signature · 3 lines
Thanks,

David / dhildenb
#64 lore raw

Re: [PATCH v6 10/10] mm/memory_hotplug: Cleanup __remove_pages()

Quoted 9 lines · David Hildenbrand
Quoted 6 lines

I'm sorry to have to correct you again for some corner cases:

ALIGN_UP(1, 4096) - 4096 = 0

Again, not as easy as it seems ...

Eh, wait, I'm messing up things. Will double check :)

Yes, makes sense, will send a patch and cc you. Thanks!

Signature · 3 lines
Thanks,

David / dhildenb
Reviewed-by on v6 10/10
#65 lore raw

Re: [PATCH v6 10/10] mm/memory_hotplug: Cleanup __remove_pages()

In reply to David Hildenbrand
Quoted 5 lines · David Hildenbrand

Let's drop the basically unused section stuff and simplify.

Also, let's use a shorter variant to calculate the number of pages to the next section boundary.

Quoted 7 lines
Cc Andrew Morton <[email protected]>
Cc Oscar Salvador <[email protected]>
Cc Michal Hocko <[email protected]>
Cc Pavel Tatashin <[email protected]>
Cc Dan Williams <[email protected]>
Cc Wei Yang <[email protected]>

Finally understand the code.

Signature · 2 lines
Wei Yang
Help you, Help me
#66 lore raw
In reply to David Hildenbrand
Quoted 84 lines · David Hildenbrand

This series fixes the access of uninitialized memmaps when shrinking zones/nodes and when removing memory. Also, it contains all fixes for crashes that can be triggered when removing certain namespace using memunmap_pages() - ZONE_DEVICE, reported by Aneesh.

We stop trying to shrink ZONE_DEVICE, as it's buggy, fixing it would be more involved (we don't have SECTION_IS_ONLINE as an indicator), and shrinking is only of limited use (set_zone_contiguous() cannot detect the ZONE_DEVICE as contiguous).

We continue shrinking !ZONE_DEVICE zones, however, I reduced the amount of code to a minimum. Shrinking is especially necessary to keep zone->contiguous set where possible, especially, on memory unplug of DIMMs at zone boundaries.

--------------------------------------------------------------------------

Zones are now properly shrunk when offlining memory blocks or when onlining failed. This allows to properly shrink zones on memory unplug even if the separate memory blocks of a DIMM were onlined to different zones or re-onlined to a different zone after offlining.

Example:

:/# cat /proc/zoneinfo
Node 1, zone  Movable
        spanned  0
        present  0
        managed  0
:/# echo "online_movable" > /sys/devices/system/memory/memory41/state
:/# echo "online_movable" > /sys/devices/system/memory/memory43/state
:/# cat /proc/zoneinfo
Node 1, zone  Movable
        spanned  98304
        present  65536
        managed  65536
:/# echo 0 > /sys/devices/system/memory/memory43/online
:/# cat /proc/zoneinfo
Node 1, zone  Movable
        spanned  32768
        present  32768
        managed  32768
:/# echo 0 > /sys/devices/system/memory/memory41/online
:/# cat /proc/zoneinfo
Node 1, zone  Movable
        spanned  0
        present  0
        managed  0

--------------------------------------------------------------------------

I tested this with DIMMs on x86. I didn't test the ZONE_DEVICE part.

v4 -> v6:
- "mm/memunmap: Don't access uninitialized memmap in memunmap_pages()" -- Minimize code changes, rephrase subject and description
- "mm/memory_hotplug: Don't access uninitialized memmaps in shrink_zone_span()" -- Add ifdef to make it compile without ZONE_DEVICE

v4 -> v5:
- "mm/memory_hotplug: Don't access uninitialized memmaps in shrink_zone_span()" -- Add more details why ZONE_DEVICE is special
- Include two patches from Aneesh
-- "mm/memunmap: Use the correct start and end pfn when removing pages
from zone"
-- "mm/memmap_init: Update variable name in memmap_init_zone"

v3 -> v4:
- Drop "mm/memremap: Get rid of memmap_init_zone_device()"
-- As Alexander noticed, it was messy either way
- Drop "mm/memory_hotplug: Exit early in __remove_pages() on BUGs"
- Drop "mm: Exit early in set_zone_contiguous() if already contiguous"
- Drop "mm/memory_hotplug: Optimize zone shrinking code when checking for holes"
- Merged "mm/memory_hotplug: Remove pages from a zone before removing memory" and "mm/memory_hotplug: Remove zone parameter from __remove_pages()" into "mm/memory_hotplug: Shrink zones when offlining memory"
- Added "mm/memory_hotplug: Poison memmap in remove_pfn_range_from_zone()"
- Stop shrinking ZONE_DEVICE
- Reshuffle patches, moving all fixes to the front. Add Fixes: tags.
- Change subject/description of various patches
- Minor changes (too many to mention)

Quoted 5 lines
Cc Aneesh Kumar K.V <[email protected]>
Cc Andrew Morton <[email protected]>
Cc Dan Williams <[email protected]>
Cc Michal Hocko <[email protected]>

So, patch #1-#4 are already upstream. The other patches have been in -next for quite a long time, and I (+other RH people) ran excessive tests on them.

Especially patch #5 is a BUG fix I want to see upstream rather sooner than later (last know "uninitialized memmap" access).

Andrew decided not to send these for 5.5 due to lack of ack/review - which is unfortunate, but the right thing to do.

@Michal, @Oscar, can some of you at least have a patch #5 now so we can proceed with that? (the other patches can stay in -next some time longer)

Signature · 3 lines
Thanks,

David / dhildenb
maintainer
#67 lore raw
Quoted 2 lines · David Hildenbrand

@Michal, @Oscar, can some of you at least have a patch #5 now so we can proceed with that? (the other patches can stay in -next some time longer)

Hi,

I will be having a look at patch#5 shortly.

Thanks for the reminder

Signature · 2 lines
Oscar Salvador
SUSE L3
maintainer
#68 lore raw
Quoted 9 lines · On Tue, 3 Dec 2019 14:36:38 +0100 Oscar Salvador <[email protected]> wrote:
Quoted 2 lines · David Hildenbrand

@Michal, @Oscar, can some of you at least have a patch #5 now so we can proceed with that? (the other patches can stay in -next some time longer)

Hi,

I will be having a look at patch#5 shortly.

Thanks for the reminder

Things haven't improved a lot :(

mm-memmap_init-update-variable-name-in-memmap_init_zone.patch mm-memory_hotplug-poison-memmap-in-remove_pfn_range_from_zone.patch mm-memory_hotplug-we-always-have-a-zone-in-find_smallestbiggest_section_pfn.patch mm-memory_hotplug-dont-check-for-all-holes-in-shrink_zone_span.patch mm-memory_hotplug-drop-local-variables-in-shrink_zone_span.patch mm-memory_hotplug-cleanup-__remove_pages.patch

The first patch has reviews, the remainder are unloved.

#69 lore raw
Quoted 14 lines · Andrew Morton
Quoted 9 lines · On Tue, 3 Dec 2019 14:36:38 +0100 Oscar Salvador <[email protected]> wrote:
Quoted 2 lines · David Hildenbrand

@Michal, @Oscar, can some of you at least have a patch #5 now so we can proceed with that? (the other patches can stay in -next some time longer)

Hi,

I will be having a look at patch#5 shortly.

Thanks for the reminder

Things haven't improved a lot :(

Quoted 8 lines

mm-memmap_init-update-variable-name-in-memmap_init_zone.patch mm-memory_hotplug-poison-memmap-in-remove_pfn_range_from_zone.patch mm-memory_hotplug-we-always-have-a-zone-in-find_smallestbiggest_section_pfn.patch mm-memory_hotplug-dont-check-for-all-holes-in-shrink_zone_span.patch mm-memory_hotplug-drop-local-variables-in-shrink_zone_span.patch mm-memory_hotplug-cleanup-__remove_pages.patch

The first patch has reviews, the remainder are unloved.

Trying hard not to rant about the review mentality on this list, but I'm afraid I can't totally bite my tongue ... :)

Now, this is an uncomfortable situation for you and me. You have to ping people about review and patches are stuck in your tree. I have a growing list of patches that are somewhat considered "done", but well, not-upstream-at-all. I have patches that are long in RHEL and were properly tested, but could get dropped any time because -ENOREVIEW.

Our process nowadays seems to be, to only upstream what has an ACK/RB (fixes/features/cleanups). I can understand this is desirable (yet, I am not sure if this makes sense with the current take-and-not-give-back review mentality on this list).

Although it will make upstreaming stuff *even harder* and *even slower*, maybe we should start to only queue patches that have an ACK/RB, so they won't get blocked by this later on? At least that makes your life easier and people won't have to eventually follow up on patches that have been in linux-next for months.

Note: the result will be that many of my patches will still not get reviewed, won't get queued/upstreamed, I will continuously ping and resend, I will lose interest because I have better things to do, I will lose interest in our code quality, I will lose interest to review.

(side note: some people might actually enjoy me sending less cleanup patches, so this approach might be desirable for some ;) )

One alternative is to send patches upstream once they have been lying around in linux-next for $RANDOM number of months, because they obviously saw some testing and nobody started to yell at them once stumbling over them on linux-mm.

Signature · 3 lines
Thanks,

David / dhildenb
#70 lore raw
Quoted 22 lines · David Hildenbrand
Quoted 21 lines · Andrew Morton
Quoted 9 lines · On Tue, 3 Dec 2019 14:36:38 +0100 Oscar Salvador <[email protected]> wrote:
Quoted 2 lines · David Hildenbrand

@Michal, @Oscar, can some of you at least have a patch #5 now so we can proceed with that? (the other patches can stay in -next some time longer)

Hi,

I will be having a look at patch#5 shortly.

Thanks for the reminder

Things haven't improved a lot :(

mm-memmap_init-update-variable-name-in-memmap_init_zone.patch mm-memory_hotplug-poison-memmap-in-remove_pfn_range_from_zone.patch mm-memory_hotplug-we-always-have-a-zone-in-find_smallestbiggest_section_pfn.patch mm-memory_hotplug-dont-check-for-all-holes-in-shrink_zone_span.patch mm-memory_hotplug-drop-local-variables-in-shrink_zone_span.patch mm-memory_hotplug-cleanup-__remove_pages.patch

Quoted 4 lines
Quoted 1 line

The first patch has reviews, the remainder are unloved.

Trying hard not to rant about the review mentality on this list, but I'm afraid I can't totally bite my tongue ... :)

I am afraid this is less about mentality than the lack of man power. This is not a new problem. We have much more code producers than reviewers.

In this particular case the review is expected from me and I am sorry that my bandwith doesn't scale with the email traffic in my inbox. I do very much appreciate the amount of work you are doing in the hotplug area but we need more reviewers here.

Quoted 11 lines

Now, this is an uncomfortable situation for you and me. You have to ping people about review and patches are stuck in your tree. I have a growing list of patches that are somewhat considered "done", but well, not-upstream-at-all. I have patches that are long in RHEL and were properly tested, but could get dropped any time because -ENOREVIEW.

Our process nowadays seems to be, to only upstream what has an ACK/RB (fixes/features/cleanups). I can understand this is desirable (yet, I am not sure if this makes sense with the current take-and-not-give-back review mentality on this list).

Quoted 5 lines

Although it will make upstreaming stuff *even harder* and *even slower*, maybe we should start to only queue patches that have an ACK/RB, so they won't get blocked by this later on? At least that makes your life easier and people won't have to eventually follow up on patches that have been in linux-next for months.

I wouldn't mind if patched got merged to mmotm less pro-actively at all. People tend to care less to follow up on patches that are in the queue already from my past experience. And also it encourages to generate more code than review.

This is certainly not a black or white of course. Some areas have barely anybody for a review except for the person actively writing code in that area so this really needs the case by case approach.

Anyway this is not a new discussion or a new problem we are facing. I believe that part of the problem is that the MM subsystem doesn't really have official maintainers so there is nobody really responsible for particular parts of the subsystem. Sure Andrew is merging patches based on the review feedback or his gut feeling but I am afraid this is not enough.

Signature · 2 lines
Michal Hocko
SUSE Labs
#71 lore raw
Quoted 8 lines
Quoted 4 lines
Quoted 1 line

The first patch has reviews, the remainder are unloved.

Trying hard not to rant about the review mentality on this list, but I'm afraid I can't totally bite my tongue ... :)

I am afraid this is less about mentality than the lack of man power. This is not a new problem. We have much more code producers than reviewers.

It's part of the "take-and-not-give-back review mentality" and I hope you "smelled" that the comment was not targeted at you.

[...]

Quoted 10 lines
Quoted 5 lines

Although it will make upstreaming stuff *even harder* and *even slower*, maybe we should start to only queue patches that have an ACK/RB, so they won't get blocked by this later on? At least that makes your life easier and people won't have to eventually follow up on patches that have been in linux-next for months.

I wouldn't mind if patched got merged to mmotm less pro-actively at all. People tend to care less to follow up on patches that are in the queue already from my past experience. And also it encourages to generate more code than review.

The current process will at least encourage me in the long term to generate less code (changes) as well. I consider patches that have not been merged upstream as on my TODO list - and it seems to keep on growing. (yes, there are people that fire-and-forget)

Then, I much rather prefer to just get no reply on my patches, ping two times, and eventually dump them into the trash. As explained, will WHP result in me generating less code changes overall. Problem partially solved (at least one producer less) :)

Quoted 4 lines

This is certainly not a black or white of course. Some areas have barely anybody for a review except for the person actively writing code in that area so this really needs the case by case approach.

Yes.

Quoted 7 lines

Anyway this is not a new discussion or a new problem we are facing. I believe that part of the problem is that the MM subsystem doesn't really have official maintainers so there is nobody really responsible for particular parts of the subsystem. Sure Andrew is merging patches based on the review feedback or his gut feeling but I am afraid this is not enough.

If we would have "official maintainers" would it really help (besides Andrew having less stuff to do of course)? E.g., if you would pick up the memory hotplug patches, you would still have to have a look at them
- it's still the producer-consumer imbalance (and you would have an even higher workload).

But yeah, we should most probably finally have official maintainers. For people sending patches, it's often not obvious whom to cc (and whom to ping).

Smells like a "LSF/MM/BPF TOPIC", but as you said, it's not a new problem/discussion. (I won't be around, so I can't bring that topic up)

Anyhow, my two cents.

Signature · 3 lines
Thanks,

David / dhildenb
maintainer
#72 lore raw
In reply to David Hildenbrand
Quoted 27 lines · On Fri, 31 Jan 2020 10:18:34 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 14 lines · Andrew Morton
Quoted 9 lines · On Tue, 3 Dec 2019 14:36:38 +0100 Oscar Salvador <[email protected]> wrote:
Quoted 2 lines · David Hildenbrand

@Michal, @Oscar, can some of you at least have a patch #5 now so we can proceed with that? (the other patches can stay in -next some time longer)

Hi,

I will be having a look at patch#5 shortly.

Thanks for the reminder

Things haven't improved a lot :(

Quoted 8 lines

mm-memmap_init-update-variable-name-in-memmap_init_zone.patch mm-memory_hotplug-poison-memmap-in-remove_pfn_range_from_zone.patch mm-memory_hotplug-we-always-have-a-zone-in-find_smallestbiggest_section_pfn.patch mm-memory_hotplug-dont-check-for-all-holes-in-shrink_zone_span.patch mm-memory_hotplug-drop-local-variables-in-shrink_zone_span.patch mm-memory_hotplug-cleanup-__remove_pages.patch

The first patch has reviews, the remainder are unloved.

Trying hard not to rant about the review mentality on this list, but I'm afraid I can't totally bite my tongue ... :)

Quoted 8 lines

Now, this is an uncomfortable situation for you and me. You have to ping people about review and patches are stuck in your tree. I have a growing list of patches that are somewhat considered "done", but well, not-upstream-at-all. I have patches that are long in RHEL and were properly tested, but could get dropped any time because -ENOREVIEW.

Our process nowadays seems to be, to only upstream what has an ACK/RB (fixes/features/cleanups).

Yes, we've been doing this for a couple of years now. I make an exception for Vitaly's zswap patches because he appears to be the only person who knows the code (since Harry's internship ended).

I think this is the first time we've hit a significant logjam. Presumably the holiday season contributed to this.

It isn't clear to me that we've gained much from this policy. But until this cycle I've seen little harm.

Quoted 9 lines

I can understand this is desirable (yet, I am
not sure if this makes sense with the current take-and-not-give-back review mentality on this list).

Although it will make upstreaming stuff *even harder* and *even slower*, maybe we should start to only queue patches that have an ACK/RB, so they won't get blocked by this later on? At least that makes your life easier and people won't have to eventually follow up on patches that have been in linux-next for months.

The merge rate would still be the review rate, but the resulting merges would be of less tested code.

Quoted 5 lines

Note: the result will be that many of my patches will still not get reviewed, won't get queued/upstreamed, I will continuously ping and resend, I will lose interest because I have better things to do, I will lose interest in our code quality, I will lose interest to review.

Quoted 7 lines

(side note: some people might actually enjoy me sending less cleanup patches, so this approach might be desirable for some ;) )

One alternative is to send patches upstream once they have been lying around in linux-next for $RANDOM number of months, because they obviously saw some testing and nobody started to yell at them once stumbling over them on linux-mm.

Yes, I think that's the case with these patches and I've sent them to Linus. Hopefully Michel will be able to find time to look them over in the next month or so.

#73 lore raw
Quoted 4 lines
Quoted 4 lines

I can understand this is desirable (yet, I am
not sure if this makes sense with the current take-and-not-give-back review mentality on this list).

Quoted 8 lines
Quoted 5 lines

Although it will make upstreaming stuff *even harder* and *even slower*, maybe we should start to only queue patches that have an ACK/RB, so they won't get blocked by this later on? At least that makes your life easier and people won't have to eventually follow up on patches that have been in linux-next for months.

The merge rate would still be the review rate, but the resulting merges would be of less tested code.

That's a valid point.

Quoted 9 lines
Quoted 9 lines

Note: the result will be that many of my patches will still not get reviewed, won't get queued/upstreamed, I will continuously ping and resend, I will lose interest because I have better things to do, I will lose interest in our code quality, I will lose interest to review.

(side note: some people might actually enjoy me sending less cleanup patches, so this approach might be desirable for some ;) )

Quoted 8 lines
Quoted 4 lines

One alternative is to send patches upstream once they have been lying around in linux-next for $RANDOM number of months, because they obviously saw some testing and nobody started to yell at them once stumbling over them on linux-mm.

Yes, I think that's the case with these patches and I've sent them to Linus. Hopefully Michel will be able to find time to look them over in the next month or so.

I really hope we'll find more reviewers in general - I'm also not happy if my patches go upstream with little/no review. However, patches shouldn't be stuck for multiple merge windows in linux-next IMHO (excluding exceptions of course) - then they should either be sent upstream (and eventually fixed later) or dropped.

Thanks Andrew!

Signature · 3 lines
Thanks,

David / dhildenb
maintainer
#74 lore raw
Quoted 5 lines · David Hildenbrand

I really hope we'll find more reviewers in general - I'm also not happy if my patches go upstream with little/no review. However, patches shouldn't be stuck for multiple merge windows in linux-next IMHO (excluding exceptions of course) - then they should either be sent upstream (and eventually fixed later) or dropped.

First of all sorry for my lack of review, as lately I have been a bit disconnected of the list because lack of time.

Lucky my I managed to find some time, so I went through the patches that did lack review (#6-#10).

I hope this helps in moving forward the series, although Michal's review would be great as well.

Signature · 2 lines
Oscar Salvador
SUSE L3