Make MIGRATE_ISOLATE a standalone bit
Subsystems (13)
Zi Yan
reviewerv10 cover Make MIGRATE_ISOLATE a standalone bit
Hi all,
This patchset moves MIGRATE_ISOLATE to a standalone bit to avoid being overwritten during pageblock isolation process. Currently, MIGRATE_ISOLATE is part of enum migratetype (in include/linux/mmzone.h), thus, setting a pageblock to MIGRATE_ISOLATE overwrites its original migratetype. This causes pageblock migratetype loss during alloc_contig_range() and memory offline, especially when the process fails due to a failed pageblock isolation and the code tries to undo the finished pageblock isolations.
It is on top of mm-new.
In terms of performance for changing pageblock types, no performance change is observed:
1. I used perf to collect stats of offlining and onlining all memory of a 40GB VM 10 times and see that get_pfnblock_flags_mask() and set_pfnblock_flags_mask() take about 0.12% and 0.02% of the whole process respectively with and without this patchset across 3 runs.
2. I used perf to collect stats of dd from /dev/random to a 40GB tmpfs file and find get_pfnblock_flags_mask() takes about 0.05% of the process with and without this patchset across 3 runs.
Changelog
===
From V9[8]:
1. changed acr_flags_t item prefix from ACR_ to ACR_FLAGS_ to avoid conflicting with drm/msm hdmi_acr_cts enum.
2. converted kexec_handover to use init_pageblock_migratetype() too.
From V8[7]:
1. made init_pageblock_migratetype() set right migratetype, when page_group_by_mobility_disabled is 1.
From V7[6]:
1. restored acr_flags_t and renamed ACR_OTHER to ACR_NONE
From V6[5]:
1. used MIGRATETYPE_AND_ISO_MASK in init_pageblock_migratetype() too.
2. fixed an indentation issue in Patch 3.
3. removed acr_flags_t and used enum pb_isolate_mode instead in alloc_contig_range().
4. collected review tags.
From V5[4]:
1. used atomic version bitops for pageblock standalone bit operations.
2. added a helper function for standalone bit check.
3. renamed PB_migrate_skip to PB_compact_skip.
4. used #define MIGRATETYPE_AND_ISO_MASK MIGRATETYPE_MASK to simplify !CONFIG_MEMORY_ISOLATION code.
5. added __MIGRATE_TYPE_END to make sure migratetypes can be stored in PB_migratetype_bits.
6. used set and clear to implement toggle_pageblock_isolate() and added VM_WARN_ONCE in __move_freepages_block_isolate() to warn isolating a isolated pageblock and unisolating a not isolated pageblock.
7. dropped toggle_pfnblock_bit().
8. made acr_flags_t an enum and added ACR_OTHER for non CMA allocation.
9. renamed pb_isolate_mode items to have PB_ISOLATE_MODE prefix.
10. collected reviewed-by.
From v4[3]:
1. cleaned up existing pageblock flag functions:
a. added {get,set}_{pfnblock,pageblock}_migratetype() to change
pageblock migratetype
b. added {get,set,clear}_pfnblock_bit() to change pageblock
standalone bit, i.e., PB_migrate_skip and PB_migrate_isolate (added in this series).
c. removed {get,set}_pfnblock_flags_mask().
2. added __NR_PAGEBLOCK_BITS to present the number of pageblock flag bits and used roundup_pow_of_two(__NR_PAGEBLOCK_BITS) as NR_PAGEBLOCK_BITS.
3. moved {get,set,clear}_pageblock_isolate() to linux/page-isolation.h.
4. added init_pageblock_migratetype() to initialize a pageblock with a migratetype and isolated. It is used by memmap_init_range(), which is called by move_pfn_range_to_zone() in online_pages() from mm/memory_hotplug.c. Other set_pageblock_migratetype() users are changed too except the ones in mm/page_alloc.c.
5. toggle_pageblock_isolate() is reimplemented using __change_bit().
6. set_pageblock_migratetype() gives a warning if a pageblock is changed from MIGRATE_ISOLATE to other migratetype.
7. added pb_isolate_mode: MEMORY_OFFLINE, CMA_ALLOCATION, ISOLATE_MODE_OTHERS to replace isolate flags.
8. REPORT_FAILURE is removed, since it is only used by MEMORY_OFFLINE.
From v3[2]:
1. kept the original is_migrate_isolate_page()
2. moved {get,set,clear}_pageblock_isolate() to mm/page_isolation.c
3. used a single version for get_pageblock_migratetype() and get_pfnblock_migratetype().
4. replace get_pageblock_isolate() with get_pageblock_migratetype() == MIGRATE_ISOLATE, a get_pageblock_isolate() becomes private in mm/page_isolation.c
5. made set_pageblock_migratetype() not accept MIGRATE_ISOLATE, so that people need to use the dedicate {get,set,clear}_pageblock_isolate() APIs.
6. changed online_page() from mm/memory_hotplug.c to first set pageblock migratetype to MIGRATE_MOVABLE, then isolate pageblocks.
7. added __maybe_unused to get_pageblock_isolate(), since it is only used in VM_BUG_ON(), which could be not present when MM debug is off. It is reported by kernel test robot.
7. fixed test_pages_isolated() type issues reported by kernel test robot.
From v2[1]:
1. Moved MIGRATETYPE_NO_ISO_MASK to Patch 2, where it is used.
2. Removed spurious changes in Patch 1.
3. Refactored code so that migratetype mask is passed properly for all callers to {get,set}_pfnblock_flags_mask().
4. Added toggle_pageblock_isolate() for setting and clearing MIGRATE_ISOLATE.
5. Changed get_pageblock_migratetype() when CONFIG_MEMORY_ISOLATION to handle MIGRATE_ISOLATE case. It acts like a parsing layer for get_pfnblock_flags_mask().
Design
===
Pageblock flags are read in words to achieve good performance and existing pageblock flags take 4 bits per pageblock. To avoid a substantial change to the pageblock flag code, 8 pageblock flag bits are used.
It might look like the pageblock flags have doubled the overhead, but in reality, the overhead is only 1 byte per 2MB/4MB (based on pageblock config), or 0.0000476 %.
Any comment and/or suggestion is welcome. Thanks.
[1] https://lore.kernel.org/linux-mm/[email protected]/
[2] https://lore.kernel.org/linux-mm/[email protected]/
[3] https://lore.kernel.org/linux-mm/[email protected]/
[4] https://lore.kernel.org/linux-mm/[email protected]/
[5] https://lore.kernel.org/linux-mm/[email protected]/
[6] https://lore.kernel.org/linux-mm/[email protected]/
[7] https://lore.kernel.org/linux-mm/[email protected]/
Zi Yan (6):
mm/page_alloc: pageblock flags functions clean up. mm/page_isolation: make page isolation a standalone bit. mm/page_alloc: add support for initializing pageblock as isolated. mm/page_isolation: remove migratetype from
move_freepages_block_isolate()
mm/page_isolation: remove migratetype from undo_isolate_page_range() mm/page_isolation: remove migratetype parameter from more functions.
Documentation/mm/physical_memory.rst | 2 +- drivers/virtio/virtio_mem.c | 2 +- include/linux/gfp.h | 7 +- include/linux/memory_hotplug.h | 3 +- include/linux/mmzone.h | 21 +- include/linux/page-isolation.h | 47 +++- include/linux/pageblock-flags.h | 48 ++-- include/trace/events/kmem.h | 14 +- kernel/kexec_handover.c | 4 +- mm/cma.c | 2 +- mm/hugetlb.c | 4 +- mm/internal.h | 3 +- mm/memory_hotplug.c | 24 +- mm/memremap.c | 2 +- mm/mm_init.c | 24 +- mm/page_alloc.c | 321 +++++++++++++++++++++------ mm/page_isolation.c | 100 ++++----- 17 files changed, 435 insertions(+), 193 deletions(-)
Signature · 1 line
2.47.2
Zi Yan
reviewerPATCH v10 1/6 mm/page_alloc: pageblock flags functions clean up.
No functional change is intended.
1. Add __NR_PAGEBLOCK_BITS for the number of pageblock flag bits and use roundup_pow_of_two(__NR_PAGEBLOCK_BITS) as NR_PAGEBLOCK_BITS to take right amount of bits for pageblock flags.
2. Rename PB_migrate_skip to PB_compact_skip.
3. Add {get,set,clear}_pfnblock_bit() to operate one a standalone bit, like PB_compact_skip.
3. Make {get,set}_pfnblock_flags_mask() internal functions and use {get,set}_pfnblock_migratetype() for pageblock migratetype operations.
4. Move pageblock flags common code to get_pfnblock_bitmap_bitidx().
3. Use MIGRATETYPE_MASK to get the migratetype of a pageblock from its flags.
4. Use PB_migrate_end in the definition of MIGRATETYPE_MASK instead of PB_migrate_bits.
5. Add a comment on is_migrate_cma_folio() to prevent one from changing it to use get_pageblock_migratetype() and causing issues.
--- Documentation/mm/physical_memory.rst | 2 +- include/linux/mmzone.h | 18 +-- include/linux/page-isolation.h | 2 +- include/linux/pageblock-flags.h | 34 +++--- mm/memory_hotplug.c | 2 +- mm/page_alloc.c | 171 +++++++++++++++++++++------ 6 files changed, 162 insertions(+), 67 deletions(-)
Documentation/mm/physical_memory.rst +1 -1
@@ -584,7 +584,7 @@ Compaction control ``compact_blockskip_flush`` Set to true when compaction migration scanner and free scanner meet, which - means the ``PB_migrate_skip`` bits should be cleared. + means the ``PB_compact_skip`` bits should be cleared. ``contiguous`` Set to true when the zone is contiguous (in other words, no hole).
include/linux/mmzone.h +10 -8
@@ -92,8 +92,12 @@ extern const char * const migratetype_names[MIGRATE_TYPES];#ifdef CONFIG_CMA # define is_migrate_cma (migratetype) unlikely((migratetype) == MIGRATE_CMA)# define is_migrate_cma_page (_page) (get_pageblock_migratetype(_page) == MIGRATE_CMA) -# define is_migrate_cma_folio (folio, pfn) (MIGRATE_CMA == \ - get_pfnblock_flags_mask(&folio->page, pfn, MIGRATETYPE_MASK)) +/* +* __dump_folio() in mm/debug.c passes a folio pointer to on-stack struct folio, +* so folio_pfn() cannot be used and pfn is needed. +*/ +# define is_migrate_cma_folio (folio, pfn) \ + (get_pfnblock_migratetype(&folio->page, pfn) == MIGRATE_CMA)#else # define is_migrate_cma (migratetype) false# define is_migrate_cma_page (_page) false
@@ -122,14 +126,12 @@ static inline bool migratetype_is_mergeable(int mt)extern int page_group_by_mobility_disabled; -#define MIGRATETYPE_MASK ((1UL << PB_migratetype_bits) - 1) +#define get_pageblock_migratetype (page) \ + get_pfnblock_migratetype(page, page_to_pfn(page)) -#define get_pageblock_migratetype (page) \ - get_pfnblock_flags_mask(page, page_to_pfn(page), MIGRATETYPE_MASK) +#define folio_migratetype (folio) \ + get_pageblock_migratetype(&folio->page) -#define folio_migratetype (folio) \ - get_pfnblock_flags_mask(&folio->page, folio_pfn(folio), \ - MIGRATETYPE_MASK)struct free_area {struct list_head free_list[MIGRATE_TYPES ];unsigned long nr_free;
include/linux/page-isolation.h +1 -1
@@ -25,7 +25,7 @@ static inline bool is_migrate_isolate(int migratetype)#define MEMORY_OFFLINE 0x1#define REPORT_FAILURE 0x2 -void set_pageblock_migratetype (struct page *page,int migratetype); +void set_pageblock_migratetype (struct page *page,enum migratetype migratetype);bool move_freepages_block_isolate (struct zone *zone,struct page *page,int migratetype);
include/linux/pageblock-flags.h +17 -17
@@ -19,15 +19,19 @@ enum pageblock_bits { PB_migrate, PB_migrate_end = PB_migrate + PB_migratetype_bits -1 ,/* 3 bits required for migrate types */ - PB_migrate_skip,/* If set the block is skipped by compaction */ + PB_compact_skip,/* If set the block is skipped by compaction */ /* * Assume the bits will always align on a word. If this assumption * changes then get/set pageblock needs updating. */ - NR_PAGEBLOCK_BITS + __NR_PAGEBLOCK_BITS }; +#define NR_PAGEBLOCK_BITS (roundup_pow_of_two(__NR_PAGEBLOCK_BITS)) + +#define MIGRATETYPE_MASK ((1UL << (PB_migrate_end + 1)) - 1) +#if defined(CONFIG_HUGETLB_PAGE )#ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE
@@ -65,27 +69,23 @@ extern unsigned int pageblock_order;/* Forward declaration */ struct page ; -unsigned long get_pfnblock_flags_mask (const struct page *page, -unsigned long pfn, -unsigned long mask); - -void set_pfnblock_flags_mask (struct page *page, -unsigned long flags, -unsigned long pfn, -unsigned long mask); +enum migratetype get_pfnblock_migratetype (const struct page *page, +unsigned long pfn); +bool get_pfnblock_bit (const struct page *page,unsigned long pfn, +enum pageblock_bits pb_bit); +void set_pfnblock_bit (const struct page *page,unsigned long pfn, +enum pageblock_bits pb_bit); +void clear_pfnblock_bit (const struct page *page,unsigned long pfn, +enum pageblock_bits pb_bit);/* Declarations for getting and setting flags. See mm/page_alloc.c */ #ifdef CONFIG_COMPACTION #define get_pageblock_skip (page) \ - get_pfnblock_flags_mask(page, page_to_pfn(page), \ - (1 << (PB_migrate_skip))) + get_pfnblock_bit(page, page_to_pfn(page), PB_compact_skip)#define clear_pageblock_skip (page) \ - set_pfnblock_flags_mask(page, 0, page_to_pfn(page), \ - (1 << PB_migrate_skip)) + clear_pfnblock_bit(page, page_to_pfn(page), PB_compact_skip)#define set_pageblock_skip (page) \ - set_pfnblock_flags_mask(page, (1 << PB_migrate_skip), \ - page_to_pfn(page), \ - (1 << PB_migrate_skip)) + set_pfnblock_bit(page, page_to_pfn(page), PB_compact_skip)#else static inline bool get_pageblock_skip (struct page *page) {
mm/memory_hotplug.c +1 -1
@@ -774,7 +774,7 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,/* * TODO now we have a visible range of pages which are not associated -* with their zone properly. Not nice but set_pfnblock_flags_mask +* with their zone properly. Not nice but set_pfnblock_migratetype() * expects the zone spans the pfn range. All the pages in the range * are reserved so nobody should be touching them so we should be safe */
mm/page_alloc.c +132 -39
241 lines not on this page · raw message
Signature · 1 line
2.47.2
Zi Yan
reviewerPATCH v10 2/6 mm/page_isolation: make page isolation a standalone bit.
In reply to Zi YanDuring page isolation, the original migratetype is overwritten, since MIGRATE_* are enums and stored in pageblock bitmaps. Change MIGRATE_ISOLATE to be stored a standalone bit, PB_migrate_isolate, like PB_compact_skip, so that migratetype is not lost during pageblock isolation.
--- include/linux/mmzone.h | 3 +++ include/linux/page-isolation.h | 16 ++++++++++++++++ include/linux/pageblock-flags.h | 14 ++++++++++++++ mm/page_alloc.c | 27 ++++++++++++++++++++++++--- 4 files changed, 57 insertions(+), 3 deletions(-)
include/linux/mmzone.h +3 -0
@@ -79,6 +79,9 @@ enum migratetype {* __free_pageblock_cma() function. */ MIGRATE_CMA , + __MIGRATE_TYPE_END =MIGRATE_CMA , +#else + __MIGRATE_TYPE_END =MIGRATE_HIGHATOMIC , #endif #ifdef CONFIG_MEMORY_ISOLATION MIGRATE_ISOLATE, /* can't allocate from here */
include/linux/page-isolation.h +16 -0
@@ -11,6 +11,12 @@ static inline bool is_migrate_isolate(int migratetype) {return migratetype ==MIGRATE_ISOLATE ; } +#define get_pageblock_isolate (page) \ + get_pfnblock_bit(page, page_to_pfn(page), PB_migrate_isolate) +#define clear_pageblock_isolate (page) \ + clear_pfnblock_bit(page, page_to_pfn(page), PB_migrate_isolate) +#define set_pageblock_isolate (page) \ + set_pfnblock_bit(page, page_to_pfn(page), PB_migrate_isolate)#else static inline bool is_migrate_isolate_page (struct page *page) {
@@ -20,6 +26,16 @@ static inline bool is_migrate_isolate(int migratetype) {return false; } +static inline bool get_pageblock_isolate (struct page *page) +{ +return false; +} +static inline void clear_pageblock_isolate (struct page *page) +{ +} +static inline void set_pageblock_isolate (struct page *page) +{ +}#endif #define MEMORY_OFFLINE 0x1
include/linux/pageblock-flags.h +14 -0
@@ -21,6 +21,13 @@ enum pageblock_bits {/* 3 bits required for migrate types */ PB_compact_skip,/* If set the block is skipped by compaction */ +#ifdef CONFIG_MEMORY_ISOLATION +/* +* Pageblock isolation is represented with a separate bit, so that +* the migratetype of a block is not overwritten by isolation. +*/ + PB_migrate_isolate, /* If set the block is isolated */ +#endif /* * Assume the bits will always align on a word. If this assumption * changes then get/set pageblock needs updating.
@@ -32,6 +39,13 @@ enum pageblock_bits {#define MIGRATETYPE_MASK ((1UL << (PB_migrate_end + 1)) - 1) +#ifdef CONFIG_MEMORY_ISOLATION +#define MIGRATETYPE_AND_ISO_MASK \ + (((1UL << (PB_migrate_end + 1)) - 1) | BIT(PB_migrate_isolate)) +#else +#define MIGRATETYPE_AND_ISO_MASK MIGRATETYPE_MASK +#endif +#if defined(CONFIG_HUGETLB_PAGE )#ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE
mm/page_alloc.c +24 -3
@@ -365,8 +365,12 @@ get_pfnblock_bitmap_bitidx(const struct page *page, unsigned long pfn,unsigned long *bitmap;unsigned long word_bitidx; +#ifdef CONFIG_MEMORY_ISOLATION +BUILD_BUG_ON (NR_PAGEBLOCK_BITS !=8 ); +#else BUILD_BUG_ON (NR_PAGEBLOCK_BITS !=4 ); -BUILD_BUG_ON (MIGRATE_TYPES > (1 << PB_migratetype_bits)); +#endif +BUILD_BUG_ON (__MIGRATE_TYPE_END >= (1 << PB_migratetype_bits));VM_BUG_ON_PAGE (!zone_spans_pfn (page_zone (page), pfn), page); bitmap =get_pageblock_bitmap (page, pfn);
@@ -439,7 +443,16 @@ bool get_pfnblock_bit(const struct page *page, unsigned long pfn, __always_inline enum migratetypeget_pfnblock_migratetype (const struct page *page,unsigned long pfn) { -return __get_pfnblock_flags_mask (page, pfn,MIGRATETYPE_MASK ); +unsigned long mask =MIGRATETYPE_AND_ISO_MASK ; +unsigned long flags; + + flags =__get_pfnblock_flags_mask (page, pfn, mask); + +#ifdef CONFIG_MEMORY_ISOLATION +if (flags &BIT (PB_migrate_isolate)) +return MIGRATE_ISOLATE ; +#endif +return flags &MIGRATETYPE_MASK ; }/**
@@ -519,8 +532,16 @@ __always_inline void set_pageblock_migratetype(struct page *page, migratetype < MIGRATE_PCPTYPES)) migratetype =MIGRATE_UNMOVABLE ; +#ifdef CONFIG_MEMORY_ISOLATION +if (migratetype ==MIGRATE_ISOLATE ) { +set_pfnblock_bit (page,page_to_pfn (page), PB_migrate_isolate); +return ; + } +/* MIGRATETYPE_AND_ISO_MASK clears PB_migrate_isolate if it is set */ +#endif __set_pfnblock_flags_mask (page,page_to_pfn (page), - (unsigned long )migratetype,MIGRATETYPE_MASK ); + (unsigned long )migratetype, +MIGRATETYPE_AND_ISO_MASK ); }#ifdef CONFIG_DEBUG_VM
Signature · 1 line
2.47.2
Zi Yan
reviewerPATCH v10 3/6 mm/page_alloc: add support for initializing pageblock as isolated.
In reply to Zi YanMIGRATE_ISOLATE is a standalone bit, so a pageblock cannot be initialized to just MIGRATE_ISOLATE. Add init_pageblock_migratetype() to enable initialize a pageblock with a migratetype and isolated.
--- include/linux/memory_hotplug.h | 3 ++- include/linux/page-isolation.h | 3 +++ kernel/kexec_handover.c | 4 ++-- mm/hugetlb.c | 4 ++-- mm/internal.h | 3 ++- mm/memory_hotplug.c | 12 ++++++++---- mm/memremap.c | 2 +- mm/mm_init.c | 24 +++++++++++++++--------- mm/page_alloc.c | 26 ++++++++++++++++++++++++++ 9 files changed, 61 insertions(+), 20 deletions(-)
include/linux/memory_hotplug.h +2 -1
@@ -314,7 +314,8 @@ extern int add_memory_driver_managed(int nid, u64 start, u64 size, mhp_t mhp_flags);extern void move_pfn_range_to_zone (struct zone *zone,unsigned long start_pfn,unsigned long nr_pages, -struct vmem_altmap *altmap,int migratetype); +struct vmem_altmap *altmap,int migratetype, +bool isolate_pageblock);extern void remove_pfn_range_from_zone (struct zone *zone,unsigned long start_pfn,unsigned long nr_pages);
include/linux/page-isolation.h +3 -0
@@ -41,6 +41,9 @@ static inline void set_pageblock_isolate(struct page *page)#define MEMORY_OFFLINE 0x1#define REPORT_FAILURE 0x2 +void __meminit init_pageblock_migratetype (struct page *page, +enum migratetype migratetype, +bool isolate);void set_pageblock_migratetype (struct page *page,enum migratetype migratetype); bool move_freepages_block_isolate(struct zone *zone, struct page *page,
kernel/kexec_handover.c +2 -2
@@ -1100,8 +1100,8 @@ static void __init kho_release_scratch(void)ulong pfn;for (pfn = start_pfn; pfn < end_pfn; pfn += pageblock_nr_pages) -set_pageblock_migratetype (pfn_to_page (pfn), -MIGRATE_CMA ); +init_pageblock_migratetype (pfn_to_page (pfn), +MIGRATE_CMA , false); } }
mm/hugetlb.c +2 -2
@@ -3299,8 +3299,8 @@ static void __init hugetlb_bootmem_init_migratetype(struct folio *folio,if (folio_test_hugetlb_cma (folio))init_cma_pageblock (folio_page (folio, i));else -set_pageblock_migratetype (folio_page (folio, i), -MIGRATE_MOVABLE ); +init_pageblock_migratetype (folio_page (folio, i), +MIGRATE_MOVABLE , false); } }
mm/internal.h +2 -1
@@ -821,7 +821,8 @@ extern void *memmap_alloc(phys_addr_t size, phys_addr_t align, int nid, bool exact_nid);void memmap_init_range (unsigned long ,int ,unsigned long ,unsigned long , -unsigned long ,enum meminit_context ,struct vmem_altmap *,int ); +unsigned long ,enum meminit_context ,struct vmem_altmap *,int , +bool );#if definedCONFIG_COMPACTION || definedCONFIG_CMA
mm/memory_hotplug.c +8 -4
@@ -747,7 +747,8 @@ static inline void section_taint_zone_device(unsigned long pfn)*/ void move_pfn_range_to_zone (struct zone *zone,unsigned long start_pfn,unsigned long nr_pages, -struct vmem_altmap *altmap,int migratetype) +struct vmem_altmap *altmap,int migratetype, +bool isolate_pageblock) {struct pglist_data *pgdat = zone->zone_pgdat;int nid = pgdat->node_id;
@@ -779,7 +780,8 @@ void move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,* are reserved so nobody should be touching them so we should be safe */ memmap_init_range (nr_pages, nid,zone_idx (zone), start_pfn,0 , -MEMINIT_HOTPLUG , altmap, migratetype); +MEMINIT_HOTPLUG , altmap, migratetype, + isolate_pageblock);set_zone_contiguous (zone); }
@@ -1104,7 +1106,8 @@ int mhp_init_memmap_on_memory(unsigned long pfn, unsigned long nr_pages,if (mhp_off_inaccessible)page_init_poison (pfn_to_page (pfn),sizeof (struct page ) * nr_pages); -move_pfn_range_to_zone (zone, pfn, nr_pages,NULL ,MIGRATE_UNMOVABLE ); +move_pfn_range_to_zone (zone, pfn, nr_pages,NULL ,MIGRATE_UNMOVABLE , + false);for (i =0 ; i < nr_pages; i++) {struct page *page =pfn_to_page (pfn + i);
@@ -1171,7 +1174,8 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,/* associate pfn range with the zone */ -move_pfn_range_to_zone (zone, pfn, nr_pages,NULL ,MIGRATE_ISOLATE ); +move_pfn_range_to_zone (zone, pfn, nr_pages,NULL ,MIGRATE_MOVABLE , + true); node_arg.nid =NUMA_NO_NODE ;if (!node_state (nid,N_MEMORY )) {
mm/memremap.c +1 -1
@@ -254,7 +254,7 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params, zone = &NODE_DATA (nid)->node_zones[ZONE_DEVICE ];move_pfn_range_to_zone (zone,PHYS_PFN (range->start),PHYS_PFN (range_len (range)), params->altmap, -MIGRATE_MOVABLE ); +MIGRATE_MOVABLE , false); }mem_hotplug_done ();
mm/mm_init.c +15 -9
@@ -685,7 +685,8 @@ void __meminit __init_page_from_nid(unsigned long pfn, int nid)__init_single_page (pfn_to_page (pfn), pfn, zid, nid);if (pageblock_aligned (pfn)) -set_pageblock_migratetype (pfn_to_page (pfn),MIGRATE_MOVABLE ); +init_pageblock_migratetype (pfn_to_page (pfn),MIGRATE_MOVABLE , + false); }#ifdef CONFIG_DEFERRED_STRUCT_PAGE_INIT
@@ -874,7 +875,8 @@ static void __init init_unavailable_range(unsigned long spfn,void __meminit memmap_init_range (unsigned long size,int nid,unsigned long zone,unsigned long start_pfn,unsigned long zone_end_pfn,enum meminit_context context, -struct vmem_altmap *altmap,int migratetype) +struct vmem_altmap *altmap,int migratetype, +bool isolate_pageblock) {unsigned long pfn, end_pfn = start_pfn + size;struct page *page;
@@ -931,7 +933,8 @@ void __meminit memmap_init_range(unsigned long size, int nid, unsigned long zone* over the place during system boot. */ if (pageblock_aligned (pfn)) { -set_pageblock_migratetype (page, migratetype); +init_pageblock_migratetype (page, migratetype, + isolate_pageblock);cond_resched (); } pfn++;
@@ -954,7 +957,8 @@ static void __init memmap_init_zone_range(struct zone *zone,return ;memmap_init_range (end_pfn - start_pfn, nid, zone_id, start_pfn, - zone_end_pfn,MEMINIT_EARLY ,NULL ,MIGRATE_MOVABLE ); + zone_end_pfn,MEMINIT_EARLY ,NULL ,MIGRATE_MOVABLE , + false);if (*hole_pfn < start_pfn)init_unavailable_range (*hole_pfn, start_pfn, zone_id, nid);
@@ -1035,7 +1039,7 @@ static void __ref __init_zone_device_page(struct page *page, unsigned long pfn,* because this is done early in section_activate() */ if (pageblock_aligned (pfn)) { -set_pageblock_migratetype (page,MIGRATE_MOVABLE ); +init_pageblock_migratetype (page,MIGRATE_MOVABLE , false);cond_resched (); }
@@ -1996,7 +2000,8 @@ static void __init deferred_free_pages(unsigned long pfn,/* Free a large naturally-aligned chunk if possible */ if (nr_pages ==MAX_ORDER_NR_PAGES &&IS_MAX_ORDER_ALIGNED (pfn)) {for (i =0 ; i < nr_pages; i += pageblock_nr_pages) -set_pageblock_migratetype (page + i,MIGRATE_MOVABLE ); +init_pageblock_migratetype (page + i,MIGRATE_MOVABLE , + false);__free_pages_core (page,MAX_PAGE_ORDER ,MEMINIT_EARLY );return ; }
@@ -2006,7 +2011,8 @@ static void __init deferred_free_pages(unsigned long pfn,for (i =0 ; i < nr_pages; i++, page++, pfn++) {if (pageblock_aligned (pfn)) -set_pageblock_migratetype (page,MIGRATE_MOVABLE ); +init_pageblock_migratetype (page,MIGRATE_MOVABLE , + false);__free_pages_core (page,0 ,MEMINIT_EARLY ); } }
@@ -2305,7 +2311,7 @@ void __init init_cma_reserved_pageblock(struct page *page)set_page_count (p,0 ); }while (++p, --i); -set_pageblock_migratetype (page,MIGRATE_CMA ); +init_pageblock_migratetype (page,MIGRATE_CMA , false);set_page_refcounted (page);/* pages were reserved and not allocated */ clear_page_tag_ref (page);
@@ -2319,7 +2325,7 @@ void __init init_cma_reserved_pageblock(struct page *page)*/ void __init init_cma_pageblock (struct page *page) { -set_pageblock_migratetype (page,MIGRATE_CMA ); +init_pageblock_migratetype (page,MIGRATE_CMA , false);adjust_managed_page_count (page, pageblock_nr_pages);page_zone (page)->cma_pages += pageblock_nr_pages; }
mm/page_alloc.c +26 -0
@@ -544,6 +544,32 @@ __always_inline void set_pageblock_migratetype(struct page *page, MIGRATETYPE_AND_ISO_MASK); } +void __meminit init_pageblock_migratetype (struct page *page, +enum migratetype migratetype, +bool isolate) +{ +unsigned long flags; + +if (unlikely (page_group_by_mobility_disabled && + migratetype <MIGRATE_PCPTYPES )) + migratetype =MIGRATE_UNMOVABLE ; + + flags = migratetype; + +#ifdef CONFIG_MEMORY_ISOLATION +if (migratetype ==MIGRATE_ISOLATE ) { +VM_WARN_ONCE ( +1 , +"Set isolate=true to isolate pageblock with a migratetype" ); +return ; + } +if (isolate) + flags |=BIT (PB_migrate_isolate); +#endif +__set_pfnblock_flags_mask (page,page_to_pfn (page), flags, +MIGRATETYPE_AND_ISO_MASK ); +} +#ifdef CONFIG_DEBUG_VM static int page_outside_zone_boundaries (struct zone *zone,struct page *page) {
Signature · 1 line
2.47.2
Zi Yan
reviewerPATCH v10 4/6 mm/page_isolation: remove migratetype from move_freepages_block_isolate()
In reply to Zi YanSince migratetype is no longer overwritten during pageblock isolation, moving a pageblock out of MIGRATE_ISOLATE no longer needs a new migratetype.
Add pageblock_isolate_and_move_free_pages() and pageblock_unisolate_and_move_free_pages() to be explicit about the page isolation operations. Both share the common code in __move_freepages_block_isolate(), which is renamed from move_freepages_block_isolate().
Add toggle_pageblock_isolate() to flip pageblock isolation bit in __move_freepages_block_isolate().
Make set_pageblock_migratetype() only accept non MIGRATE_ISOLATE types, so that one should use set_pageblock_isolate() to isolate pageblocks. As a result, move pageblock migratetype code out of __move_freepages_block().
--- include/linux/page-isolation.h | 5 +-- mm/page_alloc.c | 80 ++++++++++++++++++++++++++-------- mm/page_isolation.c | 21 +++++---- 3 files changed, 75 insertions(+), 31 deletions(-)
include/linux/page-isolation.h +2 -3
@@ -44,10 +44,9 @@ static inline void set_pageblock_isolate(struct page *page)void __meminit init_pageblock_migratetype (struct page *page,enum migratetype migratetype,bool isolate); -void set_pageblock_migratetype (struct page *page,enum migratetype migratetype); -bool move_freepages_block_isolate (struct zone *zone,struct page *page, -int migratetype); +bool pageblock_isolate_and_move_free_pages (struct zone *zone,struct page *page); +bool pageblock_unisolate_and_move_free_pages (struct zone *zone,struct page *page);int start_isolate_page_range (unsigned long start_pfn,unsigned long end_pfn,int migratetype,int flags);
mm/page_alloc.c +63 -17
157 lines not on this page · raw message
mm/page_isolation.c +10 -11
@@ -188,7 +188,7 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_ unmovable =has_unmovable_pages (check_unmovable_start, check_unmovable_end, migratetype, isol_flags);if (!unmovable) { -if (!move_freepages_block_isolate (zone, page,MIGRATE_ISOLATE )) { +if (!pageblock_isolate_and_move_free_pages (zone, page)) {spin_unlock_irqrestore (&zone->lock, flags);return -EBUSY ; }
@@ -209,7 +209,7 @@ static int set_migratetype_isolate(struct page *page, int migratetype, int isol_return -EBUSY ; } -static void unset_migratetype_isolate (struct page *page,int migratetype) +static void unset_migratetype_isolate (struct page *page) {struct zone *zone;unsigned long flags;
@@ -262,10 +262,10 @@ static void unset_migratetype_isolate(struct page *page, int migratetype)* Isolating this block already succeeded, so this * should not fail on zone boundaries. */ -WARN_ON_ONCE (!move_freepages_block_isolate (zone, page, migratetype)); +WARN_ON_ONCE (!pageblock_unisolate_and_move_free_pages (zone, page)); }else { -set_pageblock_migratetype (page, migratetype); -__putback_isolated_page (page, order, migratetype); +clear_pageblock_isolate (page); +__putback_isolated_page (page, order,get_pageblock_migratetype (page)); } zone->nr_isolate_pageblock--; out:
@@ -383,7 +383,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags,if (PageBuddy (page)) {int order =buddy_order (page); -/* move_freepages_block_isolate() handled this */ +/* pageblock_isolate_and_move_free_pages() handled this */ VM_WARN_ON_ONCE (pfn + (1 << order) > boundary_pfn); pfn +=1UL << order;
@@ -433,7 +433,7 @@ static int isolate_single_pageblock(unsigned long boundary_pfn, int flags, failed:/* restore the original migratetype */ if (!skip_isolation) -unset_migratetype_isolate (pfn_to_page (isolate_pageblock), migratetype); +unset_migratetype_isolate (pfn_to_page (isolate_pageblock));return -EBUSY ; }
@@ -504,7 +504,7 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, ret =isolate_single_pageblock (isolate_end, flags, true, skip_isolation, migratetype);if (ret) { -unset_migratetype_isolate (pfn_to_page (isolate_start), migratetype); +unset_migratetype_isolate (pfn_to_page (isolate_start));return ret; }
@@ -517,8 +517,7 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, start_pfn, end_pfn)) {undo_isolate_page_range (isolate_start, pfn, migratetype);unset_migratetype_isolate ( -pfn_to_page (isolate_end - pageblock_nr_pages), - migratetype); +pfn_to_page (isolate_end - pageblock_nr_pages));return -EBUSY ; } }
@@ -548,7 +547,7 @@ void undo_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, page =__first_valid_page (pfn, pageblock_nr_pages);if (!page || !is_migrate_isolate_page (page))continue ; -unset_migratetype_isolate (page, migratetype); +unset_migratetype_isolate (page); } }/*
Signature · 1 line
2.47.2
Zi Yan
reviewerPATCH v10 5/6 mm/page_isolation: remove migratetype from undo_isolate_page_range()
In reply to Zi YanSince migratetype is no longer overwritten during pageblock isolation, undoing pageblock isolation no longer needs which migratetype to restore.
--- include/linux/page-isolation.h | 3 +-- mm/memory_hotplug.c | 4 ++-- mm/page_alloc.c | 2 +- mm/page_isolation.c | 9 +++------ 4 files changed, 7 insertions(+), 11 deletions(-)
include/linux/page-isolation.h +1 -2
@@ -51,8 +51,7 @@ bool pageblock_unisolate_and_move_free_pages(struct zone *zone, struct page *pagint start_isolate_page_range (unsigned long start_pfn,unsigned long end_pfn,int migratetype,int flags); -void undo_isolate_page_range (unsigned long start_pfn,unsigned long end_pfn, -int migratetype); +void undo_isolate_page_range (unsigned long start_pfn,unsigned long end_pfn);int test_pages_isolated (unsigned long start_pfn,unsigned long end_pfn,int isol_flags);
mm/memory_hotplug.c +2 -2
@@ -1223,7 +1223,7 @@ int online_pages(unsigned long pfn, unsigned long nr_pages,build_all_zonelists (NULL );/* Basic onlining is complete, allow allocation of onlined pages. */ -undo_isolate_page_range (pfn, pfn + nr_pages,MIGRATE_MOVABLE ); +undo_isolate_page_range (pfn, pfn + nr_pages);/* * Freshly onlined pages aren't shuffled (e.g., all pages are placed to
@@ -2092,7 +2092,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages, failed_removal_isolated:/* pushback to free area */ -undo_isolate_page_range (start_pfn, end_pfn,MIGRATE_MOVABLE ); +undo_isolate_page_range (start_pfn, end_pfn);if (cancel_mem_notifier_on_err)memory_notify (MEM_CANCEL_OFFLINE , &mem_arg);if (cancel_node_notifier_on_err)
mm/page_alloc.c +1 -1
@@ -6984,7 +6984,7 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end, start, end, outer_start, outer_end); } done: -undo_isolate_page_range (start, end, migratetype); +undo_isolate_page_range (start, end);return ret; }EXPORT_SYMBOL (alloc_contig_range_noprof);
mm/page_isolation.c +3 -6
@@ -515,7 +515,7 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn, page =__first_valid_page (pfn, pageblock_nr_pages);if (page &&set_migratetype_isolate (page, migratetype, flags, start_pfn, end_pfn)) { -undo_isolate_page_range (isolate_start, pfn, migratetype); +undo_isolate_page_range (isolate_start, pfn);unset_migratetype_isolate (pfn_to_page (isolate_end - pageblock_nr_pages));return -EBUSY ;
@@ -528,13 +528,10 @@ int start_isolate_page_range(unsigned long start_pfn, unsigned long end_pfn,* undo_isolate_page_range - undo effects of start_isolate_page_range() * @start_pfn: The first PFN of the isolated range * @end_pfn: The last PFN of the isolated range -* @migratetype: New migrate type to set on the range * -* This finds every MIGRATE_ISOLATE page block in the given range -* and switches it to @migratetype. +* This finds and unsets every MIGRATE_ISOLATE page block in the given range */ -void undo_isolate_page_range (unsigned long start_pfn,unsigned long end_pfn, -int migratetype) +void undo_isolate_page_range (unsigned long start_pfn,unsigned long end_pfn) {unsigned long pfn;struct page *page;
Signature · 1 line
2.47.2
Zi Yan
reviewerPATCH v10 6/6 mm/page_isolation: remove migratetype parameter from more functions.
In reply to Zi Yanmigratetype is no longer overwritten during pageblock isolation, start_isolate_page_range(), has_unmovable_pages(), and set_migratetype_isolate() no longer need which migratetype to restore during isolation failure.
For has_unmoable_pages(), it needs to know if the isolation is for CMA allocation, so adding PB_ISOLATE_MODE_CMA_ALLOC provide the information. At the same time change isolation flags to enum pb_isolate_mode (PB_ISOLATE_MODE_MEM_OFFLINE, PB_ISOLATE_MODE_CMA_ALLOC, PB_ISOLATE_MODE_OTHER). Remove REPORT_FAILURE and check PB_ISOLATE_MODE_MEM_OFFLINE, since only PB_ISOLATE_MODE_MEM_OFFLINE reports isolation failures.
alloc_contig_range() no longer needs migratetype. Replace it with a newly defined acr_flags_t to tell if an allocation is for CMA. So does __alloc_contig_migrate_range(). Add ACR_FLAGS_NONE (set to 0) to indicate ordinary allocations.
--- drivers/virtio/virtio_mem.c | 2 +- include/linux/gfp.h | 7 +++- include/linux/page-isolation.h | 20 ++++++++-- include/trace/events/kmem.h | 14 ++++--- mm/cma.c | 2 +- mm/memory_hotplug.c | 6 +-- mm/page_alloc.c | 27 ++++++------- mm/page_isolation.c | 70 +++++++++++++++------------------- 8 files changed, 80 insertions(+), 68 deletions(-)
drivers/virtio/virtio_mem.c +1 -1
@@ -1243,7 +1243,7 @@ static int virtio_mem_fake_offline(struct virtio_mem *vm, unsigned long pfn,if (atomic_read (&vm->config_changed))return -EAGAIN ; - rc =alloc_contig_range (pfn, pfn + nr_pages,MIGRATE_MOVABLE , + rc =alloc_contig_range (pfn, pfn + nr_pages,ACR_FLAGS_NONE ,GFP_KERNEL );if (rc == -ENOMEM )/* whoops, out of memory */
include/linux/gfp.h +6 -1
@@ -423,9 +423,14 @@ static inline bool gfp_compaction_allowed(gfp_t gfp_mask)extern gfp_t vma_thp_gfp_mask (struct vm_area_struct *vma);#ifdef CONFIG_CONTIG_ALLOC + +typedef unsigned int __bitwise acr_flags_t ; +#define ACR_FLAGS_NONE ((__force acr_flags_t)0) // ordinary allocation request +#define ACR_FLAGS_CMA ((__force acr_flags_t)BIT(0)) // allocate for CMA +/* The below functions must be run on a range from a single zone. */ extern int alloc_contig_range_noprof (unsigned long start,unsigned long end, -unsigned migratetype,gfp_t gfp_mask); +acr_flags_t alloc_flags,gfp_t gfp_mask);#define alloc_contig_range (...) alloc_hooks(alloc_contig_range_noprof(__VA_ARGS__)) extern struct page *alloc_contig_pages_noprof(unsigned long nr_pages, gfp_t gfp_mask,
include/linux/page-isolation.h +16 -4
@@ -38,8 +38,20 @@ static inline void set_pageblock_isolate(struct page *page) }#endif -#define MEMORY_OFFLINE 0x1 -#define REPORT_FAILURE 0x2 +/* +* Pageblock isolation modes: +* PB_ISOLATE_MODE_MEM_OFFLINE - isolate to offline (!allocate) memory +* e.g., skip over PageHWPoison() pages and +* PageOffline() pages. Unmovable pages will be +* reported in this mode. +* PB_ISOLATE_MODE_CMA_ALLOC - isolate for CMA allocations +* PB_ISOLATE_MODE_OTHER - isolate for other purposes +*/ +enum pb_isolate_mode { +PB_ISOLATE_MODE_MEM_OFFLINE , +PB_ISOLATE_MODE_CMA_ALLOC , +PB_ISOLATE_MODE_OTHER , +}; void __meminit init_pageblock_migratetype(struct page *page, enum migratetype migratetype,
@@ -49,10 +61,10 @@ bool pageblock_isolate_and_move_free_pages(struct zone *zone, struct page *page)bool pageblock_unisolate_and_move_free_pages (struct zone *zone,struct page *page);int start_isolate_page_range (unsigned long start_pfn,unsigned long end_pfn, -int migratetype,int flags); +enum pb_isolate_mode mode);void undo_isolate_page_range (unsigned long start_pfn,unsigned long end_pfn);int test_pages_isolated (unsigned long start_pfn,unsigned long end_pfn, -int isol_flags); +enum pb_isolate_mode mode);#endif
include/trace/events/kmem.h +8 -6
@@ -304,6 +304,7 @@ TRACE_EVENT(mm_page_alloc_extfrag, __entry->change_ownership) ); +#ifdef CONFIG_CONTIG_ALLOC TRACE_EVENT(mm_alloc_contig_migrate_range_info, TP_PROTO(unsigned long start,
@@ -311,9 +312,9 @@ TRACE_EVENT(mm_alloc_contig_migrate_range_info,unsigned long nr_migrated, unsigned long nr_reclaimed, unsigned long nr_mapped, - int migratetype), + acr_flags_t alloc_flags), -TP_ARGS (start ,end ,nr_migrated ,nr_reclaimed ,nr_mapped ,migratetype ), +TP_ARGS (start ,end ,nr_migrated ,nr_reclaimed ,nr_mapped ,alloc_flags ),TP_STRUCT__entry ( __field(unsigned long, start)
@@ -321,7 +322,7 @@ TRACE_EVENT(mm_alloc_contig_migrate_range_info, __field(unsigned long, nr_migrated) __field(unsigned long, nr_reclaimed) __field(unsigned long, nr_mapped) - __field(int, migratetype) + __field(acr_flags_t, alloc_flags) ), TP_fast_assign(
@@ -330,17 +331,18 @@ TRACE_EVENT(mm_alloc_contig_migrate_range_info, __entry->nr_migrated = nr_migrated; __entry->nr_reclaimed = nr_reclaimed; __entry->nr_mapped = nr_mapped; - __entry->migratetype = migratetype; + __entry->alloc_flags = alloc_flags; ), -TP_printk ("start=0x%lx end=0x%lx migratetype=%d nr_migrated=%lu nr_reclaimed=%lu nr_mapped=%lu" , +TP_printk ("start=0x%lx end=0x%lx alloc_flags=%d nr_migrated=%lu nr_reclaimed=%lu nr_mapped=%lu" , __entry->start, __entry->end, - __entry->migratetype, + __entry->alloc_flags, __entry->nr_migrated, __entry->nr_reclaimed, __entry->nr_mapped) ); +#endif TRACE_EVENT(mm_setup_per_zone_wmarks,
mm/cma.c +1 -1
@@ -822,7 +822,7 @@ static int cma_range_alloc(struct cma *cma, struct cma_memrange *cmr, pfn = cmr->base_pfn + (bitmap_no << cma->order_per_bit);mutex_lock (&cma->alloc_mutex); - ret =alloc_contig_range (pfn, pfn + count,MIGRATE_CMA , gfp); + ret =alloc_contig_range (pfn, pfn + count,ACR_FLAGS_CMA , gfp);mutex_unlock (&cma->alloc_mutex);if (ret ==0 ) { page =pfn_to_page (pfn);
mm/memory_hotplug.c +3 -3
@@ -1961,8 +1961,7 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages,/* set above range as isolated */ ret =start_isolate_page_range (start_pfn, end_pfn, -MIGRATE_MOVABLE , -MEMORY_OFFLINE |REPORT_FAILURE ); +PB_ISOLATE_MODE_MEM_OFFLINE );if (ret) { reason ="failure to isolate range" ; goto failed_removal_pcplists_disabled;
@@ -2039,7 +2038,8 @@ int offline_pages(unsigned long start_pfn, unsigned long nr_pages, goto failed_removal_isolated; } - ret =test_pages_isolated (start_pfn, end_pfn,MEMORY_OFFLINE ); + ret =test_pages_isolated (start_pfn, end_pfn, +PB_ISOLATE_MODE_MEM_OFFLINE ); }while (ret);
mm/page_alloc.c +14 -13
@@ -6693,11 +6693,12 @@ static void alloc_contig_dump_pages(struct list_head *page_list)/* * [start, end) must belong to a single zone. -* @migratetype: using migratetype to filter the type of migration in +* @alloc_flags: using acr_flags_t to filter the type of migration in * trace_mm_alloc_contig_migrate_range_info. */ static int __alloc_contig_migrate_range (struct compact_control *cc, -unsigned long start,unsigned long end,int migratetype) +unsigned long start,unsigned long end, +acr_flags_t alloc_flags) {/* This function is based on compact_zone() from compaction.c. */ unsigned int nr_reclaimed;
@@ -6769,7 +6770,7 @@ static int __alloc_contig_migrate_range(struct compact_control *cc,putback_movable_pages (&cc->migratepages); } -trace_mm_alloc_contig_migrate_range_info (start, end, migratetype, +trace_mm_alloc_contig_migrate_range_info (start, end, alloc_flags, total_migrated, total_reclaimed, total_mapped);
@@ -6840,10 +6841,7 @@ static int __alloc_contig_verify_gfp_mask(gfp_t gfp_mask, gfp_t *gfp_cc_mask)* alloc_contig_range() -- tries to allocate given range of pages * @start: start PFN to allocate * @end: one-past-the-last PFN to allocate -* @migratetype: migratetype of the underlying pageblocks (either -* #MIGRATE_MOVABLE or #MIGRATE_CMA). All pageblocks -* in range must have the same migratetype and it must -* be either of the two. +* @alloc_flags: allocation information * @gfp_mask: GFP mask. Node/zone/placement hints are ignored; only some * action and reclaim modifiers are supported. Reclaim modifiers * control allocation behavior during compaction/migration/reclaim.
@@ -6860,7 +6858,7 @@ static int __alloc_contig_verify_gfp_mask(gfp_t gfp_mask, gfp_t *gfp_cc_mask)* need to be freed with free_contig_range(). */ int alloc_contig_range_noprof (unsigned long start,unsigned long end, -unsigned migratetype,gfp_t gfp_mask) +acr_flags_t alloc_flags,gfp_t gfp_mask) {unsigned long outer_start, outer_end;int ret =0 ;
@@ -6875,6 +6873,9 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end, .alloc_contig = true, };INIT_LIST_HEAD (&cc.migratepages); +enum pb_isolate_mode mode = (alloc_flags &ACR_FLAGS_CMA ) ? +PB_ISOLATE_MODE_CMA_ALLOC : +PB_ISOLATE_MODE_OTHER ; gfp_mask =current_gfp_context (gfp_mask);if (__alloc_contig_verify_gfp_mask (gfp_mask, (gfp_t *)&cc.gfp_mask))
@@ -6901,7 +6902,7 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end,* put back to page allocator so that buddy can use them. */ - ret =start_isolate_page_range (start, end, migratetype,0 ); + ret =start_isolate_page_range (start, end, mode);if (ret) goto done;
@@ -6917,7 +6918,7 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end,* allocated. So, if we fall through be sure to clear ret so that * -EBUSY is not accidentally used or returned to caller. */ - ret =__alloc_contig_migrate_range (&cc, start, end, migratetype); + ret =__alloc_contig_migrate_range (&cc, start, end, alloc_flags);if (ret && ret != -EBUSY ) goto done;
@@ -6951,7 +6952,7 @@ int alloc_contig_range_noprof(unsigned long start, unsigned long end, outer_start =find_large_buddy (start);/* Make sure the range is really isolated. */ -if (test_pages_isolated (outer_start, end,0 )) { +if (test_pages_isolated (outer_start, end, mode)) { ret = -EBUSY ; goto done; }
@@ -6994,8 +6995,8 @@ static int __alloc_contig_pages(unsigned long start_pfn, {unsigned long end_pfn = start_pfn + nr_pages; -return alloc_contig_range_noprof (start_pfn, end_pfn,MIGRATE_MOVABLE , - gfp_mask); +return alloc_contig_range_noprof (start_pfn, end_pfn,ACR_FLAGS_NONE , + gfp_mask); } static bool pfn_range_valid_contig(struct zone *z, unsigned long start_pfn,
mm/page_isolation.c +31 -39
199 lines not on this page · raw message
Signature · 1 line
2.47.2