mm: improve page allocator scalability via splitting zones
Subsystems (20)
cover mm: improve page allocator scalability via splitting zones
The patchset is based on upstream v6.3.
More and more cores are put in one physical CPU (usually one NUMA node too). In 2023, one high-end server CPU has 56, 64, or more cores. Even more cores per physical CPU are planned for future CPUs. While all cores in one physical CPU will contend for the page allocation on one zone in most cases. This causes heavy zone lock contention in some workloads. And the situation will become worse and worse in the future.
For example, on an 2-socket Intel server machine with 224 logical CPUs, if the kernel is built with make -j224, the zone lock contention cycles% can reach up to about 12.7%.
To improve the scalability of the page allocation, in this series, we will create one zone instance for each about 256 GB memory of a zone type generally. That is, one large zone type will be split into multiple zone instances. Then, different logical CPUs will prefer different zone instances based on the logical CPU No. So the total number of logical CPUs contend on one zone will be reduced. Thus the scalability is improved.
With the series, the zone lock contention cycles% reduces to less than 1.6% in the above kbuild test case when 4 zone instances are created for ZONE_NORMAL.
Also tested the series with the will-it-scale/page_fault1 with 16 processes. With the optimization, the benchmark score increases up to 18.2% and the zone lock contention reduces from 13.01% to 0.56%.
To create multiple zone instances for a zone type, another choice is to create zone instances based on the total number of logical CPUs. We choose to use memory size because it is easier to be implemented. In most cases, the more the cores, the larger the memory size is. And, on system with larger memory size, the performance requirement of the page allocator is usually higher.
Best Regards,
Huang, Ying
RFC 1/6 mm: distinguish zone type and zone instance explicitly
In current kernel, there is only 1 zone instance at most for each zone type in each NUMA node. So, zone type and zone instance are used inter-changeable in source code. For example, to specify the highest zone type to reclaim pages from via kswapd, there is "kswapd_highest_zoneidx" instead of "kswapd_highest_zone_type" field in struct pglist_data.
In the following patch in this series, multiple zone instances may be created for one zone type. So, in this patch, the zone instance and zone type usage in source code are checked and changed if necessary to distinguish them explicitly.
Although the line number of the patch is large, it only makes changes mechanically.
--- arch/x86/mm/init.c | 2 +- drivers/crypto/omap-crypto.c | 2 +- drivers/crypto/omap-sham.c | 2 +- drivers/virtio/virtio_mem.c | 2 +- include/linux/compaction.h | 8 +- include/linux/gfp.h | 4 +- include/linux/mm.h | 6 +- include/linux/mm_inline.h | 10 +- include/linux/mmzone.h | 112 +++++++++------- include/linux/page-flags-layout.h | 12 ++ include/linux/page-flags.h | 4 +- include/linux/vmstat.h | 4 +- include/trace/events/oom.h | 6 +- kernel/bounds.c | 1 + lib/show_mem.c | 2 +- mm/compaction.c | 58 ++++----- mm/internal.h | 18 +-- mm/memory_hotplug.c | 36 +++--- mm/mempolicy.c | 4 +- mm/memremap.c | 6 +- mm/migrate.c | 6 +- mm/mmzone.c | 8 +- mm/oom_kill.c | 4 +- mm/page-writeback.c | 9 +- mm/page_alloc.c | 190 ++++++++++++++------------- mm/page_isolation.c | 2 +- mm/shmem.c | 2 +- mm/slab.c | 4 +- mm/slub.c | 4 +- mm/vmscan.c | 205 ++++++++++++++++-------------- 30 files changed, 397 insertions(+), 336 deletions(-)
arch/x86/mm/init.c +1 -1
@@ -1024,7 +1024,7 @@ void __init memblock_find_dma_reserve(void)void __init zone_sizes_init (void ) { -unsigned long max_zone_pfns[MAX_NR_ZONES ]; +unsigned long max_zone_pfns[MAX_NR_ZONE_TYPES ];memset (max_zone_pfns,0 ,sizeof (max_zone_pfns));
drivers/crypto/omap-crypto.c +1 -1
@@ -102,7 +102,7 @@ static int omap_crypto_check_sg(struct scatterlist *sg, int total, int bs,if (!IS_ALIGNED (sg->length, bs))return OMAP_CRYPTO_NOT_ALIGNED ;#ifdef CONFIG_ZONE_DMA -if (page_zonenum (sg_page (sg)) !=ZONE_DMA ) +if (page_zone_type (sg_page (sg)) !=ZONE_DMA )return OMAP_CRYPTO_NOT_ALIGNED ;#endif
drivers/crypto/omap-sham.c +1 -1
@@ -782,7 +782,7 @@ static int omap_sham_align_sgs(struct scatterlist *sg, }#ifdef CONFIG_ZONE_DMA -if (page_zonenum (sg_page (sg_tmp)) !=ZONE_DMA ) { +if (page_zone_type (sg_page (sg_tmp)) !=ZONE_DMA ) { aligned = false;break ; }
drivers/virtio/virtio_mem.c +1 -1
@@ -2201,7 +2201,7 @@ static bool virtio_mem_bbm_bb_is_movable(struct virtio_mem *vm, page =pfn_to_online_page (pfn);if (!page)continue ; -if (page_zonenum (page) !=ZONE_MOVABLE ) +if (page_zone_type (page) !=ZONE_MOVABLE )return false; }
include/linux/compaction.h +4 -4
@@ -97,7 +97,7 @@ extern enum compact_result try_to_compact_pages(gfp_t gfp_mask, struct page **page);extern void reset_isolation_suitable (pg_data_t *pgdat);extern enum compact_result compaction_suitable (struct zone *zone,int order, -unsigned int alloc_flags,int highest_zoneidx); +unsigned int alloc_flags,enum zone_type highest_zone_type);extern void compaction_defer_reset (struct zone *zone,int order,bool alloc_success);
@@ -179,7 +179,7 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order,extern void kcompactd_run (int nid);extern void kcompactd_stop (int nid); -extern void wakeup_kcompactd (pg_data_t *pgdat,int order,int highest_zoneidx); +extern void wakeup_kcompactd (pg_data_t *pgdat,int order,enum zone_type highest_zone_type);#else static inline void reset_isolation_suitable (pg_data_t *pgdat)
@@ -187,7 +187,7 @@ static inline void reset_isolation_suitable(pg_data_t *pgdat) }static inline enum compact_result compaction_suitable (struct zone *zone,int order, -int alloc_flags,int highest_zoneidx) +int alloc_flags,enum zone_type highest_zone_type) {return COMPACT_SKIPPED ; }
@@ -220,7 +220,7 @@ static inline void kcompactd_stop(int nid) }static inline void wakeup_kcompactd (pg_data_t *pgdat, -int order,int highest_zoneidx) +int order,enum zone_type highest_zone_type) { }
include/linux/gfp.h +2 -2
@@ -87,11 +87,11 @@ static inline bool gfpflags_allow_blocking(const gfp_t gfp_flags)* GFP_ZONES_SHIFT must be <= 2 on 32 bit platforms. */ -#if defined(CONFIG_ZONE_DEVICE ) && (MAX_NR_ZONES -1 ) <=4 +#if defined(CONFIG_ZONE_DEVICE ) && (MAX_NR_ZONE_TYPES -1 ) <=4 /* ZONE_DEVICE is not a valid GFP zone specifier */ #define GFP_ZONES_SHIFT 2#else -#define GFP_ZONES_SHIFT ZONES_SHIFT +#define GFP_ZONES_SHIFT ZONE_TYPES_SHIFT#endif #if 16 *GFP_ZONES_SHIFT >BITS_PER_LONG
include/linux/mm.h +3 -3
@@ -1661,7 +1661,7 @@ static inline void page_kasan_tag_reset(struct page *page) { }static inline struct zone *page_zone (const struct page *page) { -return &NODE_DATA (page_to_nid (page))->node_zones[page_zonenum (page)]; +return &NODE_DATA (page_to_nid (page))->node_zones[page_zone_idx (page)]; }static inline pg_data_t *page_pgdat (const struct page *page)
@@ -2840,7 +2840,7 @@ static inline unsigned long get_num_physpages(void)* free_area_init() passing in the PFN each zone ends at. At a basic * usage, an architecture is expected to do something like * -* unsigned long max_zone_pfns[MAX_NR_ZONES] = {max_dma, max_normal_pfn, +* unsigned long max_zone_pfns[MAX_NR_ZONE_TYPES] = {max_dma, max_normal_pfn, * max_highmem_pfn}; * for_each_valid_physical_page_range() * memblock_add_node(base, size, nid, MEMBLOCK_NONE)
@@ -2866,7 +2866,7 @@ extern int __meminit early_pfn_to_nid(unsigned long pfn);#endif extern void set_dma_reserve (unsigned long new_dma_reserve); -extern void memmap_init_range (unsigned long ,int ,unsigned long , +extern void memmap_init_range (unsigned long ,int ,enum zone_type ,unsigned long ,unsigned long ,unsigned long ,enum meminit_context ,struct vmem_altmap *,int migratetype);extern void setup_per_zone_wmarks (void );
include/linux/mm_inline.h +5 -5
@@ -175,7 +175,7 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli int old_gen, int new_gen) { int type = folio_is_file_lru(folio); -int zone =folio_zonenum (folio); +int zone =folio_zone_idx (folio);int delta =folio_nr_pages (folio);enum lru_list lru = type *LRU_INACTIVE_FILE ;struct lru_gen_folio *lrugen = &lruvec->lrugen;
@@ -223,7 +223,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,unsigned long flags;int gen =folio_lru_gen (folio);int type =folio_is_file_lru (folio); -int zone =folio_zonenum (folio); +int zone =folio_zone_idx (folio);struct lru_gen_folio *lrugen = &lruvec->lrugen;VM_WARN_ON_ONCE_FOLIO (gen !=-1 , folio);
@@ -317,7 +317,7 @@ void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio)if (lru_gen_add_folio (lruvec, folio, false))return ; -update_lru_size (lruvec, lru,folio_zonenum (folio), +update_lru_size (lruvec, lru,folio_zone_idx (folio),folio_nr_pages (folio));if (lru !=LRU_UNEVICTABLE )list_add (&folio->lru, &lruvec->lists[lru]);
@@ -337,7 +337,7 @@ void lruvec_add_folio_tail(struct lruvec *lruvec, struct folio *folio)if (lru_gen_add_folio (lruvec, folio, true))return ; -update_lru_size (lruvec, lru,folio_zonenum (folio), +update_lru_size (lruvec, lru,folio_zone_idx (folio),folio_nr_pages (folio));/* This is not expected to be used on LRU_UNEVICTABLE */ list_add_tail (&folio->lru, &lruvec->lists[lru]);
@@ -353,7 +353,7 @@ void lruvec_del_folio(struct lruvec *lruvec, struct folio *folio)if (lru !=LRU_UNEVICTABLE )list_del (&folio->lru); -update_lru_size (lruvec, lru,folio_zonenum (folio), +update_lru_size (lruvec, lru,folio_zone_idx (folio), -folio_nr_pages (folio)); }
include/linux/mmzone.h +66 -46
291 lines not on this page · raw message
include/linux/page-flags-layout.h +12 -0
@@ -12,6 +12,18 @@* are used to select a priority ordered list of memory zones which * match the requested limits. See gfp_zone() in include/linux/gfp.h */ +#if MAX_NR_ZONE_TYPES <2 +#define ZONE_TYPES_SHIFT 0 +#elif MAX_NR_ZONE_TYPES <=2 +#define ZONE_TYPES_SHIFT 1 +#elif MAX_NR_ZONE_TYPES <=4 +#define ZONE_TYPES_SHIFT 2 +#elif MAX_NR_ZONE_TYPES <=8 +#define ZONE_TYPES_SHIFT 3 +#else +#error ZONE_TYPES_SHIFT "Too many zone types configured" +#endif +#if MAX_NR_ZONES <2 #define ZONES_SHIFT 0#elif MAX_NR_ZONES <=2
include/linux/page-flags.h +2 -2
@@ -530,8 +530,8 @@ PAGEFLAG(Readahead, readahead, PF_NO_COMPOUND)* Must use a macro here due to header dependency issues. page_zone() is not * available at this point. */ -#define PageHighMem (__p) is_highmem_idx(page_zonenum(__p)) -#define folio_test_highmem (__f) is_highmem_idx(folio_zonenum(__f)) +#define PageHighMem (__p) is_highmem_type(page_zone_type(__p)) +#define folio_test_highmem (__f) is_highmem_type(folio_zone_type(__f))#else PAGEFLAG_FALSE(HighMem, highmem)#endif
include/linux/vmstat.h +2 -2
@@ -125,8 +125,8 @@ static inline void vm_events_fold_cpu(int cpu)#define count_vm_tlb_events (x, y) do { (void)(y); } while (0)#endif -#define __count_zid_vm_events (item, zid, delta) \ - __count_vm_events(item##_NORMAL - ZONE_NORMAL + zid, delta) +#define __count_zt_vm_events (item, zt, delta) \ + __count_vm_events(item##_NORMAL - ZONE_NORMAL + zt, delta)/* * Zone and node-based page accounting with per cpu differentials.
include/trace/events/oom.h +3 -3
@@ -43,7 +43,7 @@ TRACE_EVENT(reclaim_retry_zone,TP_STRUCT__entry (__field (int ,node ) -__field ( int, zone_idx) +__field ( int, zone_type)__field ( int, order) __field( unsigned long, reclaimable) __field( unsigned long, available)
@@ -54,7 +54,7 @@ TRACE_EVENT(reclaim_retry_zone,TP_fast_assign ( __entry->node = zone_to_nid(zoneref->zone); - __entry->zone_idx = zoneref->zone_idx; + __entry->zone_type = zoneref->zone_type; __entry->order = order; __entry->reclaimable = reclaimable; __entry->available = available;
@@ -64,7 +64,7 @@ TRACE_EVENT(reclaim_retry_zone, ), TP_printk("node=%d zone=%-8s order=%d reclaimable=%lu available=%lu min_wmark=%lu no_progress_loops=%d wmark_check=%d", - __entry->node, __print_symbolic(__entry->zone_idx, ZONE_TYPE), + __entry->node, __print_symbolic(__entry->zone_type, ZONE_TYPE), __entry->order, __entry->reclaimable, __entry->available, __entry->min_wmark, __entry->no_progress_loops,
kernel/bounds.c +1 -0
@@ -17,6 +17,7 @@ int main(void) {/* The enum constants to put into include/generated/bounds.h */ DEFINE (NR_PAGEFLAGS , __NR_PAGEFLAGS); +DEFINE (MAX_NR_ZONE_TYPES , __MAX_NR_ZONE_TYPES);DEFINE (MAX_NR_ZONES , __MAX_NR_ZONES);#ifdef CONFIG_SMP DEFINE (NR_CPUS_BITS ,ilog2 (CONFIG_NR_CPUS ));
lib/show_mem.c +1 -1
@@ -27,7 +27,7 @@ void __show_mem(unsigned int filter, nodemask_t *nodemask, int max_zone_idx) total += zone->present_pages; reserved += zone->present_pages -zone_managed_pages (zone); -if (is_highmem_idx (zoneid)) +if (is_highmem (zone)) highmem += zone->present_pages; } }
mm/compaction.c +29 -29
202 lines not on this page · raw message
mm/internal.h +9 -9
@@ -207,10 +207,10 @@ pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address);* between functions involved in allocations, including the alloc_pages* * family of functions. * -* nodemask, migratetype and highest_zoneidx are initialized only once in +* nodemask, migratetype and highest_zone_type are initialized only once in * __alloc_pages() and then never change. * -* zonelist, preferred_zone and highest_zoneidx are set first in +* zonelist, preferred_zone and highest_zone_type are set first in * __alloc_pages() for the fast path, and might be later changed * in __alloc_pages_slowpath(). All other functions pass the whole structure * by a const pointer.
@@ -222,16 +222,16 @@ struct alloc_context {int migratetype;/* -* highest_zoneidx represents highest usable zone index of +* highest_zone_type represents highest usable zone type of * the allocation request. Due to the nature of the zone, -* memory on lower zone than the highest_zoneidx will be -* protected by lowmem_reserve[highest_zoneidx]. +* memory on lower zone than the highest_zone_type will be +* protected by lowmem_reserve[highest_zone_type]. * -* highest_zoneidx is also used by reclaim/compaction to limit -* the target zone since higher zone than this index cannot be +* highest_zone_type is also used by reclaim/compaction to limit +* the target zone since higher zone than this type cannot be * usable for this allocation request. */ -enum zone_type highest_zoneidx; +enum zone_type highest_zone_type;bool spread_dirty_pages; };
@@ -439,7 +439,7 @@ struct compact_control {int order;/* order a direct compactor needs */ int migratetype;/* migratetype of direct compactor */ const unsigned int alloc_flags;/* alloc flags of a direct compactor */ -const int highest_zoneidx;/* zone index of a direct compactor */ +const enum zone_type highest_zone_type;/* zone type of a direct compactor */ enum migrate_mode mode;/* Async or sync migration mode */ bool ignore_skip_hint;/* Scan blocks even if marked skip */ bool no_set_skip_hint;/* Don't mark blocks for skipping */
mm/memory_hotplug.c +21 -15
@@ -626,7 +626,7 @@ static void node_states_check_changes_online(unsigned long nr_pages,if (!node_state (nid,N_MEMORY )) arg->status_change_nid = nid; -if (zone_idx (zone) <=ZONE_NORMAL && !node_state (nid,N_NORMAL_MEMORY )) +if (zone_type_num (zone) <=ZONE_NORMAL && !node_state (nid,N_NORMAL_MEMORY )) arg->status_change_nid_normal = nid; }
@@ -717,7 +717,7 @@ void __ref move_pfn_range_to_zone(struct zone *zone, unsigned long start_pfn,* expects the zone spans the pfn range. All the pages in the range * are reserved so nobody should be touching them so we should be safe */ -memmap_init_range (nr_pages, nid,zone_idx (zone), start_pfn,0 , +memmap_init_range (nr_pages, nid,zone_type_num (zone),zone_idx (zone), start_pfn,0 ,MEMINIT_HOTPLUG , altmap, migratetype);set_zone_contiguous (zone);
@@ -731,7 +731,7 @@ struct auto_movable_stats {static void auto_movable_stats_account_zone (struct auto_movable_stats *stats,struct zone *zone) { -if (zone_idx (zone) ==ZONE_MOVABLE ) { +if (zone_type_num (zone) ==ZONE_MOVABLE ) { stats->movable_pages += zone->present_pages; }else { stats->kernel_early_pages += zone->present_early_pages;
@@ -837,14 +837,14 @@ static struct zone *default_kernel_zone_for_pfn(int nid, unsigned long start_pfnstruct pglist_data *pgdat =NODE_DATA (nid);int zid; -for (zid =0 ; zid <ZONE_NORMAL ; zid++) { +for (zid =0 ; zid <last_zone_idx (pgdat,ZONE_NORMAL ); zid++) {struct zone *zone = &pgdat->node_zones[zid];if (zone_intersects (zone, start_pfn, nr_pages))return zone; } -return &pgdat->node_zones[ZONE_NORMAL ]; +return &pgdat->node_zones[last_zone_idx (pgdat,ZONE_NORMAL )]; }/*
@@ -904,6 +904,7 @@ static struct zone *auto_movable_zone_for_pfn(int nid, {unsigned long online_pages =0 , max_pages, end_pfn;struct page *page; +pg_data_t *pgdat;if (!auto_movable_ratio) goto kernel_zone;
@@ -952,8 +953,9 @@ static struct zone *auto_movable_zone_for_pfn(int nid, !auto_movable_can_online_movable(nid, group, nr_pages)) goto kernel_zone;#endif /* CONFIG_NUMA */ + pgdat =NODE_DATA (nid); -return &NODE_DATA (nid)->node_zones[ZONE_MOVABLE ]; +return &pgdat->node_zones[last_zone_idx (pgdat,ZONE_MOVABLE )]; kernel_zone:return default_kernel_zone_for_pfn (nid, pfn, nr_pages); }
@@ -963,7 +965,8 @@ static inline struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn {struct zone *kernel_zone =default_kernel_zone_for_pfn (nid, start_pfn, nr_pages); -struct zone *movable_zone = &NODE_DATA (nid)->node_zones[ZONE_MOVABLE ]; +pg_data_t *pgdat =NODE_DATA (nid); +struct zone *movable_zone = &pgdat->node_zones[last_zone_idx (pgdat,ZONE_MOVABLE )];bool in_kernel =zone_intersects (kernel_zone, start_pfn, nr_pages);bool in_movable =zone_intersects (movable_zone, start_pfn, nr_pages);
@@ -986,11 +989,13 @@ struct zone *zone_for_pfn_range(int online_type, int nid, struct memory_group *group, unsigned long start_pfn, unsigned long nr_pages) { + pg_data_t *pgdat = NODE_DATA(nid); +if (online_type ==MMOP_ONLINE_KERNEL )return default_kernel_zone_for_pfn (nid, start_pfn, nr_pages);if (online_type ==MMOP_ONLINE_MOVABLE ) -return &NODE_DATA (nid)->node_zones[ZONE_MOVABLE ]; +return &pgdat->node_zones[last_zone_idx (pgdat,ZONE_MOVABLE )];if (online_policy ==ONLINE_POLICY_AUTO_MOVABLE )return auto_movable_zone_for_pfn (nid, group, start_pfn, nr_pages);
@@ -1006,7 +1011,7 @@ void adjust_present_page_count(struct page *page, struct memory_group *group, long nr_pages) {struct zone *zone =page_zone (page); -const bool movable =zone_idx (zone) ==ZONE_MOVABLE ; +const bool movable =zone_type_num (zone) ==ZONE_MOVABLE ;/* * We only support onlining/offlining/adding/removing of complete
@@ -1734,7 +1739,7 @@ static void node_states_check_changes_offline(unsigned long nr_pages, {struct pglist_data *pgdat = zone->zone_pgdat;unsigned long present_pages =0 ; -enum zone_type zt; +int zid; arg->status_change_nid =NUMA_NO_NODE ; arg->status_change_nid_normal =NUMA_NO_NODE ;
@@ -1747,9 +1752,9 @@ static void node_states_check_changes_offline(unsigned long nr_pages,* thus we can determine that we need to clear the node from * node_states[N_NORMAL_MEMORY]. */ -for (zt =0 ; zt <=ZONE_NORMAL ; zt++) - present_pages += pgdat->node_zones[zt].present_pages; -if (zone_idx (zone) <=ZONE_NORMAL && nr_pages >= present_pages) +for (zid =0 ; zid <=last_zone_idx (pgdat,ZONE_NORMAL ); zid++) + present_pages += pgdat->node_zones[zid].present_pages; +if (zone_type_num (zone) <=ZONE_NORMAL && nr_pages >= present_pages) arg->status_change_nid_normal =zone_to_nid (zone);/*
@@ -1761,7 +1766,8 @@ static void node_states_check_changes_offline(unsigned long nr_pages,* we know that the node will become empty, and so, we can clear * it for N_MEMORY as well. */ - present_pages += pgdat->node_zones[ZONE_MOVABLE ].present_pages; +for (; zid <=last_zone_idx (pgdat,ZONE_MOVABLE ); zid++) + present_pages += pgdat->node_zones[zid].present_pages;if (nr_pages >= present_pages) arg->status_change_nid =zone_to_nid (zone);
@@ -2182,7 +2188,7 @@ static int try_offline_memory_block(struct memory_block *mem, void *arg)* by offlining code ... so we don't care about that. */ page =pfn_to_online_page (section_nr_to_pfn (mem->start_section_nr)); -if (page &&zone_idx (page_zone (page)) ==ZONE_MOVABLE ) +if (page &&zone_type_num (page_zone (page)) ==ZONE_MOVABLE ) online_type =MMOP_ONLINE_MOVABLE ; rc =device_offline (&mem->dev);
mm/mempolicy.c +2 -2
@@ -1923,9 +1923,9 @@ unsigned int mempolicy_slab_node(void)* first node. */ struct zonelist *zonelist; -enum zone_type highest_zoneidx =gfp_zone (GFP_KERNEL ); +enum zone_type highest_zone_type =gfp_zone (GFP_KERNEL ); zonelist = &NODE_DATA (node)->node_zonelists[ZONELIST_FALLBACK ]; - z =first_zones_zonelist (zonelist, highest_zoneidx, + z =first_zones_zonelist (zonelist, highest_zone_type, &policy->nodes);return z->zone ?zone_to_nid (z->zone) : node; }
mm/memremap.c +4 -2
@@ -172,6 +172,7 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params,const bool is_private = pgmap->type ==MEMORY_DEVICE_PRIVATE ;struct range *range = &pgmap->ranges[range_id];struct dev_pagemap *conflict_pgmap; +pg_data_t *pgdat;int error, is_ram; if (WARN_ONCE(pgmap_altmap(pgmap) && range_id > 0,
@@ -247,10 +248,11 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params, params); } + pgdat =NODE_DATA (nid);if (!error) {struct zone *zone; - zone = &NODE_DATA (nid)->node_zones[ZONE_DEVICE ]; + zone = &pgdat->node_zones[last_zone_idx (pgdat,ZONE_DEVICE )];move_pfn_range_to_zone (zone,PHYS_PFN (range->start),PHYS_PFN (range_len (range)), params->altmap,MIGRATE_MOVABLE );
@@ -264,7 +266,7 @@ static int pagemap_range(struct dev_pagemap *pgmap, struct mhp_params *params,* Initialization of the pages has been deferred until now in order * to allow us to do the work while not holding the hotplug lock. */ -memmap_init_zone_device (&NODE_DATA (nid)->node_zones[ZONE_DEVICE ], +memmap_init_zone_device (&pgdat->node_zones[last_zone_idx (pgdat,ZONE_DEVICE )],PHYS_PFN (range->start),PHYS_PFN (range_len (range)), pgmap); if (pgmap->type != MEMORY_DEVICE_PRIVATE &&
mm/migrate.c +3 -3
@@ -2029,7 +2029,7 @@ struct page *alloc_migration_target(struct page *page, unsigned long private)struct folio *hugetlb_folio =NULL ;struct folio *new_folio =NULL ;int nid; -int zidx; +enum zone_type zt; mtc = (struct migration_target_control *)private; gfp_mask = mtc->gfp_mask;
@@ -2055,8 +2055,8 @@ struct page *alloc_migration_target(struct page *page, unsigned long private) gfp_mask |=GFP_TRANSHUGE ; order =folio_order (folio); } - zidx =zone_idx (folio_zone (folio)); -if (is_highmem_idx (zidx) || zidx ==ZONE_MOVABLE ) + zt =zone_type_num (folio_zone (folio)); +if (is_highmem_type (zt) || zt ==ZONE_MOVABLE ) gfp_mask |= __GFP_HIGHMEM; new_folio =__folio_alloc (gfp_mask, order, nid, mtc->nmask);
mm/mmzone.c +4 -4
@@ -52,9 +52,9 @@ static inline int zref_in_nodemask(struct zoneref *zref, nodemask_t *nodes)#endif /* CONFIG_NUMA */ } -/* Returns the next zone at or below highest_zoneidx in a zonelist */ +/* Returns the next zone at or below highest_zone_type in a zonelist */ struct zoneref *__next_zones_zonelist (struct zoneref *z, -enum zone_type highest_zoneidx, +enum zone_type highest_zone_type,nodemask_t *nodes) {/*
@@ -62,10 +62,10 @@ struct zoneref *__next_zones_zonelist(struct zoneref *z,* Only filter based on nodemask if it's set */ if (unlikely (nodes ==NULL )) -while (zonelist_zone_idx (z) > highest_zoneidx) +while (zonelist_zone_type (z) > highest_zone_type) z++;else -while (zonelist_zone_idx (z) > highest_zoneidx || +while (zonelist_zone_type (z) > highest_zone_type || (z->zone && !zref_in_nodemask (z, nodes))) z++;
mm/oom_kill.c +2 -2
@@ -252,7 +252,7 @@ static enum oom_constraint constrained_alloc(struct oom_control *oc) {struct zone *zone;struct zoneref *z; -enum zone_type highest_zoneidx =gfp_zone (oc->gfp_mask); +enum zone_type highest_zone_type =gfp_zone (oc->gfp_mask);bool cpuset_limited = false;int nid;
@@ -292,7 +292,7 @@ static enum oom_constraint constrained_alloc(struct oom_control *oc)/* Check this allocation failure is caused by cpuset's wall function */ for_each_zone_zonelist_nodemask (zone, z, oc->zonelist, - highest_zoneidx, oc->nodemask) + highest_zone_type, oc->nodemask)if (!cpuset_zone_allowed (zone, oc->gfp_mask)) cpuset_limited = true;
mm/page-writeback.c +6 -3
@@ -302,16 +302,19 @@ static unsigned long highmem_dirtyable_memory(unsigned long total)int node;unsigned long x =0 ;int i; +pg_data_t *pgdat;for_each_node_state (node,N_HIGH_MEMORY ) { -for (i =ZONE_NORMAL +1 ; i <MAX_NR_ZONES ; i++) { + pgdat =NODE_DATA (node); +for (i =last_zone_idx (pgdat,ZONE_NORMAL ) +1 ; i <MAX_NR_ZONES ; i++) {struct zone *z;unsigned long nr_pages; -if (!is_highmem_idx (i)) + z = &pgdat->node_zones[i]; + +if (!is_highmem (z))continue ; - z = &NODE_DATA (node)->node_zones[i];if (!populated_zone (z))continue ;
mm/page_alloc.c +99 -91
606 lines not on this page · raw message
mm/page_isolation.c +1 -1
@@ -69,7 +69,7 @@ static struct page *has_unmovable_pages(unsigned long start_pfn, unsigned long e* pages then it should be reasonably safe to assume the rest * is movable. */ -if (zone_idx (zone) ==ZONE_MOVABLE ) +if (zone_type_num (zone) ==ZONE_MOVABLE )continue ;/*
mm/shmem.c +1 -1
@@ -1622,7 +1622,7 @@ static struct folio *shmem_alloc_and_acct_folio(gfp_t gfp, struct inode *inode,*/ static bool shmem_should_replace_folio (struct folio *folio,gfp_t gfp) { -return folio_zonenum (folio) >gfp_zone (gfp); +return folio_zone_type (folio) >gfp_zone (gfp); } static int shmem_replace_folio(struct folio **foliop, gfp_t gfp,
mm/slab.c +2 -2
@@ -3072,7 +3072,7 @@ static void *fallback_alloc(struct kmem_cache *cache, gfp_t flags)struct zonelist *zonelist;struct zoneref *z;struct zone *zone; -enum zone_type highest_zoneidx =gfp_zone (flags); +enum zone_type highest_zone_type =gfp_zone (flags);void *obj =NULL ;struct slab *slab;int nid;
@@ -3090,7 +3090,7 @@ static void *fallback_alloc(struct kmem_cache *cache, gfp_t flags)* Look through allowed nodes for objects available * from existing per node queues. */ -for_each_zone_zonelist (zone, z, zonelist, highest_zoneidx) { +for_each_zone_zonelist (zone, z, zonelist, highest_zone_type) { nid =zone_to_nid (zone); if (cpuset_zone_allowed(zone, flags) &&
mm/slub.c +2 -2
@@ -2318,7 +2318,7 @@ static void *get_any_partial(struct kmem_cache *s, struct partial_context *pc)struct zonelist *zonelist;struct zoneref *z;struct zone *zone; -enum zone_type highest_zoneidx =gfp_zone (pc->flags); +enum zone_type highest_zone_type =gfp_zone (pc->flags);void *object;unsigned int cpuset_mems_cookie;
@@ -2347,7 +2347,7 @@ static void *get_any_partial(struct kmem_cache *s, struct partial_context *pc) do { cpuset_mems_cookie = read_mems_allowed_begin(); zonelist = node_zonelist(mempolicy_slab_node(), pc->flags); - for_each_zone_zonelist(zone, z, zonelist, highest_zoneidx) { + for_each_zone_zonelist(zone, z, zonelist, highest_zone_type) { struct kmem_cache_node *n; n = get_node(s, zone_to_nid(zone));
mm/vmscan.c +107 -98
645 lines not on this page · raw message
Signature · 1 line
2.39.2
RFC 2/6 mm: add struct zone_type_struct to describe zone type
In reply to Huang YingMultiple zone instances may be created for one zone type in the following patch in series. After that, struct zone is used to describe zone instance only. To describe zone type, another data structure (struct zone_type_struct) is defined in this patch. The most important information used during run time is the start and last zone indexes (start/last_zone_idx) of all zone instances of a zone type in a NUMA node. Other fields are used during system initialization to initialize struct zone.
The zone fields in page->flags are used for zone instance index. So, the zone type of a page needs to be gotten via struct zone.
--- include/linux/mm.h | 36 -------- include/linux/mmzone.h | 195 ++++++++++++++++++++++++++++------------- mm/page_alloc.c | 92 ++++++++++++++----- 3 files changed, 203 insertions(+), 120 deletions(-)
include/linux/mm.h +0 -36
@@ -1454,22 +1454,6 @@ static inline int page_zone_id(struct page *page)return (page->flags >>ZONEID_PGSHIFT ) &ZONEID_MASK ; } -#ifdef NODE_NOT_IN_PAGE_FLAGS -extern int page_to_nid (const struct page *page); -#else -static inline int page_to_nid (const struct page *page) -{ -struct page *p = (struct page *)page; - -return (PF_POISONED_CHECK (p)->flags >>NODES_PGSHIFT ) &NODES_MASK ; -} -#endif - -static inline int folio_nid (const struct folio *folio) -{ -return page_to_nid (&folio->page); -} -#ifdef CONFIG_NUMA_BALANCING /* page access time bits needs to hold at least 4 seconds */ #define PAGE_ACCESS_TIME_MIN_BITS 12
@@ -1659,26 +1643,6 @@ static inline void page_kasan_tag_reset(struct page *page) { }#endif /* CONFIG_KASAN_SW_TAGS || CONFIG_KASAN_HW_TAGS */ -static inline struct zone *page_zone (const struct page *page) -{ -return &NODE_DATA (page_to_nid (page))->node_zones[page_zone_idx (page)]; -} - -static inline pg_data_t *page_pgdat (const struct page *page) -{ -return NODE_DATA (page_to_nid (page)); -} - -static inline struct zone *folio_zone (const struct folio *folio) -{ -return page_zone (&folio->page); -} - -static inline pg_data_t *folio_pgdat (const struct folio *folio) -{ -return page_pgdat (&folio->page); -} -#ifdef SECTION_IN_PAGE_FLAGS static inline void set_page_section (struct page *page,unsigned long section) {
include/linux/mmzone.h +132 -63
244 lines not on this page · raw message
mm/page_alloc.c +71 -21
173 lines not on this page · raw message
Signature · 1 line
2.39.2
RFC 3/6 mm: support multiple zone instances per zone type in memory online
In reply to Huang YingBecause there will be multiple movable zone instances, when a memory section is onlined to movable zones, a movable zone instance needs to be chosen similar as that for kernel zones.
The online target zone instance is chosen based on the zone instance range for both movable and kernel zones.
--- mm/memory_hotplug.c | 38 ++++++++++++++++++++++++++++++-------- 1 file changed, 30 insertions(+), 8 deletions(-)
mm/memory_hotplug.c +30 -8
@@ -842,11 +842,37 @@ static struct zone *default_kernel_zone_for_pfn(int nid, unsigned long start_pfnif (zone_intersects (zone, start_pfn, nr_pages))return zone; +if (start_pfn < zone->zone_start_pfn) +return zone; }return &pgdat->node_zones[last_zone_idx (pgdat,ZONE_NORMAL )]; } +/* +* Returns a default movable memory zone for the given pfn range. +*/ +static struct zone *default_movable_zone_for_pfn (int nid,unsigned long start_pfn, +unsigned long nr_pages) +{ +struct pglist_data *pgdat =NODE_DATA (nid); +struct zone *zone; +int zid; + +for (zid =start_zone_idx (pgdat,ZONE_MOVABLE ); + zid <last_zone_idx (pgdat,ZONE_MOVABLE ); + zid++) { + zone = &pgdat->node_zones[zid]; + +if (zone_intersects (zone, start_pfn, nr_pages)) +return zone; +if (start_pfn < zone->zone_start_pfn) +return zone; + } + +return &pgdat->node_zones[last_zone_idx (pgdat,ZONE_MOVABLE )]; +} +/* * Determine to which zone to online memory dynamically based on user * configuration and system stats. We care about the following ratio:
@@ -904,7 +930,6 @@ static struct zone *auto_movable_zone_for_pfn(int nid, {unsigned long online_pages =0 , max_pages, end_pfn;struct page *page; -pg_data_t *pgdat;if (!auto_movable_ratio) goto kernel_zone;
@@ -953,9 +978,8 @@ static struct zone *auto_movable_zone_for_pfn(int nid, !auto_movable_can_online_movable(nid, group, nr_pages)) goto kernel_zone;#endif /* CONFIG_NUMA */ - pgdat =NODE_DATA (nid); -return &pgdat->node_zones[last_zone_idx (pgdat,ZONE_MOVABLE )]; +return default_movable_zone_for_pfn (nid, pfn, nr_pages); kernel_zone:return default_kernel_zone_for_pfn (nid, pfn, nr_pages); }
@@ -965,8 +989,8 @@ static inline struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn {struct zone *kernel_zone =default_kernel_zone_for_pfn (nid, start_pfn, nr_pages); -pg_data_t *pgdat =NODE_DATA (nid); -struct zone *movable_zone = &pgdat->node_zones[last_zone_idx (pgdat,ZONE_MOVABLE )]; +struct zone *movable_zone =default_movable_zone_for_pfn (nid, start_pfn, + nr_pages);bool in_kernel =zone_intersects (kernel_zone, start_pfn, nr_pages);bool in_movable =zone_intersects (movable_zone, start_pfn, nr_pages);
@@ -989,13 +1013,11 @@ struct zone *zone_for_pfn_range(int online_type, int nid, struct memory_group *group, unsigned long start_pfn, unsigned long nr_pages) { - pg_data_t *pgdat = NODE_DATA(nid); - if (online_type == MMOP_ONLINE_KERNEL) return default_kernel_zone_for_pfn(nid, start_pfn, nr_pages);if (online_type ==MMOP_ONLINE_MOVABLE ) -return &pgdat->node_zones[last_zone_idx (pgdat,ZONE_MOVABLE )]; +return default_movable_zone_for_pfn (nid, start_pfn, nr_pages);if (online_policy ==ONLINE_POLICY_AUTO_MOVABLE )return auto_movable_zone_for_pfn (nid, group, start_pfn, nr_pages);
Signature · 1 line
2.39.2
RFC 4/6 mm: avoid show invalid zone in /proc/zoneinfo
In reply to Huang YingIn the following patch of the series, one or multiple zone instances may be created for one zone type. So the total number of zone instances of a node becomes dynamic based on system configurations. So, the real zone instance number instead of static max zone number is used to show zone information in /proc/zoneinfo.
--- include/linux/mmzone.h | 5 +++++ mm/vmstat.c | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-)
include/linux/mmzone.h +5 -0
@@ -1809,6 +1809,11 @@ static inline int last_zone_idx(pg_data_t *pgdat, enum zone_type zt)return pgdat->node_zone_types[zt].last_zone_idx; } +static inline int last_zone_idx_pgdat (pg_data_t *pgdat) +{ +return last_zone_idx (pgdat,MAX_NR_ZONE_TYPES -1 ); +} +#ifdef CONFIG_SPARSEMEM #include <asm/sparsemem.h> #endif
mm/vmstat.c +2 -1
@@ -1441,8 +1441,9 @@ static void walk_zones_in_node(struct seq_file *m, pg_data_t *pgdat,struct zone *zone;struct zone *node_zones = pgdat->node_zones;unsigned long flags; +int last_zone_idx =last_zone_idx_pgdat (pgdat); -for (zone = node_zones; zone - node_zones <MAX_NR_ZONES ; ++zone) { +for (zone = node_zones; zone - node_zones <= last_zone_idx; ++zone) {if (assert_populated && !populated_zone (zone))continue ;
Signature · 1 line
2.39.2
RFC 5/6 mm: create multiple zone instances for one zone type based on memory size
In reply to Huang YingMore and more cores are put in one physical CPU (usually one NUMA node too). In 2023, one high-end server CPU has 56, 64, or more cores. Even more cores per physical CPU are planned for future CPUs. While all cores in one physical CPU will contend for the page allocation on one zone in most cases. This causes heavy zone lock contention in some workloads. And the situation will become worse and worse in the future.
For example, on an 2-socket Intel server machine with 224 logical CPUs, if the kernel is built with make -j224, the zone lock contention cycles% can reach up to about 12.7%.
To improve the scalability of the zone lock contention, in this patch, we will create one zone instance for each about 256 GB memory of a zone type generally. In next patch of the series, different logical CPUs will prefer different zone instances based on the logical CPU No. So the total number of logical CPUs contend on one zone will be reduced. Thus the scalability is improved.
Combined with the next patch in the series ("mm: prefer different zone list on different logical CPU"), the zone lock contention cycles% reduces to less than 1.6% in the above kbuild test case when 4 zone instances are created for ZONE_NORMAL.
Also tested with the will-it-scale/page_fault1 with 16 processes. With the optimization, the benchmark score increases up to 18.2% and the zone lock contention reduces from 13.01% to 0.56%.
To create multiple zone instances for a zone type, another choice is to create zone instances based on the total number of logical CPUs. We choose to use memory size because it is easier to be implemented. In most cases, the more the cores, the larger the memory size is. And, on system with larger memory size, the performance requirement of the page allocator is usually higher.
--- include/linux/mmzone.h | 13 +++- include/linux/page-flags-layout.h | 2 + mm/page_alloc.c | 104 ++++++++++++++++++++++++++---- 3 files changed, 107 insertions(+), 12 deletions(-)
include/linux/mmzone.h +12 -1
@@ -800,7 +800,12 @@ enum zone_type { }; -#define __MAX_NR_ZONES __MAX_NR_ZONE_TYPES +#ifdef CONFIG_64BIT +#define __MAX_NR_SPLIT_ZONES 4 +#else +#define __MAX_NR_SPLIT_ZONES 0 +#endif +#define __MAX_NR_ZONES (__MAX_NR_ZONE_TYPES + __MAX_NR_SPLIT_ZONES)#ifndef __GENERATING_BOUNDS_H
@@ -1106,6 +1111,12 @@ static inline bool zone_intersects(struct zone *zone,return true; } +#ifdef CONFIG_64BIT +#define MAX_NR_ZONES_PER_TYPE 4 +#else +#define MAX_NR_ZONES_PER_TYPE 1 +#endif +struct zone_type_struct {int start_zone_idx;int last_zone_idx;
include/linux/page-flags-layout.h +2 -0
@@ -32,6 +32,8 @@#define ZONES_SHIFT 2#elif MAX_NR_ZONES <=8 #define ZONES_SHIFT 3 +#elif MAX_NR_ZONES <=16 +#define ZONES_SHIFT 4#else #error ZONES_SHIFT "Too many zones configured"#endif
mm/page_alloc.c +93 -11
117 lines not on this page · raw message
Signature · 1 line
2.39.2
RFC 6/6 mm: prefer different zone list on different logical CPU
In reply to Huang YingOriginally, there are only one fallback and one no-fallback zone list for each NUMA node (pglist_data->node_zonelists). That is, all logical CPUs of a NUMA node will use one zone list during the page allocation. This isn't a problem before, because there's at most one instance for each zone type.
Now, we may create multiple zone instances for one zone type. This makes it possible for the different logical CPUs to prefer different zone instance of one zone type to improve the zone lock scalability. So, in this patch, multiple fallback and multiple no-fallback zone lists can be created for each NUMA node based on the max zone instances number for one zone type of the NUMA node. Then different logical CPUs will prefer different zone list based on logical CPU number.
Combined with the previous patches in the series, this can improve the scalability of zone lock contention effectively in the kbuild test case. Details can be found in the description of the previous patch of the series.
--- include/linux/gfp.h | 6 ++- include/linux/mmzone.h | 5 ++- mm/mempolicy.c | 6 ++- mm/mm_init.c | 2 +- mm/page_alloc.c | 93 ++++++++++++++++++++++++++++-------------- 5 files changed, 77 insertions(+), 35 deletions(-)
include/linux/gfp.h +5 -1
@@ -164,7 +164,11 @@ static inline int gfp_zonelist(gfp_t flags)*/ static inline struct zonelist *node_zonelist (int nid,gfp_t flags) { -return NODE_DATA (nid)->node_zonelists +gfp_zonelist (flags); +pg_data_t *pgdat =NODE_DATA (nid); +int li; + + li =raw_smp_processor_id () % pgdat->max_nr_zones_per_type; +return pgdat->node_zonelists[li] +gfp_zonelist (flags); }#ifndef HAVE_ARCH_FREE_PAGE
include/linux/mmzone.h +3 -2
@@ -1240,10 +1240,11 @@ typedef struct pglist_data {* Generally the first zones will be references to this node's * node_zones. */ -struct zonelist node_zonelists[MAX_ZONELISTS ]; +struct zonelist node_zonelists[MAX_NR_ZONES_PER_TYPE ][MAX_ZONELISTS ];int nr_zones;/* number of populated zones in this node */ int nr_zone_types; +int max_nr_zones_per_type;#ifdef CONFIG_FLATMEM /* means !SPARSEMEM */ struct page *node_mem_map;#ifdef CONFIG_PAGE_EXTENSION
@@ -1699,7 +1700,7 @@ static inline bool movable_only_nodes(nodemask_t *nodes)* at least one zone that can satisfy kernel allocations. */ nid =first_node (*nodes); - zonelist = &NODE_DATA (nid)->node_zonelists[ZONELIST_FALLBACK ]; + zonelist = &NODE_DATA (nid)->node_zonelists[0 ][ZONELIST_FALLBACK ]; z =first_zones_zonelist (zonelist,ZONE_NORMAL , nodes);return (!z->zone) ? true : false; }
mm/mempolicy.c +5 -1
@@ -1924,7 +1924,11 @@ unsigned int mempolicy_slab_node(void)*/ struct zonelist *zonelist;enum zone_type highest_zone_type =gfp_zone (GFP_KERNEL ); - zonelist = &NODE_DATA (node)->node_zonelists[ZONELIST_FALLBACK ]; +pg_data_t *pgdat =NODE_DATA (node); +int li; + + li =raw_smp_processor_id () % pgdat->max_nr_zones_per_type; + zonelist = &pgdat->node_zonelists[li][ZONELIST_FALLBACK ]; z =first_zones_zonelist (zonelist, highest_zone_type, &policy->nodes);return z->zone ?zone_to_nid (z->zone) : node;
mm/mm_init.c +1 -1
@@ -40,7 +40,7 @@ void __init mminit_verify_zonelist(void)/* Identify the zone and nodelist */ zoneid = i %MAX_NR_ZONES ; listid = i /MAX_NR_ZONES ; - zonelist = &pgdat->node_zonelists[listid]; + zonelist = &pgdat->node_zonelists[0 ][listid]; zone = &pgdat->node_zones[zoneid];if (!populated_zone (zone))continue ;
mm/page_alloc.c +63 -30
170 lines not on this page · raw message
Signature · 1 line
2.39.2
Quoted 37 lines · On Thu, 11 May 2023 14:56:01 +0800 Huang Ying <[email protected]> wrote:
The patchset is based on upstream v6.3.
More and more cores are put in one physical CPU (usually one NUMA node too). In 2023, one high-end server CPU has 56, 64, or more cores. Even more cores per physical CPU are planned for future CPUs. While all cores in one physical CPU will contend for the page allocation on one zone in most cases. This causes heavy zone lock contention in some workloads. And the situation will become worse and worse in the future.
For example, on an 2-socket Intel server machine with 224 logical CPUs, if the kernel is built with
make -j224, the zone lock contention cycles% can reach up to about 12.7%.To improve the scalability of the page allocation, in this series, we will create one zone instance for each about 256 GB memory of a zone type generally. That is, one large zone type will be split into multiple zone instances. Then, different logical CPUs will prefer different zone instances based on the logical CPU No. So the total number of logical CPUs contend on one zone will be reduced. Thus the scalability is improved.
With the series, the zone lock contention cycles% reduces to less than 1.6% in the above kbuild test case when 4 zone instances are created for
ZONE_NORMAL.Also tested the series with the will-it-scale/page_fault1 with 16 processes. With the optimization, the benchmark score increases up to 18.2% and the zone lock contention reduces from 13.01% to 0.56%.
To create multiple zone instances for a zone type, another choice is to create zone instances based on the total number of logical CPUs. We choose to use memory size because it is easier to be implemented. In most cases, the more the cores, the larger the memory size is. And, on system with larger memory size, the performance requirement of the page allocator is usually higher.
Quoted 3 lines
Best Regards,
Huang, Ying
Hi,
Interesting idea. I'm curious though on whether this can suffer from imbalance problems where due to uneven allocations from particular CPUs you can end up with all page faults happening in one zone and the original contention problem coming back? Or am I missing some process that will result in that imbalance being corrected?
Jonathan
Quoted 9 lines · Jonathan Cameron
Hi,
Interesting idea. I'm curious though on whether this can suffer from imbalance problems where due to uneven allocations from particular CPUs you can end up with all page faults happening in one zone and the original contention problem coming back? Or am I missing some process that will result in that imbalance being corrected?
Jonathan
Well, the first line of defense is the per cpu page lists...
it can well be that a couple of cpus all in the same zone hit some high frequency pattern... that by itself isn't the real issue. Note the "a couple". It gets to be a problem if "a high number" start hitting this...
And by splitting the total into smaller pieces, this is going to be much much less likely, since the total number per zone is just less.
Quoted 4 lines · Huang Ying
To improve the scalability of the page allocation, in this series, we will create one zone instance for each about 256 GB memory of a zone type generally. That is, one large zone type will be split into multiple zone instances.
A few anecdotes for why I think _some_ people will like this:
Some Intel hardware has a "RAM" caching mechanism. It either caches DRAM in High-Bandwidth Memory or Persistent Memory in DRAM. This cache is direct-mapped and can have lots of collisions. One way to prevent collisions is to chop up the physical memory into cache-sized zones and let users choose to allocate from one zone. That fixes the conflicts.
Some other Intel hardware a ways to chop a NUMA node representing a single socket into slices. Usually one slice gets a memory controller and its closest cores. Intel calls these approaches Cluster on Die or Sub-NUMA Clustering and users can select it from the BIOS.
In both of these cases, users have reported scalability improvements. We've gone as far as to suggest the socket-splitting options to folks today who are hitting zone scalability issues on that hardware.
That said, those _same_ users sometimes come back and say something along the lines of: "So... we've got this app that allocates a big hunk of memory. It's going slower than before." They're filling up one of the chopped-up zones, hitting _some_ kind of undesirable reclaim behavior and they want their humpty-dumpty zones put back together again ... without hurting scalability. Some people will never be happy. :)
Anyway, _if_ you do this, you might also consider being able to dynamically adjust a CPU's zonelists somehow. That would relieve pressure on one zone for those uneven allocations. That wasn't an option in the two cases above because users had ulterior motives for sticking inside a single zone. But, in your case, the zones really do have equivalent performance.
Hi, Dave,
Quoted 23 lines · Dave Hansen
Quoted 4 lines · Huang Ying
To improve the scalability of the page allocation, in this series, we will create one zone instance for each about 256 GB memory of a zone type generally. That is, one large zone type will be split into multiple zone instances.
A few anecdotes for why I think _some_ people will like this:
Some Intel hardware has a "RAM" caching mechanism. It either caches DRAM in High-Bandwidth Memory or Persistent Memory in DRAM. This cache is direct-mapped and can have lots of collisions. One way to prevent collisions is to chop up the physical memory into cache-sized zones and let users choose to allocate from one zone. That fixes the conflicts.
Some other Intel hardware a ways to chop a NUMA node representing a single socket into slices. Usually one slice gets a memory controller and its closest cores. Intel calls these approaches Cluster on Die or Sub-NUMA Clustering and users can select it from the BIOS.
In both of these cases, users have reported scalability improvements. We've gone as far as to suggest the socket-splitting options to folks today who are hitting zone scalability issues on that hardware.
Quoted 6 lines
That said, those _same_ users sometimes come back and say something along the lines of: "So... we've got this app that allocates a big hunk of memory. It's going slower than before." They're filling up one of the chopped-up zones, hitting _some_ kind of undesirable reclaim behavior and they want their humpty-dumpty zones put back together again ... without hurting scalability. Some people will never be happy. :)
Thanks a lot for your valuable input!
Quoted 6 lines
Anyway, _if_ you do this, you might also consider being able to dynamically adjust a CPU's zonelists somehow. That would relieve pressure on one zone for those uneven allocations. That wasn't an option in the two cases above because users had ulterior motives for sticking inside a single zone. But, in your case, the zones really do have equivalent performance.
Yes. For the requirements you mentioned above, we need a mechanism to adjust a CPU's zonelists dynamically. I will not implement that in this series. But I think that it's doable based on the multiple zone instances per zone type implementation in this series.
Best Regards,
Huang, Ying
Michal Hocko
maintainerQuoted 14 lines · Huang Ying
The patchset is based on upstream v6.3.
More and more cores are put in one physical CPU (usually one NUMA node too). In 2023, one high-end server CPU has 56, 64, or more cores. Even more cores per physical CPU are planned for future CPUs. While all cores in one physical CPU will contend for the page allocation on one zone in most cases. This causes heavy zone lock contention in some workloads. And the situation will become worse and worse in the future.
For example, on an 2-socket Intel server machine with 224 logical CPUs, if the kernel is built with
make -j224, the zone lock contention cycles% can reach up to about 12.7%.
Quoted 7 lines
To improve the scalability of the page allocation, in this series, we will create one zone instance for each about 256 GB memory of a zone type generally. That is, one large zone type will be split into multiple zone instances. Then, different logical CPUs will prefer different zone instances based on the logical CPU No. So the total number of logical CPUs contend on one zone will be reduced. Thus the scalability is improved.
It is not really clear to me why you need a new zone for all this rather than partition free lists internally within the zone? Essentially to increase the current two level system to 3: per cpu caches, per cpu arenas and global fallback.
I am also missing some information why pcp caches tunning is not sufficient.
Signature · 2 lines
Michal Hocko SUSE Labs
Hi, Michal,
Thanks for comments!
Quoted 23 lines · Michal Hocko
Quoted 14 lines · Huang Ying
The patchset is based on upstream v6.3.
More and more cores are put in one physical CPU (usually one NUMA node too). In 2023, one high-end server CPU has 56, 64, or more cores. Even more cores per physical CPU are planned for future CPUs. While all cores in one physical CPU will contend for the page allocation on one zone in most cases. This causes heavy zone lock contention in some workloads. And the situation will become worse and worse in the future.
For example, on an 2-socket Intel server machine with 224 logical CPUs, if the kernel is built with
make -j224, the zone lock contention cycles% can reach up to about 12.7%.Quoted 7 lines
To improve the scalability of the page allocation, in this series, we will create one zone instance for each about 256 GB memory of a zone type generally. That is, one large zone type will be split into multiple zone instances. Then, different logical CPUs will prefer different zone instances based on the logical CPU No. So the total number of logical CPUs contend on one zone will be reduced. Thus the scalability is improved.
Quoted 4 lines
It is not really clear to me why you need a new zone for all this rather than partition free lists internally within the zone? Essentially to increase the current two level system to 3: per cpu caches, per cpu arenas and global fallback.
Sorry, I didn't get your idea here. What is per cpu arenas? What's the difference between it and per cpu caches (PCP)?
Quoted 2 lines
I am also missing some information why pcp caches tunning is not sufficient.
PCP does improve the page allocation scalability greatly! But it doesn't help much for workloads that allocating pages on one CPU and free them in different CPUs. PCP tuning can improve the page allocation scalability for a workload greatly. But it's not trivial to find the best tuning parameters for various workloads and workload run time statuses (workloads may have different loads and memory requirements at different time). And we may run different workloads on different logical CPUs of the system. This also makes it hard to find the best PCP tuning globally. It would be better to find a solution to improve the page allocation scalability out of box or automatically. Do you agree?
Best Regards,
Huang, Ying
Michal Hocko
maintainerQuoted 29 lines · Ying
Hi, Michal,
Thanks for comments!
Quoted 23 lines · Michal Hocko
Quoted 21 lines · Huang Ying
The patchset is based on upstream v6.3.
More and more cores are put in one physical CPU (usually one NUMA node too). In 2023, one high-end server CPU has 56, 64, or more cores. Even more cores per physical CPU are planned for future CPUs. While all cores in one physical CPU will contend for the page allocation on one zone in most cases. This causes heavy zone lock contention in some workloads. And the situation will become worse and worse in the future.
For example, on an 2-socket Intel server machine with 224 logical CPUs, if the kernel is built with
make -j224, the zone lock contention cycles% can reach up to about 12.7%.To improve the scalability of the page allocation, in this series, we will create one zone instance for each about 256 GB memory of a zone type generally. That is, one large zone type will be split into multiple zone instances. Then, different logical CPUs will prefer different zone instances based on the logical CPU No. So the total number of logical CPUs contend on one zone will be reduced. Thus the scalability is improved.
Quoted 7 lines
Quoted 4 lines
It is not really clear to me why you need a new zone for all this rather than partition free lists internally within the zone? Essentially to increase the current two level system to 3: per cpu caches, per cpu arenas and global fallback.
Sorry, I didn't get your idea here. What is per cpu arenas? What's the difference between it and per cpu caches (PCP)?
Sorry, I didn't give this much thought than the above. Essentially, we have 2 level system right now. Pcp caches should reduce the contention on the per cpu level and that should work reasonably well, if you manage to align batch sizes to the workload AFAIK. If this is not sufficient then why to add the full zone rather than to add another level that caches across a larger than a cpu unit. Maybe a core?
This might be a wrong way around going for this but there is not much performance analysis about the source of the lock contention so I am mostly guessing.
Quoted 12 lines
Quoted 2 lines
I am also missing some information why pcp caches tunning is not sufficient.
PCP does improve the page allocation scalability greatly! But it doesn't help much for workloads that allocating pages on one CPU and free them in different CPUs. PCP tuning can improve the page allocation scalability for a workload greatly. But it's not trivial to find the best tuning parameters for various workloads and workload run time statuses (workloads may have different loads and memory requirements at different time). And we may run different workloads on different logical CPUs of the system. This also makes it hard to find the best PCP tuning globally.
Yes this makes sense. Does that mean that the global pcp tuning is not keeping up and we need to be able to do more auto-tuning on local bases rather than global?
Quoted 3 lines
It would be better to find a solution to improve
the page allocation scalability out of box or automatically. Do you agree?
Yes.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 45 lines · Michal Hocko
Quoted 29 lines · Ying
Hi, Michal,
Thanks for comments!
Quoted 23 lines · Michal Hocko
Quoted 21 lines · Huang Ying
The patchset is based on upstream v6.3.
More and more cores are put in one physical CPU (usually one NUMA node too). In 2023, one high-end server CPU has 56, 64, or more cores. Even more cores per physical CPU are planned for future CPUs. While all cores in one physical CPU will contend for the page allocation on one zone in most cases. This causes heavy zone lock contention in some workloads. And the situation will become worse and worse in the future.
For example, on an 2-socket Intel server machine with 224 logical CPUs, if the kernel is built with
make -j224, the zone lock contention cycles% can reach up to about 12.7%.To improve the scalability of the page allocation, in this series, we will create one zone instance for each about 256 GB memory of a zone type generally. That is, one large zone type will be split into multiple zone instances. Then, different logical CPUs will prefer different zone instances based on the logical CPU No. So the total number of logical CPUs contend on one zone will be reduced. Thus the scalability is improved.
Quoted 7 lines
Quoted 4 lines
It is not really clear to me why you need a new zone for all this rather than partition free lists internally within the zone? Essentially to increase the current two level system to 3: per cpu caches, per cpu arenas and global fallback.
Sorry, I didn't get your idea here. What is per cpu arenas? What's the difference between it and per cpu caches (PCP)?
Sorry, I didn't give this much thought than the above. Essentially, we have 2 level system right now. Pcp caches should reduce the contention on the per cpu level and that should work reasonably well, if you manage to align batch sizes to the workload AFAIK. If this is not sufficient then why to add the full zone rather than to add another level that caches across a larger than a cpu unit. Maybe a core?
Quoted 3 lines
This might be a wrong way around going for this but there is not much performance analysis about the source of the lock contention so I am mostly guessing.
I guess that the page allocation scalability will be improved if we put more pages in the per CPU caches, or add another level of cache for multiple logical CPUs. Because more page allocation requirements can be satisfied without acquiring zone lock.
As other caching system, there are always cases that the caches are drained and too many requirements goes to underlying slow layer (zone here). For example, if a workload needs to allocate a huge number of pages (larger than cache size) in parallel, it will run into zone lock contention finally. The situation will became worse and worse if we share one zone with more and more logical CPUs. Which is the trend in industry now. Per my understanding, we can observe the high zone lock contention cycles in kbuild test because of that.
So, per my understanding, to improve the page allocation scalability in bad situations (that is, caching doesn't work well enough), we need to restrict the number of logical CPUs that share one zone. This series is an attempt for that. Better caching can increase the good situations and reduce the bad situations. But it seems hard to eliminate all bad situations.
From another perspective, we don't install more and more memory for each logical CPU. This makes it hard to enlarge the default per-CPU cache size.
Quoted 13 lines
Quoted 12 lines
Quoted 2 lines
I am also missing some information why pcp caches tunning is not sufficient.
PCP does improve the page allocation scalability greatly! But it doesn't help much for workloads that allocating pages on one CPU and free them in different CPUs. PCP tuning can improve the page allocation scalability for a workload greatly. But it's not trivial to find the best tuning parameters for various workloads and workload run time statuses (workloads may have different loads and memory requirements at different time). And we may run different workloads on different logical CPUs of the system. This also makes it hard to find the best PCP tuning globally.
Quoted 3 lines
Yes this makes sense. Does that mean that the global pcp tuning is not keeping up and we need to be able to do more auto-tuning on local bases rather than global?
Similar as above, I think that PCP helps the good situations performance greatly, and splitting zone can help the bad situations scalability. They are working at the different levels.
As for PCP auto-tuning, I think that it's hard to implement it to resolve all problems (that is, makes PCP never be drained).
And auto-tuning doesn't sound easy. Do you have some idea of how to do that?
Quoted 5 lines
Quoted 3 lines
It would be better to find a solution to improve
the page allocation scalability out of box or automatically. Do you agree?Yes.
Best Regards,
Huang, Ying
Quoted 97 lines · Ying
Quoted 45 lines · Michal Hocko
Quoted 29 lines · Ying
Hi, Michal,
Thanks for comments!
Quoted 23 lines · Michal Hocko
Quoted 21 lines · Huang Ying
The patchset is based on upstream v6.3.
More and more cores are put in one physical CPU (usually one NUMA node too). In 2023, one high-end server CPU has 56, 64, or more cores. Even more cores per physical CPU are planned for future CPUs. While all cores in one physical CPU will contend for the page allocation on one zone in most cases. This causes heavy zone lock contention in some workloads. And the situation will become worse and worse in the future.
For example, on an 2-socket Intel server machine with 224 logical CPUs, if the kernel is built with
make -j224, the zone lock contention cycles% can reach up to about 12.7%.To improve the scalability of the page allocation, in this series, we will create one zone instance for each about 256 GB memory of a zone type generally. That is, one large zone type will be split into multiple zone instances. Then, different logical CPUs will prefer different zone instances based on the logical CPU No. So the total number of logical CPUs contend on one zone will be reduced. Thus the scalability is improved.
Quoted 7 lines
Quoted 4 lines
It is not really clear to me why you need a new zone for all this rather than partition free lists internally within the zone? Essentially to increase the current two level system to 3: per cpu caches, per cpu arenas and global fallback.
Sorry, I didn't get your idea here. What is per cpu arenas? What's the difference between it and per cpu caches (PCP)?
Sorry, I didn't give this much thought than the above. Essentially, we have 2 level system right now. Pcp caches should reduce the contention on the per cpu level and that should work reasonably well, if you manage to align batch sizes to the workload AFAIK. If this is not sufficient then why to add the full zone rather than to add another level that caches across a larger than a cpu unit. Maybe a core?
Quoted 3 lines
This might be a wrong way around going for this but there is not much performance analysis about the source of the lock contention so I am mostly guessing.
I guess that the page allocation scalability will be improved if we put more pages in the per CPU caches, or add another level of cache for multiple logical CPUs. Because more page allocation requirements can be satisfied without acquiring zone lock.
As other caching system, there are always cases that the caches are drained and too many requirements goes to underlying slow layer (zone here). For example, if a workload needs to allocate a huge number of pages (larger than cache size) in parallel, it will run into zone lock contention finally. The situation will became worse and worse if we share one zone with more and more logical CPUs. Which is the trend in industry now. Per my understanding, we can observe the high zone lock contention cycles in kbuild test because of that.
So, per my understanding, to improve the page allocation scalability in bad situations (that is, caching doesn't work well enough), we need to restrict the number of logical CPUs that share one zone. This series is an attempt for that. Better caching can increase the good situations and reduce the bad situations. But it seems hard to eliminate all bad situations.
From another perspective, we don't install more and more memory for each
logical CPU. This makes it hard to enlarge the default per-CPU cache size.Quoted 13 lines
Quoted 12 lines
Quoted 2 lines
I am also missing some information why pcp caches tunning is not sufficient.
PCP does improve the page allocation scalability greatly! But it doesn't help much for workloads that allocating pages on one CPU and free them in different CPUs. PCP tuning can improve the page allocation scalability for a workload greatly. But it's not trivial to find the best tuning parameters for various workloads and workload run time statuses (workloads may have different loads and memory requirements at different time). And we may run different workloads on different logical CPUs of the system. This also makes it hard to find the best PCP tuning globally.
Quoted 3 lines
Yes this makes sense. Does that mean that the global pcp tuning is not keeping up and we need to be able to do more auto-tuning on local bases rather than global?
Similar as above, I think that PCP helps the good situations performance greatly, and splitting zone can help the bad situations scalability. They are working at the different levels.
Quoted 5 lines
As for PCP auto-tuning, I think that it's hard to implement it to resolve all problems (that is, makes PCP never be drained).
And auto-tuning doesn't sound easy. Do you have some idea of how to do that?
If we could avoid instantiating more zones and rather improve existing mechanisms (PCP), that would be much more preferred IMHO. I'm sure it's not easy, but that shouldn't stop us from trying ;)
I did not look into the details of this proposal, but seeing the change in include/linux/page-flags-layout.h scares me. Further, I'm not so sure how that change really interacts with hot(un)plug of memory ... on a quick glimpse I feel like this series hacks the code such that such that the split works based on the boot memory size ...
I agree with Michal that looking into auto-tuning PCP would be preferred. If that can't be done, adding another layer might end up cleaner and eventually cover more use cases.
[I recall there was once a proposal to add a 3rd layer to limit fragmenation to individual memory blocks; but the granularity was rather small and there were also some concerns that I don't recall anymore]
Signature · 3 lines
Thanks, David / dhildenb
Quoted 102 lines · David Hildenbrand
Quoted 91 lines · Ying
Quoted 45 lines · Michal Hocko
Quoted 29 lines · Ying
Hi, Michal,
Thanks for comments!
Quoted 23 lines · Michal Hocko
Quoted 21 lines · Huang Ying
The patchset is based on upstream v6.3.
More and more cores are put in one physical CPU (usually one NUMA node too). In 2023, one high-end server CPU has 56, 64, or more cores. Even more cores per physical CPU are planned for future CPUs. While all cores in one physical CPU will contend for the page allocation on one zone in most cases. This causes heavy zone lock contention in some workloads. And the situation will become worse and worse in the future.
For example, on an 2-socket Intel server machine with 224 logical CPUs, if the kernel is built with
make -j224, the zone lock contention cycles% can reach up to about 12.7%.To improve the scalability of the page allocation, in this series, we will create one zone instance for each about 256 GB memory of a zone type generally. That is, one large zone type will be split into multiple zone instances. Then, different logical CPUs will prefer different zone instances based on the logical CPU No. So the total number of logical CPUs contend on one zone will be reduced. Thus the scalability is improved.
Quoted 7 lines
Quoted 4 lines
It is not really clear to me why you need a new zone for all this rather than partition free lists internally within the zone? Essentially to increase the current two level system to 3: per cpu caches, per cpu arenas and global fallback.
Sorry, I didn't get your idea here. What is per cpu arenas? What's the difference between it and per cpu caches (PCP)?
Sorry, I didn't give this much thought than the above. Essentially, we have 2 level system right now. Pcp caches should reduce the contention on the per cpu level and that should work reasonably well, if you manage to align batch sizes to the workload AFAIK. If this is not sufficient then why to add the full zone rather than to add another level that caches across a larger than a cpu unit. Maybe a core?
Quoted 3 lines
This might be a wrong way around going for this but there is not much performance analysis about the source of the lock contention so I am mostly guessing.
I guess that the page allocation scalability will be improved if we put
more pages in the per CPU caches, or add another level of cache for multiple logical CPUs. Because more page allocation requirements can be satisfied without acquiring zone lock.
As other caching system, there are always cases that the caches are drained and too many requirements goes to underlying slow layer (zone here). For example, if a workload needs to allocate a huge number of pages (larger than cache size) in parallel, it will run into zone lock contention finally. The situation will became worse and worse if we share one zone with more and more logical CPUs. Which is the trend in industry now. Per my understanding, we can observe the high zone lock contention cycles in kbuild test because of that.
So, per my understanding, to improve the page allocation scalability in
bad situations (that is, caching doesn't work well enough), we need to restrict the number of logical CPUs that share one zone. This series is an attempt for that. Better caching can increase the good situations and reduce the bad situations. But it seems hard to eliminate all bad situations.
From another perspective, we don't install more and more memory for
each
logical CPU. This makes it hard to enlarge the default per-CPU cache size.Quoted 13 lines
Quoted 12 lines
Quoted 2 lines
I am also missing some information why pcp caches tunning is not sufficient.
PCP does improve the page allocation scalability greatly! But it doesn't help much for workloads that allocating pages on one CPU and free them in different CPUs. PCP tuning can improve the page allocation scalability for a workload greatly. But it's not trivial to find the best tuning parameters for various workloads and workload run time statuses (workloads may have different loads and memory requirements at different time). And we may run different workloads on different logical CPUs of the system. This also makes it hard to find the best PCP tuning globally.
Quoted 3 lines
Yes this makes sense. Does that mean that the global pcp tuning is not keeping up and we need to be able to do more auto-tuning on local bases rather than global?
Quoted 9 lines
Similar as above, I think that PCP helps the good situations performance
greatly, and splitting zone can help the bad situations scalability. They are working at the different levels.
As for PCP auto-tuning, I think that it's hard to implement it to resolve all problems (that is, makes PCP never be drained). And auto-tuning doesn't sound easy. Do you have some idea of how to do
that?
Quoted 3 lines
If we could avoid instantiating more zones and rather improve existing mechanisms (PCP), that would be much more preferred IMHO. I'm sure it's not easy, but that shouldn't stop us from trying ;)
I do think improving PCP or adding another level of cache will help performance and scalability.
And, I think that it has value too to improve the performance of zone itself. Because there will be always some cases that the zone lock itself is contended.
That is, PCP and zone works at different level, and both deserve to be improved. Do you agree?
Quoted 2 lines
I did not look into the details of this proposal, but seeing the change in
include/linux/page-flags-layout.hscares me.
It's possible for us to use 1 more bit in page->flags. Do you think that will cause severe issue? Or you think some other stuff isn't acceptable?
Quoted 4 lines
Further, I'm not so sure how that change really interacts with hot(un)plug of memory ... on a quick glimpse I feel like this series hacks the code such that such that the split works based on the boot memory size ...
Em..., the zone stuff is kind of static now. It's hard to add a zone at run-time. So, in this series, we determine the number of zones per zone type based on boot memory size. This may be improved in the future via pre-allocate some empty zone instances during boot and hot-add some memory to these zones.
Quoted 3 lines
I agree with Michal that looking into auto-tuning PCP would be preferred. If that can't be done, adding another layer might end up cleaner and eventually cover more use cases.
I do agree that it's valuable to make PCP etc. cover more use cases. I just think that this should not prevent us from optimizing zone itself to cover remaining use cases.
Quoted 4 lines
[I recall there was once a proposal to add a 3rd layer to limit fragmenation to individual memory blocks; but the granularity was rather small and there were also some concerns that I don't recall anymore]
Best Regards,
Huang, Ying
Quoted 7 lines
Quoted 3 lines
If we could avoid instantiating more zones and rather improve existing mechanisms (PCP), that would be much more preferred IMHO. I'm sure it's not easy, but that shouldn't stop us from trying ;)
I do think improving PCP or adding another level of cache will help performance and scalability.
Quoted 6 lines
And, I think that it has value too to improve the performance of zone itself. Because there will be always some cases that the zone lock itself is contended.
That is, PCP and zone works at different level, and both deserve to be improved. Do you agree?
Spoiler: my humble opinion
Well, the zone is kind-of your "global" memory provider, and PCPs cache a fraction of that to avoid exactly having to mess with that global datastructure and lock contention.
One benefit I can see of such a "global" memory provider with caches on top is is that it is nicely integrated: for example, the concept of memory pressure exists for the zone as a whole. All memory is of the same kind and managed in a single entity, but free memory is cached for performance.
As soon as you manage the memory in multiple zones of the same kind, you lose that "global" view of your memory that is of the same kind, but managed in different bucks. You might end up with a lot of memory pressure in a single such zone, but still have plenty in another zone.
As one example, hot(un)plug of memory is easy: there is only a single zone. No need to make smart decisions or deal with having memory we're hotunplugging be stranded in multiple zones.
Quoted 7 lines
Quoted 3 lines
I did not look into the details of this proposal, but seeing the change in
include/linux/page-flags-layout.hscares me.It's possible for us to use 1 more bit in
page->flags. Do you think that will cause severe issue? Or you think some other stuff isn't acceptable?
The issue is, everybody wants to consume more bits in page->flags, so if we can get away without it that would be much better :)
The more bits you want to consume, the more people will ask for making this a compile-time option and eventually compile it out on distro kernels (e.g., with many NUMA nodes). So we end up with more code and complexity and eventually not get the benefits where we really want them.
Quoted 11 lines
Quoted 5 lines
Further, I'm not so sure how that change really interacts with hot(un)plug of memory ... on a quick glimpse I feel like this series hacks the code such that such that the split works based on the boot memory size ...
Em..., the zone stuff is kind of static now. It's hard to add a zone at run-time. So, in this series, we determine the number of zones per zone type based on boot memory size. This may be improved in the future via pre-allocate some empty zone instances during boot and hot-add some memory to these zones.
Just to give you some idea: with virtio-mem, hyper-v, daxctl, and upcoming cxl dynamic memory pooling (some day I'm sure ;) ) you might see quite a small boot memory (e.g., 4 GiB) but a significant amount of memory getting hotplugged incrementally (e.g., up to 1 TiB) -- well, and hotunplugged. With multiple zone instances you really have to be careful and might have to re-balance between the multiple zones to keep the scalability, to not create imbalances between the zones ...
Something like PCP auto-tuning would be able to handle that mostly automatically, as there is only a single memory pool.
Quoted 8 lines
Quoted 4 lines
I agree with Michal that looking into auto-tuning PCP would be preferred. If that can't be done, adding another layer might end up cleaner and eventually cover more use cases.
I do agree that it's valuable to make PCP etc. cover more use cases. I just think that this should not prevent us from optimizing zone itself to cover remaining use cases.
I really don't like the concept of replicating zones of the same kind for the same NUMA node. But that's just my personal opinion maintaining some memory hot(un)plug code :)
Having that said, some kind of a sub-zone concept (additional layer) as outlined by Michal IIUC, for example, indexed by core id/has/whatsoever could eventually be worth exploring. Yes, such a design raises various questions ... :)
Signature · 3 lines
Thanks, David / dhildenb
Quoted 26 lines · David Hildenbrand
Quoted 12 lines
Quoted 3 lines
If we could avoid instantiating more zones and rather improve existing mechanisms (PCP), that would be much more preferred IMHO. I'm sure it's not easy, but that shouldn't stop us from trying ;)
I do think improving PCP or adding another level of cache will help performance and scalability.
And, I think that it has value too to improve the performance of zone
itself. Because there will be always some cases that the zone lock itself is contended.
That is, PCP and zone works at different level, and both deserve to be
improved. Do you agree?Spoiler: my humble opinion
Well, the zone is kind-of your "global" memory provider, and PCPs cache a fraction of that to avoid exactly having to mess with that global datastructure and lock contention.
One benefit I can see of such a "global" memory provider with caches on top is is that it is nicely integrated: for example, the concept of memory pressure exists for the zone as a whole. All memory is of the same kind and managed in a single entity, but free memory is cached for performance.
Quoted 8 lines
As soon as you manage the memory in multiple zones of the same kind, you lose that "global" view of your memory that is of the same kind, but managed in different bucks. You might end up with a lot of memory pressure in a single such zone, but still have plenty in another zone.
As one example, hot(un)plug of memory is easy: there is only a single zone. No need to make smart decisions or deal with having memory we're hotunplugging be stranded in multiple zones.
I understand that there are some unresolved issues for splitting zone. I will think more about them and the possible solutions.
Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
I did not look into the details of this proposal, but seeing the change in
include/linux/page-flags-layout.hscares me.It's possible for us to use 1 more bit in
page->flags. Do you think that will cause severe issue? Or you think some other stuff isn't acceptable?The issue is, everybody wants to consume more bits in
page->flags, so if we can get away without it that would be much better :)
Yes.
Quoted 5 lines
The more bits you want to consume, the more people will ask for making this a compile-time option and eventually compile it out on distro kernels (e.g., with many NUMA nodes). So we end up with more code and complexity and eventually not get the benefits where we really want them.
That's possible. Although I think we will still use more page flags when necessary.
Quoted 12 lines
Quoted 11 lines
Quoted 5 lines
Further, I'm not so sure how that change really interacts with hot(un)plug of memory ... on a quick glimpse I feel like this series hacks the code such that such that the split works based on the boot memory size ...
Em..., the zone stuff is kind of static now. It's hard to add a zone at
run-time. So, in this series, we determine the number of zones per zone type based on boot memory size. This may be improved in the future via pre-allocate some empty zone instances during boot and hot-add some memory to these zones.
Quoted 8 lines
Just to give you some idea: with virtio-mem, hyper-v, daxctl, and upcoming cxl dynamic memory pooling (some day I'm sure ;) ) you might see quite a small boot memory (e.g., 4 GiB) but a significant amount of memory getting hotplugged incrementally (e.g., up to 1 TiB) -- well, and hotunplugged. With multiple zone instances you really have to be careful and might have to re-balance between the multiple zones to keep the scalability, to not create imbalances between the zones
...
Thanks for your information!
Quoted 2 lines
Something like PCP auto-tuning would be able to handle that mostly automatically, as there is only a single memory pool.
I agree that optimizing PCP will help performance regardless of splitting zone or not.
Quoted 9 lines
Quoted 8 lines
Quoted 4 lines
I agree with Michal that looking into auto-tuning PCP would be preferred. If that can't be done, adding another layer might end up cleaner and eventually cover more use cases.
I do agree that it's valuable to make PCP etc. cover more use cases. I
just think that this should not prevent us from optimizing zone itself to cover remaining use cases.
Quoted 8 lines
I really don't like the concept of replicating zones of the same kind for the same NUMA node. But that's just my personal opinion maintaining some memory hot(un)plug code :)
Having that said, some kind of a sub-zone concept (additional layer) as outlined by Michal IIUC, for example, indexed by core id/has/whatsoever could eventually be worth exploring. Yes, such a design raises various questions ... :)
Yes. That's another possible solution for the page allocation scalability problem.
Best Regards,
Huang, Ying
Michal Hocko
maintainer[Sorry for late reply, conferencing last 2 weeks and now catching up]
On Tue 16-05-23 12:30:21, David Hildenbrand wrote:
[...]
Quoted 6 lines
Quoted 2 lines
And auto-tuning doesn't sound easy. Do you have some idea of how to do that?
If we could avoid instantiating more zones and rather improve existing mechanisms (PCP), that would be much more preferred IMHO. I'm sure it's not easy, but that shouldn't stop us from trying ;)
Absolutely agreed. Increasing the zone number sounds like a hack to me TBH. It seems like an easier way but it allows more subtle problems later on. E.g. hard to predict per-zone memory consumption and memory reclaim disbalances.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 1 line · Michal Hocko
[Sorry for late reply, conferencing last 2 weeks and now catching up]
Never mind. And my reply is late too for sickness.
Quoted 5 lines · David Hildenbrand
[...]
Quoted 3 lines
Quoted 2 lines
And auto-tuning doesn't sound easy. Do you have some idea of how to do that?
Quoted 8 lines
Quoted 3 lines
If we could avoid instantiating more zones and rather improve existing mechanisms (PCP), that would be much more preferred IMHO. I'm sure it's not easy, but that shouldn't stop us from trying ;)
Absolutely agreed. Increasing the zone number sounds like a hack to me TBH. It seems like an easier way but it allows more subtle problems later on. E.g. hard to predict per-zone memory consumption and memory reclaim disbalances.
At least, we all think that improving PCP is something deserved to be done. I will do some experiment on that and return to you (after some time).
Best Regards,
Huang, Ying