Modify zonelist to nodelist v1
Subsystems (21)
cover Modify zonelist to nodelist v1
Motivation
----------
Currently if we want to iterate through all the nodes we have to traverse all the zones from the zonelist.
So in order to reduce the number of loops required to traverse node, this series of patches modified the zonelist to nodelist.
Two new macros have been introduced:
1) for_each_node_nlist
2) for_each_node_nlist_nodemask
Benefit
-------
1. For a NUMA system with N nodes, each node has M zones, the number of loops is reduced from N*M times to N times when traversing node.
2. The size of pg_data_t is much reduced.
Test Result
-----------
Currently I have only performed a simple page allocation benchmark test on my laptop, and the results show that the performance of a system with only one node is almost unaffected.
Others
------
Next I will do more performance testing and add it to the Test Result.
Since I don't currently have multiple node NUMA systems, I would be grateful if anyone would like to test this series of patches.
I am still not sure this series of patches is on the right way so I am sending this as an RFC.
Any comments are highly appreciated.
Pengfei Li (19):
mm, mmzone: modify zonelist to nodelist
mm, hugetlb: use for_each_node in dequeue_huge_page_nodemask() mm, oom_kill: use for_each_node in constrained_alloc() mm, slub: use for_each_node in get_any_partial()
mm, slab: use for_each_node in fallback_alloc()
mm, vmscan: use for_each_node in do_try_to_free_pages() mm, vmscan: use first_node in throttle_direct_reclaim() mm, vmscan: pass pgdat to wakeup_kswapd()
mm, vmscan: use for_each_node in shrink_zones()
mm, page_alloc: use for_each_node in wake_all_kswapds() mm, mempolicy: use first_node in mempolicy_slab_node() mm, mempolicy: use first_node in mpol_misplaced() mm, page_alloc: use first_node in local_memory_node() mm, compaction: rename compaction_zonelist_suitable mm, mm_init: rename mminit_verify_zonelist
mm, page_alloc: cleanup build_zonelists
mm, memory_hotplug: cleanup online_pages()
kernel, sysctl: cleanup numa_zonelist_order
mm, mmzone: cleanup zonelist in comments
arch/hexagon/mm/init.c | 2 +- arch/ia64/include/asm/topology.h | 2 +- arch/x86/mm/numa.c | 2 +- drivers/tty/sysrq.c | 2 +- include/linux/compaction.h | 2 +- include/linux/gfp.h | 18 +- include/linux/mmzone.h | 249 +++++++++++++------------ include/linux/oom.h | 4 +- include/linux/swap.h | 2 +- include/trace/events/oom.h | 9 +- init/main.c | 2 +- kernel/cgroup/cpuset.c | 4 +- kernel/sysctl.c | 8 +- mm/compaction.c | 20 +- mm/hugetlb.c | 21 +-- mm/internal.h | 13 +- mm/memcontrol.c | 2 +- mm/memory_hotplug.c | 24 +-- mm/mempolicy.c | 26 ++- mm/mm_init.c | 74 +++++--- mm/mmzone.c | 30 ++- mm/oom_kill.c | 16 +- mm/page_alloc.c | 301 ++++++++++++++++--------------- mm/slab.c | 13 +- mm/slub.c | 14 +- mm/vmscan.c | 149 ++++++++------- 26 files changed, 518 insertions(+), 491 deletions(-)
Signature · 1 line
2.23.0
RFC 1/19 mm, mmzone: modify zonelist to nodelist
Motivation
----------
Currently if we want to iterate through all the nodes we have to traverse all the zones from the zonelist.
So in order to reduce the number of loops required to traverse node, this series of patches modified the zonelist to nodelist.
Two new macros have been introduced:
1) for_each_node_nlist
2) for_each_node_nlist_nodemask
Benefit
-------
1. For a NUMA system with N nodes, each node has M zones, the number of loops is reduced from N*M times to N times when traversing node.
2. The size of pg_data_t is much reduced.
--- drivers/tty/sysrq.c | 2 +- include/linux/gfp.h | 10 +- include/linux/mmzone.h | 239 ++++++++++++++++++++----------------- include/linux/oom.h | 4 +- include/linux/swap.h | 2 +- include/trace/events/oom.h | 9 +- mm/compaction.c | 18 +-- mm/hugetlb.c | 8 +- mm/internal.h | 7 +- mm/memcontrol.c | 2 +- mm/mempolicy.c | 18 +-- mm/mm_init.c | 70 ++++++----- mm/mmzone.c | 30 ++--- mm/oom_kill.c | 10 +- mm/page_alloc.c | 207 +++++++++++++++++--------------- mm/slab.c | 8 +- mm/slub.c | 8 +- mm/vmscan.c | 34 +++--- 18 files changed, 367 insertions(+), 319 deletions(-)
drivers/tty/sysrq.c +1 -1
@@ -362,7 +362,7 @@ static void moom_callback(struct work_struct *ignored) {const gfp_t gfp_mask =GFP_KERNEL ;struct oom_control oc = { - .zonelist =node_zonelist (first_memory_node, gfp_mask), + .nodelist =node_nodelist (first_memory_node, gfp_mask), .nodemask =NULL , .memcg =NULL , .gfp_mask = gfp_mask,
include/linux/gfp.h +5 -5
@@ -456,13 +456,13 @@ static inline enum zone_type gfp_zone(gfp_t flags)* virtual kernel addresses to the allocated page(s). */ -static inline int gfp_zonelist (gfp_t flags) +static inline int gfp_nodelist (gfp_t flags) {#ifdef CONFIG_NUMA if (unlikely (flags & __GFP_THISNODE)) -return ZONELIST_NOFALLBACK ; +return NODELIST_NOFALLBACK ;#endif -return ZONELIST_FALLBACK ; +return NODELIST_FALLBACK ; }/*
@@ -474,9 +474,9 @@ static inline int gfp_zonelist(gfp_t flags)* For the normal case of non-DISCONTIGMEM systems the NODE_DATA() gets * optimized to &contig_page_data at compile-time. */ -static inline struct zonelist *node_zonelist (int nid,gfp_t flags) +static inline struct nodelist *node_nodelist (int nid,gfp_t flags) { -return NODE_DATA (nid)->node_zonelists +gfp_zonelist (flags); +return NODE_DATA (nid)->node_nodelists +gfp_nodelist (flags); }#ifndef HAVE_ARCH_FREE_PAGE
include/linux/mmzone.h +131 -108
289 lines not on this page · raw message
include/linux/oom.h +2 -2
@@ -10,7 +10,7 @@#include <linux/sched/coredump.h> /* MMF_* */ #include <linux/mm.h> /* VM_FAULT* */ -struct zonelist ; +struct nodelist ;struct notifier_block ;struct mem_cgroup ;struct task_struct ;
@@ -28,7 +28,7 @@ enum oom_constraint {*/ struct oom_control {/* Used to determine cpuset */ -struct zonelist *zonelist; +struct nodelist *nodelist;/* Used to determine mempolicy */ nodemask_t *nodemask;
include/linux/swap.h +1 -1
@@ -349,7 +349,7 @@ extern void lru_cache_add_active_or_unevictable(struct page *page,/* linux/mm/vmscan.c */ extern unsigned long zone_reclaimable_pages (struct zone *zone); -extern unsigned long try_to_free_pages (struct zonelist *zonelist,int order, +extern unsigned long try_to_free_pages (struct nodelist *nodelist,int order,gfp_t gfp_mask,nodemask_t *mask);extern int __isolate_lru_page (struct page *page,isolate_mode_t mode); extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,
include/trace/events/oom.h +5 -4
@@ -31,7 +31,8 @@ TRACE_EVENT(oom_score_adj_update, TRACE_EVENT(reclaim_retry_zone, - TP_PROTO(struct zoneref *zoneref, + TP_PROTO(int node, + struct zone *zone, int order, unsigned long reclaimable, unsigned long available,
@@ -39,7 +40,7 @@ TRACE_EVENT(reclaim_retry_zone,int no_progress_loops, bool wmark_check), -TP_ARGS (zoneref ,order ,reclaimable ,available ,min_wmark ,no_progress_loops ,wmark_check ), +TP_ARGS (node ,zone ,order ,reclaimable ,available ,min_wmark ,no_progress_loops ,wmark_check ),TP_STRUCT__entry ( __field( int, node)
@@ -53,8 +54,8 @@ TRACE_EVENT(reclaim_retry_zone, ), TP_fast_assign( - __entry->node = zone_to_nid(zoneref->zone); - __entry->zone_idx = zoneref->zone_idx; + __entry->node = node; + __entry->zone_idx =zone_idx (zone); __entry->order = order; __entry->reclaimable = reclaimable; __entry->available = available;
mm/compaction.c +9 -9
@@ -2039,15 +2039,15 @@ enum compact_result compaction_suitable(struct zone *zone, int order,bool compaction_zonelist_suitable (struct alloc_context *ac,int order,int alloc_flags) { +struct nlist_traverser t;struct zone *zone; -struct zoneref *z;/* * Make sure at least one zone would pass __compaction_suitable if we continue * retrying the reclaim. */ -for_each_zone_zonelist_nodemask (zone, z, ac->zonelist, ac->high_zoneidx, - ac->nodemask) { +for_each_zone_nlist_nodemask (zone, &t, ac->nodelist, + ac->high_zoneidx, ac->nodemask) {unsigned long available;enum compact_result compact_result;
@@ -2060,7 +2060,7 @@ bool compaction_zonelist_suitable(struct alloc_context *ac, int order, available =zone_reclaimable_pages (zone) / order; available +=zone_page_state_snapshot (zone,NR_FREE_PAGES ); compact_result =__compaction_suitable (zone, order, alloc_flags, -ac_classzone_idx (ac), available); + ac->classzone_idx, available);if (compact_result !=COMPACT_SKIPPED )return true; }
@@ -2341,9 +2341,9 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, enum compact_priority prio, struct page **capture) { int may_perform_io = gfp_mask & __GFP_IO; -struct zoneref *z; -struct zone *zone;enum compact_result rc =COMPACT_SKIPPED ; +struct nlist_traverser t; +struct zone *zone;/* * Check if the GFP flags allow compaction - GFP_NOIO is really
@@ -2355,8 +2355,8 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order,trace_mm_compaction_try_to_compact_pages (order, gfp_mask, prio);/* Compact each zone in the list */ -for_each_zone_zonelist_nodemask (zone, z, ac->zonelist, ac->high_zoneidx, - ac->nodemask) { +for_each_zone_nlist_nodemask (zone, &t, ac->nodelist, + ac->high_zoneidx, ac->nodemask) {enum compact_result status; if (prio > MIN_COMPACT_PRIORITY
@@ -2366,7 +2366,7 @@ enum compact_result try_to_compact_pages(gfp_t gfp_mask, unsigned int order, } status =compact_zone_order (zone, order, gfp_mask, prio, - alloc_flags,ac_classzone_idx (ac), capture); + alloc_flags, ac->classzone_idx, capture); rc =max (status, rc);/* The allocation should succeed, stop compacting */
mm/hugetlb.c +4 -4
@@ -848,16 +848,16 @@ static struct page *dequeue_huge_page_nodemask(struct hstate *h, gfp_t gfp_mask, nodemask_t *nmask) {unsigned int cpuset_mems_cookie; -struct zonelist *zonelist; +struct nodelist *nodelist; +struct nlist_traverser t;struct zone *zone; -struct zoneref *z;int node =NUMA_NO_NODE ; - zonelist =node_zonelist (nid, gfp_mask); + nodelist =node_nodelist (nid, gfp_mask); retry_cpuset: cpuset_mems_cookie =read_mems_allowed_begin (); -for_each_zone_zonelist_nodemask (zone, z, zonelist,gfp_zone (gfp_mask), nmask) { +for_each_zone_nlist_nodemask (zone, &t, nodelist,gfp_zone (gfp_mask), nmask) {struct page *page;if (!cpuset_zone_allowed (zone, gfp_mask))
mm/internal.h +3 -4
@@ -111,16 +111,15 @@ extern pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address);* by a const pointer. */ struct alloc_context { -struct zonelist *zonelist; +struct nodelist *nodelist;nodemask_t *nodemask; -struct zoneref *preferred_zoneref; +struct zone *preferred_zone; +enum zone_type classzone_idx;int migratetype;enum zone_type high_zoneidx;bool spread_dirty_pages; }; -#define ac_classzone_idx (ac) zonelist_zone_idx(ac->preferred_zoneref) -/* * Locate the struct page for both the matching buddy in our * pair (buddy1) and the combined O(n+1) page they form (page).
mm/memcontrol.c +1 -1
@@ -1558,7 +1558,7 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, int order) {struct oom_control oc = { - .zonelist =NULL , + .nodelist =NULL , .nodemask =NULL , .memcg = memcg, .gfp_mask = gfp_mask,
mm/mempolicy.c +9 -9
@@ -1876,18 +1876,18 @@ unsigned int mempolicy_slab_node(void)return interleave_nodes (policy);case MPOL_BIND : { -struct zoneref *z; +struct zone *zone;/* * Follow bind policy behavior and start allocation at the * first node. */ -struct zonelist *zonelist; +struct nodelist *nodelist;enum zone_type highest_zoneidx =gfp_zone (GFP_KERNEL ); - zonelist = &NODE_DATA (node)->node_zonelists[ZONELIST_FALLBACK ]; - z =first_zones_zonelist (zonelist, highest_zoneidx, + nodelist = &NODE_DATA (node)->node_nodelists[NODELIST_FALLBACK ]; + zone =first_zone_nlist_nodemask (nodelist, highest_zoneidx, &policy->v.nodes); -return z->zone ?zone_to_nid (z->zone) : node; +return zone ?zone_to_nid (zone) : node; }default :
@@ -2404,7 +2404,7 @@ static void sp_free(struct sp_node *n)int mpol_misplaced (struct page *page,struct vm_area_struct *vma,unsigned long addr) {struct mempolicy *pol; -struct zoneref *z; +struct zone *zone;int curnid =page_to_nid (page);unsigned long pgoff;int thiscpu =raw_smp_processor_id ();
@@ -2440,11 +2440,11 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long*/ if (node_isset (curnid, pol->v.nodes)) goto out; - z =first_zones_zonelist ( -node_zonelist (numa_node_id (),GFP_HIGHUSER ), + zone =first_zone_nlist_nodemask ( +node_nodelist (numa_node_id (),GFP_HIGHUSER ),gfp_zone (GFP_HIGHUSER ), &pol->v.nodes); - polnid =zone_to_nid (z->zone); + polnid =zone_to_nid (zone);break ;default :
mm/mm_init.c +43 -27
86 lines not on this page · raw message
mm/mmzone.c +12 -18
@@ -43,33 +43,27 @@ struct zone *next_zone(struct zone *zone)return zone; } -static inline int zref_in_nodemask (struct zoneref *zref,nodemask_t *nodes) +static inline bool node_in_nodemask (int nid,nodemask_t *nodemask) {#ifdef CONFIG_NUMA -return node_isset (zonelist_node_idx (zref), *nodes); +return node_isset (nid, *nodemask);#else -return 1 ; +return true;#endif /* CONFIG_NUMA */ } -/* Returns the next zone at or below highest_zoneidx in a zonelist */ -struct zoneref *__next_zones_zonelist (struct zoneref *z, -enum zone_type highest_zoneidx, -nodemask_t *nodes) +/* Returns the next nlist_entry at or below highest_zoneidx in a nodelist */ +void __next_nlist_entry_nodemask (struct nlist_traverser *t, +nodemask_t *nodemask) {/* -* Find the next suitable zone to use for the allocation. -* Only filter based on nodemask if it's set +* Find the next suitable nlist_entry to use for the allocation. +* Only filter based on nodemask */ -if (unlikely (nodes ==NULL )) -while (zonelist_zone_idx (z) > highest_zoneidx) - z++; -else -while (zonelist_zone_idx (z) > highest_zoneidx || - (z->zone && !zref_in_nodemask (z, nodes))) - z++; - -return z; +while (!node_has_usable_zones (t) || + (traverser_node (t) !=NUMA_NO_NODE && + !node_in_nodemask (traverser_node (t), nodemask))) + t->entry++; }#ifdef CONFIG_ARCH_HAS_HOLES_MEMORYMODEL
mm/oom_kill.c +5 -5
@@ -251,10 +251,10 @@ static const char * const oom_constraint_text[] = {*/ static enum oom_constraint constrained_alloc (struct oom_control *oc) { -struct zone *zone; -struct zoneref *z;enum zone_type high_zoneidx =gfp_zone (oc->gfp_mask);bool cpuset_limited = false; +struct nlist_traverser t; +struct zone *zone;int nid;if (is_memcg_oom (oc)) {
@@ -268,7 +268,7 @@ static enum oom_constraint constrained_alloc(struct oom_control *oc)if (!IS_ENABLED (CONFIG_NUMA ))return CONSTRAINT_NONE ; -if (!oc->zonelist) +if (!oc->nodelist)return CONSTRAINT_NONE ;/* * Reach here only when __GFP_NOFAIL is used. So, we should avoid
@@ -292,7 +292,7 @@ static enum oom_constraint constrained_alloc(struct oom_control *oc) }/* Check this allocation failure is caused by cpuset's wall function */ -for_each_zone_zonelist_nodemask (zone, z, oc->zonelist, +for_each_zone_nlist_nodemask (zone, &t, oc->nodelist, high_zoneidx, oc->nodemask)if (!cpuset_zone_allowed (zone, oc->gfp_mask)) cpuset_limited = true;
@@ -1120,7 +1120,7 @@ bool out_of_memory(struct oom_control *oc)void pagefault_out_of_memory (void ) {struct oom_control oc = { - .zonelist =NULL , + .nodelist =NULL , .nodemask =NULL , .memcg =NULL , .gfp_mask =0 ,
mm/page_alloc.c +111 -96
458 lines not on this page · raw message
mm/slab.c +4 -4
@@ -3103,8 +3103,8 @@ static void *alternate_node_alloc(struct kmem_cache *cachep, gfp_t flags)*/ static void *fallback_alloc (struct kmem_cache *cache,gfp_t flags) { -struct zonelist *zonelist; -struct zoneref *z; +struct nodelist *nodelist; +struct nlist_traverser t;struct zone *zone;enum zone_type high_zoneidx =gfp_zone (flags);void *obj =NULL ;
@@ -3117,14 +3117,14 @@ static void *fallback_alloc(struct kmem_cache *cache, gfp_t flags) retry_cpuset: cpuset_mems_cookie =read_mems_allowed_begin (); - zonelist =node_zonelist (mempolicy_slab_node (), flags); + nodelist =node_nodelist (mempolicy_slab_node (), flags); retry:/* * Look through allowed nodes for objects available * from existing per node queues. */ -for_each_zone_zonelist (zone, z, zonelist, high_zoneidx) { +for_each_zone_nlist (zone, &t, nodelist, high_zoneidx) { nid =zone_to_nid (zone); if (cpuset_zone_allowed(zone, flags) &&
mm/slub.c +4 -4
@@ -1905,8 +1905,8 @@ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, struct kmem_cache_cpu *c) {#ifdef CONFIG_NUMA -struct zonelist *zonelist; -struct zoneref *z; +struct nodelist *nodelist; +struct nlist_traverser t;struct zone *zone;enum zone_type high_zoneidx =gfp_zone (flags);void *object;
@@ -1936,8 +1936,8 @@ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, do { cpuset_mems_cookie = read_mems_allowed_begin(); - zonelist = node_zonelist(mempolicy_slab_node(), flags); - for_each_zone_zonelist(zone, z, zonelist, high_zoneidx) { + nodelist = node_nodelist(mempolicy_slab_node(), flags); + for_each_zone_nlist(zone, &t, nodelist, high_zoneidx) { struct kmem_cache_node *n; n = get_node(s, zone_to_nid(zone));
mm/vmscan.c +17 -17
@@ -2918,9 +2918,9 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc)* If a zone is deemed to be full of pinned pages then just give it a light * scan then give up on it. */ -static void shrink_zones (struct zonelist *zonelist,struct scan_control *sc) +static void shrink_zones (struct nodelist *nodelist,struct scan_control *sc) { -struct zoneref *z; +struct nlist_traverser t;struct zone *zone;unsigned long nr_soft_reclaimed;unsigned long nr_soft_scanned;
@@ -2938,7 +2938,7 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc) sc->reclaim_idx =gfp_zone (sc->gfp_mask); } -for_each_zone_zonelist_nodemask (zone, z, zonelist, +for_each_zone_nlist_nodemask (zone, &t, nodelist, sc->reclaim_idx, sc->nodemask) {/* * Take care memory controller reclaiming has small influence
@@ -3029,12 +3029,12 @@ static void snapshot_refaults(struct mem_cgroup *target_memcg, pg_data_t *pgdat)* returns: 0, if no pages reclaimed * else, the number of pages reclaimed */ -static unsigned long do_try_to_free_pages (struct zonelist *zonelist, +static unsigned long do_try_to_free_pages (struct nodelist *nodelist,struct scan_control *sc) {int initial_priority = sc->priority;pg_data_t *last_pgdat; -struct zoneref *z; +struct nlist_traverser t;struct zone *zone; retry:delayacct_freepages_start ();
@@ -3046,7 +3046,7 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist,vmpressure_prio (sc->gfp_mask, sc->target_mem_cgroup, sc->priority); sc->nr_scanned =0 ; -shrink_zones (zonelist, sc); +shrink_zones (nodelist, sc);if (sc->nr_reclaimed >= sc->nr_to_reclaim)break ;
@@ -3063,7 +3063,7 @@ static unsigned long do_try_to_free_pages(struct zonelist *zonelist, }while (--sc->priority >=0 ); last_pgdat =NULL ; -for_each_zone_zonelist_nodemask (zone, z, zonelist, sc->reclaim_idx, +for_each_zone_nlist_nodemask (zone, &t, nodelist, sc->reclaim_idx, sc->nodemask) {if (zone->zone_pgdat == last_pgdat)continue ;
@@ -3166,10 +3166,10 @@ static bool allow_direct_reclaim(pg_data_t *pgdat)* Returns true if a fatal signal was delivered during throttling. If this * happens, the page allocator should not consider triggering the OOM killer. */ -static bool throttle_direct_reclaim (gfp_t gfp_mask,struct zonelist *zonelist, +static bool throttle_direct_reclaim (gfp_t gfp_mask,struct nodelist *nodelist,nodemask_t *nodemask) { -struct zoneref *z; +struct nlist_traverser t;struct zone *zone;pg_data_t *pgdat =NULL ;
@@ -3204,7 +3204,7 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,* for remote pfmemalloc reserves and processes on different nodes * should make reasonable progress. */ -for_each_zone_zonelist_nodemask (zone, z, zonelist, +for_each_zone_nlist_nodemask (zone, &t, nodelist,gfp_zone (gfp_mask), nodemask) {if (zone_idx (zone) >ZONE_NORMAL )continue ;
@@ -3250,7 +3250,7 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct zonelist *zonelist,return false; } -unsigned long try_to_free_pages (struct zonelist *zonelist,int order, +unsigned long try_to_free_pages (struct nodelist *nodelist,int order,gfp_t gfp_mask,nodemask_t *nodemask) {unsigned long nr_reclaimed;
@@ -3279,13 +3279,13 @@ unsigned long try_to_free_pages(struct zonelist *zonelist, int order,* 1 is returned so that the page allocator does not OOM kill at this * point. */ -if (throttle_direct_reclaim (sc.gfp_mask, zonelist, nodemask)) +if (throttle_direct_reclaim (sc.gfp_mask, nodelist, nodemask))return 1 ;set_task_reclaim_state (current, &sc.reclaim_state);trace_mm_vmscan_direct_reclaim_begin (order, sc.gfp_mask); - nr_reclaimed =do_try_to_free_pages (zonelist, &sc); + nr_reclaimed =do_try_to_free_pages (nodelist, &sc);trace_mm_vmscan_direct_reclaim_end (nr_reclaimed);set_task_reclaim_state (current,NULL );
@@ -3363,7 +3363,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,* equal pressure on all the nodes. This is based on the assumption that * the reclaim does not bail out early. */ -struct zonelist *zonelist =node_zonelist (numa_node_id (), sc.gfp_mask); +struct nodelist *nodelist =node_nodelist (numa_node_id (), sc.gfp_mask);set_task_reclaim_state (current, &sc.reclaim_state);
@@ -3374,7 +3374,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg,psi_memstall_enter (&pflags); noreclaim_flag =memalloc_noreclaim_save (); - nr_reclaimed =do_try_to_free_pages (zonelist, &sc); + nr_reclaimed =do_try_to_free_pages (nodelist, &sc);memalloc_noreclaim_restore (noreclaim_flag);psi_memstall_leave (&pflags);
@@ -4033,7 +4033,7 @@ unsigned long shrink_all_memory(unsigned long nr_to_reclaim) .may_swap = 1, .hibernation_mode = 1, }; -struct zonelist *zonelist =node_zonelist (numa_node_id (), sc.gfp_mask); +struct nodelist *nodelist =node_nodelist (numa_node_id (), sc.gfp_mask);unsigned long nr_reclaimed;unsigned int noreclaim_flag;
@@ -4041,7 +4041,7 @@ unsigned long shrink_all_memory(unsigned long nr_to_reclaim) noreclaim_flag =memalloc_noreclaim_save ();set_task_reclaim_state (current, &sc.reclaim_state); - nr_reclaimed =do_try_to_free_pages (zonelist, &sc); + nr_reclaimed =do_try_to_free_pages (nodelist, &sc);set_task_reclaim_state (current,NULL );memalloc_noreclaim_restore (noreclaim_flag);
Signature · 1 line
2.23.0
kbuild test robot
build botRe: [RFC v1 01/19] mm, mmzone: modify zonelist to nodelist
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please…
Hi Pengfei,
[FYI, it's a private test report for your RFC patch.]
[auto build test WARNING on next-20191122]
[cannot apply to tty/tty-testing tip/perf/core linus/master v5.4-rc8 v5.4-rc7 v5.4-rc6 v5.4] [if your patch is applied to the wrong git tree, please drop us a note to help improve the system. BTW, we also suggest to use '--base' option to specify the base tree in git format-patch, please see https://stackoverflow.com/a/37406982]
url: https://github.com/0day-ci/linux/commits/Pengfei-Li/Modify-zonelist-to-nodelist-v1/20191124-070128 base: b9d3d01405061bb42358fe53f824e894a1922ced reproduce: # apt-get install sparse # sparse version: v0.6.1-36-g9305d48-dirty make ARCH=x86_64 allmodconfig make C=1 CF='-fdiagnostic-prefix -D__CHECK_ENDIAN__'
If you fix the issue, kindly add following tag
sparse warnings: (new ones prefixed by >>)
>> mm/mm_init.c:21:12: sparse: sparse: symbol 'nodelist_name' was not declared. Should it be static? >> mm/mm_init.c:32:13: sparse: sparse: symbol 'mminit_print_nodelist' was not declared. Should it be static?
Please review and possibly fold the followup patch.
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/hyperkitty/list/kbuild-all(a)lists.01.org Intel Corporation
kbuild test robot
build bot[RFC PATCH] mm, mmzone: nodelist_name[] can be static
In reply to Pengfei LiFixes: 6ba7c7dd07ae ("mm, mmzone: modify zonelist to nodelist")
--- mm_init.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
mm/mm_init.c +2 -2
@@ -18,7 +18,7 @@#ifdef CONFIG_DEBUG_MEMORY_INIT int __meminitdata mminit_loglevel; -const char *nodelist_name[MAX_NODELISTS ]__meminitconst = { +static const char *nodelist_name[MAX_NODELISTS ]__meminitconst = {"general" , #ifdef CONFIG_NUMA "thisnode"
@@ -29,7 +29,7 @@ const char *nodelist_name[MAX_NODELISTS] __meminitconst = {#define SECTIONS_SHIFT 0#endif -void __init mminit_print_nodelist (int nid,int nl_type) +static void __init mminit_print_nodelist (int nid,int nl_type) {pg_data_t *pgdat =NODE_DATA (nid);struct nodelist *nodelist;
RFC 2/19 mm, hugetlb: use for_each_node in dequeue_huge_page_nodemask()
In reply to Pengfei LiIn dequeue_huge_page_nodemask(), we want to traverse node instead of zone, so use for_each_node instead of for_each_zone.
--- mm/hugetlb.c | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-)
mm/hugetlb.c +4 -11
@@ -850,25 +850,18 @@ static struct page *dequeue_huge_page_nodemask(struct hstate *h, gfp_t gfp_mask,unsigned int cpuset_mems_cookie;struct nodelist *nodelist;struct nlist_traverser t; -struct zone *zone; -int node =NUMA_NO_NODE ; +int node; nodelist =node_nodelist (nid, gfp_mask); retry_cpuset: cpuset_mems_cookie =read_mems_allowed_begin (); -for_each_zone_nlist_nodemask (zone, &t, nodelist,gfp_zone (gfp_mask), nmask) { +for_each_node_nlist_nodemask (node, &t, nodelist, +gfp_zone (gfp_mask), nmask) {struct page *page; -if (!cpuset_zone_allowed (zone, gfp_mask)) -continue ; -/* -* no need to ask again on the same node. Pool is node rather than -* zone aware -*/ -if (zone_to_nid (zone) == node) +if (!cpuset_node_allowed (node, gfp_mask))continue ; - node =zone_to_nid (zone); page =dequeue_huge_page_node_exact (h, node);if (page)
Signature · 1 line
2.23.0
RFC 3/19 mm, oom_kill: use for_each_node in constrained_alloc()
In reply to Pengfei LiIn constrained_alloc(), we want to traverse node instead of zone, so use for_each_node instead of for_each_zone.
--- mm/oom_kill.c | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-)
mm/oom_kill.c +6 -4
@@ -254,7 +254,6 @@ static enum oom_constraint constrained_alloc(struct oom_control *oc)enum zone_type high_zoneidx =gfp_zone (oc->gfp_mask);bool cpuset_limited = false;struct nlist_traverser t; -struct zone *zone;int nid;if (is_memcg_oom (oc)) {
@@ -292,10 +291,13 @@ static enum oom_constraint constrained_alloc(struct oom_control *oc) }/* Check this allocation failure is caused by cpuset's wall function */ -for_each_zone_nlist_nodemask (zone, &t, oc->nodelist, - high_zoneidx, oc->nodemask) -if (!cpuset_zone_allowed (zone, oc->gfp_mask)) +for_each_node_nlist_nodemask (nid, &t, oc->nodelist, + high_zoneidx, oc->nodemask) { +if (!cpuset_node_allowed (nid, oc->gfp_mask)) { cpuset_limited = true; +break ; + } + }if (cpuset_limited) { oc->totalpages = total_swap_pages;
Signature · 1 line
2.23.0
RFC 4/19 mm, slub: use for_each_node in get_any_partial()
In reply to Pengfei LiIn get_any_partial(), we want to traverse node instead of zone, so use for_each_node instead of for_each_zone.
--- mm/slub.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
mm/slub.c +5 -5
@@ -1905,10 +1905,10 @@ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, struct kmem_cache_cpu *c) {#ifdef CONFIG_NUMA +enum zone_type high_zoneidx =gfp_zone (flags);struct nodelist *nodelist;struct nlist_traverser t; -struct zone *zone; -enum zone_type high_zoneidx =gfp_zone (flags); +int node;void *object;unsigned int cpuset_mems_cookie;
@@ -1937,12 +1937,12 @@ static void *get_any_partial(struct kmem_cache *s, gfp_t flags, do { cpuset_mems_cookie = read_mems_allowed_begin(); nodelist = node_nodelist(mempolicy_slab_node(), flags); - for_each_zone_nlist(zone, &t, nodelist, high_zoneidx) { + for_each_node_nlist(node, &t, nodelist, high_zoneidx) { struct kmem_cache_node *n; - n = get_node(s, zone_to_nid(zone)); + n = get_node(s, node); - if (n && cpuset_zone_allowed(zone, flags) && + if (n && cpuset_node_allowed(node, flags) && n->nr_partial > s->min_partial) { object = get_partial_node(s, n, c, flags); if (object) {
Signature · 1 line
2.23.0
RFC 5/19 mm, slab: use for_each_node in fallback_alloc()
In reply to Pengfei LiIn fallback_alloc(), we want to traverse node instead of zone, so use for_each_node instead of for_each_zone.
--- mm/slab.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
mm/slab.c +4 -7
@@ -3103,12 +3103,11 @@ static void *alternate_node_alloc(struct kmem_cache *cachep, gfp_t flags)*/ static void *fallback_alloc (struct kmem_cache *cache,gfp_t flags) { -struct nodelist *nodelist; -struct nlist_traverser t; -struct zone *zone;enum zone_type high_zoneidx =gfp_zone (flags);void *obj =NULL ;struct page *page; +struct nodelist *nodelist; +struct nlist_traverser t;int nid;unsigned int cpuset_mems_cookie;
@@ -3124,10 +3123,8 @@ static void *fallback_alloc(struct kmem_cache *cache, gfp_t flags)* Look through allowed nodes for objects available * from existing per node queues. */ -for_each_zone_nlist (zone, &t, nodelist, high_zoneidx) { - nid =zone_to_nid (zone); - -if (cpuset_zone_allowed (zone, flags) && +for_each_node_nlist (nid, &t, nodelist, high_zoneidx) { +if (cpuset_node_allowed (nid, flags) &&get_node (cache, nid) &&get_node (cache, nid)->free_objects) { obj = ____cache_alloc_node(cache,
Signature · 1 line
2.23.0
RFC 6/19 mm, vmscan: use for_each_node in do_try_to_free_pages()
In reply to Pengfei LiIn do_try_to_free_pages(), we want to traverse node instead of zone, so use for_each_node instead of for_each_zone.
--- mm/vmscan.c | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-)
mm/vmscan.c +7 -10
@@ -3033,9 +3033,9 @@ static unsigned long do_try_to_free_pages(struct nodelist *nodelist, struct scan_control *sc) {int initial_priority = sc->priority; -pg_data_t *last_pgdat; +pg_data_t *pgdat;struct nlist_traverser t; -struct zone *zone; +int node; retry:delayacct_freepages_start ();
@@ -3062,20 +3062,17 @@ static unsigned long do_try_to_free_pages(struct nodelist *nodelist, sc->may_writepage =1 ; }while (--sc->priority >=0 ); - last_pgdat =NULL ; -for_each_zone_nlist_nodemask (zone, &t, nodelist, sc->reclaim_idx, - sc->nodemask) { -if (zone->zone_pgdat == last_pgdat) -continue ; - last_pgdat = zone->zone_pgdat; +for_each_node_nlist_nodemask (node, &t, nodelist, + sc->reclaim_idx, sc->nodemask) { + pgdat =NODE_DATA (node); -snapshot_refaults (sc->target_mem_cgroup, zone->zone_pgdat); +snapshot_refaults (sc->target_mem_cgroup, pgdat);if (cgroup_reclaim (sc)) {struct lruvec *lruvec; lruvec =mem_cgroup_lruvec (sc->target_mem_cgroup, - zone->zone_pgdat); + pgdat);clear_bit (LRUVEC_CONGESTED , &lruvec->flags); } }
Signature · 1 line
2.23.0
RFC 7/19 mm, vmscan: use first_node in throttle_direct_reclaim()
In reply to Pengfei LiIn throttle_direct_reclaim(), we want to access first node instead of first zone, so use first_node_nlist instead of first_zone_zonelist.
--- mm/vmscan.c | 41 +++++++++++++++++++---------------------- 1 file changed, 19 insertions(+), 22 deletions(-)
mm/vmscan.c +19 -22
@@ -3166,9 +3166,9 @@ static bool allow_direct_reclaim(pg_data_t *pgdat)static bool throttle_direct_reclaim (gfp_t gfp_mask,struct nodelist *nodelist,nodemask_t *nodemask) { -struct nlist_traverser t; -struct zone *zone; -pg_data_t *pgdat =NULL ; +pg_data_t *pgdat; +enum zone_type high_idx; +int node;/* * Kernel threads should not be throttled as they may be indirectly
@@ -3178,21 +3178,24 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct nodelist *nodelist,* processes to block on log_wait_commit(). */ if (current->flags &PF_KTHREAD ) - goto out; +return false;/* * If a fatal signal is pending, this process should not throttle. * It should return quickly so it can exit and free its memory */ if (fatal_signal_pending (current)) - goto out; +return false;/* * Check if the pfmemalloc reserves are ok by finding the first node * with a usable ZONE_NORMAL or lower zone. The expectation is that * GFP_KERNEL will be required for allocating network buffers when * swapping over the network so ZONE_HIGHMEM is unusable. -* +*/ + high_idx = min_t(enum zone_type, ZONE_NORMAL, gfp_zone(gfp_mask)); + +/* * Throttling is based on the first usable node and throttled processes * wait on a queue until kswapd makes progress and wakes them. There * is an affinity then between processes waking up and where reclaim
@@ -3201,21 +3204,16 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct nodelist *nodelist,* for remote pfmemalloc reserves and processes on different nodes * should make reasonable progress. */ -for_each_zone_nlist_nodemask (zone, &t, nodelist, -gfp_zone (gfp_mask), nodemask) { -if (zone_idx (zone) >ZONE_NORMAL ) -continue ; - -/* Throttle based on the first usable node */ - pgdat = zone->zone_pgdat; -if (allow_direct_reclaim (pgdat)) - goto out; -break ; - } + node =first_node_nlist_nodemask (nodelist, high_idx, nodemask);/* If no zone was usable by the allocation flags then do not throttle */ -if (!pgdat) - goto out; +if (node ==NUMA_NO_NODE ) +return false; + + pgdat =NODE_DATA (node); +/* Throttle based on the first usable node */ +if (allow_direct_reclaim (pgdat)) +return false;/* Account for the throttling */ count_vm_event (PGSCAN_DIRECT_THROTTLE );
@@ -3236,14 +3234,13 @@ static bool throttle_direct_reclaim(gfp_t gfp_mask, struct nodelist *nodelist, }/* Throttle until kswapd wakes the process */ -wait_event_killable (zone->zone_pgdat->pfmemalloc_wait, -allow_direct_reclaim (pgdat)); +wait_event_killable (pgdat->pfmemalloc_wait, +allow_direct_reclaim (pgdat)); check_pending:if (fatal_signal_pending (current))return true; -out:return false; }
Signature · 1 line
2.23.0
RFC 8/19 mm, vmscan: pass pgdat to wakeup_kswapd()
In reply to Pengfei LiThis is a preparation patch. Passing pgdat to wakeup_kswapd() and avoiding indirect access to pgdat via zone->zone_pgdat.
--- include/linux/mmzone.h | 2 +- mm/page_alloc.c | 4 ++-- mm/vmscan.c | 12 ++++-------- 3 files changed, 7 insertions(+), 11 deletions(-)
include/linux/mmzone.h +1 -1
@@ -814,7 +814,7 @@ static inline bool pgdat_is_empty(pg_data_t *pgdat)#include <linux/memory_hotplug.h> void build_all_zonelists (pg_data_t *pgdat); -void wakeup_kswapd (struct zone *zone,gfp_t gfp_mask,int order, +void wakeup_kswapd (pg_data_t *pgdat,gfp_t gfp_mask,int order,enum zone_type classzone_idx); bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark, int classzone_idx, unsigned int alloc_flags,
mm/page_alloc.c +2 -2
@@ -3306,7 +3306,7 @@ struct page *rmqueue(struct zone *preferred_zone,/* Separate test+clear to avoid unnecessary atomics */ if (test_bit (ZONE_BOOSTED_WATERMARK , &zone->flags)) {clear_bit (ZONE_BOOSTED_WATERMARK , &zone->flags); -wakeup_kswapd (zone,0 ,0 ,zone_idx (zone)); +wakeup_kswapd (zone->zone_pgdat,0 ,0 ,zone_idx (zone)); }VM_BUG_ON_PAGE (page &&bad_range (zone, page), page);
@@ -4173,7 +4173,7 @@ static void wake_all_kswapds(unsigned int order, gfp_t gfp_mask,for_each_zone_nlist_nodemask (zone, &t, ac->nodelist, high_zoneidx, ac->nodemask) {if (last_pgdat != zone->zone_pgdat) -wakeup_kswapd (zone, gfp_mask, order, high_zoneidx); +wakeup_kswapd (zone->zone_pgdat, gfp_mask, order, high_zoneidx); last_pgdat = zone->zone_pgdat; } }
mm/vmscan.c +4 -8
@@ -3964,17 +3964,13 @@ static int kswapd(void *p)* has failed or is not needed, still wake up kcompactd if only compaction is * needed. */ -void wakeup_kswapd (struct zone *zone,gfp_t gfp_flags,int order, +void wakeup_kswapd (pg_data_t *pgdat,gfp_t gfp_flags,int order,enum zone_type classzone_idx) { -pg_data_t *pgdat; - -if (!managed_zone (zone)) -return ; +int node = pgdat->node_id; -if (!cpuset_zone_allowed (zone, gfp_flags)) +if (!cpuset_node_allowed (node, gfp_flags))return ; - pgdat = zone->zone_pgdat;if (pgdat->kswapd_classzone_idx ==MAX_NR_ZONES ) pgdat->kswapd_classzone_idx = classzone_idx;
@@ -4001,7 +3997,7 @@ void wakeup_kswapd(struct zone *zone, gfp_t gfp_flags, int order,return ; } -trace_mm_vmscan_wakeup_kswapd (pgdat->node_id, classzone_idx, order, +trace_mm_vmscan_wakeup_kswapd (node, classzone_idx, order, gfp_flags);wake_up_interruptible (&pgdat->kswapd_wait); }
Signature · 1 line
2.23.0
RFC 9/19 mm, vmscan: use for_each_node in shrink_zones()
In reply to Pengfei LiIn shrink_zones(), we want to traverse node instead of zone, so use for_each_node instead of for_each_zone.
--- mm/vmscan.c | 53 ++++++++++++++++++++++++++++++----------------------- 1 file changed, 30 insertions(+), 23 deletions(-)
mm/vmscan.c +30 -23
@@ -2910,6 +2910,25 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc)return zone_watermark_ok_safe (zone,0 , watermark, sc->reclaim_idx); } +static bool +node_compaction_ready (struct nlist_traverser *t,struct scan_control *sc) +{ +bool node_ready = true; +struct zone *zone; + +do { + zone =traverser_zone (t); + +if (compaction_ready (zone, sc)) + sc->compaction_ready = true; +else + node_ready = false; + + }while (t->usable_zones); + +return node_ready; +} +/* * This is the direct reclaim path, for page-allocating processes. We only * try to reclaim pages from zones which will satisfy the caller's allocation
@@ -2920,12 +2939,12 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc)*/ static void shrink_zones (struct nodelist *nodelist,struct scan_control *sc) { -struct nlist_traverser t; -struct zone *zone;unsigned long nr_soft_reclaimed;unsigned long nr_soft_scanned;gfp_t orig_mask; -pg_data_t *last_pgdat =NULL ; +pg_data_t *pgdat; +struct nlist_traverser t; +int node;/* * If the number of buffer_heads in the machine exceeds the maximum
@@ -2938,14 +2957,17 @@ static void shrink_zones(struct nodelist *nodelist, struct scan_control *sc) sc->reclaim_idx =gfp_zone (sc->gfp_mask); } -for_each_zone_nlist_nodemask (zone, &t, nodelist, +for_each_node_nlist_nodemask (node, &t, nodelist, sc->reclaim_idx, sc->nodemask) { + + pgdat =NODE_DATA (node); +/* * Take care memory controller reclaiming has small influence * to global LRU. */ if (!cgroup_reclaim (sc)) { -if (!cpuset_zone_allowed (zone, +if (!cpuset_node_allowed (node,GFP_KERNEL | __GFP_HARDWALL))continue ;
@@ -2960,18 +2982,7 @@ static void shrink_zones(struct nodelist *nodelist, struct scan_control *sc)*/ if (IS_ENABLED (CONFIG_COMPACTION ) && sc->order >PAGE_ALLOC_COSTLY_ORDER && -compaction_ready (zone, sc)) { - sc->compaction_ready = true; -continue ; - } - -/* -* Shrink each node in the zonelist once. If the -* zonelist is ordered by zone (not the default) then a -* node may be shrunk multiple times but in that case -* the user prefers lower zones being preserved. -*/ -if (zone->zone_pgdat == last_pgdat) +node_compaction_ready (&t, sc))continue ;/*
@@ -2981,7 +2992,7 @@ static void shrink_zones(struct nodelist *nodelist, struct scan_control *sc)* and balancing, not for a memcg's limit. */ nr_soft_scanned =0 ; - nr_soft_reclaimed =mem_cgroup_soft_limit_reclaim (zone->zone_pgdat, + nr_soft_reclaimed =mem_cgroup_soft_limit_reclaim (pgdat, sc->order, sc->gfp_mask, &nr_soft_scanned); sc->nr_reclaimed += nr_soft_reclaimed;
@@ -2989,11 +3000,7 @@ static void shrink_zones(struct nodelist *nodelist, struct scan_control *sc)/* need some check for avoid more shrink_zone() */ } -/* See comment about same check for global reclaim above */ -if (zone->zone_pgdat == last_pgdat) -continue ; - last_pgdat = zone->zone_pgdat; -shrink_node (zone->zone_pgdat, sc); +shrink_node (pgdat, sc); }/*
Signature · 1 line
2.23.0
RFC 10/19 mm, page_alloc: use for_each_node in wake_all_kswapds()
In reply to Pengfei LiIn wake_all_kswapds(), we want to traverse node instead of zone, so use for_each_node instead of for_each_zone.
--- mm/page_alloc.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
mm/page_alloc.c +4 -7
@@ -4166,15 +4166,12 @@ static void wake_all_kswapds(unsigned int order, gfp_t gfp_mask, const struct alloc_context *ac) {enum zone_type high_zoneidx = ac->high_zoneidx; -pg_data_t *last_pgdat =NULL ;struct nlist_traverser t; -struct zone *zone; +int node; -for_each_zone_nlist_nodemask (zone, &t, ac->nodelist, high_zoneidx, - ac->nodemask) { -if (last_pgdat != zone->zone_pgdat) -wakeup_kswapd (zone->zone_pgdat, gfp_mask, order, high_zoneidx); - last_pgdat = zone->zone_pgdat; +for_each_node_nlist_nodemask (node, &t, ac->nodelist, + high_zoneidx, ac->nodemask) { +wakeup_kswapd (NODE_DATA (node), gfp_mask, order, high_zoneidx); } }
Signature · 1 line
2.23.0
RFC 11/19 mm, mempolicy: use first_node in mempolicy_slab_node()
In reply to Pengfei LiIn mempolicy_slab_node(), we want to access first node instead of first zone, so use first_node_nlist instead of first_zone_nlist.
--- mm/mempolicy.c | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-)
mm/mempolicy.c +9 -9
@@ -1855,8 +1855,10 @@ static unsigned interleave_nodes(struct mempolicy *policy)*/ unsigned int mempolicy_slab_node (void ) { -struct mempolicy *policy;int node =numa_mem_id (); +struct mempolicy *policy; +struct nodelist *nodelist; +int first_node;if (in_interrupt ())return node;
@@ -1876,18 +1878,16 @@ unsigned int mempolicy_slab_node(void)return interleave_nodes (policy);case MPOL_BIND : { -struct zone *zone; -/* * Follow bind policy behavior and start allocation at the * first node. */ -struct nodelist *nodelist; -enum zone_type highest_zoneidx =gfp_zone (GFP_KERNEL ); - nodelist = &NODE_DATA (node)->node_nodelists[NODELIST_FALLBACK ]; - zone =first_zone_nlist_nodemask (nodelist, highest_zoneidx, - &policy->v.nodes); -return zone ?zone_to_nid (zone) : node; + nodelist =node_nodelist (node,GFP_KERNEL ); + + first_node =first_node_nlist_nodemask (nodelist, +gfp_zone (GFP_KERNEL ), &policy->v.nodes); + +return (first_node !=NUMA_NO_NODE ) ? first_node : node; }default :
Signature · 1 line
2.23.0
RFC 12/19 mm, mempolicy: use first_node in mpol_misplaced()
In reply to Pengfei LiIn mpol_misplaced(), we want to access first node instead of first zone, so use first_node_nlist instead of first_zone_nlist.
--- mm/mempolicy.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-)
mm/mempolicy.c +1 -3
@@ -2404,7 +2404,6 @@ static void sp_free(struct sp_node *n)int mpol_misplaced (struct page *page,struct vm_area_struct *vma,unsigned long addr) {struct mempolicy *pol; -struct zone *zone;int curnid =page_to_nid (page);unsigned long pgoff;int thiscpu =raw_smp_processor_id ();
@@ -2440,11 +2439,10 @@ int mpol_misplaced(struct page *page, struct vm_area_struct *vma, unsigned long*/ if (node_isset (curnid, pol->v.nodes)) goto out; - zone =first_zone_nlist_nodemask ( + polnid =first_node_nlist_nodemask (node_nodelist (numa_node_id (),GFP_HIGHUSER ),gfp_zone (GFP_HIGHUSER ), &pol->v.nodes); - polnid =zone_to_nid (zone);break ;default :
Signature · 1 line
2.23.0
RFC 13/19 mm, page_alloc: use first_node in local_memory_node()
In reply to Pengfei LiIn local_memory_node(), we want to access first node instead of first zone, so use first_node_nlist instead of first_zone_nlist.
--- mm/page_alloc.c | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-)
mm/page_alloc.c +2 -6
@@ -5673,12 +5673,8 @@ static void build_zonelists(pg_data_t *pgdat)*/ int local_memory_node (int node) { -struct zone *zone; - - zone =first_zone_nlist_nodemask (node_nodelist (node,GFP_KERNEL ), -gfp_zone (GFP_KERNEL ), -NULL ); -return zone_to_nid (zone); +return first_node_nlist_nodemask (node_nodelist (node,GFP_KERNEL ), +gfp_zone (GFP_KERNEL ),NULL ); }#endif
Signature · 1 line
2.23.0
RFC 14/19 mm, compaction: rename compaction_zonelist_suitable
In reply to Pengfei LiThis is a cleanup patch. Rename compaction_zonelist_suitable to compaction_nodelist_suitable.
--- include/linux/compaction.h | 2 +- mm/compaction.c | 2 +- mm/page_alloc.c | 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
include/linux/compaction.h +1 -1
@@ -177,7 +177,7 @@ static inline bool compaction_withdrawn(enum compact_result result) } -bool compaction_zonelist_suitable (struct alloc_context *ac,int order, +bool compaction_nodelist_suitable (struct alloc_context *ac,int order,int alloc_flags);extern int kcompactd_run (int nid);
mm/compaction.c +1 -1
@@ -2036,7 +2036,7 @@ enum compact_result compaction_suitable(struct zone *zone, int order,return ret; } -bool compaction_zonelist_suitable (struct alloc_context *ac,int order, +bool compaction_nodelist_suitable (struct alloc_context *ac,int order,int alloc_flags) {struct nlist_traverser t;
mm/page_alloc.c +1 -1
@@ -3967,7 +3967,7 @@ should_compact_retry(struct alloc_context *ac, int order, int alloc_flags,* to work with, so we retry only if it looks like reclaim can help. */ if (compaction_needs_reclaim (compact_result)) { - ret =compaction_zonelist_suitable (ac, order, alloc_flags); + ret =compaction_nodelist_suitable (ac, order, alloc_flags); goto out; }
Signature · 1 line
2.23.0
RFC 15/19 mm, mm_init: rename mminit_verify_zonelist
In reply to Pengfei LiThis is a cleanup patch. Rename mminit_verify_zonelist to mminit_verify_nodelist.
--- mm/internal.h | 4 ++-- mm/mm_init.c | 2 +- mm/page_alloc.c | 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
mm/internal.h +2 -2
@@ -443,7 +443,7 @@ do { \ } while (0) extern void mminit_verify_pageflags_layout(void); -extern void mminit_verify_zonelist (void ); +extern void mminit_verify_nodelist (void );#else static inline void mminit_dprintk(enum mminit_level level,
@@ -455,7 +455,7 @@ static inline void mminit_verify_pageflags_layout(void) { } -static inline void mminit_verify_zonelist (void ) +static inline void mminit_verify_nodelist (void ) { }#endif /* CONFIG_DEBUG_MEMORY_INIT */
mm/mm_init.c +1 -1
@@ -59,7 +59,7 @@ void __init mminit_print_nodelist(int nid, int nl_type) }/* The zonelists are simply reported, validation is manual. */ -void __init mminit_verify_zonelist (void ) +void __init mminit_verify_nodelist (void ) {int nid;
mm/page_alloc.c +1 -1
@@ -5803,7 +5803,7 @@ build_all_zonelists_init(void)for_each_possible_cpu (cpu)setup_pageset (&per_cpu (boot_pageset, cpu),0 ); -mminit_verify_zonelist (); +mminit_verify_nodelist ();cpuset_init_current_mems_allowed (); }
Signature · 1 line
2.23.0
RFC 16/19 mm, page_alloc: cleanup build_zonelists
In reply to Pengfei LiThis is a cleanup patch.
Renaming some functions and variables when initializing nodelist is xxx_nodelist instead of xxx_zonelist.
--- include/linux/mmzone.h | 2 +- init/main.c | 2 +- mm/memory_hotplug.c | 6 +++--- mm/page_alloc.c | 20 ++++++++++---------- 4 files changed, 15 insertions(+), 15 deletions(-)
include/linux/mmzone.h +1 -1
@@ -813,7 +813,7 @@ static inline bool pgdat_is_empty(pg_data_t *pgdat)#include <linux/memory_hotplug.h> -void build_all_zonelists (pg_data_t *pgdat); +void build_all_nodelists (pg_data_t *pgdat);void wakeup_kswapd (pg_data_t *pgdat,gfp_t gfp_mask,int order,enum zone_type classzone_idx); bool __zone_watermark_ok(struct zone *z, unsigned int order, unsigned long mark,
init/main.c +1 -1
@@ -602,7 +602,7 @@ asmlinkage __visible void __init start_kernel(void)smp_prepare_boot_cpu ();/* arch-specific boot-cpu hooks */ boot_cpu_hotplug_init (); -build_all_zonelists (NULL ); +build_all_nodelists (NULL );page_alloc_init ();pr_notice ("Kernel command line: %s\n" , boot_command_line);
mm/memory_hotplug.c +3 -3
@@ -821,7 +821,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typnode_states_set_node (nid, &arg);if (need_zonelists_rebuild) -build_all_zonelists (NULL ); +build_all_nodelists (NULL );else zone_pcp_update (zone);
@@ -904,7 +904,7 @@ static pg_data_t __ref *hotadd_new_pgdat(int nid, u64 start)* The node we allocated has no zone fallback lists. For avoiding * to access not-initialized zonelist, build here. */ -build_all_zonelists (pgdat); +build_all_nodelists (pgdat);/* * When memory is hot-added, all the memory is in offline state. So
@@ -1565,7 +1565,7 @@ static int __ref __offline_pages(unsigned long start_pfn,if (!populated_zone (zone)) {zone_pcp_reset (zone); -build_all_zonelists (NULL ); +build_all_nodelists (NULL ); }else zone_pcp_update (zone);
mm/page_alloc.c +10 -10
@@ -5631,7 +5631,7 @@ static void build_thisnode_nodelists(pg_data_t *pgdat)* may still exist in local DMA zone. */ -static void build_zonelists (pg_data_t *pgdat) +static void build_nodelists (pg_data_t *pgdat) {static int node_order[MAX_NUMNODES ];int node, load, nr_nodes =0 ;
@@ -5682,7 +5682,7 @@ static void setup_min_unmapped_ratio(void);static void setup_min_slab_ratio (void );#else /* CONFIG_NUMA */ -static void build_zonelists (pg_data_t *pgdat) +static void build_nodelists (pg_data_t *pgdat) {int node, local_node;struct nlist_entry *entry;
@@ -5737,7 +5737,7 @@ static void setup_pageset(struct per_cpu_pageset *p, unsigned long batch); static DEFINE_PER_CPU(struct per_cpu_pageset, boot_pageset); static DEFINE_PER_CPU(struct per_cpu_nodestat, boot_nodestats); -static void __build_all_zonelists (void *data) +static void __build_all_nodelists (void *data) {int nid;int __maybe_unused cpu;
@@ -5755,12 +5755,12 @@ static void __build_all_zonelists(void *data)* building zonelists is fine - no need to touch other nodes. */ if (self && !node_online (self->node_id)) { -build_zonelists (self); +build_nodelists (self); }else {for_each_online_node (nid) {pg_data_t *pgdat =NODE_DATA (nid); -build_zonelists (pgdat); +build_nodelists (pgdat); }#ifdef CONFIG_HAVE_MEMORYLESS_NODES
@@ -5781,11 +5781,11 @@ static void __build_all_zonelists(void *data) } static noinline void __init -build_all_zonelists_init (void ) +build_all_nodelists_init (void ) {int cpu; -__build_all_zonelists (NULL ); +__build_all_nodelists (NULL );/* * Initialize the boot_pagesets that are going to be used
@@ -5813,12 +5813,12 @@ build_all_zonelists_init(void)* __ref due to call of __init annotated helper build_all_zonelists_init * [protected by SYSTEM_BOOTING]. */ -void __ref build_all_zonelists (pg_data_t *pgdat) +void __ref build_all_nodelists (pg_data_t *pgdat) {if (system_state ==SYSTEM_BOOTING ) { -build_all_zonelists_init (); +build_all_nodelists_init (); }else { -__build_all_zonelists (pgdat); +__build_all_nodelists (pgdat);/* cpuset refresh routine should be here */ } vm_total_pages =nr_free_pagecache_pages ();
Signature · 1 line
2.23.0
RFC 17/19 mm, memory_hotplug: cleanup online_pages()
In reply to Pengfei LiThis is a cleanup patch.
In online_pages(), rename the local variable need_zonelists_rebuild to need_nodelists_rebuild.
--- mm/memory_hotplug.c | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
mm/memory_hotplug.c +5 -5
@@ -760,10 +760,10 @@ struct zone * zone_for_pfn_range(int online_type, int nid, unsigned start_pfn,int __ref online_pages (unsigned long pfn,unsigned long nr_pages,int online_type) { -unsigned long flags; +bool need_nodelists_rebuild = false;unsigned long onlined_pages =0 ; +unsigned long flags;struct zone *zone; -int need_zonelists_rebuild =0 ;int nid;int ret;struct memory_notify arg;
@@ -798,7 +798,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ* So, zonelist must be updated after online. */ if (!populated_zone (zone)) { - need_zonelists_rebuild =1 ; + need_nodelists_rebuild = true;setup_zone_pageset (zone); }
@@ -806,7 +806,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ online_pages_range);if (ret) {/* not a single memory resource was applicable */ -if (need_zonelists_rebuild) +if (need_nodelists_rebuild)zone_pcp_reset (zone); goto failed_addition; }
@@ -820,7 +820,7 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typshuffle_zone (zone);node_states_set_node (nid, &arg); -if (need_zonelists_rebuild) +if (need_nodelists_rebuild)build_all_nodelists (NULL );else zone_pcp_update (zone);
Signature · 1 line
2.23.0
RFC 18/19 kernel, sysctl: cleanup numa_zonelist_order
In reply to Pengfei LiThis is a cleanup patch.
Rename numa_zonelist_order to numa_nodelist_order, and clean up some zonelist_xxx accordingly.
--- include/linux/mmzone.h | 6 +++--- kernel/sysctl.c | 8 ++++---- mm/page_alloc.c | 16 ++++++++-------- 3 files changed, 15 insertions(+), 15 deletions(-)
include/linux/mmzone.h +3 -3
@@ -959,10 +959,10 @@ int sysctl_min_unmapped_ratio_sysctl_handler(struct ctl_table *, int,int sysctl_min_slab_ratio_sysctl_handler (struct ctl_table *,int ,void __user *,size_t *,loff_t *); -extern int numa_zonelist_order_handler (struct ctl_table *,int , +extern int numa_nodelist_order_handler (struct ctl_table *,int ,void __user *,size_t *,loff_t *); -extern char numa_zonelist_order[]; -#define NUMA_ZONELIST_ORDER_LEN 16 +extern char numa_nodelist_order[]; +#define NUMA_NODELIST_ORDER_LEN 16#ifndef CONFIG_NEED_MULTIPLE_NODES
kernel/sysctl.c +4 -4
@@ -1648,11 +1648,11 @@ static struct ctl_table vm_table[] = {#endif #ifdef CONFIG_NUMA { - .procname = "numa_zonelist_order", - .data = &numa_zonelist_order, - .maxlen = NUMA_ZONELIST_ORDER_LEN, + .procname = "numa_nodelist_order", + .data = &numa_nodelist_order, + .maxlen = NUMA_NODELIST_ORDER_LEN, .mode = 0644, - .proc_handler = numa_zonelist_order_handler, + .proc_handler = numa_nodelist_order_handler, }, #endif #if (defined(CONFIG_X86_32) && !defined(CONFIG_UML))|| \
mm/page_alloc.c +8 -8
@@ -5479,7 +5479,7 @@ static inline void set_last_nlist_entry(struct nlist_entry *entry)#ifdef CONFIG_NUMA -static int __parse_numa_zonelist_order (char *s) +static int __parse_numa_nodelist_order (char *s) {/* * We used to support different zonlists modes but they turned
@@ -5488,27 +5488,27 @@ static int __parse_numa_zonelist_order(char *s)* not fail it silently */ if (!(*s =='d' || *s =='D' || *s =='n' || *s =='N' )) { -pr_warn ("Ignoring unsupported numa_zonelist_order value: %s\n" , s); +pr_warn ("Ignoring unsupported numa_nodelist_order value: %s\n" , s);return -EINVAL ; }return 0 ; } -static __init int setup_numa_zonelist_order (char *s) +static __init int setup_numa_nodelist_order (char *s) {if (!s)return 0 ; -return __parse_numa_zonelist_order (s); +return __parse_numa_nodelist_order (s); } -early_param ("numa_zonelist_order" , setup_numa_zonelist_order); +early_param ("numa_nodelist_order" , setup_numa_nodelist_order); -char numa_zonelist_order[] ="Node" ; +char numa_nodelist_order[] ="Node" ;/* * sysctl handler for numa_zonelist_order */ -int numa_zonelist_order_handler (struct ctl_table *table,int write, +int numa_nodelist_order_handler (struct ctl_table *table,int write,void __user *buffer,size_t *length,loff_t *ppos) {
@@ -5521,7 +5521,7 @@ int numa_zonelist_order_handler(struct ctl_table *table, int write,if (IS_ERR (str))return PTR_ERR (str); - ret =__parse_numa_zonelist_order (str); + ret =__parse_numa_nodelist_order (str);kfree (str);return ret; }
Signature · 1 line
2.23.0
RFC 19/19 mm, mmzone: cleanup zonelist in comments
In reply to Pengfei LiThis is a cleanup patch.
Clean up all comments that contain the zonelist.
--- arch/hexagon/mm/init.c | 2 +- arch/ia64/include/asm/topology.h | 2 +- arch/x86/mm/numa.c | 2 +- include/linux/gfp.h | 8 +++--- include/linux/mmzone.h | 4 +-- kernel/cgroup/cpuset.c | 4 +-- mm/internal.h | 2 +- mm/memory_hotplug.c | 8 +++--- mm/mempolicy.c | 2 +- mm/mm_init.c | 2 +- mm/page_alloc.c | 45 ++++++++++++++++---------------- mm/vmscan.c | 2 +- 12 files changed, 41 insertions(+), 42 deletions(-)
arch/hexagon/mm/init.c +1 -1
@@ -103,7 +103,7 @@ void __init paging_init(void) zones_sizes[ZONE_NORMAL ] = max_low_pfn; -free_area_init (zones_sizes);/* sets up the zonelists and mem_map */ +free_area_init (zones_sizes);/* sets up the nodelists and mem_map */ /* * Start of high memory area. Will probably need something more
arch/ia64/include/asm/topology.h +1 -1
@@ -13,7 +13,7 @@#ifdef CONFIG_NUMA -/* Nodes w/o CPUs are preferred for memory allocations, see build_zonelists */ +/* Nodes w/o CPUs are preferred for memory allocations, see build_nodelists */ #define PENALTY_FOR_NODE_WITH_CPUS 255/*
arch/x86/mm/numa.c +1 -1
@@ -728,7 +728,7 @@ static void __init init_memory_less_node(int nid)free_area_init_node (nid, zones_size,0 , zholes_size);/* -* All zonelists will be built later in start_kernel() after per cpu +* All nodelists will be built later in start_kernel() after per cpu * areas are initialized. */ }
include/linux/gfp.h +4 -4
@@ -466,10 +466,10 @@ static inline int gfp_nodelist(gfp_t flags) }/* -* We get the zone list from the current node and the gfp_mask. -* This zone list contains a maximum of MAXNODES*MAX_NR_ZONES zones. -* There are two zonelists per node, one for all zones with memory and -* one containing just zones from the node the zonelist belongs to. +* We get the node list from the current node and the gfp_mask. +* This node list contains a maximum of MAXNODES nodes. +* There are two nodelists per node, one for all zones with memory and +* one containing just zones from the node the nodelist belongs to. * * For the normal case of non-DISCONTIGMEM systems the NODE_DATA() gets * optimized to &contig_page_data at compile-time.
include/linux/mmzone.h +2 -2
@@ -648,7 +648,7 @@ enum { NODELIST_FALLBACK, /* nodelist with fallback */#ifdef CONFIG_NUMA /* -* The NUMA zonelists are doubled because we need nodelists that +* The NUMA nodelists are doubled because we need nodelists that * restrict the allocations to a single node for __GFP_THISNODE. */ NODELIST_NOFALLBACK, /* nodelist without fallback (__GFP_THISNODE) */
@@ -657,7 +657,7 @@ enum { };/* -* This struct contains information about a node in a zonelist. It is stored +* This struct contains information about a node in a nodelist. It is stored * here to avoid dereferences into large structures and lookups of tables */ struct nlist_entry {
kernel/cgroup/cpuset.c +2 -2
@@ -3393,7 +3393,7 @@ static struct cpuset *nearest_hardwall_ancestor(struct cpuset *cs)* __alloc_pages() routine only calls here with __GFP_HARDWALL bit * _not_ set if it's a GFP_KERNEL allocation, and all nodes in the * current tasks mems_allowed came up empty on the first pass over -* the zonelist. So only GFP_KERNEL allocations, if all nodes in the +* the nodelist. So only GFP_KERNEL allocations, if all nodes in the * cpuset are short of memory, might require taking the callback_lock. * * The first call here from mm/page_alloc:get_page_from_freelist()
@@ -3467,7 +3467,7 @@ bool __cpuset_node_allowed(int node, gfp_t gfp_mask)* should not be possible for the following code to return an * offline node. But if it did, that would be ok, as this routine * is not returning the node where the allocation must be, only -* the node where the search should start. The zonelist passed to +* the node where the search should start. The nodelist passed to * __alloc_pages() will include all nodes. If the slab allocator * is passed an offline node, it will fall back to the local node. * See kmem_cache_alloc_node().
mm/internal.h +1 -1
@@ -105,7 +105,7 @@ extern pmd_t *mm_find_pmd(struct mm_struct *mm, unsigned long address);* nodemask, migratetype and high_zoneidx are initialized only once in * __alloc_pages_nodemask() and then never change. * -* zonelist, preferred_zone and classzone_idx are set first in +* nodelist, preferred_zone and classzone_idx are set first in * __alloc_pages_nodemask() for the fast path, and might be later changed * in __alloc_pages_slowpath(). All other functions pass the whole strucure * by a const pointer.
mm/memory_hotplug.c +4 -4
@@ -793,9 +793,9 @@ int __ref online_pages(unsigned long pfn, unsigned long nr_pages, int online_typ goto failed_addition;/* -* If this zone is not populated, then it is not in zonelist. +* If this zone is not populated, then it is not in nodelist. * This means the page allocator ignores this zone. -* So, zonelist must be updated after online. +* So, nodelist must be updated after online. */ if (!populated_zone (zone)) { need_nodelists_rebuild = true;
@@ -901,8 +901,8 @@ static pg_data_t __ref *hotadd_new_pgdat(int nid, u64 start)free_area_init_core_hotplug (nid);/* -* The node we allocated has no zone fallback lists. For avoiding -* to access not-initialized zonelist, build here. +* The node we allocated has no node fallback lists. For avoiding +* to access not-initialized nodelist, build here. */ build_all_nodelists (pgdat);
mm/mempolicy.c +1 -1
@@ -1950,7 +1950,7 @@ static inline unsigned interleave_nid(struct mempolicy *pol,* Returns a nid suitable for a huge page allocation and a pointer * to the struct mempolicy for conditional unref after allocation. * If the effective policy is 'BIND, returns a pointer to the mempolicy's -* @nodemask for filtering the zonelist. +* @nodemask for filtering the nodelist. * * Must be protected by read_mems_allowed_begin() */
mm/mm_init.c +1 -1
@@ -58,7 +58,7 @@ void __init mminit_print_nodelist(int nid, int nl_type) } } -/* The zonelists are simply reported, validation is manual. */ +/* The nodelists are simply reported, validation is manual. */ void __init mminit_verify_nodelist (void ) {int nid;
mm/page_alloc.c +22 -23
166 lines not on this page · raw message
mm/vmscan.c +1 -1
@@ -3360,7 +3360,7 @@ unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, .may_swap = may_swap, };/* -* Traverse the ZONELIST_FALLBACK zonelist of the current node to put +* Traverse the NODELIST_FALLBACK nodelist of the current node to put * equal pressure on all the nodes. This is based on the assumption that * the reclaim does not bail out early. */
Signature · 1 line
2.23.0
Michal Hocko
maintainerOn Thu 21-11-19 23:17:52, Pengfei Li wrote:
[...]
Quoted 2 lines
Since I don't currently have multiple node NUMA systems, I would be grateful if anyone would like to test this series of patches.
I didn't really get to think about the actual patchset. From a very quick glance I am wondering whether we need to optimize as there are usually only small amount of numa nodes. But I am quite busy so I cannot really do any claims.
Anyway, you can test this even without NUMA HW. Have a look at numa=fake option (numa_emu_cmdline). Or you can use kvm/qemu which provides easy ways to setup a NUMA capable virtual machine.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 9 lines · On Thu, 21 Nov 2019 19:04:01 +0100 Michal Hocko <[email protected]> wrote:
On Thu 21-11-19 23:17:52, Pengfei Li wrote:
[...]Quoted 2 lines
Since I don't currently have multiple node NUMA systems, I would be grateful if anyone would like to test this series of patches.
I didn't really get to think about the actual patchset. From a very quick glance I am wondering whether we need to optimize as there are usually only small amount of numa nodes. But I am quite busy so I cannot really do any claims.
Thanks for your comments.
I think it's time to modify the zonelist to nodelist because the zonelist is always in node order and the page reclamation is based on node.
I will do more performance testing to show that multi-node systems will benefit from this series of patches.
Quoted 3 lines
Anyway, you can test this even without NUMA HW. Have a look at numa=fake option (
numa_emu_cmdline). Or you can use kvm/qemu which provides easy ways to setup a NUMA capable virtual machine.
Thanks a lot. I will use the numa=fake option to do more performance testing.
Michal Hocko
maintainerQuoted 13 lines · Pengfei Li
Quoted 9 lines · On Thu, 21 Nov 2019 19:04:01 +0100 Michal Hocko <[email protected]> wrote:
On Thu 21-11-19 23:17:52, Pengfei Li wrote:
[...]Quoted 2 lines
Since I don't currently have multiple node NUMA systems, I would be grateful if anyone would like to test this series of patches.
I didn't really get to think about the actual patchset. From a very quick glance I am wondering whether we need to optimize as there are usually only small amount of numa nodes. But I am quite busy so I cannot really do any claims.
Quoted 8 lines
Thanks for your comments.
I think it's time to modify the zonelist to nodelist because the zonelist is always in node order and the page reclamation is based on node.
I will do more performance testing to show that multi-node systems will benefit from this series of patches.
Sensible performance numbers on multiple workloads (ideally some real world ones rather than artificial microbenchmarks) is essential for a performance optimization that is this large.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 21 lines · On Mon, 25 Nov 2019 09:40:58 +0100 Michal Hocko <[email protected]> wrote:
Quoted 20 lines · Pengfei Li
Quoted 10 lines · On Thu, 21 Nov 2019 19:04:01 +0100 Michal Hocko <[email protected]> wrote:
On Thu 21-11-19 23:17:52, Pengfei Li wrote:
[...]Quoted 3 lines
Since I don't currently have multiple node NUMA systems, I would be grateful if anyone would like to test this series of patches.
I didn't really get to think about the actual patchset. From a very quick glance I am wondering whether we need to optimize as there are usually only small amount of numa nodes. But I am quite busy so I cannot really do any claims.
Thanks for your comments.
I think it's time to modify the zonelist to nodelist because the zonelist is always in node order and the page reclamation is based on node.
Quoted 6 lines
Quoted 2 lines
I will do more performance testing to show that multi-node systems will benefit from this series of patches.
Sensible performance numbers on multiple workloads (ideally some real world ones rather than artificial microbenchmarks) is essential for a performance optimization that is this large.
Thank you for your suggestion.
But this is probably a bit difficult because I don't have a NUMA server to do real-world workload testing.
I will do as many performance benchmarks as possible, just like Mel Gorman's "Move LRU page reclaim from zones to nodes v9"
(https://lwn.net/Articles/694121/).
Signature · 1 line
Pengfei
Michal Hocko
maintainerQuoted 32 lines · Pengfei Li
Quoted 21 lines · On Mon, 25 Nov 2019 09:40:58 +0100 Michal Hocko <[email protected]> wrote:
Quoted 20 lines · Pengfei Li
Quoted 10 lines · On Thu, 21 Nov 2019 19:04:01 +0100 Michal Hocko <[email protected]> wrote:
On Thu 21-11-19 23:17:52, Pengfei Li wrote:
[...]Quoted 3 lines
Since I don't currently have multiple node NUMA systems, I would be grateful if anyone would like to test this series of patches.
I didn't really get to think about the actual patchset. From a very quick glance I am wondering whether we need to optimize as there are usually only small amount of numa nodes. But I am quite busy so I cannot really do any claims.
Thanks for your comments.
I think it's time to modify the zonelist to nodelist because the zonelist is always in node order and the page reclamation is based on node.
Quoted 6 lines
Quoted 2 lines
I will do more performance testing to show that multi-node systems will benefit from this series of patches.
Sensible performance numbers on multiple workloads (ideally some real world ones rather than artificial microbenchmarks) is essential for a performance optimization that is this large.
Quoted 4 lines
Thank you for your suggestion.
But this is probably a bit difficult because I don't have a NUMA server to do real-world workload testing.
For this particular feature you really do not need any real NUMA server. Your patch shouldn't introduce NUMA locality. All you are aiming for is to optimize the zone list iteration.
Quoted 3 lines
I will do as many performance benchmarks as possible, just like Mel Gorman's "Move LRU page reclaim from zones to nodes v9"
(https://lwn.net/Articles/694121/).
Be aware that this will be quite time consuming and non-trivial to process/evaluate. Not that I want to discourage you from this endeavor but it is always good to think whether your final goal really has a potential to a visible optimization. I might be wrong but only the page allocator should really be the hot path which iterates over zonelist so a microbenchmark targeting this path would be something I would start with. Unless there are some really nice results from there I would lose more time with other benchmarks.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 13 lines · On 2019-11-21 at 23:17 Pengfei Li wrote:
Motivation
----------
Currently if we want to iterate through all the nodes we have to traverse all the zones from the zonelist.So in order to reduce the number of loops required to traverse node, this series of patches modified the zonelist to nodelist.
Two new macros have been introduced:
1)for_each_node_nlist
2)for_each_node_nlist_nodemask
Quoted 5 lines
Benefit
-------
1. For a NUMA system with N nodes, each node has M zones, the number of loops is reduced from N*M times to N times when traversing node.
It looks to me that we don't really have system which has N nodes and each node with M zones in its address range.
We may have systems which has several nodes, but only the first node has all zone types, other nodes only have NORMAL zone. (Evenly distribute the !NORMAL zones on all nodes is not reasonable, as those zones have limited size)
So iterate over zones to reach nodes should at N level, not M*N level.
- Xinhai
Quoted 75 lines
2. The size of
pg_data_tis much reduced.Test Result
-----------
Currently I have only performed a simple page allocation benchmark test on my laptop, and the results show that the performance of a system with only one node is almost unaffected.Others
------
Next I will do more performance testing and add it to the Test Result.Since I don't currently have multiple node NUMA systems, I would be grateful if anyone would like to test this series of patches.
I am still not sure this series of patches is on the right way so I am sending this as an RFC.
Any comments are highly appreciated.
Pengfei Li (19):
mm, mmzone: modify zonelist to nodelist
mm, hugetlb: usefor_each_nodeindequeue_huge_page_nodemask()mm,oom_kill: usefor_each_nodeinconstrained_alloc()mm, slub: usefor_each_nodeinget_any_partial()
mm, slab: usefor_each_nodeinfallback_alloc()
mm, vmscan: usefor_each_nodeindo_try_to_free_pages()mm, vmscan: usefirst_nodeinthrottle_direct_reclaim()mm, vmscan: pass pgdat towakeup_kswapd()
mm, vmscan: usefor_each_nodeinshrink_zones()
mm,page_alloc: usefor_each_nodeinwake_all_kswapds()mm, mempolicy: usefirst_nodeinmempolicy_slab_node()mm, mempolicy: usefirst_nodeinmpol_misplaced()mm,page_alloc: usefirst_nodeinlocal_memory_node()mm, compaction: renamecompaction_zonelist_suitablemm,mm_init: renamemminit_verify_zonelist
mm,page_alloc: cleanupbuild_zonelists
mm,memory_hotplug: cleanuponline_pages()
kernel, sysctl: cleanupnuma_zonelist_order
mm, mmzone: cleanup zonelist in commentsarch/hexagon/mm/init.c | 2 +-
arch/ia64/include/asm/topology.h| 2 +- arch/x86/mm/numa.c | 2 +- drivers/tty/sysrq.c | 2 +- include/linux/compaction.h | 2 +- include/linux/gfp.h | 18 +- include/linux/mmzone.h | 249 +++++++++++++------------ include/linux/oom.h | 4 +- include/linux/swap.h | 2 +- include/trace/events/oom.h | 9 +-
init/main.c | 2 +- kernel/cgroup/cpuset.c | 4 +-
kernel/sysctl.c | 8 +-
mm/compaction.c | 20 +-
mm/hugetlb.c | 21 +--
mm/internal.h | 13 +-
mm/memcontrol.c | 2 +- mm/memory_hotplug.c | 24 +--
mm/mempolicy.c | 26 ++-
mm/mm_init.c | 74 +++++---
mm/mmzone.c | 30 ++-
mm/oom_kill.c | 16 +-
mm/page_alloc.c | 301 ++++++++++++++++--------------- mm/slab.c | 13 +-
mm/slub.c | 14 +-
mm/vmscan.c | 149 ++++++++-------26 files changed, 518 insertions(+), 491 deletions(-)--
2.23.0
Quoted 20 lines · On 22.11.19 08:25, [email protected] wrote:
Quoted 13 lines · On 2019-11-21 at 23:17 Pengfei Li wrote:
Motivation
----------
Currently if we want to iterate through all the nodes we have to traverse all the zones from the zonelist.So in order to reduce the number of loops required to traverse node, this series of patches modified the zonelist to nodelist.
Two new macros have been introduced:
1)for_each_node_nlist
2)for_each_node_nlist_nodemaskQuoted 5 lines
Benefit
-------
1. For a NUMA system with N nodes, each node has M zones, the number of loops is reduced from N*M times to N times when traversing node.
Quoted 7 lines
It looks to me that we don't really have system which has N nodes and each node with M zones in its address range.
We may have systems which has several nodes, but only the first node has all zone types, other nodes only have NORMAL zone. (Evenly distribute the !NORMAL zones on all nodes is not reasonable, as those zones have limited size)
So iterate over zones to reach nodes should at N level, not M*N level.
I guess NORMAL/MOVABLE/DEVICE would be common for most nodes, while I do agree that usually we will only have 1 or 2 zones per node (when we have many nodes). So it would be something like c*N, whereby c is most probably on average 2.
Signature · 3 lines
Thanks, David / dhildenb
Quoted 21 lines · On Fri, 22 Nov 2019 15:25:00 +0800 "[email protected]"…
Quoted 13 lines · On 2019-11-21 at 23:17 Pengfei Li wrote:
Motivation
----------
Currently if we want to iterate through all the nodes we have to traverse all the zones from the zonelist.So in order to reduce the number of loops required to traverse node, this series of patches modified the zonelist to nodelist.
Two new macros have been introduced:
1)for_each_node_nlist
2)for_each_node_nlist_nodemaskQuoted 6 lines
Benefit
-------
1. For a NUMA system with N nodes, each node has M zones, the number of loops is reduced from N*M times to N times when traversing node.
Quoted 8 lines
It looks to me that we don't really have system which has N nodes and each node with M zones in its address range.
We may have systems which has several nodes, but only the first node has all zone types, other nodes only have NORMAL zone. (Evenly distribute the !NORMAL zones on all nodes is not reasonable, as those zones have limited size)
So iterate over zones to reach nodes should at N level, not M*N level.
Thanks for your comments.
In the case you said, the number of loops required to traverse all nodes is similar to traversing all zones.
I have two main reasons to explain that this series of patches is beneficial.
1. When node has more than one zone, it will take fewer cycles to traverse all nodes. (for example, ZONE_MOVABLE?)
2. Using zonelist to traverse all nodes is inefficient, pgdat must be obtained indirectly via zone->zone_pgdat, and additional judgment must be made.
E.g
1) Using zonelist to traverse all nodes
last_pgdat = NULL;
for_each_zone_zonelist(zone, xxx) {
pgdat = zone->zone_pgdat;
if (pgdat == last_pgdat)
continue;
last_pgdat = pgdat;
do_something(pgdat);
}2) Using nodelist to traverse all nodes
for_each_node_nodelist(node, xxx) {
do_something(NODE_INFO(node));
}Quoted 35 lines · Pengfei Li
Quoted 21 lines · On Fri, 22 Nov 2019 15:25:00 +0800 "[email protected]"…
Quoted 13 lines · On 2019-11-21 at 23:17 Pengfei Li wrote:
Motivation
----------
Currently if we want to iterate through all the nodes we have to traverse all the zones from the zonelist.So in order to reduce the number of loops required to traverse node, this series of patches modified the zonelist to nodelist.
Two new macros have been introduced:
1)for_each_node_nlist
2)for_each_node_nlist_nodemaskQuoted 6 lines
Benefit
-------
1. For a NUMA system with N nodes, each node has M zones, the number of loops is reduced from N*M times to N times when traversing node.Quoted 8 lines
It looks to me that we don't really have system which has N nodes and each node with M zones in its address range.
We may have systems which has several nodes, but only the first node has all zone types, other nodes only have NORMAL zone. (Evenly distribute the !NORMAL zones on all nodes is not reasonable, as those zones have limited size)
So iterate over zones to reach nodes should at N level, not M*N level.Thanks for your comments.
Quoted 8 lines
In the case you said, the number of loops required to traverse all nodes is similar to traversing all zones.
I have two main reasons to explain that this series of patches is beneficial.
1. When node has more than one zone, it will take fewer cycles to traverse all nodes. (for example,
ZONE_MOVABLE?)
ZONE_MOVABLE is broken for ages (non-movable allocations are there all the time last time I tried) which indicates there is very few people care about it, so it is rather weak to use that as a justification for the churns it might cause.
Quoted 24 lines
2. Using zonelist to traverse all nodes is inefficient, pgdat must be obtained indirectly via
zone->zone_pgdat, and additional judgment must be made.E.g
1) Using zonelist to traverse all nodeslast_pgdat = NULL; for_each_zone_zonelist(zone, xxx) { pgdat = zone->zone_pgdat; if (pgdat == last_pgdat) continue; last_pgdat = pgdat; do_something(pgdat); }2) Using nodelist to traverse all nodes
for_each_node_nodelist(node, xxx) { do_something(NODE_INFO(node)); }
Quoted 43 lines · On Fri, 22 Nov 2019 10:53:22 -0500 Qian Cai <[email protected]> wrote:
Quoted 42 lines · Pengfei Li
Quoted 21 lines · On Fri, 22 Nov 2019 15:25:00 +0800 "[email protected]"…
Quoted 13 lines · On 2019-11-21 at 23:17 Pengfei Li wrote:
Motivation
----------
Currently if we want to iterate through all the nodes we have to traverse all the zones from the zonelist.So in order to reduce the number of loops required to traverse node, this series of patches modified the zonelist to nodelist.
Two new macros have been introduced:
1)for_each_node_nlist
2)for_each_node_nlist_nodemaskQuoted 6 lines
Benefit
-------
1. For a NUMA system with N nodes, each node has M zones, the number of loops is reduced from N*M times to N times when traversing node.Quoted 9 lines
It looks to me that we don't really have system which has N nodes and each node with M zones in its address range.
We may have systems which has several nodes, but only the first node has all zone types, other nodes only have NORMAL zone. (Evenly distribute the !NORMAL zones on all nodes is not reasonable, as those zones have limited size)
So iterate over zones to reach nodes should at N level, not M*N level.Thanks for your comments.
In the case you said, the number of loops required to traverse all nodes is similar to traversing all zones.
I have two main reasons to explain that this series of patches is beneficial.
Quoted 8 lines
Quoted 2 lines
1. When node has more than one zone, it will take fewer cycles to traverse all nodes. (for example,
ZONE_MOVABLE?)
ZONE_MOVABLEis broken for ages (non-movable allocations are there all the time last time I tried) which indicates there is very few people care about it, so it is rather weak to use that as a justification for the churns it might cause.
Thanks for your comments.
Yes, if node has only NORMAL ZONE, then the zonelist is actually a nodelist. This series of patches really only benefits nodes with more than one zone.
Quoted 24 lines
Quoted 24 lines
2. Using zonelist to traverse all nodes is inefficient, pgdat must be obtained indirectly via
zone->zone_pgdat, and additional judgment must be made.E.g
1) Using zonelist to traverse all nodeslast_pgdat = NULL; for_each_zone_zonelist(zone, xxx) { pgdat = zone->zone_pgdat; if (pgdat == last_pgdat) continue; last_pgdat = pgdat; do_something(pgdat); }2) Using nodelist to traverse all nodes
for_each_node_nodelist(node, xxx) { do_something(NODE_INFO(node)); }
Signature · 1 line
Pengfei
Michal Hocko
maintainerOn Fri 22-11-19 10:53:22, Qian Cai wrote:
[...]
Quoted 3 lines
ZONE_MOVABLEis broken for ages (non-movable allocations are there all the time last time I tried) which indicates there is very few people care about it, so it is rather weak to use that as a justification for the churns it might cause.
People do care about ZONE_MOVABLE and if there is a non-movable memory sitting there then it is a bug. Please report that.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 6 lines · Michal Hocko
People do care about
ZONE_MOVABLEand if there is a non-movable memory sitting there then it is a bug. Please report that.
It is trivial to test yourself if you ever care. Just pass kernelcore= to as many NUMA machines you could find, an then test if ever possible to offline those memory.
Michal Hocko
maintainerQuoted 10 lines · Qian Cai
Quoted 6 lines · Michal Hocko
People do care about
ZONE_MOVABLEand if there is a non-movable memory sitting there then it is a bug. Please report that.It is trivial to test yourself if you ever care. Just pass kernelcore= to as many NUMA machines you could find, an then test if ever possible to offline those memory.
I definitely do care if you can provide more details (ideally in a separate email thread). I am using movable memory for memory hotplug usecases and so far I do not remember any kernel/non-movable allocations would make it in - modulo bugs when somebody might use __GFP_MOVABLE when it is not appropriate.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 16 lines · Michal Hocko
Quoted 10 lines · Qian Cai
Quoted 6 lines · Michal Hocko
People do care about
ZONE_MOVABLEand if there is a non-movable memory sitting there then it is a bug. Please report that.It is trivial to test yourself if you ever care. Just pass kernelcore= to as many NUMA machines you could find, an then test if ever possible to offline those memory.
Quoted 6 lines
I definitely do care if you can provide more details (ideally in a separate email thread). I am using movable memory for memory hotplug usecases and so far I do not remember any
kernel/non-movableallocations would make it in - modulo bugs when somebody might use__GFP_MOVABLEwhen it is not appropriate.
I don’t think it is anything to do with __GFP_MOVABLE. It is about booting a kernel with either passing kernelcore= or movablecore=. Then, those ZONE_MOVABLE will have non-movable pages which looks like those from vmemmap_populate(). How do you create ZONE_MOVALBLE in this first place?
Michal Hocko
maintainerQuoted 23 lines · Qian Cai
Quoted 16 lines · Michal Hocko
Quoted 10 lines · Qian Cai
Quoted 6 lines · Michal Hocko
People do care about
ZONE_MOVABLEand if there is a non-movable memory sitting there then it is a bug. Please report that.It is trivial to test yourself if you ever care. Just pass kernelcore= to as many NUMA machines you could find, an then test if ever possible to offline those memory.
Quoted 6 lines
I definitely do care if you can provide more details (ideally in a separate email thread). I am using movable memory for memory hotplug usecases and so far I do not remember any
kernel/non-movableallocations would make it in - modulo bugs when somebody might use__GFP_MOVABLEwhen it is not appropriate.
Quoted 3 lines
I don’t think it is anything to do with
__GFP_MOVABLE. It is about booting a kernel with either passing kernelcore= or movablecore=. Then, thoseZONE_MOVABLEwill have non-movable pages which looks like those fromvmemmap_populate().
OK, I see. This looks like a bug in kernelcore/movablecore. And honestly I wouldn't be surprised because these are hacks that should have been removed. I have even attempted to do that because their main usecase is mostly gone. There were some objections though...
Quoted 1 line
How do you create
ZONE_MOVALBLEin this first place?
The most common usecase I work with is to use movable_node parameter which makes whole nodes marked as hotplugable to be moved to ZONE_MOVABLE.
Signature · 2 lines
Michal Hocko SUSE Labs
Quoted 18 lines · Pengfei Li
Motivation
----------
Currently if we want to iterate through all the nodes we have to traverse all the zones from the zonelist.So in order to reduce the number of loops required to traverse node, this series of patches modified the zonelist to nodelist.
Two new macros have been introduced:
1)for_each_node_nlist
2)for_each_node_nlist_nodemaskBenefit
-------
1. For a NUMA system with N nodes, each node has M zones, the number of loops is reduced from N*M times to N times when traversing node.
Quoted 9 lines
2. The size of
pg_data_tis much reduced.Test Result
-----------
Currently I have only performed a simple page allocation benchmark test on my laptop, and the results show that the performance of a system with only one node is almost unaffected.
So you are seeing no performance changes. I am wondering why do we need this, then - because your motivation sounds like a performance improvement? (not completely against this, just trying to understand the value of this :) )
Signature · 3 lines
Thanks, David / dhildenb
Quoted 30 lines · On Fri, 22 Nov 2019 11:03:30 +0100 David Hildenbrand <[email protected]> wrote:
Quoted 19 lines · Pengfei Li
Motivation
----------
Currently if we want to iterate through all the nodes we have to traverse all the zones from the zonelist.So in order to reduce the number of loops required to traverse node, this series of patches modified the zonelist to nodelist.
Two new macros have been introduced:
1)for_each_node_nlist
2)for_each_node_nlist_nodemaskBenefit
-------
1. For a NUMA system with N nodes, each node has M zones, the number of loops is reduced from N*M times to N times when traversing
node.Quoted 9 lines
2. The size of
pg_data_tis much reduced.Test Result
-----------
Currently I have only performed a simple page allocation benchmark test on my laptop, and the results show that the performance of a system with only one node is almost unaffected.
Quoted 4 lines
So you are seeing no performance changes. I am wondering why do we need this, then - because your motivation sounds like a performance improvement? (not completely against this, just trying to understand the value of this :) )
Thanks for your comments.
I am sorry that I did not make it clear. I want to express this series of patches will benefit NUMA systems with multiple nodes.
The main benefit is that it will be more efficient when traversing all nodes (for example when performing page reclamation).
Quoted 2 lines · Pengfei Li
I am sorry that I did not make it clear. I want to express this series of patches will benefit NUMA systems with multiple nodes.
Ok but that benefit needs to be quantified somehow.
Quoted 2 lines
The main benefit is that it will be more efficient when traversing all nodes (for example when performing page reclamation).
And you loose the prioritization of allocations through these different zones. We create zonelists with a certain sequence of the zones in order to prefer allocations from certain zones. This is in particular important for the smaller DMA zones which may be easily exhausted.
Quoted 7 lines · Christopher Lameter
Quoted 2 lines · Pengfei Li
I am sorry that I did not make it clear. I want to express this series of patches will benefit NUMA systems with multiple nodes.
Ok but that benefit needs to be quantified somehow.
Quoted 8 lines
Quoted 2 lines
The main benefit is that it will be more efficient when traversing all nodes (for example when performing page reclamation).
And you loose the prioritization of allocations through these different zones. We create zonelists with a certain sequence of the zones in order to prefer allocations from certain zones. This is in particular important for the smaller DMA zones which may be easily exhausted.
That's an important point also when MOVABLE is in place. You really don't want to go to random other zones before considering all MOVABLE zones. Same with DMA and NORMAL.
Signature · 3 lines
Thanks, David / dhildenb
Quoted 7 lines · On Fri, 22 Nov 2019 15:53:57 +0000 (UTC) Christopher Lameter <[email protected]>…
Quoted 2 lines · Pengfei Li
I am sorry that I did not make it clear. I want to express this series of patches will benefit NUMA systems with multiple nodes.
Ok but that benefit needs to be quantified somehow.
Thanks for your comments.
Yes, I will add detailed performance test data in v2.
Quoted 5 lines
Quoted 2 lines
The main benefit is that it will be more efficient when traversing all nodes (for example when performing page reclamation).
And you loose the prioritization of allocations through these different zones.
Sorry, I forgot to mention that the information about the zones that are available to the node is still there.
The old for_each_zone_zonelist has been replaced with for_each_zone_nodelist.
I will add some key implementation details in v2.
Quoted 5 lines
We create zonelists with a certain sequence of the
zones in order to prefer allocations from certain zones. This is in particular important for the smaller DMA zones which may be easily exhausted.
I'm not sure if I understand what you mean, but since commit c9bff3eebc09 ("mm, page_alloc: rip out ZONELIST_ORDER_ZONE"), the zonelist is always in "Node" order, so building the nodelist is fine.
Signature · 1 line
Pengfei
Quoted 3 lines · Pengfei Li
I'm not sure if I understand what you mean, but since commit
c9bff3eebc09("mm,page_alloc: rip outZONELIST_ORDER_ZONE"), the zonelist is always in "Node" order, so building the nodelist is fine.
Sounds good. Just wonder how the allocations that are constrained to certain physical addresses (16M via DMA and 4G via DMA32) will work in that case.