Cleanup and optimise the page allocator V3
Subsystems (19)
cover Cleanup and optimise the page allocator V3
Here is V3 of an attempt to cleanup and optimise the page allocator and should be ready for general testing. The page allocator is now faster (16% reduced time overall for kernbench on one machine) and it has a smaller cache footprint (16.5% less L1 cache misses and 19.5% less L2 cache misses for kernbench on one machine). The text footprint has unfortunately increased, largely due to the introduction of a form of lazy buddy merging mechanism that avoids cache misses by postponing buddy merging until a high-order allocation needs it.
I tested the patchset with kernbench, hackbench, sysbench-postgres and netperf UDP and TCP with a variety of sizes. Many machines and loads showed improved performance *however* it was not universal. On some machines, one load would be faster and another slower (perversely, sometimes netperf-UDP would be faster with netperf-TCP slower). On an different machines, the workloads that gained or lost would differ. I haven't fully pinned down why this is yet but I have observed on at least one machine lock contention is higher and more time is spent in functions like rb_erase(), both which might imply some sort of scheduling artifact. I've also noted that while the allocator incurs fewer cache misses, sometimes cache misses overall are increased for the workload but the increased lock contention might account for this.
In some cases, more time is spent in copy_user_generic_string()[1] which might imply that strings are getting the same colour with the greater effort spent giving back hot pages but theories as to why this is not a universal effect are welcome. I've also noted that machines with many CPUs with different caches suffer because struct page is not cache-aligned but aligning it hurts other machines so I left it alone. Finally, the performance characteristics are vary depending on if you use SLAB, SLUB or SLQB.
So, while the page allocator is faster in most cases, making all workloads universally go faster needs to now look at other areas like the sl*b allocator and the scheduler.
Here is the patchset as it stands and I think it's ready for wider testing and to be considered for merging depending on the outcome of testing and reviews.
[1] copy_user_generic_unrolled on one machine was slowed down by an extreme amount. I did not check if there was a pattern of slowdowns versus which version of copy_user_generic() was used
Changes since V2
o Remove brances by treating watermark flags as array indices
o Remove branch by assuming __GFP_HIGH == ALLOC_HIGH
o Do not check for compound on every page free
o Remove branch by always ensuring the migratetype is known on free
o Simplify buffered_rmqueue further
o Reintroduce improved version of batched bulk free of pcp pages
o Use allocation flags as an index to zone watermarks
o Work out __GFP_COLD only once
o Reduce the number of times zone stats are updated
o Do not dump reserve pages back into the allocator. Instead treat them as MOVABLE so that MIGRATE_RESERVE gets used on the max-order-overlapped boundaries without causing trouble
o Allow pages up to PAGE_ALLOC_COSTLY_ORDER to use the per-cpu allocator. order-1 allocations are frequently enough in particular to justify this
o Rearrange inlining such that the hot-path is inlined but not in a way that increases the text size of the page allocator
o Make the check for needing additional zonelist filtering due to NUMA or cpusets as light as possible
o Do not destroy compound pages going to the PCP lists
o Delay the merging of buddies until a high-order allocation needs them or anti-fragmentation is being forced to fallback
o Count high-order pages as 1
Changes since V1
o Remove the ifdef CONFIG_CPUSETS from inside get_page_from_freelist()
o Use non-lock bit operations for clearing the mlock flag
o Factor out alloc_flags calculation so it is only done once (Peter)
o Make gfp.h a bit prettier and clear-cut (Peter)
o Instead of deleting a debugging check, replace page_count() in the free path with a version that does not check for compound pages (Nick)
o Drop the alteration for hot/cold page freeing until we know if it helps or not
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 1/35 Replace __alloc_pages_internal() with __alloc_pages_nodemask()
__alloc_pages_internal is the core page allocator function but essentially it is an alias of __alloc_pages_nodemask. Naming a publicly available and exported function "internal" is also a big ugly. This patch renames __alloc_pages_internal() to __alloc_pages_nodemask() and deletes the old nodemask function.
Warning - This patch renames an exported symbol. No kernel driver is affected by external drivers calling __alloc_pages_internal() should change the call to __alloc_pages_nodemask() without any alteration of parameters.
--- include/linux/gfp.h | 12 ++---------- mm/page_alloc.c | 4 ++-- 2 files changed, 4 insertions(+), 12 deletions(-)
include/linux/gfp.h +2 -10
@@ -168,24 +168,16 @@ static inline void arch_alloc_page(struct page *page, int order) { }#endif struct page * -__alloc_pages_internal (gfp_t gfp_mask,unsigned int order, +__alloc_pages_nodemask (gfp_t gfp_mask,unsigned int order,struct zonelist *zonelist,nodemask_t *nodemask);static inline struct page *__alloc_pages (gfp_t gfp_mask,unsigned int order,struct zonelist *zonelist) { -return __alloc_pages_internal (gfp_mask, order, zonelist,NULL ); +return __alloc_pages_nodemask (gfp_mask, order, zonelist,NULL ); } -static inline struct page * -__alloc_pages_nodemask (gfp_t gfp_mask,unsigned int order, -struct zonelist *zonelist,nodemask_t *nodemask) -{ -return __alloc_pages_internal (gfp_mask, order, zonelist, nodemask); -} - -static inline struct page *alloc_pages_node (int nid,gfp_t gfp_mask,unsigned int order) {
mm/page_alloc.c +2 -2
@@ -1464,7 +1464,7 @@ try_next_zone:* This is the 'heart' of the zoned buddy allocator. */ struct page * -__alloc_pages_internal (gfp_t gfp_mask,unsigned int order, +__alloc_pages_nodemask (gfp_t gfp_mask,unsigned int order,struct zonelist *zonelist,nodemask_t *nodemask) {const gfp_t wait = gfp_mask & __GFP_WAIT;
@@ -1670,7 +1670,7 @@ nopage: got_pg:return page; } -EXPORT_SYMBOL (__alloc_pages_internal); +EXPORT_SYMBOL (__alloc_pages_nodemask);/* * Common helper functions.
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 01/35] Replace __alloc_pages_internal() with __alloc_pages_nodemask()
PATCH 2/35 Do not sanity check order in the fast path
In reply to Mel GormanNo user of the allocator API should be passing in an order >= MAX_ORDER but we check for it on each and every allocation. Delete this check and make it a VM_BUG_ON check further down the call path.
--- include/linux/gfp.h | 6 ------ mm/page_alloc.c | 2 ++ 2 files changed, 2 insertions(+), 6 deletions(-)
include/linux/gfp.h +0 -6
@@ -181,9 +181,6 @@ __alloc_pages(gfp_t gfp_mask, unsigned int order,static inline struct page *alloc_pages_node (int nid,gfp_t gfp_mask,unsigned int order) { -if (unlikely (order >=MAX_ORDER )) -return NULL ; -/* Unknown node is current node */ if (nid <0 ) nid =numa_node_id ();
@@ -197,9 +194,6 @@ extern struct page *alloc_pages_current(gfp_t gfp_mask, unsigned order);static inline struct page *alloc_pages (gfp_t gfp_mask,unsigned int order) { -if (unlikely (order >=MAX_ORDER )) -return NULL ; -return alloc_pages_current (gfp_mask, order); } extern struct page *alloc_page_vma(gfp_t gfp_mask,
mm/page_alloc.c +2 -0
@@ -1407,6 +1407,8 @@ get_page_from_freelist(gfp_t gfp_mask, nodemask_t *nodemask, unsigned int order, classzone_idx =zone_idx (preferred_zone); +VM_BUG_ON (order >=MAX_ORDER ); + zonelist_scan:/* * Scan zonelist, looking for a zone with enough free.
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 02/35] Do not sanity check order in the fast path
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 3/35 Do not check NUMA node ID when the caller knows the node is valid
In reply to Mel GormanCallers of alloc_pages_node() can optionally specify -1 as a node to mean "allocate from the current node". However, a number of the callers in fast paths know for a fact their node is valid. To avoid a comparison and branch, this patch adds alloc_pages_exact_node() that only checks the nid with VM_BUG_ON(). Callers that know their node is valid are then converted.
--- arch/ia64/hp/common/sba_iommu.c | 2 +- arch/ia64/kernel/mca.c | 3 +-- arch/ia64/kernel/uncached.c | 3 ++- arch/ia64/sn/pci/pci_dma.c | 3 ++- arch/powerpc/platforms/cell/ras.c | 2 +- arch/x86/kvm/vmx.c | 2 +- drivers/misc/sgi-gru/grufile.c | 2 +- drivers/misc/sgi-xp/xpc_uv.c | 2 +- include/linux/gfp.h | 9 +++++++++ include/linux/mm.h | 1 - kernel/profile.c | 8 ++++---- mm/filemap.c | 2 +- mm/hugetlb.c | 4 ++-- mm/mempolicy.c | 2 +- mm/migrate.c | 2 +- mm/slab.c | 4 ++-- mm/slob.c | 4 ++-- 17 files changed, 32 insertions(+), 23 deletions(-)
arch/ia64/hp/common/sba_iommu.c +1 -1
@@ -1116,7 +1116,7 @@ sba_alloc_coherent (struct device *dev, size_t size, dma_addr_t *dma_handle, gfp#ifdef CONFIG_NUMA {struct page *page; - page =alloc_pages_node (ioc->node ==MAX_NUMNODES ? + page =alloc_pages_exact_node (ioc->node ==MAX_NUMNODES ?numa_node_id () : ioc->node, flags,get_order (size));
arch/ia64/kernel/mca.c +1 -2
@@ -1829,8 +1829,7 @@ ia64_mca_cpu_init(void *cpu_data) data =mca_bootmem (); first_time =0 ; }else - data =page_address (alloc_pages_node (numa_node_id (), -GFP_KERNEL ,get_order (sz))); + data =__get_free_pages (GFP_KERNEL ,get_order (sz));if (!data)panic ("Could not allocate MCA memory for cpu %d\n" , cpu);
arch/ia64/kernel/uncached.c +2 -1
@@ -98,7 +98,8 @@ static int uncached_add_chunk(struct uncached_pool *uc_pool, int nid)/* attempt to allocate a granule's worth of cached memory pages */ - page =alloc_pages_node (nid,GFP_KERNEL | __GFP_ZERO |GFP_THISNODE , + page =alloc_pages_exact_node (nid, +GFP_KERNEL | __GFP_ZERO |GFP_THISNODE ,IA64_GRANULE_SHIFT -PAGE_SHIFT );if (!page) {mutex_unlock (&uc_pool->add_chunk_mutex);
arch/ia64/sn/pci/pci_dma.c +2 -1
@@ -91,7 +91,8 @@ void *sn_dma_alloc_coherent(struct device *dev, size_t size,*/ node =pcibus_to_node (pdev->bus);if (likely (node >=0 )) { -struct page *p =alloc_pages_node (node, flags,get_order (size)); +struct page *p =alloc_pages_exact_node (node, + flags,get_order (size));if (likely (p)) cpuaddr =page_address (p);
arch/powerpc/platforms/cell/ras.c +1 -1
@@ -122,7 +122,7 @@ static int __init cbe_ptcal_enable_on_node(int nid, int order) area->nid = nid; area->order = order; - area->pages =alloc_pages_node (area->nid,GFP_KERNEL , area->order); + area->pages =alloc_pages_exact_node (area->nid,GFP_KERNEL , area->order);if (!area->pages) goto out_free_area;
arch/x86/kvm/vmx.c +1 -1
@@ -1244,7 +1244,7 @@ static struct vmcs *alloc_vmcs_cpu(int cpu)struct page *pages;struct vmcs *vmcs; - pages =alloc_pages_node (node,GFP_KERNEL , vmcs_config.order); + pages =alloc_pages_exact_node (node,GFP_KERNEL , vmcs_config.order);if (!pages)return NULL ; vmcs =page_address (pages);
drivers/misc/sgi-gru/grufile.c +1 -1
@@ -309,7 +309,7 @@ static int gru_init_tables(unsigned long gru_base_paddr, void *gru_base_vaddr) pnode =uv_node_to_pnode (nid);if (gru_base[bid])continue ; - page =alloc_pages_node (nid,GFP_KERNEL , order); + page =alloc_pages_exact_node (nid,GFP_KERNEL , order);if (!page) goto fail; gru_base[bid] =page_address (page);
drivers/misc/sgi-xp/xpc_uv.c +1 -1
@@ -184,7 +184,7 @@ xpc_create_gru_mq_uv(unsigned int mq_size, int cpu, char *irq_name, mq->mmr_blade =uv_cpu_to_blade_id (cpu); nid =cpu_to_node (cpu); - page =alloc_pages_node (nid,GFP_KERNEL | __GFP_ZERO |GFP_THISNODE , + page =alloc_pages_exact_node (nid,GFP_KERNEL | __GFP_ZERO |GFP_THISNODE , pg_order);if (page ==NULL ) { dev_err(xpc_part, "xpc_create_gru_mq_uv() failed to alloc %d "
include/linux/gfp.h +9 -0
@@ -4,6 +4,7 @@#include <linux/mmzone.h> #include <linux/stddef.h> #include <linux/linkage.h> +#include <linux/mmdebug.h> struct vm_area_struct ;
@@ -188,6 +189,14 @@ static inline struct page *alloc_pages_node(int nid, gfp_t gfp_mask,return __alloc_pages (gfp_mask, order,node_zonelist (nid, gfp_mask)); } +static inline struct page *alloc_pages_exact_node (int nid,gfp_t gfp_mask, +unsigned int order) +{ +VM_BUG_ON (nid <0 || nid >=MAX_NUMNODES ); + +return __alloc_pages (gfp_mask, order,node_zonelist (nid, gfp_mask)); +} +#ifdef CONFIG_NUMA extern struct page *alloc_pages_current (gfp_t gfp_mask,unsigned order);
include/linux/mm.h +0 -1
@@ -7,7 +7,6 @@#include <linux/gfp.h> #include <linux/list.h> -#include <linux/mmdebug.h> #include <linux/mmzone.h> #include <linux/rbtree.h> #include <linux/prio_tree.h>
kernel/profile.c +4 -4
@@ -371,7 +371,7 @@ static int __cpuinit profile_cpu_callback(struct notifier_block *info, node =cpu_to_node (cpu);per_cpu (cpu_profile_flip, cpu) =0 ;if (!per_cpu (cpu_profile_hits, cpu)[1 ]) { - page =alloc_pages_node (node, + page =alloc_pages_exact_node (node,GFP_KERNEL | __GFP_ZERO,0 );if (!page)
@@ -379,7 +379,7 @@ static int __cpuinit profile_cpu_callback(struct notifier_block *info,per_cpu (cpu_profile_hits, cpu)[1 ] =page_address (page); }if (!per_cpu (cpu_profile_hits, cpu)[0 ]) { - page =alloc_pages_node (node, + page =alloc_pages_exact_node (node,GFP_KERNEL | __GFP_ZERO,0 );if (!page)
@@ -570,14 +570,14 @@ static int create_hash_tables(void)int node =cpu_to_node (cpu);struct page *page; - page =alloc_pages_node (node, + page =alloc_pages_exact_node (node,GFP_KERNEL | __GFP_ZERO |GFP_THISNODE ,0 );if (!page) goto out_cleanup;per_cpu (cpu_profile_hits, cpu)[1 ] = (struct profile_hit *)page_address (page); - page =alloc_pages_node (node, + page =alloc_pages_exact_node (node,GFP_KERNEL | __GFP_ZERO |GFP_THISNODE ,0 );if (!page)
mm/filemap.c +1 -1
@@ -519,7 +519,7 @@ struct page *__page_cache_alloc(gfp_t gfp) {if (cpuset_do_page_mem_spread ()) {int n =cpuset_mem_spread_node (); -return alloc_pages_node (n, gfp,0 ); +return alloc_pages_exact_node (n, gfp,0 ); }return alloc_pages (gfp,0 ); }
mm/hugetlb.c +2 -2
@@ -630,7 +630,7 @@ static struct page *alloc_fresh_huge_page_node(struct hstate *h, int nid)if (h->order >=MAX_ORDER )return NULL ; - page =alloc_pages_node (nid, + page =alloc_pages_exact_node (nid, htlb_alloc_mask|__GFP_COMP|__GFP_THISNODE| __GFP_REPEAT|__GFP_NOWARN,huge_page_order (h));
@@ -649,7 +649,7 @@ static struct page *alloc_fresh_huge_page_node(struct hstate *h, int nid)* Use a helper variable to find the next node and then * copy it back to hugetlb_next_nid afterwards: * otherwise there's a window in which a racer might -* pass invalid nid MAX_NUMNODES to alloc_pages_node. +* pass invalid nid MAX_NUMNODES to alloc_pages_exact_node. * But we don't need to use a spin_lock here: it really * doesn't matter if occasionally a racer chooses the * same nid as we do. Move nid forward in the mask even
mm/mempolicy.c +1 -1
@@ -767,7 +767,7 @@ static void migrate_page_add(struct page *page, struct list_head *pagelist,static struct page *new_node_page (struct page *page,unsigned long node,int **x) { -return alloc_pages_node (node,GFP_HIGHUSER_MOVABLE ,0 ); +return alloc_pages_exact_node (node,GFP_HIGHUSER_MOVABLE ,0 ); }/*
mm/migrate.c +1 -1
@@ -802,7 +802,7 @@ static struct page *new_page_node(struct page *p, unsigned long private, *result = &pm->status; -return alloc_pages_node (pm->node, +return alloc_pages_exact_node (pm->node,GFP_HIGHUSER_MOVABLE |GFP_THISNODE ,0 ); }
mm/slab.c +2 -2
@@ -1680,7 +1680,7 @@ static void *kmem_getpages(struct kmem_cache *cachep, gfp_t flags, int nodeid)if (cachep->flags &SLAB_RECLAIM_ACCOUNT ) flags |= __GFP_RECLAIMABLE; - page =alloc_pages_node (nodeid, flags, cachep->gfporder); + page =alloc_pages_exact_node (nodeid, flags, cachep->gfporder);if (!page)return NULL ;
@@ -3210,7 +3210,7 @@ retry:if (local_flags & __GFP_WAIT)local_irq_enable ();kmem_flagcheck (cache, flags); - obj =kmem_getpages (cache, local_flags,-1 ); + obj =kmem_getpages (cache, local_flags,numa_node_id ());if (local_flags & __GFP_WAIT)local_irq_disable ();if (obj) {
mm/slob.c +2 -2
@@ -46,7 +46,7 @@* NUMA support in SLOB is fairly simplistic, pushing most of the real * logic down to the page allocator, and simply doing the node accounting * on the upper levels. In the event that a node id is explicitly -* provided, alloc_pages_node() with the specified node id is used +* provided, alloc_pages_exact_node() with the specified node id is used * instead. The common case (or when the node id isn't explicitly provided) * will default to the current node, as per numa_node_id(). *
@@ -236,7 +236,7 @@ static void *slob_new_page(gfp_t gfp, int order, int node)#ifdef CONFIG_NUMA if (node !=-1 ) - page =alloc_pages_node (node, gfp, order); + page =alloc_pages_exact_node (node, gfp, order); else#endif page =alloc_pages (gfp, order);
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 4/35 Check only once if the zonelist is suitable for the allocation
In reply to Mel GormanIt is possible with __GFP_THISNODE that no zones are suitable. This patch makes sure the check is only made once.
--- mm/page_alloc.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
mm/page_alloc.c +3 -3
@@ -1486,9 +1486,8 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,if (should_fail_alloc_page (gfp_mask, order))return NULL ; -restart: - z = zonelist->_zonerefs;/* the list of zones suitable for gfp_mask */ - +/* the list of zones suitable for gfp_mask */ + z = zonelist->_zonerefs;if (unlikely (!z->zone)) {/* * Happens if we have an empty zonelist as a result of
@@ -1497,6 +1496,7 @@ restart:return NULL ; } +restart: page =get_page_from_freelist (gfp_mask|__GFP_HARDWALL, nodemask, order, zonelist, high_zoneidx,ALLOC_WMARK_LOW |ALLOC_CPUSET );if (page)
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 5/35 Break up the allocator entry point into fast and slow paths
In reply to Mel GormanThe core of the page allocator is one giant function which allocates memory on the stack and makes calculations that may not be needed for every allocation. This patch breaks up the allocator path into fast and slow paths for clarity. Note the slow paths are still inlined but the entry is marked unlikely. If they were not inlined, it actally increases text size to generate the as there is only one call site.
--- mm/page_alloc.c | 348 ++++++++++++++++++++++++++++++++++--------------------- 1 files changed, 218 insertions(+), 130 deletions(-)
mm/page_alloc.c +218 -130
@@ -1462,45 +1462,171 @@ try_next_zone:return page; } -/* -* This is the 'heart' of the zoned buddy allocator. -*/ -struct page * -__alloc_pages_nodemask (gfp_t gfp_mask,unsigned int order, -struct zonelist *zonelist,nodemask_t *nodemask) +static inline int +should_alloc_retry (gfp_t gfp_mask,unsigned int order, +unsigned long pages_reclaimed) { -const gfp_t wait = gfp_mask & __GFP_WAIT; -enum zone_type high_zoneidx =gfp_zone (gfp_mask); -struct zoneref *z; -struct zone *zone; -struct page *page; -struct reclaim_state reclaim_state; -struct task_struct *p = current; -int do_retry; -int alloc_flags; -unsigned long did_some_progress; -unsigned long pages_reclaimed =0 ; +/* Do not loop if specifically requested */ +if (gfp_mask & __GFP_NORETRY) +return 0 ; -might_sleep_if (wait); +/* +* In this implementation, order <= PAGE_ALLOC_COSTLY_ORDER +* means __GFP_NOFAIL, but that may not be true in other +* implementations. +*/ +if (order <=PAGE_ALLOC_COSTLY_ORDER ) +return 1 ; -if (should_fail_alloc_page (gfp_mask, order)) -return NULL ; +/* +* For order > PAGE_ALLOC_COSTLY_ORDER, if __GFP_REPEAT is +* specified, then we retry until we no longer reclaim any pages +* (above), or we've reclaimed an order of pages at least as +* large as the allocation's order. In both cases, if the +* allocation still fails, we stop retrying. +*/ +if (gfp_mask & __GFP_REPEAT && pages_reclaimed < (1 << order)) +return 1 ; -/* the list of zones suitable for gfp_mask */ - z = zonelist->_zonerefs; -if (unlikely (!z->zone)) { -/* -* Happens if we have an empty zonelist as a result of -* GFP_THISNODE being used on a memoryless node -*/ +/* +* Don't let big-order allocations loop unless the caller +* explicitly requests that. +*/ +if (gfp_mask & __GFP_NOFAIL) +return 1 ; + +return 0 ; +} + +static inline struct page * +__alloc_pages_may_oom (gfp_t gfp_mask,unsigned int order, +struct zonelist *zonelist,enum zone_type high_zoneidx, +nodemask_t *nodemask) +{ +struct page *page; + +/* Acquire the OOM killer lock for the zones in zonelist */ +if (!try_set_zone_oom (zonelist, gfp_mask)) { +schedule_timeout_uninterruptible (1 );return NULL ; } -restart: - page =get_page_from_freelist (gfp_mask|__GFP_HARDWALL, nodemask, order, - zonelist, high_zoneidx,ALLOC_WMARK_LOW |ALLOC_CPUSET ); +/* +* Go through the zonelist yet one more time, keep very high watermark +* here, this is only to catch a parallel oom killing, we must fail if +* we're still under heavy pressure. +*/ + page =get_page_from_freelist (gfp_mask|__GFP_HARDWALL, nodemask, + order, zonelist, high_zoneidx, +ALLOC_WMARK_HIGH |ALLOC_CPUSET );if (page) - goto got_pg; + goto out; + +/* The OOM killer will not help higher order allocs */ +if (order >PAGE_ALLOC_COSTLY_ORDER ) + goto out; + +/* Exhausted what can be done so it's blamo time */ +out_of_memory (zonelist, gfp_mask, order); + +out: +clear_zonelist_oom (zonelist, gfp_mask); +return page; +} + +/* The really slow allocator path where we enter direct reclaim */ +static inline struct page * +__alloc_pages_direct_reclaim (gfp_t gfp_mask,unsigned int order, +struct zonelist *zonelist,enum zone_type high_zoneidx, +nodemask_t *nodemask,int alloc_flags,unsigned long *did_some_progress) +{ +struct page *page =NULL ; +struct reclaim_state reclaim_state; +struct task_struct *p = current; + +cond_resched (); + +/* We now go into synchronous reclaim */ +cpuset_memory_pressure_bump (); + +/* +* The task's cpuset might have expanded its set of allowable nodes +*/ +cpuset_update_task_memory_state (); + p->flags |=PF_MEMALLOC ; + reclaim_state.reclaimed_slab =0 ; + p->reclaim_state = &reclaim_state; + + *did_some_progress =try_to_free_pages (zonelist, order, gfp_mask); + + p->reclaim_state =NULL ; + p->flags &= ~PF_MEMALLOC ; + +cond_resched (); + +if (order !=0 ) +drain_all_pages (); + +if (likely (*did_some_progress)) + page =get_page_from_freelist (gfp_mask, nodemask, order, + zonelist, high_zoneidx, alloc_flags); +return page; +} + +static inline int +is_allocation_high_priority (struct task_struct *p,gfp_t gfp_mask) +{ +if (((p->flags &PF_MEMALLOC ) ||unlikely (test_thread_flag (TIF_MEMDIE ))) + && !in_interrupt ()) +if (!(gfp_mask & __GFP_NOMEMALLOC)) +return 1 ; +return 0 ; +} + +/* +* This is called in the allocator slow-path if the allocation request is of +* sufficient urgency to ignore watermarks and take other desperate measures +*/ +static inline struct page * +__alloc_pages_high_priority (gfp_t gfp_mask,unsigned int order, +struct zonelist *zonelist,enum zone_type high_zoneidx, +nodemask_t *nodemask) +{ +struct page *page; + +do { + page =get_page_from_freelist (gfp_mask, nodemask, order, + zonelist, high_zoneidx,ALLOC_NO_WATERMARKS ); + +if (!page && gfp_mask & __GFP_NOFAIL) +congestion_wait (WRITE ,HZ /50 ); + }while (!page && (gfp_mask & __GFP_NOFAIL)); + +return page; +} + +static inline +void wake_all_kswapd (unsigned int order,struct zonelist *zonelist, +enum zone_type high_zoneidx) +{ +struct zoneref *z; +struct zone *zone; + +for_each_zone_zonelist (zone, z, zonelist, high_zoneidx) +wakeup_kswapd (zone, order); +} + +static inline struct page * +__alloc_pages_slowpath (gfp_t gfp_mask,unsigned int order, +struct zonelist *zonelist,enum zone_type high_zoneidx, +nodemask_t *nodemask) +{ +const gfp_t wait = gfp_mask & __GFP_WAIT; +struct page *page =NULL ; +int alloc_flags; +unsigned long pages_reclaimed =0 ; +unsigned long did_some_progress; +struct task_struct *p = current;/* * GFP_THISNODE (meaning __GFP_THISNODE, __GFP_NORETRY and
@@ -1513,8 +1639,7 @@ restart:if (NUMA_BUILD && (gfp_mask &GFP_THISNODE ) ==GFP_THISNODE ) goto nopage; -for_each_zone_zonelist (zone, z, zonelist, high_zoneidx) -wakeup_kswapd (zone, order); +wake_all_kswapd (order, zonelist, high_zoneidx);/* * OK, we're below the kswapd watermark and have kicked background
@@ -1534,6 +1659,7 @@ restart:if (wait) alloc_flags |=ALLOC_CPUSET ; +restart:/* * Go through the zonelist again. Let __GFP_HIGH and allocations * coming from realtime tasks go deeper into reserves.
@@ -1547,118 +1673,47 @@ restart:if (page) goto got_pg; -/* This allocation should allow future memory freeing. */ - -rebalance: -if (((p->flags &PF_MEMALLOC ) ||unlikely (test_thread_flag (TIF_MEMDIE ))) - && !in_interrupt ()) { -if (!(gfp_mask & __GFP_NOMEMALLOC)) { -nofail_alloc: -/* go through the zonelist yet again, ignoring mins */ - page =get_page_from_freelist (gfp_mask, nodemask, order, - zonelist, high_zoneidx,ALLOC_NO_WATERMARKS ); -if (page) - goto got_pg; -if (gfp_mask & __GFP_NOFAIL) { -congestion_wait (WRITE ,HZ /50 ); - goto nofail_alloc; - } - } - goto nopage; - } +/* Allocate without watermarks if the context allows */ +if (is_allocation_high_priority (p, gfp_mask)) + page =__alloc_pages_high_priority (gfp_mask, order, + zonelist, high_zoneidx, nodemask); +if (page) + goto got_pg;/* Atomic allocations - we can't balance anything */ if (!wait) goto nopage; -cond_resched (); +/* Try direct reclaim and then allocating */ + page =__alloc_pages_direct_reclaim (gfp_mask, order, + zonelist, high_zoneidx, + nodemask, + alloc_flags, &did_some_progress); +if (page) + goto got_pg; -/* We now go into synchronous reclaim */ -cpuset_memory_pressure_bump ();/* -* The task's cpuset might have expanded its set of allowable nodes +* If we failed to make any progress reclaiming, then we are +* running out of options and have to consider going OOM */ -cpuset_update_task_memory_state (); - p->flags |=PF_MEMALLOC ; - reclaim_state.reclaimed_slab =0 ; - p->reclaim_state = &reclaim_state; - - did_some_progress =try_to_free_pages (zonelist, order, gfp_mask); - - p->reclaim_state =NULL ; - p->flags &= ~PF_MEMALLOC ; - -cond_resched (); - -if (order !=0 ) -drain_all_pages (); +if (!did_some_progress) { +if ((gfp_mask & __GFP_FS) && !(gfp_mask & __GFP_NORETRY)) { + page =__alloc_pages_may_oom (gfp_mask, order, + zonelist, high_zoneidx, + nodemask); +if (page) + goto got_pg; -if (likely (did_some_progress)) { - page =get_page_from_freelist (gfp_mask, nodemask, order, - zonelist, high_zoneidx, alloc_flags); -if (page) - goto got_pg; - }else if ((gfp_mask & __GFP_FS) && !(gfp_mask & __GFP_NORETRY)) { -if (!try_set_zone_oom (zonelist, gfp_mask)) { -schedule_timeout_uninterruptible (1 ); goto restart; } - -/* -* Go through the zonelist yet one more time, keep -* very high watermark here, this is only to catch -* a parallel oom killing, we must fail if we're still -* under heavy pressure. -*/ - page =get_page_from_freelist (gfp_mask|__GFP_HARDWALL, nodemask, - order, zonelist, high_zoneidx, -ALLOC_WMARK_HIGH |ALLOC_CPUSET ); -if (page) { -clear_zonelist_oom (zonelist, gfp_mask); - goto got_pg; - } - -/* The OOM killer will not help higher order allocs so fail */ -if (order >PAGE_ALLOC_COSTLY_ORDER ) { -clear_zonelist_oom (zonelist, gfp_mask); - goto nopage; - } - -out_of_memory (zonelist, gfp_mask, order); -clear_zonelist_oom (zonelist, gfp_mask); - goto restart; } -/* -* Don't let big-order allocations loop unless the caller explicitly -* requests that. Wait for some write requests to complete then retry. -* -* In this implementation, order <= PAGE_ALLOC_COSTLY_ORDER -* means __GFP_NOFAIL, but that may not be true in other -* implementations. -* -* For order > PAGE_ALLOC_COSTLY_ORDER, if __GFP_REPEAT is -* specified, then we retry until we no longer reclaim any pages -* (above), or we've reclaimed an order of pages at least as -* large as the allocation's order. In both cases, if the -* allocation still fails, we stop retrying. -*/ +/* Check if we should retry the allocation */ pages_reclaimed += did_some_progress; - do_retry =0 ; -if (!(gfp_mask & __GFP_NORETRY)) { -if (order <=PAGE_ALLOC_COSTLY_ORDER ) { - do_retry =1 ; - }else { -if (gfp_mask & __GFP_REPEAT && - pages_reclaimed < (1 << order)) - do_retry =1 ; - } -if (gfp_mask & __GFP_NOFAIL) - do_retry =1 ; - } -if (do_retry) { +if (should_alloc_retry (gfp_mask, order, pages_reclaimed)) { +/* Wait for some write requests to complete then retry */ congestion_wait (WRITE ,HZ /50 ); - goto rebalance; + goto restart; } nopage:
@@ -1671,6 +1726,39 @@ nopage: } got_pg:return page; + +} + +/* +* This is the 'heart' of the zoned buddy allocator. +*/ +struct page * +__alloc_pages_nodemask (gfp_t gfp_mask,unsigned int order, +struct zonelist *zonelist,nodemask_t *nodemask) +{ +enum zone_type high_zoneidx =gfp_zone (gfp_mask); +struct page *page; + +might_sleep_if (gfp_mask & __GFP_WAIT); + +if (should_fail_alloc_page (gfp_mask, order)) +return NULL ; + +/* +* Check the zones suitable for the gfp_mask contain at least one +* valid zone. It's possible to have an empty zonelist as a result +* of GFP_THISNODE and a memoryless node +*/ +if (unlikely (!zonelist->_zonerefs->zone)) +return NULL ; + + page =get_page_from_freelist (gfp_mask|__GFP_HARDWALL, nodemask, order, + zonelist, high_zoneidx,ALLOC_WMARK_LOW |ALLOC_CPUSET ); +if (unlikely (!page)) + page =__alloc_pages_slowpath (gfp_mask, order, + zonelist, high_zoneidx, nodemask); + +return page; }EXPORT_SYMBOL (__alloc_pages_nodemask);
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 6/35 Move check for disabled anti-fragmentation out of fastpath
In reply to Mel GormanOn low-memory systems, anti-fragmentation gets disabled as there is nothing it can do and it would just incur overhead shuffling pages between lists constantly. Currently the check is made in the free page fast path for every page. This patch moves it to a slow path. On machines with low memory, there will be small amount of additional overhead as pages get shuffled between lists but it should quickly settle.
--- include/linux/mmzone.h | 3 --- mm/page_alloc.c | 4 ++++ 2 files changed, 4 insertions(+), 3 deletions(-)
include/linux/mmzone.h +0 -3
@@ -50,9 +50,6 @@ extern int page_group_by_mobility_disabled;static inline int get_pageblock_migratetype (struct page *page) { -if (unlikely (page_group_by_mobility_disabled)) -return MIGRATE_UNMOVABLE ; -return get_pageblock_flags_group (page, PB_migrate, PB_migrate_end); }
mm/page_alloc.c +4 -0
@@ -171,6 +171,10 @@ int page_group_by_mobility_disabled __read_mostly;static void set_pageblock_migratetype (struct page *page,int migratetype) { + +if (unlikely (page_group_by_mobility_disabled)) + migratetype =MIGRATE_UNMOVABLE ; +set_pageblock_flags_group (page, (unsigned long )migratetype, PB_migrate, PB_migrate_end); }
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 06/35] Move check for disabled anti-fragmentation out of fastpath
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 7/35 Check in advance if the zonelist needs additional filtering
In reply to Mel GormanZonelist are filtered based on nodemasks for memory policies normally. It can be additionally filters on cpusets if they exist as well as noting when zones are full. These simple checks are expensive enough to be noticed in profiles. This patch checks in advance if zonelist filtering will ever be needed. If not, then the bulk of the checks are skipped.
--- include/linux/cpuset.h | 2 ++ mm/page_alloc.c | 37 ++++++++++++++++++++++++++----------- 2 files changed, 28 insertions(+), 11 deletions(-)
include/linux/cpuset.h +2 -0
@@ -83,6 +83,8 @@ extern void cpuset_print_task_mems_allowed(struct task_struct *p);#else /* !CONFIG_CPUSETS */ +#define number_of_cpusets (0) +static inline int cpuset_init_early (void ) {return 0 ; }static inline int cpuset_init (void ) {return 0 ; }static inline void cpuset_init_smp (void ) {}
mm/page_alloc.c +26 -11
@@ -1139,7 +1139,11 @@ failed:#define ALLOC_WMARK_HIGH 0x08/* use pages_high watermark */ #define ALLOC_HARDER 0x10/* try to alloc harder */ #define ALLOC_HIGH 0x20/* __GFP_HIGH set */ +#ifdef CONFIG_CPUSETS #define ALLOC_CPUSET 0x40/* check for correct cpuset */ +#else +#define ALLOC_CPUSET 0x00 +#endif /* CONFIG_CPUSETS */ #ifdef CONFIG_FAIL_PAGE_ALLOC
@@ -1403,6 +1407,7 @@ get_page_from_freelist(gfp_t gfp_mask, nodemask_t *nodemask, unsigned int order,nodemask_t *allowednodes =NULL ;/* zonelist_cache approximation */ int zlc_active =0 ;/* set if using zonelist_cache */ int did_zlc_setup =0 ;/* just call zlc_setup() one time */ +int zonelist_filter =0 ; (void )first_zones_zonelist (zonelist, high_zoneidx, nodemask, &preferred_zone);
@@ -1413,6 +1418,10 @@ get_page_from_freelist(gfp_t gfp_mask, nodemask_t *nodemask, unsigned int order,VM_BUG_ON (order >=MAX_ORDER ); +/* Determine in advance if the zonelist needs filtering */ +if ((alloc_flags &ALLOC_CPUSET ) &&unlikely (number_of_cpusets >1 )) + zonelist_filter =1 ; + zonelist_scan:/* * Scan zonelist, looking for a zone with enough free.
@@ -1420,12 +1429,16 @@ zonelist_scan:*/ for_each_zone_zonelist_nodemask (zone, z, zonelist, high_zoneidx, nodemask) { -if (NUMA_BUILD && zlc_active && - !zlc_zone_worth_trying (zonelist, z, allowednodes)) -continue ; -if ((alloc_flags &ALLOC_CPUSET ) && - !cpuset_zone_allowed_softwall (zone, gfp_mask)) - goto try_next_zone; + +/* Ignore the additional zonelist filter checks if possible */ +if (zonelist_filter) { +if (NUMA_BUILD && zlc_active && + !zlc_zone_worth_trying (zonelist, z, allowednodes)) +continue ; +if ((alloc_flags &ALLOC_CPUSET ) && + !cpuset_zone_allowed_softwall (zone, gfp_mask)) + goto try_next_zone; + }if (!(alloc_flags &ALLOC_NO_WATERMARKS )) {unsigned long mark;
@@ -1447,13 +1460,15 @@ zonelist_scan:if (page)break ; this_zone_full: -if (NUMA_BUILD ) +if (NUMA_BUILD && zonelist_filter)zlc_mark_zone_full (zonelist, z); try_next_zone: -if (NUMA_BUILD && !did_zlc_setup) { -/* we do zlc_setup after the first zone is tried */ - allowednodes =zlc_setup (zonelist, alloc_flags); - zlc_active =1 ; +if (NUMA_BUILD && zonelist_filter) { +if (!did_zlc_setup) { +/* do zlc_setup after the first zone is tried */ + allowednodes =zlc_setup (zonelist, alloc_flags); + zlc_active =1 ; + } did_zlc_setup =1 ; } }
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 8/35 Calculate the preferred zone for allocation only once
In reply to Mel Gormanget_page_from_freelist() can be called multiple times for an allocation. Part of this calculates the preferred_zone which is the first usable zone in the zonelist. This patch calculates preferred_zone once.
--- mm/page_alloc.c | 53 ++++++++++++++++++++++++++++++++--------------------- 1 files changed, 32 insertions(+), 21 deletions(-)
mm/page_alloc.c +32 -21
@@ -1398,24 +1398,19 @@ static void zlc_mark_zone_full(struct zonelist *zonelist, struct zoneref *z)*/ static struct page *get_page_from_freelist (gfp_t gfp_mask,nodemask_t *nodemask,unsigned int order, -struct zonelist *zonelist,int high_zoneidx,int alloc_flags) +struct zonelist *zonelist,int high_zoneidx,int alloc_flags, +struct zone *preferred_zone) {struct zoneref *z;struct page *page =NULL ;int classzone_idx; -struct zone *zone, *preferred_zone; +struct zone *zone;nodemask_t *allowednodes =NULL ;/* zonelist_cache approximation */ int zlc_active =0 ;/* set if using zonelist_cache */ int did_zlc_setup =0 ;/* just call zlc_setup() one time */ int zonelist_filter =0 ; - (void )first_zones_zonelist (zonelist, high_zoneidx, nodemask, - &preferred_zone); -if (!preferred_zone) -return NULL ; - classzone_idx =zone_idx (preferred_zone); -VM_BUG_ON (order >=MAX_ORDER );/* Determine in advance if the zonelist needs filtering */
@@ -1520,7 +1515,7 @@ should_alloc_retry(gfp_t gfp_mask, unsigned int order,static inline struct page *__alloc_pages_may_oom (gfp_t gfp_mask,unsigned int order,struct zonelist *zonelist,enum zone_type high_zoneidx, -nodemask_t *nodemask) +nodemask_t *nodemask,struct zone *preferred_zone) {struct page *page;
@@ -1537,7 +1532,8 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order,*/ page =get_page_from_freelist (gfp_mask|__GFP_HARDWALL, nodemask, order, zonelist, high_zoneidx, -ALLOC_WMARK_HIGH |ALLOC_CPUSET ); +ALLOC_WMARK_HIGH |ALLOC_CPUSET , + preferred_zone);if (page) goto out;
@@ -1557,7 +1553,8 @@ out:static inline struct page *__alloc_pages_direct_reclaim (gfp_t gfp_mask,unsigned int order,struct zonelist *zonelist,enum zone_type high_zoneidx, -nodemask_t *nodemask,int alloc_flags,unsigned long *did_some_progress) +nodemask_t *nodemask,int alloc_flags,struct zone *preferred_zone, +unsigned long *did_some_progress) {struct page *page =NULL ;struct reclaim_state reclaim_state;
@@ -1588,7 +1585,8 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,if (likely (*did_some_progress)) page =get_page_from_freelist (gfp_mask, nodemask, order, - zonelist, high_zoneidx, alloc_flags); + zonelist, high_zoneidx, + alloc_flags, preferred_zone);return page; }
@@ -1609,13 +1607,14 @@ is_allocation_high_priority(struct task_struct *p, gfp_t gfp_mask)static inline struct page *__alloc_pages_high_priority (gfp_t gfp_mask,unsigned int order,struct zonelist *zonelist,enum zone_type high_zoneidx, -nodemask_t *nodemask) +nodemask_t *nodemask,struct zone *preferred_zone) {struct page *page; do { page = get_page_from_freelist(gfp_mask, nodemask, order, - zonelist, high_zoneidx, ALLOC_NO_WATERMARKS); + zonelist, high_zoneidx, ALLOC_NO_WATERMARKS, + preferred_zone); if (!page && gfp_mask & __GFP_NOFAIL) congestion_wait(WRITE, HZ/50);
@@ -1638,7 +1637,7 @@ void wake_all_kswapd(unsigned int order, struct zonelist *zonelist,static inline struct page *__alloc_pages_slowpath (gfp_t gfp_mask,unsigned int order,struct zonelist *zonelist,enum zone_type high_zoneidx, -nodemask_t *nodemask) +nodemask_t *nodemask,struct zone *preferred_zone) {const gfp_t wait = gfp_mask & __GFP_WAIT;struct page *page =NULL ;
@@ -1688,14 +1687,15 @@ restart:* See also cpuset_zone_allowed() comment in kernel/cpuset.c. */ page =get_page_from_freelist (gfp_mask, nodemask, order, zonelist, - high_zoneidx, alloc_flags); + high_zoneidx, alloc_flags, + preferred_zone);if (page) goto got_pg;/* Allocate without watermarks if the context allows */ if (is_allocation_high_priority (p, gfp_mask)) page =__alloc_pages_high_priority (gfp_mask, order, - zonelist, high_zoneidx, nodemask); + zonelist, high_zoneidx, nodemask, preferred_zone);if (page) goto got_pg;
@@ -1707,7 +1707,8 @@ restart: page =__alloc_pages_direct_reclaim (gfp_mask, order, zonelist, high_zoneidx, nodemask, - alloc_flags, &did_some_progress); + alloc_flags, preferred_zone, + &did_some_progress);if (page) goto got_pg;
@@ -1719,7 +1720,7 @@ restart:if ((gfp_mask & __GFP_FS) && !(gfp_mask & __GFP_NORETRY)) { page =__alloc_pages_may_oom (gfp_mask, order, zonelist, high_zoneidx, - nodemask); + nodemask, preferred_zone);if (page) goto got_pg;
@@ -1756,6 +1757,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, nodemask_t *nodemask) { enum zone_type high_zoneidx = gfp_zone(gfp_mask); +struct zone *preferred_zone;struct page *page;might_sleep_if (gfp_mask & __GFP_WAIT);
@@ -1771,11 +1773,20 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,if (unlikely (!zonelist->_zonerefs->zone))return NULL ; +/* The preferred zone is used for statistics later */ + (void )first_zones_zonelist (zonelist, high_zoneidx, nodemask, + &preferred_zone); +if (!preferred_zone) +return NULL ; + +/* First allocation attempt */ page =get_page_from_freelist (gfp_mask|__GFP_HARDWALL, nodemask, order, - zonelist, high_zoneidx,ALLOC_WMARK_LOW |ALLOC_CPUSET ); + zonelist, high_zoneidx,ALLOC_WMARK_LOW |ALLOC_CPUSET , + preferred_zone);if (unlikely (!page)) page =__alloc_pages_slowpath (gfp_mask, order, - zonelist, high_zoneidx, nodemask); + zonelist, high_zoneidx, nodemask, + preferred_zone);return page; }
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 9/35 Calculate the migratetype for allocation only once
In reply to Mel GormanGFP mask is converted into a migratetype when deciding which pagelist to take a page from. However, it is happening multiple times per allocation, at least once per zone traversed. Calculate it once.
--- mm/page_alloc.c | 43 ++++++++++++++++++++++++++----------------- 1 files changed, 26 insertions(+), 17 deletions(-)
mm/page_alloc.c +26 -17
@@ -1067,13 +1067,13 @@ void split_page(struct page *page, unsigned int order)* or two. */ static struct page *buffered_rmqueue (struct zone *preferred_zone, -struct zone *zone,int order,gfp_t gfp_flags) +struct zone *zone,int order,gfp_t gfp_flags, +int migratetype) {unsigned long flags;struct page *page;int cold = !!(gfp_flags & __GFP_COLD);int cpu; -int migratetype =allocflags_to_migratetype (gfp_flags); again: cpu =get_cpu ();
@@ -1399,7 +1399,7 @@ static void zlc_mark_zone_full(struct zonelist *zonelist, struct zoneref *z)static struct page *get_page_from_freelist (gfp_t gfp_mask,nodemask_t *nodemask,unsigned int order,struct zonelist *zonelist,int high_zoneidx,int alloc_flags, -struct zone *preferred_zone) +struct zone *preferred_zone,int migratetype) {struct zoneref *z;struct page *page =NULL ;
@@ -1451,7 +1451,8 @@ zonelist_scan: } } - page =buffered_rmqueue (preferred_zone, zone, order, gfp_mask); + page =buffered_rmqueue (preferred_zone, zone, order, + gfp_mask, migratetype);if (page)break ; this_zone_full:
@@ -1515,7 +1516,8 @@ should_alloc_retry(gfp_t gfp_mask, unsigned int order,static inline struct page *__alloc_pages_may_oom (gfp_t gfp_mask,unsigned int order,struct zonelist *zonelist,enum zone_type high_zoneidx, -nodemask_t *nodemask,struct zone *preferred_zone) +nodemask_t *nodemask,struct zone *preferred_zone, +int migratetype) {struct page *page;
@@ -1533,7 +1535,7 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, page =get_page_from_freelist (gfp_mask|__GFP_HARDWALL, nodemask, order, zonelist, high_zoneidx,ALLOC_WMARK_HIGH |ALLOC_CPUSET , - preferred_zone); + preferred_zone, migratetype);if (page) goto out;
@@ -1554,7 +1556,7 @@ static inline struct page * __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, enum zone_type high_zoneidx, nodemask_t *nodemask, int alloc_flags, struct zone *preferred_zone, - unsigned long *did_some_progress) + int migratetype, unsigned long *did_some_progress) { struct page *page = NULL;struct reclaim_state reclaim_state;
@@ -1586,7 +1588,8 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,if (likely (*did_some_progress)) page =get_page_from_freelist (gfp_mask, nodemask, order, zonelist, high_zoneidx, - alloc_flags, preferred_zone); + alloc_flags, preferred_zone, + migratetype);return page; }
@@ -1607,14 +1610,15 @@ is_allocation_high_priority(struct task_struct *p, gfp_t gfp_mask)static inline struct page *__alloc_pages_high_priority (gfp_t gfp_mask,unsigned int order,struct zonelist *zonelist,enum zone_type high_zoneidx, -nodemask_t *nodemask,struct zone *preferred_zone) +nodemask_t *nodemask,struct zone *preferred_zone, +int migratetype) {struct page *page; do { page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist, high_zoneidx, ALLOC_NO_WATERMARKS, - preferred_zone); + preferred_zone, migratetype); if (!page && gfp_mask & __GFP_NOFAIL) congestion_wait(WRITE, HZ/50);
@@ -1637,7 +1641,8 @@ void wake_all_kswapd(unsigned int order, struct zonelist *zonelist,static inline struct page *__alloc_pages_slowpath (gfp_t gfp_mask,unsigned int order,struct zonelist *zonelist,enum zone_type high_zoneidx, -nodemask_t *nodemask,struct zone *preferred_zone) +nodemask_t *nodemask,struct zone *preferred_zone, +int migratetype) {const gfp_t wait = gfp_mask & __GFP_WAIT;struct page *page =NULL ;
@@ -1688,14 +1693,16 @@ restart:*/ page =get_page_from_freelist (gfp_mask, nodemask, order, zonelist, high_zoneidx, alloc_flags, - preferred_zone); + preferred_zone, + migratetype);if (page) goto got_pg;/* Allocate without watermarks if the context allows */ if (is_allocation_high_priority (p, gfp_mask)) page =__alloc_pages_high_priority (gfp_mask, order, - zonelist, high_zoneidx, nodemask, preferred_zone); + zonelist, high_zoneidx, nodemask, preferred_zone, + migratetype);if (page) goto got_pg;
@@ -1708,7 +1715,7 @@ restart: zonelist, high_zoneidx, nodemask, alloc_flags, preferred_zone, - &did_some_progress); + migratetype, &did_some_progress);if (page) goto got_pg;
@@ -1720,7 +1727,8 @@ restart:if ((gfp_mask & __GFP_FS) && !(gfp_mask & __GFP_NORETRY)) { page =__alloc_pages_may_oom (gfp_mask, order, zonelist, high_zoneidx, - nodemask, preferred_zone); + nodemask, preferred_zone, + migratetype);if (page) goto got_pg;
@@ -1759,6 +1767,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,enum zone_type high_zoneidx =gfp_zone (gfp_mask);struct zone *preferred_zone;struct page *page; +int migratetype =allocflags_to_migratetype (gfp_mask);might_sleep_if (gfp_mask & __GFP_WAIT);
@@ -1782,11 +1791,11 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,/* First allocation attempt */ page =get_page_from_freelist (gfp_mask|__GFP_HARDWALL, nodemask, order, zonelist, high_zoneidx,ALLOC_WMARK_LOW |ALLOC_CPUSET , - preferred_zone); + preferred_zone, migratetype);if (unlikely (!page)) page =__alloc_pages_slowpath (gfp_mask, order, zonelist, high_zoneidx, nodemask, - preferred_zone); + preferred_zone, migratetype);return page; }
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 10/35 Calculate the alloc_flags for allocation only once
In reply to Mel GormanFactor out the mapping between GFP and alloc_flags only once. Once factored out, it only needs to be calculated once but some care must be taken.
[[email protected] says]
As the test:
- if (((p->flags & PF_MEMALLOC) || unlikely(test_thread_flag(TIF_MEMDIE)))
- && !in_interrupt()) {
- if (!(gfp_mask & __GFP_NOMEMALLOC)) {
has been replaced with a slightly weaker one:
+ if (alloc_flags & ALLOC_NO_WATERMARKS) {
we need to ensure we don't recurse when PF_MEMALLOC is set.
From: Peter Zijlstra <[email protected]>
--- mm/page_alloc.c | 88 +++++++++++++++++++++++++++++++----------------------- 1 files changed, 50 insertions(+), 38 deletions(-)
mm/page_alloc.c +50 -38
@@ -1593,16 +1593,6 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order,return page; } -static inline int -is_allocation_high_priority (struct task_struct *p,gfp_t gfp_mask) -{ -if (((p->flags &PF_MEMALLOC ) ||unlikely (test_thread_flag (TIF_MEMDIE ))) - && !in_interrupt ()) -if (!(gfp_mask & __GFP_NOMEMALLOC)) -return 1 ; -return 0 ; -} -/* * This is called in the allocator slow-path if the allocation request is of * sufficient urgency to ignore watermarks and take other desperate measures
@@ -1638,6 +1628,42 @@ void wake_all_kswapd(unsigned int order, struct zonelist *zonelist,wakeup_kswapd (zone, order); } +static inline int +gfp_to_alloc_flags (gfp_t gfp_mask) +{ +struct task_struct *p = current; +int alloc_flags =ALLOC_WMARK_MIN |ALLOC_CPUSET ; +const gfp_t wait = gfp_mask & __GFP_WAIT; + +/* +* The caller may dip into page reserves a bit more if the caller +* cannot run direct reclaim, or if the caller has realtime scheduling +* policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will +* set both ALLOC_HARDER (!wait) and ALLOC_HIGH (__GFP_HIGH). +*/ +if (gfp_mask & __GFP_HIGH) + alloc_flags |=ALLOC_HIGH ; + +if (!wait) { + alloc_flags |=ALLOC_HARDER ; +/* +* Ignore cpuset if GFP_ATOMIC (!wait) rather than fail alloc. +* See also cpuset_zone_allowed() comment in kernel/cpuset.c. +*/ + alloc_flags &= ~ALLOC_CPUSET ; + }else if (unlikely (rt_task (p)) && !in_interrupt ()) + alloc_flags |=ALLOC_HARDER ; + +if (likely (!(gfp_mask & __GFP_NOMEMALLOC))) { +if (!in_interrupt () && + ((p->flags &PF_MEMALLOC ) || +unlikely (test_thread_flag (TIF_MEMDIE )))) + alloc_flags |=ALLOC_NO_WATERMARKS ; + } + +return alloc_flags; +} +static inline struct page * __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, enum zone_type high_zoneidx,
@@ -1668,48 +1694,34 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,* OK, we're below the kswapd watermark and have kicked background * reclaim. Now things get more complex, so set up alloc_flags according * to how we want to proceed. -* -* The caller may dip into page reserves a bit more if the caller -* cannot run direct reclaim, or if the caller has realtime scheduling -* policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will -* set both ALLOC_HARDER (!wait) and ALLOC_HIGH (__GFP_HIGH). */ - alloc_flags =ALLOC_WMARK_MIN ; -if ((unlikely (rt_task (p)) && !in_interrupt ()) || !wait) - alloc_flags |=ALLOC_HARDER ; -if (gfp_mask & __GFP_HIGH) - alloc_flags |=ALLOC_HIGH ; -if (wait) - alloc_flags |=ALLOC_CPUSET ; + alloc_flags =gfp_to_alloc_flags (gfp_mask); restart: -/* -* Go through the zonelist again. Let __GFP_HIGH and allocations -* coming from realtime tasks go deeper into reserves. -* -* This is the last chance, in general, before the goto nopage. -* Ignore cpuset if GFP_ATOMIC (!wait) rather than fail alloc. -* See also cpuset_zone_allowed() comment in kernel/cpuset.c. -*/ +/* This is the last chance, in general, before the goto nopage. */ page =get_page_from_freelist (gfp_mask, nodemask, order, zonelist, - high_zoneidx, alloc_flags, - preferred_zone, - migratetype); + high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS , + preferred_zone, migratetype);if (page) goto got_pg;/* Allocate without watermarks if the context allows */ -if (is_allocation_high_priority (p, gfp_mask)) +if (alloc_flags &ALLOC_NO_WATERMARKS ) { page =__alloc_pages_high_priority (gfp_mask, order, - zonelist, high_zoneidx, nodemask, preferred_zone, - migratetype); -if (page) - goto got_pg; + zonelist, high_zoneidx, nodemask, + preferred_zone, migratetype); +if (page) + goto got_pg; + }/* Atomic allocations - we can't balance anything */ if (!wait) goto nopage; +/* Avoid recursion of direct reclaim */ +if (p->flags &PF_MEMALLOC ) + goto nopage; +/* Try direct reclaim and then allocating */ page = __alloc_pages_direct_reclaim(gfp_mask, order, zonelist, high_zoneidx,
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 11/35 Calculate the cold parameter for allocation only once
In reply to Mel GormanGFP mask is checked for __GFP_COLD has been specified when deciding which end of the PCP lists to use. However, it is happening multiple times per allocation, at least once per zone traversed. Calculate it once.
--- mm/page_alloc.c | 35 ++++++++++++++++++----------------- 1 files changed, 18 insertions(+), 17 deletions(-)
mm/page_alloc.c +18 -17
@@ -1068,11 +1068,10 @@ void split_page(struct page *page, unsigned int order)*/ static struct page *buffered_rmqueue (struct zone *preferred_zone,struct zone *zone,int order,gfp_t gfp_flags, -int migratetype) +int migratetype,int cold) {unsigned long flags;struct page *page; -int cold = !!(gfp_flags & __GFP_COLD);int cpu; again:
@@ -1399,7 +1398,7 @@ static void zlc_mark_zone_full(struct zonelist *zonelist, struct zoneref *z)static struct page *get_page_from_freelist (gfp_t gfp_mask,nodemask_t *nodemask,unsigned int order,struct zonelist *zonelist,int high_zoneidx,int alloc_flags, -struct zone *preferred_zone,int migratetype) +struct zone *preferred_zone,int migratetype,int cold) {struct zoneref *z;struct page *page =NULL ;
@@ -1452,7 +1451,7 @@ zonelist_scan: } page =buffered_rmqueue (preferred_zone, zone, order, - gfp_mask, migratetype); + gfp_mask, migratetype, cold);if (page)break ; this_zone_full:
@@ -1517,7 +1516,7 @@ static inline struct page * __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, enum zone_type high_zoneidx, nodemask_t *nodemask, struct zone *preferred_zone, - int migratetype) + int migratetype, int cold) { struct page *page;
@@ -1535,7 +1534,7 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order, page =get_page_from_freelist (gfp_mask|__GFP_HARDWALL, nodemask, order, zonelist, high_zoneidx,ALLOC_WMARK_HIGH |ALLOC_CPUSET , - preferred_zone, migratetype); + preferred_zone, migratetype, cold);if (page) goto out;
@@ -1556,7 +1555,7 @@ static inline struct page * __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, enum zone_type high_zoneidx, nodemask_t *nodemask, int alloc_flags, struct zone *preferred_zone, - int migratetype, unsigned long *did_some_progress) + int migratetype, int cold, unsigned long *did_some_progress) { struct page *page = NULL;struct reclaim_state reclaim_state;
@@ -1589,7 +1588,7 @@ __alloc_pages_direct_reclaim(gfp_t gfp_mask, unsigned int order, page =get_page_from_freelist (gfp_mask, nodemask, order, zonelist, high_zoneidx, alloc_flags, preferred_zone, - migratetype); + migratetype, cold);return page; }
@@ -1601,14 +1600,14 @@ static inline struct page * __alloc_pages_high_priority(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, enum zone_type high_zoneidx, nodemask_t *nodemask, struct zone *preferred_zone, - int migratetype) + int migratetype, int cold) { struct page *page; do { page = get_page_from_freelist(gfp_mask, nodemask, order, zonelist, high_zoneidx, ALLOC_NO_WATERMARKS, - preferred_zone, migratetype); + preferred_zone, migratetype, cold); if (!page && gfp_mask & __GFP_NOFAIL) congestion_wait(WRITE, HZ/50);
@@ -1668,7 +1667,7 @@ static inline struct page * __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order, struct zonelist *zonelist, enum zone_type high_zoneidx, nodemask_t *nodemask, struct zone *preferred_zone, - int migratetype) + int migratetype, int cold) { const gfp_t wait = gfp_mask & __GFP_WAIT;struct page *page =NULL ;
@@ -1701,7 +1700,7 @@ restart:/* This is the last chance, in general, before the goto nopage. */ page =get_page_from_freelist (gfp_mask, nodemask, order, zonelist, high_zoneidx, alloc_flags & ~ALLOC_NO_WATERMARKS , - preferred_zone, migratetype); + preferred_zone, migratetype, cold);if (page) goto got_pg;
@@ -1709,7 +1708,7 @@ restart:if (alloc_flags &ALLOC_NO_WATERMARKS ) { page =__alloc_pages_high_priority (gfp_mask, order, zonelist, high_zoneidx, nodemask, - preferred_zone, migratetype); + preferred_zone, migratetype, cold);if (page) goto got_pg; }
@@ -1727,7 +1726,8 @@ restart: zonelist, high_zoneidx, nodemask, alloc_flags, preferred_zone, - migratetype, &did_some_progress); + migratetype, cold, + &did_some_progress);if (page) goto got_pg;
@@ -1740,7 +1740,7 @@ restart: page =__alloc_pages_may_oom (gfp_mask, order, zonelist, high_zoneidx, nodemask, preferred_zone, - migratetype); + migratetype, cold);if (page) goto got_pg;
@@ -1780,6 +1780,7 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,struct zone *preferred_zone;struct page *page;int migratetype =allocflags_to_migratetype (gfp_mask); +int cold = gfp_mask & __GFP_COLD;might_sleep_if (gfp_mask & __GFP_WAIT);
@@ -1803,11 +1804,11 @@ __alloc_pages_nodemask(gfp_t gfp_mask, unsigned int order,/* First allocation attempt */ page =get_page_from_freelist (gfp_mask|__GFP_HARDWALL, nodemask, order, zonelist, high_zoneidx,ALLOC_WMARK_LOW |ALLOC_CPUSET , - preferred_zone, migratetype); + preferred_zone, migratetype, cold);if (unlikely (!page)) page =__alloc_pages_slowpath (gfp_mask, order, zonelist, high_zoneidx, nodemask, - preferred_zone, migratetype); + preferred_zone, migratetype, cold);return page; }
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 12/35 Remove a branch by assuming __GFP_HIGH == ALLOC_HIGH
In reply to Mel GormanAllocations that specify __GFP_HIGH get the ALLOC_HIGH flag. If these flags are equal to each other, we can eliminate a branch.
--- mm/page_alloc.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
mm/page_alloc.c +2 -2
@@ -1640,8 +1640,8 @@ gfp_to_alloc_flags(gfp_t gfp_mask)* policy or is asking for __GFP_HIGH memory. GFP_ATOMIC requests will * set both ALLOC_HARDER (!wait) and ALLOC_HIGH (__GFP_HIGH). */ -if (gfp_mask & __GFP_HIGH) - alloc_flags |=ALLOC_HIGH ; +VM_BUG_ON (__GFP_HIGH !=ALLOC_HIGH ); + alloc_flags |= (gfp_mask & __GFP_HIGH);if (!wait) { alloc_flags |=ALLOC_HARDER ;
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 13/35 Inline __rmqueue_smallest()
In reply to Mel GormanInline __rmqueue_smallest by altering flow very slightly so that there is only one call site. This allows the function to be inlined without additional text bloat.
--- mm/page_alloc.c | 23 ++++++++++++++++++----- 1 files changed, 18 insertions(+), 5 deletions(-)
mm/page_alloc.c +18 -5
@@ -664,7 +664,8 @@ static int prep_new_page(struct page *page, int order, gfp_t gfp_flags)* Go through the free lists for the given migratetype and remove * the smallest available page from the freelists */ -static struct page *__rmqueue_smallest (struct zone *zone,unsigned int order, +static inline +struct page *__rmqueue_smallest (struct zone *zone,unsigned int order,int migratetype) {unsigned int current_order;
@@ -834,24 +835,36 @@ static struct page *__rmqueue_fallback(struct zone *zone, int order, } } -/* Use MIGRATE_RESERVE rather than fail an allocation */ -return __rmqueue_smallest (zone, order,MIGRATE_RESERVE ); +return NULL ; }/* * Do the hard work of removing an element from the buddy allocator. * Call me with the zone->lock already held. */ -static struct page *__rmqueue (struct zone *zone,unsigned int order, +static inline +struct page *__rmqueue (struct zone *zone,unsigned int order,int migratetype) {struct page *page; +retry_reserve: page =__rmqueue_smallest (zone, order, migratetype); -if (unlikely (!page)) +if (unlikely (!page) && migratetype !=MIGRATE_RESERVE ) { page =__rmqueue_fallback (zone, order, migratetype); +/* +* Use MIGRATE_RESERVE rather than fail an allocation. goto +* is used because __rmqueue_smallest is an inline function +* and we want just one call site +*/ +if (!page) { + migratetype =MIGRATE_RESERVE ; + goto retry_reserve; + } + } +return page; }
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 14/35 Inline buffered_rmqueue()
In reply to Mel Gormanbuffered_rmqueue() is in the fast path so inline it. Because it only has one call site, this actually should reduce text bloat instead of increase it.
--- mm/page_alloc.c | 3 ++- 1 files changed, 2 insertions(+), 1 deletions(-)
mm/page_alloc.c +2 -1
@@ -1079,7 +1079,8 @@ void split_page(struct page *page, unsigned int order)* we cheat by calling it from here, in the order > 0 path. Saves a branch * or two. */ -static struct page *buffered_rmqueue (struct zone *preferred_zone, +static inline +struct page *buffered_rmqueue (struct zone *preferred_zone,struct zone *zone,int order,gfp_t gfp_flags,int migratetype,int cold) {
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 15/35 Inline __rmqueue_fallback()
In reply to Mel Gorman__rmqueue() is in the slow path but has only one call site. It actually reduces text if it's inlined.
--- mm/page_alloc.c | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-)
mm/page_alloc.c +2 -2
@@ -774,8 +774,8 @@ static int move_freepages_block(struct zone *zone, struct page *page, }/* Remove an element from the buddy allocator from the fallback list */ -static struct page *__rmqueue_fallback (struct zone *zone,int order, -int start_migratetype) +static inline struct page * +__rmqueue_fallback (struct zone *zone,int order,int start_migratetype) {struct free_area * area;int current_order;
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 15/35] Inline __rmqueue_fallback()
Quoted 2 lines · Mel Gorman
__rmqueue()is in the slow path but has only one call site. It actually reduces text if it's inlined.
This is modifying __rmqueue_fallback() not __rmqueue().
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 15/35] Inline __rmqueue_fallback()
Quoted 7 lines · Christoph Lameter
Quoted 2 lines · Mel Gorman
__rmqueue()is in the slow path but has only one call site. It actually reduces text if it's inlined.This is modifying
__rmqueue_fallback()not__rmqueue().
Fixed, thanks
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 16/35 Save text by reducing call sites of __rmqueue()
In reply to Mel Gorman__rmqueue is inlined in the fast path but it has two call sites, the low order and high order paths. However, a slight modification to the high-order path reduces the call sites of __rmqueue. This reduces text at the slight increase of complexity of the high-order allocation path.
--- mm/page_alloc.c | 11 +++++++---- 1 files changed, 7 insertions(+), 4 deletions(-)
mm/page_alloc.c +7 -4
@@ -1123,11 +1123,14 @@ again:list_del (&page->lru); pcp->count--; }else { -spin_lock_irqsave (&zone->lock, flags); - page =__rmqueue (zone, order, migratetype); -spin_unlock (&zone->lock); -if (!page) +LIST_HEAD (list); +local_irq_save (flags); + +/* Calling __rmqueue would bloat text, hence this */ +if (!rmqueue_bulk (zone, order,1 , &list, migratetype)) goto failed; + page = list_entry(list.next, struct page, lru); +list_del (&page->lru); }__count_zone_vm_events (PGALLOC , zone,1 << order);
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 17/35 Do not call get_pageblock_migratetype() more than necessary
In reply to Mel Gormanget_pageblock_migratetype() is potentially called twice for every page free. Once, when being freed to the pcp lists and once when being freed back to buddy. When freeing from the pcp lists, it is known what the pageblock type was at the time of free so use it rather than rechecking. In low memory situations under memory pressure, this might skew anti-fragmentation slightly but the interference is minimal and decisions that are fragmenting memory are being made anyway.
--- mm/page_alloc.c | 16 ++++++++++------ 1 files changed, 10 insertions(+), 6 deletions(-)
mm/page_alloc.c +10 -6
@@ -455,16 +455,18 @@ static inline int page_is_buddy(struct page *page, struct page *buddy,*/ static inline void __free_one_page (struct page *page, -struct zone *zone,unsigned int order) +struct zone *zone,unsigned int order, +int migratetype) {unsigned long page_idx;int order_size =1 << order; -int migratetype =get_pageblock_migratetype (page);if (unlikely (PageCompound (page)))if (unlikely (destroy_compound_page (page, order)))return ; +VM_BUG_ON (migratetype ==-1 ); + page_idx =page_to_pfn (page) & ((1 <<MAX_ORDER ) -1 );VM_BUG_ON (page_idx & (order_size -1 ));
@@ -533,17 +535,18 @@ static void free_pages_bulk(struct zone *zone, int count, page = list_entry(list->prev, struct page, lru);/* have to delete it as __free_one_page list manipulates */ list_del (&page->lru); -__free_one_page (page, zone, order); +__free_one_page (page, zone, order,page_private (page)); }spin_unlock (&zone->lock); } -static void free_one_page (struct zone *zone,struct page *page,int order) +static void free_one_page (struct zone *zone,struct page *page,int order, +int migratetype) {spin_lock (&zone->lock);zone_clear_flag (zone,ZONE_ALL_UNRECLAIMABLE ); zone->pages_scanned =0 ; -__free_one_page (page, zone, order); +__free_one_page (page, zone, order, migratetype);spin_unlock (&zone->lock); }
@@ -568,7 +571,8 @@ static void __free_pages_ok(struct page *page, unsigned int order)local_irq_save (flags);__count_vm_events (PGFREE ,1 << order); -free_one_page (page_zone (page), page, order); +free_one_page (page_zone (page), page, order, +get_pageblock_migratetype (page));local_irq_restore (flags); }
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 17/35] Do not call get_pageblock_migratetype() more than necessary
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 18/35 Do not disable interrupts in free_page_mlock()
In reply to Mel Gormanfree_page_mlock() tests and clears PG_mlocked using locked versions of the bit operations. If set, it disables interrupts to update counters and this happens on every page free even though interrupts are disabled very shortly afterwards a second time. This is wasteful.
This patch splits what free_page_mlock() does. The bit check is still made. However, the update of counters is delayed until the interrupts are disabled and the non-lock version for clearing the bit is used. One potential weirdness with this split is that the counters do not get updated if the bad_page() check is triggered but a system showing bad pages is getting screwed already.
--- mm/internal.h | 11 +++-------- mm/page_alloc.c | 8 +++++++- 2 files changed, 10 insertions(+), 9 deletions(-)
mm/internal.h +3 -8
@@ -155,14 +155,9 @@ static inline void mlock_migrate_page(struct page *newpage, struct page *page)*/ static inline void free_page_mlock (struct page *page) { -if (unlikely (TestClearPageMlocked (page))) { -unsigned long flags; - -local_irq_save (flags); -__dec_zone_page_state (page,NR_MLOCK ); -__count_vm_event (UNEVICTABLE_MLOCKFREED ); -local_irq_restore (flags); - } +__ClearPageMlocked (page); +__dec_zone_page_state (page,NR_MLOCK ); +__count_vm_event (UNEVICTABLE_MLOCKFREED ); }#else /* CONFIG_UNEVICTABLE_LRU */
mm/page_alloc.c +7 -1
@@ -498,7 +498,6 @@ static inline void __free_one_page(struct page *page,static inline int free_pages_check (struct page *page) { -free_page_mlock (page); if (unlikely(page_mapcount(page) | (page->mapping != NULL) | (page_count(page) != 0) |
@@ -555,6 +554,7 @@ static void __free_pages_ok(struct page *page, unsigned int order)unsigned long flags;int i;int bad =0 ; +int clearMlocked =PageMlocked (page);for (i =0 ; i < (1 << order) ; ++i) bad +=free_pages_check (page + i);
@@ -570,6 +570,8 @@ static void __free_pages_ok(struct page *page, unsigned int order)kernel_map_pages (page,1 << order,0 );local_irq_save (flags); +if (clearMlocked) +free_page_mlock (page);__count_vm_events (PGFREE ,1 << order);free_one_page (page_zone (page), page, order,get_pageblock_migratetype (page));
@@ -1020,6 +1022,7 @@ static void free_hot_cold_page(struct page *page, int cold)struct zone *zone =page_zone (page);struct per_cpu_pages *pcp;unsigned long flags; +int clearMlocked =PageMlocked (page);if (PageAnon (page)) page->mapping =NULL ;
@@ -1036,6 +1039,9 @@ static void free_hot_cold_page(struct page *page, int cold) pcp = &zone_pcp (zone,get_cpu ())->pcp;local_irq_save (flags);__count_vm_event (PGFREE ); +if (clearMlocked) +free_page_mlock (page); +if (cold)list_add_tail (&page->lru, &pcp->list);else
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 18/35] Do not disable interrupts in free_page_mlock()
mm/page_alloc.c static void __free_pages_ok(struct page *page, unsigned int order) · line 577
Quoted 9 lines · Mel Gorman
mm/page_alloc.c +2
-0@@ -570,6 +570,8 @@ static void __free_pages_ok(struct page *page, unsigned int order)kernel_map_pages (page,1 << order,0 );local_irq_save (flags); +if (clearMlocked) +free_page_mlock (page);__count_vm_events (PGFREE ,1 << order);free_one_page (page_zone (page), page, order,get_pageblock_migratetype (page));
Add an unlikely(clearMblocked) here?
mm/page_alloc.c static void free_hot_cold_page(struct page *page, int cold) · line 1047
Quoted 11 lines
mm/page_alloc.c +3
-0@@ -1036,6 +1039,9 @@ static void free_hot_cold_page(struct page *page, int cold) pcp = &zone_pcp (zone,get_cpu ())->pcp;local_irq_save (flags);__count_vm_event (PGFREE ); +if (clearMlocked) +free_page_mlock (page); +if (cold)list_add_tail (&page->lru, &pcp->list);else
Same here also make sure tha the __count_vm_events(PGFREE) comes after the free_pages_mlock() to preserve symmetry with __free_pages_ok() and maybe allow the compiler to do CSE between two invocations of __count_vm_events().
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 18/35] Do not disable interrupts in free_page_mlock()
Quoted 5 lines · Christoph Lameter
Quoted 3 lines · Mel Gorman
mm/page_alloc.c +0
-0@@ -570,6 +570,8 @@ static void __free_pages_ok(struct page *page, unsigned int order)kernel_map_pages (page,1 << order,0 );
Quoted 9 lines
mm/page_alloc.c static void __free_pages_ok(struct page *page, unsigned int order) · line 577
Quoted 6 lines
mm/page_alloc.c +2
-0local_irq_save (flags); +if (clearMlocked) +free_page_mlock (page);__count_vm_events (PGFREE ,1 << order);free_one_page (page_zone (page), page, order,get_pageblock_migratetype (page));Add an unlikely(clearMblocked) here?
I wasn't sure at the time of writing how likely the case really is but it makes sense that mlocked() pages are rarely freed. On reflection though, it makes sense to mark this unlikely().
Quoted 8 lines
Quoted 8 lines
mm/page_alloc.c +3
-0@@ -1036,6 +1039,9 @@ static void free_hot_cold_page(struct page *page, int cold) pcp = &zone_pcp (zone,get_cpu ())->pcp;local_irq_save (flags);__count_vm_event (PGFREE ); +if (clearMlocked) +free_page_mlock (page); +if (cold)
Quoted 9 lines
mm/page_alloc.c static void free_hot_cold_page(struct page *page, int cold) · line 1047
Quoted 3 lines
mm/page_alloc.c +0
-0list_add_tail (&page->lru, &pcp->list); elseSame here also make sure tha the
__count_vm_events(PGFREE)comes after thefree_pages_mlock()to preserve symmetry with__free_pages_ok()and maybe allow the compiler to do CSE between two invocations of__count_vm_events().
Whatever about the latter reasoning about CSE, the symmetry makes sense. I've made the change. Thanks
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 19/35 Do not setup zonelist cache when there is only one node
In reply to Mel GormanThere is a zonelist cache which is used to track zones that are not in the allowed cpuset or found to be recently full. This is to reduce cache footprint on large machines. On smaller machines, it just incurs cost for no gain. This patch only uses the zonelist cache when there are NUMA nodes.
--- mm/page_alloc.c | 10 ++++++++-- 1 files changed, 8 insertions(+), 2 deletions(-)
mm/page_alloc.c +8 -2
@@ -1442,6 +1442,8 @@ get_page_from_freelist(gfp_t gfp_mask, nodemask_t *nodemask, unsigned int order,/* Determine in advance if the zonelist needs filtering */ if ((alloc_flags &ALLOC_CPUSET ) &&unlikely (number_of_cpusets >1 )) zonelist_filter =1 ; +if (num_online_nodes () >1 ) + zonelist_filter =1 ; zonelist_scan:/*
@@ -1486,8 +1488,12 @@ this_zone_full:zlc_mark_zone_full (zonelist, z); try_next_zone:if (NUMA_BUILD && zonelist_filter) { -if (!did_zlc_setup) { -/* do zlc_setup after the first zone is tried */ +if (!did_zlc_setup &&num_online_nodes () >1 ) { +/* +* do zlc_setup after the first zone is tried +* but only if there are multiple nodes to make +* it worthwhile +*/ allowednodes =zlc_setup (zonelist, alloc_flags); zlc_active =1 ; }
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 19/35] Do not setup zonelist cache when there is only one node
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 20/35 Use a pre-calculated value for num_online_nodes()
In reply to Mel Gormannum_online_nodes() is called by the page allocator to decide whether the zonelist needs to be filtered based on cpusets or the zonelist cache. This is actually a heavy function and touches a number of cache lines. This patch stores the number of online nodes at boot time and when nodes get onlined and offlined.
--- include/linux/nodemask.h | 16 ++++++++++++++-- mm/page_alloc.c | 6 ++++-- 2 files changed, 18 insertions(+), 4 deletions(-)
include/linux/nodemask.h +14 -2
@@ -449,13 +449,25 @@ static inline int num_node_state(enum node_states state) node; \ }) +/* Recorded value for num_online_nodes() */ +extern int static_num_online_nodes; +#define num_online_nodes () num_node_state(N_ONLINE)#define num_possible_nodes () num_node_state(N_POSSIBLE)#define node_online (node) node_state((node), N_ONLINE)#define node_possible (node) node_state((node), N_POSSIBLE) -#define node_set_online (node) node_set_state((node), N_ONLINE) -#define node_set_offline (node) node_clear_state((node), N_ONLINE) +static inline void node_set_online (int nid) +{ +node_set_state (nid,N_ONLINE ); + static_num_online_nodes =num_node_state (N_ONLINE ); +} + +static inline void node_set_offline (int nid) +{ +node_clear_state (nid,N_ONLINE ); + static_num_online_nodes =num_node_state (N_ONLINE ); +}#define for_each_node (node) for_each_node_state(node, N_POSSIBLE)#define for_each_online_node (node) for_each_node_state(node, N_ONLINE)
mm/page_alloc.c +4 -2
@@ -70,6 +70,7 @@ EXPORT_SYMBOL(node_states);unsigned long totalram_pages__read_mostly ;unsigned long totalreserve_pages__read_mostly ;unsigned long highest_memmap_pfn__read_mostly ; +int static_num_online_nodes__read_mostly ;int percpu_pagelist_fraction;#ifdef CONFIG_HUGETLB_PAGE_SIZE_VARIABLE
@@ -1442,7 +1443,7 @@ get_page_from_freelist(gfp_t gfp_mask, nodemask_t *nodemask, unsigned int order,/* Determine in advance if the zonelist needs filtering */ if ((alloc_flags &ALLOC_CPUSET ) &&unlikely (number_of_cpusets >1 )) zonelist_filter =1 ; -if (num_online_nodes () >1 ) +if (static_num_online_nodes >1 ) zonelist_filter =1 ; zonelist_scan:
@@ -1488,7 +1489,7 @@ this_zone_full:zlc_mark_zone_full (zonelist, z); try_next_zone:if (NUMA_BUILD && zonelist_filter) { -if (!did_zlc_setup &&num_online_nodes () >1 ) { +if (!did_zlc_setup && static_num_online_nodes >1 ) {/* * do zlc_setup after the first zone is tried * but only if there are multiple nodes to make
@@ -2645,6 +2646,7 @@ void build_all_zonelists(void)else page_group_by_mobility_disabled =0 ; + static_num_online_nodes =num_node_state (N_ONLINE ); printk("Built %i zonelists in %s order, mobility grouping %s. " "Total pages: %ld\n", num_online_nodes(),
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 17 lines · On Monday 16 March 2009 20:46:15 Mel Gorman wrote:
num_online_nodes()is called by the page allocator to decide whether the zonelist needs to be filtered based on cpusets or the zonelist cache. This is actually a heavy function and touches a number of cache lines. This patch stores the number of online nodes at boot time and when nodes get onlined and offlined.Signed-off-by Mel Gorman <[email protected]>--- include/linux/nodemask.h | 16 ++++++++++++++-- mm/page_alloc.c | 6 ++++-- 2 files changed, 18 insertions(+), 4 deletions(-)include/linux/nodemask.h +0
-0
include/linux/nodemask.h static inline int num_node_state(enum node_states · line 453
Quoted 5 lines
include/linux/nodemask.h +2
-0@@ -449,13 +449,25 @@ static inline int num_node_state(enum node_states state) node; \ }) +/* Recorded value for num_online_nodes() */ +extern int static_num_online_nodes;
__read_mostly, please. Check this for any other place you've added global cachelines that are referenced by the allocator.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 21 lines · Nick Piggin
Quoted 20 lines · On Monday 16 March 2009 20:46:15 Mel Gorman wrote:
num_online_nodes()is called by the page allocator to decide whether the zonelist needs to be filtered based on cpusets or the zonelist cache. This is actually a heavy function and touches a number of cache lines. This patch stores the number of online nodes at boot time and when nodes get onlined and offlined.Signed-off-by Mel Gorman <[email protected]>--- include/linux/nodemask.h | 16 ++++++++++++++-- mm/page_alloc.c | 6 ++++-- 2 files changed, 18 insertions(+), 4 deletions(-)include/linux/nodemask.h +0
-0@@ -449,13 +449,25 @@ static inline int num_node_state(enum node_states state) node; \ })
Quoted 5 lines
include/linux/nodemask.h static inline int num_node_state(enum node_states state) · line 453
Quoted 2 lines
include/linux/nodemask.h +2
-0+/* Recorded value for num_online_nodes() */ +extern int static_num_online_nodes;
__read_mostly, please. Check this for any other place you've added global cachelines that are referenced by the allocator.
OK I'm blind, sorry ignore that :P
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
In reply to Mel GormanQuoted 1 line · Mel Gorman
+extern int
static_num_online_nodes;
Strange name. Could we name this nr_online_nodes or so?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 6 lines · Christoph Lameter
Quoted 1 line · Mel Gorman
+extern int
static_num_online_nodes;Strange name. Could we name this
nr_online_nodesor so?
It's to match the function name. Arguably I could also have replaced the implementation of num_online_nodes() with a version that uses the static variable.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 11 lines · Mel Gorman
Quoted 6 lines · Christoph Lameter
Quoted 1 line · Mel Gorman
+extern int
static_num_online_nodes;Strange name. Could we name this
nr_online_nodesor so?It's to match the function name. Arguably I could also have replaced the implementation of
num_online_nodes()with a version that uses the static variable.
We have nr_node_ids etc. It would be consistant with that naming.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 7 lines · Christoph Lameter
Quoted 5 lines · Mel Gorman
Quoted 4 lines · Christoph Lameter
Quoted 1 line · Mel Gorman
+extern int
static_num_online_nodes;
Quoted 9 lines
Quoted 6 lines
Quoted 2 lines
Strange name. Could we name this
nr_online_nodesor so?It's to match the function name. Arguably I could also have replaced the implementation of
num_online_nodes()with a version that uses the static variable.We have
nr_node_idsetc. It would be consistant with that naming.
Naming has never been great, but in this case the static value is a direct replacement of num_online_nodes(). I think having a similarly-named-but-still-different name obscures more than it helps.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 3 lines · Mel Gorman
Naming has never been great, but in this case the static value is a direct replacement of
num_online_nodes(). I think having a similarly-named-but-still-different name obscures more than it helps.
Creates a weird new name. Please use nr_online_nodes. Its useful elsewhere too.
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 9 lines · Christoph Lameter
Quoted 3 lines · Mel Gorman
Naming has never been great, but in this case the static value is a direct replacement of
num_online_nodes(). I think having a similarly-named-but-still-different name obscures more than it helps.Creates a weird new name. Please use
nr_online_nodes. Its useful elsewhere too.
Ok, I'm dropping this patch for the first pass altogether and will deal with it later.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 7 lines · Mel Gorman
Quoted 6 lines · Christoph Lameter
Quoted 3 lines · Mel Gorman
Naming has never been great, but in this case the static value is a direct replacement of
num_online_nodes(). I think having a similarly-named-but-still-different name obscures more than it helps.
Quoted 6 lines
Quoted 3 lines
Creates a weird new name. Please use
nr_online_nodes. Its useful elsewhere too.Ok, I'm dropping this patch for the first pass altogether and will deal with it later.
This is important infrastructure stuff. num_online_nodes is used in a couple of other places where it could be replaced by nr_online_nodes.
7 grufile.c gru_get_config_info 181 if (num_online_nodes() > 1 &&
8 grufile.c gru_get_config_info 187 info.nodes = num_online_nodes();
9 hugetlb.c return_unused_surplus_pages 878 unsigned long remaining_iterations = num_online_nodes();
a hugetlb.c return_unused_surplus_pages 907 remaining_iterations = num_online_nodes();
b page_alloc.c MAX_NODE_LOAD 2115 #define MAX_NODE_LOAD (num_online_nodes())
c page_alloc.c build_zonelists 2324 load = num_online_nodes();
d page_alloc.c build_all_zonelists 2475 num_online_nodes(),
e slub.c list_locations 3651 if (num_online_nodes() > 1 && !nodes_empty(l->nodes) &&
f svc.c svc_pool_map_choose_mode 127 if (num_online_nodes() > 1) {In other places its avoided because deemed to be too expensive.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Trying to the same in the style of nr_node_ids etc.
Subject: Provide nr_online_nodes and nr_possible_nodes
It seems that its beneficial to have a less expensive way to check for the number of currently active and possible nodes in a NUMA system. This will simplify further kernel optimizations for the cases in which only a single node is online or possible.
include/linux/nodemask.h +17 -3
@@ -408,6 +408,20 @@#define next_online_node (nid) next_node((nid), node_states[N_ONLINE])extern int nr_node_ids; +extern int nr_online_nodes; +extern int nr_possible_nodes; + +static inline void node_set_online (int nid) +{ +node_set_state (nid,N_ONLINE ); + nr_online_nodes =num_node_state (N_ONLINE ); +} + +static inline void node_set_offline (int nid) +{ +node_clear_state (nid,N_ONLINE ); + nr_online_nodes =num_node_state (N_ONLINE ); +}#else static inline int node_state (int node,enum node_states state)
@@ -434,7 +448,8 @@#define first_online_node 0#define next_online_node (nid) (MAX_NUMNODES)#define nr_node_ids 1 - +#define nr_online_nodes 1 +#define nr_possible_nodes 1#endif #define node_online_map node_states[N_ONLINE]
@@ -454,8 +469,7 @@#define node_online (node) node_state((node), N_ONLINE)#define node_possible (node) node_state((node), N_POSSIBLE) -#define node_set_online (node) node_set_state((node), N_ONLINE) -#define node_set_offline (node) node_clear_state((node), N_ONLINE) +#define for_each_node (node) for_each_node_state(node, N_POSSIBLE)#define for_each_online_node (node) for_each_node_state(node, N_ONLINE)
mm/hugetlb.c +2 -2
@@ -875,7 +875,7 @@* can no longer free unreserved surplus pages. This occurs when * the nodes with surplus pages have no free pages. */ -unsigned long remaining_iterations =num_online_nodes (); +unsigned long remaining_iterations = nr_online_nodes;/* Uncommit the reservation */ h->resv_huge_pages -= unused_resv_pages;
@@ -904,7 +904,7 @@ h->surplus_huge_pages--; h->surplus_huge_pages_node[nid]--; nr_pages--; - remaining_iterations =num_online_nodes (); + remaining_iterations = nr_online_nodes; } } }
mm/page_alloc.c +5 -1
@@ -168,6 +168,10 @@#if MAX_NUMNODES >1 int nr_node_ids__read_mostly =MAX_NUMNODES ;EXPORT_SYMBOL (nr_node_ids); +int nr_online_nodes__read_mostly =1 ; +EXPORT_SYMBOL (nr_online_nodes); +int nr_possible_nodes__read_mostly =MAX_NUMNODES ; +EXPORT_SYMBOL (nr_possible_nodes);#endif int page_group_by_mobility_disabled__read_mostly ;
@@ -2115,7 +2119,7 @@ } -#define MAX_NODE_LOAD (num_online_nodes()) +#define MAX_NODE_LOAD nr_online_nodesstatic int node_load[MAX_NUMNODES ];/**
mm/slab.c +2 -4
@@ -881,7 +881,6 @@*/ static int use_alien_caches__read_mostly =1 ; -static int numa_platform__read_mostly =1 ;static int __init noaliencache_setup (char *s) { use_alien_caches =0 ;
@@ -1434,9 +1433,8 @@int order;int node; -if (num_possible_nodes () ==1 ) { +if (nr_possible_nodes ==1 ) { use_alien_caches =0 ; - numa_platform =0 ; }for (i =0 ; i <NUM_INIT_LISTS ; i++) {
@@ -3526,7 +3524,7 @@* variable to skip the call, which is mostly likely to be present in * the cache. */ -if (numa_platform &&cache_free_alien (cachep, objp)) +if (nr_possible_nodes >1 &&cache_free_alien (cachep, objp))return ;if (likely (ac->avail < ac->limit)) {
mm/slub.c +1 -1
@@ -3648,7 +3648,7 @@ to_cpumask(l->cpus)); } -if (num_online_nodes () >1 && !nodes_empty (l->nodes) && +if (nr_online_nodes >1 && !nodes_empty (l->nodes) && len <PAGE_SIZE -60 ) { len +=sprintf (buf + len," nodes=" ); len += nodelist_scnprintf(buf + len, PAGE_SIZE - len - 50,
net/sunrpc/svc.c +1 -1
@@ -124,7 +124,7 @@ {unsigned int node; -if (num_online_nodes () >1 ) { +if (nr_online_nodes >1 ) {/* * Actually have multiple NUMA nodes, * so split pools on NUMA node boundaries
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 12 lines · Christoph Lameter
Trying to the same in the style of
nr_node_idsetc.Subject: Provide
nr_online_nodesandnr_possible_nodesIt seems that its beneficial to have a less expensive way to check for the number of currently active and possible nodes in a NUMA system. This will simplify further kernel optimizations for the cases in which only a single node is online or possible.
Signed-off-by Christoph Lameter <[email protected]>
heh, funningily enough we ended up doing something similar because I couldn't leave the cost of num_online_nodes() so high.
This patch actually alters the API. node_set_online() called when MAX_NUMNODES == 1 will now fail to compile. That situation wouldn't make any sense anyway but is it intentional?
For reference here is the patch I had for a similar goal which kept the API as it was. I'll drop it if you prefer your own version.
================
commit d767ff28677178659e3260b04b6535e608af68b7
Author: Mel Gorman <[email protected]> Date: Wed Mar 18 21:12:31 2009 +0000
Use a pre-calculated value for num_online_nodes()
num_online_nodes() is called in a number of places but most often by the page allocator when deciding whether the zonelist needs to be filtered based on cpusets or the zonelist cache. This is actually a heavy function and touches a number of cache lines. This patch stores the number of online nodes at boot time and updates the value when nodes get onlined and offlined.
include/linux/nodemask.h +11 -2
@@ -381,6 +381,10 @@ enum node_states {extern nodemask_t node_states[NR_NODE_STATES ];#if MAX_NUMNODES >1 + +extern int nr_node_ids; +extern int nr_online_nodes; +static inline int node_state (int node,enum node_states state) {return node_isset (node, node_states[state]);
@@ -389,11 +393,15 @@ static inline int node_state(int node, enum node_states state)static inline void node_set_state (int node,enum node_states state) {__node_set (node, &node_states[state]); +if (state ==N_ONLINE ) + nr_online_nodes =num_node_state (N_ONLINE ); }static inline void node_clear_state (int node,enum node_states state) {__node_clear (node, &node_states[state]); +if (state ==N_ONLINE ) + nr_online_nodes =num_node_state (N_ONLINE ); }static inline int num_node_state (enum node_states state)
@@ -407,7 +415,6 @@ static inline int num_node_state(enum node_states state)#define first_online_node first_node(node_states[N_ONLINE])#define next_online_node (nid) next_node((nid), node_states[N_ONLINE]) -extern int nr_node_ids;#else static inline int node_state (int node,enum node_states state)
@@ -434,6 +441,7 @@ static inline int num_node_state(enum node_states state)#define first_online_node 0#define next_online_node (nid) (MAX_NUMNODES)#define nr_node_ids 1 +#define nr_online_nodes 1#endif
@@ -449,7 +457,8 @@ static inline int num_node_state(enum node_states state) node; \ }) -#define num_online_nodes () num_node_state(N_ONLINE) + +#define num_online_nodes () (nr_online_nodes)#define num_possible_nodes () num_node_state(N_POSSIBLE)#define node_online (node) node_state((node), N_ONLINE)#define node_possible (node) node_state((node), N_POSSIBLE)
mm/page_alloc.c +2 -0
@@ -164,7 +164,9 @@ static unsigned long __meminitdata dma_reserve;#if MAX_NUMNODES >1 int nr_node_ids__read_mostly =MAX_NUMNODES ; +int nr_online_nodes__read_mostly ;EXPORT_SYMBOL (nr_node_ids); +EXPORT_SYMBOL (nr_online_nodes);#endif int page_group_by_mobility_disabled__read_mostly ;
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 3 lines · Mel Gorman
This patch actually alters the API.
node_set_online()called whenMAX_NUMNODES== 1 will now fail to compile. That situation wouldn't make any sense anyway but is it intentional?
Yes MAX_NUMNODES means that this is not a NUMA configuration. Setting an ode online would make no sense. Node 0 is always online.
Quoted 2 lines
For reference here is the patch I had for a similar goal which kept the API as it was. I'll drop it if you prefer your own version.
Lets look through it and get the best pieces from both.
include/linux/nodemask.h static inline int node_state(int node, enum node_states state) · line 398
Quoted 6 lines
static inline void
node_set_state(int node, enum node_states state){include/linux/nodemask.h +2
-0__node_set (node, &node_states[state]); +if (state ==N_ONLINE ) + nr_online_nodes =num_node_state (N_ONLINE ); }
That assumes uses of node_set_state N_ONLINE. Are there such users or are all using node_set_online()?
include/linux/nodemask.h static inline int num_node_state(enum node_states state) · line 464
Quoted 10 lines
include/linux/nodemask.h +2
-1@@ -449,7 +457,8 @@ static inline int num_node_state(enum node_states state) node; \ }) -#define num_online_nodes () num_node_state(N_ONLINE) + +#define num_online_nodes () (nr_online_nodes)#define num_possible_nodes () num_node_state(N_POSSIBLE)#define node_online (node) node_state((node), N_ONLINE)#define node_possible (node) node_state((node), N_POSSIBLE)
Hmmmm... Yes we could get rid of those.
I'd also like to see nr_possible_nodes(). nr_possible_nodes is important if you want to check if the system could ever bring up a second node (which would make the current optimization not viable) whereas nr_online_nodes is the check for how many nodes are currently online.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 9 lines · Christoph Lameter
Quoted 3 lines · Mel Gorman
This patch actually alters the API.
node_set_online()called whenMAX_NUMNODES== 1 will now fail to compile. That situation wouldn't make any sense anyway but is it intentional?Yes
MAX_NUMNODESmeans that this is not a NUMA configuration. Setting an ode online would make no sense. Node 0 is always online.
Right.
Quoted 5 lines
Quoted 2 lines
For reference here is the patch I had for a similar goal which kept the API as it was. I'll drop it if you prefer your own version.
Lets look through it and get the best pieces from both.
I posted an amalgamation. Sorry for the cross-over mails but I wanted to get tests going before I fell asleep. They take a few hours to complete.
Quoted 10 lines
include/linux/nodemask.h static inline int node_state(int node, enum node_states state) · line 398
Quoted 6 lines
static inline void
node_set_state(int node, enum node_states state){include/linux/nodemask.h +2
-0__node_set (node, &node_states[state]); +if (state ==N_ONLINE ) + nr_online_nodes =num_node_state (N_ONLINE ); }That assumes uses of
node_set_stateN_ONLINE. Are there such users or are all usingnode_set_online()?
node_set_online() calls node_set_state(node, N_ONLINE) so it should have worked out.
Quoted 13 lines
include/linux/nodemask.h static inline int num_node_state(enum node_states state) · line 464
Quoted 10 lines
include/linux/nodemask.h +2
-1@@ -449,7 +457,8 @@ static inline int num_node_state(enum node_states state) node; \ }) -#define num_online_nodes () num_node_state(N_ONLINE) + +#define num_online_nodes () (nr_online_nodes)#define num_possible_nodes () num_node_state(N_POSSIBLE)#define node_online (node) node_state((node), N_ONLINE)#define node_possible (node) node_state((node), N_POSSIBLE)Hmmmm... Yes we could get rid of those.
Quoted 5 lines
I'd also like to see
nr_possible_nodes().nr_possible_nodesis important if you want to check if the system could ever bring up a second node (which would make the current optimization not viable) whereasnr_online_nodesis the check for how many nodes are currently online.
I redid your patch to drop the nr_possible_nodes() because I couldn't convince myself it was correct in all cases and it isn't as important as avoiding num_online_nodes() in fast paths.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 14 lines · Mel Gorman
I posted an amalgamation. Sorry for the cross-over mails but I wanted to get tests going before I fell asleep. They take a few hours to complete.
Quoted 10 lines
include/linux/nodemask.h static inline int node_state(int node, enum node_states state) · line 398
Quoted 6 lines
static inline void
node_set_state(int node, enum node_states state){include/linux/nodemask.h +2
-0__node_set (node, &node_states[state]); +if (state ==N_ONLINE ) + nr_online_nodes =num_node_state (N_ONLINE ); }That assumes uses of
node_set_stateN_ONLINE. Are there such users or are all usingnode_set_online()?
Quoted 2 lines
node_set_online()callsnode_set_state(node, N_ONLINE)so it should have worked out.
But this adds a surprising side effect to all uses of node_set_state. Node_set_state is generating more code now.
Quoted 8 lines
Quoted 4 lines
if you want to check if the system could ever bring up a second node (which would make the current optimization not viable) whereas
nr_online_nodesis the check for how many nodes are currently online.I redid your patch to drop the
nr_possible_nodes()because I couldn't convince myself it was correct in all cases and it isn't as important as avoidingnum_online_nodes()in fast paths.
I was more thinking about getting the infrastructure right so that we can avoid future hacks like the one in slab.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 16 lines · Christoph Lameter
Quoted 14 lines · Mel Gorman
I posted an amalgamation. Sorry for the cross-over mails but I wanted to get tests going before I fell asleep. They take a few hours to complete.
Quoted 10 lines
include/linux/nodemask.h static inline int node_state(int node, enum node_states state) · line 398
Quoted 6 lines
static inline void
node_set_state(int node, enum node_states state){include/linux/nodemask.h +2
-0__node_set (node, &node_states[state]); +if (state ==N_ONLINE ) + nr_online_nodes =num_node_state (N_ONLINE ); }That assumes uses of
node_set_stateN_ONLINE. Are there such users or are all usingnode_set_online()?
Quoted 6 lines
Quoted 2 lines
node_set_online()callsnode_set_state(node, N_ONLINE)so it should have worked out.But this adds a surprising side effect to all uses of
node_set_state.Node_set_stateis generating more code now.
Fair point.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
if you want to check if the system could ever bring up a second node (which would make the current optimization not viable) whereas
nr_online_nodesis the check for how many nodes are currently online.
Quoted 7 lines
Quoted 3 lines
I redid your patch to drop the
nr_possible_nodes()because I couldn't convince myself it was correct in all cases and it isn't as important as avoidingnum_online_nodes()in fast paths.I was more thinking about getting the infrastructure right so that we can avoid future hacks like the one in slab.
Which is fair enough and you're right in that it's worth fixing. One horribly large patchset and associcate thread at a time though so I'll be putting it on the wrong finger rather than adding this to the pile right now :).
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
In reply to Christoph LameterQuoted 13 lines · Christoph Lameter
It seems that its beneficial to have a less expensive way to check for the number of currently active and possible nodes in a NUMA system. This will simplify further kernel optimizations for the cases in which only a single node is online or possible.
Signed-off-by Christoph Lameter <[email protected]>Index:
linux-2.6/include/linux/nodemask.h
===================================================================
include/linux/nodemask.h line 412
Quoted 5 lines
include/linux/nodemask.h +2
-0@@ -408,6 +408,20 @@#define next_online_node (nid) next_node((nid), node_states[N_ONLINE])extern int nr_node_ids; +extern int nr_online_nodes; +extern int nr_possible_nodes;
Have you tested the nr_possible_nodes aspects of this patch? I can see where it gets initialised but nothing that updates it. It would appear that nr_possible_nodes() and num_possible_nodes() can return different values.
Quoted 140 lines
include/linux/nodemask.h +15
-3+ +static inline void node_set_online (int nid) +{ +node_set_state (nid,N_ONLINE ); + nr_online_nodes =num_node_state (N_ONLINE ); +} + +static inline void node_set_offline (int nid) +{ +node_clear_state (nid,N_ONLINE ); + nr_online_nodes =num_node_state (N_ONLINE ); +}#else static inline int node_state (int node,enum node_states state)@@ -434,7 +448,8 @@#define first_online_node 0#define next_online_node (nid) (MAX_NUMNODES)#define nr_node_ids 1 - +#define nr_online_nodes 1 +#define nr_possible_nodes 1#endif #define node_online_map node_states[N_ONLINE]@@ -454,8 +469,7 @@#define node_online (node) node_state((node), N_ONLINE)#define node_possible (node) node_state((node), N_POSSIBLE) -#define node_set_online (node) node_set_state((node), N_ONLINE) -#define node_set_offline (node) node_clear_state((node), N_ONLINE) +#define for_each_node (node) for_each_node_state(node, N_POSSIBLE)#define for_each_online_node (node) for_each_node_state(node, N_ONLINE)Index:
linux-2.6/mm/hugetlb.c
===================================================================mm/hugetlb.c +2
-2@@ -875,7 +875,7 @@* can no longer free unreserved surplus pages. This occurs when * the nodes with surplus pages have no free pages. */ -unsigned long remaining_iterations =num_online_nodes (); +unsigned long remaining_iterations = nr_online_nodes;/* Uncommit the reservation */ h->resv_huge_pages -= unused_resv_pages;@@ -904,7 +904,7 @@ h->surplus_huge_pages--; h->surplus_huge_pages_node[nid]--; nr_pages--; - remaining_iterations =num_online_nodes (); + remaining_iterations = nr_online_nodes; } } }Index:
linux-2.6/mm/page_alloc.c
===================================================================mm/page_alloc.c +5
-1@@ -168,6 +168,10 @@#if MAX_NUMNODES >1 int nr_node_ids__read_mostly =MAX_NUMNODES ;EXPORT_SYMBOL (nr_node_ids); +int nr_online_nodes__read_mostly =1 ; +EXPORT_SYMBOL (nr_online_nodes); +int nr_possible_nodes__read_mostly =MAX_NUMNODES ; +EXPORT_SYMBOL (nr_possible_nodes);#endif int page_group_by_mobility_disabled__read_mostly ;@@ -2115,7 +2119,7 @@ } -#define MAX_NODE_LOAD (num_online_nodes()) +#define MAX_NODE_LOAD nr_online_nodesstatic int node_load[MAX_NUMNODES ];/** Index:
linux-2.6/mm/slab.c
===================================================================mm/slab.c +2
-4@@ -881,7 +881,6 @@*/ static int use_alien_caches__read_mostly =1 ; -static int numa_platform__read_mostly =1 ;static int __init noaliencache_setup (char *s) { use_alien_caches =0 ;@@ -1434,9 +1433,8 @@int order;int node; -if (num_possible_nodes () ==1 ) { +if (nr_possible_nodes ==1 ) { use_alien_caches =0 ; - numa_platform =0 ; }for (i =0 ; i <NUM_INIT_LISTS ; i++) {@@ -3526,7 +3524,7 @@* variable to skip the call, which is mostly likely to be present in * the cache. */ -if (numa_platform &&cache_free_alien (cachep, objp)) +if (nr_possible_nodes >1 &&cache_free_alien (cachep, objp))return ;if (likely (ac->avail < ac->limit)) {Index:
linux-2.6/mm/slub.c
===================================================================mm/slub.c +1
-1@@ -3648,7 +3648,7 @@ to_cpumask(l->cpus)); } -if (num_online_nodes () >1 && !nodes_empty (l->nodes) && +if (nr_online_nodes >1 && !nodes_empty (l->nodes) && len <PAGE_SIZE -60 ) { len +=sprintf (buf + len," nodes=" ); len += nodelist_scnprintf(buf + len, PAGE_SIZE - len - 50,Index:
linux-2.6/net/sunrpc/svc.c
===================================================================net/sunrpc/svc.c +1
-1@@ -124,7 +124,7 @@ {unsigned int node; -if (num_online_nodes () >1 ) { +if (nr_online_nodes >1 ) {/* * Actually have multiple NUMA nodes, * so split pools on NUMA node boundaries
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 8 lines · Mel Gorman
include/linux/nodemask.h line 412
Quoted 4 lines
extern int
nr_node_ids;include/linux/nodemask.h +2
-0+extern int nr_online_nodes; +extern int nr_possible_nodes;Have you tested the
nr_possible_nodesaspects of this patch? I can see where it gets initialised but nothing that updates it. It would appear thatnr_possible_nodes()andnum_possible_nodes()can return different values.
Right now we bypass the helper functions.... The only places where the possible map is modified are:
./arch/x86/mm/numa_64.c: node_set(0, node_possible_map);./arch/x86/mm/k8topology_64.c: node_set(nodeid, node_possible_map);./arch/x86/mm/srat_64.c: node_possible_map = nodes_parsed;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
In reply to Christoph LameterQuoted 2 lines · Christoph Lameter
Trying to the same in the style of
nr_node_idsetc.
Because of some issues with the patch and what it does for possible nodes, I reworked the patch slightly into the following and is what I'm actually testing.
commit 58652d22798591d4df5421a90d65cbd7e60b9048
Author: Christoph Lameter <[email protected]> Date: Thu Mar 19 21:34:17 2009 +0000
Use a pre-calculated value for num_online_nodes()
num_online_nodes() is called in a number of places but most often by the page allocator when deciding whether the zonelist needs to be filtered based on cpusets or the zonelist cache. This is actually a heavy function and touches a number of cache lines. This patch stores the number of online nodes at boot time and updates the value when nodes get onlined and offlined.
include/linux/nodemask.h +14 -1
@@ -408,6 +408,19 @@ static inline int num_node_state(enum node_states state)#define next_online_node (nid) next_node((nid), node_states[N_ONLINE])extern int nr_node_ids; +extern int nr_online_nodes; + +static inline void node_set_online (int nid) +{ +node_set_state (nid,N_ONLINE ); + nr_online_nodes =num_node_state (N_ONLINE ); +} + +static inline void node_set_offline (int nid) +{ +node_clear_state (nid,N_ONLINE ); + nr_online_nodes =num_node_state (N_ONLINE ); +}#else static inline int node_state (int node,enum node_states state)
@@ -434,7 +447,7 @@ static inline int num_node_state(enum node_states state)#define first_online_node 0#define next_online_node (nid) (MAX_NUMNODES)#define nr_node_ids 1 - +#define nr_online_nodes 1#endif #define node_online_map node_states[N_ONLINE]
mm/hugetlb.c +2 -2
@@ -875,7 +875,7 @@ static void return_unused_surplus_pages(struct hstate *h,* can no longer free unreserved surplus pages. This occurs when * the nodes with surplus pages have no free pages. */ -unsigned long remaining_iterations =num_online_nodes (); +unsigned long remaining_iterations = nr_online_nodes;/* Uncommit the reservation */ h->resv_huge_pages -= unused_resv_pages;
@@ -904,7 +904,7 @@ static void return_unused_surplus_pages(struct hstate *h, h->surplus_huge_pages--; h->surplus_huge_pages_node[nid]--; nr_pages--; - remaining_iterations =num_online_nodes (); + remaining_iterations = nr_online_nodes; } } }
mm/page_alloc.c +7 -5
@@ -164,7 +164,9 @@ static unsigned long __meminitdata dma_reserve;#if MAX_NUMNODES >1 int nr_node_ids__read_mostly =MAX_NUMNODES ; +int nr_online_nodes__read_mostly =1 ;EXPORT_SYMBOL (nr_node_ids); +EXPORT_SYMBOL (nr_online_nodes);#endif int page_group_by_mobility_disabled__read_mostly ;
@@ -1445,7 +1447,7 @@ get_page_from_freelist(gfp_t gfp_mask, nodemask_t *nodemask, unsigned int order,/* Determine in advance if the zonelist needs filtering */ if ((alloc_flags &ALLOC_CPUSET ) &&unlikely (number_of_cpusets >1 )) zonelist_filter =1 ; -if (num_online_nodes () >1 ) +if (nr_online_nodes >1 ) zonelist_filter =1 ; zonelist_scan:
@@ -1486,7 +1488,7 @@ this_zone_full:zlc_mark_zone_full (zonelist, z); try_next_zone:if (NUMA_BUILD && zonelist_filter) { -if (!did_zlc_setup &&num_online_nodes () >1 ) { +if (!did_zlc_setup && nr_online_nodes >1 ) {/* * do zlc_setup after the first zone is tried * but only if there are multiple nodes to make
@@ -2285,7 +2287,7 @@ int numa_zonelist_order_handler(ctl_table *table, int write, } -#define MAX_NODE_LOAD (num_online_nodes()) +#define MAX_NODE_LOAD (nr_online_nodes)static int node_load[MAX_NUMNODES ];/**
@@ -2494,7 +2496,7 @@ static void build_zonelists(pg_data_t *pgdat)/* NUMA-aware ordering of nodes */ local_node = pgdat->node_id; - load =num_online_nodes (); + load = nr_online_nodes; prev_node = local_node;nodes_clear (used_mask);
@@ -2645,7 +2647,7 @@ void build_all_zonelists(void)printk ("Built %i zonelists in %s order, mobility grouping %s. " "Total pages: %ld\n" , -num_online_nodes (), + nr_online_nodes, zonelist_order_name[current_zonelist_order], page_group_by_mobility_disabled ?"off" :"on" , vm_total_pages);
mm/slab.c +1 -1
@@ -3526,7 +3526,7 @@ static inline void __cache_free(struct kmem_cache *cachep, void *objp)* variable to skip the call, which is mostly likely to be present in * the cache. */ -if (numa_platform &&cache_free_alien (cachep, objp)) +if (numa_platform >1 &&cache_free_alien (cachep, objp))return ;if (likely (ac->avail < ac->limit)) {
mm/slub.c +1 -1
@@ -3648,7 +3648,7 @@ static int list_locations(struct kmem_cache *s, char *buf, to_cpumask(l->cpus)); } -if (num_online_nodes () >1 && !nodes_empty (l->nodes) && +if (nr_online_nodes >1 && !nodes_empty (l->nodes) && len <PAGE_SIZE -60 ) { len +=sprintf (buf + len," nodes=" ); len += nodelist_scnprintf(buf + len, PAGE_SIZE - len - 50,
net/sunrpc/svc.c +1 -1
@@ -124,7 +124,7 @@ svc_pool_map_choose_mode(void) {unsigned int node; -if (num_online_nodes () >1 ) { +if (nr_online_nodes >1 ) {/* * Actually have multiple NUMA nodes, * so split pools on NUMA node boundaries
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 7 lines · Mel Gorman
Quoted 2 lines · Christoph Lameter
Trying to the same in the style of
nr_node_idsetc.Because of some issues with the patch and what it does for possible nodes, I reworked the patch slightly into the following and is what I'm actually testing.
Ok. It also removes the slab bits etc.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 20/35] Use a pre-calculated value for num_online_nodes()
Quoted 12 lines · Christoph Lameter
Quoted 7 lines · Mel Gorman
Quoted 2 lines · Christoph Lameter
Trying to the same in the style of
nr_node_idsetc.Because of some issues with the patch and what it does for possible nodes, I reworked the patch slightly into the following and is what I'm actually testing.
Ok. It also removes the slab bits etc.
Well ... yes.
One of the slab changes removed a variable called numa_platform. From your patch, this appears to have some relation to nr_possible_nodes but it's not obvious at all if that is true or not.
The second change replaced num_possible_nodes() with nr_possible_nodes() but it wasn't clear this was equivalent because nr_possible_nodes() doesn't get updated from the call sites affecting the "possible" bitmap.
Both of those changes belong in a different patch and need explaination. The bits left alter just nr_online_nodes and use it where it's important.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 21/35 Do not check for compound pages during the page allocator sanity checks
In reply to Mel GormanA number of sanity checks are made on each page allocation and free including that the page count is zero. page_count() checks for compound pages and checks the count of the head page if true. However, in these paths, we do not care if the page is compound or not as the count of each tail page should also be zero.
This patch makes two changes to the use of page_count() in the free path. It converts one check of page_count() to a VM_BUG_ON() as the count should have been unconditionally checked earlier in the free path. It also avoids checking for compound pages.
--- mm/page_alloc.c | 6 +++--- 1 files changed, 3 insertions(+), 3 deletions(-)
mm/page_alloc.c +3 -3
@@ -425,7 +425,7 @@ static inline int page_is_buddy(struct page *page, struct page *buddy,return 0 ;if (PageBuddy (buddy) &&page_order (buddy) == order) { -BUG_ON (page_count (buddy) !=0 ); +VM_BUG_ON (page_count (buddy) !=0 );return 1 ; }return 0 ;
@@ -501,7 +501,7 @@ static inline int free_pages_check(struct page *page) {if (unlikely (page_mapcount (page) | (page->mapping !=NULL ) | - (page_count (page) !=0 ) | + (atomic_read (&page->_count) !=0 ) | (page->flags &PAGE_FLAGS_CHECK_AT_FREE ))) {bad_page (page);return 1 ;
@@ -646,7 +646,7 @@ static int prep_new_page(struct page *page, int order, gfp_t gfp_flags) {if (unlikely (page_mapcount (page) | (page->mapping !=NULL ) | - (page_count (page) !=0 ) | + (atomic_read (&page->_count) !=0 ) | (page->flags &PAGE_FLAGS_CHECK_AT_PREP ))) {bad_page (page);return 1 ;
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 21/35] Do not check for compound pages during the page allocator sanity checks
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 22/35 Use allocation flags as an index to the zone watermark
In reply to Mel GormanALLOC_WMARK_MIN, ALLOC_WMARK_LOW and ALLOC_WMARK_HIGH determin whether pages_min, pages_low or pages_high is used as the zone watermark when allocating the pages. Two branches in the allocator hotpath determine which watermark to use. This patch uses the flags as an array index and places the three watermarks in a union with an array so it can be offset. This means the flags can be used as an array index and reduces the branches taken.
--- include/linux/mmzone.h | 8 +++++++- mm/page_alloc.c | 18 ++++++++---------- 2 files changed, 15 insertions(+), 11 deletions(-)
include/linux/mmzone.h +7 -1
@@ -275,7 +275,13 @@ struct zone_reclaim_stat {struct zone {/* Fields commonly accessed by the page allocator */ -unsigned long pages_min, pages_low, pages_high; +union { +struct { +unsigned long pages_min, pages_low, pages_high; + }; +unsigned long pages_mark[3 ]; + }; +/* * We don't know if the memory that we're going to allocate will be freeable * or/and it will be released eventually, so to avoid totally wasting several
mm/page_alloc.c +8 -10
@@ -1160,10 +1160,13 @@ failed:return NULL ; } -#define ALLOC_NO_WATERMARKS 0x01/* don't check watermarks at all */ -#define ALLOC_WMARK_MIN 0x02/* use pages_min watermark */ -#define ALLOC_WMARK_LOW 0x04/* use pages_low watermark */ -#define ALLOC_WMARK_HIGH 0x08/* use pages_high watermark */ +/* The WMARK bits are used as an index zone->pages_mark */ +#define ALLOC_WMARK_MIN 0x00/* use pages_min watermark */ +#define ALLOC_WMARK_LOW 0x01/* use pages_low watermark */ +#define ALLOC_WMARK_HIGH 0x02/* use pages_high watermark */ +#define ALLOC_NO_WATERMARKS 0x08/* don't check watermarks at all */ +#define ALLOC_WMARK_MASK 0x07/* Mask to get the watermark bits */ +#define ALLOC_HARDER 0x10/* try to alloc harder */ #define ALLOC_HIGH 0x20/* __GFP_HIGH set */ #ifdef CONFIG_CPUSETS
@@ -1466,12 +1469,7 @@ zonelist_scan:if (!(alloc_flags &ALLOC_NO_WATERMARKS )) {unsigned long mark; -if (alloc_flags &ALLOC_WMARK_MIN ) - mark = zone->pages_min; -else if (alloc_flags &ALLOC_WMARK_LOW ) - mark = zone->pages_low; -else - mark = zone->pages_high; + mark = zone->pages_mark[alloc_flags &ALLOC_WMARK_MASK ];if (!zone_watermark_ok (zone, order, mark, classzone_idx, alloc_flags)) { if (!zone_reclaim_mode ||
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 22/35] Use allocation flags as an index to the zone watermark
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 23/35 Update NR_FREE_PAGES only as necessary
In reply to Mel GormanWhen pages are being freed to the buddy allocator, the zone NR_FREE_PAGES counter must be updated. In the case of bulk per-cpu page freeing, it's updated once per page. This retouches cache lines more than necessary. Update the counters one per per-cpu bulk free.
--- mm/page_alloc.c | 12 ++++++------ 1 files changed, 6 insertions(+), 6 deletions(-)
mm/page_alloc.c +6 -6
@@ -460,7 +460,6 @@ static inline void __free_one_page(struct page *page, int migratetype) {unsigned long page_idx; -int order_size =1 << order;if (unlikely (PageCompound (page)))if (unlikely (destroy_compound_page (page, order)))
@@ -470,10 +469,9 @@ static inline void __free_one_page(struct page *page, page_idx =page_to_pfn (page) & ((1 <<MAX_ORDER ) -1 ); -VM_BUG_ON (page_idx & (order_size -1 )); +VM_BUG_ON (page_idx & ((1 << order) -1 ));VM_BUG_ON (bad_range (zone, page)); -__mod_zone_page_state (zone,NR_FREE_PAGES , order_size);while (order <MAX_ORDER -1 ) {unsigned long combined_idx;struct page *buddy;
@@ -528,6 +526,8 @@ static void free_pages_bulk(struct zone *zone, int count,spin_lock (&zone->lock);zone_clear_flag (zone,ZONE_ALL_UNRECLAIMABLE ); zone->pages_scanned =0 ; + +__mod_zone_page_state (zone,NR_FREE_PAGES , count);while (count--) {struct page *page;
@@ -546,6 +546,8 @@ static void free_one_page(struct zone *zone, struct page *page, int order,spin_lock (&zone->lock);zone_clear_flag (zone,ZONE_ALL_UNRECLAIMABLE ); zone->pages_scanned =0 ; + +__mod_zone_page_state (zone,NR_FREE_PAGES ,1 << order);__free_one_page (page, zone, order, migratetype);spin_unlock (&zone->lock); }
@@ -690,7 +692,6 @@ struct page *__rmqueue_smallest(struct zone *zone, unsigned int order,list_del (&page->lru);rmv_page_order (page); area->nr_free--; -__mod_zone_page_state (zone,NR_FREE_PAGES , - (1UL << order));expand (zone, page, order, current_order, area, migratetype);return page; }
@@ -830,8 +831,6 @@ __rmqueue_fallback(struct zone *zone, int order, int start_migratetype)/* Remove the page from the freelists */ list_del (&page->lru);rmv_page_order (page); -__mod_zone_page_state (zone,NR_FREE_PAGES , - -(1UL << order));if (current_order == pageblock_order) set_pageblock_migratetype(page,
@@ -905,6 +904,7 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,set_page_private (page, migratetype); list = &page->lru; } +__mod_zone_page_state (zone,NR_FREE_PAGES , -(1UL << order) * i);spin_unlock (&zone->lock);return i; }
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 23/35] Update NR_FREE_PAGES only as necessary
Quoted 4 lines · Mel Gorman
When pages are being freed to the buddy allocator, the zone
NR_FREE_PAGEScounter must be updated. In the case of bulk per-cpu page freeing, it's updated once per page. This retouches cache lines more than necessary. Update the counters one per per-cpu bulk free.
Not sure about the reasoning here since the individual updates are batched and you are touching the same cacheline as the pcp you are operating on and have to touch anyways.
But if its frequent that __rmqueue_smallest() and free_pages_bulk() are called with multiple pages then its always a win.
mm/page_alloc.c static int rmqueue_bulk(struct zone *zone, unsigned int order, · line 907
Quoted 1 line
mm/page_alloc.c +1
-0+__mod_zone_page_state (zone,NR_FREE_PAGES , -(1UL << order) * i);
A multiplication? Okay with contemporary cpus I guess.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 23/35] Update NR_FREE_PAGES only as necessary
Quoted 8 lines · Christoph Lameter
Quoted 4 lines · Mel Gorman
When pages are being freed to the buddy allocator, the zone
NR_FREE_PAGEScounter must be updated. In the case of bulk per-cpu page freeing, it's updated once per page. This retouches cache lines more than necessary. Update the counters one per per-cpu bulk free.Not sure about the reasoning here since the individual updates are batched
Each update take places between lots of other work with different cache lines. With enough buddy merging, I believed the line holding the counters could at least get pushed out of L1 although probably not L2 cache.
Quoted 6 lines
and you are touching the same cacheline as the pcp you are operating on and have to touch anyways.
But if its frequent that
__rmqueue_smallest()andfree_pages_bulk()are called with multiple pages then its always a win.
It's frequent enough that it showed up in profiles
Quoted 6 lines
Reviewed-by Christoph Lameter <[email protected]>mm/page_alloc.c static int rmqueue_bulk(struct zone *zone, unsigned int order, · line 907
Quoted 1 line
mm/page_alloc.c +1
-0+__mod_zone_page_state (zone,NR_FREE_PAGES , -(1UL << order) * i);A multiplication? Okay with contemporary cpus I guess.
Replaced with
__mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Re: [PATCH 23/35] Update NR_FREE_PAGES only as necessary
Quoted 3 lines · Mel Gorman
Replaced with
__mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));
A later patch does that also.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 23/35] Update NR_FREE_PAGES only as necessary
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Mel Gorman
Replaced with
__mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order));A later patch does that also.
Silly of me. That later patch is in the "controversial" pile but the change is now moved back here where it belongs.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 24/35 Convert gfp_zone() to use a table of precalculated values
In reply to Mel GormanEvery page allocation uses gfp_zone() to calcuate what the highest zone allowed by a combination of GFP flags is. This is a large number of branches to have in a fast path. This patch replaces the branches with a lookup table that is calculated at boot-time and stored in the read-mostly section so it can be shared. This requires __GFP_MOVABLE to be redefined but it's debatable as to whether it should be considered a zone modifier or not.
--- include/linux/gfp.h | 28 +++++++++++----------------- init/main.c | 1 + mm/page_alloc.c | 36 +++++++++++++++++++++++++++++++++++- 3 files changed, 47 insertions(+), 18 deletions(-)
include/linux/gfp.h +11 -17
@@ -16,6 +16,10 @@ struct vm_area_struct;* Do not put any conditional on these. If necessary modify the definitions * without the underscores and use the consistently. The definitions here may * be used in bit comparisons. +* +* Note that __GFP_MOVABLE uses the next available bit but it is not +* a zone modifier. It uses the fourth bit so that the calculation of +* gfp_zone() can use a table rather than a series of comparisons */ #define __GFP_DMA ((__force gfp_t)0x01u)#define __GFP_HIGHMEM ((__force gfp_t)0x02u)
@@ -50,7 +54,7 @@ struct vm_area_struct;#define __GFP_HARDWALL ((__force gfp_t)0x20000u)/* Enforce hardwall cpuset memory allocs */ #define __GFP_THISNODE ((__force gfp_t)0x40000u)/* No fallback, no policies */ #define __GFP_RECLAIMABLE ((__force gfp_t)0x80000u)/* Page is reclaimable */ -#define __GFP_MOVABLE ((__force gfp_t)0x100000u)/* Page is movable */ +#define __GFP_MOVABLE ((__force gfp_t)0x08u)/* Page is movable */ #define __GFP_BITS_SHIFT 21/* Room for 21 __GFP_FOO bits */ #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
@@ -77,6 +81,9 @@ struct vm_area_struct;#define GFP_THISNODE ((__force gfp_t)0)#endif +/* This is a mask of all modifiers affecting gfp_zonemask() */ +#define GFP_ZONEMASK (__GFP_DMA | __GFP_HIGHMEM | __GFP_DMA32 | __GFP_MOVABLE) +/* This mask makes up all the page movable related flags */ #define GFP_MOVABLE_MASK (__GFP_RECLAIMABLE|__GFP_MOVABLE)
@@ -112,24 +119,11 @@ static inline int allocflags_to_migratetype(gfp_t gfp_flags) ((gfp_flags & __GFP_RECLAIMABLE) !=0 ); } +extern int gfp_zone_table[GFP_ZONEMASK ]; +void init_gfp_zone_table (void );static inline enum zone_type gfp_zone (gfp_t flags) { -#ifdef CONFIG_ZONE_DMA -if (flags & __GFP_DMA) -return ZONE_DMA ; -#endif -#ifdef CONFIG_ZONE_DMA32 -if (flags & __GFP_DMA32) -return ZONE_DMA32 ; -#endif -if ((flags & (__GFP_HIGHMEM | __GFP_MOVABLE)) == - (__GFP_HIGHMEM | __GFP_MOVABLE)) -return ZONE_MOVABLE ; -#ifdef CONFIG_HIGHMEM -if (flags & __GFP_HIGHMEM) -return ZONE_HIGHMEM ; -#endif -return ZONE_NORMAL ; +return gfp_zone_table[flags &GFP_ZONEMASK ]; }/*
init/main.c +1 -0
@@ -573,6 +573,7 @@ asmlinkage void __init start_kernel(void)* fragile until we cpu_idle() for the first time. */ preempt_disable (); +init_gfp_zone_table ();build_all_zonelists ();page_alloc_init ();printk (KERN_NOTICE "Kernel command line: %s\n" , boot_command_line);
mm/page_alloc.c +35 -1
@@ -70,6 +70,7 @@ EXPORT_SYMBOL(node_states);unsigned long totalram_pages__read_mostly ;unsigned long totalreserve_pages__read_mostly ;unsigned long highest_memmap_pfn__read_mostly ; +int gfp_zone_table[GFP_ZONEMASK ]__read_mostly ;int static_num_online_nodes__read_mostly ;int percpu_pagelist_fraction;
@@ -4569,7 +4570,7 @@ static void setup_per_zone_inactive_ratio(void)* 8192MB: 11584k * 16384MB: 16384k */ -static int __init init_per_zone_pages_min (void ) +static int init_per_zone_pages_min (void ) {unsigned long lowmem_kbytes;
@@ -4587,6 +4588,39 @@ static int __init init_per_zone_pages_min(void) }module_init (init_per_zone_pages_min) +static inline int __init gfp_flags_to_zone(gfp_t flags) +{ +#ifdef CONFIG_ZONE_DMA +if (flags & __GFP_DMA) +return ZONE_DMA ; +#endif +#ifdef CONFIG_ZONE_DMA32 +if (flags & __GFP_DMA32) +return ZONE_DMA32 ; +#endif +if ((flags & (__GFP_HIGHMEM | __GFP_MOVABLE)) == + (__GFP_HIGHMEM | __GFP_MOVABLE)) +return ZONE_MOVABLE ; +#ifdef CONFIG_HIGHMEM +if (flags & __GFP_HIGHMEM) +return ZONE_HIGHMEM ; +#endif +return ZONE_NORMAL ; +} + +/* +* For each possible combination of zone modifier flags, we calculate +* what zone it should be using. This consumes a cache line in most +* cases but avoids a number of branches in the allocator fast path +*/ +void __init init_gfp_zone_table (void ) +{ +gfp_t gfp_flags; + +for (gfp_flags =0 ; gfp_flags <GFP_ZONEMASK ; gfp_flags++) + gfp_zone_table[gfp_flags] =gfp_flags_to_zone (gfp_flags); +} +/* * min_free_kbytes_sysctl_handler - just a wrapper around proc_dointvec() so * that we can call two helper functions whenever min_free_kbytes
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 24/35] Convert gfp_zone() to use a table of precalculated values
Quoted 60 lines · Mel Gorman
init/main.c +1
-0@@ -573,6 +573,7 @@ asmlinkage void __init start_kernel(void)* fragile until we cpu_idle() for the first time. */ preempt_disable (); +init_gfp_zone_table ();build_all_zonelists ();page_alloc_init ();printk (KERN_NOTICE "Kernel command line: %s\n" , boot_command_line);mm/page_alloc.c +25
-1@@ -70,6 +70,7 @@ EXPORT_SYMBOL(node_states);unsigned long totalram_pages__read_mostly ;unsigned long totalreserve_pages__read_mostly ;unsigned long highest_memmap_pfn__read_mostly ; +int gfp_zone_table[GFP_ZONEMASK ]__read_mostly ;int static_num_online_nodes__read_mostly ;int percpu_pagelist_fraction;@@ -4569,7 +4570,7 @@ static void setup_per_zone_inactive_ratio(void)* 8192MB: 11584k * 16384MB: 16384k */ -static int __init init_per_zone_pages_min (void ) +static int init_per_zone_pages_min (void ) {unsigned long lowmem_kbytes;@@ -4587,6 +4588,39 @@ static int __init init_per_zone_pages_min(void) }module_init (init_per_zone_pages_min) +static inline int __init gfp_flags_to_zone(gfp_t flags) +{ +#ifdef CONFIG_ZONE_DMA +if (flags & __GFP_DMA) +return ZONE_DMA ; +#endif +#ifdef CONFIG_ZONE_DMA32 +if (flags & __GFP_DMA32) +return ZONE_DMA32 ; +#endif +if ((flags & (__GFP_HIGHMEM | __GFP_MOVABLE)) == + (__GFP_HIGHMEM | __GFP_MOVABLE)) +return ZONE_MOVABLE ; +#ifdef CONFIG_HIGHMEM +if (flags & __GFP_HIGHMEM) +return ZONE_HIGHMEM ; +#endif +return ZONE_NORMAL ; +} + +/* +* For each possible combination of zone modifier flags, we calculate +* what zone it should be using. This consumes a cache line in most
mm/page_alloc.c static int __init init_per_zone_pages_min(void) · line 4623
Quoted 10 lines
mm/page_alloc.c +10
-0+* cases but avoids a number of branches in the allocator fast path +*/ +void __init init_gfp_zone_table (void ) +{ +gfp_t gfp_flags; + +for (gfp_flags =0 ; gfp_flags <GFP_ZONEMASK ; gfp_flags++) + gfp_zone_table[gfp_flags] =gfp_flags_to_zone (gfp_flags); +} +
This is all known at compile time. The table can be calculated at compile time with some ifdefs and then we do not need the init_gfp_zone_table().
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 24/35] Convert gfp_zone() to use a table of precalculated values
Quoted 67 lines · Christoph Lameter
Quoted 65 lines · Mel Gorman
init/main.c +1
-0@@ -573,6 +573,7 @@ asmlinkage void __init start_kernel(void)* fragile until we cpu_idle() for the first time. */ preempt_disable (); +init_gfp_zone_table ();build_all_zonelists ();page_alloc_init ();printk (KERN_NOTICE "Kernel command line: %s\n" , boot_command_line);mm/page_alloc.c +30
-1@@ -70,6 +70,7 @@ EXPORT_SYMBOL(node_states);unsigned long totalram_pages__read_mostly ;unsigned long totalreserve_pages__read_mostly ;unsigned long highest_memmap_pfn__read_mostly ; +int gfp_zone_table[GFP_ZONEMASK ]__read_mostly ;int static_num_online_nodes__read_mostly ;int percpu_pagelist_fraction;@@ -4569,7 +4570,7 @@ static void setup_per_zone_inactive_ratio(void)* 8192MB: 11584k * 16384MB: 16384k */ -static int __init init_per_zone_pages_min (void ) +static int init_per_zone_pages_min (void ) {unsigned long lowmem_kbytes;@@ -4587,6 +4588,39 @@ static int __init init_per_zone_pages_min(void) }module_init (init_per_zone_pages_min) +static inline int __init gfp_flags_to_zone(gfp_t flags) +{ +#ifdef CONFIG_ZONE_DMA +if (flags & __GFP_DMA) +return ZONE_DMA ; +#endif +#ifdef CONFIG_ZONE_DMA32 +if (flags & __GFP_DMA32) +return ZONE_DMA32 ; +#endif +if ((flags & (__GFP_HIGHMEM | __GFP_MOVABLE)) == + (__GFP_HIGHMEM | __GFP_MOVABLE)) +return ZONE_MOVABLE ; +#ifdef CONFIG_HIGHMEM +if (flags & __GFP_HIGHMEM) +return ZONE_HIGHMEM ; +#endif +return ZONE_NORMAL ; +} + +/* +* For each possible combination of zone modifier flags, we calculate +* what zone it should be using. This consumes a cache line in most +* cases but avoids a number of branches in the allocator fast path +*/ +void __init init_gfp_zone_table (void ) +{ +gfp_t gfp_flags;
Quoted 9 lines
mm/page_alloc.c static int __init init_per_zone_pages_min(void) · line 4623
Quoted 5 lines
mm/page_alloc.c +5
-0+ +for (gfp_flags =0 ; gfp_flags <GFP_ZONEMASK ; gfp_flags++) + gfp_zone_table[gfp_flags] =gfp_flags_to_zone (gfp_flags); +} +This is all known at compile time. The table can be calculated at compile time with some ifdefs and then we do not need the
init_gfp_zone_table().
The last discussion didn't come up with something that could generate such a table and still be correct in all cases. I guess someone could do it all by hand but it was going to be an error prone exercise. I left the patch as-is as a result.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 25/35 Re-sort GFP flags and fix whitespace alignment for easier reading.
In reply to Mel GormanResort the GFP flags after __GFP_MOVABLE got redefined so how the bits are used are a bit cleared.
From: Peter Zijlstra <[email protected]>
--- include/linux/gfp.h | 9 +++++---- 1 files changed, 5 insertions(+), 4 deletions(-)
include/linux/gfp.h +5 -4
@@ -25,6 +25,8 @@ struct vm_area_struct;#define __GFP_HIGHMEM ((__force gfp_t)0x02u)#define __GFP_DMA32 ((__force gfp_t)0x04u) +#define __GFP_MOVABLE ((__force gfp_t)0x08u)/* Page is movable */ +/* * Action modifiers - doesn't change the zoning *
@@ -50,11 +52,10 @@ struct vm_area_struct;#define __GFP_NORETRY ((__force gfp_t)0x1000u)/* See above */ #define __GFP_COMP ((__force gfp_t)0x4000u)/* Add compound page metadata */ #define __GFP_ZERO ((__force gfp_t)0x8000u)/* Return zeroed page on success */ -#define __GFP_NOMEMALLOC ((__force gfp_t)0x10000u)/* Don't use emergency reserves */ -#define __GFP_HARDWALL ((__force gfp_t)0x20000u)/* Enforce hardwall cpuset memory allocs */ -#define __GFP_THISNODE ((__force gfp_t)0x40000u)/* No fallback, no policies */ +#define __GFP_NOMEMALLOC ((__force gfp_t)0x10000u)/* Don't use emergency reserves */ +#define __GFP_HARDWALL ((__force gfp_t)0x20000u)/* Enforce hardwall cpuset memory allocs */ +#define __GFP_THISNODE ((__force gfp_t)0x40000u)/* No fallback, no policies */ #define __GFP_RECLAIMABLE ((__force gfp_t)0x80000u)/* Page is reclaimable */ -#define __GFP_MOVABLE ((__force gfp_t)0x08u)/* Page is movable */ #define __GFP_BITS_SHIFT 21/* Room for 21 __GFP_FOO bits */ #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1))
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 26/35 Use the per-cpu allocator for orders up to PAGE_ALLOC_COSTLY_ORDER
In reply to Mel GormanThe per-cpu allocator is used to store order-0 pages for fast allocation without using the zone locks. There are a number of cases where order-1 allocations are frequent. Obviously there are 8K stacks, but also the signal handlers can be sufficiently large as well as some networking-related structures.
This patch allows orders up to PAGE_ALLOC_COSTLY_ORDER to be stored on the per-cpu lists which should help workloads making frequent order-1 allocations such as fork-heavy workloads on x86-64. It is somewhat simplified in that no splitting occurs of high-order pages on the lists but care is taken to account for the larger size of those pages so that the same amount of memory can end up on the per-cpu lists. If a high-order allocation would fail, then the lists get drained so it is not expected to cause any unusual OOM problems.
--- mm/page_alloc.c | 118 +++++++++++++++++++++++++++++++++++++------------------ 1 files changed, 80 insertions(+), 38 deletions(-)
mm/page_alloc.c +80 -38
@@ -469,6 +469,7 @@ static inline void __free_one_page(struct page *page,VM_BUG_ON (migratetype ==-1 ); page_idx =page_to_pfn (page) & ((1 <<MAX_ORDER ) -1 ); + page->index =0 ;VM_BUG_ON (page_idx & ((1 << order) -1 ));VM_BUG_ON (bad_range (zone, page));
@@ -510,8 +511,31 @@ static inline int free_pages_check(struct page *page)return 0 ; } +static inline void rmv_pcp_page (struct per_cpu_pages *pcp,struct page *page) +{ +list_del (&page->lru); + pcp->count -=1 << page->index; +} + +static inline void add_pcp_page (struct per_cpu_pages *pcp, +struct page *page, +int cold) +{ +if (cold) +list_add_tail (&page->lru, &pcp->list); +else +list_add (&page->lru, &pcp->list); + pcp->count +=1 << page->index; +} + +static inline void bulk_add_pcp_page (struct per_cpu_pages *pcp, +int order,int count) +{ + pcp->count += count << order; +} +/* -* Frees a list of pages. +* Frees a number of pages from the PCP lists * Assumes all pages on list are in same zone, and of same order. * count is the number of pages to free. *
@@ -522,23 +546,28 @@ static inline int free_pages_check(struct page *page)* pinned" detection logic. */ static void free_pages_bulk (struct zone *zone,int count, -struct list_head *list,int order) +struct per_cpu_pages *pcp) { +unsigned int freed =0 ; +struct list_head *list = &pcp->list; +spin_lock (&zone->lock);zone_clear_flag (zone,ZONE_ALL_UNRECLAIMABLE ); zone->pages_scanned =0 ; -__mod_zone_page_state (zone,NR_FREE_PAGES , count); -while (count--) { +while (freed < count) {struct page *page;VM_BUG_ON (list_empty (list)); page = list_entry(list->prev, struct page, lru); -/* have to delete it as __free_one_page list manipulates */ -list_del (&page->lru); -__free_one_page (page, zone, order,page_private (page)); +rmv_pcp_page (pcp, page); + + freed +=1 << page->index; +__free_one_page (page, zone, page->index,page_private (page)); }spin_unlock (&zone->lock); + +__mod_zone_page_state (zone,NR_FREE_PAGES , freed); } static void free_one_page(struct zone *zone, struct page *page, int order,
@@ -655,6 +684,7 @@ static int prep_new_page(struct page *page, int order, gfp_t gfp_flags)return 1 ; } + page->index =0 ;set_page_private (page,0 );set_page_refcounted (page);
@@ -903,9 +933,10 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order,*/ list_add (&page->lru, list);set_page_private (page, migratetype); + page->index = order; list = &page->lru; } -__mod_zone_page_state (zone,NR_FREE_PAGES , -(1UL << order) * i); +__mod_zone_page_state (zone,NR_FREE_PAGES , -(i << order));spin_unlock (&zone->lock);return i; }
@@ -929,8 +960,7 @@ void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp) to_drain = pcp->batch; else to_drain = pcp->count; -free_pages_bulk (zone, to_drain, &pcp->list,0 ); - pcp->count -= to_drain; +free_pages_bulk (zone, to_drain, &pcp->list); local_irq_restore(flags); }#endif
@@ -958,8 +988,8 @@ static void drain_pages(unsigned int cpu) pcp = &pset->pcp;local_irq_save (flags); -free_pages_bulk (zone, pcp->count, &pcp->list,0 ); - pcp->count =0 ; +free_pages_bulk (zone, pcp->count, pcp); +BUG_ON (pcp->count);local_irq_restore (flags); } }
@@ -1019,13 +1049,18 @@ void mark_free_pages(struct zone *zone)/* * Free a 0-order page */ -static void free_hot_cold_page (struct page *page,int cold) +static void free_hot_cold_page (struct page *page,int order,int cold) {struct zone *zone =page_zone (page);struct per_cpu_pages *pcp;unsigned long flags;int clearMlocked =PageMlocked (page); +/* SLUB can return lowish-order compound pages that need handling */ +if (order >0 &&unlikely (PageCompound (page))) +if (unlikely (destroy_compound_page (page, order))) +return ; +if (PageAnon (page)) page->mapping =NULL ;if (free_pages_check (page))
@@ -1035,8 +1070,8 @@ static void free_hot_cold_page(struct page *page, int cold)debug_check_no_locks_freed (page_address (page),PAGE_SIZE );debug_check_no_obj_freed (page_address (page),PAGE_SIZE ); } -arch_free_page (page,0 ); -kernel_map_pages (page,1 ,0 ); +arch_free_page (page, order); +kernel_map_pages (page,1 << order,0 ); pcp = &zone_pcp (zone,get_cpu ())->pcp;local_irq_save (flags);
@@ -1044,28 +1079,24 @@ static void free_hot_cold_page(struct page *page, int cold)if (clearMlocked)free_page_mlock (page); -if (cold) -list_add_tail (&page->lru, &pcp->list); -else -list_add (&page->lru, &pcp->list);set_page_private (page,get_pageblock_migratetype (page)); - pcp->count++; -if (pcp->count >= pcp->high) { -free_pages_bulk (zone, pcp->batch, &pcp->list,0 ); - pcp->count -= pcp->batch; - } + page->index = order; +add_pcp_page (pcp, page, cold); + +if (pcp->count >= pcp->high) +free_pages_bulk (zone, pcp->batch, pcp);local_irq_restore (flags);put_cpu (); }void free_hot_page (struct page *page) { -free_hot_cold_page (page,0 ); +free_hot_cold_page (page,0 ,0 ); }void free_cold_page (struct page *page) { -free_hot_cold_page (page,1 ); +free_hot_cold_page (page,0 ,1 ); }/*
@@ -1086,6 +1117,11 @@ void split_page(struct page *page, unsigned int order)set_page_refcounted (page + i); } +static inline int pcp_page_suit (struct page *page,int migratetype,int order) +{ +return page_private (page) == migratetype && page->index == order; +} +/* * Really, prep_compound_page() should be called from __rmqueue_bulk(). But * we cheat by calling it from here, in the order > 0 path. Saves a branch
@@ -1102,14 +1138,18 @@ struct page *buffered_rmqueue(struct zone *preferred_zone, again: cpu =get_cpu (); -if (likely (order ==0 )) { +if (likely (order <=PAGE_ALLOC_COSTLY_ORDER )) {struct per_cpu_pages *pcp; +int batch; +int delta; pcp = &zone_pcp (zone, cpu)->pcp; + batch =max (1 , pcp->batch >> order);local_irq_save (flags);if (!pcp->count) { - pcp->count =rmqueue_bulk (zone,0 , - pcp->batch, &pcp->list, migratetype); + delta =rmqueue_bulk (zone, order, batch, + &pcp->list, migratetype); +bulk_add_pcp_page (pcp, order, delta);if (unlikely (!pcp->count)) goto failed; }
@@ -1117,23 +1157,25 @@ again:/* Find a page of the appropriate migrate type */ if (cold) {list_for_each_entry_reverse (page, &pcp->list, lru) -if (page_private (page) == migratetype) +if (pcp_page_suit (page, migratetype, order))break ; }else {list_for_each_entry (page, &pcp->list, lru) -if (page_private (page) == migratetype) +if (pcp_page_suit (page, migratetype, order))break ; }/* Allocate more to the pcp list if necessary */ if (unlikely (&page->lru == &pcp->list)) { - pcp->count +=rmqueue_bulk (zone,0 , - pcp->batch, &pcp->list, migratetype); + delta =rmqueue_bulk (zone, order, batch, + &pcp->list, migratetype); +bulk_add_pcp_page (pcp, order, delta); page = list_entry(pcp->list.next, struct page, lru); +if (!pcp_page_suit (page, migratetype, order)) + goto failed; } -list_del (&page->lru); - pcp->count--; +rmv_pcp_page (pcp, page); }else {LIST_HEAD (list);local_irq_save (flags);
@@ -1884,14 +1926,14 @@ void __pagevec_free(struct pagevec *pvec)int i =pagevec_count (pvec);while (--i >=0 ) -free_hot_cold_page (pvec->pages[i], pvec->cold); +free_hot_cold_page (pvec->pages[i],0 , pvec->cold); }void __free_pages (struct page *page,unsigned int order) {if (put_page_testzero (page)) { -if (order ==0 ) -free_hot_page (page); +if (order <=PAGE_ALLOC_COSTLY_ORDER ) +free_hot_cold_page (page, order,0 );else __free_pages_ok (page, order); }
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 26/35] Use the per-cpu allocator for orders up to PAGE_ALLOC_COSTLY_ORDER
mm/page_alloc.c void mark_free_pages(struct zone *zone) · line 1063
Quoted 13 lines · Mel Gorman
mm/page_alloc.c +6
-1-static void free_hot_cold_page (struct page *page,int cold) +static void free_hot_cold_page (struct page *page,int order,int cold) {struct zone *zone =page_zone (page);struct per_cpu_pages *pcp;unsigned long flags;int clearMlocked =PageMlocked (page); +/* SLUB can return lowish-order compound pages that need handling */ +if (order >0 &&unlikely (PageCompound (page))) +if (unlikely (destroy_compound_page (page, order))) +return ; +
Isnt that also true for stacks and generic network objects ==- 8k?
Quoted 9 lines
again:
mm/page_alloc.c +3
-1cpu =get_cpu (); -if (likely (order ==0 )) { +if (likely (order <=PAGE_ALLOC_COSTLY_ORDER )) {struct per_cpu_pages *pcp; +int batch; +int delta; pcp = &zone_pcp (zone, cpu)->pcp;
mm/page_alloc.c struct page *buffered_rmqueue(struct zone *preferred_zone, · line 1154
Quoted 10 lines
mm/page_alloc.c +4
-2+ batch =max (1 , pcp->batch >> order);local_irq_save (flags);if (!pcp->count) { - pcp->count =rmqueue_bulk (zone,0 , - pcp->batch, &pcp->list, migratetype); + delta =rmqueue_bulk (zone, order, batch, + &pcp->list, migratetype); +bulk_add_pcp_page (pcp, order, delta);if (unlikely (!pcp->count)) goto failed;
The pcp adds a series of order N pages if an order N alloc occurs and the queue is empty?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 26/35] Use the per-cpu allocator for orders up to PAGE_ALLOC_COSTLY_ORDER
Quoted 10 lines · Christoph Lameter
Quoted 8 lines · Mel Gorman
mm/page_alloc.c +1
-1-static void free_hot_cold_page (struct page *page,int cold) +static void free_hot_cold_page (struct page *page,int order,int cold) {struct zone *zone =page_zone (page);struct per_cpu_pages *pcp;unsigned long flags;int clearMlocked =PageMlocked (page);
Quoted 8 lines
mm/page_alloc.c void mark_free_pages(struct zone *zone) · line 1063
Quoted 5 lines
mm/page_alloc.c +5
-0+/* SLUB can return lowish-order compound pages that need handling */ +if (order >0 &&unlikely (PageCompound (page))) +if (unlikely (destroy_compound_page (page, order))) +return ; +Isnt that also true for stacks and generic network objects ==- 8k?
I think they are vanilla high-order pages, not compound pages.
Quoted 8 lines
Quoted 8 lines
again:
mm/page_alloc.c +3
-1cpu =get_cpu (); -if (likely (order ==0 )) { +if (likely (order <=PAGE_ALLOC_COSTLY_ORDER )) {struct per_cpu_pages *pcp; +int batch; +int delta;
Quoted 15 lines
mm/page_alloc.c struct page *buffered_rmqueue(struct zone *preferred_zone, · line 1154
Quoted 11 lines
mm/page_alloc.c +4
-2pcp = &zone_pcp (zone, cpu)->pcp; + batch =max (1 , pcp->batch >> order);local_irq_save (flags);if (!pcp->count) { - pcp->count =rmqueue_bulk (zone,0 , - pcp->batch, &pcp->list, migratetype); + delta =rmqueue_bulk (zone, order, batch, + &pcp->list, migratetype); +bulk_add_pcp_page (pcp, order, delta);if (unlikely (!pcp->count)) goto failed;The pcp adds a series of order N pages if an order N alloc occurs and the queue is empty?
Nope, that would be bad. The calculation of batch is made above and is
batch = max(1, pcp->batch >> order);
so order is taken into account when deciding how many pages to allocate.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 27/35 Split per-cpu list into one-list-per-migrate-type
In reply to Mel GormanCurrently the per-cpu page allocator searches the PCP list for pages of the correct migrate-type to reduce the possibility of pages being inappropriate placed from a fragmentation perspective. This search is potentially expensive in a fast-path and undesirable. Splitting the per-cpu list into multiple lists increases the size of a per-cpu structure and this was potentially a major problem at the time the search was introduced. These problem has been mitigated as now only the necessary number of structures is allocated for the running system.
This patch replaces a list search in the per-cpu allocator with one list per migrate type. It is still searched but for a page of the correct order which is expected to be a lot less costly.
--- include/linux/mmzone.h | 5 ++- mm/page_alloc.c | 82 +++++++++++++++++++++++++++++++++-------------- 2 files changed, 61 insertions(+), 26 deletions(-)
include/linux/mmzone.h +4 -1
@@ -38,6 +38,7 @@#define MIGRATE_UNMOVABLE 0#define MIGRATE_RECLAIMABLE 1#define MIGRATE_MOVABLE 2 +#define MIGRATE_PCPTYPES 3/* the number of types on the pcp lists */ #define MIGRATE_RESERVE 3#define MIGRATE_ISOLATE 4/* can't allocate from here */ #define MIGRATE_TYPES 5
@@ -167,7 +168,9 @@ struct per_cpu_pages {int count;/* number of pages in the list */ int high;/* high watermark, emptying needed */ int batch;/* chunk size for buddy add/remove */ -struct list_head list;/* the list of pages */ + +/* Lists of pages, one per migrate type stored on the pcp-lists */ +struct list_head lists[MIGRATE_PCPTYPES ]; };struct per_cpu_pageset {
mm/page_alloc.c +57 -25
182 lines not on this page · raw message
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 28/35 Batch free pages from migratetype per-cpu lists
In reply to Mel GormanWhen the PCP lists are too large, a number of pages are freed in bulk. Currently the free lists are examined in a round-robin fashion but this touches more cache lines than necessary. This patch frees pages from one list at a time and uses the migratetype most recently used as the starting point.
--- mm/page_alloc.c | 36 +++++++++++++++++++++++------------- 1 files changed, 23 insertions(+), 13 deletions(-)
mm/page_alloc.c +23 -13
@@ -547,32 +547,42 @@ static inline void bulk_add_pcp_page(struct per_cpu_pages *pcp,* pinned" detection logic. */ static void free_pcppages_bulk (struct zone *zone,int count, -struct per_cpu_pages *pcp) +struct per_cpu_pages *pcp, +int migratetype) { -int migratetype =0 ;unsigned int freed =0 ; +unsigned int bulkcount; +struct list_head *list;spin_lock (&zone->lock);zone_clear_flag (zone,ZONE_ALL_UNRECLAIMABLE ); zone->pages_scanned =0 ; + list = &pcp->lists[migratetype]; + bulkcount =1 + (count / (MIGRATE_PCPTYPES *2 ));while (freed < count) {struct page *page; -struct list_head *list; +int thisfreed; -/* Remove pages from lists in a round-robin fashion */ -do { -if (migratetype ==MIGRATE_PCPTYPES ) +/* +* Move to another migratetype if this list is depleted or +* we've freed enough in this batch +*/ +while (list_empty (list) || bulkcount <0 ) { + bulkcount =1 + (count / (MIGRATE_PCPTYPES *2 )); +if (++migratetype ==MIGRATE_PCPTYPES ) migratetype =0 ; list = &pcp->lists[migratetype]; - migratetype++; - }while (list_empty (list)); + } +/* Remove from list and update counters */ page = list_entry(list->prev, struct page, lru);rmv_pcp_page (pcp, page); + thisfreed =1 << page->index; + freed += thisfreed; + bulkcount -= thisfreed; - freed +=1 << page->index; -__free_one_page (page, zone, page->index,page_private (page)); +__free_one_page (page, zone, page->index, migratetype); }spin_unlock (&zone->lock);
@@ -969,7 +979,7 @@ void drain_zone_pages(struct zone *zone, struct per_cpu_pages *pcp) to_drain = pcp->batch; else to_drain = pcp->count; -free_pcppages_bulk (zone, to_drain, pcp); +free_pcppages_bulk (zone, to_drain, pcp,0 ); local_irq_restore(flags); }#endif
@@ -997,7 +1007,7 @@ static void drain_pages(unsigned int cpu) pcp = &pset->pcp;local_irq_save (flags); -free_pcppages_bulk (zone, pcp->count, pcp); +free_pcppages_bulk (zone, pcp->count, pcp,0 );BUG_ON (pcp->count);local_irq_restore (flags); }
@@ -1110,7 +1120,7 @@ static void free_hot_cold_page(struct page *page, int order, int cold) page->index = order;add_pcp_page (pcp, page, cold);if (pcp->count >= pcp->high) -free_pcppages_bulk (zone, pcp->batch, pcp); +free_pcppages_bulk (zone, pcp->batch, pcp, migratetype); out:local_irq_restore (flags);
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 29/35 Do not store the PCP high and batch watermarks in the per-cpu structure
In reply to Mel GormanCurrently, there are high and batch counters in the per-cpu structure. This might have made sense when there was hot and cold per-cpu structures but that is no longer the case. In practice, all the per-cpu structures for a zone contain the same values and they are read-mostly. This patch stores them in the zone with the watermarks which are also read-mostly.
--- include/linux/mmzone.h | 8 ++++++-- mm/page_alloc.c | 43 +++++++++++++++++++++++-------------------- mm/vmstat.c | 4 ++-- 3 files changed, 31 insertions(+), 24 deletions(-)
include/linux/mmzone.h +6 -2
@@ -166,8 +166,6 @@ static inline int is_unevictable_lru(enum lru_list l)struct per_cpu_pages {int count;/* number of pages in the list */ -int high;/* high watermark, emptying needed */ -int batch;/* chunk size for buddy add/remove */ /* Lists of pages, one per migrate type stored on the pcp-lists */ struct list_head lists[MIGRATE_PCPTYPES ];
@@ -285,6 +283,12 @@ struct zone {unsigned long pages_mark[3 ]; }; +/* high watermark for per-cpu lists, emptying needed */ +u16 pcp_high; + +/* chunk size for buddy add/remove to per-cpu lists*/ +u16 pcp_batch; +/* * We don't know if the memory that we're going to allocate will be freeable * or/and it will be released eventually, so to avoid totally wasting several
mm/page_alloc.c +23 -20
118 lines not on this page · raw message
mm/vmstat.c +2 -2
@@ -766,8 +766,8 @@ static void zoneinfo_show_print(struct seq_file *m, pg_data_t *pgdat,"\n batch: %i" , i, pageset->pcp.count, - pageset->pcp.high, - pageset->pcp.batch); + zone->pcp_high, + zone->pcp_batch);#ifdef CONFIG_SMP seq_printf (m,"\n vm stats threshold: %d" , pageset->stat_threshold);
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 29/35] Do not store the PCP high and batch watermarks in the per-cpu structure
include/linux/mmzone.h static inline int is_unevictable_lru(enum lru_list l) · line 170
Quoted 8 lines · Mel Gorman
include/linux/mmzone.h +0
-2@@ -166,8 +166,6 @@ static inline int is_unevictable_lru(enum lru_list l)struct per_cpu_pages {int count;/* number of pages in the list */ -int high;/* high watermark, emptying needed */ -int batch;/* chunk size for buddy add/remove */
There is a hole here on 64 bit systems.
include/linux/mmzone.h struct zone { · line 291
Quoted 13 lines
include/linux/mmzone.h +6
-0/* Lists of pages, one per migrate type stored on the pcp-lists */ struct list_head lists[MIGRATE_PCPTYPES ];@@ -285,6 +283,12 @@ struct zone {unsigned long pages_mark[3 ]; }; +/* high watermark for per-cpu lists, emptying needed */ +u16 pcp_high; + +/* chunk size for buddy add/remove to per-cpu lists*/ +u16 pcp_batch; +
Move this up to fill the hole?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 30/35 Skip the PCP list search by counting the order and type of pages on list
In reply to Mel GormanThe PCP lists are searched for free pages of the right size but due to multiple orders, the list may be searched uselessly. This patch records how many pages there are of each order and migratetype on the list to determine if a list search will succeed.
--- include/linux/mmzone.h | 6 +++++- mm/page_alloc.c | 46 +++++++++++++++++++++++++--------------------- 2 files changed, 30 insertions(+), 22 deletions(-)
include/linux/mmzone.h +5 -1
@@ -165,7 +165,11 @@ static inline int is_unevictable_lru(enum lru_list l) }struct per_cpu_pages { -int count;/* number of pages in the list */ +/* The total number of pages on the PCP lists */ +int count; + +/* Count of each migratetype and order */ +u8 mocount[MIGRATE_PCPTYPES ][PAGE_ALLOC_COSTLY_ORDER +1 ];/* Lists of pages, one per migrate type stored on the pcp-lists */ struct list_head lists[MIGRATE_PCPTYPES ];
mm/page_alloc.c +25 -21
92 lines not on this page · raw message
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 30/35] Skip the PCP list search by counting the order and type of pages on list
include/linux/mmzone.h static inline int is_unevictable_lru(enum lru_list l) · line 172
Quoted 8 lines · Mel Gorman
struct
per_cpu_pages{include/linux/mmzone.h +5
-1-int count;/* number of pages in the list */ +/* The total number of pages on the PCP lists */ +int count; + +/* Count of each migratetype and order */ +u8 mocount[MIGRATE_PCPTYPES ][PAGE_ALLOC_COSTLY_ORDER +1 ];
What about overflow? You could have more than 255 pages of a given type in a pcp.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 30/35] Skip the PCP list search by counting the order and type of pages on list
Quoted 14 lines · Christoph Lameter
include/linux/mmzone.h static inline int is_unevictable_lru(enum lru_list l) · line 172
Quoted 8 lines · Mel Gorman
struct
per_cpu_pages{include/linux/mmzone.h +5
-1-int count;/* number of pages in the list */ +/* The total number of pages on the PCP lists */ +int count; + +/* Count of each migratetype and order */ +u8 mocount[MIGRATE_PCPTYPES ][PAGE_ALLOC_COSTLY_ORDER +1 ];What about overflow? You could have more than 255 pages of a given type in a pcp.
Are you sure? The maximum size I would expect is pcp->high which is 6 * pcp->batch which is limited in size. I might have missed something though.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 31/35 Optimistically check the first page on the PCP free list is suitable
In reply to Mel GormanThe PCP lists are searched for a page of the suitable order. However, the majority of pages are still expected to be order-0 pages and the setup for the search is a bit expensive. This patch optimistically checks if the first page is suitable for use in the hot-page allocation path.
--- mm/page_alloc.c | 6 ++++++ 1 files changed, 6 insertions(+), 0 deletions(-)
mm/page_alloc.c +6 -0
@@ -1212,6 +1212,12 @@ again:if (pcp_page_suit (page, order))break ; }else { +/* Optimistic before we start a list search */ + page = list_entry(list->next, struct page, lru); +if (pcp_page_suit (page, order)) + goto found; + +/* Do the search */ list_for_each_entry (page, list, lru)if (pcp_page_suit (page, order))break ;
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 31/35] Optimistically check the first page on the PCP free list is suitable
Quoted 7 lines · Mel Gorman
mm/page_alloc.c +0
-0@@ -1212,6 +1212,12 @@ again:if (pcp_page_suit (page, order))break ;
mm/page_alloc.c again: · line 1223
Quoted 10 lines
mm/page_alloc.c +6
-0}else { +/* Optimistic before we start a list search */ + page = list_entry(list->next, struct page, lru); +if (pcp_page_suit (page, order)) + goto found; + +/* Do the search */ list_for_each_entry (page, list, lru)if (pcp_page_suit (page, order))break ;
I am not convinced that this is beneficial. If it would then the compiler would unroll the loop.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 31/35] Optimistically check the first page on the PCP free list is suitable
Quoted 14 lines · Christoph Lameter
Quoted 12 lines · Mel Gorman
mm/page_alloc.c +4
-0@@ -1212,6 +1212,12 @@ again:if (pcp_page_suit (page, order))break ; }else { +/* Optimistic before we start a list search */ + page = list_entry(list->next, struct page, lru); +if (pcp_page_suit (page, order)) + goto found;
Quoted 9 lines
mm/page_alloc.c again: · line 1223
Quoted 5 lines
mm/page_alloc.c +2
-0+ +/* Do the search */ list_for_each_entry (page, list, lru)if (pcp_page_suit (page, order))break ;I am not convinced that this is beneficial. If it would then the compiler would unroll the loop.
It hit a large number of times when I checked (although I don't have the figures any more) and the list_for_each_entry() does a prefetch possibly fetching in a line we don't need.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 32/35 Inline next_zones_zonelist() of the zonelist scan in the fastpath
In reply to Mel GormanThe zonelist walkers call next_zones_zonelist() to find the next zone that is allowed by the nodemask. It's not inlined because the number of call-sites bloats text but it is not free to call a function either. This patch inlines next_zones_zonelist() only for the page allocator fastpath. All other zonelist walkers use an uninlined version.
--- include/linux/mmzone.h | 6 ++++++ mm/mmzone.c | 31 ------------------------------- mm/page_alloc.c | 40 +++++++++++++++++++++++++++++++++++++++- 3 files changed, 45 insertions(+), 32 deletions(-)
include/linux/mmzone.h +6 -0
@@ -895,6 +895,12 @@ static inline struct zoneref *first_zones_zonelist(struct zonelist *zonelist,zone ; \ z = next_zones_zonelist(++z, highidx, nodemask, &zone)) \ +/* Only available to the page allocator fast-path */ +#define fast_foreach_zone_zonelist_nodemask (zone, z, zlist, highidx, nodemask) \ + for (z = first_zones_zonelist(zlist, highidx, nodemask, &zone); \ + zone; \ + z = __next_zones_zonelist(++z, highidx, nodemask, &zone)) \ +/** * for_each_zone_zonelist - helper macro to iterate over valid zones in a zonelist at or below a given zone index * @zone - The current zone in the iterator
mm/mmzone.c +0 -31
@@ -41,34 +41,3 @@ struct zone *next_zone(struct zone *zone) }return zone; } - -static inline int zref_in_nodemask (struct zoneref *zref,nodemask_t *nodes) -{ -#ifdef CONFIG_NUMA -return node_isset (zonelist_node_idx (zref), *nodes); -#else -return 1 ; -#endif /* CONFIG_NUMA */ -} - -/* Returns the next zone at or below highest_zoneidx in a zonelist */ -struct zoneref *next_zones_zonelist (struct zoneref *z, -enum zone_type highest_zoneidx, -nodemask_t *nodes, -struct zone **zone) -{ -/* -* Find the next suitable zone to use for the allocation. -* Only filter based on nodemask if it's set -*/ -if (likely (nodes ==NULL )) -while (zonelist_zone_idx (z) > highest_zoneidx) - z++; -else -while (zonelist_zone_idx (z) > highest_zoneidx || - (z->zone && !zref_in_nodemask (z, nodes))) - z++; - - *zone =zonelist_zone (z); -return z; -}
mm/page_alloc.c +39 -1
@@ -1514,6 +1514,44 @@ static void zlc_mark_zone_full(struct zonelist *zonelist, struct zoneref *z) }#endif /* CONFIG_NUMA */ +static inline int +zref_in_nodemask (struct zoneref *zref,nodemask_t *nodes) +{ +#ifdef CONFIG_NUMA +return node_isset (zonelist_node_idx (zref), *nodes); +#else +return 1 ; +#endif /* CONFIG_NUMA */ +} + +/* Returns the next zone at or below highest_zoneidx in a zonelist */ +static inline struct zoneref * +__next_zones_zonelist (struct zoneref *z,enum zone_type highest_zoneidx, +nodemask_t *nodes,struct zone **zone) +{ +/* +* Find the next suitable zone to use for the allocation. +* Only filter based on nodemask if it's set +*/ +if (likely (nodes ==NULL )) +while (zonelist_zone_idx (z) > highest_zoneidx) + z++; +else +while (zonelist_zone_idx (z) > highest_zoneidx || + (z->zone && !zref_in_nodemask (z, nodes))) + z++; + + *zone =zonelist_zone (z); +return z; +} + +struct zoneref * +next_zones_zonelist (struct zoneref *z,enum zone_type highest_zoneidx, +nodemask_t *nodes,struct zone **zone) +{ +return __next_zones_zonelist (z, highest_zoneidx, nodes, zone); +} +/* * get_page_from_freelist goes through the zonelist trying to allocate * a page.
@@ -1546,7 +1584,7 @@ zonelist_scan:* Scan zonelist, looking for a zone with enough free. * See also cpuset_zone_allowed() comment in kernel/cpuset.c. */ -for_each_zone_zonelist_nodemask (zone, z, zonelist, +fast_foreach_zone_zonelist_nodemask (zone, z, zonelist, high_zoneidx, nodemask) {/* Ignore the additional zonelist filter checks if possible */
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 33/35 Do not merge buddies until they are needed by a high-order allocation or anti-fragmentation
In reply to Mel GormanFreeing and allocating pages from the buddy lists can incur a number of cache misses as the struct pages are written to. This patch only merges buddies up to PAGE_ALLOC_COSTLY_ORDER. High-order allocations are then required to do the actual merging. This punishes high-order allocations somewhat but they are expected to be relatively rare and should be avoided in general.
--- include/linux/mmzone.h | 7 ++++ mm/page_alloc.c | 91 +++++++++++++++++++++++++++++++++++++++++------ 2 files changed, 86 insertions(+), 12 deletions(-)
include/linux/mmzone.h +7 -0
@@ -35,6 +35,13 @@*/ #define PAGE_ALLOC_COSTLY_ORDER 3 +/* +* PAGE_ALLOC_MERGE_ORDER is the order at which pages get merged together +* but not merged further unless explicitly needed by a high-order allocation. +* The value is to merge to larger than the PCP batch refill size +*/ +#define PAGE_ALLOC_MERGE_ORDER 5 +#define MIGRATE_UNMOVABLE 0#define MIGRATE_RECLAIMABLE 1#define MIGRATE_MOVABLE 2
mm/page_alloc.c +79 -12
126 lines not on this page · raw message
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 34/35 Allow compound pages to be stored on the PCP lists
In reply to Mel GormanThe SLUB allocator frees and allocates compound pages. The setup costs for compound pages are noticeable in profiles and incur cache misses as every struct page has to be checked and written. This patch allows compound pages to be stored on the PCP list to save on teardown and setup time.
--- include/linux/page-flags.h | 4 ++- mm/page_alloc.c | 50 +++++++++++++++++++++++++++++--------------- 2 files changed, 36 insertions(+), 18 deletions(-)
include/linux/page-flags.h +3 -1
@@ -388,7 +388,9 @@ static inline void __ClearPageTail(struct page *page)* Pages being prepped should not have any flags set. It they are set, * there has been a kernel bug or struct page corruption. */ -#define PAGE_FLAGS_CHECK_AT_PREP ((1 << NR_PAGEFLAGS) - 1) +#define PAGE_FLAGS_CHECK_AT_PREP_BUDDY ((1 << NR_PAGEFLAGS) - 1) +#define PAGE_FLAGS_CHECK_AT_PREP (((1 << NR_PAGEFLAGS) - 1) & \ + ~(1 << PG_head | 1 << PG_tail))#endif /* !__GENERATING_BOUNDS_H */ #endif /* PAGE_FLAGS_H */
mm/page_alloc.c +33 -17
94 lines not on this page · raw message
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Re: [PATCH 34/35] Allow compound pages to be stored on the PCP lists
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
PATCH 35/35 Allow up to 4MB PCP lists due to compound pages
In reply to Mel GormanCompound pages from SLUB on the free lists can occupy a fair percentage of the 512K that is currently allowed on the PCP lists. This can push out cache hot order-0 pages even though the compound page may be relatively sparsely used in the short term. This patch changes pcp->count to count pages (1 per page regardless of order) instead of accounting for the number of base pages on the list. This keeps cache hot pages on the list at the cost of the PCP lists being up to 4MB in size instead of 512K.
--- mm/page_alloc.c | 23 +++++++++-------------- 1 files changed, 9 insertions(+), 14 deletions(-)
mm/page_alloc.c +9 -14
@@ -572,11 +572,10 @@ static inline int check_freepage_flags(struct page *page,static inline void rmv_pcp_page (struct per_cpu_pages *pcp,struct page *page) {int migratetype =page_private (page); -int basepage_count =1 << page->index;list_del (&page->lru); - pcp->count -= basepage_count; - pcp->mocount[migratetype][page->index] -= basepage_count; + pcp->count--; + pcp->mocount[migratetype][page->index]--; } static inline void add_pcp_page(struct per_cpu_pages *pcp,
@@ -584,22 +583,20 @@ static inline void add_pcp_page(struct per_cpu_pages *pcp, int cold) {int migratetype =page_private (page); -int basepage_count =1 << page->index;if (cold)list_add_tail (&page->lru, &pcp->lists[migratetype]);else list_add (&page->lru, &pcp->lists[migratetype]); - pcp->count += basepage_count; - pcp->mocount[migratetype][page->index] += basepage_count; + pcp->count++; + pcp->mocount[migratetype][page->index]++; }static inline void bulk_add_pcp_page (struct per_cpu_pages *pcp,int migratetype,int order,int count) { -int basepage_count = count << order; - pcp->count += basepage_count; - pcp->mocount[migratetype][order] += basepage_count; + pcp->count += count; + pcp->mocount[migratetype][order] += count; }/*
@@ -627,9 +624,8 @@ static void free_pcppages_bulk(struct zone *zone, int count, list = &pcp->lists[migratetype]; bulkcount =1 + (count / (MIGRATE_PCPTYPES *2 )); -while (freed < count) { +while (count--) {struct page *page; -int thisfreed;/* * Move to another migratetype if this list is depleted or
@@ -645,9 +641,8 @@ static void free_pcppages_bulk(struct zone *zone, int count,/* Remove from list and update counters */ page = list_entry(list->prev, struct page, lru);rmv_pcp_page (pcp, page); - thisfreed =1 << page->index; - freed += thisfreed; - bulkcount -= thisfreed; + freed +=1 << page->index; + bulkcount--;__free_one_page (page, zone, page->index, migratetype); }
Signature · 7 lines
1.5.6.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Mel Gorman
Here is V3 of an attempt to cleanup and optimise the page allocator and should be ready for general testing. The page allocator is now faster (16% reduced time overall for kernbench on one machine) and it has a smaller cache footprint (16.5% less L1 cache misses and 19.5% less L2 cache misses for kernbench on one machine). The text footprint has unfortunately increased, largely due to the introduction of a form of lazy buddy merging mechanism that avoids cache misses by postponing buddy merging until a high-order allocation needs it.
You!? You want to do lazy buddy? ;) That's wonderful, but it would significantly increase the fragmentation problem, wouldn't it? (although pcp lists are conceptually a form of lazy buddy already)
No objections from me of course, if it is making significant speedups. I assume you mean overall time on kernbench is overall sys time?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 11 lines · Nick Piggin
Quoted 8 lines · Mel Gorman
Here is V3 of an attempt to cleanup and optimise the page allocator and should be ready for general testing. The page allocator is now faster (16% reduced time overall for kernbench on one machine) and it has a smaller cache footprint (16.5% less L1 cache misses and 19.5% less L2 cache misses for kernbench on one machine). The text footprint has unfortunately increased, largely due to the introduction of a form of lazy buddy merging mechanism that avoids cache misses by postponing buddy merging until a high-order allocation needs it.
You!? You want to do lazy buddy? ;)
I'm either a man of surprises, an idiot or just plain inconsistent :)
Quoted 2 lines
That's wonderful, but it would
significantly increase the fragmentation problem, wouldn't it?
Not necessarily, anti-fragmentation groups movable pages within a hugepage-aligned block and high-order allocations will trigger a merge of buddies from PAGE_ALLOC_MERGE_ORDER (defined in the relevant patch) up to MAX_ORDER-1. Critically, a merge is also triggered when anti-fragmentation wants to fallback to another migratetype to satisfy an allocation. As long as the grouping works, it doesn't matter if they were only merged up to PAGE_ALLOC_MERGE_ORDER as a full merge will still free up hugepages. So two slow paths are made slower but the fast path should be faster and it should be causing fewer cache line bounces due to writes to struct page.
The success rate of high-order allocations should be roughly the same but they will be slower, particularly as I remerge more often than required. This slowdown is undesirable but the assumptions are that either a) it's the static hugepage pool being resized in which case the delay is irrelevant or
b) the pool is being dynamically resized but the expected lifetime of the page far exceeds the allocation cost of merging.
I fully agree with you that it's more important that order-0 allocations are faster than order-9 allocations but I'm not totally off high-order allocs either. You'll see the patchset allows higher-order pages (up to PAGE_ALLOC_COSTLY_ORDER) onto the PCP lists for order-1 allocations used by sig handlers, stacks and the like (important for fork-heavy loads I am guessing) and because SLUB uses high-order allocations that are currently bypassing the PCP lists altogether. I haven't measured it but SLUB-heavy workloads must be contending on the zone->lock to some extent.
When I last checked (about 10 days) ago, I hadn't damaged anti-fragmentation but that was a lot of revisions ago. I'm redoing the tests to make sure anti-fragmentation is still ok.
Quoted 2 lines
(although pcp lists are conceptually a form of lazy buddy already)
Indeed.
Quoted 3 lines
No objections from me of course, if it is making significant speedups. I assume you mean overall time on kernbench is overall sys time?
Both, I should be clearer. The amount of oprofile samples measured in the page allocator is reduced by a large amount but it does not always translate into overall speedups although it did for 8 out of 9 machines I tested. On most machines the overall "System Time" for kernbench is reduced but in 2 out of 9 test machines, the elapsed time increases due to some other caching weirdness or due to a change in timing with respect to locking.
Pinning down the exact problem is tricky as profile sometimes reverses the performance effects. i.e. without profiling I'll see a slowdown and with profiling I see significant speedups so I can't measure what is going on.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 4 lines · Mel Gorman
Quoted 2 lines · Nick Piggin
That's wonderful, but it would
significantly increase the fragmentation problem, wouldn't it?
Quoted 9 lines
Not necessarily, anti-fragmentation groups movable pages within a hugepage-aligned block and high-order allocations will trigger a merge of buddies from
PAGE_ALLOC_MERGE_ORDER(defined in the relevant patch) up to MAX_ORDER-1. Critically, a merge is also triggered when anti-fragmentation wants to fallback to another migratetype to satisfy an allocation. As long as the grouping works, it doesn't matter if they were only merged up toPAGE_ALLOC_MERGE_ORDERas a full merge will still free up hugepages. So two slow paths are made slower but the fast path should be faster and it should be causing fewer cache line bounces due to writes to struct page.
Oh, but the anti-fragmentation stuff is orthogonal to this. Movable groups should always be defragmentable (at some cost)... the bane of anti-frag is fragmentation of the non-movable groups.
And one reason why buddy is so good at avoiding fragmentation is because it will pick up _any_ pages that go past the allocator if they have any free buddies. And it hands out ones that don't have free buddies. So in that way it is naturally continually filtering out pages which can be merged.
Wheras if you defer this until the point you need a higher order page, the only thing you have to work with are the pages that are free *right now*.
It will almost definitely increase fragmentation of non movable zones, and if you have a workload doing non-trivial, non movable higher order allocations that are likely to cause fragmentation, it will result in these allocations eating movable groups sooner I think.
Quoted 3 lines
When I last checked (about 10 days) ago, I hadn't damaged anti-fragmentation but that was a lot of revisions ago. I'm redoing the tests to make sure anti-fragmentation is still ok.
Your anti-frag tests probably don't stress this long term fragmentation problem.
Still, it's significant enough that I think it should be made optional (and arguably default to on) even if it does harm higher order allocations a bit.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 15 lines · Nick Piggin
Quoted 4 lines · Mel Gorman
Quoted 2 lines · Nick Piggin
That's wonderful, but it would
significantly increase the fragmentation problem, wouldn't it?Quoted 9 lines
Not necessarily, anti-fragmentation groups movable pages within a hugepage-aligned block and high-order allocations will trigger a merge of buddies from
PAGE_ALLOC_MERGE_ORDER(defined in the relevant patch) up to MAX_ORDER-1. Critically, a merge is also triggered when anti-fragmentation wants to fallback to another migratetype to satisfy an allocation. As long as the grouping works, it doesn't matter if they were only merged up toPAGE_ALLOC_MERGE_ORDERas a full merge will still free up hugepages. So two slow paths are made slower but the fast path should be faster and it should be causing fewer cache line bounces due to writes to struct page.
Quoted 4 lines
Oh, but the anti-fragmentation stuff is orthogonal to this. Movable groups should always be defragmentable (at some cost)... the bane of anti-frag is fragmentation of the non-movable groups.
True, the reclaimable area has varying degrees of success and the non-movable groups are almost unworkable and depend on how much of them depend on pagetables.
Quoted 10 lines
And one reason why buddy is so good at avoiding fragmentation is because it will pick up _any_ pages that go past the allocator if they have any free buddies. And it hands out ones that don't have free buddies. So in that way it is naturally continually filtering out pages which can be merged.
Wheras if you defer this until the point you need a higher order page, the only thing you have to work with are the pages that are free *right now*.
Well, buddy always uses the smallest available page first. Even with deferred coalescing, it will merge up to order-5 at least. Lets say they could have merged up to order-10 in ordinary circumstances, they are still avoided for as long as possible. Granted, it might mean that an order-5 is split that could have been merged but it's hard to tell how much of a difference that makes.
Quoted 5 lines
It will almost definitely increase fragmentation of non movable zones, and if you have a workload doing non-trivial, non movable higher order allocations that are likely to cause fragmentation, it will result in these allocations eating movable groups sooner I think.
I think the effect will be same unless the non-movable high-order allocations are order-5 or higher in which case we are likely going to hit trouble anyway.
Quoted 8 lines
Quoted 4 lines
When I last checked (about 10 days) ago, I hadn't damaged anti-fragmentation but that was a lot of revisions ago. I'm redoing the tests to make sure anti-fragmentation is still ok.
Your anti-frag tests probably don't stress this long term fragmentation problem.
Probably not, but we have little data on long-term fragmentation other than anecdotal evidence that it's ok these days.
Quoted 4 lines
Still, it's significant enough that I think it should be made optional (and arguably default to on) even if it does harm higher order allocations a bit.
I could make PAGE_ORDER_MERGE_ORDER a proc tunable? If it's placed as a read-mostly variable beside the gfp_zone table, it might even fit in the same cache line.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · Mel Gorman
Quoted 4 lines · Nick Piggin
Wheras if you defer this until the point you need a higher order page, the only thing you have to work with are the pages that are free *right now*.
Quoted 6 lines
Well, buddy always uses the smallest available page first. Even with deferred coalescing, it will merge up to order-5 at least. Lets say they could have merged up to order-10 in ordinary circumstances, they are still avoided for as long as possible. Granted, it might mean that an order-5 is split that could have been merged but it's hard to tell how much of a difference that makes.
But the kinds of pages *you* are interested in are order-10, right?
Quoted 6 lines
Quoted 3 lines
Your anti-frag tests probably don't stress this long term fragmentation problem.
Probably not, but we have little data on long-term fragmentation other than anecdotal evidence that it's ok these days.
Well, I think before anti-frag there was lots of anecdotal evidence that it's "ok", except for loads heavily using large higher order allocations. I don't know if we'd have many systems running with hundreds of days of uptime on such workloads post-anti-frag?
Google might? But I don't know how long their uptimes are. I expect we'd have a better idea in a couple more years after the next enterprise distro release cycles with anti-frag.
Quoted 8 lines
Quoted 4 lines
Still, it's significant enough that I think it should be made optional (and arguably default to on) even if it does harm higher order allocations a bit.
I could make
PAGE_ORDER_MERGE_ORDERa proc tunable? If it's placed as a read-mostly variable beside thegfp_zonetable, it might even fit in the same cache line.
Hmm, possibly. OTOH I don't like tunables. If you don't think it will be a problem for hugepage allocations, then I would prefer just to leave it on and 5 by default (or even less? COSTLY_ORDER?)
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 7 lines · Nick Piggin
Quoted 6 lines · Mel Gorman
Quoted 4 lines · Nick Piggin
Wheras if you defer this until the point you need a higher order page, the only thing you have to work with are the pages that are free *right now*.
Quoted 9 lines
Quoted 6 lines
Well, buddy always uses the smallest available page first. Even with deferred coalescing, it will merge up to order-5 at least. Lets say they could have merged up to order-10 in ordinary circumstances, they are still avoided for as long as possible. Granted, it might mean that an order-5 is split that could have been merged but it's hard to tell how much of a difference that makes.
But the kinds of pages *you* are interested in are order-10, right?
Yes, but my expectation is that multiple free order-5 pages can be merged to make up an order-10. If they can't, then lumpy reclaim kicks in as normal. My expectation actually is that order-10 allocations often end up using lumpy reclaim and the pages are not automatically available.
As it is though, I have done something wrong and success rates have dropped where they were ok 10 days ago. I need to investigate further but as the first cut-off point at 25 patches is before the lazy buddy patch, it's not an immediate problem.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
Your anti-frag tests probably don't stress this long term fragmentation problem.
Quoted 8 lines
Quoted 2 lines
Probably not, but we have little data on long-term fragmentation other than anecdotal evidence that it's ok these days.
Well, I think before anti-frag there was lots of anecdotal evidence that it's "ok", except for loads heavily using large higher order allocations. I don't know if we'd have many systems running with hundreds of days of uptime on such workloads post-anti-frag?
I doubt it. I probably won't see proper reports on how it behaves until it's part of a major distro release.
Quoted 4 lines
Google might? But I don't know how long their uptimes are. I expect we'd have a better idea in a couple more years after the next enterprise distro release cycles with anti-frag.
Exactly.
Quoted 11 lines
Quoted 9 lines
Quoted 5 lines
Still, it's significant enough that I think it should be made optional (and arguably default to on) even if it does harm higher order allocations a bit.
I could make
PAGE_ORDER_MERGE_ORDERa proc tunable? If it's placed as a read-mostly variable beside thegfp_zonetable, it might even fit in the same cache line.Hmm, possibly. OTOH I don't like tunables.
Neither do I, but in this case it would make it easier to test where the proper cut-off point is without requiring kernel recompiles and make a final static decision later.
Quoted 4 lines
If you don't think it will
be a problem for hugepage allocations, then I would prefer just to leave it on and 5 by default (or even less?COSTLY_ORDER?)
I went with 5 because it means we merge up to at least the size the pcp->batch size. As the page allocator gives back pages in contiguous order if a buddy split occured, it made sense that pcp batch refills are contiguous where possible.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Mel Gorman
Quoted 7 lines · Nick Piggin
Quoted 6 lines
Well, buddy always uses the smallest available page first. Even with deferred coalescing, it will merge up to order-5 at least. Lets say they could have merged up to order-10 in ordinary circumstances, they are still avoided for as long as possible. Granted, it might mean that an order-5 is split that could have been merged but it's hard to tell how much of a difference that makes.
Quoted 5 lines
Quoted 2 lines
But the kinds of pages *you* are interested in are order-10, right?
Yes, but my expectation is that multiple free order-5 pages can be merged to make up an order-10.
Yes, but lazy buddy will give out part of an order-10 free area to an order-5 request even when there are genuine order-5,6,7,8,9 free areas available.
Now it could be assumed that not too much else in the kernel asks for anything over order-3, so you are unlikely to get these kinds of requests. But it's worse than that actually, because lazy buddy will also split half of an order-10 free area in order to satisfy an order-0 allocation in cases that there are no smaller orders than 5 available.
So yes definitely I think there should be a very real impact on higher order coalescing no matter what you do.
Quoted 4 lines
If they can't, then lumpy reclaim kicks
in as normal. My expectation actually is that order-10 allocations often end up using lumpy reclaim and the pages are not automatically available.
movable zone is less interesting, although it will make it harder to allocate these guys from movable zone. But the pages are movable so eventually they should be able to be reclaimed.
unmovable zone fragmentation is more important point because it eventually can destroy the movable zone.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 12 lines · Nick Piggin
Quoted 11 lines · Mel Gorman
Quoted 9 lines · Nick Piggin
Quoted 6 lines
Well, buddy always uses the smallest available page first. Even with deferred coalescing, it will merge up to order-5 at least. Lets say they could have merged up to order-10 in ordinary circumstances, they are still avoided for as long as possible. Granted, it might mean that an order-5 is split that could have been merged but it's hard to tell how much of a difference that makes.
But the kinds of pages *you* are interested in are order-10, right?
Quoted 7 lines
Quoted 2 lines
Yes, but my expectation is that multiple free order-5 pages can be merged to make up an order-10.
Yes, but lazy buddy will give out part of an order-10 free area to an order-5 request even when there are genuine order-5,6,7,8,9 free areas available.
True.
Quoted 3 lines
Now it could be assumed that not too much else in the kernel asks for anything over order-3, so you are unlikely to get these kinds of requests.
Which is an assumption I was working with.
Quoted 5 lines
But it's worse than that actually, because
lazy buddy will also split half of an order-10 free area in order to satisfy an order-0 allocation in cases that there are no smaller orders than 5 available.
Also true. In movable areas it probably makes no difference but it might if large high-order unmovable allocations were common.
Quoted 3 lines
So yes definitely I think there should be a very real impact on higher order coalescing no matter what you do.
Because this is not straight-forward at all, I'll put lazy buddy onto the back-burner and exhaust all other possibilities before revisiting it again.
Quoted 10 lines
Quoted 5 lines
If they can't, then lumpy reclaim kicks
in as normal. My expectation actually is that order-10 allocations often end up using lumpy reclaim and the pages are not automatically available.movable zone is less interesting, although it will make it harder to allocate these guys from movable zone. But the pages are movable so eventually they should be able to be reclaimed.
Exactly.
Quoted 3 lines
unmovable zone fragmentation is more important point because it eventually can destroy the movable zone.
Which is why rmqueue_fallback() also merges up all buddies before making any decisions but I accept your points. This is hard enough to mind-experiment with that it should be considered last.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 8 lines · Mel Gorman
Quoted 3 lines · Nick Piggin
So yes definitely I think there should be a very real impact on higher order coalescing no matter what you do.
Because this is not straight-forward at all, I'll put lazy buddy onto the back-burner and exhaust all other possibilities before revisiting it again.
If it is such a big improvement, I expect *most* people will want it and we probably should do it. But just that it will not play nicely with fragmentation and so you'd need to look into it and devise some way those users can tune it to be nicer.
Quoted 6 lines
Quoted 3 lines
unmovable zone fragmentation is more important point because it eventually can destroy the movable zone.
Which is why
rmqueue_fallback()also merges up all buddies before making any decisions but I accept your points.
Right, that merge of buddies will only be able to look at what is currently free. Wheras non-lazy buddy can pull out higher orders before reallocating them.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 6 lines · Nick Piggin
Quoted 5 lines · Mel Gorman
Quoted 3 lines · Nick Piggin
So yes definitely I think there should be a very real impact on higher order coalescing no matter what you do.
Quoted 6 lines
Quoted 3 lines
Because this is not straight-forward at all, I'll put lazy buddy onto the back-burner and exhaust all other possibilities before revisiting it again.
If it is such a big improvement, I expect *most* people will want it and we probably should do it.
I'll be reinvestigating it in isolation. It's possible that high-order and compound pages on the PCP lists is enough of a delayed buddy merging that the benefit from lazy buddy is marginal.
Quoted 4 lines
But just that it will not play
nicely with fragmentation and so you'd need to look into it and devise some way those users can tune it to be nicer.
Which is why I'm going to postpone it for now.
Quoted 11 lines
Quoted 6 lines
Quoted 3 lines
unmovable zone fragmentation is more important point because it eventually can destroy the movable zone.
Which is why
rmqueue_fallback()also merges up all buddies before making any decisions but I accept your points.Right, that merge of buddies will only be able to look at what is currently free. Wheras non-lazy buddy can pull out higher orders before reallocating them.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Quoted 8 lines · Mel Gorman
Here is V3 of an attempt to cleanup and optimise the page allocator and should be ready for general testing. The page allocator is now faster (16% reduced time overall for kernbench on one machine) and it has a smaller cache footprint (16.5% less L1 cache misses and 19.5% less L2 cache misses for kernbench on one machine). The text footprint has unfortunately increased, largely due to the introduction of a form of lazy buddy merging mechanism that avoids cache misses by postponing buddy merging until a high-order allocation needs it.
BTW. I would feel better about this if it gets merged in stages, with functional changes split out, and also code optimisations and omore obvious performace improvements split out and preferably merged first.
At a very quick glance, the first 25 or so patches should go in first, and that gives a much better base to compare subsequent functional changes with. Patch 18 for example is really significant, and should almost be 2.6.29/-stable material IMO.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 10 lines · Nick Piggin
Quoted 8 lines · Mel Gorman
Here is V3 of an attempt to cleanup and optimise the page allocator and should be ready for general testing. The page allocator is now faster (16% reduced time overall for kernbench on one machine) and it has a smaller cache footprint (16.5% less L1 cache misses and 19.5% less L2 cache misses for kernbench on one machine). The text footprint has unfortunately increased, largely due to the introduction of a form of lazy buddy merging mechanism that avoids cache misses by postponing buddy merging until a high-order allocation needs it.
Quoted 4 lines
BTW. I would feel better about this if it gets merged in stages, with functional changes split out, and also code optimisations and omore obvious performace improvements split out and preferably merged first.
The ordering of the patches was such that least-controversial stuff is at the start of the patchset. The intention was to be able to select a cut-off point and say "that's enough for now"
Quoted 3 lines
At a very quick glance, the first 25 or so patches should go in first, and that gives a much better base to compare subsequent functional changes with.
That's reasonable. I've requeued tests for the patchset up to 25 to see what that looks like. There is also a part of a later patch that reduces how much time is spent with interrupts disabled. I should split that out and move it back to within the cut-off point as something that is "obviously good".
Quoted 3 lines
Patch 18 for example is really significant, and should almost be 2.6.29/-stable material IMO.
My impression was that -stable was only for functional regressions where as this is really a performance thing.
Signature · 9 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 20 lines · Mel Gorman
Quoted 10 lines · Nick Piggin
Quoted 8 lines · Mel Gorman
Here is V3 of an attempt to cleanup and optimise the page allocator and should be ready for general testing. The page allocator is now faster (16% reduced time overall for kernbench on one machine) and it has a smaller cache footprint (16.5% less L1 cache misses and 19.5% less L2 cache misses for kernbench on one machine). The text footprint has unfortunately increased, largely due to the introduction of a form of lazy buddy merging mechanism that avoids cache misses by postponing buddy merging until a high-order allocation needs it.
Quoted 4 lines
BTW. I would feel better about this if it gets merged in stages, with functional changes split out, and also code optimisations and omore obvious performace improvements split out and preferably merged first.
The ordering of the patches was such that least-controversial stuff is at the start of the patchset. The intention was to be able to select a cut-off point and say "that's enough for now"
Quoted 8 lines
Quoted 3 lines
At a very quick glance, the first 25 or so patches should go in first, and that gives a much better base to compare subsequent functional changes with.
That's reasonable. I've requeued tests for the patchset up to 25 to see what that looks like. There is also a part of a later patch that reduces how much time is spent with interrupts disabled. I should split that out and move it back to within the cut-off point as something that is "obviously good".
OK cool. It also means we can start getting benefit of some of them sooner. I hope most of the obvious ones can be merged in 2.6.30.
Quoted 6 lines
Quoted 3 lines
Patch 18 for example is really significant, and should almost be 2.6.29/-stable material IMO.
My impression was that -stable was only for functional regressions where as this is really a performance thing.
A performance regression like this in the core page allocator is a pretty important problem. The fix is obvious. But maybe you're right.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>