mm: Convert global dirty_exceeded flag to per-node node_dirty_exceeded
Subsystems (2)
PATCH 1/1 mm: Convert global dirty_exceeded flag to per-node node_dirty_exceeded
This is a repost. I did not get any comments when I last posted this about a month back. So I guess this patch is all good :)
Andrew, can this get some exposure in -mm?
Thanks,
Kiran
Convert global dirty_exceeded flag to per-node node_dirty_exceeded.
dirty_exceeded ping pongs between nodes in order to force all cpus in the system to increase the frequency of calls to balance_dirty_pages.
Currently dirty_exceeded is used by balance_dirty_pages_ratelimited to force all CPUs in the system call balance_dirty_pages often, in order to reduce the amount of dirty pages in the entire system (based on dirty_thresh and one CPU exceeding thee ratelimits). As dirty_exceeded is a global variable, it will ping-pong between nodes of a NUMA system which is not good.
We think it is OK to reduce the affect of dirty_exceeded to node basis. That will give us on one hand having multiple CPUs (of that node) calling balance_dirty_pages often (based on same logic as before) and on the other hand not causing ping-pong of dirty_exceeded with the other nodes.
Following patch changes the global dirty_exceeded flag to a per-node flag. Hence, balance_dirty_pages is called more often only on the node which exceeded the dirty_threshold. The SMP case is left as was before.
Comments?
mm/page-writeback.c +17 -4
@@ -46,7 +46,20 @@static long ratelimit_pages =32 ;static long total_pages;/* The total number of pages in the machine. */ -static int dirty_exceeded;/* Dirty mem may be over limit */ + +/* +* node_dirty_exceeded is used to indicate that dirty mem in the current node +* maybe over the limit and we need to increase the frequency of calls to +* balance_dirty_pages +*/ +#ifdef CONFIG_NUMA +static DEFINE_PER_CPU(int, _dirty_exceeded) = { 0 }; +#define node_dirty_exceeded \ + per_cpu(_dirty_exceeded, node_to_first_cpu(numa_node_id())) +#else +static int node_dirty_exceeded; +#endif +/* * When balance_dirty_pages decides that the caller needs to perform some
@@ -212,7 +225,7 @@ static void balance_dirty_pages(struct aif (nr_reclaimable + wbs.nr_writeback <= dirty_thresh)break ; - dirty_exceeded =1 ; + node_dirty_exceeded =1 ;/* Note: nr_reclaimable denotes nr_dirty + nr_unstable. * Unstable writes are a feature of certain networked
@@ -235,7 +248,7 @@ static void balance_dirty_pages(struct a }if (nr_reclaimable + wbs.nr_writeback <= dirty_thresh) - dirty_exceeded =0 ; + node_dirty_exceeded =0 ;if (writeback_in_progress (bdi))return ;/* pdflush is already working this queue */
@@ -272,7 +285,7 @@ void balance_dirty_pages_ratelimited(strlong ratelimit; ratelimit = ratelimit_pages; -if (dirty_exceeded) +if (node_dirty_exceeded) ratelimit =8 ;/*
Quoted 4 lines · Ravikiran G Thirumalai
This is a repost. I did not get any comments when I last posted this about a month back. So I guess this patch is all good :)
I thought it was fairly ghastly, sorry ;)
Quoted 11 lines
Convert global
dirty_exceededflag to per-nodenode_dirty_exceeded.
dirty_exceededping pongs between nodes in order to force all cpus in the system to increase the frequency of calls tobalance_dirty_pages.Currently
dirty_exceededis used bybalance_dirty_pages_ratelimitedto force all CPUs in the system callbalance_dirty_pagesoften, in order to reduce the amount of dirty pages in the entire system (based ondirty_threshand one CPU exceeding thee ratelimits). Asdirty_exceededis a global variable, it will ping-pong between nodes of a NUMA system which is not good.
Did you not test this obvious little optimisation?
mm/page-writeback.c renamed from mm/page-writeback.c~mm-dirty_exceeded-speedup +3 -2
@@ -212,7 +212,8 @@ static void balance_dirty_pages(struct aif (nr_reclaimable + wbs.nr_writeback <= dirty_thresh)break ; - dirty_exceeded =1 ; +if (!dirty_exceeded) + dirty_exceeded =1 ;/* Note: nr_reclaimable denotes nr_dirty + nr_unstable. * Unstable writes are a feature of certain networked
@@ -234,7 +235,7 @@ static void balance_dirty_pages(struct ablk_congestion_wait (WRITE ,HZ /10 ); } -if (nr_reclaimable + wbs.nr_writeback <= dirty_thresh) +if (nr_reclaimable + wbs.nr_writeback <= dirty_thresh && dirty_exceeded) dirty_exceeded =0 ;if (writeback_in_progress (bdi))
_
Quoted 7 lines · Andrew Morton
Quoted 6 lines · Ravikiran G Thirumalai
Convert global
dirty_exceededflag to per-nodenode_dirty_exceeded.
dirty_exceededping pongs between nodes in order to force all cpus in the system to increase the frequency of calls tobalance_dirty_pages.
Quoted 8 lines
Quoted 6 lines
Currently
dirty_exceededis used bybalance_dirty_pages_ratelimitedto force all CPUs in the system callbalance_dirty_pagesoften, in order to reduce the amount of dirty pages in the entire system (based ondirty_threshand one CPU exceeding thee ratelimits). Asdirty_exceededis a global variable, it will ping-pong between nodes of a NUMA system which is not good.Did you not test this obvious little optimisation?
We ran the test we encountered this problem on with your patch. At first it looked like it did not help. But later we found that there was false sharing on this variable. So padding dirty_exceeded to cacheline along with your patch helps.
Thanks,
Kiran
Avoid false sharing on dirty_exceeded.
mm/page-writeback.c +1 -1
@@ -46,7 +46,7 @@static long ratelimit_pages =32 ;static long total_pages;/* The total number of pages in the machine. */ -static int dirty_exceeded;/* Dirty mem may be over limit */ +static int dirty_exceeded __cacheline_aligned_in_smp; /* Dirty mem may be over limit *//* * When balance_dirty_pages decides that the caller needs to perform some
Quoted 16 lines · Ravikiran G Thirumalai
Quoted 14 lines · Andrew Morton
Quoted 12 lines · Ravikiran G Thirumalai
Convert global
dirty_exceededflag to per-nodenode_dirty_exceeded.
dirty_exceededping pongs between nodes in order to force all cpus in the system to increase the frequency of calls tobalance_dirty_pages.Currently
dirty_exceededis used bybalance_dirty_pages_ratelimitedto force all CPUs in the system callbalance_dirty_pagesoften, in order to reduce the amount of dirty pages in the entire system (based ondirty_threshand one CPU exceeding thee ratelimits). Asdirty_exceededis a global variable, it will ping-pong between nodes of a NUMA system which is not good.
Quoted 5 lines
Quoted 1 line
Did you not test this obvious little optimisation?
We ran the test we encountered this problem on with your patch. At first it looked like it did not help. But later we found that there was false sharing on this variable.
OK. That's a bit nasty, isn't it? It can work well or poorly for different people depending upon vagaries of .config and the linker.
We should find out what it was sharing _with_. Could you please run
nm -n vmlinux| grep -C5 dirty_exceeded
Quoted 23 lines · Andrew Morton
Quoted 16 lines · Ravikiran G Thirumalai
Quoted 14 lines · Andrew Morton
Quoted 12 lines · Ravikiran G Thirumalai
Convert global
dirty_exceededflag to per-nodenode_dirty_exceeded.
dirty_exceededping pongs between nodes in order to force all cpus in the system to increase the frequency of calls tobalance_dirty_pages.Currently
dirty_exceededis used bybalance_dirty_pages_ratelimitedto force all CPUs in the system callbalance_dirty_pagesoften, in order to reduce the amount of dirty pages in the entire system (based ondirty_threshand one CPU exceeding thee ratelimits). Asdirty_exceededis a global variable, it will ping-pong between nodes of a NUMA system which is not good.Quoted 5 lines
Quoted 1 line
Did you not test this obvious little optimisation?
We ran the test we encountered this problem on with your patch. At first it looked like it did not help. But later we found that there was false sharing on this variable.
Quoted 6 lines
OK. That's a bit nasty, isn't it? It can work well or poorly for different people depending upon vagaries of .config and the linker.
We should find out what it was sharing _with_. Could you please run
nm -n vmlinux| grep -C5
dirty_exceeded
ffffffff805290c0 b irq_dirffffffff80529838 b root_irq_dirffffffff80529880 B max_pfnffffffff80529888 B min_low_pfnffffffff80529890 B max_low_pfnffffffff80529900 B nr_pagecacheffffffff80529908 B nr_swap_pagesffffffff80529980 b boot_pagesetffffffff8052a980 B laptop_modeffffffff8052a984 B block_dumpffffffff8052a988 b dirty_exceededffffffff8052a990 b total_pagesffffffff8052a998 B nr_pdflush_threadsffffffff8052a9a0 b last_empty_jifsffffffff8052a9c0 B slab_reclaim_pagesffffffff8052a9c4 b slab_break_gfp_orderffffffff8052a9c8 b g_cpucache_upffffffff8052a9d0 b cache_chainffffffff8052a9e0 b cache_chain_semffffffff8052aa00 b offslab_limitffffffff8052aa08 B page_cluster
Maybe slab_reclaim_pages is the culprit?
Thanks,
Kiran
Quoted 30 lines · Ravikiran G Thirumalai
Quoted 7 lines
OK. That's a bit nasty, isn't it? It can work well or poorly for different people depending upon vagaries of .config and the linker.
We should find out what it was sharing _with_. Could you please run
nm -n vmlinux| grep -C5
dirty_exceeded
ffffffff805290c0birq_dirffffffff80529838broot_irq_dirffffffff80529880Bmax_pfnffffffff80529888Bmin_low_pfnffffffff80529890Bmax_low_pfnffffffff80529900Bnr_pagecacheffffffff80529908Bnr_swap_pagesffffffff80529980bboot_pagesetffffffff8052a980Blaptop_modeffffffff8052a984Bblock_dumpffffffff8052a988bdirty_exceededffffffff8052a990btotal_pagesffffffff8052a998Bnr_pdflush_threadsffffffff8052a9a0blast_empty_jifsffffffff8052a9c0Bslab_reclaim_pagesffffffff8052a9c4bslab_break_gfp_orderffffffff8052a9c8bg_cpucache_upffffffff8052a9d0bcache_chainffffffff8052a9e0bcache_chain_semffffffff8052aa00boffslab_limitffffffff8052aa08Bpage_cluster
Quoted 2 lines
Maybe
slab_reclaim_pagesis the culprit?
I guess so - pretty much everything else there should be in __read_mostly anwyay, apart from nr_pagecache.
slab_reclaim_pages is just a silly beancounting thing for overcommit. We could give it the approximate-counting treatment like pagecache_acct() I guess.