powerpc: Set a smaller value for RECLAIM_DISTANCE to enable zone reclaim
PATCH 1/1 powerpc: Set a smaller value for RECLAIM_DISTANCE to enable zone reclaim
I noticed /proc/sys/vm/zone_reclaim_mode was 0 on a ppc64 NUMA box. It gets enabled via this:
/*
* If another node is sufficiently far away then it is better
* to reclaim pages in a zone before going off node. */
if (distance > RECLAIM_DISTANCE)
zone_reclaim_mode = 1;
Since we use the default value of 20 for REMOTE_DISTANCE and 20 for RECLAIM_DISTANCE it never kicks in.
The local to remote bandwidth ratios can be quite large on System p machines so it makes sense for us to reclaim clean pagecache locally before going off node.
The patch below sets a smaller value for RECLAIM_DISTANCE and thus enables zone reclaim.
---
arch/powerpc/include/asm/topology.h +10 -0
@@ -8,6 +8,16 @@ struct device_node;#ifdef CONFIG_NUMA +/* +* Before going off node we want the VM to try and reclaim from the local +* node. It does this if the remote distance is larger than RECLAIM_DISTANCE. +* With the default REMOTE_DISTANCE of 20 and the default RECLAIM_DISTANCE of +* 20, we never reclaim and go off node straight away. +* +* To fix this we choose a smaller value of RECLAIM_DISTANCE. +*/ +#define RECLAIM_DISTANCE 10 +#include <asm/mmzone.h> static inline int cpu_to_node (int cpu)
Hi,
Quoted 2 lines
The patch below sets a smaller value for
RECLAIM_DISTANCEand thus enables zone reclaim.
FYI even with this enabled I could trip it up pretty easily with a multi threaded application. I tried running stream across all threads in node 0. The machine looks like:
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 node 0 free: 30254 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 node 1 free: 31832 MB
Now create some clean pagecache on node 0:
# taskset -c 0 dd if=/dev/zero of=/tmp/bigfile bs=1G count=16 # sync
node 0 free: 12880 MB
node 1 free: 31830 MB
I built stream to use about 25GB of memory. I then ran stream across all threads in node 0:
# OMP_NUM_THREADS=16 taskset -c 0-15 ./stream
We exhaust all memory on node 0, and start using memory on node 1:
node 0 free: 0 MB
node 1 free: 20795 MB
ie about 10GB of node 1. Now if we run the same test with one thread:
# OMP_NUM_THREADS=1 taskset -c 0 ./stream
things are much better:
node 0 free: 11 MB
node 1 free: 31552 MB
Interestingly enough it takes two goes to get completely onto node 0, even with one thread. The second run looks like:
node 0 free: 14 MB
node 1 free: 31811 MB
I had a quick look at the page allocation logic and I think I understand why we would have issues with multple threads all trying to allocate at once.
- The ZONE_RECLAIM_LOCKED flag allows only one thread into zone reclaim at a time, and whatever thread is in zone reclaim probably only frees a small amount of memory. Certainly not enough to satisfy all 16 threads.
- We seem to end up racing between zone_watermark_ok, zone_reclaim and buffered_rmqueue. Since everyone is in here the memory one thread reclaims may be stolen by another thread.
I'm not sure if there is an easy way to fix this without penalising other workloads though.
Anton
Quoted 6 lines · Anton Blanchard
Hi,
Quoted 2 lines
The patch below sets a smaller value for
RECLAIM_DISTANCEand thus enables zone reclaim.
I've no problem with the patch anyway.
Quoted 45 lines
FYI even with this enabled I could trip it up pretty easily with a multi threaded application. I tried running stream across all threads in node 0. The machine looks like:
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 node 0 free: 30254 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 node 1 free: 31832 MBNow create some clean pagecache on node 0:
# taskset -c 0 dd if=/dev/zero of=/tmp/bigfile bs=1G count=16 # syncnode 0 free: 12880 MB
node 1 free: 31830 MBI built stream to use about 25GB of memory. I then ran stream across all threads in node 0:
# OMP_NUM_THREADS=16 taskset -c 0-15 ./stream
We exhaust all memory on node 0, and start using memory on node 1:
node 0 free: 0 MB
node 1 free: 20795 MBie about 10GB of node 1. Now if we run the same test with one thread:
# OMP_NUM_THREADS=1 taskset -c 0 ./stream
things are much better:
node 0 free: 11 MB
node 1 free: 31552 MBInterestingly enough it takes two goes to get completely onto node 0, even with one thread. The second run looks like:
node 0 free: 14 MB
node 1 free: 31811 MBI had a quick look at the page allocation logic and I think I understand why we would have issues with multple threads all trying to allocate at once.
Quoted 8 lines
- The
ZONE_RECLAIM_LOCKEDflag allows only one thread into zone reclaim at a time, and whatever thread is in zone reclaim probably only frees a small amount of memory. Certainly not enough to satisfy all 16 threads.- We seem to end up racing between
zone_watermark_ok,zone_reclaimandbuffered_rmqueue. Since everyone is in here the memory one thread reclaims may be stolen by another thread.
You're pretty much on the button here. Only one thread at a time enters zone_reclaim. The others back off and try the next zone in the zonelist instead. I'm not sure what the original intention was but most likely it was to prevent too many parallel reclaimers in the same zone potentially dumping out way more data than necessary.
Quoted 3 lines
I'm not sure if there is an easy way to fix this without penalising other workloads though.
You could experiment with waiting on the bit if the GFP flags allowi it? The expectation would be that the reclaim operation does not take long. Wait on the bit, if you are making the forward progress, recheck the watermarks before continueing.
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Quoted 5 lines · Mel Gorman
Quoted 3 lines
Quoted 2 lines
The patch below sets a smaller value for
RECLAIM_DISTANCEand thus enables zone reclaim.I've no problem with the patch anyway.
Nor do I.
Quoted 10 lines
Quoted 4 lines
- We seem to end up racing between
zone_watermark_ok,zone_reclaimandbuffered_rmqueue. Since everyone is in here the memory one thread reclaims may be stolen by another thread.You're pretty much on the button here. Only one thread at a time enters
zone_reclaim. The others back off and try the next zone in the zonelist instead. I'm not sure what the original intention was but most likely it was to prevent too many parallel reclaimers in the same zone potentially dumping out way more data than necessary.
Yes it was to prevent concurrency slowing down reclaim. At that time the number of processors per NUMA node was 2 or so. The number of pages that are reclaimed is limited to avoid tossing too many page cache pages.
Quoted 4 lines
You could experiment with waiting on the bit if the GFP flags allowi it? The expectation would be that the reclaim operation does not take long. Wait on the bit, if you are making the forward progress, recheck the watermarks before continueing.
You could reclaim more pages during a zone reclaim pass? Increase the nr_to_reclaim in __zone_reclaim() and see if that helps. One zone reclaim pass should reclaim enough local pages to keep the processors on a node happy for a reasonable interval. Maybe do a fraction of a zone? 1/16th?
Quoted 3 lines · Christoph Lameter
Quoted 1 line · Mel Gorman
Quoted 1 line
Quoted 1 line
The patch below sets a smaller value for
RECLAIM_DISTANCEand thus e=
nables
Quoted 9 lines
Quoted 4 lines
Quoted 2 lines
Quoted 1 line
zone reclaim.
I've no problem with the patch anyway.
Nor do I.
Quoted 2 lines
Quoted 2 lines
- We seem to end up racing between
zone_watermark_ok,zone_reclaimand =A0buffered_rmqueue. Since everyone is in here the memory one thread =
reclaims
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
=A0 may be stolen by another thread.
You're pretty much on the button here. Only one thread at a time enters
zone_reclaim. The others back off and try the next zone in the zonelist instead. I'm not sure what the original intention was but most likely it was to prevent too many parallel reclaimers in the same zone potentially dumping out way more data than necessary.
Quoted 4 lines
Yes it was to prevent concurrency slowing down reclaim. At that time the number of processors per NUMA node was 2 or so. The number of pages that are reclaimed is limited to avoid tossing too many page cache pages.
That is interesting, I always thought it was to try and free page cache first. For example with zone->min_unmapped_pages, if zone_pagecache_reclaimable is greater than unmapped pages, we start reclaim the cached pages first. The min_unmapped_pages almost sounds like the higher level watermark - or am I misreading the code.
Balbir Singh
Quoted 10 lines · Balbir Singh
Quoted 9 lines
Quoted 4 lines
zone_reclaim. The others back off and try the next zone in the zonelist instead. I'm not sure what the original intention was but most likely it was to prevent too many parallel reclaimers in the same zone potentially dumping out way more data than necessary.Yes it was to prevent concurrency slowing down reclaim. At that time the number of processors per NUMA node was 2 or so. The number of pages that are reclaimed is limited to avoid tossing too many page cache pages.
Quoted 5 lines
That is interesting, I always thought it was to try and free page cache first. For example with
zone->min_unmapped_pages, ifzone_pagecache_reclaimableis greater than unmapped pages, we start reclaim the cached pages first. Themin_unmapped_pagesalmost sounds like the higher level watermark - or am I misreading the code.
Indeed the purpose is to free *old* page cache pages.
The min_unmapped_pages is to protect a mininum of the page cache pages / fs metadata from zone reclaim so that ongoing file I/O is not impacted.
Quoted 18 lines · Christoph Lameter
Quoted 10 lines · Balbir Singh
Quoted 9 lines
Quoted 4 lines
zone_reclaim. The others back off and try the next zone in the zonelist instead. I'm not sure what the original intention was but most likely it was to prevent too many parallel reclaimers in the same zone potentially dumping out way more data than necessary.Yes it was to prevent concurrency slowing down reclaim. At that time the number of processors per NUMA node was 2 or so. The number of pages that are reclaimed is limited to avoid tossing too many page cache pages.
Quoted 5 lines
That is interesting, I always thought it was to try and free page cache first. For example with
zone->min_unmapped_pages, ifzone_pagecache_reclaimableis greater than unmapped pages, we start reclaim the cached pages first. Themin_unmapped_pagesalmost sounds like the higher level watermark - or am I misreading the code.
Quoted 4 lines
Indeed the purpose is to free *old* page cache pages.
The
min_unmapped_pagesis to protect a mininum of the page cache pages / fs metadata from zone reclaim so that ongoing file I/O is not impacted.
Thanks for the explanation!
Signature · 2 lines
Three Cheers, Balbir
Hi Mel,
Quoted 6 lines
You're pretty much on the button here. Only one thread at a time enters
zone_reclaim. The others back off and try the next zone in the zonelist instead. I'm not sure what the original intention was but most likely it was to prevent too many parallel reclaimers in the same zone potentially dumping out way more data than necessary.
Quoted 8 lines
Quoted 3 lines
I'm not sure if there is an easy way to fix this without penalising other workloads though.
You could experiment with waiting on the bit if the GFP flags allowi it? The expectation would be that the reclaim operation does not take long. Wait on the bit, if you are making the forward progress, recheck the watermarks before continueing.
Thanks to you and Christoph for some suggestions to try. Attached is a chart showing the results of the following tests:
baseline.txt
The current ppc64 default of zone_reclaim_mode = 0. As expected we see no change in remote node memory usage even after 10 iterations.
zone_reclaim_mode.txt
Now we set zone_reclaim_mode = 1. On each iteration we continue to improve, but even after 10 runs of stream we have > 10% remote node memory usage.
reclaim_4096_pages.txt
Instead of reclaiming 32 pages at a time, we try for a much larger batch of 4096. The slope is much steeper but it still takes around 6 iterations to get almost all local node memory.
wait_on_busy_flag.txt
Here we busy wait if the ZONE_RECLAIM_LOCKED flag is set. As you suggest we would need to check the GFP flags etc, but so far it looks the most promising. We only get a few percent of remote node memory on the first iteration and get all local node by the second.
Perhaps a combination of larger batch size and waiting on the busy flag is the way to go?
Anton
vmscan.c renamed from vmscan.c~ +1 -1
@@ -2534,7 +2534,7 @@ .may_unmap = !!(zone_reclaim_mode & RECLAIM_SWAP), .may_swap = 1, .nr_to_reclaim = max_t(unsigned long, nr_pages, - SWAP_CLUSTER_MAX), + 4096), .gfp_mask = gfp_mask, .swappiness = vm_swappiness, .order = order,
vmscan.c renamed from vmscan.c~ +2 -2
@@ -2634,8 +2634,8 @@if (node_state (node_id,N_CPU ) && node_id !=numa_node_id ())return ZONE_RECLAIM_NOSCAN ; -if (zone_test_and_set_flag (zone,ZONE_RECLAIM_LOCKED )) -return ZONE_RECLAIM_NOSCAN ; +while (zone_test_and_set_flag (zone,ZONE_RECLAIM_LOCKED )) +cpu_relax (); ret =__zone_reclaim (zone, gfp_mask, order);zone_clear_flag (zone,ZONE_RECLAIM_LOCKED );
Quoted 3 lines · Anton Blanchard
Hi Mel,
I'm afraid I'm on vacation at the moment. This mail is costing me shots with penaltys every minute it's open. It'll be early next week before I can look at this closely.
Sorry.
Quoted 70 lines
Quoted 6 lines
You're pretty much on the button here. Only one thread at a time enters
zone_reclaim. The others back off and try the next zone in the zonelist instead. I'm not sure what the original intention was but most likely it was to prevent too many parallel reclaimers in the same zone potentially dumping out way more data than necessary.Quoted 8 lines
Quoted 3 lines
I'm not sure if there is an easy way to fix this without penalising other workloads though.
You could experiment with waiting on the bit if the GFP flags allowi it? The expectation would be that the reclaim operation does not take long. Wait on the bit, if you are making the forward progress, recheck the watermarks before continueing.
Thanks to you and Christoph for some suggestions to try. Attached is a chart showing the results of the following tests:
baseline.txt
The current ppc64 default ofzone_reclaim_mode= 0. As expected we see no change in remote node memory usage even after 10 iterations.zone_reclaim_mode.txt
Now we setzone_reclaim_mode= 1. On each iteration we continue to improve, but even after 10 runs of stream we have > 10% remote node memory usage.reclaim_4096_pages.txt
Instead of reclaiming 32 pages at a time, we try for a much larger batch of 4096. The slope is much steeper but it still takes around 6 iterations to get almost all local node memory.wait_on_busy_flag.txt
Here we busy wait if theZONE_RECLAIM_LOCKEDflag is set. As you suggest we would need to check the GFP flags etc, but so far it looks the most promising. We only get a few percent of remote node memory on the first iteration and get all local node by the second.Perhaps a combination of larger batch size and waiting on the busy flag is the way to go?
Anton
vmscan.c renamed from vmscan.c~ +1
-1@@ -2534,7 +2534,7 @@ .may_unmap = !!(zone_reclaim_mode & RECLAIM_SWAP), .may_swap = 1, .nr_to_reclaim = max_t(unsigned long, nr_pages, - SWAP_CLUSTER_MAX), + 4096), .gfp_mask = gfp_mask, .swappiness = vm_swappiness, .order = order,vmscan.c renamed from vmscan.c~ +2
-2@@ -2634,8 +2634,8 @@if (node_state (node_id,N_CPU ) && node_id !=numa_node_id ())return ZONE_RECLAIM_NOSCAN ; -if (zone_test_and_set_flag (zone,ZONE_RECLAIM_LOCKED )) -return ZONE_RECLAIM_NOSCAN ; +while (zone_test_and_set_flag (zone,ZONE_RECLAIM_LOCKED )) +cpu_relax (); ret =__zone_reclaim (zone, gfp_mask, order);zone_clear_flag (zone,ZONE_RECLAIM_LOCKED );
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Quoted 3 lines · Anton Blanchard
zone_reclaim_mode.txt
Now we setzone_reclaim_mode= 1. On each iteration we continue to improve, but even after 10 runs of stream we have > 10% remote node memory usage.
The intend of zone reclaim was never to allocate all memory from on node. You should not expect all memory to come from the node even if zone reclaim works.
Quoted 4 lines
reclaim_4096_pages.txt
Instead of reclaiming 32 pages at a time, we try for a much larger batch of 4096. The slope is much steeper but it still takes around 6 iterations to get almost all local node memory.
"almost all"? How much do you want?
Quoted 5 lines
wait_on_busy_flag.txt
Here we busy wait if theZONE_RECLAIM_LOCKEDflag is set. As you suggest we would need to check the GFP flags etc, but so far it looks the most promising. We only get a few percent of remote node memory on the first iteration and get all local node by the second.
This would significantly impact performance. Zone reclaim should reclaim with minimal overhead. If zone reclaim is running on another processor then the OS already takes measures against the shortage of node local memory. The right thing to do is to take what is currently available which may be off node memory.
--- mm/vmscan.c~ 2010-02-21 23:47:14.000000000 -0600
+++ mm/vmscan.c 2010-02-22 03:22:01.000000000 -0600
@@ -2534,7 +2534,7 @@
.may_unmap = !!(zone_reclaim_mode & RECLAIM_SWAP),
.may_swap = 1,
.nr_to_reclaim = max_t(unsigned long, nr_pages,
- SWAP_CLUSTER_MAX),
+ 4096),
.gfp_mask = gfp_mask,
.swappiness = vm_swappiness,
.order = order,
--- mm/vmscan.c~ 2010-02-21 23:47:14.000000000 -0600
+++ mm/vmscan.c 2010-02-21 23:47:31.000000000 -0600
@@ -2634,8 +2634,8 @@
if (node_state(node_id, N_CPU) && node_id != numa_node_id())
return ZONE_RECLAIM_NOSCAN;
- if (zone_test_and_set_flag(zone, ZONE_RECLAIM_LOCKED))
- return ZONE_RECLAIM_NOSCAN;
+ while (zone_test_and_set_flag(zone, ZONE_RECLAIM_LOCKED))
+ cpu_relax();
ret = __zone_reclaim(zone, gfp_mask, order);
zone_clear_flag(zone, ZONE_RECLAIM_LOCKED);
Quoted 3 lines · Anton Blanchard
Hi Mel,
I'm back but a bit vague. Am on painkillers for the bashing I gave myself down the hills.
Quoted 19 lines
Quoted 6 lines
You're pretty much on the button here. Only one thread at a time enters
zone_reclaim. The others back off and try the next zone in the zonelist instead. I'm not sure what the original intention was but most likely it was to prevent too many parallel reclaimers in the same zone potentially dumping out way more data than necessary.Quoted 8 lines
Quoted 3 lines
I'm not sure if there is an easy way to fix this without penalising other workloads though.
You could experiment with waiting on the bit if the GFP flags allowi it? The expectation would be that the reclaim operation does not take long. Wait on the bit, if you are making the forward progress, recheck the watermarks before continueing.
Thanks to you and Christoph for some suggestions to try. Attached is a chart showing the results of the following tests:
Quoted 8 lines
baseline.txt
The current ppc64 default ofzone_reclaim_mode= 0. As expected we see no change in remote node memory usage even after 10 iterations.zone_reclaim_mode.txt
Now we setzone_reclaim_mode= 1. On each iteration we continue to improve, but even after 10 runs of stream we have > 10% remote node memory usage.
Ok, so how reasonable would it be to expect that the rate of "improvement" to be related to the ratio between "available free node memory at start - how many pages the benchmark requires" and the number of pages zone_reclaim reclaims on the local node?
The exact rate of improvement is complicated by multiple threads so it won't be exact.
Quoted 11 lines
reclaim_4096_pages.txt
Instead of reclaiming 32 pages at a time, we try for a much larger batch of 4096. The slope is much steeper but it still takes around 6 iterations to get almost all local node memory.wait_on_busy_flag.txt
Here we busy wait if theZONE_RECLAIM_LOCKEDflag is set. As you suggest we would need to check the GFP flags etc, but so far it looks the most promising. We only get a few percent of remote node memory on the first iteration and get all local node by the second.
If the above expectation is reasonable, a better alternative may be to adapt the number of pages reclaimed to the number of callers to __zone_reclaim() and allow parallel reclaimers.
e.g.
1 thread 128 2 threads 64 3 threads 32 4 threads 16
etc
The exact starting batch count needs more careful thinking than what I'm giving it currently and maybe the decay ratio too to work out what the worst-case scenario for dumping node-local memory is but you get the idea.
The downside is that this requires a per-zone counter to count the number of parallel reclaimers.
Quoted 4 lines
Perhaps a combination of larger batch size and waiting on the busy flag is the way to go?
I think a static increase on the batch size runs three risks. The first of parallel reclaimers dumping too much of local memory although it could be mitigated by checking the watermarks after waiting on the bit lock. The second is that the thread doing the reclaiming is penalised with higher reclaim costs while other CPUs remain idle. The third is that there could be latency snags with a thread spinning that would previously have gone off-node.
Not sure what the impact of the third risk but it might be noticeable on latency-sensitive machines where the off-node cost is not significant enough to justify a delay.
Christoph, how feasible would it be to allow parallel reclaimers in __zone_reclaim() that back off at a rate depending on the number of reclaimers?
Quoted 25 lines
vmscan.c renamed from vmscan.c~ +1
-1@@ -2534,7 +2534,7 @@ .may_unmap = !!(zone_reclaim_mode & RECLAIM_SWAP), .may_swap = 1, .nr_to_reclaim = max_t(unsigned long, nr_pages, - SWAP_CLUSTER_MAX), + 4096), .gfp_mask = gfp_mask, .swappiness = vm_swappiness, .order = order,vmscan.c renamed from vmscan.c~ +2
-2@@ -2634,8 +2634,8 @@if (node_state (node_id,N_CPU ) && node_id !=numa_node_id ())return ZONE_RECLAIM_NOSCAN ; -if (zone_test_and_set_flag (zone,ZONE_RECLAIM_LOCKED )) -return ZONE_RECLAIM_NOSCAN ; +while (zone_test_and_set_flag (zone,ZONE_RECLAIM_LOCKED )) +cpu_relax (); ret =__zone_reclaim (zone, gfp_mask, order);zone_clear_flag (zone,ZONE_RECLAIM_LOCKED );
Signature · 3 lines
Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab
Quoted 3 lines · Mel Gorman
Christoph, how feasible would it be to allow parallel reclaimers in
__zone_reclaim()that back off at a rate depending on the number of reclaimers?
Not too hard. Zone locking is there but there may be a lot of bouncing cachelines if you run it concurrently.
Quoted 2 lines · Anton Blanchard
I noticed
/proc/sys/vm/zone_reclaim_modewas 0 on a ppc64 NUMA box. It ge=
ts
Quoted 4 lines
enabled via this:
=A0 =A0 =A0 =A0/*
=A0 =A0 =A0 =A0 * If another node is sufficiently far away then it is bet=
ter
Quoted 10 lines
=A0 =A0 =A0 =A0 * to reclaim pages in a zone before going off node. =A0 =A0 =A0 =A0 */
=A0 =A0 =A0 =A0if (distance >RECLAIM_DISTANCE)
=A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0zone_reclaim_mode =3D 1;Since we use the default value of 20 for
REMOTE_DISTANCEand 20 forRECLAIM_DISTANCEit never kicks in.The local to remote bandwidth ratios can be quite large on System p machines so it makes sense for us to reclaim clean pagecache locally befo=
re
Quoted 3 lines
going off node.
The patch below sets a smaller value for
RECLAIM_DISTANCEand thus enable=
s
Quoted 2 lines
zone reclaim.
A reclaim distance of 10 implies a ratio of 1, that means we'll always do zone_reclaim() to free page cache and slab cache before moving on to another node?
Balbir Singh.
Hi Balbir,
Quoted 3 lines
A reclaim distance of 10 implies a ratio of 1, that means we'll always do
zone_reclaim()to free page cache and slab cache before moving on to another node?
I want to make an effort to reclaim local pagecache before ever going off node. As an example, a completely off node stream result is almost 3x slower than on node on my test box.
Anton