Might refill_inactive_zone () be too aggressive?

DISCUSSION Marc Singer 2 participants 13 messages Apr 17, 2004

Subsystems (5)

Memory management MGLRU MM reclaim MM swap +1 more subsystems

lore

Share & tools

lore

Marc Singer

author

2004-04-17 06:09 UTC #0 lore raw

On my target board, an ARM cpu with 32MiB or RAM, I'm finding that the performance is quite poor once RAM fills with IO pages and the code pages of the program being executed are evicted.

In my test setup, rootfs is mounted over NFS. The degenerate example is a simple program that copies a 40MiB file over NFS using a read/write loop. As it runs and as memory fills with NFS cached pages, I can watched the VM evict the code that is executing the loop. Since there are no other programs running (no TLB flushes from context switching), there is nothing to stop the VM from aging the code pages. During the copy, it may evict this same page a dozen times. While I understand that this setup by design, I wonder if there isn't something that can (or should) be done to reduce this behavior.

There are a couple of other things to keep in mind.

1) This is an embedded system.
2) The root filesystem will not be NFS mounted in production. The root is most likely to be stored in bootflash.
3) Some of these systems may perform significant amounts of IO, but almost none will be filesystem IO. Thus, there is unlikely to be much hanging about the page cache.
4) Performance in my test scenarios is quite poor. Once I've copied the 40MiB file, executing an 'ls' command may take several seconds while the machine reloads libraries from the NFS server. The cached IO pages hang about in RAM for some time such that any programs executed will experience code page evictions.
5) Removing the reclaim_mapped=1 line improves system response dramatically...just as I'd expect.

So, is this something to worry about? Should it be a tunable feature? Should this be something addressed in the platform specific VM code? --
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

William Lee Irwin III

2004-04-17 06:18 UTC #1 lore raw

Quoted 4 lines · Marc Singer

5) Removing the reclaim_mapped=1 line improves system response dramatically...just as I'd expect.
So, is this something to worry about? Should it be a tunable feature? Should this be something addressed in the platform specific VM code?

A very interesting point there. The tendency to set reclaim_mapped = 1 is controlled by /proc/sys/vm/swappiness; setting that to 0 may improve your performance or behave closer to how the case you cited where vmscan.c never sets reclaim_mapped = 1 improved performance.

The default value is 60, which begins unmapping mapped memory about when 40% of memory is mapped by userspace.

-- wli

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Marc Singer

author

2004-04-17 14:08 UTC #2 lore raw

Quoted 6 lines · William Lee Irwin III

Quoted 4 lines · Marc Singer
5) Removing the reclaim_mapped=1 line improves system response dramatically...just as I'd expect.
So, is this something to worry about? Should it be a tunable feature? Should this be something addressed in the platform specific VM code?

Quoted 7 lines

A very interesting point there. The tendency to set reclaim_mapped = 1 is controlled by /proc/sys/vm/swappiness; setting that to 0 may improve your performance or behave closer to how the case you cited where vmscan.c never sets reclaim_mapped = 1 improved performance.
The default value is 60, which begins unmapping mapped memory about when 40% of memory is mapped by userspace.

I did a little more looking at when reclaim_mapped is set to one. In my case, I don't think that very much memory is mapped. I've got one program running that has one or two code pages, there may be some libraries. The system has 28MiB of free RAM. I don't see how I could be getting more than 20% of RAM mapped.

William Lee Irwin III

2004-04-17 14:21 UTC #3 lore raw

Quoted 6 lines · William Lee Irwin III

Quoted 6 lines
A very interesting point there. The tendency to set reclaim_mapped = 1 is controlled by /proc/sys/vm/swappiness; setting that to 0 may improve your performance or behave closer to how the case you cited where vmscan.c never sets reclaim_mapped = 1 improved performance.
The default value is 60, which begins unmapping mapped memory about when 40% of memory is mapped by userspace.

Quoted 5 lines · Marc Singer

I did a little more looking at when reclaim_mapped is set to one. In my case, I don't think that very much memory is mapped. I've got one program running that has one or two code pages, there may be some libraries. The system has 28MiB of free RAM. I don't see how I could be getting more than 20% of RAM mapped.

So adjusting swappiness didn't help? Does adjusting this help? (uncompiled, untested)

-- wli

include/linux/sysctl.h +1 -0

Index: singer-2.6.5-mm6/include/linux/sysctl.h
===================================================================
@@ -161,6 +161,7 @@
 	VM_MAX_MAP_COUNT=22,	/* int: Maximum number of mmaps/address-space */
 	VM_LAPTOP_MODE=23,	/* vm laptop mode */
 	VM_BLOCK_DUMP=24,	/* block dump mode */
+	VM_MAPPED_SCALE_FACTOR=25, /* scale factor for mapped ratio */
 };

include/linux/swap.h +1 -1

Index: singer-2.6.5-mm6/include/linux/swap.h
===================================================================
@@ -174,7 +174,7 @@
 /* linux/mm/vmscan.c */
 extern int try_to_free_pages(struct zone **, unsigned int, unsigned int);
 extern int shrink_all_memory(int);
-extern int vm_swappiness;
+extern int vm_swappiness, vm_mapped_scale_factor;
 
 #ifdef CONFIG_MMU
 /* linux/mm/shmem.c */

kernel/sysctl.c +11 -0

Index: singer-2.6.5-mm6/kernel/sysctl.c
===================================================================
@@ -766,6 +766,17 @@
 		.strategy	= &sysctl_intvec,
 		.extra1		= &zero,
 	},
+	{
+		.ctl_name	= VM_MAPPED_SCALE_FACTOR,
+		.procname	= "mapped_scale_factor",
+		.data		= &vm_mapped_scale_factor,
+		.maxlen		= sizeof(vm_mapped_scale_factor),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec_minmax,
+		.strategy	= &sysctl_intvec,
+		.extra1		= &zero,
+		.extra2		= &one_hundred,
+	},
 	{ .ctl_name = 0 }
 };

mm/vmscan.c +3 -2

Index: singer-2.6.5-mm6/mm/vmscan.c
===================================================================
@@ -43,6 +43,7 @@
  * From 0 .. 100.  Higher means more swappy.
  */
 int vm_swappiness = 60;
+int vm_mapped_scale_factor = 50;
 static long total_memory;
 
 #define lru_to_page(_head) (list_entry((_head)->prev, struct page, lru))

@@ -625,7 +626,7 @@
 	 * mapped memory instead of just pagecache.  Work out how much memory
 	 * is mapped.
 	 */
-	mapped_ratio = (ps->nr_mapped * 100) / total_memory;
+	mapped_ratio = (ps->nr_mapped * vm_mapped_scale_factor) / total_memory;
 
 	/*
 	 * Now decide how much we really want to unmap some pages.  The mapped

@@ -636,7 +637,7 @@
 	 *
 	 * A 100% value of vm_swappiness overrides this algorithm altogether.
 	 */
-	swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
+	swap_tendency = mapped_ratio + distress + vm_swappiness;
 
 	/*
 	 * Now use this metric to decide whether to start moving mapped memory

William Lee Irwin III

2004-04-17 17:16 UTC #4 lore raw

In reply to Marc Singer

Quoted 6 lines · William Lee Irwin III

Quoted 6 lines
A very interesting point there. The tendency to set reclaim_mapped = 1 is controlled by /proc/sys/vm/swappiness; setting that to 0 may improve your performance or behave closer to how the case you cited where vmscan.c never sets reclaim_mapped = 1 improved performance.
The default value is 60, which begins unmapping mapped memory about when 40% of memory is mapped by userspace.

Quoted 5 lines · Marc Singer

I did a little more looking at when reclaim_mapped is set to one. In my case, I don't think that very much memory is mapped. I've got one program running that has one or two code pages, there may be some libraries. The system has 28MiB of free RAM. I don't see how I could be getting more than 20% of RAM mapped.

Also, it would be helpful to log periodic snapshots of /proc/meminfo and /proc/vmstat to see what's being fed to the various heuristics.

-- wli

Marc Singer

author

2004-04-17 17:57 UTC #5 lore raw

In reply to William Lee Irwin III

Quoted 6 lines · William Lee Irwin III

Quoted 4 lines · Marc Singer
5) Removing the reclaim_mapped=1 line improves system response dramatically...just as I'd expect.
So, is this something to worry about? Should it be a tunable feature? Should this be something addressed in the platform specific VM code?

Quoted 7 lines

A very interesting point there. The tendency to set reclaim_mapped = 1 is controlled by /proc/sys/vm/swappiness; setting that to 0 may improve your performance or behave closer to how the case you cited where vmscan.c never sets reclaim_mapped = 1 improved performance.
The default value is 60, which begins unmapping mapped memory about when 40% of memory is mapped by userspace.

I don't think that's the whole story. I printed distress, mapped_ratio, and swappiness when vmscan starts trying to reclaim mapped pages.

reclaim_mapped: distress 50 mapped_ratio 0 swappiness 60

50 + 60 > 100

So, part of the problem is swappiness. I could set that value to 25, for example, to stop the machine from swapping.

I'd be fine stopping here, except for you comment about what swappiness means. In my case, nearly none of memory is mapped. It is zone priority which has dropped to 1 that is precipitating the eviction. Is this what you expect and want?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

William Lee Irwin III

2004-04-17 18:10 UTC #6 lore raw

Quoted 11 lines · Marc Singer

I don't think that's the whole story. I printed distress, mapped_ratio, and swappiness when vmscan starts trying to reclaim mapped pages.
reclaim_mapped: distress 50 mapped_ratio 0 swappiness 60
50 + 60 > 100
So, part of the problem is swappiness. I could set that value to 25, for example, to stop the machine from swapping.
I'd be fine stopping here, except for you comment about what swappiness means. In my case, nearly none of memory is mapped. It is zone priority which has dropped to 1 that is precipitating the eviction. Is this what you expect and want?

I'm not sure it's expected. Maybe this patch fares better?

-- wli

mm/vmscan.c +1 -1

Index: singer-2.6.5-mm6/mm/vmscan.c
===================================================================
@@ -636,7 +636,7 @@
 	 *
 	 * A 100% value of vm_swappiness overrides this algorithm altogether.
 	 */
-	swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
+	swap_tendency = mapped_ratio / 2 + max(distress, vm_swappiness);
 
 	/*
 	 * Now use this metric to decide whether to start moving mapped memory

Marc Singer

author

2004-04-17 18:28 UTC #7 lore raw

Quoted 13 lines · William Lee Irwin III

Quoted 11 lines · Marc Singer
I don't think that's the whole story. I printed distress, mapped_ratio, and swappiness when vmscan starts trying to reclaim mapped pages.
reclaim_mapped: distress 50 mapped_ratio 0 swappiness 60
50 + 60 > 100
So, part of the problem is swappiness. I could set that value to 25, for example, to stop the machine from swapping.
I'd be fine stopping here, except for you comment about what swappiness means. In my case, nearly none of memory is mapped. It is zone priority which has dropped to 1 that is precipitating the eviction. Is this what you expect and want?

Quoted 1 line

I'm not sure it's expected. Maybe this patch fares better?

Ah, that's a much different thing. That works for me. Is that something you'd want to put into the kernel?

Quoted 10 lines

-- wli
Index: singer-2.6.5-mm6/mm/vmscan.c
===================================================================

mm/vmscan.c line 642

Quoted 8 lines

mm/vmscan.c +1 -1

@@ -636,7 +636,7 @@
 	 *
 	 * A 100% value of vm_swappiness overrides this algorithm altogether.
 	 */
-	swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
+	swap_tendency = mapped_ratio / 2 + max(distress, vm_swappiness);
 
 	/*
 	 * Now use this metric to decide whether to start moving mapped memory

William Lee Irwin III

2004-04-17 18:33 UTC #8 lore raw

Quoted 1 line · William Lee Irwin III

Quoted 1 line
I'm not sure it's expected. Maybe this patch fares better?

Quoted 2 lines · Marc Singer

Ah, that's a much different thing. That works for me. Is that something you'd want to put into the kernel?

Since we have a coherent story about this working for you, I think we should probably send it upstream for review. I don't have a particular opinion about it being the right thing to do, as since it's a policy decision, it's rather arbitrary.

If this is important to you, it may help to numerically quantify your results, e.g. some before/after benchmark/throughput/whatever numbers.

-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Marc Singer

author

2004-04-17 18:44 UTC #9 lore raw

Quoted 7 lines · William Lee Irwin III

Quoted 1 line · William Lee Irwin III
Quoted 1 line
I'm not sure it's expected. Maybe this patch fares better?
Quoted 2 lines · Marc Singer
Ah, that's a much different thing. That works for me. Is that something you'd want to put into the kernel?

Quoted 7 lines

Since we have a coherent story about this working for you, I think we should probably send it upstream for review. I don't have a particular opinion about it being the right thing to do, as since it's a policy decision, it's rather arbitrary.
If this is important to you, it may help to numerically quantify your results, e.g. some before/after benchmark/throughput/whatever numbers.

That's a difficult thing to do. My test setup uses an NFS root and the IO is over NFS. Due to some oddities in the NFS code, performance is variable to a degree that does not make for good timing comparisons. I'm looking for a way to enable TCP nfsroot mounts. Once this is working, I may be able to get some reliable numbers.

It's your call about waiting for performance numbers. As soon as I have better data, I'll post it. Setting the swappiness flag does work, so I can give my users something for now. It is possible that this will work for all cases that I'm ever going to see. Setting swappiness to zero and the user mapping 50% of RAM will once again cause reclaim_mapped to go into action. The difference is that with swappiness of 60, I'm not allowed to keep any mapped pages in RAM. At swappiness of 0, I'm allows to keep half of total RAM mapped.

Cheers.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

William Lee Irwin III

2004-04-17 19:19 UTC #10 lore raw

Quoted 5 lines · Marc Singer

That's a difficult thing to do. My test setup uses an NFS root and the IO is over NFS. Due to some oddities in the NFS code, performance is variable to a degree that does not make for good timing comparisons. I'm looking for a way to enable TCP nfsroot mounts. Once this is working, I may be able to get some reliable numbers.

Quoted 8 lines

It's your call about waiting for performance numbers. As soon as I have better data, I'll post it. Setting the swappiness flag does work, so I can give my users something for now. It is possible that this will work for all cases that I'm ever going to see. Setting swappiness to zero and the user mapping 50% of RAM will once again cause reclaim_mapped to go into action. The difference is that with swappiness of 60, I'm not allowed to keep any mapped pages in RAM. At swappiness of 0, I'm allows to keep half of total RAM mapped.

That's something of a normative question about the heuristics, and I try to steer clear of those, though I'm not entirely sure that's how I would interpret it the tunings for your descriptive parts.

In the absence of "hard" numbers, you might still be able to use things like wall clock timings. Another thing that would help is to expose the thing to a variety of workloads/etc. For that, I guess I post to lkml.

Marc Singer

author

2004-04-17 19:25 UTC #11 lore raw

Quoted 15 lines · William Lee Irwin III

Quoted 5 lines · Marc Singer
That's a difficult thing to do. My test setup uses an NFS root and the IO is over NFS. Due to some oddities in the NFS code, performance is variable to a degree that does not make for good timing comparisons. I'm looking for a way to enable TCP nfsroot mounts. Once this is working, I may be able to get some reliable numbers.
Quoted 8 lines
It's your call about waiting for performance numbers. As soon as I have better data, I'll post it. Setting the swappiness flag does work, so I can give my users something for now. It is possible that this will work for all cases that I'm ever going to see. Setting swappiness to zero and the user mapping 50% of RAM will once again cause reclaim_mapped to go into action. The difference is that with swappiness of 60, I'm not allowed to keep any mapped pages in RAM. At swappiness of 0, I'm allows to keep half of total RAM mapped.

Quoted 3 lines

That's something of a normative question about the heuristics, and I try to steer clear of those, though I'm not entirely sure that's how I would interpret it the tunings for your descriptive parts.

The more I think about it, the more I think that there is something awry. Once distress reaches 50, swappiness is going to rule the ability of the system to keep pages mapped. If swappiness is then 50 or more, vmscan is going age and then purge every mapped page.

Quoted 3 lines

In the absence of "hard" numbers, you might still be able to use things like wall clock timings. Another thing that would help is to expose the thing to a variety of workloads/etc. For that, I guess I post to lkml.

Are you suggesting that I post to LKML to get some ideas about other workloads?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

William Lee Irwin III

2004-04-17 19:45 UTC #12 lore raw

Quoted 3 lines · William Lee Irwin III

Quoted 3 lines
That's something of a normative question about the heuristics, and I try to steer clear of those, though I'm not entirely sure that's how I would interpret it the tunings for your descriptive parts.

Quoted 4 lines · Marc Singer

The more I think about it, the more I think that there is something awry. Once distress reaches 50, swappiness is going to rule the ability of the system to keep pages mapped. If swappiness is then 50 or more, vmscan is going age and then purge every mapped page.

There's not any a priori reason to believe this is wrong that I know of, though it looks like things do better for you when slightly rearranged.

Quoted 3 lines · William Lee Irwin III

Quoted 3 lines
In the absence of "hard" numbers, you might still be able to use things like wall clock timings. Another thing that would help is to expose the thing to a variety of workloads/etc. For that, I guess I post to lkml.

Quoted 2 lines · Marc Singer

Are you suggesting that I post to LKML to get some ideas about other workloads?

Well, yes, but I just did so myself.