swapping and the value of /proc/sys/vm/swappiness

Andrew (et al),

The attached results started as an exercise to try to understand what value of "swappiness" we should be recommending to our Altix customers when they start running Linux 2.6 kernels. The benchmark is very simple -- a task first mallocs around 90% of memory, touches all of the memory, then sleeps forever. After the task begins to sleep, we start up a bunch of "dd" copies. When the dd's all complete, we record the amount of swap used, the size of the page cache, and the data rates for the dd's. (Exact details are given in the attachment.) The benchmark was repeated for swappiness values of 0, 20, 40, 60, 80, 100, for a number of recent 2.6 kernels.

What is unexpected is that the amount of swap space used at a particular
swappiness setting varies dramatically with the kernel version being tested, in spite of the fact that the basic swap_tendency calculation in refile_ianctive_zone() is unchanged. (Other, subtle changes in the vm as a whole and this routine in particular clearly effect the impact of that computation.)

For example, at a swappiness value of 0, Kernel 2.6.5 swapped out 0 bytes, whereas Kernel 2.6.9-rc1-mm3 swapped out 10 GB. Similarly, most kernels have a significant change in behavior for swappiness values near 100, but for SLES9 the change point occurs at swappness=60.

A scan of the change logs for swappiness related changes shows nothing that might explain these changes. My question is: "Is this change in behavior deliberate, or just a side effect of other changes that were made in the vm?" and "What kind of swappiness behavior might I expect to find in future kernels?".

Lots more detail is in the attachment.

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

Swappiness Tests
----------------

The following was the result of attempt to quantify what swappiness values people should be using on Altix. As can be seen from the tables, swappiness seems to have undergone a minor change in meaning or implementation between kernel 2.6.6 and 2.6.7, with some minor changes in behavior after that.

The benchmark was to do the following:

(1) Malloc and touch around 90% of memory. (The test system has 29.4 GB of RAM, of which 27.7 were "free" after system boot. The test program malloc'd 25.3 GB, leaving 2.4 GB free, or 8.6% memory free.) Once the memory has been touched, the malloc program sleeps forever, not touching the memory again.
(2) Start up 8 dd copies, copying around 17 GB of data from /dev/zero to a disk file; each dd was targeted to a different physical disk.
(3) When the dd's completed, record the amount of swap used and

     the amount of page cache used.  Also record the run times (

(4) Kill off the malloc process, remove the dd output files, clear
swap by swapon/swapoff. (Removing the files frees up any
remaining buffer cache associated with those files.)
(5) Repeat above so that we have 5 trials of each case.

The test system was a 32 CPU Altix system.

The benchmarks were repeated for swappiness values of 0, 20, 40, 60, 80, and 100. The tests are moderately time consuming, so that limited the number of swappiness values that could be benchmarked.

The tables below show total I/O rate in MB/s for the dd commands, avg swap (of the 5 trials), min and max observed swap, average page cache size, min and max page cache size (all of the latter are in MB). Also, as a general rule, the data shows it is better from an I/O bandwidth standpoint to run with a smaller page cache and not allow swapping to occur.

Here is a discussion of the results in the tables:

(1) 2.6.5 and 2.6.6: These two kernels behave about the same, with practically nothing swapped out until swappiness is 100, at which point the entire malloc'd area is swapped out. The I/O rate for 2.6.6 is significantly below that of SLES9.

(2) 2.6.5-7.97-default (SLES9) Swaps nothing until swappiness is >=60 at which point all of the malloc'd area is swapped out. I/O rate
is high until swapping starts. This kernel is the only one
tested fow which the swappiness change point was swappiness=60 instead of swappiness=100.

(3) 2.6.7: Avg swapout is 1600-1900 MB for all values of swappiness below 100, but there is wide variation in the trials (see the max and min values). Max I/O rate is significantly better than for previous kernels. At swappiness of 100, entire malloc'd area is swapped out.

(4) 2.6.8.1-mm4: much like 2.6.7, except the average swap is smaller below swappiness of 100. I/O rate is similar to that of 2.6.7. Still significant variations among the trials, but not quite as severe as 2.6.7.

(5) 2.6.9-rc1-mm3: Well, it almost looks as if the swappiness code is broken in this version. Even for swappiness of 0, it swaps out 10 GB worth of data. Can this be right?

A comparison of the refill_inactive_zone() routines from the above kernel versions shows that there are subtle differences in the scanning loop, but that the base calculation for swap_tendency (and hence for the influence of swappiness on the decision to set reclaim_mapped) are the same.

Scanning the changelogs for swappiness doesn't bring up any changes, so I wonder if this change was deliberate or inadvertent?

So my question is, is all of this intended, and which variation of swappiness behavior is the one I should expect in future kernels?

Kernel Version 2.6.5:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   279.56 MB/s      0 MB (     0,     0)   3121 MB (  3009,  3233)
  20   273.99 MB/s      0 MB (     0,     0)   3060 MB (  3024,  3104)
  40   285.12 MB/s      0 MB (     0,     0)   2996 MB (  2945,  3057)
  60   289.46 MB/s    115 MB (    62,   185)   3120 MB (  3056,  3167)
  80   288.58 MB/s    103 MB (    68,   212)   3181 MB (  3104,  3265)
 100   151.31 MB/s  25513 MB ( 25380, 25699)  27760 MB ( 27569, 28093)

Kernel Version 2.6.5-7.97-default (SLES9):
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   273.57 MB/s      0 MB (     0,     0)   3191 MB (  3168,  3229)
  20   273.75 MB/s      0 MB (     0,     0)   3151 MB (  3088,  3180)
  40   273.52 MB/s      0 MB (     0,     0)   3096 MB (  3076,  3124)
  60   229.01 MB/s  23068 MB ( 22042, 24195)  25564 MB ( 24578, 26689)
  80   195.63 MB/s  25587 MB ( 25227, 25815)  28046 MB ( 27681, 28260)
 100   184.30 MB/s  26006 MB ( 26006, 26006)  28388 MB ( 28349, 28434)

Kernel Version 2.6.6:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   242.47 MB/s      0 MB (     0,     0)   3195 MB (  3138,  3266)
  20   256.06 MB/s      0 MB (     0,     0)   3170 MB (  3074,  3234)
  40   267.29 MB/s      0 MB (     0,     0)   3189 MB (  3137,  3234)
  60   289.43 MB/s    666 MB (    72,  1680)   3847 MB (  3296,  4817)
  80   286.49 MB/s    170 MB (    86,   393)   3211 MB (  2897,  3618)
 100   154.87 MB/s  24663 MB ( 24320, 25054)  26708 MB ( 26274, 27154)

Kernel Version 2.6.7:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   287.52 MB/s   1688 MB (  1590,  2069)   4966 MB (  4819,  5363)
  20   289.80 MB/s   1838 MB (    25,  3741)   5081 MB (  3265,  6932)
  40   290.39 MB/s   1970 MB (  1593,  3453)   5270 MB (  4899,  6660)
  60   290.25 MB/s   1271 MB (     4,  1591)   4559 MB (  3297,  4915)
  80   288.89 MB/s   1599 MB (     6,  3220)   4876 MB (  3345,  6436)
 100   158.67 MB/s  25768 MB ( 25474, 26004)  28363 MB ( 27968, 28753)

Kernel Version 2.6.8.1-mm4:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   287.28 MB/s    710 MB (    46,  3060)   4082 MB (  3426,  6308)
  20   288.05 MB/s    508 MB (    94,  1417)   3848 MB (  3442,  4739)
  40   287.03 MB/s    588 MB (   199,  1251)   3909 MB (  3570,  4515)
  60   290.08 MB/s    640 MB (   210,  1190)   3976 MB (  3538,  4531)
  80   287.73 MB/s    693 MB (   316,  1195)   4049 MB (  3713,  4545)
 100   166.17 MB/s  26001 MB ( 26001, 26002)  28798 MB ( 28740, 28852)

Kernel Version 2.6.9-rc1-mm3:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   274.80 MB/s  10511 MB (  5644, 14492)  13293 MB (  8596, 17156)
  20   267.02 MB/s  12624 MB (  5578, 16287)  15298 MB (  8468, 18889)
  40   267.66 MB/s  13541 MB (  6619, 17461)  16199 MB (  9393, 20044)
  60   233.73 MB/s  18094 MB ( 16550, 19676)  20629 MB ( 19103, 22192)
  80   213.64 MB/s  20950 MB ( 15844, 22977)  23450 MB ( 18496, 25440)
 100   164.58 MB/s  26004 MB ( 26004, 26004)  28410 MB ( 28327, 28455)

It'll be accidental side-effects arising from changes to other parts of the page reclaim code.

Hopefully very little. Unless we choose to deliberately change the swapout behaviour. The code in there is complex and as you've seen, has surprising interactions. And changes have been made without sufficiently broad testing.

So I'll be setting the bar much higher for changes to vmscan.c. It takes a *lot* of work to demonstrate that a change in there does what it's supposed to do without breaking other things.

That being said, your tests are interesting. There's a wide spread of results across different kernel versions and across different swappiness settings. But the question is: which behaviour is correct for your users, and why?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andrew,

Behavior more like that of 2.6.5 and 2.6.6 is what we would like to see, I think. We have had problems in the past with a single large HPC application that runs for a long time then wants to push its data out quickly. What happens to us in 2.4.21 is that the page cache pages swap out the user pages, and that is somethine we would like to avoid, since it can reduce the data rate significantly.

We were planning on suggesting that such users set swappiness=0 to give
user pages priority over the page cache pages. But it doesn't look like that works very well in the more recent kernels.

One (perhaps) desirable feature would be for intermediate values of swappiness to have behavior in between the two extremes (mapped pages have higher priority vs page cache pages having priority over unreferenced mapped pages), so that one would have finer grain control over the amount of swap used. I'm not sure how to achieve such a goal, however. :-)

On a separate issue, the response to my proposal for a mempolicy to control allocation of page cache pages has been <ahem> underwhelming.

(See: http://marc.theaimsgroup.com/?l=linux-mm&m=109416852113561&w=2
and http://marc.theaimsgroup.com/?l=linux-mm&m=109416852416997&w=2 )

I wonder if this is because I just posted it to linux-mm or its not fleshed out enough yet to be interesting?

Thanks,

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

You probably need to decrease /proc/sys/vm/dirty_ratio and dirty_background_ratio by a lot. That will reduce the amount of unreclaimable pagecache and will take pressure off page reclaim.

Also, converting the application to explicitly tell the kernel that it doesn't want certain data cached will help things a lot. posix_fadvise(POSIX_FADV_DONTNEED), preferably preceded by fsync().

As I say above: avoiding putting all that pressure onto page reclaim in the first case would be preferable to trying to fix stuff up after it has happened.

General brain-fry, I expect. There's a lot happening. --
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Priority paging again? A perennial suggestion.

It was very noncontroversial. Since it's apparently useful to someone and generally low-impact it should probably be merged.

-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I guess reclaim_mapped is effectively priority paging. But is reasonably fragile I guess.

My mapped_page_cost stuff is possibly (I hope) more robust in theory, but the change from never scanning mapped pages until some point, to always scanning mapped pages slowly necessitated non trivial changes to things like handling of use-once pages.

Yeah, I couldn't see any reason to not go ahead with it, which is why I didn't say anything :)

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Cool. I'll go ahead and finish it and make it something useful then.

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

IIRC no deliberate /proc/sys/vm/swappiness semantic changes were merged. The policy tweakers have something to answer for here unless some stats they rely upon have since been flubbed. Logging periodic snapshots of /proc/vmstat for these benchmarks may be helpful to implicate specific statistics' bungling or rule out statistic miscalculation as causes.

-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

The change was not deliberate but there have been some other people report significant changes in the swappiness behaviour as well (see archives). It has usually been of the increased swapping variety lately. It has been annoying enough to the bleeding edge desktop users for a swag of out-of-tree hacks to start appearing (like mine).

Cheers,
Con

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

All of which is largely wasted effort. It would be much more useful to get down and identify which patch actually caused the behavioural change. --
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I don't disagree. Is there anyone who has the time and is willing to do the regression testing? This is a general appeal to the mailing list.

Cheers,
Con

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi kernel fellows,

I volunteer. I'll try something tomorrow to compare swappiness of older kernels like 2.6.5 and 2.6.6, which were fine on SGI's Altix tests, up to current newer kernels (on small memory boxes of course).

Someone needs to write a vmstat-like tool to parse /proc/vmstat. The statistics in there allows us to watch the behaviour of VM page reclaim code.

Con, if you could compile a list of reports we would be very grateful.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Apart from lots of "soft" reports I've been getting, the most obvious one recently on the mailing list is this:

http://marc.theaimsgroup.com/?l=linux-kernel&m=109237941331221&w=2

and no, I'm not referring to this thread because he tried one of my patches; that's an old patch that I'm not even pushing any more.

Cheers,
Con

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hi Marcelo,

Just a suggestion - I'd look at the thrashing control patch first. I bet that's the cause.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Good point!

I recall one of my users found his workload which often hit swap lightly was swapping much heavier and his performance dropped dramatically until I stopped including the swap thrash control patch. I informed Rik about it some time back so I'm not sure if he addressed it in the meantime.

Cheers,
Con

Swap thrashing code doesnt affect anything, at least on my simple contained test. With the same test, the amount of swapped out memory with 2.6.6/2.6.7 is 100-150MB,
while 2.6.8/2.6.9-mm* swaps out around 250MB.

I tried 2.6.7's "vmscan.c" on 2.6.8 without noticeable difference, I wonder why.

What I've noticed before with the swap token code is total crap interactivity when memory hog is running. Which doesnt happen without it.

Con, I've seen your hard swappiness patch, why do you remove the current swap_tendency calculation? Can you give us some insight into it?

The thing is, if the user thinks the machine is swapping out too heavily he can always decrease vm_swappinness. Whatever change that might happen on VM swapout policy can be tuned with vm_swappinness.

It works - its not very smooth, changing from "53" to "50" causes the amount of swapped data to be 4 times smaller (due to
if (swap_tendency >= 100) I believe). Apart from that its fine, and behaves as expected.

Maybe the current value of "60" is too high for most desktop users, if so it can be decreased a little bit.

But whats the point of your patch?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sure. It was painfully simple. The swap tendency worked basically the same but did not take into account distress. ie It made the "swappiness" knob purely dependant on mapped ratio. For whatever reason, if the swappiness value is the same in later kernels but swaps more, there is more "distress" meaning we are priority scanning much more aggressively.

Cheers,
Con

The token based thrashing control patch is also in 2.6.8.1-mm4, and that kernel doesn't behave nearly as badly as 2.5.9-rc1-mm3, so I don't think that is the culprit in that case.

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Spent some time doing a few tests.

A contained test (512MB box, allocate 450MB of memory, touch and sleep + huge dd) show's that 2.6.6 and 2.6.7 are equivalent (swapout 120-150M when the "dd" starts writing out data).

2.6.9-rc1 swaps out 350M on the same test. Doenst seem nice, I'll try to isolate what causes this tomorrow (this is probably what is hitting desktop users which is fixed by Con's patches). Thats problem #1. Something needs a little tweaking I think.

Ray, I see the additional swapouts increase the dd performance for your particular testcase:

on 2.6.6:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   242.47 MB/s      0 MB (     0,     0)   3195 MB (  3138,  3266)
  20   256.06 MB/s      0 MB (     0,     0)   3170 MB (  3074,  3234)
  40   267.29 MB/s      0 MB (     0,     0)   3189 MB (  3137,  3234)
  60   289.43 MB/s    666 MB (    72,  1680)   3847 MB (  3296,  4817)		<---------- 

So for this one testcase it is being beneficial.

However, decreasing the "swappiness" value does not seem to make much of a difference:

Kernel Version 2.6.8.1-mm4:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   287.28 MB/s    710 MB (    46,  3060)   4082 MB (  3426,  6308)
  20   288.05 MB/s    508 MB (    94,  1417)   3848 MB (  3442,  4739)
  40   287.03 MB/s    588 MB (   199,  1251)   3909 MB (  3570,  4515)
  60   290.08 MB/s    640 MB (   210,  1190)   3976 MB (  3538,  4531)
  80   287.73 MB/s    693 MB (   316,  1195)   4049 MB (  3713,  4545)
 100   166.17 MB/s  26001 MB ( 26001, 26002)  28798 MB ( 28740, 28852)

Kernel Version 2.6.9-rc1-mm3:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   274.80 MB/s  10511 MB (  5644, 14492)  13293 MB (  8596, 17156)
  20   267.02 MB/s  12624 MB (  5578, 16287)  15298 MB (  8468, 18889)
  40   267.66 MB/s  13541 MB (  6619, 17461)  16199 MB (  9393, 20044)
  60   233.73 MB/s  18094 MB ( 16550, 19676)  20629 MB ( 19103, 22192)
  80   213.64 MB/s  20950 MB ( 15844, 22977)  23450 MB ( 18496, 25440)
 100   164.58 MB/s  26004 MB ( 26004, 26004)  28410 MB ( 28327, 28455)

And that is a problem #2 - swappinness not being honoured. Guys, any ideas on the reason for that?

Andrew, dirty_ratio and dirty_background_ratio (as low as 5% each) did not significantly affect the amount of swapped out data, only a small effect on _how soon_ anonymous memory was swapped out.

And finally, Ray, the difference you see between 2.6.6 and 2.6.7 can be explained, as noted by others in this thread, to vmscan.c changes (page replacement/scanning policy changes were made).

Will continue with more tests tomorrow.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Hum I just tested it on my 512MB box and swappiness is working fine on 2.6.8.. (it has expected effects), unlike Ray's tests.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I looked at the get_dirty_limits() code and for the test cases I was running,
we have mapped > 90% of memory. So what will happen is that dirty_ratio will be thresholded at 5%, and background_ratio will be 1%. Changing values in /proc won't modify this at all (well, you could force background_ratio to 0%.)

It seems to me that the 5% number in there is more or less arbitrary. If we are on a big memory Altix (4 TB), 5% of memory would be 200 GB. That is a lot of page cache.

It seems get_dirty_limits() would be a lot simpler (and automatically scale as memory is mapped) if the limits were interpreted as being in terms of the amount of unmapped memory. A patch that implements this idea is attached. (Andrew -- if it comes to that I can submit this patch inline -- this is just for talking at the moment).

I'll run a few of the tests with this modified kernel and see if they are any different.

Yep. I can probably live with those minor differences though. I would be happier if the system didn't swap anything at all for low values of swappiness, though.

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

Index: linux-2.6.9-rc1-mm3-kdb/mm/page-writeback.c
===================================================================
@@ -135,32 +135,19 @@
 static void
 get_dirty_limits(struct writeback_state *wbs, long *pbackground, long *pdirty)
 {
-	int background_ratio;		/* Percentages */
-	int dirty_ratio;
-	int unmapped_ratio;
+	int unmapped;
 	long background;
 	long dirty;
 	struct task_struct *tsk;
 
 	get_writeback_state(wbs);
 
-	unmapped_ratio = 100 - (wbs->nr_mapped * 100) / total_pages;
 
-	dirty_ratio = vm_dirty_ratio;
-	if (dirty_ratio > unmapped_ratio / 2)
-		dirty_ratio = unmapped_ratio / 2;
+	unmapped = total_pages - wbs->nr_mapped;
 
-	if (dirty_ratio < 5)
-		dirty_ratio = 5;
+	background = (dirty_background_ratio * unmapped) / 100;
+	dirty      = (vm_dirty_ratio * unmapped) / 100;
 
-	/*
-	 * Keep the ratio between dirty_ratio and background_ratio roughly
-	 * what the sysctls are after dirty_ratio has been scaled (above).
-	 */
-	background_ratio = dirty_background_ratio * dirty_ratio/vm_dirty_ratio;
-
-	background = (background_ratio * total_pages) / 100;
-	dirty = (dirty_ratio * total_pages) / 100;
 	tsk = current;
 	if (tsk->flags & PF_LESS_THROTTLE || rt_task(tsk)) {
 		background += background / 4;

On such huge memory machines I guess you have no choice but scale down the dirty limits for them to be "equivalent" with reference to IO device speed.

And as Martin says it depends on the workload also.

Huh, that changes the meaning of the dirty limits. Dont think its suitable for mainline.

Now that must work - if its not we have a problem.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

The change is, in fact, not much different from what is already actually there. The code in get_dirty_limits() adjusts the value of the user supplied parameters in /proc/sys/vm depending on how much mapped memory there is. If you undo the convoluted arithmetic that is in there, one finds that if you are using the default dirty_ratio of 40%, then if the unmapped_ratio is between 80% and 10%, then

   dirty_ratio = unmapped_ratio / 2;

and, a little bit of algebra later:

dirty = (total_pages - wbs->nr_mapped)/2

and

background = dirty_background_ratio/vm_background_ratio * (total_pages
- wbs->nr_mapped)

That is, for a wide range of memory usage, you are really running with an
dirty ratio of 50% stated in terms of the number of unmapped pages, and there is no direct way to override this.

Of course, at the edges, the code changes these calculations. It just seems to me that rather than continue the convoluted calculation that is in get_dirty_limits(), we just make the outcome more explicit and tell the user
what is really going on.

We'd still have to figure out how to encourage a minimum page cache size of some kind, which is what I understand the 5% min value for dirty_ratio is in there for.

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

OK I see, yes.

For the user "dirty_ratio" and "dirty_background_ratio" means "percentage of total memory" (thats how it has been traditionally in Linux). And right now,
as you noted, we dont do that way.

There's probably a good reason for the "no more than half of unmapped memory". Andrew ?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Marcelo,

For what it is worth, here are the benchmark results for the kernel with the patch I discussed before, along with the previous 2.6.9-rc1-mm3 results:

Kernel Version 2.6.9-rc1-mm3:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   274.80 MB/s  10511 MB (  5644, 14492)  13293 MB (  8596, 17156)
  20   267.02 MB/s  12624 MB (  5578, 16287)  15298 MB (  8468, 18889)
  40   267.66 MB/s  13541 MB (  6619, 17461)  16199 MB (  9393, 20044)
  60   233.73 MB/s  18094 MB ( 16550, 19676)  20629 MB ( 19103, 22192)
  80   213.64 MB/s  20950 MB ( 15844, 22977)  23450 MB ( 18496, 25440)
 100   164.58 MB/s  26004 MB ( 26004, 26004)  28410 MB ( 28327, 28455)

Kernel Version 2.6.9-rc1-mm3-kdb-nrmap:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   286.93 MB/s   7288 MB (  4847, 14536)  10122 MB (  7771, 17138)
  20   252.43 MB/s  13305 MB (  3950, 18337)  15938 MB (  6866, 20876)
  40   268.52 MB/s  11538 MB (  5333, 17298)  14238 MB (  8247, 19836)
  60   242.72 MB/s  16367 MB (  8652, 21217)  18909 MB ( 11514, 23561)
  80   212.94 MB/s  19424 MB (  5632, 24047)  21937 MB (  8567, 26469)
 100   161.66 MB/s  26006 MB ( 26004, 26007)  28445 MB ( 28407, 28471)

Except for the swappiness = 20 case, things are a smallish bit better for
the modified kernel than 2.6.9-rc1-mm3. Clearly we haven't found the root of this problem yet.

Have you still been unable to duplicate this problem on a small i386 platform?

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

Indeed.

Yes right, I have been unable to duplicate the problem on small i386 box. What about your tests?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I haven't had time to try on i386 yet. I guess I will have to. Thanks for trying, anyway.

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

[...]

Please log periodic snapshots of /proc/vmstat during runs on kernel versions before and after major behavioral shifts.

-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

wli,

Attached is the output you requested for two kernel versions: 2.6.8.1-mm4 and 2.6.9-rc1-mm3 + the nrmap_patch (that patch didn't make much difference so this should be good enough for comparison purposes, and it was the kernel I had built.)

Because there are so many parameters in /proc/vmstat, I had to split the
output up (more or less arbitarily) into three files to get something you could actually look at with an editor. Even then it requires 100 columns or so.

Data in the files was observed every 10 s for the duration of the runs. The first line of each file is a summary line, the next lines are incremental, except for the nr_* stats, which are assumed to be absolute numbers. Corresponding lines of each of the three output files per run were printed. at the same time.

Data file names in the attached archive are of the form:

vmstat.$s.$t.$f.$v

where s=swappiness, here 0, 60, or 100.

      t=trial       here always 5
      f=file        1, 2, or 3 for the three output files per run
      v=version     kernel version

There is a script, compare.csh in the archive that can be run to edit corresponding files for each version. It cycles through the list examining corresponding files foreach swapiness level and foreach output file.

The benchmark output for kernel version v is in reduce.out.$v, for reference.

I can post the perl script that was used to create this output, if there is interest.

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

This will do fine. I'll examine these for anomalous maintenance of statistics and/or operational variables used to drive page replacement.

Thanks.

-- wli
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Nick,

As reported to you elsewhere (and duplicated here to get on this thread), application of the patch you sent (attached) dramatically changes the swappiness behavior of the 2.6.9-rc1 (and presumably the rc2) kernel.

Here are the updated results:

Previously:

Kernel Version 2.6.9-rc1-mm3:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   274.80 MB/s  10511 MB (  5644, 14492)  13293 MB (  8596, 17156)
  20   267.02 MB/s  12624 MB (  5578, 16287)  15298 MB (  8468, 18889)
  40   267.66 MB/s  13541 MB (  6619, 17461)  16199 MB (  9393, 20044)
  60   233.73 MB/s  18094 MB ( 16550, 19676)  20629 MB ( 19103, 22192)
  80   213.64 MB/s  20950 MB ( 15844, 22977)  23450 MB ( 18496, 25440)
 100   164.58 MB/s  26004 MB ( 26004, 26004)  28410 MB ( 28327, 28455)

With Nick Piggin et al fix:

Kernel Version: linux-2.6.9-rc1-mm3-kswapdfix

Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   279.97 MB/s     89 MB (    12,   265)   3062 MB (  2947,  3267)
  20   283.55 MB/s    161 MB (    15,   372)   3190 MB (  3011,  3427)
  40   282.32 MB/s    204 MB (     6,   407)   3187 MB (  2995,  3331)
  60   279.42 MB/s     72 MB (    15,   171)   3091 MB (  3027,  3155)
  80   283.34 MB/s    920 MB (   144,  3028)   3904 MB (  3106,  5957)
 100   160.55 MB/s  26008 MB ( 26007, 26008)  28473 MB ( 28455, 28487)

(The drop at swappiness of 60 may just be randomness, not sure it is significant, but these results are all based on 5 trials.)

At any rate, this patch appears to fix the problems I was seeing before. (See

	http://marc.theaimsgroup.com/?l=linux-kernel&m=109449778320333&w=2

for further details of the benchmark and the test environment).

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

---

 linux-2.6-npiggin/mm/vmscan.c |   14 +++++++++++++-
 1 files changed, 13 insertions(+), 1 deletion(-)

diff -puN mm/vmscan.c~vm-no-wild-kswapd mm/vmscan.c
@@ -993,10 +993,13 @@ static int balance_pgdat(pg_data_t *pgda
 	int to_free = nr_pages;
 	int priority;
 	int i;
-	int total_scanned = 0, total_reclaimed = 0;
+	int total_scanned, total_reclaimed;
 	struct reclaim_state *reclaim_state = current->reclaim_state;
 	struct scan_control sc;
 
+loop_again:
+	total_scanned = 0;
+	total_reclaimed = 0;
 	sc.gfp_mask = GFP_KERNEL;
 	sc.may_writepage = 0;
 	sc.nr_mapped = read_page_state(nr_mapped);

@@ -1095,6 +1098,15 @@ scan:
 		 */
 		if (total_scanned && priority < DEF_PRIORITY - 2)
 			blk_congestion_wait(WRITE, HZ/10);
+
+		/*
+		 * We do this so kswapd doesn't build up large priorities for
+		 * example when it is freeing in parallel with allocators. It
+		 * matches the direct reclaim path behaviour in terms of impact
+		 * on zone->*_priority.
+		 */
+		if (total_reclaimed >= 32)
+			goto loop_again;
 	}
 out:
 	for (i = 0; i < pgdat->nr_zones; i++) {

_

Thanks Ray. From looking over your old results, it appears that -kswapdfix probably has the nicest swappiness ramp, which is probably to be expected, as the problem that is being fixed did exist in all other kernels you tested, but the later ones just had other aggrivating changes.

The swappiness=60 weirdness might just be some obscure interaction with the workload. If that is the case, it is probably not too important, however it could be due to a possible oversight in my patch....

I'm not in front of the code right now, so I can't give you a new patch to try yet... if you're up for modifying it yourself though: we possibly should be updating "zone->prev_priority" after each 32 (SWAP_CLUSTER_MAX) pages freed. So change the following:

==> if (total_reclaimed >= 32)
==> break;
}
out:

   for (i = 0; i < pgdat->nr_zones; i++) {
      ... /* this updates zone->prev_priority */

}
=> if (!all_zones_ok)
=> goto loop_again;
return total_reclaimed;
}

so it looks something like that.

If you could run your tests again on that, it would be great.

Thanks
Nick
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Here is a patch on top of the last one - if you can give it a test some time, that would be great.

Thanks
Nick

---

 linux-2.6-npiggin/mm/vmscan.c |    9 +++++++--
 1 files changed, 7 insertions(+), 2 deletions(-)

diff -puN mm/vmscan.c~vm-no-wild-kswapd2 mm/vmscan.c
@@ -991,6 +991,7 @@ out:
 static int balance_pgdat(pg_data_t *pgdat, int nr_pages)
 {
 	int to_free = nr_pages;
+	int all_zones_ok;
 	int priority;
 	int i;
 	int total_scanned, total_reclaimed;

@@ -1013,10 +1014,11 @@ loop_again:
 	}
 
 	for (priority = DEF_PRIORITY; priority >= 0; priority--) {
-		int all_zones_ok = 1;
 		int end_zone = 0;	/* Inclusive.  0 = ZONE_DMA */
 		unsigned long lru_pages = 0;
 
+		all_zones_ok = 1;
+
 		if (nr_pages == 0) {
 			/*
 			 * Scan in the highmem->dma direction for the highest

@@ -1106,7 +1108,7 @@ scan:
 		 * on zone->*_priority.
 		 */
 		if (total_reclaimed >= SWAP_CLUSTER_MAX)
-			goto loop_again;
+			break;
 	}
 out:
 	for (i = 0; i < pgdat->nr_zones; i++) {

@@ -1114,6 +1116,9 @@ out:
 
 		zone->prev_priority = zone->temp_priority;
 	}
+	if (!all_zones_ok)
+		goto loop_again;
+
 	return total_reclaimed;
 }
 

_

Nick,

I'll put it on my list of TODO's. FYI, a full test like the ones I have been running, at 6 swappiness levels, takes around 5 hours of machine time, so it sometimes we have to wait for a slot that big to open up. :-)

Ray

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

Hi Nick,

Here are the results for the kswapd2 patch, along with the previous version and the original 2.6.9-rc1-mm3 test. It pretty much looks like a wash between the kswapd and kswapd2 patches.

Kernel Version 2.6.9-rc1-mm3:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   274.80 MB/s  10511 MB (  5644, 14492)  13293 MB (  8596, 17156)
  20   267.02 MB/s  12624 MB (  5578, 16287)  15298 MB (  8468, 18889)
  40   267.66 MB/s  13541 MB (  6619, 17461)  16199 MB (  9393, 20044)
  60   233.73 MB/s  18094 MB ( 16550, 19676)  20629 MB ( 19103, 22192)
  80   213.64 MB/s  20950 MB ( 15844, 22977)  23450 MB ( 18496, 25440)
 100   164.58 MB/s  26004 MB ( 26004, 26004)  28410 MB ( 28327, 28455)

Kernel Version 2.6.9-rc1-mm3-kdb-kswapdfix:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   279.97 MB/s     89 MB (    12,   265)   3062 MB (  2947,  3267)
  20   283.55 MB/s    161 MB (    15,   372)   3190 MB (  3011,  3427)
  40   282.32 MB/s    204 MB (     6,   407)   3187 MB (  2995,  3331)
  60   279.42 MB/s     72 MB (    15,   171)   3091 MB (  3027,  3155)
  80   283.34 MB/s    920 MB (   144,  3028)   3904 MB (  3106,  5957)
 100   160.55 MB/s  26008 MB ( 26007, 26008)  28473 MB ( 28455, 28487)

Kernel Version 2.6.9-rc1-mm3-kdb-kswapdfix2:
Total I/O Avg Swap min max pg cache min max
----------- --------- ------- ------ --------- ------- -------

   0   282.71 MB/s    247 MB (     8,   691)   3248 MB (  2995,  3636)
  20   281.96 MB/s    127 MB (     3,   300)   3097 MB (  2931,  3283)
  40   282.24 MB/s    252 MB (    16,   417)   3270 MB (  3059,  3475)
  60   281.27 MB/s    404 MB (   106,  1070)   3414 MB (  3139,  4068)
  80   281.30 MB/s    494 MB (   204,   662)   3485 MB (  3187,  3635)
 100   160.35 MB/s  26003 MB ( 26003, 26003)  28477 MB ( 28407, 28519)

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

For HPC, maybe. For a fileserver, it might be far too little. That's the trouble ... it's all dependant on the workload. Personally, I'd prefer to get rid of manual tweakables (which are a pain in the ass in the field anyway), and try to have the kernel react to what the customer is doing. I guess we can leave them there for overrides, but a self-tunable default would be most desirable.

For instance, would be nice if we started doing writeback to the spindles that weren't busy much earlier than if the disks were thrashing.

M.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Agreed. Many of these things should be self-tunable pretty easily, too...

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

El Wed, 08 Sep 2004 14:04:31 -0400 (EDT) Rik van Riel <[email protected]> escribio:

I know this has been discussed before, but could a userspace daemon which autotunes the tweakables do a better job wrt. to adapting the kernel behaviour depending on the workload? Just like these days we have irqbalance instead of a in-kernel "irq balancer". It's a alternative worth of look at?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I really don't see any point in pushing the self-tuning of the kernel out into userspace. What are you hoping to achieve?

M.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

El Wed, 08 Sep 2004 14:10:32 -0700 "Martin J. Bligh" <[email protected]> escribio:

Well your own words explain it, I think. "it's all dependant on the workload", which means that only the user knows what he is going to do with the machine and that the kernel doesn't knows that, so the algoritms built in the kernel may be "not perfect" in their auto-tuning job. The point would be to be able to take decisions the kernel can't take because userspace would know better how the system should behave, say stupids things like "I want to have this set of tunables which make compile jobs 0.01% faster at 12:00 because at that time a cron job autocompiles cvs snapshots of some project, and at 6:00 those jobs have already finished so at that time I want a set of tunables optimized for my everyday desktop work which make everthing 0.01% slower but the system feels a 5% more reponsive". (well, for that a shell script is enought) Kernel however could try to adapt itself to those changes, and do it well...I don't really know. This came to my mind when I was thinking about irqbalance case, which was somewhat similar, I also remember a discussion about a "ktuned" in the mailing lists...I guess it's a matter of coding it and get some numbers :-/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Oh, I see what you mean. I think we're much better off sticking the mechanism for autotuning stuff in the kernel - if we want to feed in policy from userspace (which 99.9% of people will never do), it can be through such things as /proc/sys/vm/swapiness ... that could just fix them statically if people insisted on such things.

Having the kernel do something sensible by default is what we aim for ... overrides still are possible if the sysadmin really thinks they're smarter ;-)

M.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Agreed. Autotuning like this appears to work best by having a self adjusting algorithm, often negative feedback loops so things get balanced out automagically.

Works way better than anything looking at indirect data and then tweaking some magic knobs...

"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Don't need to be smarter, just to know more about program behaviour and desired system response than the o/s. Think of it as providing more information to the o/s and letting the o/s autotune with the additional information if that makes you feel better ;-)

The only tune I really want, and I patched it in 2.4.18 or so, is the ability to reserve some memory which will never be used for anything but program pages. Or some way to stop a program making a single pass through a large file (80GB) from pushing programs out of memory. I'm watching that now as I read the list, vmstat says si/so are about 800 each, on a system which runs at zero otherwise.

I'm going to look at capabilities when I get that mythical free time, and see if I can let a few programs lock their ass in memory, having 60-80% of RAM dedicated to once-read pages is silly.

Sorry for the rant, but this is the single most common problem I see with performance.

   -bill davidsen ([email protected])
"The secret to procrastination is to put things off until the
 last possible moment - but no longer"  -me

What if there is more than one right answer to "self-tune" policy. Also what if you want an application to tweak the tuning in ways that are different to general policy ?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

It's still overridable from userspace, I'd think. But having a sensible default in the kernel makes a crapload of sense to me. We have better faster access to data from there - if there are really things that aren't just parameters to the tuning algorithm it'd have to repeatedly poke values into hard overrides. Do-able, but not what we want by default, I'd think.

M.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

I agree that tunables are a pain in the butt, but a quick fix would to be at least to add that 5% to the set of stuff settable in /proc/sys/vm. Most workloads/systems won't need to change it. Very large Altix systems could change it if needed.

I don't think that is at the root of the swappiness problems with 2.6.9-rc1-mm3, though.

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

True enough, but the general trend is that increasing swapping decreases data rate. This is even more true for the real applications that we are modelling with this simple benchmark. In thosec cases, the user has a lot of mapped data that they then write out using buffered I/O. If the mapped data gets swapped out, then it may have to be swapped back in to be written out to the file system. It would be faster to keep the mapped data from being swapped out at all provided that there is enough page cache space to keep the devices running at full speed.

(And yes, we've suggested that they mmap() the data files -- but sometimes this is an ISV's code that it causing the problem and we can't necessarily get them to update their codes to use the API's we want.)

Best Regards,
Ray
-----------------------------------------------
                  Ray Bryant
512-453-9679 (work)         512-507-7807 (cell)
[email protected]             [email protected]
The box said: "Requires Windows 98 or better",
           so I installed Linux.
-----------------------------------------------

one to pass judgement.

without a disk cache" perspective, definitely not true.

I've found a situation where the vanilla kernel has a behaviour that makes no sense:

http://marc.theaimsgroup.com/?l=linux-kernel&m=109237941331221&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=109237959719868&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=109238126314192&w=2

A patch by Con Kolivas fixed it:

http://marc.theaimsgroup.com/?l=linux-kernel&m=109410526607990&w=2

I cannot offer more details, i have no time for experiments, i just need a system that works. The vanilla kernel does not.

Florin Andrei

http://florin.myip.org/

Have you tried to decrease the value of /proc/sys/vm/swappiness to say 30 and see what you get?

Andrew's point is that we should identify the problem - Con's patch rewrites swapping policy.

I already answered this. That hard swappiness patch does not really rewrite swapping policy. It identifies exactly what has changed because it does not count "distress in the swap tendency". Therefore if the swappiness value is the same, the mapped ratio is the same (in the workload) yet the vm is swappinig more, it is getting into more "distress". The mapped ratio is the same but the "distress" is for some reason much higher in later kernels, meaning the priority of our scanning is getting more and more intense. This should help direct your searches.

These are the relevant lines of code _from mainline_:

distress = 100 >> zone->prev_priority
mapped_ratio = (sc->nr_mapped * 100) / total_memory; swap_tendency = mapped_ratio / 2 + distress + vm_swappiness if (swap_tendency >= 100)
- reclaim_mapped = 1;

That hard swappiness patch effectively made "distress == 0" always.

Con

OK.

So isnt it true that decreasing vm_swappiness should compensate distress and have the same effect of your patch?

To be fair I'm just arguing, haven't really looked at the code.

Nope. We swap large amounts with the wrong workload at swappiness==0 where we wouldn't before at swappiness==60. ie there is no workaround possible without changing the code in some way.

Thats cool ;)

Cheers,
Con

Con!

Spent some time reading your patch...

Well if "distress" is getting higher (with similar workload/pressure) thats because VM is having a harder time freeing pages (priority increases, distress increases).

You say "distress is getting higher in later kernels". Can you expand more on that? How did you find this out, and can you be more especific wrt "later kernels".

"we wouldn't before" refering to older kernel versions?

I see you add a "z->nr_unmapped" watermark a bit above "z->pages_high", and use that to set "pgdat->mapped_nrpages" to what needs to be freed so z->free_pages reaches "z->nr_unmapped".

And then you use that per-pgdat "mapped_nrpages" count to avoid:

- moving mapped pages to inactive list (wasting the swappiness algorithm)
- swapping out pages at shrink_list

Those two only happen when pgdat->mapped_nrpages is zero, which becomes true when we go below pages_low.

To resume, deactivation/swapout of mapped pages only happens when we go any zone pages_low.

Correct?

Now with v2.6 stock kernel, kswapd will deactivate (using vm_swappiness algorithm) and swapout pages between the low and high zone watermarks.

That avoids swapping out as hard as possible until we go below pages_low.

IMHO this might be OK for common desktop workloads where people complain about swap, but might be harmful for other workloads where swapping out on advance unused anonymous process memory is a _gain_.

I dont understand this check on balance_pgdat (kswapd worker function):

+ /*

+        * kswapd does a light balance_pgdat() when there is less than 1/3
+        * ram free provided there is less than vm_mapped % of that ram
+        * mapped.
+        */

+ if (maplimit && sc.nr_mapped * 100 / total_memory > vm_mapped)
+ return 0;
+

So "if not any zone is under pages_low, and more than vm_mapped % of ram is mapped, bail out."

I dont get what you're trying to achieve with this.

I still think swapout behaviour can be correctly tuned with vm_swappiness, and agree with Andrew on that we should not change anything in the algorithm if this can be tuned.

Andrew, maybe decrease vm_swappiness to 50 on the next -mm for a test?

Great!

When I say earlier kernels I mean prior to 2.6.8.

I'm still referring to the "hard_swappiness patch" that Florin was using to fix his problem which diagnosed that distress increased. I'm sorry if my newer patch confuses the issue. hard_swappiness effectively changed this:

-distress = 100 >> zone->prev_priority -mapped_ratio = (sc->nr_mapped * 100) / total_memory; -swap_tendency = mapped_ratio / 2 + distress + vm_swappiness -if (swap_tendency >= 100)
- reclaim_mapped = 1;

into this:

+mapped_ratio = (sc->nr_mapped * 100) / total_memory; +swap_tendency = mapped_ratio / 2 + vm_swappiness +if (swap_tendency >= 100)
+ reclaim_mapped = 1;

This made swap_tendency dependant _only_ on the mapped_ratio. Now if you load up the same desktop and applications your mapped_ratio will be virtually identical regardless of the kernel. If you then copy a large file or convert a large video file etc, then the mapped ratio will be unchanged. Therefore if the swapping increased with this workload in 2.6.8 and later kernels but did _not_ increase with hard_swappiness it must be the "distress" value which is entirely dependant on zone->prev_priority. Does that make my conclusion clearer?

Below here you're referring to my mapped_watermark patch so I'll address that separately to avoid confusion.

Yes apart from one big caveat. scanning is expensive, so it only scans at lowest priority (DEF_PRIORITY). If it fails to release enough memory it simply returns quietly. This means that if vm pressure is hard enough and occurs frequently/fast enough it will still drop down below pages_high even if the watermarks have not been re-achieved. Then the normal algorithm will take over.

As I said, it only does it lightly, and it's tunable.

This will only be hit if "maplimit" is true. This means we have entered balance_pgdat only due to the unmapped watermark (zone->pages_min * 4). Here is where the real "tunable" comes into play. If greater than vm_mapped % of ram is mapped (ie application) pages, it will not do anything at this watermark. By default it is set to 66%. Setting it to 0 inactivates this patch entirely and makes the vm behave much like setting swappiness to 100 in mainline.

I agree it can be, but something in the logic has definitely changed, and a different value is not giving users like Florin the desired result any more.

Cheers,
Con

I cannot repeat the experiment (due to lack of time) but i remember having severe issues with low /proc/*/swappiness values on vanilla kernels. I don't recall the nature of the problems, though, because it's been a while. Sorry. But that was what prompted me to search further. The search ended when i found and tested Con's patch.

The current swapping policy of the vanilla 2.6 kernels is broken. Badly.

At least my system works properly now. <shrug>

Florin Andrei

http://florin.myip.org/