2.5.40-mm2 with contest

Here are the latest results including 2.5.40-mm2 with contest v0.50
(http://contest.kolivas.net)

noload:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio 
2.4.19 [3]              67.7    98      0       0       1.01 
2.5.38 [3]              72.0    93      0       0       1.07 
2.5.39 [2]              72.2    93      0       0       1.07 
2.5.40 [1]              72.5    93      0       0       1.08 
2.5.40-mm1 [1]          72.9    93      0       0       1.09 
2.5.40-mm2 [1]          72.2    93      0       0       1.07 

process_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio 
2.4.19 [3]              106.5   59      112     43      1.59 
2.5.38 [3]              89.5    74      34      28      1.33 
2.5.39 [2]              91.2    73      36      28      1.36 
2.5.40 [2]              82.8    80      25      23      1.23 
2.5.40-mm1 [2]          86.9    77      30      25      1.29 
2.5.40-mm2 [1]          98.0    69      45      33      1.46 

io_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio 
2.4.19 [3]              492.6   14      38      10      7.33 
2.5.38 [1]              4000.0  1       500     1       59.55 
2.5.39 [2]              423.9   18      30      11      6.31 
2.5.40 [1]              315.7   25      22      10      4.70 
2.5.40-mm1 [1]          326.2   24      23      11      4.86 
2.5.40-mm2 [2]          208.0   38      12      10      3.10 

mem_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio 
2.4.19 [3]              100.0   72      33      3       1.49 
2.5.38 [3]              107.3   70      34      3       1.60 
2.5.39 [2]              103.1   72      31      3       1.53 
2.5.40 [2]              102.5   72      31      3       1.53 
2.5.40-mm1 [2]          107.7   68      29      2       1.60 
2.5.40-mm2 [2]          165.1   44      38      2       2.46 

Well something happened here. The tuning under IO load has relaxed the pressure even more to allow kernel compilation to proceed. Mem load seems to have changed dramatically though with a disproportionately long increase in kernel compilation time given only a modest increase in the amount of work done by mem load. Process_load seems proportionately longer than mm1 with an appropriate rise in load work done.

Below are also the experimental results with the newer loads:

tarc_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio 
2.4.19 [2]              106.5   70      1       8       1.59 
2.5.38 [1]              97.2    79      1       6       1.45 
2.5.39 [1]              91.8    83      1       6       1.37 
2.5.40 [1]              96.9    80      1       6       1.44 
2.5.40-mm1 [1]          94.4    81      1       6       1.41 
2.5.40-mm2 [1]          91.9    82      1       6       1.37 

tarx_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio 
2.4.19 [1]              132.4   55      2       9       1.97 
2.5.38 [1]              120.5   63      2       8       1.79 
2.5.39 [1]              108.3   69      1       6       1.61 
2.5.40 [1]              110.7   68      1       6       1.65 
2.5.40-mm1 [1]          191.5   39      3       7       2.85 
2.5.40-mm2 [1]          188.1   39      3       7       2.80 

read_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio 
2.4.19 [2]              134.1   54      14      5       2.00 
2.5.38 [2]              100.5   76      9       5       1.50 
2.5.39 [2]              101.3   74      14      6       1.51 
2.5.40 [1]              101.5   73      13      5       1.51 
2.5.40-mm1 [1]          104.5   74      9       5       1.56 
2.5.40-mm2 [1]          102.7   75      7       4       1.53 

lslr_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio 
2.4.19 [1]              89.8    77      1       20      1.34 
2.5.38 [1]              99.1    71      1       20      1.48 
2.5.39 [1]              101.3   70      2       24      1.51 
2.5.40 [1]              97.0    72      1       21      1.44 
2.5.40-mm1 [1]          96.6    73      1       22      1.44 
2.5.40-mm2 [1]          94.3    75      1       21      1.40 

These do not appear significantly different.

Con

-mm2 has the "don't swap so much" knob. By default it is set at 50%. The VM _wants_ to reclaim lots of memory from mem_load so that gcc has some cache to chew on. But you the operator have said "I know better and I don't want you to do that".

Because it is prevented from building enough cache, gcc is issuing a ton of reads, which are hampering the swapstorm which is happening at the other end of the disk. It's a lose-lose.

There's not much that can be done about that really (apart from some heavy-handed load control) - if you want to optimise for throughput above all else,

echo 100 > /proc/sys/vm/swappiness

(I suspect our swap performance right now is fairly poor, in terms of block allocation, readaround, etc. Nobody has looked at that in some time afaik. But tuning in there is unlikely to make a huge difference).

Andrew

Quoting Andrew Morton <[email protected]>:

I like the idea of the swappiness switch. It seems to me that this shouldn't be a magic number though. I've experimented with making it auto-regulating and found that with a positive feedback arm being ten times greater than the negative feedback arm it gives good results. Here is a patch describing that:

@@ -44,7 +44,7 @@
 /*
  * From 0 .. 100.  Higher means more swappy.
  */
-int vm_swappiness = 50;
+int vm_swappiness = 0;
 
 #ifdef ARCH_HAS_PREFETCH
 #define prefetch_prev_lru_page(_page, _base, _field)                   \

@@ -535,7 +535,13 @@
         * A 100% value of vm_swappiness will override this algorithm almost
         * altogether.
         */
-       swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
+       swap_tendency = mapped_ratio / 2 + distress ;
+       if (swap_tendency > 50){
+               if (vm_swappiness <= 990) vm_swappiness+=10;
+               }
+               else
+               if (vm_swappiness > 0) vm_swappiness--;
+       swap_tendency += (vm_swappiness / 10);
        if (akpm_print) printk(" st:%ld\n", swap_tendency);
 
        if (akpm_print) printk("\n");

----------------------

And here are the results I have obtained with that (mm2v is mm2 with variable vm_swappiness):

noload:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.5.40-mm1 [1]          72.9    93      0       0       1.09
2.5.40-mm2 [1]          72.2    93      0       0       1.07
2.5.40-mm2v [2]         73.1    92      0       0       1.09

process_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.5.40-mm1 [2]          86.9    77      30      25      1.29
2.5.40-mm2 [1]          98.0    69      45      33      1.46
2.5.40-mm2v [2]         85.6    77      29      25      1.27

tarc_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.5.40-mm1 [1]          94.4    81      1       6       1.41
2.5.40-mm2 [1]          91.9    82      1       6       1.37
2.5.40-mm2v [2]         91.2    82      1       6       1.36

tarx_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.5.40-mm1 [1]          191.5   39      3       7       2.85
2.5.40-mm2 [1]          188.1   39      3       7       2.80
2.5.40-mm2v [2]         174.6   46      2       7       2.60

io_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.5.40-mm1 [1]          326.2   24      23      11      4.86
2.5.40-mm2 [2]          208.0   38      12      10      3.10
2.5.40-mm2v [3]         254.0   31      15      10      3.78

read_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.5.40-mm1 [1]          104.5   74      9       5       1.56
2.5.40-mm2 [1]          102.7   75      7       4       1.53
2.5.40-mm2v [2]         105.0   72      7       4       1.56

lslr_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.5.40-mm1 [1]          96.6    73      1       22      1.44
2.5.40-mm2 [1]          94.3    75      1       21      1.40
2.5.40-mm2v [2]         97.9    71      1       20      1.46

mem_load:

Kernel [runs]           Time    CPU%    Loads   LCPU%   Ratio
2.5.40-mm1 [2]          107.7   68      29      2       1.60
2.5.40-mm2 [2]          165.1   44      38      2       2.46
2.5.40-mm2v [3]         118.1   62      30      2       1.76

Most of the time it seems to hover around the 500 number during normal use (equivalent to 50 of the original vm_swappiness).

What do you think?
Con

Quoting Andrew Morton <[email protected]>:

Exactly, which made complete sense to me.

But do they really want that or do they think they want that without knowing the consequences of such a setting?

Well I made it as simple as I possibly could. It seems to do what they want (not swappy) but not at the expense of making the machine never swapping when it really needs to - and the performance seems to be better all round in real usage. I guess the only thing is it isn't a fixed number... unless we set a maximum swappiness level or... but then it starts getting unnecessarily complicated with questionable benefits.

/me waits but is a little disappointed

Con

I have been able to tune bdflush in 2.4-aa kernels to be much more aggressive about moving data to disk under write pressure, and that has been a big plus in terms of both getting the write completed in less real time and fewer big pauses doing trivial things like uncovering a window. I see less swap used.

I'm going to try this patch, but building a kernel on my standard test machine is painfully slow, so it will come after 41-ac2. It appears to address the balance issue dynamically.

I actually like the idea of writer throttling, I just wonder how it will work at the corner cases like only one big writer (mkisofs) or the alternative, lots of little writers.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've been playing with the feedback loop a bit more and made it respond more relative to the pressure present, and not have the "magic number" of 10 times the gain on the positive arm. I'm getting good results with it. Check it out below. I think if you want to give the users a "knob" then limiting the max_vm_swappiness rather than the current vm_swappiness would work.

Worth trying it out I guess.

- --- linux-2.5.41/mm/vmscan.c 2002-10-11 09:11:20.000000000 +1000
+++ linux-2.5.41-new/mm/vmscan.c 2002-10-11 00:51:06.000000000 +1000 @@ -44,7 +44,8 @@
/*
* From 0 .. 100. Higher means more swappy.
*/
- -int vm_swappiness = 50;
+int vm_swappiness = 0;
+int vm_swap_feedback;
static long total_memory;

#ifdef ARCH_HAS_PREFETCH
@@ -587,7 +588,18 @@
* A 100% value of vm_swappiness will override this algorithm almost
* altogether.
*/
- - swap_tendency = mapped_ratio / 2 + distress + vm_swappiness;
+ swap_tendency = mapped_ratio / 2 + distress;
+

+        vm_swap_feedback = (swap_tendency - 50)/10;
+        vm_swappiness += vm_swap_feedback;
+        if (vm_swappiness < 0){
+               vm_swappiness = 0;
+       }
+       else
+       if (vm_swappiness > 100){

+ vm_swappiness = 100;
+ }
+ swap_tendency += vm_swappiness;

/*
* Well that all made sense. Now for some magic numbers. Use the

Con
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.7 (GNU/Linux)

iD8DBQE9pgqLF6dfvkL3i1gRAo87AKCT2QL/4yihdGoRQZFxFL4/Az9RNgCeJjhJ Xew01Xff9t31p9Bi2TODj44=
=OEcn
-----END PGP SIGNATURE-----

It is really desirable to improve write intense performance in 2.5. My response benchmark shows that 2.5.xx is seriously worse under heavy write load than 2.4. And in 2.4 it is desirable to do tuning of bdflush for write loads, to keep performance up in -aa kernels. Andrea was kind enough to provide me some general hints in this area.

Here's what I think is happening.

1 - the kernel is buffering too much data in the hope that it will possibly be reread. This is fine, but it results in swapping a lot of programs to make room, and finally a big cleanup to disk, which triggers...

2 - without the io scheduler having a bunch of writes has a very bad effect on read performance, including swap-in. While it's hard to be sure, I think I see a program getting a fault to page in a data page (while massive write load is present) and while blocked some of the code pages are released.

I think there's room for improving the performance, as the "swappiness" patch shows. I played with trying to block a process after it had a certain amount of data buffered for write, but it didn't do what I wanted. I think the total buffered data in the system needs to be considered as well.

I believe one of the people who actually works on this stuff regularly has mentioned this, I don't find the post quickly.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

2.5 and 2.5-mm are very different in this area. You did not specify.

This is why 2.5.41-mm2 has improved writer throttling, and it's why it adjusts the throttling threshold down when the amount of mapped memory is high.

Yes, that happens quite a lot.

It does. The throttling of write(2) callers is a critical part
of the VM. Large amounts of dirty data cause lots of problems.