Rik spreading bullshit about VM

I read here:

http://linux.html.it/articoli/rik_van_riel_ita1.htm

[..] La nuova VM ha migliori performance rispetto alla vecchia sui tipici sistemi desktop ... ma fa fiasco terribilmente su più sistemi di quanto non lo facesse la vecchia VM. Redhat, per esempio, non ha potuto inserire la nuova VM nella sua distribuzione perché cadrebbe a pezzi per i database server, [..]

This is total bullshit. If there's something where the -aa VM is good are the DBMS, that was designed for it basically, very lightweight, basically no VM overhead also under very heavy I/O.

If redhat doesn't use the -aa VM into their kernels that's either a political decision or they're not good enough at the VM. I can tell you there are an huge number of users very happy about the -aa VM mainly on DBMS, I know some redhat partner is also using the -aa VM for their internal DBMS work (incidentally I assume rh kernels aren't good enough for them?). I don't care much of interviews, real people will make their choice about the VM based on facts anyways, you may forbid someone to try or you may convince someone that doesn't need much VM anyways saying weird things like "collapse under high load or breaks on databases", but still it's annoying to read such a bullshit and have people sending emails to you about stuff that it isn't true.

So if you can proof any of your statements with numbers about regressions with DB with -aa VM compared to 2.4.9 or 2.4.9+ac/rh/whatever please let us know. I have an huge amount of doc to proof the exact opposite (the first few impressive emails I found are attached) and most important I don't have a single bugreport about the current 2.4.18pre2aa2 VM (except perhaps the bdflush wakeup that seems to be a little too late and that deals to lower numbers with slow write load etc.., fixable with bdflush tuning). Mainline VM kills too easily, this is fixed in -aa VM and -aa VM has a number of other issues resolved, but mainline 2.4 vm isn't that far either. In the last few days I was playing with pte-highmem, soon I will spend some time merging -aa VM into mainline with Marcelo if he likes to.

Andrea

PS. I know the interviewer and he's usually very accurate, so I don't think this could be a misunderstanding where you say one thing and they writer another one just to create troubles.

Hi,

on LKML there were many discussions about the performance of recent kernels. We did some test, too, and perhaps their outcome is interesting to someone:

we tested in long time runs the performance of distributor provide kernels based on 2.4.7 vs. the new stock 2.4.14 (pre 8).

1) Test description:
The test suite we use for our QA and stress tests consists of a so-called "standard SAP SD benchmark". This benchmark tries to emulate a typical user load on a SAP system. It consists of a benchmark driver which simulates multiple users logging into the system and starting transactions which consist of several dialog steps. The relevant performance quantity is therefore based on the throughput, i.e. dialog steps per second. More technically speaking the typical work load this benchmark generates is dominated by m[map, copy], string operations and integer operations. It has a huge memory footprint / working set (approx. 2 GB). It tests mainly the CPU performance and the VM behaviour. Network performance and disk IO (for the database) are much less important.

Test hardware:
4 way Dell, 4 GB physical RAM, SCSI/RAID subsystem, DB runs on FS.

Test setting:
we configured the SAP system for 4GB and measured the dialog steps per second for a situation where only 1 GB was activated at boot time. This led to heavy swapping load over several hours to days. The results were checked several times.

2) Main result:
Dialog steps per second of a typical test series:

	2.4.7			2.4.14
-----------------------------------------
	8.59			15.47
	4.40			14.76
	2.67			11.61
	2.30			14.63
	1.87			14.42
	1.65			15.30	
	1.26			15.02	
	1.68			14.53
	1.17 			15.23
	1.80			12.82
	1.10			14.59
	1.20			16.09
	1.26			14.38
	1.34			15.41

Two aspects seem to be reproduceable:
1) whereas 2.4.7 shows a significant degradation over time 2.4.14 remains stable.

2) the asymptotic behaviour of the 2.4.14 based kernel is faster
than 2.4.7 by a _factor_ of around _10_. Compared to
our previous experiences with 2.2 ad 2.4 based this is incredible... (OK, I don't want to become enthusiastic :-))

3) Details of typical runs:

2.4.7:
------

a)
vmstat 5:

procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in
cs us

       sy  id
       0 18  0 1718772   4692    380   9404  56  45    60    48   43    71
1   0  98
       0 15  1 1725360   5824    384   9384 5102 692  5109   713  410  1155
       2   1  96
       0 13  0 1723960   4824    372   9308 3316 467  3316   478  390   925
       2   2  96
       0 21  1 1719720   5576    368   9320 3973 1162  3978  1197  494 
  887
        3   2  95
       0 21  1 1727092   4732    368   9316 1539 1187  1539  1191  419 
  399
        1   1  98
       0 16  1 1726992   4752    368   9268 4510 951  4517   982  432   956
       4   3  94

b)
Top 10 of /proc/profile:
5785008 default_idle 90390.7500
10132 blk_get_queue 126.6500
6744 swap_out_pmd 22.1842
5839 refill_inactive_scan 19.2072
3451 __wake_up 17.9740
9004 __get_swap_page 13.7256
402 __free_pages 12.5625
1143 __remove_inode_page 10.2054
469 add_wait_queue_exclusive 9.7708
3398 bounce_end_io_read 9.2337

c)
meminfo during run:
total: used: free: shared: buffers: cached:

Mem:  1051394048 1047044096  4349952 443473920  1052672 294608896
Swap: 4294934528 1314680832 2980253696
MemTotal:      1026752 kB
MemFree:          4248 kB
MemShared:      433080 kB
Buffers:          1028 kB
Cached:          20900 kB
SwapCached:     266804 kB
Active:         694700 kB
Inact_dirty:     19296 kB
Inact_clean:      7816 kB
Inact_target:    12836 kB
HighTotal:      131072 kB
HighFree:         1460 kB
LowTotal:       895680 kB
LowFree:          2788 kB
SwapTotal:     4194272 kB
SwapFree:      2910404 kB
NrSwapPages:    727602 pages

###################################################################

2.4.14:
--------

a) vmstat 5:
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in
cs us
sy id
22 0 3 898668 6248 472 682160 273 108 298 187 67 391 55

        3  42
11  1  5 898696   2676    836 684692 2661 489  2813  1576  564  2570  91
        9   0
21  3  4 899164   5824    512 684520 2863 666  2947  2096  595  2517  77
        9  14
15  2  2 899628   5212    440 683816 3134 658  3260  2023  647  2631  89
        4   7
23  5  4 899796   5284    452 684860 2726 824  2842  2297  641  2552  81
        5  14
16  7  4 899700   6056    400 683916 3250 788  3346  2079  656  2527  89
        5   6

b) /proc/profile:
219002 default_idle 3421.9062

         604 fget                                       9.4375
         452 system_call                                8.0714
         354 sock_poll                                  7.3750
         488 wake_up_process                            5.0833
        2123 scsi_dispatch_cmd                          4.9144
         230 add_wait_queue_exclusive                   4.7917
         269 mark_page_accessed                         4.2031
          55 RDOUTDOOR                                  3.4375
         197 __generic_copy_to_user                     3.0781

Despite a comparable swapping rate the 2.4.14 runs much more smooth. One reason could be that the VM has stepped almost completely out of the way...

c)
meminfo during run:
total: used: free: shared: buffers: cached:

Mem:  1052712960 1046528000  6184960        0   319488 856850432
Swap: 4294934528 1313320960 2981613568
MemTotal:      1028040 kB
MemFree:          6040 kB
MemShared:           0 kB
Buffers:           312 kB
Cached:         714576 kB
SwapCached:     122192 kB
Active:         851084 kB
Inactive:       113592 kB
HighTotal:      131072 kB
HighFree:         2044 kB
LowTotal:       896968 kB
LowFree:          3996 kB
SwapTotal:     4194272 kB
SwapFree:      2911732 kB

###########################################################################

4) Our conclusion:

Although we still see some problems with the 2.4.14 based kernel it looks really promising for us. A _stable_ increase of a factor of 10 in memory critical situations is impressive. Especially since our customer tend to steer every system finally into this load region ;-)

Great work!

Best regards
Willi Nuesser
SAP LinuxLab

PS:
Please CC me, since I'm not on LKML

PPS:
In case you need any further info please mail me. We can put more detailed data on our ftp server.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hi,

We did this, you know ...

The results were comparable to the 2.4.7 results, at least for the kernels 2.4.9-[1,6]enterprise.

Best regards
Willi
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hi,

after my first posting to lkml where we compared distributor provided kernels vs. a plain 2.4.14-pre8 it was pointed out that between PAE and non-PAE kernels some performance differences might exist.

We checked this last night and here are the first results. Again,
a) the relevant quantity dialog steps per second is a measure for the throughput our application servers runs.
b) our application server and the corresponding database (SAP DB) run on 4 way Dell, 1 GB at boot time enabled.

Results:
---------

2.4.7

	2.4.14p8 PAE		2.4.14p4 non- PAE

-------------------------------------------------------------
1.80 13.42 15.47
1.10 13.28 14.76
1.20 14.08 14.63
1.26 13.17 15.30
1.35 13.41 14.51

This means that we did see a performance decrease of about 6 % compared to 2.4.14p8 nonPAE but still 2.4.14p8 is an order of magnitude faster than 2.4.7

Best regards
	Willi

-----------------------------------
Willi Nuesser
SAP Linuxlab

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Yes, I know the difference. The reason for this post was not blind
curiosity but the presumption that the great performance increase of 2.4.14p8 we reported was mainly due to the PAE enabling in the 2.4.7 and non-PAE-enabling 2.4.14p8 in our first test.

The tests above show that this is not the case. Even with PAE 2.4.14 is faster by an order of magnitude.

The other info we did find interesting is the actual amount of difference between PAE and nonPAE. The 6% we got fit very well in some estimates I found on lkml.

Best regards
Willi

-----------------------------------
Willi Nuesser
SAP Linuxlab

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

I'm not a programmer. I'm not a kernel hacker. I don't like _very_ valuable people wasting their time reading messages like mine. I give credit to all of you (Andrea, Rik, Linus, Alan and everyine else), and sincerely thank you for your hard and good work. I don't try to start or give fuel to a flame war.

But, Andrea, sometimes people like me _feel_ that bugs supposedly related to your VM implementation, and reported in this list, go without your attention. I'm completely sure you read all bug reports, and work "off-line" to solve them (-aa seems to be where they live). I am not of those who like talking a lot, but would be very nice (and it will sure decrease repeated bug reports and neverending threads about known problems) to sometimes answer some bug reports.

I try to keep in sync with the list, and it's very difficult to find if those "corner cases" your VM seem to still suffer under are solved in your three, known but not solved, solved and merged, solved and sent to Marcelo, etc.

In short, is more a feeling of believing that no one cares about the few problems that still exist than "objective unstability" of the VM.

Just my 0.02.

José Luis Domingo López
Linux Registered User #189436     Debian Linux Woody (P166 64 MB RAM)
 
jdomingo AT internautas DOT   org  => Spam at your own risk

[---SNIP---]

[---SNIP---]

[--SNIP--]

Andrea your VM also works perfectly for me, and I think you are doing a brilliant job on it. The only thing that I have noticed that worries me is the lack of MemShared. I wass about to study the code and try to find out why it is always 0 kB on my PC, but these stats also show the same results. Do you have any idea why this is so. I will still study the code (just for the fun of it), maybe I might learn something about te VM system.

-Bongani

http://www.tux.org/lkml/#s14-3

regards
john

"Now why did you have to go and mess up the child's head, so you can get another gold waterbed ?
 You fake-hair contact-wearing liposuction carnival exhibit, listen to my rhyme ..."

Thank you I missed that, its been a while since I read the FAQ

Thank
Bongani

I don't know why you're so upset. As far as I can tell, Rik has warmly praised your VM in the above message. Of course, since you didn't provide an English translation, I can't really be sure. Perhaps Rik was talking about Virtual Machines, and not Virtual Memory? Or perhaps Virgin Mary?

				Regards,

					Richard....
Permanent: [email protected]
Current:   [email protected]

English (Well, some crazy moon-language dialect of english) translation:-
http://linux.html.it/articoli/rik_van_riel_en1.htm

| Dave Jones.        http://www.codemonkey.org.uk
| SuSE Labs

It seems the IRC log of the journalist in question was missing some lines of what I said and he just glued together the remaining parts of the paragraph for that particular question ;)

The rest of the interview seems to have survived pretty ok, though.

cheers,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

It also looked to me like Rik was talking about the mainline VM, not any VM that exists in the many not-yet-integrated -aa patchsets.

The argument always feels to me like:

Joe Luser: "I downloaded a 2.4 kernel and boy does the VM bite!"

AA: "That's not true at all, but you need to apply these thirteen -aa
patches from my site."

Joe Luser: <runs away in fear>

ian

Don't worry about it. Everyone knows (or should know) that you and Andrea both have the best interest of the kernel vm in mind. It's normal for both of you to have strong feelings about something that you both take pride in. What would be great would be seeing both of you come together and really kick some serious vm ass and prove that linux can compete with any OS. Maybe you two can come to a compromise along the lines of "we'll try it your way for a while, and I'll back you, then we'll try it my way for a while and you back me. We'll go with the one that works best for Linux."

I'm sure there are hundreds if not thousands of members on this list that respect you both and believe that either one of you can provide a great vm.

I'm behind both of you.

my .02

Dan

err, the english version is the original one...

English translation:
http://linux.html.it/articoli/rik_van_riel_en1.htm

Andrea Arcangeli wrote:
<snip>

I don't know if this is a reference to the issue I reported under the "Writeout in recent kernels..." thread or not. If not, my apologies for clogging up this new "discussion".

As reported[0] in the above-mentioned thread, the bdflush tuning parameters you suggested made no difference in my test case other than slightly adjusting the temporal relationship between writeout and file transfer. -aa still performs slightly worse than both 2.4.17 stock and -rmap. 2.4.13-ac7 currently beats all competitors.

--Adam

[0] http://www.kroptech.com:8300/mailimport/showmsg.php?msg_id=49746&db_name=linux_kernel

Well, I haven't reported it yet, but booting my box with mem=4M gave as result: (running 2.4.18-pre2aa2):
diego# cat /var/log/messages | grep gfp

Jan 13 15:37:10 localhost kernel: __alloc_pages: 0-order allocation failed
(gfp=0xf0/0)
Jan 15 16:06:28 localhost kernel: __alloc_pages: 0-order allocation failed
(gfp=0xf0/0)
Jan 15 18:37:21 localhost kernel: __alloc_pages: 0-order allocation failed
(gfp=0xf0/0)
Jan 15 21:58:32 localhost kernel: __alloc_pages: 0-order allocation failed
(gfp=0xf0/0)
Jan 15 21:58:33 localhost kernel: __alloc_pages: 0-order allocation failed

(gfp=0xf0/0)
diego#

Each script of /etc/rc.d was killed by VM when it was started, there wasn't any "OOM", just
"VM killed..." or something similar.
As /etc/rc.d scripts were killed, I couldn't start swap.

The gfp=0x... numbers were not always the same, but I can't remember them because syslogd wasn't running.
I can repeat this if you want and I'll copy all messages.

..I remember running 2.2.14 in a 386 box with 4MB of RAM and 8 or 16 of swap. It was veeery slow, but even I could run apache :-)...

It seems Andrea's patch backs out a bugfix for this problem which marcelo and me put into the normal 2.4 kernel ...

regards,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

hmm, is this the bugfix you mean? that shouldn't really matter to me as far I can tell, I did it in an alternate way since the first place.

diff -urN 2.4.17pre8/mm/vmscan.c 2.4.17/mm/vmscan.c
@@ -338,7 +338,7 @@
 {
        struct list_head * entry;
        int max_scan = nr_inactive_pages / priority;
-       int max_mapped = nr_pages << (9 - priority);
+       int max_mapped = min((nr_pages << (10 - priority)), max_scan / 10);
 
        spin_lock(&pagemap_lru_lock);
        while (--max_scan >= 0 && (entry = inactive_list.prev) != &inactive_list) {

furthmore I hate those "10" hardwirded magic numbers that you keep adding. The less of them the better. At least I put those magics in sysctl.

see what my max_mapped is:

int orig_max_mapped = SWAP_CLUSTER_MAX * vm_mapped_ratio,

It is controlled by the vm_mapped_ratio and by the swap-cluster. So we unmap one swap cluster at every vm_mapped_ratio of pages scanned that were mapped. This ensure we unmap when there's some relevant work to do. The lower the vm_mapped_ratio, the earlier the kernel will start swapping/paging. (ah, and of course also the SWAP_CLUSTER_MAX would better be a sysctl but it isn't yet)

Andrea

It matters a lot since without this change max_mapped will always be larger than max_scan and swap_out() will NEVER be called.

If this is fixed in another way in -aa I must have missed that piece of code, I only stared at the patch for about 10 minutes before writing this email.

Absolutely agreed ... if it helps you, it was marcelo who changed the 9 to 10 ;)

Ideally we'd have a VM which runs ok without magic numbers, or at least one where changing the magic numbers has extremely little influence, the defaults work and the sysctl switches don't require you to learn how all the VM internals work.

Yes, but what happens when orig_max_mapped gets larger than max_scan ? How does the -aa VM protect against that ?

regards,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

I see, thanks for the explanation.

yes, -aa handles differently the case where max_scan expires.

Never mind :)

100% agreed.

in short I recall swap_out also from the outside of shrink_caches, if it fails (if max_scan timeouts it means shrink_caches will fail).

Andrea

Well, I haven't reported it yet, but booting my box with mem=4M gave as result: (running 2.4.18-pre2aa2):
diego# cat /var/log/messages | grep gfp

Jan 13 15:37:10 localhost kernel: __alloc_pages: 0-order allocation failed
(gfp=0xf0/0)
Jan 15 16:06:28 localhost kernel: __alloc_pages: 0-order allocation failed
(gfp=0xf0/0)
Jan 15 18:37:21 localhost kernel: __alloc_pages: 0-order allocation failed
(gfp=0xf0/0)
Jan 15 21:58:32 localhost kernel: __alloc_pages: 0-order allocation failed
(gfp=0xf0/0)
Jan 15 21:58:33 localhost kernel: __alloc_pages: 0-order allocation failed

(gfp=0xf0/0)
diego#

Each script of /etc/rc.d was killed by VM when it was started, there wasn't any "OOM", just
"VM killed..." or something similar.
As /etc/rc.d scripts were killed, I couldn't start swap.

The gfp=0x... numbers were not always the same, but I can't remember them because syslogd wasn't running.
I can repeat this if you want and I'll copy all messages.

..I remember running 2.2.14 in a 386 box with 4MB of RAM and 8 or 16 of swap. It was veeery slow, but even I could run apache :-)...

Can we first make sure that the other factors dont plat a rol in this benchmark? I have a couple (14+) Dell servers here, and i know for a fact that most of their RAID systems are heavely borked in the performance department.

All kernels upto 2.4.1x performed horibly, and all kernels after 2.4.16 or so perform horibly again! Somewhere inbetween some magic seemed to happen in the block layer / elevator code / etc, that caused performance to increase upto 100% on the Dell PERC adapters. (started @ the first release of the AA VM). However after a few small releases, the performance went down to the same old horible level again.

So it might well be (very likely actualy) that the tested redhat 2.4.14 is a performance 'sweet spot' kernel, where kernels < 2.4.13 and > 2.4.15 or 16 are definatly not.

The raid performance is a whole issue on its self. part seems to be block IO / Elevator / driver related, and a part seems to be adapter firmware related. (And adaptec refusing to release their drivers).

However since both 2.4.17 and 2.4.7 have the same horible RAID performance, i do not think the VM is responcible for that part ;-)

A good test would be to configure those disks on a normal AIC7xxx adapter, and software raiding them together. The performance of that is 'equal' between those different kernels, and much much higher then the hardware raid. Benchmarking with this would give much better results for benchmarking VM's

-- Chris

[...]

Sorry, but in my opinion Rik's rmap VM still beats your VM under IO load. My benchmark is very simple: import a kernel tree into a CVS tree that already contains about 470 other kernel trees. Both the import directory and the CVS root are on the same disk. With 2.4.17 the mp3 player stutters, I can't even read email or edit a couple of files with XEmacs at the same time. With 2.4.17-rmap-11a the mp3 player runs smoothly and email and XEmacs are usable again.

Some time ago Linus made the important observation that we shouldn't tune the scheduler for SMP systems simply because 99.9% of the systems in the world running linux have a single CPU. IMHO an equally well observation would be that we shouldn't tune the VM for the 0.1% of the systems in this world that run large DMBSes. The 99.9% majority is much more important.

Just my 0.02.

Erik

J.A.K. (Erik) Mouw, Information and Communication Theory Group, Faculty
of Information Technology and Systems, Delft University of Technology,
PO BOX 5031, 2600 GA Delft, The Netherlands  Phone: +31-15-2783635
Fax: +31-15-2781843  Email: [email protected]
WWW: http://www-ict.its.tudelft.nl/~erik/

Try 2.4.17-aa as well - the -aa stuff generally behaves better than 2.4.17 base

There is a way to fulfill both needs. If my systems are part of that 0.1%, I have to disagree with you. :)

There is no way to make one good VM for all possible situations. But, you can tune/make one VM to work great on large DBMS (e.g.) and tune/make another one to work great on ordinary desktop systems (playing mp3s & co). So, add different VMs as kernel-config options. The 'default' one should be VM for 99.9% users. Everybody happy? :)

Greetings

This is an interesting assertion ... but up to date nobody has been able to tell me what exactly should be different between these two mythical VMs ;)

regards,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Well what is the different requirements for desktop use versus, the server in terms of virtual memory? What I am doing on my system with MP3's playing, running Xfree86 on a SuSE 7.3 system will be different from a server running DMBS's.

Another issue, would it be possible to select one or the other VM's like you do for example the CPU type?

Kind regards,

Mayy

That's a good question ....

.... especially because I haven't seen any suggestion on how these different workloads would get translated into different VM requirements.

regards,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

So, your argument basically boils down to "the differences will be that it's, well, different." Please do us a favour and never join a debating team of any sort.

-Ryan

Did you get out of the wrong side of the bed or something? Geez, try and help then get flamed by a troll for your efforts...Plus the fact I was not even arguing! I was asking questions...

Further, I was postulating to see what the differences (or indeed similarities) are between the VM's to cope with desktop use vs server use. No-one seems to know right now.

How does one test the VM precisley? Sorry for the ignorance on this subject.

Kind regards,

Matt
PS If you're going to reply with a childish, unproffessional remark do us all a favor and be quiet. If I say something stupid and nobody replies I get the hint usually.

Having been active doing just that, I have been triggering my favorite bad behaviour and trying to evaluate which (if either) makes the system run better, as defined by both measurements and "feel."

The two problems I have been seeing are (1) load with low to moderate memory, and (2) sudden i/o bursts freezing the system when doing large writes (CD image creation).

My test for #1 is simple, I compile the 2.4.16 stock kernel after booting with mem=64m or mem=128m options. I have a batch of files to hack into something I can post, and I ran on an Athlon 1400 and dual Celeron 500 system, so I have moderate UP and SMP machines of similar performance when full memory is used. I compile with:
make clean; make dep
make bzImage modules MAKE='make -j7'

I took all these numbers with the intent of posting, but the runs finished at 0630 this morning and I haven't the time yet. Gut feeling is that 17-rmap-11c works better under small memory, 18pre2aa2 was better when creating CD images on the Athlon, ran out of time for the SMP.

Neither crashed, hung, or caused the OOM to commit procedural genocide, which plain 2.4.17 does. the -aa kernel was also tested with my own patch for intermediate disk loads, I will post when I'm sure it's actually better by enough to matter. I believe the extra tuning in -aa allows better large i/o performance if you match bdflush to your load.

Chech large i/o by sync() followed by a fast CD build from wav files, on fast disk. When the disk light come on hard, grab a window and try to wave it around, or change X virtual desktops. Chances are that you will get bad to nil response if you have fast disk and CPU. I get a whole 600MB in memory before the light comes on :-(

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Slight typo In last line, should read "What am I" not "What I am" perhaps this lead to some confusion. Shouldn't write to this list after driving for almost 12 hours :).

Matt

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I can see two different "VMs". I say "VMs" because it could be the same code with different magic numbers to control its behavior.

aspect. Both disk and network throughput. Interactive response on such systems isn't as critical as most of the time it will sit there processing queries or sending files.

disk and network throughput also have to have a fine balance. Maybe the balance is three way here between interactive response, disk throughput and network throughput.

Perhaps having a VM system that you select your main focus server vs desktop would be the way to go. Also the end-user should be able to adjust this balance. Say a person selected desktop, and is a graphic artist, they may not care as much about network thoroughput and rather push up interactive response and disk throughput at the expense of the network thoroughput.

Regards.
PS: IANAVMP (I Am Not A VM Programmer)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iEYEARECAAYFAjxIRZAACgkQwtU6L/A4vVDUTQCdG4Pg4hYGPvRXN9kBVfDyWBbD bnsAnigMlPA21izLJUhKjZcTeeaaK9IC
=EKri
-----END PGP SIGNATURE-----

There is another VM that has a property that people would like:
deterministically handling memory exhaustion. Unfortunately, that VM probably can't co-exist with over-commit and the performance gains that affords.

 "Love the dolphins," she advised him. "Write by W.A.S.T.E.."

It can definitely co-exist. Overcommit control is just a book keeping exercise on address space commits.

A bookkeeping technique developed by Arthur Anderson.

DS

Hardly, and for many workloads its actually a very good thing to do. Of course there is always a small mostly theoretical risk of doing an Enron

And that's a _definitely_; other OS's have done it. Digital Unix, for one,
on the basis of a file called 'swapdefault', swapped between overcommit and precommit modes. I was rather disappointed when I first tried to enable overcommit mode on Solaris 2.x (where 'x' was probably somewhere around 4) and searched for quite some time before giving up and deciding it wasn't a tunable option...

Although I've never actually deliberately run a system in precommit mode,
it always used to be the first thing to "fix" on a Digital Unix box, and when I discovered Solaris had the same "flaw" my suggestion was to move the affected applications (large applications which fork before exec'ing or fork short lived-children, at around 1/2Gb+ each, which should just be short-lived COW shared mappings and not exhaust memory) to Linux.

And while precommit may be something people ask for, I'd have to say many
of them would, having experienced the difference on identical hardware, then realise what a bad idea it was and go back to the current mode. That is, it sounds like a big waste of time to implement the 'traditional'
behaviour which Linux is already so much better than.

David.
--

David Luyer                                     Phone:   +61 3 9674 7525
Network Manager                P A C I F I C    Fax:     +61 3 9699 8693
Pacific Internet (Australia)  I N T E R N E T   Mobile:  +61 4 1111 BYTE
http://www.pacific.net.au/                      NASDAQ:  PCNTF

Precommit basically just asks the application to die when it first allocates memory if it's even possible for it to die in the most pathlogical usage case of that memory.

I.E. "die up front" instead of "die while running". (There's no possible way this can improve performance. If you want to never swap, just don't mount a swap partition.) You don't even have to change the VM's behavior, it can still copy on write and such. You just add a test to cause allocations to fail unnecessarily at times.

Throwing in a little extra code on mmaps and allocations to kill a process wouldn't be too hard. It would be stupid outside of something like a financial transaction system (and probably even in there), but it technically shouldn't be all that hard to do.

Unless I missed something...?

Rob

Why don't we all follow the MSCommit method of VM? We simply allocate 99% of physical RAM for cache and other non-userspace purposes, and whenever an application needs memory, pop up a message:
printk("Your system is out of virtual memory. Linux is increasing
your virtual memory size. During this time, memory allocation requests may fail.\n");

Then spend a few minutes doing hard disk I/O, while exposing bugs in programs that don't check to make sure that malloc succeeded.

;)

--
Stevie-O

Real programmers use COPY CON PROGRAM.EXE

I have no clue about VM's but I can imagine that for example the following situations have different requirements:
1-Desktop: many "small" apps, I believe exe's remain in memory and data is written to disk? Anyway I can imagine fragmentation and latency is an issue here.
2-DBMS: 1 or 2 big programs which sometimes even do their own memory management.Fragmentation and latency isn't issue here I think however moving ltos of data to and from swap is.
3-Webserver: for example apache with many childs being created under high load and killed under low load. The data is always small (in case of static pages). So a lot of small swaps? Latency is not as much as un issue but I can imagine that fragmentation can be an issue?

I think these 3 situations behave very differently, but then again it's just what I think. I can also imagine that more situations are possible but not many. I also indictated that we have a few parameters we can optimise for like latency, fragmentation and moving a lot small chunks, or occasionally 1 big chunk.

I know from an AI perspective that optimize for 3 different parameters is difficult.

There is another situation to consider which I think more typical in the "Real World"

4 - The Typical Web server Environment, in many many companies that I have seen.
The machine runs both the DBMS(mysql, Oracle, etc) and the web server
Apache. Therefore a balence of memory use needs to be struck between the two applications. Then ususally the system also has a development environment at the ready incase changes need to be made quickly from the console. While this may not be true as things scale up for a business and functions get separated to different machines. Many startups and small businesses that I have worked with that have turned to LINUX for the OS becuase the TCO, it offers over the other guys, tend to have just that single does everything box.

-Bill

A lot of times a DBMS is bulit that way because they assume they know better than the OS designer how memory should be managed. Same reason they usually use raw writting the the drive instead of using the OS calls.

Is this right or fair? I don't know. But it does imply that if a
VM or FS layer for an OS performs well enough, that a DBM system might be built that would be built to take advantage of the OS's VM and FS layer.

Ciao!

"When you have to shoot, shoot! Don't talk."
		--Tuco (The Good, The Bad, and The Ugly)

The Doctor What: Kaboom!                         http://docwhat.gerf.org/
[email protected]                                                   KF6VNC

Actually this isn't true. DBMS usually handle their own memory because everything is done in blocks of the same size. Since this is configured as part of the DBMS' parameters, it is much better at handling this than the OS ever could be, once it has garnered the original memory from the OS. Remember, this space is normally shared memory as well.

As for the FS, DBMS' prefer raw devices for consistency issues (as well as speed). Raw devices prevent the use of the kernel's internal buffers for files (thus reducing the number of memory copies involved).

/Mike

Nice try, but do the test again with 2.4.18-pre2-aa2 -

2.4.17 doesn't neccesarily have all andrea's fixes.

Just my .02

jjs

1) the journalist may be good, but his english was far from fluent, there was some confusion at times during the interview

2) the interview was done quite informally on IRC, with me replying in normal IRC style ... it appears however that the sentence fragments were just cut'n'pasted together into something gramatically dubious, this has messed up the contents in some places ;)

3) I guess this whole stuff was converted to gramatically correct Italian, possibly meaning something slightly different from the text in (2), definately something else than what I wanted to say.

I guess this is the last time I'm giving an interview to a journalist who isn't fluent in any of the languages I'm fluent in ;)

cheers,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Then...

What did you want to say exactly?

Why aren't you rushing out to provide corrections?

What are you waiting for?

Nicolas

It's just an article, it's just this week.

Next week people will be all upset about the next block io layer flamewar or maybe the politicians will have time again for the next round of DMCA.

I'm not a good writer, by the time I'd have written any correction or follow-up people would have mostly forgotten this thing anyway.

regards,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Wrong century, Rik.
Google is your friend.

But anyway, hope we all have learned something.

Regards,
Stephan

This email is just to let you all know that the original interview was the english one. I published it as it was in my IRC log, without correcting any grammatical/syntactical error, removing emoticons, changing any word.

Rik, I asked you to choose between an email or IRC interview, you chose IRC, so from that moment if you used an informal language, it was out of my job. If you aren't able to take your responsabilities for your actions or words, this probably means you're not enough mature to be a maintainer. If you are sure to be a victim, publish your logs, i'll be happy to publish mine.

The interview was translated to italian by a highly qualified person. Anyway, both italian and english interviews are on the web, i'm sure Andrea, or whoever, will be happy to tell you if anything was changed.

Best regards,
Andrea Scrimieri

I'm willing to take full responsability for what I mean, not for other people's interpretations of my words.

It seems IRC isn't a good medium to do interviews so I won't do that again.

regards,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

For the last time, i didn't change the interview text or cut anything. If you want to continue throwing shit to me because you are not responsible of your words, please publish the logs. I'm beginning to get tired of sentences like: "That assumes it is my assertion, it appears the journalist in question is missing a few lines from his IRC log though...". That's not true.

If you saw some differences beetween my version an yours, why didn't you tell me that? Maybe because in reality there were not? This controversy is started almost 2 days after the publishing of the article. If lies or inaccuracies would be written on it you'd have all the time you needed to tell me. You started to affirm that there were cuts only after the first Andrea's email. Please be fair and get your own resposibilities.

You chose IRC, I don't.

Regards,

Andrea

Which part of "to interprete" do you not understand ?

Interpreting my text is reading it and building a meaning for the words in your head. It doesn't mean the text needs to be changed ... the meaning can be different for each person reading the exact same words.

I guess this is why politicians are very careful to not attach any meaning at all to their words, however I'm a programmer, not a politician ;)

And that was my mistake, indeed.

I think you're reading things into my words that I didn't want to put there.

regards,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

Rik - bury this for good. Tell the world what you did mean to port there

To an extant anyone is responsible for how another takes (interprets) what they say. Culture is that combined collection of knowledge of the human race. It is your responsibility as a human to be as clear as possible (within reason).

Were you that clear? It seems not, but I don't know. Just don't say you have no responsibility for how others take what you say, because that is just dog crap. No communication or collaboration would ever take place if we all had that attitude. (It most likely couldn't take place, let alone wouldn't.)

Trever Adams

Let us not forget about 'deconstructionism':

deconstructionism -- (a philosophical theory of criticism ... that seeks to expose deep-seated contradictions in a work by delving below its surface meaning)

It doesn't even matter what Rik thinks he wrote. The real meaning is what *we*, the readers, can devolve from his writings.

8-)

- Brian Litzinger <[email protected]>

Nice try, but 2.4.17-rmap-11a doesn't have all Rik's fixes either (rmap-11b is available, but I don't feel like running a new kernel every few days).

I've been running a couple of 2.4.17-pre kernels on my laptop (which is my primary machine), but each time they made me switch back to good old 2.4.13-ac5 simply because its VM (read: Rik's VM) was much smoother.

It's not that I think Andrea's VM is bad, it's just that a VM should be tuned for the common cases, not for the power users that want to squeeze every last drop out of it. It's fine with me if somebody wants to design a VM for the niche XYZ, but do that as a separate patch and don't clutter up the mainline kernel with it.

Erik
[sleep(7*3600);]

J.A.K. (Erik) Mouw, Information and Communication Theory Group, Faculty
of Information Technology and Systems, Delft University of Technology,
PO BOX 5031, 2600 GA Delft, The Netherlands  Phone: +31-15-2783635
Fax: +31-15-2781843  Email: [email protected]
WWW: http://www-ict.its.tudelft.nl/~erik/

In article <[email protected]> you write:

| It's not that I think Andrea's VM is bad, it's just that a VM should be
| tuned for the common cases, not for the power users that want to
| squeeze every last drop out of it. It's fine with me if somebody wants
| to design a VM for the niche XYZ, but do that as a separate patch and
| don't clutter up the mainline kernel with it.

I have a few points of disagreement with that. I have no idea what
prices are doing elsewhere, but in the USA memory prices are around $250-400/GB for memory (from crap to decent ECC) and there are a lot more machines which live in the "power user" range then there used to be. And when memory was really cheap, ~$150/GB, many people built big Athlon systems for small $$. I totally agree that Linux should run in 4MB, but that's not typical anymore.

My real disagreement is that we should be doing worst case analysis on
the VM and scheduler rather than trying to go for best at one thing, calling that typical, and then letting all other loads take the leavings. I like to be able to tune, the the kernel should do a decent job with systems having any reasonable mix of large and small, i/o and CPU bound jobs. I don't like even the implication that it's okay for performance to suck, or for new kernels to be worse than 2.4.14 or so. Alan Cox had some somments on this, and has started the -ac series again because of it.

So far I find 18pre2aa2 with some setting for bdflush to work
acceptably on several largish machines and one small system with many processes and working set 3-4x physical memory. More later.

Let's aim for "doesn't suck" instead of "perfect for XXX" and more
people will be satisfied if not delighted.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Hi Chris,

We run these tests regularly on different hardware - We have hardware from Compaq, Dell, HP, IBM and FSC.

All tests so far showed the same overall performance tendencies.

Greetings
Christoph

If you want to insult the Red Hat people please don't do it from a SuSE address. There are some great people at SuSE and I somehow doubt you speak for the management or major stockholders (ibm etc)

When Red Hat shipped 7.2 the -aa vm didn't even exist. It was 2.4.7 era - so the choices were Rik's stuff half missing and mangled by Linus versus Riks stuff proper (2.4.7-ac). The latter passed QA the formed choked and died. The same basically applied for the 2.4.9 based errata - combined with a desire to reduce unneeded change, because paying corporate customers want certainly not neat toys. Shipping 2.4.10 to customers would have been pretty irresponsible when it seems that in some cases even things like fsync() didnt actually work. The O_DIRECT security bug with /dev alone I think justified that caution.

When we tested 2.4.17 during evaluation we found it 20% slower on many I/O heavy workloads. I don't know if -aa ever got that far in QA testing. I do know 2.4.17+rmap11* passes Cerberus.

2.4.17 base until you get high loads then it gets ugly. Clearly it has a lot of things right.

At the moment both 2.4.18pre+rmap and -aa are better than base 2.4.18pre3, so its in your interest to actually send Marcelo the changes one at a time with explanations to get 2.4.18 better and better, and with luck find which change is causing the horribly heavy swap behaviour and "0 order allocation failed" cases along the way.

Alan

if you're using the blkdev directly, then please try to mount the blkdev with a 4k filesystem before making your benchmark, that should give you the magic performance back. 2.4.10 intentionally were defaulting to 4k I/O, this is probably what made the difference for you.

Andrea

yes, I was thinking about you report.

Then can you verify the bandwith you get out of the network card is the same across 2.4.13-ac7 and all the other kernels you are trying. Also please check with an hdparm -t the speed you get out of IDE is the same. This sounds like some driver changed (note that -ac is used to queue lots of driver updates) and that made the difference. Otherwise if we wakeup bdflush early enough I don't see why it takes more time.

Andrea

0xf0 shouldn't lead to an oom killing, there should be some other failure before the killing. The above are normal warnings, they're KERN_NOTICE, not KERN_WARNING nor KERN_ERROR.

That means there wasn't enough memory, sounds like your bootup sequence is broken and startup something big before activating swap, either that or you start something that takes more than 16+4m, note that with any recent distribution it is very easy that you need 16+4 after a little time after boot.

If you could provide a vmstat trace during the VM killing, that could show better if the VM is the culprit or if it did the right thing.

I know for experience at the first VM killing people tends to point the finger at the VM (me too sometime at first, see the pte-highmem thread) but at least in my tree that never turned out to be the case yet.

Can you try to boot with emergency, then activate swap, and then check if it runs oom again despite lots of free swap available etc...? thanks,

:)

Andrea

do you plan to sue me as well? :)

"If redhat doesn't use the -aa VM " was a short form of "if redhat cannot see the goodness of all the bugfixing work that happened between the 2.4.9 VM and any current branch 2.4, and so if they keep shipping 2.4.9 VM as the best one for DBMS and critical VM apps like the SAP benchmark".

I think it's fair enough to say that if you plan to keep shipping 2.4.9 VM with all its troubles like I understood yesterday (starting from VM highmem deadlocks, to kswapd looping into ZONE_DMA etc..., swap storms throwing the realistic SAP benchmark to /dev/null) that was not usable on long uptimes on big DBMS with several gigabytes of ram.

Somebody else also complained me about this saying that from what I said it looks like the -aa VM is the best thing possible which is obviously not true. In such two lines I said -aa VM just to go short. The -aa VM in 2.4.18pre2aa2 is obviously certainly not the best that you can make and I suggest everybody to try to make things better and invent and try new algorithm etc... that is just the best compromise that _I_ could make so far. So it is obvious if anybody doesn't use the -aa VM in 2.4.18pre2aa2 it doesn't mean he doesn't understand about VM. as far I can tell rmap could be an order of magnitude better of -aa VM in 2.4.18pre2aa2, it's just I didn't checked it yet (because of all the non obvious implication the rmap design adds, see DaveM emails of one year back to linux-mm). All my wondering in my previous email was between 2.4.9 VM with all its known troubles and a sane version of the current vm like in 2.4.18pre2aa2. So about the past and the present, not about the present and the future. I thought it was obvious from the context of the email. I said this in two lines and apparently RedHat didn't like it, I'm sorry, but quite frankly I think that was quite fair enough, at least with this additional clarification added.

Andrea

Redhat's 2.4.9 is about as close to 2.4.9 as your 2.4.18-aa is to 2.4.17.

If you want to judge Redhat by vanilla 2.4.9, I guess we should start judging -aa based on measuring 2.4.17 ;)

regards,

Rik

"Linux holds advantages over the single-vendor commercial OS"
    -- Microsoft's "Competing with Linux" document

http://www.surriel.com/		http://distro.conectiva.com/

The RH VM is totally unrelated to the crap in 2.4.9 vanilla. The SAP comment begs a question. 2.4.10 seems to have problems remembering to actually do fsync()'s. How much of your SAP benchmark is from fsync's that dont happen ? Do you get the same values with 2.4.18-aa ?

Alan

Well, basically we checked the thing many times with quite different kernels.
Our current tests - which show exactly the same results as 2.4.[10,14,15] - run
on the new "official" SuSE kernel 2.4.16. Again, we observe a performance increase
in high swap situations of about a factor of ten compared to 2.4.[7,9].

IMO, this shows that errors like fsync etc. are _not_ responsible for the improved
performance.

But of course, we will check the newer kernels as well. I think we could live
with another factor of ten ...

Best regards
Willi

-----------------------------------
Willi Nuesser
SAP Linuxlab

Well, basically we checked the thing many times with quite different kernels. Our current tests - which show exactly the same results as 2.4.[10,14,15] - run on the new "official" SuSE kernel 2.4.16. Again, we observe a performance increase in high swap situations of about a factor of ten compared to 2.4.[7,9].

IMO, this shows that errors like fsync etc. are _not_ responsible for the improved performance.

But of course, we will check the newer kernels as well. I think we could live with another factor of ten ...

Best regards
Willi

-----------------------

Willi Nüßer
SAP LinuxLab

and I assume you were using either ext2 or reiserfs anyways, so the fsync problem never affected you since the first place (also with older kernels) I believe.

Andrea

Hi Andrea,

It was done on ext2 _and_ against raw devices. Same dendency on both setups.

Further on I doubt the test is very depended on fsync. It should be swap io limited since it runs with a way too small memory configuration.

If you have enough memory the test is not very IO intensive either despite the fact that a big database is running. To bring the database really into IO you have to add application servers. (Fujitsu Siemens took 160 4way Linux servers to saturate a database server running Solaris on 64way FSC Primepower.)

BTW since we are just bashing VMs: I always hear that 2.2 is so much
better: The first 2.2 kernel which could really survive this test was 2.2.19!

Greetings
Christoph

That I can believe. With the exception of the dcache balancing problem the 2.2.19/20 VM basically eliminated all 2.2 bug reports on VM behaviour.

Alan

can you reproduce the dcache problem with this patch applied?

	ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.2/2.2.20aa1/00_inode-boot-dynamic-3

The grow of the dcache in 2.2 is bound to the grow of the icache which is not dynamic in 2.2. (modulo hardlinks but I don't think the guy was using hardlinks)

Andrea

AFIK the bench was not with 2.4.10 (not that I remeber any missing fsync anyways, actually MS_ASYNC is broken and this is fixed between in 18pre2aa2 from Andrew Morton, but that was broken in 2.4.[79] too). The bench in 2.2 was delivering much better performance than with 2.4 (I don't recall the exact number) and 2.2 definitely is not missing fsync etc... furthmore the 2.2 numbers were reproducible. the benchmark swaps heavily shm etc... and the 2.4.[79] vm was collapsing at the second pass (I think first throughput was 5 then 1 1 1 1), if you swapout always the wrong part and you start trashing because of unbalance of aging it is very easy to make a x10 difference in the final numbers. I think a sane vm should run faster than 2.2 and to be reproducible as 2.2. I tend to like such test, also because it is a real life test (unlike what somebody thought). The huge regression in such test was one of the main reasons that made me to realize the brokeness of the vm algorithms.

Andrea

deals to

adjusting

performs

I'll check that and get back to you.

There is no IDE in the system. The destination for the file transfer is on cpqarray RAID5. Do you have a recommendation for how I test the transfer rate of that without stressing the VM?

One of my original tests[0] was to take the cpqarray update from -ac and bring it forward to 2.4.17. I saw about 20 sec improvement with that, still not competitive overall with -ac performance. I'll try doing the same with eepro driver, which is the NIC I'm using.

--Adam

[0]
http://www.kroptech.com:8300/mailimport/showmsg.php?msg_id=49714&db_name=linux_k
ernel