Skip to contentSkip to conversation

Multiqueue VM Patch OK? Does page-aging really work?

DISCUSSION Benjamin Redelings I 2 participants 6 messages Sep 8, 2000 – Sep 16, 2000 Memory management

lore
Share & tools
lore
#0 lore raw

Hi guys -
I just want to remind people that dbench, bonnie, mmap2, and stuff are
only one type of way to measure the "performance" or the multiqueue patch. Even if these give good numbers (which they apparently don't always), we still have major problems with the multiqueue patch, that I think prevent it from being a candidate for 2.4.0 in its current state. Simply fixing the crashes is not good enough:

	Problem #1: 
	Simply untarring a large file (e.g. linux-2.4.0-test7.tar) evicts most

of the working set. Example: I was running a few programs, including netscape, with little swap in use. Then I untarred a linux source tree, and lo and behold, look at this! 44Mb swap!
telomere:~> cat free_after_untar
total used free shared buffers
cached
Mem: 62872 61432 1440 0 1588
41800

-/+ buffers/cache:      18044      44828
Swap:       128484      44044      84440

Now I have __44Mb__ of swap?? Interestingly there is about 41Mb cached. One question I guess that needs to be asked is: is this cached data "dirty data" in the newly created "linux/" tree , or is it "clean data" from "linux.tar"?

Maybe the problem is that dirty data is given preference over other data, and tyrannically takes over the cache. (But that couldn't be the whole problem, see rest of this e-mail)

Anyway, for comparison, here is test8 after untarring a similar file telomere:/usr/src/temp> free
total used free shared buffers
cached
Mem: 62944 60980 1964 0 752
30900
-/+ buffers/cache: 29328 33616
Swap: 128484 13724 114760

	Problem #2:
	Programs that are basically unused, like (for example) an apache server

that I am running on my home computer, no longer have their RSS go down to 4K. Simply put, unused processes are not evicted, while data from used processes IS evicted.

Conclusion:
1. Aren't BOTH these problems supposed to be solved by page aging?
Then, shouldn't the multiqueue-patched kernel do BETTER than test8? Apparently page-aging is not quite doing its job. Why?
2. With the drop_behind stuff, I'm sure the kernel will perform better,
at least with problem #1, to some extent. However, even if the drop_behind stuff is moved to the VMA level, I still think this is a "special case hack". I am not trying to be overly negative or critical
- it is just that the NEED for drop_behind this indicates that page aging (the general solution) is not working. Or am I missing something?

In any case, what I am not missing is the fact that the multi-queue
patch is failing to reform the VM system in some of its most important aspects. I don't see how it could go into 2.4.0 in its current state.

keep up the good work!
-BenRI

Signature · 8 lines
"I want to be in the light, as He is in the Light,
 I want to shine like the stars in the heavens." - DC Talk, "In the
Light"
Benjamin Redelings I      <><     http://www.bol.ucla.edu/~bredelin/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
#1 lore raw
Quoted 10 lines · Benjamin Redelings I
	Problem #1: 
	Simply untarring a large file (e.g. linux-2.4.0-test7.tar) evicts most

of the working set. Example: I was running a few programs, including netscape, with little swap in use. Then I untarred a linux source tree, and lo and behold, look at this! 44Mb swap!

Now I have __44Mb__ of swap?? Interestingly there is about 41Mb cached. One question I guess that needs to be asked is: is this cached data "dirty data" in the newly created "linux/" tree , or is it "clean data" from "linux.tar"?

Indeed, this is something which still has to be fixed. You have to keep in mind that the new VM has received about 5 days of performance tweaking right now and has only received very limited testing. Most of the time working on the new VM has been spent tracking down the (now fixed) SMP bug...

Quoted 2 lines

Maybe the problem is that dirty data is given preference over other data, and tyrannically takes over the cache.

Please try to understand the code before saying something like this.

We age all active pages the same and the distiction is only made between pages which are already "evicted" from the active queue and moved to an inactive queue, so aging and flushing definately do NOT interfere with each other.

Quoted 5 lines
	Problem #2:
	Programs that are basically unused, like (for example) an

apache server that I am running on my home computer, no longer have their RSS go down to 4K. Simply put, unused processes are not evicted, while data from used processes IS evicted.

This is not a problem but only looks like one. We only evict pages which are not in active use anywhere, but keep shared pages from glibc in memory (and mapped) when they are in active use in other processes.

Quoted 4 lines

Conclusion:
1. Aren't BOTH these problems supposed to be solved by page aging?
Then, shouldn't the multiqueue-patched kernel do BETTER than test8? Apparently page-aging is not quite doing its job. Why?

The one problem you saw is indeed in need of some tuning and I'm working on it right now. One of the things which would help identifying all the problems (so we can get them fixed faster) would be having a larger test base..

Quoted 6 lines

2. With the drop_behind stuff, I'm sure the kernel will perform better,
at least with problem #1, to some extent. However, even if the drop_behind stuff is moved to the VMA level, I still think this is a "special case hack". I am not trying to be overly negative or critical
- it is just that the NEED for drop_behind this indicates that page aging (the general solution) is not working. Or am I missing something?

While page aging can be used to make the distinction between pages which have been used once and pages which are being used all the time, every use-once page puts a little bit of "pressure" on the working set and it is still possible to completely overwhelm (and evict) the working set by doing very fast streaming IO.

Drop-behind is a way to detect the difference between pages which are used once and pages which are used more often, while at the same time not putting the pages we just read ahead in jeopardy of being evicted before they are being put to use.

Quoted 4 lines

In any case, what I am not missing is the fact that the
multi-queue patch is failing to reform the VM system in some of its most important aspects. I don't see how it could go into 2.4.0 in its current state.

Define "most important". You only gave me one thing which still needs to be tweaked a bit and an optical illusion with no memory use in reality.

Things like interactive latency and speed increase for workloads which exhibit locality of reference (say, big compiles) are important as well...

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
-- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/ http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

#2 lore raw

Happiness with t8-vmpatch4 (was Re: Does page-aging really work?)

Hi guys -
I have done a few small tests on t8-vmpatch4 (64Mb RAM, 166Mhz UP
PPro), and I must say that I am very happy with the results. More comparison with test8 vanilla confirms that Rik's version is much more responsive, also that Rik is (or course) correct - unused process are indeed swapped out, but shared pages from (I assume) libc remain.

Observations:
1. test8-vmpatch4 does not swap very much at first, but then swaps a
lot of memory in a short time when triggered.
Specifically, I untarred a kernel source tree and saw NO swapping .
The working set was NOT evicted :) I then started more programs, and saw that the cache shrunk a lot: still no swapping. Then I ran the untar again, and the kernel swapped out 22Mb very quickly. However, I must say that it was still very smooth - I didn't even notice the swapping until I looked at xosview; normally it is quite audible. Also, the choice of pages for swapping out seemed to be VERY accurate - there was virtually no page-in as long as I was watching...

2. I guess we could wish that unused programs got swapped a bit sooner
instead of all at once - but presumably that can be tuned.

Congrats on the great code! Tuning does indeed seem to be fixing the
problems that I saw before.

thanks again,
-BenRI

Signature · 8 lines
"I want to be in the light, as He is in the Light,
 I want to shine like the stars in the heavens." - DC Talk, "In the
Light"
Benjamin Redelings I      <><     http://www.bol.ucla.edu/~bredelin/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
#3 lore raw

Re: Happiness with t8-vmpatch4 (was Re: Does page-aging really work?)

Quoted 6 lines · Benjamin Redelings I

1. test8-vmpatch4 does not swap very much at first, but then
swaps a lot of memory in a short time when triggered.

2. I guess we could wish that unused programs got swapped a
bit sooner instead of all at once - but presumably that can be tuned.

This is indeed something to look at. Maybe we could give idle processes (sleeping for more than 20 seconds?) a "full" swap_cnt instead of swap_cnt = rss >> SWAP_SHIFT ?

And we could also start swapping a bit earlier when the cache is getting small, but I'm not sure about how to do this or exactly what performance benefits that would give ...

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
-- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/ http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/

#4 lore raw

Re: Happiness with t8-vmpatch4 (was Re: Does page-aging really work?)

Quoted 3 lines

This is indeed something to look at. Maybe we could give idle processes (sleeping for more than 20 seconds?) a "full" swap_cnt instead of swap_cnt = rss >> SWAP_SHIFT ?

This doesn't seem like it should be necessary. Right now, unused processes ARE swapped preferentially (and completely) - its just that swapping happens all of a sudden.

Quoted 4 lines

And we could also start swapping a bit earlier when the cache is getting small, but I'm not sure about how to do this or exactly what performance benefits that would give ...

Evicting unused pages, either from the cache or from process can have
significant benefits on my machine (64Mb). Once swapping triggered, 20Mb were paged out, and stayed out. If these 20 Mb had been paged out before, then I would have had 20Mb more cache to work with, which is 31% of my memory. Go figure :)
While I agree with Byron that on low memory machines a smaller cache
can be a good thing - this DOES depend on the amount of RAM, and on the workload. But on higher memory machines, more aggressiveness against program code is good. Is there a way to make this adjust itself dynamically - e.g. by measuring page faults?
Anyway - I have seen the behavior that Byron described, where 'used'
increases as a result of a 'find'. I think that maybe some more aggressiveness against code pages (is that the right phrase?) might solve the swapping problem and improve performance on low memory machines also (since some code pages are simply evicted).

BTW, with test8-vmpatch4, I am gettings zillions of "VM: page_launder, found pre-cleaned page ?!" messages.

-BenRI

Signature · 8 lines
"I want to be in the light, as He is in the Light,
 I want to shine like the stars in the heavens." - DC Talk, "In the
Light"
Benjamin Redelings I      <><     http://www.bol.ucla.edu/~bredelin/
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/
#5 lore raw

Re: Happiness with t8-vmpatch4 (was Re: Does page-aging really work?)

Quoted 7 lines · Benjamin Redelings I
Quoted 3 lines

This is indeed something to look at. Maybe we could give idle processes (sleeping for more than 20 seconds?) a "full" swap_cnt instead of swap_cnt = rss >> SWAP_SHIFT ?

This doesn't seem like it should be necessary. Right now,
unused processes ARE swapped preferentially (and completely) - its just that swapping happens all of a sudden.

We may be able to fix that by swapping long-idle processes before we have eaten all of the cache.

Quoted 6 lines

Evicting unused pages, either from the cache or from
process can have significant benefits on my machine (64Mb). Once swapping triggered, 20Mb were paged out, and stayed out. If these 20 Mb had been paged out before, then I would have had 20Mb more cache to work with, which is 31% of my memory. Go figure :)

*nod*

Quoted 2 lines

BTW, with test8-vmpatch4, I am gettings zillions of "VM:
page_launder, found pre-cleaned page ?!" messages.

No you're not. That's test8-vmpatch3.
This is fixed in -vmpatch4...

(and the comments in the code have been
updated to explain how and what)

regards,

Rik
--
"What you're running that piece of shit Gnome?!?!"
-- Miguel de Icaza, UKUUG 2000

http://www.conectiva.com/ http://www.surriel.com/

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux.eu.org/Linux-MM/