2.6.10-rc3: kswapd eats CPU on start of memory-eating task
Subsystems (2)
Bingo.
[PATCH] token based thrashing control
http://marc.theaimsgroup.com/?l=bk-commits-head&m=109330925227996&w=2
Backing that one out 2.6.9-rc1 behaves just like 2.6.8.1-bk2, ie no freezes and swapping done in 1 minute in my testcase. Tested both with and without lapic=lapic (due to my own mind demons ;-)
If someone doubt my ability to back out a patch, here's how it looked:
root:loke:/usr/src/debug/1-mydebug/linux-2.6.9-rc1-debug-notoken# patch - Rp1 -i ../token.patch patching file include/linux/sched.h patching file include/linux/swap.h patching file kernel/fork.c patching file mm/Makefile patching file mm/filemap.c Hunk #1 succeeded at 1246 (offset 51 lines). patching file mm/memory.c patching file mm/rmap.c patching file mm/thrash.c
Then I diffed the original tree and this notoken-tree and eyeball-compared it with the patch (had to first delete a mm/filemap.c~ backup left by the patch program). Was all OK.
Would be nice though if someone else could verify...
This is also that time of the year when no strict timetables can be made. I'll be available on and off for the next 48 hours if some testing needs to be done.
Mvh
Mats Johannesson
Quoted 2 lines · Voluspa
Would be nice though if someone else could verify...
Well I'd love to, but afaik the only workloads which we currently know of involve complex userspace apps which I have no experience running.
Did anyone come up with a simple step-by-step procedure for reproducing the problem? It would be good if someone could do this, because I don't think we understand the root cause yet?
Quoted 5 lines · Andrew Morton
Quoted 1 line · Voluspa
Would be nice though if someone else could verify...
Quoted 7 lines
Well I'd love to, but afaik the only workloads which we currently know of involve complex userspace apps which I have no experience running.
Did anyone come up with a simple step-by-step procedure for reproducing the problem? It would be good if someone could do this, because I don't think we understand the root cause yet?
I admit to generally being in the same boat as you with respect to running complex userspace apps.
However, based on this and other scattered reports, I'd say it seems quite likely that token based thrashing control is the culprit. Based on the cost/benefit, I wonder if we should disable TBTC by default for 2.6.10, rather than trying to fix it, and try again for 2.6.11?
Rik? Andrew?
Also, it would be nice to have a sysctl to *completely* disable TBTC, that would make testing easier.
Nick
Quoted 24 lines · Nick Piggin
Quoted 6 lines · Andrew Morton
Quoted 1 line · Voluspa
Would be nice though if someone else could verify...
Quoted 7 lines
Well I'd love to, but afaik the only workloads which we currently know of involve complex userspace apps which I have no experience running.
Did anyone come up with a simple step-by-step procedure for reproducing the problem? It would be good if someone could do this, because I don't think we understand the root cause yet?
I admit to generally being in the same boat as you with respect to running complex userspace apps.
However, based on this and other scattered reports, I'd say it seems quite likely that token based thrashing control is the culprit. Based on the cost/benefit, I wonder if we should disable TBTC by default for 2.6.10, rather than trying to fix it, and try again for 2.6.11?
Quoted 4 lines
Rik? Andrew?
Also, it would be nice to have a sysctl to *completely* disable TBTC, that would make testing easier.
Logistically what makes sense is if a timeout of 0 is used as a test that completely disables it (avoids another sysctl too). In time for 2.6.10 we should disable it by default until the regressions are better understood. Tuning it into a useful "on" position can happen later and I suspect requires more code.
Con
Quoted 1 line · Con Kolivas
Logistically what makes sense is if a timeout of 0 is used as a test that completely disables it (avoids another sysctl too). In time for 2.6.10 we should disable it by default until the regressions are better understood. Tuning it into a useful "on" position can happen later and I suspect requires more code.
This patch should have the desired effect.
Con
mm/rmap.c +3 -0
@@ -395,6 +395,9 @@ int page_referenced(struct page *page, i {int referenced =0 ; +if (!swap_token_default_timeout) + ignore_token =1 ; +if (page_test_and_clear_young (page)) referenced++;
mm/thrash.c +4 -1
@@ -19,7 +19,10 @@ unsigned long swap_token_check;struct mm_struct * swap_token_mm = &init_mm;#define SWAP_TOKEN_CHECK_INTERVAL (HZ * 2) -#define SWAP_TOKEN_TIMEOUT (HZ * 300) +#define SWAP_TOKEN_TIMEOUT 0 +/* +* Currently disabled; Needs further code to work at HZ * 300. +*/ unsigned long swap_token_default_timeout =SWAP_TOKEN_TIMEOUT ;/*
Quoted 1 line · Con Kolivas
This patch should have the desired effect.
Look great Con, thanks.
Quoted 22 lines · Nick Piggin
Quoted 5 lines · Andrew Morton
Quoted 1 line · Voluspa
Would be nice though if someone else could verify...
Quoted 7 lines
Well I'd love to, but afaik the only workloads which we currently know of involve complex userspace apps which I have no experience running.
Did anyone come up with a simple step-by-step procedure for reproducing the problem? It would be good if someone could do this, because I don't think we understand the root cause yet?
I admit to generally being in the same boat as you with respect to running complex userspace apps.
However, based on this and other scattered reports, I'd say it seems quite likely that token based thrashing control is the culprit. Based on the cost/benefit, I wonder if we should disable TBTC by default for 2.6.10, rather than trying to fix it, and try again for 2.6.11?
Quoted 4 lines
Rik? Andrew?
Also, it would be nice to have a sysctl to *completely* disable TBTC, that would make testing easier.
Except that disabling it (with 0) reportedly did not solve the problem. There is a possibility that its a more complex issue...
Ed Tomlinson
Quoted 23 lines · Ed Tomlinson
Quoted 22 lines · Nick Piggin
Quoted 5 lines · Andrew Morton
Quoted 1 line · Voluspa
Would be nice though if someone else could verify...
Quoted 7 lines
Well I'd love to, but afaik the only workloads which we currently know of involve complex userspace apps which I have no experience running.
Did anyone come up with a simple step-by-step procedure for reproducing the problem? It would be good if someone could do this, because I don't think we understand the root cause yet?
I admit to generally being in the same boat as you with respect to running complex userspace apps.
However, based on this and other scattered reports, I'd say it seems quite likely that token based thrashing control is the culprit. Based on the cost/benefit, I wonder if we should disable TBTC by default for 2.6.10, rather than trying to fix it, and try again for 2.6.11?
Quoted 6 lines
Quoted 4 lines
Rik? Andrew?
Also, it would be nice to have a sysctl to *completely* disable TBTC, that would make testing easier.
Except that disabling it (with 0) reportedly did not solve the problem. There is a possibility that its a more complex issue...
Disabling it is more than setting it to 0. Removing the patch disables it and this does fix the problem. We need it to be truly possible to disable it. See the patch I posted on this thread later.
Con
Quoted 1 line · Nick Piggin
Andrew Morton wrote:
[snip]
Quoted 6 lines
Quoted 5 lines
Did anyone come up with a simple step-by-step procedure for reproducing the
problem? It would be good if someone could do this, because I don't think
we understand the root cause yet?
Quoted 7 lines
I admit to generally being in the same boat as you with respect to running complex userspace apps.
However, based on this and other scattered reports, I'd say it seems quite likely that token based thrashing control is the culprit. Based on the cost/benefit, I wonder if we should disable TBTC by default for 2.6.10, rather than trying to fix it, and try again for 2.6.11?
Hello,
I imagine that the issue might occur when only one process holds almost all memory and has swap token too long time.
However, TBTC has a good effect in my workload. So, I think that it is better to keep VM tunable using TBTC.
It may be a good idea to set 0 to default swap_token_timeout until we find the root cause.
Best regards,
Hideo AOKI
Quoted 20 lines · Hideo AOKI
Quoted 1 line · Nick Piggin
Andrew Morton wrote:
[snip]
Quoted 11 lines
Quoted 3 lines
Did anyone come up with a simple step-by-step procedure for reproducing the problem? It would be good if someone could do this, because I don't think we understand the root cause yet?
I admit to generally being in the same boat as you with respect to running complex userspace apps.
However, based on this and other scattered reports, I'd say it seems quite likely that token based thrashing control is the culprit. Based on the cost/benefit, I wonder if we should disable TBTC by default for 2.6.10, rather than trying to fix it, and try again for 2.6.11?
Quoted 8 lines
Hello,
I imagine that the issue might occur when only one process holds almost all memory and has swap token too long time.
However, TBTC has a good effect in my workload. So, I think that it is better to keep VM tunable using TBTC.
It may be a good idea to set 0 to default
swap_token_timeoutuntil we find the root cause.
Yes, with Con's patch to have TBTC turned off when swap_token_timeout is set to zero. It causes unacceptable regressions, so that is the best way to go.
It would be great to get it fixed, but I would be worried about putting in new patches for it now, right before 2.6.10.
Hello,
Quoted 4 lines
Did anyone come up with a simple step-by-step procedure for reproducing the
problem? It would be good if someone could do this, because I don't think we understand the root cause yet?
Here's a step-by-step explanation of the way I test this:
- Get the Memory Eater and compile it:
http://lkml.org/lkml/2004/12/13/272
- Do a clean boot
- Start top, and some app that has a clock and preferrably a CPU graph (to monitor screen freezes and CPU load; it's IceWM for me)
- Start the Memory Eater
- Give it an amount of megabytes that is more than the actual RAM size. I use a value of 300, as my computer has 256 M RAM.
- Enjoy :) "eatmemory" will slowly eat up more and more RAM (visible in top as RSS); under 2.6.8.1 no screen freezes come, and under 2.6.9 and 2.6.10-rc3 they do come; under 2.6.10-rc3 I also see high CPU periods for kswapd.
Yours, Mikhail Ramendik
Quoted 4 lines · On Mon, 20 Dec 2004 [email protected] wrote:
- Enjoy :) "eatmemory" will slowly eat up more and more RAM (visible in top as RSS); under 2.6.8.1 no screen freezes come, and under 2.6.9 and 2.6.10-rc3 they do come; under 2.6.10-rc3 I also see high CPU periods for kswapd.
The high cpu use for kswapd should be fixed by applying the vm-pageout-throttling.patch patch from -mm.
I'll also come up with a patch to not have the swap token used when the system is not under a swapin load...
Signature · 7 lines
He did not think of himself as a tourist; he was a traveler. The difference is partly one of time, he would explain. Where as the tourist generally hurries back home at the end of a few weeks or months, the traveler belonging no more to one place than to the next, moves slowly, over periods of years, from one part of the earth to another. Indeed, he would have found it difficult to tell, among the many places he had lived, precisely where it was he had felt most at home. -- Paul Bowles