Skip to contentSkip to conversation

2.6.10-rc3: kswapd eats CPU on start of memory-eating task

DISCUSSION Voluspa 9 participants 12 messages Dec 20, 2004

Subsystems (2)
lore
Share & tools
lore

Voluspa

author
#0 lore raw

Bingo.

[PATCH] token based thrashing control
http://marc.theaimsgroup.com/?l=bk-commits-head&m=109330925227996&w=2

Backing that one out 2.6.9-rc1 behaves just like 2.6.8.1-bk2, ie no freezes and swapping done in 1 minute in my testcase. Tested both with and without lapic=lapic (due to my own mind demons ;-)

If someone doubt my ability to back out a patch, here's how it looked:

root:loke:/usr/src/debug/1-mydebug/linux-2.6.9-rc1-debug-notoken# patch -
Rp1 -i
../token.patch
patching file include/linux/sched.h
patching file include/linux/swap.h
patching file kernel/fork.c
patching file mm/Makefile
patching file mm/filemap.c
Hunk #1 succeeded at 1246 (offset 51 lines).
patching file mm/memory.c
patching file mm/rmap.c
patching file mm/thrash.c

Then I diffed the original tree and this notoken-tree and eyeball-compared it with the patch (had to first delete a mm/filemap.c~ backup left by the patch program). Was all OK.

Would be nice though if someone else could verify...

This is also that time of the year when no strict timetables can be made. I'll be available on and off for the next 48 hours if some testing needs to be done.

Mvh
Mats Johannesson

#1 lore raw
Quoted 2 lines · Voluspa

Would be nice though if someone else could verify...

Well I'd love to, but afaik the only workloads which we currently know of involve complex userspace apps which I have no experience running.

Did anyone come up with a simple step-by-step procedure for reproducing the problem? It would be good if someone could do this, because I don't think we understand the root cause yet?

#2 lore raw
Quoted 5 lines · Andrew Morton
Quoted 1 line · Voluspa

Would be nice though if someone else could verify...

Quoted 7 lines

Well I'd love to, but afaik the only workloads which we currently know of involve complex userspace apps which I have no experience running.

Did anyone come up with a simple step-by-step procedure for reproducing the problem? It would be good if someone could do this, because I don't think we understand the root cause yet?

I admit to generally being in the same boat as you with respect to running complex userspace apps.

However, based on this and other scattered reports, I'd say it seems quite likely that token based thrashing control is the culprit. Based on the cost/benefit, I wonder if we should disable TBTC by default for 2.6.10, rather than trying to fix it, and try again for 2.6.11?

Rik? Andrew?

Also, it would be nice to have a sysctl to *completely* disable TBTC, that would make testing easier.

Nick

#3 lore raw
Quoted 24 lines · Nick Piggin
Quoted 6 lines · Andrew Morton
Quoted 1 line · Voluspa

Would be nice though if someone else could verify...

Quoted 7 lines

Well I'd love to, but afaik the only workloads which we currently know of involve complex userspace apps which I have no experience running.

Did anyone come up with a simple step-by-step procedure for reproducing the problem? It would be good if someone could do this, because I don't think we understand the root cause yet?

I admit to generally being in the same boat as you with respect to running complex userspace apps.

However, based on this and other scattered reports, I'd say it seems quite likely that token based thrashing control is the culprit. Based on the cost/benefit, I wonder if we should disable TBTC by default for 2.6.10, rather than trying to fix it, and try again for 2.6.11?

Quoted 4 lines

Rik? Andrew?

Also, it would be nice to have a sysctl to *completely* disable TBTC, that would make testing easier.

Logistically what makes sense is if a timeout of 0 is used as a test that completely disables it (avoids another sysctl too). In time for 2.6.10 we should disable it by default until the regressions are better understood. Tuning it into a useful "on" position can happen later and I suspect requires more code.

Con

#4 lore raw
Quoted 1 line · Con Kolivas

Logistically what makes sense is if a timeout of 0 is used as a test that completely disables it (avoids another sysctl too). In time for 2.6.10 we should disable it by default until the regressions are better understood. Tuning it into a useful "on" position can happen later and I suspect requires more code.

This patch should have the desired effect.

Con

mm/rmap.c +3 -0
Index: linux-2.6.10-rc3/mm/rmap.c
===================================================================
@@ -395,6 +395,9 @@ int page_referenced(struct page *page, i
 {
 	int referenced = 0;
 
+	if (!swap_token_default_timeout)
+		ignore_token = 1;
+
 	if (page_test_and_clear_young(page))
 		referenced++;
 
mm/thrash.c +4 -1
Index: linux-2.6.10-rc3/mm/thrash.c
===================================================================
@@ -19,7 +19,10 @@ unsigned long swap_token_check;
 struct mm_struct * swap_token_mm = &init_mm;
 
 #define SWAP_TOKEN_CHECK_INTERVAL (HZ * 2)
-#define SWAP_TOKEN_TIMEOUT (HZ * 300)
+#define SWAP_TOKEN_TIMEOUT	0
+/*
+ * Currently disabled; Needs further code to work at HZ * 300.
+ */
 unsigned long swap_token_default_timeout = SWAP_TOKEN_TIMEOUT;
 
 /*
#5 lore raw
Quoted 1 line · Con Kolivas

This patch should have the desired effect.

Look great Con, thanks.

#6 lore raw
In reply to Nick Piggin
Quoted 22 lines · Nick Piggin
Quoted 5 lines · Andrew Morton
Quoted 1 line · Voluspa

Would be nice though if someone else could verify...

Quoted 7 lines

Well I'd love to, but afaik the only workloads which we currently know of involve complex userspace apps which I have no experience running.

Did anyone come up with a simple step-by-step procedure for reproducing the problem? It would be good if someone could do this, because I don't think we understand the root cause yet?

I admit to generally being in the same boat as you with respect to running complex userspace apps.

However, based on this and other scattered reports, I'd say it seems quite likely that token based thrashing control is the culprit. Based on the cost/benefit, I wonder if we should disable TBTC by default for 2.6.10, rather than trying to fix it, and try again for 2.6.11?

Quoted 4 lines

Rik? Andrew?

Also, it would be nice to have a sysctl to *completely* disable TBTC, that would make testing easier.

Except that disabling it (with 0) reportedly did not solve the problem. There is a possibility that its a more complex issue...

Ed Tomlinson

#7 lore raw
Quoted 23 lines · Ed Tomlinson
Quoted 22 lines · Nick Piggin
Quoted 5 lines · Andrew Morton
Quoted 1 line · Voluspa

Would be nice though if someone else could verify...

Quoted 7 lines

Well I'd love to, but afaik the only workloads which we currently know of involve complex userspace apps which I have no experience running.

Did anyone come up with a simple step-by-step procedure for reproducing the problem? It would be good if someone could do this, because I don't think we understand the root cause yet?

I admit to generally being in the same boat as you with respect to running complex userspace apps.

However, based on this and other scattered reports, I'd say it seems quite likely that token based thrashing control is the culprit. Based on the cost/benefit, I wonder if we should disable TBTC by default for 2.6.10, rather than trying to fix it, and try again for 2.6.11?

Quoted 6 lines
Quoted 4 lines

Rik? Andrew?

Also, it would be nice to have a sysctl to *completely* disable TBTC, that would make testing easier.

Except that disabling it (with 0) reportedly did not solve the problem. There is a possibility that its a more complex issue...

Disabling it is more than setting it to 0. Removing the patch disables it and this does fix the problem. We need it to be truly possible to disable it. See the patch I posted on this thread later.

Con

#8 lore raw
In reply to Nick Piggin
Quoted 1 line · Nick Piggin

Andrew Morton wrote:

[snip]

Quoted 6 lines
Quoted 5 lines

Did anyone come up with a simple step-by-step procedure for reproducing the
problem? It would be good if someone could do this, because I don't think
we understand the root cause yet?

Quoted 7 lines

I admit to generally being in the same boat as you with respect to running complex userspace apps.

However, based on this and other scattered reports, I'd say it seems quite likely that token based thrashing control is the culprit. Based on the cost/benefit, I wonder if we should disable TBTC by default for 2.6.10, rather than trying to fix it, and try again for 2.6.11?

Hello,

I imagine that the issue might occur when only one process holds almost all memory and has swap token too long time.

However, TBTC has a good effect in my workload. So, I think that it is better to keep VM tunable using TBTC.

It may be a good idea to set 0 to default swap_token_timeout until we find the root cause.

Best regards,
Hideo AOKI

#9 lore raw
Quoted 20 lines · Hideo AOKI
Quoted 1 line · Nick Piggin

Andrew Morton wrote:

[snip]

Quoted 11 lines
Quoted 3 lines

Did anyone come up with a simple step-by-step procedure for reproducing the problem? It would be good if someone could do this, because I don't think we understand the root cause yet?

I admit to generally being in the same boat as you with respect to running complex userspace apps.

However, based on this and other scattered reports, I'd say it seems quite likely that token based thrashing control is the culprit. Based on the cost/benefit, I wonder if we should disable TBTC by default for 2.6.10, rather than trying to fix it, and try again for 2.6.11?

Quoted 8 lines

Hello,

I imagine that the issue might occur when only one process holds almost all memory and has swap token too long time.

However, TBTC has a good effect in my workload. So, I think that it is better to keep VM tunable using TBTC.

It may be a good idea to set 0 to default swap_token_timeout until we find the root cause.

Yes, with Con's patch to have TBTC turned off when swap_token_timeout is set to zero. It causes unacceptable regressions, so that is the best way to go.

It would be great to get it fixed, but I would be worried about putting in new patches for it now, right before 2.6.10.

#10 lore raw
In reply to Andrew Morton

Hello,

Quoted 4 lines

Did anyone come up with a simple step-by-step procedure for reproducing the
problem? It would be good if someone could do this, because I don't think we understand the root cause yet?

Here's a step-by-step explanation of the way I test this:

- Get the Memory Eater and compile it:

http://lkml.org/lkml/2004/12/13/272

- Do a clean boot

- Start top, and some app that has a clock and preferrably a CPU graph (to monitor screen freezes and CPU load; it's IceWM for me)

- Start the Memory Eater

- Give it an amount of megabytes that is more than the actual RAM size. I use a value of 300, as my computer has 256 M RAM.

- Enjoy :) "eatmemory" will slowly eat up more and more RAM (visible in top as RSS); under 2.6.8.1 no screen freezes come, and under 2.6.9 and 2.6.10-rc3 they do come; under 2.6.10-rc3 I also see high CPU periods for kswapd.

Yours, Mikhail Ramendik

#11 lore raw
Quoted 4 lines · On Mon, 20 Dec 2004 [email protected] wrote:

- Enjoy :) "eatmemory" will slowly eat up more and more RAM (visible in top as RSS); under 2.6.8.1 no screen freezes come, and under 2.6.9 and 2.6.10-rc3 they do come; under 2.6.10-rc3 I also see high CPU periods for kswapd.

The high cpu use for kswapd should be fixed by applying the vm-pageout-throttling.patch patch from -mm.

I'll also come up with a patch to not have the swap token used when the system is not under a swapin load...

Signature · 7 lines
He did not think of himself as a tourist; he was a traveler. The difference is
partly one of time, he would explain. Where as the tourist generally hurries
back home at the end of a few weeks or months, the traveler belonging no more
to one place than to the next, moves slowly, over periods of years, from one
part of the earth to another. Indeed, he would have found it difficult to tell,
among the many places he had lived, precisely where it was he had felt most at
home.  -- Paul Bowles