Skip to contentSkip to conversation

2.6.10-rc3: kswapd eats CPU on start of memory-eating task

DISCUSSION Voluspa 7 participants 12 messages Dec 19, 2004 – Dec 20, 2004

Subsystems (2)
lore
Share & tools
lore

Voluspa

author
#0 lore raw

Found the first kernel version with the regression. It's linux-2.6.9-rc1

Perusing lkml from august there was a short thread about the oopses and loss of keyboard in X. Applying that information in a crude hack I was able to test the effected 2.6.9-rc1 and three -bk forward:

http://marc.theaimsgroup.com/?t=109357291300002&r=1&w=2

diff -Naur linux-2.6.9-rc1/net/sunrpc/svcauth_unix.c linux-2.6.9-rc1-debug/net/sunrpc/svcauth_unix.c
--- linux-2.6.9-rc1/net/sunrpc/svcauth_unix.c 2004-12-15 18:39:28.000000000
+0100
+++ linux-2.6.9-rc1-debug/net/sunrpc/svcauth_unix.c 2004-12-19 19:01:
53.000000000 +0100
@@ -104,7 +104,6 @@

                if (test_bit(CACHE_VALID, &item->flags) &&
                    !test_bit(CACHE_NEGATIVE, &item->flags))
                        auth_domain_put(&im->m_client->h);
-               kfree(im->m_class);
                kfree(im);
        }

}

I've since tested and retested for several hours on the different kernels. At one point I thought the usage of lapic=lapic made a difference, but it turned out to be a red herring.

2.6.8.1-bk2 is without doubt the last kernel to handle my testcase "properly". There are no freezes whatsoever and the swapping is finished within 1 minute and some seconds.

2.6.9-rc1 and forward all have the freezes. Swapping and readback takes from 3 to 6 minutes. I can't find a pattern in the time differences.

What's left now is to find some repository which has the gargantuan 2.6.9-rc1 broken out in its pieces (and I guess 2.6.8.1-bk1 and 2 must be subtracted from that). Then reverting patches. A process where I'd need some handholding as to what would be likely candidates.

An innocent one is Ingo's "context-switching overhead in X, ioport()" patch. I added it to 2.6.8.1-bk2 and it didn't break my testcase.

Ah, well. It's that time of the year, so I won't be able to do any testing until the madness is over.

Mvh
Mats Johannesson

#1 lore raw
Quoted 2 lines · Voluspa

Found the first kernel version with the regression. It's linux-2.6.9-rc1

Thanks!

"[PATCH] token based thrashing control" would be a prime suspect.

None of my infamous VM patches (which did cause random problems) had gone into 2.6.9-rc1. The first ones were in 2.6.9-rc2.

Well, "[PATCH] make shrinker_sem an rwsem" was in -rc1; I guess that would be worthwhile testing, if only because it touches vmscan.c

It would be nice to find out what is going on before 2.6.10 gets released, but Mats isn't going to be able to do any more testing for the moment. Andrew, what should we do?

#2 lore raw
Quoted 3 lines · Nick Piggin

It would be nice to find out what is going on before 2.6.10 gets released, but Mats isn't going to be able to do any more testing for the moment. Andrew, what should we do?

I am ready to do the testing with the memory eater, and with "a complie in the background plus the memory eater".

I'm not as good at kernel code management as Mats and did not do the regression tests on -bk and past -rc versions, nor can I duplicate the X hack on 2.6.9-rc1. But if you give me a patch against any numbered version (2.6.8.1, 2.6.9) or against 2.6.10-rc3 , I'll gladly test it, with probably no more than 24 h response time ;)

Signature · 1 line
Yours, Mikhail Ramendik
#3 lore raw
In reply to Nick Piggin
Quoted 2 lines · Nick Piggin

Andrew, what should we do?

- Ask Voluspa to do

echo 0 > /proc/sys/vm/swap_token_timeout

on 2.6.10-rc3 and retest.

- Dig out Rik's token-timeout-autotuning patch, make it apply, test it, then ask Volupsa and others to test that.

Have you time to look into the latter?

(We still don't know why it chews tons of CPU, do we?)

#4 lore raw
Quoted 5 lines · Andrew Morton

- Ask Voluspa to do

echo 0 > /proc/sys/vm/swap_token_timeout

on 2.6.10-rc3 and retest.

He did, and I did (but I have not sent my report to lkml). In both cases, screen freezes remained but were now less in duration (up to 10-20 sec). In mu case I also monitored CPU loading and the big load peaks were there (the biggest one was in the beginning).

Quoted 1 line

(We still don't know why it chews tons of CPU, do we?)

It does! Any way to dig into this?

Signature · 1 line
Yours, Mikhail Ramendik
#5 lore raw
Quoted 10 lines · Mikhail Ramendik
Quoted 5 lines · Andrew Morton

- Ask Voluspa to do

echo 0 > /proc/sys/vm/swap_token_timeout

on 2.6.10-rc3 and retest.

Quoted 8 lines

He did, and I did (but I have not sent my report to lkml). In both cases, screen freezes remained but were now less in duration (up to 10-20 sec). In mu case I also monitored CPU loading and the big load peaks were there (the biggest one was in the beginning).

Quoted 1 line

(We still don't know why it chews tons of CPU, do we?)

It does! Any way to dig into this?

I still suspect the thrash token patch even with the swap token timeout at 0. Is it completely disabled at 0 or does it still do something?

Con

#6 lore raw
Quoted 1 line · Con Kolivas

I still suspect the thrash token patch even with the swap token timeout at 0. Is it completely disabled at 0 or does it still do something?

It makes it harder to page out pages from the task holding the token. I wonder if kswapd should try to steal the token away from the task holding it, so in effect nobody holds the token when the system isn't under a heavy swapping load.

Signature · 7 lines
He did not think of himself as a tourist; he was a traveler. The difference is
partly one of time, he would explain. Where as the tourist generally hurries
back home at the end of a few weeks or months, the traveler belonging no more
to one place than to the next, moves slowly, over periods of years, from one
part of the earth to another. Indeed, he would have found it difficult to tell,
among the many places he had lived, precisely where it was he had felt most at
home.  -- Paul Bowles
#7 lore raw
Quoted 10 lines · Rik van Riel
Quoted 1 line · Con Kolivas

I still suspect the thrash token patch even with the swap token timeout at 0. Is it completely disabled at 0 or does it still do something?

It makes it harder to page out pages from the task holding the token. I wonder if kswapd should try to steal the token away from the task holding it, so in effect nobody holds the token when the system isn't under a heavy swapping load.

What if the token isn't handed out at all until a heavy swapping load starts? A slight delay in thrash control would be worth it.

Con

#8 lore raw
Quoted 1 line · Con Kolivas

What if the token isn't handed out at all until a heavy swapping load starts? A slight delay in thrash control would be worth it.

How do you define "heavy swapping" ?

How would you measure it ?

How would you relinquish the token after the "heavy swapping" load stopped ?

Signature · 7 lines
He did not think of himself as a tourist; he was a traveler. The difference is
partly one of time, he would explain. Where as the tourist generally hurries
back home at the end of a few weeks or months, the traveler belonging no more
to one place than to the next, moves slowly, over periods of years, from one
part of the earth to another. Indeed, he would have found it difficult to tell,
among the many places he had lived, precisely where it was he had felt most at
home.  -- Paul Bowles
#9 lore raw
Quoted 5 lines · Rik van Riel
Quoted 1 line · Con Kolivas

What if the token isn't handed out at all until a heavy swapping load starts? A slight delay in thrash control would be worth it.

Quoted 7 lines

How do you define "heavy swapping" ?

How would you measure it ?

How would you relinquish the token after the "heavy swapping" load stopped ?

N F I

Cheers,
Con

#10 lore raw
In reply to Rik van Riel
Quoted 10 lines · Rik van Riel
Quoted 2 lines · Con Kolivas

I still suspect the thrash token patch even with the swap token timeout at 0. Is it completely disabled at 0 or does it still do something?

It makes it harder to page out pages from the task holding the token. I wonder if kswapd should try to steal the token away from the task holding it, so in effect nobody holds the token when the system isn't under a heavy swapping load.

In that case, the first thing we need to do is disable thrash token completely, and retest that. We still don't know for sure that it is the problem.

I don't have the code in front of me at the moment, but I'll be able to send a patch to do that in a couple of hours, if nobody beats me to it.

Nick

#11 lore raw
Quoted 13 lines · Nick Piggin
Quoted 10 lines · Rik van Riel
Quoted 2 lines · Con Kolivas

I still suspect the thrash token patch even with the swap token timeout at 0. Is it completely disabled at 0 or does it still do something?

It makes it harder to page out pages from the task holding the token. I wonder if kswapd should try to steal the token away from the task holding it, so in effect nobody holds the token when the system isn't under a heavy swapping load.

Quoted 7 lines

In that case, the first thing we need to do is disable thrash token completely, and retest that. We still don't know for sure that it is the problem.

I don't have the code in front of me at the moment, but I'll be able to send a patch to do that in a couple of hours, if nobody beats me to it.

This should disable the thrashing control code?

mm/rmap.c renamed from mm/rmap.c~a +2 -0
@@ -395,6 +395,8 @@ int page_referenced(struct page *page, i
 {
 	int referenced = 0;
 
+	ignore_token = 1;
+
 	if (page_test_and_clear_young(page))
 		referenced++;
 

_