Skip to contentSkip to conversation

Unusually high system CPU usage with recent kernels

DISCUSSION Tibor Billes 1 participant 1 message Sep 28, 2013

lore
Share & tools
lore
#0 lore raw

Hi Paul,

I was just wondering if you received my last mail, because I haven't heard from you for a while now.

Tibor

Quoted 94 lines

----- Original Message -----
From: Tibor Billes
Sent: 09/14/13 03:59 PM
To: [email protected]
Subject: Re: Unusually high system CPU usage with recent kernels

Quoted 62 lines

From: Paul E. McKenney Sent: 09/13/13 02:19 AM

Quoted 52 lines · Tibor Billes
Quoted 52 lines

From: Paul E. McKenney Sent: 09/09/13 10:44 PM

Quoted 10 lines · Tibor Billes
Quoted 8 lines

From: Paul E. McKenney Sent: 09/08/13 08:43 PM

Quoted 2 lines · Tibor Billes

Good news Paul, the above patch did solve this issue :) I see no extra context switches, no extra CPU usage and no extra compile time.

Woo-hoo!!! ;-)

May I add your Tested-by to the fix?

Yes, please :)

Done! ;-)

Quoted 23 lines
Quoted 6 lines
Quoted 1 line

Any idea why couldn't you reproduce this? Why did it only hit my system?

Timing, maybe? Another question is "Why didn't lots of people complain about this?" It would be good to find out because it is quite possible that there is some other bug that this patch is masking -- or even just making less probable.

Good point!

Quoted 10 lines

If you are interested, please build one of the old kernels but with CONFIG_RCU_TRACE=y. Then run something like the following as root concurrently with your workload:

sleep 10
echo 1 > /sys/kernel/debug/tracing/events/rcu/enable sleep 0.01
echo 0 > /sys/kernel/debug/tracing/events/rcu/enable cat /sys/kernel/debug/tracing/trace > /tmp/trace

Quoted 3 lines

Send me the /tmp/trace file, which will probably be a few megabytes in size, so please compress it before sending. ;-) A good compression tool should be able to shrink it by a factor of 20 or thereabouts.

Quoted 8 lines

Ok, I did that. Twice! The first is with commit 910ee45db2f4837c8440e770474758493ab94bf7, which was the first bad commit according to the bisection I did initially. Second with the current mainline 3.11. I have little idea of what the fields and lines mean in the RCU trace files, so I'm not going to guess if they are essentially the same or not, but it may provide more information to you. Both files were created by using a shell script containing the commands you suggested.

So traces both correspond to bad cases, correct? They are both quite impressive -- looks like you have quite the interrupt rate going on there! Almost looks like interrupts are somehow getting enabled on the path to/from idle.

Quoted 7 lines
Quoted 1 line

Could you please also send along a trace with the fix applied?

Sure. The attached tar file contains traces of good kernels. The first is with version 3.9.7 (no patch applied) which was the last stable kernel I tried and didn't have this issue. The second is version 3.11 with your fix applied. Judging by the size of the traces, 3.11.0+ is still doing more work than 3.9.7.

Quoted 4 lines

Indeed, though quite a bit less than the problematic traces.

Did you have all three patches applied to 3.11.0, or just the last one? If the latter, could you please try it with all three?

Only the last one was applied to 3.11.0. The attachement now contains the RCU trace with all thee applied. It seems to be smaller in size, but still not close to 3.9.7.

Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines

I'm not sure about LKML policies about attaching not-so-small files to emails, so I've dropped LKML from the CC list. Please CC the mailing list in your reply.

Done!

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

Another approach is to post the traces on the web and send the URL to LKML. But whatever works for you is fine by me.

Sending directly to you won again :) Could you please CC the list in your reply?

Done! ;-)

Could you please CC the list in your reply again? :)

Tibor