Skip to contentSkip to conversation

question about contest benchmark

DISCUSSION Haoqiang Zheng 4 participants 6 messages May 3, 2005

lore
Share & tools
lore
#0 lore raw

I am wondering how we should interpret the CONTEST benchmark results. I tried CONTEST with process_load on 2.6.12-rc3 (single CPU, P4 2.8G, 1G RAM). The CPU usage of kernel compiling is 28.9%, the load consumes 70.1% and the ratio is 3.98. Based on what Con says, the result is bad since the ratio is high. I did some tracing and found the background load (contest) runs at a dynamic priority of 115-120, which is often higher than the dynamic priority of the kernel compiling processes. This explains why the process_load consumes so much CPU.

My question is why is the result bad at all? One could certainly
argue that contest processes shouldn't consume so much CPU time since they are considered to be background jobs. But why is kernel compiling considered foreground jobs? Why making kernel compiling faster is better? Actually, I am wondering if CONTEST is an appropriate benchmark to report system responsiveness at all?

Any comments?

BTW, what benchmark do you guys use to test system responsiveness?

Haoqiang

#1 lore raw
Quoted 5 lines · Haoqiang Zheng

My question is why is the result bad at all? One could certainly
argue that contest processes shouldn't consume so much CPU time since they are considered to be background jobs. But why is kernel compiling considered foreground jobs? Why making kernel compiling faster is better?

I have reported the same problem before. CPU bound processes like kernel compiles will sometimes starve interactive processes like my mail client (Evolution).

Actually, I discovered a horrible performance bug in Evolution (suboptimal "hide junk" implementation, and searches for "" not properly optimized away, see evolution-hackers for a rough patch) that was responsible for the massive CPU suckage.

But, it seems to me that even if an interactive process briefly goes CPU bound (due to bloat, bugs, or intent), it should still be scheduled preferentially to a pure CPU bound process like a build.

Lee

#2 lore raw
Quoted 3 lines · Lee Revell

But, it seems to me that even if an interactive process briefly goes CPU bound (due to bloat, bugs, or intent), it should still be scheduled preferentially to a pure CPU bound process like a build.

So you want it to schedule that big image (Evolution) that's already used 5 minutes of CPU since it started (this morning, admittedly) in preference to that cc1 process that will be gone before it's used 2 seconds of CPU, plus all the disk I/O that cc1 performs (hopefully the cache will help here, but it may indeed go to disk to read the source files)?

#3 lore raw
Quoted 6 lines · On Tue, 2005-05-03 at 16:09 -0400, [email protected] wrote:
Quoted 3 lines · Lee Revell

But, it seems to me that even if an interactive process briefly goes CPU bound (due to bloat, bugs, or intent), it should still be scheduled preferentially to a pure CPU bound process like a build.

Quoted 6 lines

So you want it to schedule that big image (Evolution) that's already used 5 minutes of CPU since it started (this morning, admittedly) in preference to that cc1 process that will be gone before it's used 2 seconds of CPU, plus all the disk I/O that cc1 performs (hopefully the cache will help here, but it may indeed go to disk to read the source files)?

Yes. Almost no one will notice whether that build took 2 or 4 seconds. But a few seconds is an eternity when you are staring at a blank pane, waiting for it to render the message list. And I found that when waiting for the message list to render, backgrounding the build causes it to render in a second or two, while if I just wait for it it might take 20 seconds. This implies that the scheduler could do the same thing.

Anyway, this was not a great example, as the problem turned out to be an application bug. If I can find a non-pathological case that demonstrates a scheduler problem, I'll post it.

Lee

#4 lore raw
In reply to Haoqiang Zheng
Quoted 9 lines · Haoqiang Zheng

I am wondering how we should interpret the CONTEST benchmark results. I tried CONTEST with process_load on 2.6.12-rc3 (single CPU, P4 2.8G, 1G RAM). The CPU usage of kernel compiling is 28.9%, the load consumes 70.1% and the ratio is 3.98. Based on what Con says, the result is bad since the ratio is high. I did some tracing and found the background load (contest) runs at a dynamic priority of 115-120, which is often higher than the dynamic priority of the kernel compiling processes. This explains why the process_load consumes so much CPU.

Quoted 6 lines

My question is why is the result bad at all? One could certainly
argue that contest processes shouldn't consume so much CPU time since they are considered to be background jobs. But why is kernel compiling considered foreground jobs? Why making kernel compiling faster is better? Actually, I am wondering if CONTEST is an appropriate benchmark to report system responsiveness at all?

I don't think in my readme do I say anywhere what is the ideal balance. Process_load is a uniquely different load to the other loads which are various combinations of cpu and i/o. It spawns processes that wake up, hand their data off to another process and go to sleep. Thus the processes behave like interactive one with their frequent waiting, but share their effective group cpu usage amongst all the process_child processes running so none of them is actually seen as cpu bound. Furthermore there are massive numbers of context switches between them meaning there is a large in-kernel "system" load that is done on behalf of the process_child ren. The purpose of the process_load in contest is to ensure that an interactive design is not DoS'able by processes behaving like this. Process_load spawns 4 times as many processes as the timed 'make' in contest so theoretically ideal cpu balance between them should show process_load having 4x as much cpu as the make. Because their cpu binding is so intermittent it's hard to balance them perfectly. Anyway the balance in your output seems pretty good. When the interactive design goes horribly wrong process_load consumes 100 times as much cpu as the 'make'.

Quoted 4 lines

Any comments?

BTW, what benchmark do you guys use to test system responsiveness?

Note that interactivity is not responsiveness which some people try to measure with contest, and there is still no interactivity benchmark. Responsiveness is the ability of the system to continue performing tasks at a reasonable pace under various system loads. Interactivity is having low scheduling latency and jitter in those tasks where human interaction would notice the latency and jitter - and what constitutes and interactive tasks has not been quantified although we all know what they are when using the pc.

Cheers,
Con

#5 lore raw

Con,

Thanks so much for your response. The confusion was caused by what you said at http://members.optusnet.com.au/ckolivas/contest/:
"The lower the time (and ratio) the better, and the higher the cpu percentage the better. ".

Quoted 1 line

From http://kerneltrap.org/node/465, I also found: "

Time needs to be as low as possible (and therefore ratio as close to 1) CPU% needs to be as high as possible
Loads needs to be as high as possible
LCPU% needs to be as high as possible.
".

However, from what you described in the email, this doesn't sound to be true anymore. I think it will be hard to interpret the results if it's the "balance" that matters. For example, if CPU% is 40 in case A and 60 in case B. If lower is better, then A is better. If it's balance that matters, I would say A is about the same as B since in both case MAX/MIN = 1.5. (Suppose the total usage is 100%). So which answer is correct?

BTW, I like your clarification about interactivity vs. responsiveness.
But considering the work you have done on improving Linux
interactivity, there's no wonder that people think CONTEST is also an interactivity benchmark. :-)

Haoqiang

Quoted 53 lines · Con Kolivas
Quoted 9 lines · Haoqiang Zheng

I am wondering how we should interpret the CONTEST benchmark results. I tried CONTEST with process_load on 2.6.12-rc3 (single CPU, P4 2.8G, 1G RAM). The CPU usage of kernel compiling is 28.9%, the load consumes 70.1% and the ratio is 3.98. Based on what Con says, the result is bad since the ratio is high. I did some tracing and found the background load (contest) runs at a dynamic priority of 115-120, which is often higher than the dynamic priority of the kernel compiling processes. This explains why the process_load consumes so much CPU.

Quoted 6 lines

My question is why is the result bad at all? One could certainly
argue that contest processes shouldn't consume so much CPU time since they are considered to be background jobs. But why is kernel compiling considered foreground jobs? Why making kernel compiling faster is better? Actually, I am wondering if CONTEST is an appropriate benchmark to report system responsiveness at all?

I don't think in my readme do I say anywhere what is the ideal balance. Process_load is a uniquely different load to the other loads which are various combinations of cpu and i/o. It spawns processes that wake up, hand their data off to another process and go to sleep. Thus the processes behave like interactive one with their frequent waiting, but share their effective group cpu usage amongst all the process_child processes running so none of them is actually seen as cpu bound. Furthermore there are massive numbers of context switches between them meaning there is a large in-kernel "system" load that is done on behalf of the process_child ren. The purpose of the process_load in contest is to ensure that an interactive design is not DoS'able by processes behaving like this. Process_load spawns 4 times as many processes as the timed 'make' in contest so theoretically ideal cpu balance between them should show process_load having 4x as much cpu as the make. Because their cpu binding is so intermittent it's hard to balance them perfectly. Anyway the balance in your output seems pretty good. When the interactive design goes horribly wrong process_load consumes 100 times as much cpu as the 'make'.

Quoted 4 lines

Any comments?

BTW, what benchmark do you guys use to test system responsiveness?

Note that interactivity is not responsiveness which some people try to measure with contest, and there is still no interactivity benchmark. Responsiveness is the ability of the system to continue performing tasks at a reasonable pace under various system loads. Interactivity is having low scheduling latency and jitter in those tasks where human interaction would notice the latency and jitter - and what constitutes and interactive tasks has not been quantified although we all know what they are when using the pc.

Cheers,
Con