2.6.12 Performance problems
Quoted 48 lines · --- Jesper Juhl
Quoted 1 line · Jesper Juhl
On 8/21/05, Danial Thom
Quoted 7 lines · <[email protected]> wrote:
Quoted 7 lines
Quoted 1 line
Ok, well you'll have to explain this one:
"Low latency comes at the cost of decreased
throughput - can't have both"Seems to be a bit backwards. Threading the
kernel
Quoted 1 line
Quoted 1 line
adds latency, so its the additional latency
in
Quoted 1 line
Quoted 1 line
the kernel that causes the drop in
throughput. Do
Quoted 2 lines
Quoted 2 lines
you mean that kernel performance has been
sacrificed in order to be able to serviceother
Quoted 1 line
Quoted 1 line
threads more quickly, even when there are
no
Quoted 4 lines
Quoted 2 lines
other threads to be serviced?
Ok, let me start with the way HZ influences
things.
Quoted 1 line
[snip]
A small followup.
I'm not saying that the value of HZ or your
preempt setting (whatever
it may be) is definately the cause of your
problem. All I'm saying is
that it might be a contributing factor, so it's
probably something
that's worth a little bit of time testing.In many cases people running a server on
resonably new hardware with
HZ=1000 and full preempt won't even notice, but
that's depending on
the load on the server and what jobs it has.
For some tasks it
matters, for some the differences in
performance is negligible.
Quoted 17 lines
You problem could very well be something else
entirely, but try a
kernel build withPREEMPT_NONEand HZ=100 and
see if it makes a big
difference (or if that's your current config,
then try the opposite,
HZ=1000 and PREEMPT). If it does make a
difference, then that's a
valuable piece of information to report on the
list. If it turns out
it makes next to no difference at all, then
that as well is relevant
information as then people will know that HZ &
preempt is not the
cause and can focus on finding the problem
elsewhere.
Yes. Hz isn't going to make much difference on a 2.0Ghz opteron, but I can see how premption can cause packet loss. Shouldn't packet processing be the highest priority process? It seems pointless to "keep the audio buffers full" if you're dropping packets as a result.
Also some clown typing on the keyboard shouldn't cause packet loss. Trading network integrity for snappy responsiveness is a bad trade.
Danial
__________________________________
Yahoo! Mail for Mobile
Take Yahoo! Mail with you! Check email on your mobile phone.
http://mobile.yahoo.com/learn/mail
Quoted 76 lines · Danial Thom
Quoted 48 lines · --- Jesper Juhl
Quoted 1 line · Jesper Juhl
On 8/21/05, Danial Thom
Quoted 7 lines · <[email protected]> wrote:
Quoted 7 lines
Quoted 1 line
Ok, well you'll have to explain this one:
"Low latency comes at the cost of decreased
throughput - can't have both"Seems to be a bit backwards. Threading the
kernel
Quoted 1 line
Quoted 1 line
adds latency, so its the additional latency
in
Quoted 1 line
Quoted 1 line
the kernel that causes the drop in
throughput. Do
Quoted 2 lines
Quoted 2 lines
you mean that kernel performance has been
sacrificed in order to be able to serviceother
Quoted 1 line
Quoted 1 line
threads more quickly, even when there are
no
Quoted 4 lines
Quoted 2 lines
other threads to be serviced?
Ok, let me start with the way HZ influences
things.
Quoted 1 line
[snip]
A small followup.
I'm not saying that the value of HZ or your
preempt setting (whatever
it may be) is definately the cause of your
problem. All I'm saying is
that it might be a contributing factor, so it's
probably something
that's worth a little bit of time testing.In many cases people running a server on
resonably new hardware with
HZ=1000 and full preempt won't even notice, but
that's depending on
the load on the server and what jobs it has.
For some tasks it
matters, for some the differences in
performance is negligible.Quoted 17 lines
You problem could very well be something else
entirely, but try a
kernel build withPREEMPT_NONEand HZ=100 and
see if it makes a big
difference (or if that's your current config,
then try the opposite,
HZ=1000 and PREEMPT). If it does make a
difference, then that's a
valuable piece of information to report on the
list. If it turns out
it makes next to no difference at all, then
that as well is relevant
information as then people will know that HZ &
preempt is not the
cause and can focus on finding the problem
elsewhere.Yes. Hz isn't going to make much difference on a
2.0Ghz opteron, but I can see how premption can
cause packet loss. Shouldn't packet processing be
the highest priority process? It seems pointless
to "keep the audio buffers full" if you're
dropping packets as a result.
Quoted 4 lines
Also some clown typing on the keyboard shouldn't
cause packet loss. Trading network integrity for
snappy responsiveness is a bad trade.
Since you choose to reply to a public list on a private email I guess I should post the bit I'd [snip]'d above, so everyone else can get the whole picture :
Quoted 89 lines · Jesper Juhl
Quoted 6 lines · Danial Thom
Quoted 1 line
Ok, well you'll have to explain this one:
"Low latency comes at the cost of decreased
throughput - can't have both"Quoted 8 lines
Seems to be a bit backwards. Threading the kernel
adds latency, so its the additional latency in
the kernel that causes the drop in throughput. Do
you mean that kernel performance has been
sacrificed in order to be able to service other
threads more quickly, even when there are no
other threads to be serviced?Ok, let me start with the way HZ influences things.
The value of HZ was increased from 100 to 1000 in 2.6.x (and later made a config option so people can choose at compile time between 100, 250 & 1000).
This means that in 2.6 you have ten times as many timer interrupts happening every second compared to 2.4. This is great for things that need to be served on short notice, like audio buffers that need to be filled *now* before the music skips and can't wait 10ms bacause the user will notice. But, all those interrupts add overhead and thus the total amount of work that can be done every second will be slightly
lower even though response times are good. For a desktop HZ=1000 is
usually a very nice thing, but for a server that doesn't need good response times (low latency) HZ=100 or HZ=250 will provide better overall throughput and is often preferable.It's more or less the same thing with kernel preemption. When the kernel is fully preemtible it means that kernel functions can be interrupted at (almost) any time if there happens to be a higher priority process/thread that is runnable. When the kernel is not preemtible, system calls etc always run to completion before being interrupted.
Preemption is great for interactive applications where you want user interfaces to respond fast, want multimedia applications to be able to refill buffers at regular intervals (with great precision on the timing) etc. With preempt, such high applications will be able to interrupt/preempt other lower priority processes (like a background run of updatedb running from cron for example) even when those processes are in the middle of a system call, whereas without preemption they are forced to wait several milliseconds for the other apps syscall to complete before they can get a chance to run - which the user notices as sluggish gui response, music or video skips etc.
There are a few prices you pay for preemption. 1) extra overhead for bookkeeping (is it safe to preempt now, more locks need to be taken and dropped to make it safe etc). 2) preempted processes don't get to run for the entire duration of their timeslice in one go. 3) You get more context switch overhead - when the kernel has to switch between apps more often it needs to save and restore process context more often.
All of this hurts overall throughput but gives you nice interactive response times (low latency).So, you can't have both at once. You can't get maximum throughput at the same time as you get very low response times. It's one or the other depending on your need.
With 2.4 you have HZ=100, no preemption. So there you have a kernel optimized for maximum throughput, but latencies are high which is a pain for interactive and multimedia apps.
With 2.6, the default is HZ=1000 and preempt =
PREEMPT_NONEwhich is a compromise.
But luckily you can modify the default. As I already mentioned above, you can pick between HZ=100, 250 & 1000 these days, and the preemtion model gives you 3 choices :
PREEMPT_NONE- No Forced Preemption (Server)PREEMPT_VOLUNTARY- Voluntary Kernel Preemption (Desktop) PREEMPT - Preemptible Kernel (Low-Latency Desktop)So you are free to mix and match to get the kernel behaviour that suits you best for any given task.
Personally I run (most) my servers with HZ=100 and
PREEMPT_NONEand my personal desktop at home has HZ=1000 and PREEMPT.Did that make things a little more clear?
Signature · 3 lines
Jesper Juhl <[email protected]> Don't top-post http://www.catb.org/~esr/jargon/html/T/top-post.html Plain text mails only, please http://www.expita.com/nomime.html
Quoted 24 lines · Danial Thom
Quoted 8 lines
You problem could very well be something else
entirely, but try a
kernel build withPREEMPT_NONEand HZ=100 and
see if it makes a big
difference (or if that's your current config,
then try the opposite,
HZ=1000 and PREEMPT). If it does make a
difference, then that's aQuoted 8 lines
valuable piece of information to report on the
list. If it turns out
it makes next to no difference at all, then
that as well is relevant
information as then people will know that HZ &
preempt is not the
cause and can focus on finding the problem
elsewhere.Yes. Hz isn't going to make much difference on a 2.0Ghz opteron, but I can see how premption can cause packet loss. Shouldn't packet processing be the highest priority process? It seems pointless to "keep the audio buffers full" if you're dropping packets as a result.
Quoted 3 lines
Also some clown typing on the keyboard shouldn't cause packet loss. Trading network integrity for snappy responsiveness is a bad trade.
You do not need to argue about usefulness of preempt (or lack thereof). You need to try non-PREEMPT kernel as suggested (if you really are interested in fixing performance degradation you observe, that is).