2.6.12 Performance problems
A message that is not in the archive (sent off-list or not yet received): [email protected]
*confused by the top-posting..*
--- Luigi Genoni
Quoted 46 lines · <[email protected]> wrote:
maybe it is possible to be more clear.
voluntary kernel preemption adds explicit
preemption points into the
kernel and full kernel preemption makes all
kernel code preemptible. This
way even when a process is executing some
syscall in kernel space, it can
be volontary or involontary preempted.For interactive users the systems seems to be
smarter, but when the system
is doing a lot of work in kernel space, then
you of course have to lose
something.Also just to check id it's the case or not to
preempt means you lose
something. This something is usually troughput.
In your case I would not
use a preemptible kernel.SOmething similar could be said about Timer
frequency, but here the lost
is connected to the number to interrupts you
have to manage.The point is that a desktop where the users
simple need a smooth sysstem
to be userd interactivelly, but not real CPU
power, and a server where you
need hourse power are different topics and need
different kernel
behaviour.Quoted 6 lines · Danial Thom
Ok, well you'll have to explain this one:
"Low latency comes at the cost of decreased
throughput - can't have both"
Quoted 11 lines
Quoted 1 line
Seems to be a bit backwards. Threading the
kernel
Quoted 1 line
adds latency, so its the additional latency
in the kernel that causes the
Quoted 1 line
drop in throughput. Do you mean that kernel
performance has been sacrificed
Quoted 1 line
in order to be able to service other threads
more quickly, even when there
Quoted 3 lines
are no other threads to be serviced?
Danial
The issue I have with that logic is that you seem to use "kernel" in a general sense without regard to what its doing. Dropping packets is always detrimental to the user regardless of what he's using the computer for. An audio stream that drops packets isn't going to be "smooth" at the user level.
All of this aside, I need to measure the raw capabilities of the kernel. With 'bsd OSes I can tell what the breaking point is by driving the machine to livelock. Linux seems to have a soft, floating capacity in that it will drop packets here and there for no isolatable reason. I'm having difficulty making a case for its use in a networking appliance, as dropped packets are not acceptable. How do I tune the "its ok to drop packets when x occurs" algorithm to be "its never ok to drop packets unless x occurs" (such as a queue depth)? Is it possible?
Danial
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Quoted 9 lines · Danial Thom
...
The issue I have with that logic is that you seem to use "kernel" in a general sense without regard to what its doing. Dropping packets is always detrimental to the user regardless of what he's using the computer for. An audio stream that drops packets isn't going to be "smooth" at the user level.
That's not always true.
Imagine a slow computer with a GBit ethernet connection, where the user is downloading files from a server that can utilize the full network connection while listening to music from his local disk with XMMS.
In this case, the audio stream is not depending on the network connection. And the user might prefer dropped packages over a stuttering XMMS.
Quoted 4 lines
All of this aside, I need to measure the raw capabilities of the kernel. With 'bsd OSes I can tell what the breaking point is by driving the machine to livelock. Linux seems to have a soft,
Quoted 8 lines
floating capacity in that it will drop packets here and there for no isolatable reason. I'm having difficulty making a case for its use in a networking appliance, as dropped packets are not acceptable. How do I tune the "its ok to drop packets when x occurs" algorithm to be "its never ok to drop packets unless x occurs" (such as a queue depth)? Is it possible?
What do you want to achieve with this thread?
1. Try to proof that Linux is inferior to *BSD?
2. Get help with your problem?
If your goal is 1. feel free to do so, but please do so on more appropriate lists.
If your goal is 2., you must help the people who are trying to help you with _your_ problem.
E.g. Patrick McHardy asked you:
<-- snip -->
In that case please send more information, like both 2.4 and 2.6 configs, dmesg output from booting, lspci output, other hardware details, ..
<-- snip -->
Crystal balls are rare, and if your goal is really to get problem solved you should send the information other people require for debugging your problem.
Quoted 1 line
Danial
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 27 lines · --- Adrian Bunk
Quoted 3 lines · Danial Thom
...
The issue I have with that logic is that you
seem
Quoted 1 line
to use "kernel" in a general sense without
regard
Quoted 2 lines
to what its doing. Dropping packets is always
detrimental to the user regardless of whathe's
Quoted 2 lines
using the computer for. An audio stream that
drops packets isn't going to be "smooth" atthe
Quoted 1 line
user level.
That's not always true.
Imagine a slow computer with a GBit ethernet connection, where the user
is downloading files from a server that can utilize the full
network connection while listening to music from his local disk with
XMMS.
Quoted 6 lines
In this case, the audio stream is not depending on the network
connection. And the user might prefer dropped packages over a stuttering
XMMS.
Audio connections are going to be windowed/flowed in some way (thats how the internet works) so you're not dealing with real capacity issues in that example. You're not going to overrun your ethernet adapter with an application; the issue is running applications on servers that are also doing a lot of other things. You can't isolate a particular application at the device level, so you drop packets randomly; not just ones you don't care about. If your application can't run well on your machine without dropping real traffic, then you need a faster machine, not an OS that compensates in a negative way.
DT
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
Quoted 19 lines · Danial Thom
Quoted 11 lines · --- Adrian Bunk
That's not always true.
Imagine a slow computer with a GBit ethernet connection, where the user
is downloading files from a server that can utilize the full
network connection while listening to music from his local disk with
XMMS.Quoted 5 lines
In this case, the audio stream is not depending on the network
connection. And the user might prefer dropped packages over a stuttering
XMMS.
Quoted 3 lines
Audio connections are going to be windowed/flowed
in some way (thats how the internet works) so
...
I was talking about an audio stream coming from a file on the "local disk", IOW something like an mp3 file.
But the most interesting thing about your email is not what you were answering to, but which part of my email you silently omitted. Since you are not answering questions that might help to debug the problem you claim to have, it seems your intention is not getting a Linux problem fixed...
Quoted 1 line
DT
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 33 lines · --- Adrian Bunk
Quoted 6 lines · Danial Thom
Quoted 4 lines · --- Adrian Bunk
That's not always true.
Imagine a slow computer with a GBit
ethernet
Quoted 8 lines
Quoted 8 lines
connection, where the user
is downloading files from a server that can
utilize the full
network connection while listening to music
from his local disk with
XMMS.In this case, the audio stream is not
depending
Quoted 2 lines
Quoted 2 lines
on the network
connection. And the user might preferdropped
Quoted 4 lines
Quoted 2 lines
packages over a stuttering
XMMS.Audio connections are going to be
windowed/flowed
Quoted 2 lines
in some way (thats how the internet works) so
...I was talking about an audio stream coming from
a file on the
"local disk", IOW something like an mp3 file.
Quoted 9 lines
But the most interesting thing about your email
is not what you were
answering to, but which part of my email you
silently omitted. Since you
are not answering questions that might help to
debug the problem you
claim to have, it seems your intention is not
getting a Linux problem
fixed...
I don't think I'm obligated to answer every single person who pipes into a thread. People who say "show me your config and dmesg" are not useful. Linux has long had a philisophical problem of dropping packets as a "performance feature", and we've already established I think that you can't eliminate it altogether, if you read the thread carefully.
Also, if you're on a slow machine you can't download faster than your machine can handle it, because the "server on a gig network" can't send faster then you tell it to, not matter how fast it is. You'll never overrun a machine with 1 or 2 or 5 connections. Having a clue how things work is important.
DT
____________________________________________________
Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs
Quoted 10 lines · Danial Thom
...
I don't think I'm obligated to answer every single person who pipes into a thread. People who say "show me your config and dmesg" are not useful. Linux has long had a philisophical problem of dropping packets as a "performance feature", and we've already established I think that you can't eliminate it altogether, if you read the thread carefully.
...
You say you have observed for a long time a problem.
The only person participating in this thread who is one of the networking maintainers is Patrick McHardy.
Did _he_ say this was a known and unfixable problem?
He wanted to help you and you pissed him off because you refuxed to give him dmesg, .config and other information that would have helped him to debug your problem. If you don't feel obliged to help the persons responsible for the part of the kernel you have a problem with to debug your problem your whole initial mail was pointless.
Quoted 1 line
DT
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 19 lines · --- Adrian Bunk
Quoted 3 lines · Danial Thom
...
I don't think I'm obligated to answer every single person who pipes into a thread. Peoplewho
Quoted 4 lines
say "show me your config and dmesg" are not useful. Linux has long had a philisophical problem of dropping packets as a "performance feature", and we've already established I
think
Quoted 1 line
that you can't eliminate it altogether, if
you
Quoted 2 lines
read the thread carefully.
...You say you have observed for a long time a
problem.
Quoted 6 lines
The only person participating in this thread
who is one of the
networking maintainers is Patrick McHardy.Did _he_ say this was a known and unfixable
problem?
I don't know. He said that the network stack has absolute priority, so perhaps he can explain why compiling a kernel causes an exponential increase in packet loss?
The FreeBSD "network maintainers" are largely clueless about just about every problem in 5.x, so I'm not sure the title qualifies someone as knowing all there is to know. I haven't heard him claim that he can do the things I'm doing without any receive errors or drops. I'd like to hear about what test he runs to test this sort of problem.
Before you can solve a problem you have to have a clear understanding of what the problem is, and admit that you have a problem.
Quoted 11 lines
He wanted to help you and you pissed him off
because you refuxed to give
him dmesg, .config and other information that
would have helped him to
debug your problem. If you don't feel obliged
to help the persons
responsible for the part of the kernel you have
a problem with to debug
your problem your whole initial mail was
pointless.
I didn't refuse. I just chose to take help from Ben, because Ben took the time to reproduce the problem and to provide useful settings that made sense to me. There's nothing wrong with my machine.
I don't know who is who; I judge people based on the intelligence of their analysis. People responsible for the network stack may not be the right people, because this is more likely a scheduler issue. There's probably nothing wrong with the stack or the drivers; the machine just doesn't react fast enough to avert ring overruns.
DT
__________________________________
Yahoo! Mail
Stay connected, organized, and protected. Take the tour:
http://tour.mail.yahoo.com/mailtour.html
Quoted 5 lines · Danial Thom
I didn't refuse. I just chose to take help from Ben, because Ben took the time to reproduce the problem and to provide useful settings that made sense to me. There's nothing wrong with my machine.
Well, I didn't see the slowdown on my system when I tried 2.6 v/s 2.4. You reported a 3x slowdown. In your original mail, you didn't mention trying to do compiles at the same time as your network test. I *did* see a pathetic slowdown (from 250kpps bridging to about 6kpps getting through the bridge) when I coppied a 512MB CDROM to the HD, but let's get the pure network slowdown on your system figured out before we start looking at the impact of disk IO.
So, if you see a 3x slowdown on an unloaded machine when going from 2.4 to 2.6, then there is something unique about your system and we should figure out what it is.
Also, my numbers (about 250kpps with moderate amount of drops) was a lot better than the 100kpps you reported for 2.6. My hardware is different (P-IV, HT, 3.0Ghz 1MB Cache, 1GB RAM, dual Intel pro/1000 NIC in PCI-X 66/133 slot), but I would not expect that to be 2x faster than your Opteron (or whatever you had).
You could mention what you are using to generate traffic. (I was using pktgen to generate a stream of 60 byte pkts.)
I think you should give us some better information on exactly what you are doing on 2.4 v/s 2.6. And for heaven's sake, don't hesitate to send kernel configs and hardware listings to folks that ask. Ruling out problems is as useful as finding problems in this sort of thing.
Thanks,
Ben
Signature · 2 lines
Ben Greear <[email protected]> Candela Technologies Inc http://www.candelatech.com
Quoted 49 lines · --- Ben Greear
Quoted 1 line · Danial Thom
I didn't refuse. I just chose to take help
from
Quoted 1 line
Ben, because Ben took the time to reproduce
the
Quoted 1 line
problem and to provide useful settings that
made
Quoted 2 lines
sense to me. There's nothing wrong with my
machine.Well, I didn't see the slowdown on my system
when I tried 2.6
v/s 2.4. You reported a 3x slowdown. In your
original mail,
you didn't mention trying to do compiles at the
same time
as your network test. I *did* see a pathetic
slowdown (from 250kpps
bridging to about 6kpps getting through the
bridge) when I coppied
a 512MB CDROM to the HD, but let's get the pure
network slowdown
on your system figured out before we start
looking at the impact
of disk IO.So, if you see a 3x slowdown on an unloaded
machine when going
from 2.4 to 2.6, then there is something unique
about your system
and we should figure out what it is.Also, my numbers (about 250kpps with moderate
amount of drops)
was a lot better than the 100kpps you reported
for 2.6. My
hardware is different (P-IV, HT, 3.0Ghz 1MB
Cache, 1GB RAM,
dual Intel pro/1000 NIC in PCI-X 66/133 slot),
but I would not
expect that to be 2x faster than your Opteron
(or whatever you had).You could mention what you are using to
generate traffic. (I was
using pktgen to generate a stream of 60 byte
pkts.)
Quoted 9 lines
I think you should give us some better
information on exactly
what you are doing on 2.4 v/s 2.6. And for
heaven's sake,
don't hesitate to send kernel configs and
hardware listings
to folks that ask. Ruling out problems is as
useful as finding
problems in this sort of thing.
Ok, good point, but I think the original issue has evolved into "why does linux drop packets even when its not close to capacity". Its not really a "slow down" now that we've been discussing it, its the point at which packet loss begins to occur. And since it appears that packet loss can occur at usage levels not even close to capacity, I can't really determine the capacity. Since CPU usage also doesn't seem to register correctly, I can't get a handle on the load for various activities. So it may not have anything to do with "slowness". I feel like I'm trying to weigh a big blob of Jell-o that keeps falling off the scale.
The bottom line is that if I can't get it to stop dropping packets, the performance is moot, because its simply not suitable for my needs.
DT
____________________________________________________
Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs