Skip to contentSkip to conversation

2.6.12 Performance problems

DISCUSSION Danial Thom 3 participants 9 messages Aug 22, 2005 – Aug 27, 2005 1 message not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

*confused by the top-posting..*

--- Luigi Genoni

Quoted 46 lines · <[email protected]> wrote:

maybe it is possible to be more clear.

voluntary kernel preemption adds explicit
preemption points into the
kernel and full kernel preemption makes all
kernel code preemptible. This
way even when a process is executing some
syscall in kernel space, it can
be volontary or involontary preempted.

For interactive users the systems seems to be
smarter, but when the system
is doing a lot of work in kernel space, then
you of course have to lose
something.

Also just to check id it's the case or not to
preempt means you lose
something. This something is usually troughput.
In your case I would not
use a preemptible kernel.

SOmething similar could be said about Timer
frequency, but here the lost
is connected to the number to interrupts you
have to manage.

The point is that a desktop where the users
simple need a smooth sysstem
to be userd interactivelly, but not real CPU
power, and a server where you
need hourse power are different topics and need
different kernel
behaviour.

Quoted 6 lines · Danial Thom

Ok, well you'll have to explain this one:

"Low latency comes at the cost of decreased
throughput - can't have both"

Quoted 11 lines
Quoted 1 line

Seems to be a bit backwards. Threading the

kernel

Quoted 1 line

adds latency, so its the additional latency

in the kernel that causes the

Quoted 1 line

drop in throughput. Do you mean that kernel

performance has been sacrificed

Quoted 1 line

in order to be able to service other threads

more quickly, even when there

Quoted 3 lines

are no other threads to be serviced?

Danial

The issue I have with that logic is that you seem to use "kernel" in a general sense without regard to what its doing. Dropping packets is always detrimental to the user regardless of what he's using the computer for. An audio stream that drops packets isn't going to be "smooth" at the user level.

All of this aside, I need to measure the raw capabilities of the kernel. With 'bsd OSes I can tell what the breaking point is by driving the machine to livelock. Linux seems to have a soft, floating capacity in that it will drop packets here and there for no isolatable reason. I'm having difficulty making a case for its use in a networking appliance, as dropped packets are not acceptable. How do I tune the "its ok to drop packets when x occurs" algorithm to be "its never ok to drop packets unless x occurs" (such as a queue depth)? Is it possible?

Danial

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#2 lore raw
Quoted 9 lines · Danial Thom

...

The issue I have with that logic is that you seem to use "kernel" in a general sense without regard to what its doing. Dropping packets is always detrimental to the user regardless of what he's using the computer for. An audio stream that drops packets isn't going to be "smooth" at the user level.

That's not always true.

Imagine a slow computer with a GBit ethernet connection, where the user is downloading files from a server that can utilize the full network connection while listening to music from his local disk with XMMS.

In this case, the audio stream is not depending on the network connection. And the user might prefer dropped packages over a stuttering XMMS.

Quoted 4 lines

All of this aside, I need to measure the raw capabilities of the kernel. With 'bsd OSes I can tell what the breaking point is by driving the machine to livelock. Linux seems to have a soft,

Quoted 8 lines

floating capacity in that it will drop packets here and there for no isolatable reason. I'm having difficulty making a case for its use in a networking appliance, as dropped packets are not acceptable. How do I tune the "its ok to drop packets when x occurs" algorithm to be "its never ok to drop packets unless x occurs" (such as a queue depth)? Is it possible?

What do you want to achieve with this thread?
1. Try to proof that Linux is inferior to *BSD?
2. Get help with your problem?

If your goal is 1. feel free to do so, but please do so on more appropriate lists.

If your goal is 2., you must help the people who are trying to help you with _your_ problem.

E.g. Patrick McHardy asked you:

<-- snip -->

In that case please send more information, like both 2.4 and 2.6 configs, dmesg output from booting, lspci output, other hardware details, ..

<-- snip -->

Crystal balls are rare, and if your goal is really to get problem solved you should send the information other people require for debugging your problem.

Quoted 1 line

Danial

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#3 lore raw
Quoted 27 lines · --- Adrian Bunk
Quoted 3 lines · Danial Thom

...

The issue I have with that logic is that you

seem

Quoted 1 line

to use "kernel" in a general sense without

regard

Quoted 2 lines

to what its doing. Dropping packets is always
detrimental to the user regardless of what

he's

Quoted 2 lines

using the computer for. An audio stream that
drops packets isn't going to be "smooth" at

the

Quoted 1 line

user level.

That's not always true.

Imagine a slow computer with a GBit ethernet connection, where the user
is downloading files from a server that can utilize the full
network connection while listening to music from his local disk with
XMMS.

Quoted 6 lines

In this case, the audio stream is not depending on the network
connection. And the user might prefer dropped packages over a stuttering
XMMS.

Audio connections are going to be windowed/flowed in some way (thats how the internet works) so you're not dealing with real capacity issues in that example. You're not going to overrun your ethernet adapter with an application; the issue is running applications on servers that are also doing a lot of other things. You can't isolate a particular application at the device level, so you drop packets randomly; not just ones you don't care about. If your application can't run well on your machine without dropping real traffic, then you need a faster machine, not an OS that compensates in a negative way.

DT

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#4 lore raw
Quoted 19 lines · Danial Thom
Quoted 11 lines · --- Adrian Bunk

That's not always true.

Imagine a slow computer with a GBit ethernet connection, where the user
is downloading files from a server that can utilize the full
network connection while listening to music from his local disk with
XMMS.

Quoted 5 lines

In this case, the audio stream is not depending on the network
connection. And the user might prefer dropped packages over a stuttering
XMMS.

Quoted 3 lines

Audio connections are going to be windowed/flowed
in some way (thats how the internet works) so
...

I was talking about an audio stream coming from a file on the "local disk", IOW something like an mp3 file.

But the most interesting thing about your email is not what you were answering to, but which part of my email you silently omitted. Since you are not answering questions that might help to debug the problem you claim to have, it seems your intention is not getting a Linux problem fixed...

Quoted 1 line

DT

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#5 lore raw
Quoted 33 lines · --- Adrian Bunk
Quoted 6 lines · Danial Thom
Quoted 4 lines · --- Adrian Bunk

That's not always true.

Imagine a slow computer with a GBit

ethernet

Quoted 8 lines
Quoted 8 lines

connection, where the user
is downloading files from a server that can
utilize the full
network connection while listening to music
from his local disk with
XMMS.

In this case, the audio stream is not

depending

Quoted 2 lines
Quoted 2 lines

on the network
connection. And the user might prefer

dropped

Quoted 4 lines
Quoted 2 lines

packages over a stuttering
XMMS.

Audio connections are going to be

windowed/flowed

Quoted 2 lines

in some way (thats how the internet works) so
...

I was talking about an audio stream coming from
a file on the
"local disk", IOW something like an mp3 file.

Quoted 9 lines

But the most interesting thing about your email
is not what you were
answering to, but which part of my email you
silently omitted. Since you
are not answering questions that might help to
debug the problem you
claim to have, it seems your intention is not
getting a Linux problem
fixed...

I don't think I'm obligated to answer every single person who pipes into a thread. People who say "show me your config and dmesg" are not useful. Linux has long had a philisophical problem of dropping packets as a "performance feature", and we've already established I think that you can't eliminate it altogether, if you read the thread carefully.

Also, if you're on a slow machine you can't download faster than your machine can handle it, because the "server on a gig network" can't send faster then you tell it to, not matter how fast it is. You'll never overrun a machine with 1 or 2 or 5 connections. Having a clue how things work is important.

DT

____________________________________________________
Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs

#6 lore raw
Quoted 10 lines · Danial Thom

...
I don't think I'm obligated to answer every single person who pipes into a thread. People who say "show me your config and dmesg" are not useful. Linux has long had a philisophical problem of dropping packets as a "performance feature", and we've already established I think that you can't eliminate it altogether, if you read the thread carefully.
...

You say you have observed for a long time a problem.

The only person participating in this thread who is one of the networking maintainers is Patrick McHardy.

Did _he_ say this was a known and unfixable problem?

He wanted to help you and you pissed him off because you refuxed to give him dmesg, .config and other information that would have helped him to debug your problem. If you don't feel obliged to help the persons responsible for the part of the kernel you have a problem with to debug your problem your whole initial mail was pointless.

Quoted 1 line

DT

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#7 lore raw
Quoted 19 lines · --- Adrian Bunk
Quoted 3 lines · Danial Thom

...
I don't think I'm obligated to answer every single person who pipes into a thread. People

who

Quoted 4 lines

say "show me your config and dmesg" are not useful. Linux has long had a philisophical problem of dropping packets as a "performance feature", and we've already established I

think

Quoted 1 line

that you can't eliminate it altogether, if

you

Quoted 2 lines

read the thread carefully.
...

You say you have observed for a long time a
problem.

Quoted 6 lines

The only person participating in this thread
who is one of the
networking maintainers is Patrick McHardy.

Did _he_ say this was a known and unfixable
problem?

I don't know. He said that the network stack has absolute priority, so perhaps he can explain why compiling a kernel causes an exponential increase in packet loss?

The FreeBSD "network maintainers" are largely clueless about just about every problem in 5.x, so I'm not sure the title qualifies someone as knowing all there is to know. I haven't heard him claim that he can do the things I'm doing without any receive errors or drops. I'd like to hear about what test he runs to test this sort of problem.

Before you can solve a problem you have to have a clear understanding of what the problem is, and admit that you have a problem.

Quoted 11 lines

He wanted to help you and you pissed him off
because you refuxed to give
him dmesg, .config and other information that
would have helped him to
debug your problem. If you don't feel obliged
to help the persons
responsible for the part of the kernel you have
a problem with to debug
your problem your whole initial mail was
pointless.

I didn't refuse. I just chose to take help from Ben, because Ben took the time to reproduce the problem and to provide useful settings that made sense to me. There's nothing wrong with my machine.

I don't know who is who; I judge people based on the intelligence of their analysis. People responsible for the network stack may not be the right people, because this is more likely a scheduler issue. There's probably nothing wrong with the stack or the drivers; the machine just doesn't react fast enough to avert ring overruns.

DT

__________________________________
Yahoo! Mail
Stay connected, organized, and protected. Take the tour:
http://tour.mail.yahoo.com/mailtour.html

#8 lore raw
Quoted 5 lines · Danial Thom

I didn't refuse. I just chose to take help from Ben, because Ben took the time to reproduce the problem and to provide useful settings that made sense to me. There's nothing wrong with my machine.

Well, I didn't see the slowdown on my system when I tried 2.6 v/s 2.4. You reported a 3x slowdown. In your original mail, you didn't mention trying to do compiles at the same time as your network test. I *did* see a pathetic slowdown (from 250kpps bridging to about 6kpps getting through the bridge) when I coppied a 512MB CDROM to the HD, but let's get the pure network slowdown on your system figured out before we start looking at the impact of disk IO.

So, if you see a 3x slowdown on an unloaded machine when going from 2.4 to 2.6, then there is something unique about your system and we should figure out what it is.

Also, my numbers (about 250kpps with moderate amount of drops) was a lot better than the 100kpps you reported for 2.6. My hardware is different (P-IV, HT, 3.0Ghz 1MB Cache, 1GB RAM, dual Intel pro/1000 NIC in PCI-X 66/133 slot), but I would not expect that to be 2x faster than your Opteron (or whatever you had).

You could mention what you are using to generate traffic. (I was using pktgen to generate a stream of 60 byte pkts.)

I think you should give us some better information on exactly what you are doing on 2.4 v/s 2.6. And for heaven's sake, don't hesitate to send kernel configs and hardware listings to folks that ask. Ruling out problems is as useful as finding problems in this sort of thing.

Thanks,
Ben

Signature · 2 lines
Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com
#9 lore raw
Quoted 49 lines · --- Ben Greear
Quoted 1 line · Danial Thom

I didn't refuse. I just chose to take help

from

Quoted 1 line

Ben, because Ben took the time to reproduce

the

Quoted 1 line

problem and to provide useful settings that

made

Quoted 2 lines

sense to me. There's nothing wrong with my
machine.

Well, I didn't see the slowdown on my system
when I tried 2.6
v/s 2.4. You reported a 3x slowdown. In your
original mail,
you didn't mention trying to do compiles at the
same time
as your network test. I *did* see a pathetic
slowdown (from 250kpps
bridging to about 6kpps getting through the
bridge) when I coppied
a 512MB CDROM to the HD, but let's get the pure
network slowdown
on your system figured out before we start
looking at the impact
of disk IO.

So, if you see a 3x slowdown on an unloaded
machine when going
from 2.4 to 2.6, then there is something unique
about your system
and we should figure out what it is.

Also, my numbers (about 250kpps with moderate
amount of drops)
was a lot better than the 100kpps you reported
for 2.6. My
hardware is different (P-IV, HT, 3.0Ghz 1MB
Cache, 1GB RAM,
dual Intel pro/1000 NIC in PCI-X 66/133 slot),
but I would not
expect that to be 2x faster than your Opteron
(or whatever you had).

You could mention what you are using to
generate traffic. (I was
using pktgen to generate a stream of 60 byte
pkts.)

Quoted 9 lines

I think you should give us some better
information on exactly
what you are doing on 2.4 v/s 2.6. And for
heaven's sake,
don't hesitate to send kernel configs and
hardware listings
to folks that ask. Ruling out problems is as
useful as finding
problems in this sort of thing.

Ok, good point, but I think the original issue has evolved into "why does linux drop packets even when its not close to capacity". Its not really a "slow down" now that we've been discussing it, its the point at which packet loss begins to occur. And since it appears that packet loss can occur at usage levels not even close to capacity, I can't really determine the capacity. Since CPU usage also doesn't seem to register correctly, I can't get a handle on the load for various activities. So it may not have anything to do with "slowness". I feel like I'm trying to weigh a big blob of Jell-o that keeps falling off the scale.

The bottom line is that if I can't get it to stop dropping packets, the performance is moot, because its simply not suitable for my needs.

DT

____________________________________________________
Start your day with Yahoo! - make it your home page
http://www.yahoo.com/r/hs