userspace irq balancer
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 10 lines · Badari Pulavarty
---------- Forwarded Message ----------
Subject: Re: 2.5 closeout
Date: Mon, 19 May 2003 11:40:05 -0700
From: Andrew Morton <[email protected]>
To: Badari Pulavarty <[email protected]>The attribution wasn't very accurate. That's me, Arjan, perhaps Jeff Garzik.
Quoted 6 lines
The problem is that kird does the wrong thing for some workloads (packet forwarding mainly - routing). We'll never get that right, so we'd like to deprecate the in-kernel IRQ balancer and merge Arjan's user-space balancer into the kernel tree instead.
The only thing I'm concerned about is how it's going to be packaged. I'm envisioning explaining how to get the daemon out of its initrd image, set it up and run it, especially before distros have it integrated. The stuff that's in the kernel now isn't horribly broken; it's just not optimal for some relatively unusual cases.
Can we leave the current code as-is, and make the added intelligence from the userspace one an optional thing for those with unusal setups?
Signature · 2 lines
Dave Hansen [email protected]
Quoted 5 lines · Dave Hansen
The only thing I'm concerned about is how it's going to be packaged. I'm envisioning explaining how to get the daemon out of its initrd image, set it up and run it, especially before distros have it integrated. The stuff that's in the kernel now isn't horribly broken; it's just not optimal for some relatively unusual cases.
as for distros: RHL8 and later ship with it on the RH side (default enabled as of RHL9).
As for where to start it: I really think an initscript is the logical place; there has been some discussion about doing it
from the initramfs but I don't see real benifit from that; from starting init to running the initscripts isn't exactly THIS interrupt/performance heavy.
Quoted 10 lines · Arjan van de Ven
Quoted 5 lines · Dave Hansen
The only thing I'm concerned about is how it's going to be packaged. I'm envisioning explaining how to get the daemon out of its initrd image, set it up and run it, especially before distros have it integrated. The stuff that's in the kernel now isn't horribly broken; it's just not optimal for some relatively unusual cases.
as for distros: RHL8 and later ship with it on the RH side (default enabled as of RHL9).
But, do you see the need for ripping out the current code? For those of us that are still running a slightly more primitive distro, it would be nice to have some pretty effective default behavior, like what is in the kernel now.
Quoted 5 lines
As for where to start it: I really think an initscript is the logical place; there has been some discussion about doing it
from the initramfs but I don't see real benifit from that; from starting init to running the initscripts isn't exactly THIS interrupt/performance heavy.
Yeah, I don't think we need it the second the kernel boots :) Do you really think this is a 2.6 showstopper? Since it will require distro cooperation anyway, and those are many months from releasing a 2.6 distro, do we really need it in place for 2.6.0?
Signature · 2 lines
Dave Hansen [email protected]
Quoted 7 lines · Dave Hansen
Quoted 2 lines
as for distros: RHL8 and later ship with it on the RH side (default enabled as of RHL9).
But, do you see the need for ripping out the current code? For those of us that are still running a slightly more primitive distro, it would be nice to have some pretty effective default behavior, like what is in the kernel now.
You have to install new modutils to even use modules with the 2.5.x kernel, given that why are we even talking about the "inconvenience" of installing the usermode IRQ balancer as being a blocker for ripping out the in-kernel stuff?
The in-kernel stuff MUST go. It went in because "some benchmark went faster", but with no "why" describing why it might have improved performance. We KNOW it absolutely sucks for routing and firewall applications. The in-kernel bits were all a shamans dance, with zero technical "here is why this makes things go faster" description attached. If I remember properly, the changelog message when the in-kernel irq balancing went in was of the form "this makes some specweb run go faster".
Signature · 1 line
David S. Miller <[email protected]>
Quoted 8 lines · David S. Miller
The in-kernel stuff MUST go. It went in because "some benchmark went faster", but with no "why" describing why it might have improved performance. We KNOW it absolutely sucks for routing and firewall applications. The in-kernel bits were all a shamans dance, with zero technical "here is why this makes things go faster" description attached. If I remember properly, the changelog message when the in-kernel irq balancing went in was of the form "this makes some specweb run go faster".
Absolutely. Not to mention the code for the in-kernel algorithm has historically broken i386 ports using certain modes of Intel's interrupt controllers.
Far better would be to validate that the affinity specified is feasible to program into the interrupt controller in a system call and leave the algorithm to userspace.
-- wli
Quoted 10 lines · William Lee Irwin III
Quoted 8 lines · David S. Miller
The in-kernel stuff MUST go. It went in because "some benchmark went faster", but with no "why" describing why it might have improved performance. We KNOW it absolutely sucks for routing and firewall applications. The in-kernel bits were all a shamans dance, with zero technical "here is why this makes things go faster" description attached. If I remember properly, the changelog message when the in-kernel irq balancing went in was of the form "this makes some specweb run go faster".
Quoted 3 lines
Absolutely. Not to mention the code for the in-kernel algorithm has historically broken i386 ports using certain modes of Intel's interrupt controllers.
OK, I just went and actually looked at the code again. After suppressing my gag reflex, I started to remember all of the problems we've had with it, including fixing it for Intel's own clustered APIC mode.
Does anyone have a patch to tear it out already? Is the current proc interface acceptable, or do we want a syscall interface like wli suggests?
Signature · 2 lines
Dave Hansen [email protected]
--Dave Hansen <[email protected]> wrote (on Monday, May 19, 2003 22:03:50 -0700):
Quoted 15 lines · William Lee Irwin III
Quoted 10 lines · David S. Miller
Quoted 8 lines
The in-kernel stuff MUST go. It went in because "some benchmark went faster", but with no "why" describing why it might have improved performance. We KNOW it absolutely sucks for routing and firewall applications. The in-kernel bits were all a shamans dance, with zero technical "here is why this makes things go faster" description attached. If I remember properly, the changelog message when the in-kernel irq balancing went in was of the form "this makes some specweb run go faster".
Quoted 3 lines
Absolutely. Not to mention the code for the in-kernel algorithm has historically broken i386 ports using certain modes of Intel's interrupt controllers.
Quoted 8 lines
OK, I just went and actually looked at the code again. After suppressing my gag reflex, I started to remember all of the problems we've had with it, including fixing it for Intel's own clustered APIC mode.
Does anyone have a patch to tear it out already? Is the current proc interface acceptable, or do we want a syscall interface like wli suggests?
I have no frigging idea why you'd want to tear something out that works well already, and has a shitload of work put into it.
Make it a config option if you don't like it, Keith has a patch to do that already - it's trivial. That way everyone can have what they want.
M.
From: "Martin J. Bligh" <[email protected]>
Date: Mon, 19 May 2003 22:53:11 -0700
I have no frigging idea why you'd want to tear something out that works well already, and has a shitload of work put into it.
It's pretty fundamentally broken for having had so much work put into it. Show me something other than "SpecWEB run for IBM ran faster" as a reason for keeping this code in there. Can you even do this?
It is crap, from the very beginning, would you like to know why?
How does the in-kernel IRQ load balancing measure "load" and "busyness"? Herein lies the most absolutely fundamental problem with this code, it fails to recognize that we end up with most of our networking "load" from softint context.
We can process thousands of packets for one hardware interrupt. Are you able to comprehend this?
Measuring hardware interrupts in some was as "load" is about as far from the truth as you can get.
This is just the tip of the iceberg.
rm -rf in-kernel-irqbalance;
And hey, if _YOU_ want a broken system which uses this bogus algorithm, YOU CAN DO THIS with the userland thing if you want.
Quoted 10 lines · David S. Miller
From: "Martin J. Bligh" <[email protected]>
Date: Mon, 19 May 2003 22:53:11 -0700I have no frigging idea why you'd want to tear something out that works well already, and has a shitload of work put into it.
It's pretty fundamentally broken for having had so much work put into it. Show me something other than "SpecWEB run for IBM ran faster" as a reason for keeping this code in there. Can you even do this?
I don't even think we can do that. That code was being integrated around the same time that our Specweb setup decided to go south on us and start physically frying itself. We never got a chance to run it. BTW, I don't think there are any other kernel developers running Specweb on 2.5 kernels. If there are, please speak up!
Andrew Theurer posted some positive results here, which were quite marginal in the case with 1 nic. 4.7% with two.
http://marc.theaimsgroup.com/?l=linux-kernel&m=104212930819212&w=2
Signature · 2 lines
Dave Hansen [email protected]
From: Dave Hansen <[email protected]>
Date: 19 May 2003 23:36:23 -0700
I don't even think we can do that. That code was being integrated around the same time that our Specweb setup decided to go south on us and start physically frying itself.
This gets more amusing by the second. Let's kill this code already. People who like the current algorithms can push them into the userspace solution.
Quoted 10 lines · David S. Miller
From: Dave Hansen <[email protected]>
Date: 19 May 2003 23:36:23 -0700I don't even think we can do that. That code was being integrated around the same time that our Specweb setup decided to go south on us and start physically frying itself.
This gets more amusing by the second. Let's kill this code already. People who like the current algorithms can push them into the userspace solution.
Remember this all started with some idea of "fairness" among cpus and very
little to do with performance. particularly on P4 with HT, where the first
logical cpu got all the ints and tasks running on that cpu were slower than other cpus. This was in most cases the highest performing situation, -but- it was unfair to the tasks running on cpu0. irq_balance fixed this with a random target cpu that was in theory supposed to not change often enough to preserve cache warmth. In practice is the target cpus changed too often which thrashed cache and the HW overhead of changing the destination that often was way way to high.
Although kirq was a step in the right direction (compared to irq_balance), I'd rather see it in user space in the long term, too. That way we can make policy changes much much easier. IMO, networking performance was always better with all net card ints going to only one cpu, -until- that cpu would
be saturated. This situation point can come much sooner with HT since the
core is shared, and as far as I know, there is no way to bias the core to the one sibling handling ints when int load is high. The only thing better than all ints to cpu0 is aligning irq a process affinity together, which is 99% unrealistic for all actual workloads.
Now, if someone can figure out how/when the first cpu is saturated, and measure int load properly, maybe we can have a policy that keeps all ints on cpu0, spills some ints to another cpu when that cpu is saturated, -and- modifies find_busiest_queue to compensate nr_running on cpus with high int load to make the process thingy more fair.
If kirq gets ripped out, at least have some default policy that is somewhat
harmless, like destination cpu = int_number % nr_cpus. I think Suse8 had
this, and it performed reasonably well.
-Andrew Theurer
Quoted 12 lines · Andrew Theurer
Quoted 10 lines · David S. Miller
From: Dave Hansen <[email protected]>
Date: 19 May 2003 23:36:23 -0700I don't even think we can do that. That code was being integrated around the same time that our Specweb setup decided to go south on us and start physically frying itself.
This gets more amusing by the second. Let's kill this code already. People who like the current algorithms can push them into the userspace solution.
Quoted 9 lines
Remember this all started with some idea of "fairness" among cpus and very
little to do with performance. particularly on P4 with HT, where the first
logical cpu got all the ints and tasks running on that cpu were slower than other cpus. This was in most cases the highest performing situation, -but- it was unfair to the tasks running on cpu0.irq_balancefixed this with a random target cpu that was in theory supposed to not change often enough to preserve cache warmth. In practice is the target cpus changed too often which thrashed cache and the HW overhead of changing the destination that often was way way to high.
You call that a fix? ;-) I call that working around a bug.
If tasks run slower on cpuX than cpuY because of a heavier int load, that's the fault of the scheduler not the irqbalancer, be it in-kernel or userspace. If there's a lesser-utilized cpu the task needs to be migrated to that cpu from the irq-loaded one, when CPU accounting notices the kernel interrupt handling having an impact.
Jeff
Quoted 23 lines · Jeff Garzik
Quoted 12 lines · Andrew Theurer
Quoted 10 lines · David S. Miller
From: Dave Hansen <[email protected]>
Date: 19 May 2003 23:36:23 -0700I don't even think we can do that. That code was being integrated around the same time that our Specweb setup decided to go south on
us and start physically frying itself.This gets more amusing by the second. Let's kill this code already. People who like the current algorithms can push them into the userspace solution.
Quoted 9 lines
Remember this all started with some idea of "fairness" among cpus and
very little to do with performance. particularly on P4 with HT, where
the first logical cpu got all the ints and tasks running on that cpu were slower than other cpus. This was in most cases the highest performing situation, -but- it was unfair to the tasks running on cpu0.irq_balancefixed this with a random target cpu that was in theory supposed to not change often enough to preserve cache warmth. In practice is the target cpus changed too often which thrashed cache and the HW overhead of changing the destination that often was way way to high.
Quoted 7 lines
You call that a fix? ;-) I call that working around a bug.
If tasks run slower on cpuX than cpuY because of a heavier int load, that's the fault of the scheduler not the irqbalancer, be it in-kernel or userspace. If there's a lesser-utilized cpu the task needs to be migrated to that cpu from the irq-loaded one, when CPU accounting notices the kernel interrupt handling having an impact.
On paper it sounds good but it may be more difficult (and not perfect) in practice. For example, if you have one runnable task per cpu, moving it to another cpu would not help. To preserve fairness, all you could do is swap the tasks around, which would thrash the cache pretty well. Also, the int load may be very dynamic, so you need to be really careful and make sure you are measuring sustained int load.
However I do agree, we need something in the scheduler, in particular something to modify max_load and load in find_busiest_queue, based on a int load average for those cpus.
-Andrew
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines
There is zero reason why IRQ balancing should be in any way different. It's POLICY, and POLICY == USERSPACE. It is the end of the argument.
Despite whatever political wrangling there is between userspace and kernelspace implementations (and some very valid points about other arches), there is still a dearth of testing, as far as I can see.
I can't see anything wrong with making it a config option for now, and letting people choose what they want to do, until we have more information as to which performs better under a variety of workloads. That seems the most pragmatic way forward.
M.
From: "Martin J. Bligh" <[email protected]>
Date: Wed, 21 May 2003 07:58:11 -0700
Despite whatever political wrangling there is between userspace and kernelspace implementations (and some very valid points about other arches), there is still a dearth of testing, as far as I can see.
I've never in my life heard the argument that we kept something in the kernel that didn't belong there due to "userland testing". That's a bogus argument.
When I ripped RARP out of the kernel, we didn't immediately have a replacement, but one showed up shortly. So what?
And in this ase we already have Arjan's stuff. So start testing his code instead of whining about keeping the current stuff in the tree.
Quoted 13 lines · David S. Miller
From: Dave Hansen <[email protected]>
Date: 19 May 2003 23:36:23 -0700I don't even think we can do that. That code was being integrated around the same time that our Specweb setup decided to go south on us and start physically frying itself.
This gets more amusing by the second. Let's kill this code already. People who like the current algorithms can push them into the userspace solution.
That´s also my feeling.
After having more and more performance loss (at least for my system/usage) and no real good solution in view to fix the problems at this location in the kernel, I would prefer to get this thing ripped of as soon as possible. By now I do not see any gain with that piece of code (relative to process/interrupt distribution). And please don´t show me benchmarks - I know what I feel.
Userland would be first of all the right place to do any further testing.
And please - don´t tell me it´s that complicated to remove this piece of code from source tree.
Kai.
Quoted 4 lines
How does the in-kernel IRQ load balancing measure "load" and "busyness"? Herein lies the most absolutely fundamental problem with this code, it fails to recognize that we end up with most of our networking "load" from softint context.
OK, that's a great observation, and probably fixable. What were the author's comments when you told him that?
Quoted 1 line
rm -rf in-kernel-irqbalance;
It's *very* late in the day to be ripping out such chunks of code.
1. Prove new code works better for you => make it a config option.
2. Prove new code works better for everyone => rip it out.
I think we're at 1, not 2.
Note that the userspace stuff doesn't even require that the kernel stuff be disabled ... it should just override it (I can believe there maybe is a bug that needs fixing, but it works by design).
M.
Quoted 3 lines · Dave Hansen
Does anyone have a patch to tear it out already? Is the current proc interface acceptable, or do we want a syscall interface like wli suggests?
I have no problems with the proc interface; it's ascii so reasonably extendible in the future for, say, when 64 cpus on
32 bit linux get supported. It's also not THAT inefficient since my code only uses it when some binding changes, not all the time.
Quoted 3 lines · Dave Hansen
Quoted 3 lines
Does anyone have a patch to tear it out already? Is the current proc interface acceptable, or do we want a syscall interface like wli suggests?
Quoted 4 lines · Arjan van de Ven
I have no problems with the proc interface; it's ascii so reasonably extendible in the future for, say, when 64 cpus on
32 bit linux get supported. It's also not THAT inefficient since my code only uses it when some binding changes, not all the time.
Sorry about that; I forgot about the /proc/ part and thought the thing was based on system calls as it stood. I wouldn't want a redundant interface to be added.
My current cpumask_t patches handle extending the /proc/ interface to handle an arbitrary-sized cpumask, so I should have realized this.
-- wli
Quoted 10 lines · Arjan van de Ven
Quoted 3 lines · Dave Hansen
Does anyone have a patch to tear it out already? Is the current proc interface acceptable, or do we want a syscall interface like wli suggests?
I have no problems with the proc interface; it's ascii so reasonably extendible in the future for, say, when 64 cpus on
32 bit linux get supported. It's also not THAT inefficient since my code only uses it when some binding changes, not all the time.
Concerns have been expressed that the /proc interface may be a bit racy. One thing we do need to do is to write a /proc stresstest tool which pokes numbers into the /proc files at high rates, run that under traffic for a few hours.
There is no need to pull out the existing balancer until the userspace solution is proven - it can be turned off with `noirqbalance' until that work has been performed.
Nobody has tried improving the current balancer. From a quick look it appears that it could work reasonably for the problematic packet-forwarding workload if the when-to-start-balancing threshold is reduced from 1000/sec to (say) 10/sec. Don't know - I've never seen a description of how the algorithm should be improved.
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 8 lines · David S. Miller
From: Andrew Morton <[email protected]>
Date: Tue, 20 May 2003 02:17:12 -0700Concerns have been expressed that the /proc interface may be a bit racy. One thing we do need to do is to write a /proc stresstest tool which
pokes numbers into the /proc files at high rates, run that under traffic for a few hours.
Quoted 8 lines
This issue is %100 independant of whether policy belongs in the kernel or not. Also, the /proc race problem exists and should be fixed regardless.
Nobody has tried improving the current balancer.
Policy does not belong in the kernel. I don't care what algorithm people decide to use, but such decisions do NOT belong in the kernel.
You keep saying that, but suppose I want to try HW IRQ balancing using the TPR registers. How could I do that from userspace? And if I could, wouldn't the benefit of real time IRQ routing be lost?
It seems to me that only long term interrupt policy can be done from userland. Anything that does fast responses to fluctuating load must be inside the kernel.
At the moment we don't do any fast IRQ policy. Even the original irq_balance only looked for idle CPUs after an interrupt was serviced. However, suppose you had a P4 with hyperthreading turned on. If an IRQ is to be delivered to the main thread but it is busy and its sibling is idle, why shouldn't we deliver the interrupt to the idle sibling? They both share the same caches, etc, so cache warmth isn't a problem.
Signature · 3 lines
James Cleverdon
IBM xSeries Linux Solutions
{jamesclv(Unix, preferred), cleverdj(Notes)} at us dot ibm dot com