Process priority in 2.4.18 (RedHat 7.3)
Quoted 6 lines · Andrew Morton
Quoted 3 lines · Paul Faure
Just installed RedHat 7.3 with kernel 2.4.18 and noticed that my network no longer functions when my CPU usage is at 100%.
ugh. Which NIC are you using?
ne.o
Its an ISA card.
Signature · 4 lines
Paul N. Faure 613.266.3286 EngSoc Administrator paul-at-engsoc-dot-org Chief Technical Officer, CertainKey Inc. paul-at-certainkey-dot-com Carleton University Systems Eng. 4th Year paul-at-faure-dot-ca
Quoted 5 lines · Paul Faure
Quoted 2 lines · Andrew Morton
Quoted 1 line · Paul Faure
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
Just installed RedHat 7.3 with kernel 2.4.18 and noticed that my network no longer functions when my CPU usage is at 100%.
ugh. Which NIC are you using?
ne.o
Its an ISA card.
It's possible that this particular driver is flunking on a lot of transmit attempts and is relying on ksoftirqd to transmit.
Are you able to test with a moderately respectable NIC? tulip, 3com, realtek, eepro?
-
Quoted 1 line · Andrew Morton
of transmit attempts and is relying on ksoftirqd to transmit.
ksoftirqd or not the softirq are guaranteed to keep running even if there's a task in loop with SCHED_FIFO, ksoftirqd only enhance/polish the case of a recursive softirq, or a very big flood of softirq events, it is not required to run softirqs.
Andrea
It would seem that it only occurs when running the application (that takes 100% of the CPU) as root.
As for testing it with other cards, I only have this one card.
Thanks for your time... its not a big issue now since I can run my application as a non privileged user.
Quoted 10 lines · Andrea Arcangeli
Quoted 1 line · Andrew Morton
of transmit attempts and is relying on ksoftirqd to transmit.
ksoftirqd or not the softirq are guaranteed to keep running even if there's a task in loop with
SCHED_FIFO, ksoftirqd only enhance/polish the case of a recursive softirq, or a very big flood of softirq events, it is not required to run softirqs.Andrea
Signature · 4 lines
Paul N. Faure 613.266.3286 EngSoc Administrator paul-at-engsoc-dot-org Chief Technical Officer, CertainKey Inc. paul-at-certainkey-dot-com Carleton University Systems Eng. 4th Year paul-at-faure-dot-ca
Quoted 3 lines · Paul Faure
It would seem that it only occurs when running the application (that takes 100% of the CPU) as root.
That's because without root, the application cannot raise its scheuling priority and it cannot change to realtime policy.
So the problem would appear to be that your networking *requires* ksoftirqd services to function. Either:
1) The driver is bust - its hard_start_xmit() function is failing frequently, and relying on ksoftirqd to get things done (I think; it's been a while). Or
2) Something is wrong with the ksoftirqd design. Or
3) Red Hat fiddled with ksoftirqd and broke it.
I'd be inclined to suspect 1).
Quoted 2 lines
As for testing it with other cards, I only have this one card.
That's a shame.
-
Quoted 6 lines
So the problem would appear to be that your networking *requires* ksoftirqd services to function. Either:
1) The driver is bust - its
hard_start_xmit()function is failing frequently, and relying on ksoftirqd to get things done (I think; it's been a while). Or
The ne2k card has little buffering.
Quoted 1 line
2) Something is wrong with the ksoftirqd design. Or
I think its mostly #2. We invoke ksoftirq far far too easily.
Quoted 1 line · Alan Cox
I think its mostly #2. We invoke ksoftirq far far too easily.
ksoftirqd + SCHED_FIFO is like no ksoftirqd at all, provided the ne card is irq driven (it is) everything works like it was working in 2.4.0.
Andrea
Quoted 5 lines · Alan Cox
Quoted 1 line
I think its mostly #2. We invoke ksoftirq far far too easily.
ksoftirqd +
SCHED_FIFOis like no ksoftirqd at all, provided the ne card is irq driven (it is) everything works like it was working in 2.4.0.
For a 10Mbit ne2k it ought to be if its done with sched fifo. For serious devices its not. The ksoftirqd bounce blows everything out of cache and is easily measured
Quoted 9 lines · Alan Cox
Quoted 5 lines · Alan Cox
Quoted 1 line
I think its mostly #2. We invoke ksoftirq far far too easily.
ksoftirqd +
SCHED_FIFOis like no ksoftirqd at all, provided the ne card is irq driven (it is) everything works like it was working in 2.4.0.For a 10Mbit ne2k it ought to be if its done with sched fifo. For serious devices its not. The ksoftirqd bounce blows everything out of cache and is easily measured
if you're under a flood of irq ksoftirqd or not won't make differences to the softirq handling, and yes in such case ksoftirq cannot help because you are under a flood of do_softirq anyways run from irq context and it is only a minor scheduler overhead in such case, but it gets right all and polishes all the other "recursion" cases like NAPI.
But that has nothing to do with this case, here the userspace runs with SCHED_FIFO in a loop so ksoftirqd cannot make any difference compared to 2.4.0 if the device is irq driven, so I don't see your point in mentioning minor performance regressions while under a flood of irqs due the minor scheduler overhead, here the minor scheduler overhead cannot apply because ksoftirqd has not a chance to run at all.
Also I'd be nice if he could try with mainline (or 2.4.19pre8aa3) too just in case, we didn't had any confirm that such proggy uses SCHED_FIFO or SCHED_RR, even if I of course agree about the supposions made by Andrew without having access to additional informations.
Andrea
Quoted 5 lines
Quoted 3 lines
For a 10Mbit ne2k it ought to be if its done with sched fifo. For serious devices its not. The ksoftirqd bounce blows everything out of cache and is easily measured
if you're under a flood of irq ksoftirqd or not won't make differences
I didnt mention a flood of irqs. If stuff falls back to softirqd it materially harms throughput
Quoted 8 lines · Alan Cox
Quoted 5 lines
Quoted 3 lines
For a 10Mbit ne2k it ought to be if its done with sched fifo. For serious devices its not. The ksoftirqd bounce blows everything out of cache and is easily measured
if you're under a flood of irq ksoftirqd or not won't make differences
I didnt mention a flood of irqs. If stuff falls back to softirqd it materially harms throughput
You did implicitly becuse if there's not a flood of irq or recursive softirqs it cannot fall to sofitrqd.
Andrea
Quoted 5 lines
Quoted 2 lines
I didnt mention a flood of irqs. If stuff falls back to softirqd it materially harms throughput
You did implicitly becuse if there's not a flood of irq or recursive softirqs it cannot fall to sofitrqd.
Hardly takes a flood of IRQ's. A tiny burst of interrupts will happily trigger it
Quoted 2 lines · Alan Cox
Hardly takes a flood of IRQ's. A tiny burst of interrupts will happily trigger it
hmm, tiny != burst. of course sometime ksoftirqd will kick in when it notices a burst. But it is irrelevant to this thread about SCHED_FIFO + ksoftirqd.
If there's SCHED_FIFO app in loop, ksoftirqd never runs and we only rely on the support from irq that we had in 2.4.0 and previous too.
returning to the other issue about the fake burst that stops immediatly, it should be easy to write a small heuristic to avoid scheduling ksoftirqd in those cases, two different ways could be to simply count the number of do_softirq invocations outside ksofitrqd per jiffy and to skip waking up ksoftirqd if the frequency is high enough, this would fix the minor schedule overhead during the fake burst, however this may not be optimal for NAPI if for whatever reason there's an irq flood from some other device in background, so I would actually prefer to detect when ksoftirqd stops immediatly, it's a bit more complicated but more generic, if it always stops immediatly then we make it harder to kick it in. Note that I also benchmarked the effect softirqd in the early days and I couldn't notice any performance drawback with using netpipe on 100Mbit tulip both on the small fragments (higher burst) and on the big fragments, so whatever regression it is it is a minor issue and it definitely pays off the polishing of all the other important cases so cleanly (NAPI, rcu-poll, ppa, irq flood even on 10Mbit, even if you block the hard irq with ratelimit you still can hang in the softirq code without ksofitrqd otherwise you risk to introduce a 10msec latency to the softirq processing, you don't know how long the softirq load takes etc..). It is true that even with ksoftirqd you may hang if the softirq load takes exactly the same frequency of the do_softirq calls, but that's not a pratical case, you cannot stay exactly on the border for a relevant time, while you can pratically go over the border as long as you want.
Andrea
A little more information:
On a dual system:
Running as root -> Single process app (50% total CPU)-> locks out network Running as root -> Multi process app (100% total CPU) -> locks out network Running unprivileged -> Single process(50%) -> Network works fine Running unprivileged -> Multi process(100%) -> locks out network
Going to try a few other network cards if I find one.
Quoted 13 lines · Alan Cox
Quoted 6 lines
So the problem would appear to be that your networking *requires* ksoftirqd services to function. Either:
1) The driver is bust - its
hard_start_xmit()function is failing frequently, and relying on ksoftirqd to get things done (I think; it's been a while). OrThe ne2k card has little buffering.
Quoted 1 line
2) Something is wrong with the ksoftirqd design. Or
I think its mostly #2. We invoke ksoftirq far far too easily.
Signature · 4 lines
Paul N. Faure 613.266.3286 EngSoc Administrator paul-at-engsoc-dot-org Chief Technical Officer, CertainKey Inc. paul-at-certainkey-dot-com Carleton University Systems Eng. 4th Year paul-at-faure-dot-ca
Quoted 3 lines
hmm, tiny != burst. of course sometime ksoftirqd will kick in when it notices a burst. But it is irrelevant to this thread about
SCHED_FIFO+ ksoftirqd.
Agreed
Quoted 2 lines
If there's
SCHED_FIFOapp in loop, ksoftirqd never runs and we only rely on the support from irq that we had in 2.4.0 and previous too.
Yes
My brother dropped off a network card (smc-ultra) and it works fine with the 2.4.18-4 redhat kernel. If you are still interested in testing the ne.o card with the latest official kernel, I can do it.
Thanks for all your help.
Quoted 11 lines · Alan Cox
Quoted 3 lines
hmm, tiny != burst. of course sometime ksoftirqd will kick in when it notices a burst. But it is irrelevant to this thread about
SCHED_FIFO+ ksoftirqd.Agreed
Quoted 2 lines
If there's
SCHED_FIFOapp in loop, ksoftirqd never runs and we only rely on the support from irq that we had in 2.4.0 and previous too.Yes
Signature · 4 lines
Paul N. Faure 613.266.3286 EngSoc Administrator paul-at-engsoc-dot-org Chief Technical Officer, CertainKey Inc. paul-at-certainkey-dot-com Carleton University Systems Eng. 4th Year paul-at-faure-dot-ca
My brother dropped of a network card (smc-ultra) and it works fine with the 2.4.18-4 redhat kernel. If you are still interested in testing the ne.o card with the latest official kernel, I can do so.
Thanks for all your help.
Quoted 11 lines · Alan Cox
Quoted 3 lines
hmm, tiny != burst. of course sometime ksoftirqd will kick in when it notices a burst. But it is irrelevant to this thread about
SCHED_FIFO+ ksoftirqd.Agreed
Quoted 2 lines
If there's
SCHED_FIFOapp in loop, ksoftirqd never runs and we only rely on the support from irq that we had in 2.4.0 and previous too.Yes
Signature · 4 lines
Paul N. Faure 613.266.3286 EngSoc Administrator paul-at-engsoc-dot-org Chief Technical Officer, CertainKey Inc. paul-at-certainkey-dot-com Carleton University Systems Eng. 4th Year paul-at-faure-dot-ca