Net softirq deferring to ksoftirqd

Warning: Not merge-ready

I. Current workflow of ksoftirqd.
Softirqs are processed in the context of ksoftirqd iff they are being raised very frequently. How it works:
do_softirq() and invoke_softirq() deffer pending softirq iff ksoftirqd is in runqueue. Ksoftirqd is scheduled mostly in the end of processed softirqs if 2ms were not enough to process all pending softirqs.

Here is pseudo-picture of the workflow (for simplicity on UMP):

  -------------      ------------------      ------------------
  | ksoftirqd |      | User's process |      |   Softirqs     |
  -------------      ------------------      ------------------
   Not scheduled          Running
                             |
                             o------------------------o
                                                      |
                                                __do_softirq()
                                                      |
                                              2ms & softirq pending?
                                              Schedule ksoftirqd
                                                      |
    Scheduled                o------------------------o
                             |
        o--------------------o
        |
     Running             Scheduled
        |
        o--------------------o
                             |

Not scheduled Running

Timegraph for the workflow,
dash (-) means ksoftirqd not scheduled;
equal(=) ksoftirqd is scheduled, a softirq may still be pending

Pending softirqs
| | | | | | | | |
v v v v | | | | v

   Processing   o-----o           | | | |       o--o
    softirqs    |     |           | | | |       |  |
                |     |           | | | |       |  |
                |     |           | | | |       |  |
   Userspace  o-o     o=========o | | | |  o----o  o---------o
                <-2ms->         | | | | |  |
                                | v v v v  |
   Ksoftirqd                    o----------o

II. Corner-conditions.
During testing of commit [1] on some non-mainstream driver, I've found that due to platform specifics, the IRQ is being raised too late (after softirq has been processed).

In result softirqs steal time from userspace process, leaving it starving for CPU time and never/rarely scheduling ksoftirqd:

2018-01-09 14:36 UTC+01:00, Dmitry Safonov <[email protected]>:

Quoted 51 lines

Warning: Not merge-ready

I. Current workflow of ksoftirqd.
Softirqs are processed in the context of ksoftirqd iff they are being raised very frequently. How it works:
do_softirq() and invoke_softirq() deffer pending softirq iff ksoftirqd is in runqueue. Ksoftirqd is scheduled mostly in the end of processed softirqs if 2ms were not enough to process all pending softirqs.

Here is pseudo-picture of the workflow (for simplicity on UMP):

  -------------      ------------------      ------------------
  | ksoftirqd |      | User's process |      |   Softirqs     |
  -------------      ------------------      ------------------
   Not scheduled          Running
                             |
                             o------------------------o
                                                      |
                                                __do_softirq()
                                                      |
                                              2ms & softirq
pending?
                                              Schedule ksoftirqd
                                                      |
    Scheduled                o------------------------o
                             |
        o--------------------o
        |
     Running             Scheduled
        |
        o--------------------o
                             |

Not scheduled Running

Timegraph for the workflow,
dash (-) means ksoftirqd not scheduled; equal(=) ksoftirqd is scheduled, a softirq may still be
pending

Pending softirqs
| | | | | | | | |
v v v v | | | | v

   Processing   o-----o           | | | |       o--o
    softirqs    |     |           | | | |       |  |
                |     |           | | | |       |  |
                |     |           | | | |       |  |
   Userspace  o-o     o=========o | | | |  o----o  o---------o
                <-2ms->         | | | | |  |
                                | v v v v  |
   Ksoftirqd                    o----------o

Quoted 4 lines

II. Corner-conditions.
During testing of commit [1] on some non-mainstream driver, I've found that due to platform specifics, the IRQ is being raised too late (after softirq has been processed).

I'm a bit confused about that part. I would expect the softirq to be raised by the IRQ.

So I guess in this scenario the softirq is raised by something else and you expect the upcoming IRQ to handle the softirq, right? (sorry I'm not used to networking code).

Warning: Not merge-ready

I. Current workflow of ksoftirqd.
Softirqs are processed in the context of ksoftirqd iff they are being raised very frequently. How it works:
do_softirq() and invoke_softirq() deffer pending softirq iff

Note, that I tested in VMs and I've found that if I produce more hw irqs on the host, than the results for master are not that dramatically bad, but still much worse then with RFC. By that reason I have qualms if my test's results are correct.

Quoted 7 lines · Dmitry Safonov

Warning: Not merge-ready

I. Current workflow of ksoftirqd.
Softirqs are processed in the context of ksoftirqd iff they are being raised very frequently. How it works:
do_softirq() and invoke_softirq() deffer pending softirq iff

...

Quoted 5 lines

Note, that I tested in VMs and I've found that if I produce more hw irqs on the host, than the results for master are not that dramatically bad, but still much worse then with RFC. By that reason I have qualms if my test's results are correct.

Note that deferring all NET RX/TX to ksoftirqd is going to dramatically hurt tail latencies.

You really should test with RPC like workloads (netperf -t TCP_RR) and
hundred of threads per cpu :/

It seems we are going to revert/adapt 4cd13c21b2 , not defer more stuff to ksoftirqd.

Hmm, what if we use some other logic for deferring/non-deferring like checking how many softirqs where serviced during process's timeslice and decide if proceed with __do_softirq() or defer it not to starve a task? Might that make sense?

So just saying "hey, ksoftirq is runnable - but maybe not running _now"" and ignoring softirqs entirely is just stupid. Even if we could easily do another small bunch of them, at least the non-networking ones.

So maybe that "ksoftirqd_running()" check should actually be something like

  static bool ksoftirqd_running(void)
  {
        struct task_struct *tsk = __this_cpu_read(ksoftirqd);

        return tsk == current;

}

which actually checks that ksoftirq is running right *now*, and not scheduled away because somebody is running a perl script.

Makes sense, but I think you need to keep the TASK_RUNNING check.

Quoted 2 lines · Frederic Weisbecker

Makes sense, but I think you need to keep the TASK_RUNNING check.

Yes, good point.

kernel/softirq.c static bool ksoftirqd_running(void) · line 88

Quoted 4 lines

So perhaps it should be:

kernel/softirq.c +1 -1

-       return tsk && (tsk->state == TASK_RUNNING);
+       return (tsk == current) && (tsk->state == TASK_RUNNING);

Looks good to me - definitely worth trying.

Maybe that weakens the thing so much that it doesn't actually help the UDP packet storm case?

And maybe it's not sufficient for the dvb issue.

But I think it's worth at least testing. Maybe it makes neither side entirely happy, but maybe it might be a good halfway point?

Quoted 18 lines · Linus Torvalds

Quoted 2 lines · Frederic Weisbecker

Makes sense, but I think you need to keep the TASK_RUNNING check.

Yes, good point.

kernel/softirq.c static bool ksoftirqd_running(void) · line 88

Quoted 4 lines

So perhaps it should be:

kernel/softirq.c +1 -1

-       return tsk && (tsk->state == TASK_RUNNING);
+       return (tsk == current) && (tsk->state == TASK_RUNNING);

Looks good to me - definitely worth trying.

Maybe that weakens the thing so much that it doesn't actually help the
UDP packet storm case?

Quoted 5 lines

And maybe it's not sufficient for the dvb issue.

But I think it's worth at least testing. Maybe it makes neither side
entirely happy, but maybe it might be a good halfway point?

Yes I believe Dmitry is facing a different problem where he would rather
see ksoftirqd scheduled more often to handle the queue as a deferred batch
instead of having it served one by one on the tails of IRQ storms. (Dmitry correct me if I misunderstood).

But your patch still seems to make sense for the case you described:
when
ksoftirqd is voluntarily preempted off and the current IRQ could handle the
queue.

Quoted 25 lines · Frederic Weisbecker

Quoted 18 lines · Linus Torvalds

Quoted 2 lines · Frederic Weisbecker

Makes sense, but I think you need to keep the TASK_RUNNING check.

Yes, good point.

Review

Quoted 4 lines

So perhaps it should be:

(quoted diff) +1 -1

-       return tsk && (tsk->state == TASK_RUNNING);
+       return (tsk == current) && (tsk->state == TASK_RUNNING);

Looks good to me - definitely worth trying.

Maybe that weakens the thing so much that it doesn't actually help the
UDP packet storm case?

Quoted 5 lines

And maybe it's not sufficient for the dvb issue.

But I think it's worth at least testing. Maybe it makes neither side
entirely happy, but maybe it might be a good halfway point?

Quoted 6 lines

Yes I believe Dmitry is facing a different problem where he would rather
see ksoftirqd scheduled more often to handle the queue as a deferred batch
instead of having it served one by one on the tails of IRQ storms. (Dmitry correct me if I misunderstood).

Quite so, what I see is that ksoftirqd is rarely (close to never) scheduled in case of UDP packet storm. That's because the up coming irq is too late in __do_softirq().
So, there is no wakeup on UDP storm here:
: pending = local_softirq_pending();
: if (pending & mask) {
: if (time_before(jiffies, end) && !need_resched() &&
: --max_restart)
: goto restart;
:
: wakeup_softirqd();
: }
(as there is yet no pending softirq). It comes a bit late to schedule ksoftirqd and in result the next softirq is processed on the context of the task again, not in the scheduled ksoftirqd.
That results in cpu-time starvation for the process on irq storm.

While I saw that on out-of-tree driver, I believe that on some frequencies (lower than storm) one can observe the same on mainstream drivers. And I *think* that I've reproduced that on mainstream with virtio driver and package size of 1500 in VMs (thou I don't quite like the perf testing in VMs).

So, ITOW, maybe there is a bit better way to *detect* that cpu time spent on serving softirqs is close to storm and that userspace starts starving? (and launch ksoftirqd in the result or balance between deferring and serving softirq right-there).

Quoted 5 lines

But your patch still seems to make sense for the case you described:
when
ksoftirqd is voluntarily preempted off and the current IRQ could handle the
queue.

kernel/softirq.c +0 -0

index 2f5e87f1bae22f3df44fa4493fcc8b255882267f..d2f20daf77d14dc8ebde00d7c4a0237152d082ba

100644

kernel/softirq.c +1 -1

@@ -192,7 +192,7 @@ EXPORT_SYMBOL(__local_bh_enable_ip);
 
 /*
  * We restart softirq processing for at most MAX_SOFTIRQ_RESTART times,
- * but break the loop if need_resched() is set or after 2 ms.
+ * but break the loop after 2 ms.
  * The MAX_SOFTIRQ_TIME provides a nice upper bound in most cases, but in
  * certain cases, such as stop_machine(), jiffies may cease to
  * increment and so we need the MAX_SOFTIRQ_RESTART limit as

You just introduced a 2ms preempt-disable region I think, that's not cool for PREEMPT and a plain bug on PREEMPT_RT.

From: Peter Zijlstra <[email protected]>
Date: Thu, 11 Jan 2018 17:32:04 +0100

Quoted 2 lines

You just introduced a 2ms preempt-disable region I think, that's not cool for PREEMPT and a plain bug on PREEMPT_RT.

I'd just like to know what the difference is between ksoftirqd running softirq work for 2ms and the real softirq context doing it. The latter already has this "2ms preempt-disable region" of sorts, right?

Quoted 23 lines · Eric Dumazet

kernel/softirq.c +0 -0

index 2f5e87f1bae22f3df44fa4493fcc8b255882267f..d2f20daf77d14dc8ebde00d7c4a0237152d082ba

100644

kernel/softirq.c +2 -3

@@ -192,7 +192,7 @@ EXPORT_SYMBOL(__local_bh_enable_ip);
 
 /*
  * We restart softirq processing for at most MAX_SOFTIRQ_RESTART times,
- * but break the loop if need_resched() is set or after 2 ms.
+ * but break the loop after 2 ms.
  * The MAX_SOFTIRQ_TIME provides a nice upper bound in most cases, but in
  * certain cases, such as stop_machine(), jiffies may cease to
  * increment and so we need the MAX_SOFTIRQ_RESTART limit as

@@ -299,8 +299,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)
 
        pending = local_softirq_pending();
        if (pending) {
-               if (time_before(jiffies, end) && !need_resched() &&
-                   --max_restart)
+               if (time_before(jiffies, end) && --max_restart)
                        goto restart;
 

kernel/softirq.c asmlinkage __visible void __softirq_entry __do_softirq(void) · line 305

Quoted 1 line

kernel/softirq.c +0 -0

                wakeup_softirqd();

You just introduced a 2ms preempt-disable region I think, that's not cool for PREEMPT and a plain bug on PREEMPT_RT.

Quoted 24 lines · Peter Zijlstra

Quoted 23 lines · Eric Dumazet

kernel/softirq.c +0 -0

index 2f5e87f1bae22f3df44fa4493fcc8b255882267f..d2f20daf77d14dc8ebde00d7c4a0237152d082ba

100644

kernel/softirq.c +2 -3

@@ -192,7 +192,7 @@ EXPORT_SYMBOL(__local_bh_enable_ip);
 
 /*
  * We restart softirq processing for at most MAX_SOFTIRQ_RESTART times,
- * but break the loop if need_resched() is set or after 2 ms.
+ * but break the loop after 2 ms.
  * The MAX_SOFTIRQ_TIME provides a nice upper bound in most cases, but in
  * certain cases, such as stop_machine(), jiffies may cease to
  * increment and so we need the MAX_SOFTIRQ_RESTART limit as

@@ -299,8 +299,7 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)
 
        pending = local_softirq_pending();
        if (pending) {
-               if (time_before(jiffies, end) && !need_resched() &&
-                   --max_restart)
+               if (time_before(jiffies, end) && --max_restart)
                        goto restart;
 

Quoted 4 lines

kernel/softirq.c asmlinkage __visible void __softirq_entry __do_softirq(void) · line 305

Quoted 1 line

kernel/softirq.c +0 -0

                wakeup_softirqd();

You just introduced a 2ms preempt-disable region I think, that's not cool for PREEMPT and a plain bug on PREEMPT_RT.

I actually think that he whole "time_before()" is garbage and should be removed.

If doing 10 rounds of softirq processing takes several jiffies, you have bigger issue.

Looking at the history, the original time_before() was _replacing_ the "max_retry". See commit c10d73671ad3 ("softirq: reduce latencies").

But then "max_restart" was put back in because jiffies just isn't reliable in the first place.

And I think *this* is the real issue. Networking does a *shitton* of things in softirq context, and the problem is not the softirq code, it's the networking code. The softirq code doesn't understand that the networking code does billions of operations, and one softirq can take ages and ages because it is handling millions of packets.

My feeling is that it's the networking code that should notice "I have a billion packets in my softirq queue, and I just keep getting more, now *I* should start doing things in a thread instead".

Because none of those limits really make sense for any of the other softirq users. Everybody else just wants a low-latency "soft interrupt", not some stupid thread. Who knew? Maybe the name should have given the thing away?

If you want a thread, use a tasklet and add_task_work().

Maybe networking should reconsider its use of softirqs entirely?

How are we supposed to know this particular workload is problematic for innocent user threads on the same cpu ?

Quoted 3 lines · Eric Dumazet

How are we supposed to know this particular workload is problematic for innocent user threads on the same cpu ?

Put that question another way: how is the _softirq_ code supposed to know?

If you can't know, then the softirq code definitely can't know either. It has even less information. It doesn't know about NAPI, it doesn't know about irq steering, it doesn't know anything at all.

That was the purpose on the last patch : As soon as ksoftirqd is scheduled (by some kind of jitter in the 99,000 pps workload, or antagonist wakeup), we then switch to a mode where process scheduler can make decisions based on threads prios and cpu usage.

Then, as soon as the load was able to finish in its quantum the pending irqs, we re-enter the mode
where softirq are immediately serviced.

Quoted 4 lines · Eric Dumazet

That was the purpose on the last patch : As soon as ksoftirqd is scheduled (by some kind of jitter in the 99,000 pps workload, or antagonist wakeup), we then switch to a mode where process scheduler can make decisions based on threads prios and cpu usage.

Yeah, but that really screws up everybody else.

It really is a soft *interrupt*. That was what it was designed for. The thread handling is not primary, it's literally a fallback to avoid complete starvation.

The fact that networking has now - for several years - tried to make it some kind of thread and get fairness with user threads is all entirely antithetical to what softirq was designed for.

Quoted 3 lines

Then, as soon as the load was able to finish in its quantum the pending irqs, we re-enter the mode
where softirq are immediately serviced.

Except that's not at all how the code works.

As I pointed out, the softirq thread can be scheduled away, but the "softiq_running()" wil stilll return true - and the networking code has now screwed up all the *other* softirqs too!

I really suspect that what networking wants is more like the workqueues. Or at least more isolation between different softirq users, but that's fairly fundamentally hard, given how softirqs are designed.

Note that when I implemented TCP Small queues, I did experiments between using a work queue or a tasklet, and workqueues added unacceptable P99 latencies, when many user threads are competing with kernel threads.

Quoted 7 lines · Eric Dumazet

Note that when I implemented TCP Small queues, I did experiments between
using a work queue or a tasklet, and workqueues added unacceptable P99
latencies, when many user threads are competing with kernel threads.

Yes.

So I think one solution might be to have a hybrid system, where we do the softirq's synchronously normally (which is what you really want for good latency).

But then fall down on a threaded model - but that fallback case should
be per-softirq, not global. So if one softirq uses a lot of CPU time, that shouldn't affect the latency of other softirqs.

So maybe we could get rid of the per-cpu ksoftirqd entirely, and replace it with with per-cpu and per-softirq workqueues?

Would something like that sound sane?

Just a SMOP/SMOT (small matter of programming/testing).

Quoted 26 lines · Linus Torvalds

Quoted 7 lines · Eric Dumazet

Note that when I implemented TCP Small queues, I did experiments between
using a work queue or a tasklet, and workqueues added unacceptable P99
latencies, when many user threads are competing with kernel threads.

Yes.

So I think one solution might be to have a hybrid system, where we do the softirq's synchronously normally (which is what you really want for good latency).

But then fall down on a threaded model - but that fallback case should
be per-softirq, not global. So if one softirq uses a lot of CPU time, that shouldn't affect the latency of other softirqs.

So maybe we could get rid of the per-cpu ksoftirqd entirely, and replace it with with per-cpu and per-softirq workqueues?

Would something like that sound sane?

Quoted 1 line

Just a SMOP/SMOT (small matter of programming/testing).

I could try to write a PoC for that..
What should be the trigger to fall into workqueue? How to tell if there're too many softirqs of the kind? Current logic with if (pending) in the end of __do_softirq() looks working selectively..
It looks to be still possible to starve a cpu.

I could try to write a PoC for that..
What should be the trigger to fall into workqueue? How to tell if there're too many softirqs of the kind?

Quoted 4 lines · Dmitry Safonov

I could try to write a PoC for that..
What should be the trigger to fall into workqueue? How to tell if there're too many softirqs of the kind?

I suspect it would have to be time-based, probably using the scheduler clock.

Most softirqs are really really small. So just counting them probably isn't all that meaningful, although the count is good as a fallback (as shown by the jiffy issues).

The good news is that we only have a fairly small handful of softirqs,
so counting/timing them separately is still mainly a pretty small array (which needs to be percpu, of course).

Quoted 7 lines · Linus Torvalds

Quoted 4 lines · Dmitry Safonov

I could try to write a PoC for that..
What should be the trigger to fall into workqueue? How to tell if there're too many softirqs of the kind?

Quoted 2 lines

I suspect it would have to be time-based, probably using the scheduler clock.

I thought about this, but I was a bit afraid of how much pricey it would be recalculate it each clock. Well, might just try to write that and measure the impact.

Quoted 8 lines

Most softirqs are really really small. So just counting them probably isn't all that meaningful, although the count is good as a fallback (as shown by the jiffy issues).

The good news is that we only have a fairly small handful of softirqs,
so counting/timing them separately is still mainly a pretty small array (which needs to be percpu, of course).

Quoted 9 lines · Dmitry Safonov

Quoted 8 lines · Linus Torvalds

On Thu, Jan 11, 2018 at 12:34 PM, Dmitry Safonov <[email protected]

Quoted 1 line

Quoted 4 lines · wrote:

I could try to write a PoC for that..
What should be the trigger to fall into workqueue? How to tell if there're too many softirqs of the kind?

Quoted 18 lines

Quoted 2 lines

I suspect it would have to be time-based, probably using the scheduler clock.

I thought about this, but I was a bit afraid of how much pricey it would be recalculate it each clock. Well, might just try to write that
and measure the impact.

Quoted 10 lines

Most softirqs are really really small. So just counting them probably
isn't all that meaningful, although the count is good as a fallback
(as shown by the jiffy issues).

The good news is that we only have a fairly small handful of softirqs,
so counting/timing them separately is still mainly a pretty small array (which needs to be percpu, of course).

Note that using (scheduler) clock might also help to break net_rx_action()
not on a stupid netdev_budget, but on a more precise time limit as well.

netdev_budget of 300 packets is quite big :/

(The time_limit based on jiffies + 2 does not work on hosts with one cpu, since jiffies wont make progress while net_rx_action() is running)

Quoted 35 lines · Eric Dumazet

Quoted 9 lines · Dmitry Safonov

Quoted 8 lines · Linus Torvalds

On Thu, Jan 11, 2018 at 12:34 PM, Dmitry Safonov <[email protected]

Quoted 1 line

Quoted 4 lines · wrote:

I could try to write a PoC for that..
What should be the trigger to fall into workqueue? How to tell if there're too many softirqs of the kind?

Quoted 18 lines

Quoted 2 lines

I suspect it would have to be time-based, probably using the scheduler clock.

I thought about this, but I was a bit afraid of how much pricey it would be recalculate it each clock. Well, might just try to write that
and measure the impact.

Quoted 10 lines

Most softirqs are really really small. So just counting them probably
isn't all that meaningful, although the count is good as a fallback
(as shown by the jiffy issues).

The good news is that we only have a fairly small handful of softirqs,
so counting/timing them separately is still mainly a pretty small array (which needs to be percpu, of course).

Note that using (scheduler) clock might also help to break net_rx_action()
not on a stupid netdev_budget, but on a more precise time limit as well.

Quoted 5 lines

netdev_budget of 300 packets is quite big :/

(The time_limit based on jiffies + 2 does not work on hosts with one cpu, since jiffies wont make progress while net_rx_action() is running)

Thanks for the details, Eric.
I'll try to come up with poc if no one beats me at it.

Quoted 4 lines · Eric Dumazet

Note that when I implemented TCP Small queues, I did experiments between using a work queue or a tasklet, and workqueues added unacceptable P99 latencies, when many user threads are competing with kernel threads.

Yes.

So I think one solution might be to have a hybrid system, where we do the softirq's synchronously normally (which is what you really want for good latency).

But then fall down on a threaded model - but that fallback case should be per-softirq, not global. So if one softirq uses a lot of CPU time, that shouldn't affect the latency of other softirqs.

So maybe we could get rid of the per-cpu ksoftirqd entirely, and replace it with with per-cpu and per-softirq workqueues?

Quoted 16 lines · Linus Torvalds

Quoted 4 lines · Eric Dumazet

Note that when I implemented TCP Small queues, I did experiments between using a work queue or a tasklet, and workqueues added unacceptable P99 latencies, when many user threads are competing with kernel threads.

Yes.

So I think one solution might be to have a hybrid system, where we do the softirq's synchronously normally (which is what you really want for good latency).

But then fall down on a threaded model - but that fallback case should be per-softirq, not global. So if one softirq uses a lot of CPU time, that shouldn't affect the latency of other softirqs.

Quoted 2 lines

So maybe we could get rid of the per-cpu ksoftirqd entirely, and replace it with with per-cpu and per-softirq workqueues?

How would that be better than what RT used to do, and I still do for my RT kernels via boot option, namely split ksoftirqd into per-softirq threads.

Quoted 4 lines · Mike Galbraith

How would that be better than what RT used to do, and I still do for my RT kernels via boot option, namely split ksoftirqd into per-softirq threads.

Since we mention it; one of the problems RT has is that all RX is through a single softirq context, which generates a priority inversion between devices.

Quoted 9 lines · Peter Zijlstra

Quoted 4 lines · Mike Galbraith

How would that be better than what RT used to do, and I still do for my RT kernels via boot option, namely split ksoftirqd into per-softirq threads.

Since we mention it; one of the problems RT has is that all RX is through a single softirq context, which generates a priority inversion between devices.

Oh, yes, that is complete and utter shit. Not acceptable at all.

As mentioned, it really would need to be per-cpu _and_ per-softirq.

Which is why I thought workqueues might be the thing. Whatever RT is doing is apparently just pure and utter garbage.

Nah, a misunderstanding happened. RT that still offers full threading creates per-softirq threads per cpu. The regular trees split ksoftirqd into only two threads per cpu, one processes timer/hrtimer softriqs, the other processes the rest.

Quoted 5 lines · Mike Galbraith

Nah, a misunderstanding happened. RT that still offers full threading creates per-softirq threads per cpu. The regular trees split ksoftirqd into only two threads per cpu, one processes timer/hrtimer softriqs, the other processes the rest.

Ok, that sounds like it should work, but it also sounds like it's very specific to RT itself.

For example, the dvb issue was not about the timer softirqs, but about the tasklet ones.

So maybe we wouldn't need to split it for _every_ softirq, but we'd need to split it more than just along the timer case.

And it does sound a bit excessive to have ten fixed threads for every CPU. The days when tens of CPU's meant "huge system" are gone. These days it can be a phone.

Quoted 16 lines · Linus Torvalds

Quoted 5 lines · Mike Galbraith

Nah, a misunderstanding happened. RT that still offers full threading creates per-softirq threads per cpu. The regular trees split ksoftirqd into only two threads per cpu, one processes timer/hrtimer softriqs, the other processes the rest.

Ok, that sounds like it should work, but it also sounds like it's very specific to RT itself.

For example, the dvb issue was not about the timer softirqs, but about the tasklet ones.

So maybe we wouldn't need to split it for _every_ softirq, but we'd need to split it more than just along the timer case.

Quoted 3 lines

And it does sound a bit excessive to have ten fixed threads for every CPU. The days when tens of CPU's meant "huge system" are gone. These days it can be a phone.

Yeah, it is excessive more often than not. You get to prioritize, and segregate, which is nice, but you pay for it.

Quoted 17 lines · Linus Torvalds

Quoted 16 lines · Mike Galbraith

Quoted 5 lines

Nah, a misunderstanding happened. RT that still offers full threading creates per-softirq threads per cpu. The regular trees split ksoftirqd into only two threads per cpu, one processes timer/hrtimer softriqs, the other processes the rest.

Ok, that sounds like it should work, but it also sounds like it's very specific to RT itself.

For example, the dvb issue was not about the timer softirqs, but about the tasklet ones.

So maybe we wouldn't need to split it for _every_ softirq, but we'd need to split it more than just along the timer case.

Quoted 6 lines

Quoted 3 lines

And it does sound a bit excessive to have ten fixed threads for every CPU. The days when tens of CPU's meant "huge system" are gone. These days it can be a phone.

Yeah, it is excessive more often than not. You get to prioritize, and segregate, which is nice, but you pay for it.

BTW, much of the softirq load in RT is processed by the raising task.

      tbench_srv-6985  [000] d...112   293.902511: softirq_raise: vec=3 [action=NET_RX]
      tbench_srv-6985  [000] .....13   293.902511: softirq_entry: vec=3 [action=NET_RX]
      tbench_srv-6985  [000] .....13   293.902515: softirq_exit: vec=3 [action=NET_RX]
          tbench-6984  [003] d...112   293.902520: softirq_raise: vec=3 [action=NET_RX]
          tbench-6984  [003] .....13   293.902520: softirq_entry: vec=3 [action=NET_RX]
          tbench-6984  [003] .....13   293.902523: softirq_exit: vec=3 [action=NET_RX]

I wanted to chime in about this earlier, and make it clear that it isn't just IRQs that can trigger softirqs. User context actions in the kernel can trigger softirqs too.

It was in some way always a "poor mans interrupt thread" (with no blocking like a real thread context, but at least not impacting actual interrupt latency).

That said, this made me wonder a bit. I wonder how bounded the latency is for raising a softirq from process context. We only _check_ the softirq on the last hardirq exit, I think.

I wonder if we should run softirqs on return to user mode (and make softirq set a thread flag if not in interrupt context).

From: Linus Torvalds <[email protected]>
Date: Wed, 17 Jan 2018 13:06:58 -0800

Quoted 3 lines

It was in some way always a "poor mans interrupt thread" (with no blocking like a real thread context, but at least not impacting actual interrupt latency).

Or in this loopback device case (and tunnel decapsulation) a poor man's longjmp, releasing the current stack frame to keep the depth in check.

Anyways...

Quoted 3 lines

That said, this made me wonder a bit. I wonder how bounded the latency is for raising a softirq from process context. We only _check_ the softirq on the last hardirq exit, I think.

System call return checks it, otherwise this situation would be completely bolixed.

Quoted 3 lines

I wonder if we should run softirqs on return to user mode (and make softirq set a thread flag if not in interrupt context).

I'm pretty sure we already do.

raise_softirq() -> raise_softirq_irqoff()

        set_softirq_bit();

        if (!in_interrupt())
                wake_softirqd();

So if the caller is not in hard or soft interrupt context, which includes bottom half disabled regions softirqd is woken.

Quoted 9 lines · Thomas Gleixner

raise_softirq() -> raise_softirq_irqoff()

        set_softirq_bit();

        if (!in_interrupt())
                wake_softirqd();

So if the caller is not in hard or soft interrupt context, which includes bottom half disabled regions softirqd is woken.

That does seem unnecessarily expensive, and maybe we could just do it with thread flag (TIF_NOTIFY_RESUME or whatever).

In fact, that was what I *thought* we did. Maybe I just remember some historical behavior.

Since networking seems to largely prefer softirqd anyway, maybe that wake_softirqd() is the right thing to do anyway.

Quoted 17 lines · Linus Torvalds

Quoted 9 lines · Thomas Gleixner

raise_softirq() -> raise_softirq_irqoff()

        set_softirq_bit();

        if (!in_interrupt())
                wake_softirqd();

So if the caller is not in hard or soft interrupt context, which includes bottom half disabled regions softirqd is woken.

That does seem unnecessarily expensive, and maybe we could just do it with thread flag (TIF_NOTIFY_RESUME or whatever).

In fact, that was what I *thought* we did. Maybe I just remember some historical behavior.

Quoted 2 lines

Since networking seems to largely prefer softirqd anyway, maybe that wake_softirqd() is the right thing to do anyway.

Well, but we only do it when we are not in a bh disabled region. The places where thread context raises the network softirqs is usually inside a bh disabled region, so the softirq is executed on local_bh_enable(). The thread is woken up rarely.

Quoted 19 lines · Linus Torvalds

Quoted 18 lines · Thomas Gleixner

Quoted 9 lines

raise_softirq() -> raise_softirq_irqoff()

        set_softirq_bit();

        if (!in_interrupt())
                wake_softirqd();

So if the caller is not in hard or soft interrupt context, which includes bottom half disabled regions softirqd is woken.

That does seem unnecessarily expensive, and maybe we could just do it with thread flag (TIF_NOTIFY_RESUME or whatever).

In fact, that was what I *thought* we did. Maybe I just remember some historical behavior.

Quoted 7 lines

Quoted 2 lines

Since networking seems to largely prefer softirqd anyway, maybe that wake_softirqd() is the right thing to do anyway.

Well, but we only do it when we are not in a bh disabled region. The places where thread context raises the network softirqs is usually inside a bh disabled region, so the softirq is executed on local_bh_enable(). The thread is woken up rarely.

There is also the netif_rx_ni() stuff.

Can't remember right now why it is not using local_bh_{diable,enable}() pair instead
of preempt_disable() ... if (local_softirq_pending()) do_softirq();

From: Eric Dumazet <[email protected]>
Date: Wed, 17 Jan 2018 14:02:43 -0800

Quoted 30 lines · Thomas Gleixner

Quoted 19 lines · Linus Torvalds

Quoted 18 lines · Thomas Gleixner

Quoted 9 lines

raise_softirq() -> raise_softirq_irqoff()

        set_softirq_bit();

        if (!in_interrupt())
                wake_softirqd();

So if the caller is not in hard or soft interrupt context, which includes bottom half disabled regions softirqd is woken.

That does seem unnecessarily expensive, and maybe we could just do it with thread flag (TIF_NOTIFY_RESUME or whatever).

In fact, that was what I *thought* we did. Maybe I just remember some historical behavior.

Quoted 7 lines

Quoted 2 lines

Since networking seems to largely prefer softirqd anyway, maybe that wake_softirqd() is the right thing to do anyway.

Well, but we only do it when we are not in a bh disabled region. The places where thread context raises the network softirqs is usually inside a bh disabled region, so the softirq is executed on local_bh_enable(). The thread is woken up rarely.

There is also the netif_rx_ni() stuff.

Quoted 3 lines

Can't remember right now why it is not using local_bh_{diable,enable}() pair instead
of preempt_disable() ... if (local_softirq_pending()) do_softirq();

Hmmm, that code predates the initial GIT repository build.

I do remember we had some back and forth with that stuff.

From: Eric Dumazet <[email protected]>
Date: Wed, 17 Jan 2018 14:02:43 -0800

Quoted 5 lines

There is also the netif_rx_ni() stuff.

Can't remember right now why it is not using local_bh_{diable,enable}() pair instead
of preempt_disable() ... if (local_softirq_pending()) do_softirq();

Hmmm, that code predates the initial GIT repository build.

I do remember we had some back and forth with that stuff.

Quoted 8 lines · David Miller

From: Eric Dumazet <[email protected]>
Date: Wed, 17 Jan 2018 14:02:43 -0800

Quoted 5 lines

There is also the netif_rx_ni() stuff.

Can't remember right now why it is not using local_bh_{diable,enable}() pair instead
of preempt_disable() ... if (local_softirq_pending()) do_softirq();

Quoted 3 lines

Hmmm, that code predates the initial GIT repository build.

I do remember we had some back and forth with that stuff.

So I did a little research and tried to replace preempt_disable() with local_bh_disable() [0] a while ago.

[0] https://lkml.kernel.org/r/[email protected]

Quoted 4 lines

That said, this made me wonder a bit. I wonder how bounded the latency is for raising a softirq from process context. We only _check_ the softirq on the last hardirq exit, I think.

System call return checks it, otherwise this situation would be completely bolixed.

Quoted 5 lines · David Miller

Quoted 4 lines

That said, this made me wonder a bit. I wonder how bounded the latency is for raising a softirq from process context. We only _check_ the softirq on the last hardirq exit, I think.

Quoted 2 lines

System call return checks it, otherwise this situation would be completely bolixed.

That's what I thought too. But then I went and looked, and I can't find it.

But you're probably right, and I just missed it.

Quoted 5 lines · Mike Galbraith

Nah, a misunderstanding happened. RT that still offers full threading creates per-softirq threads per cpu. The regular trees split ksoftirqd into only two threads per cpu, one processes timer/hrtimer softriqs, the other processes the rest.

Ok, that sounds like it should work, but it also sounds like it's very specific to RT itself.

For example, the dvb issue was not about the timer softirqs, but about the tasklet ones.

So maybe we wouldn't need to split it for _every_ softirq, but we'd need to split it more than just along the timer case.

And it does sound a bit excessive to have ten fixed threads for every CPU. The days when tens of CPU's meant "huge system" are gone. These days it can be a phone.

Quoted 13 lines · Linus Torvalds

Quoted 5 lines · Mike Galbraith

Nah, a misunderstanding happened. RT that still offers full threading creates per-softirq threads per cpu. The regular trees split ksoftirqd into only two threads per cpu, one processes timer/hrtimer softriqs, the other processes the rest.

Ok, that sounds like it should work, but it also sounds like it's very specific to RT itself.

For example, the dvb issue was not about the timer softirqs, but about the tasklet ones.

Quoted 6 lines

So maybe we wouldn't need to split it for _every_ softirq, but we'd need to split it more than just along the timer case.

And it does sound a bit excessive to have ten fixed threads for every CPU. The days when tens of CPU's meant "huge system" are gone. These days it can be a phone.

That's true.

One thing which might worth a try is utilizing the threaded irq infrastructure and that might also pave the way to address Peters request for per device splitting. I've experimented with that in the past but never got around to finish it completely. I should have half baken patches somewhere in the poison cabinet.

Quoted 16 lines · Linus Torvalds

Quoted 4 lines · Eric Dumazet

Note that when I implemented TCP Small queues, I did experiments between using a work queue or a tasklet, and workqueues added unacceptable P99 latencies, when many user threads are competing with kernel threads.

Yes.

So I think one solution might be to have a hybrid system, where we do the softirq's synchronously normally (which is what you really want for good latency).

But then fall down on a threaded model - but that fallback case should be per-softirq, not global. So if one softirq uses a lot of CPU time, that shouldn't affect the latency of other softirqs.

Quoted 2 lines

So maybe we could get rid of the per-cpu ksoftirqd entirely, and replace it with with per-cpu and per-softirq workqueues?

How would that be better than what RT used to do, and I still do for my RT kernels via boot option, namely split ksoftirqd into per-softirq threads.

Quoted 20 lines · Mike Galbraith

Quoted 18 lines · Linus Torvalds

Quoted 4 lines · Eric Dumazet

Note that when I implemented TCP Small queues, I did experiments between using a work queue or a tasklet, and workqueues added unacceptable P99 latencies, when many user threads are competing with kernel threads.

Yes.

So I think one solution might be to have a hybrid system, where we do the softirq's synchronously normally (which is what you really want for good latency).

But then fall down on a threaded model - but that fallback case should be per-softirq, not global. So if one softirq uses a lot of CPU time, that shouldn't affect the latency of other softirqs.

So maybe we could get rid of the per-cpu ksoftirqd entirely, and replace it with with per-cpu and per-softirq workqueues?

Quoted 3 lines

How would that be better than what RT used to do, and I still do for my RT kernels via boot option, namely split ksoftirqd into per-softirq threads.

Workqueue are probably more simple. Unless you need to set specific prios to your ksoftirqds? Not sure if that's tunable on workqueues.

Quoted 21 lines · Frederic Weisbecker

Quoted 20 lines · Mike Galbraith

Quoted 18 lines · Linus Torvalds

Quoted 4 lines · Eric Dumazet

Note that when I implemented TCP Small queues, I did experiments between using a work queue or a tasklet, and workqueues added unacceptable P99 latencies, when many user threads are competing with kernel threads.

Yes.

So I think one solution might be to have a hybrid system, where we do the softirq's synchronously normally (which is what you really want for good latency).

But then fall down on a threaded model - but that fallback case should be per-softirq, not global. So if one softirq uses a lot of CPU time, that shouldn't affect the latency of other softirqs.

So maybe we could get rid of the per-cpu ksoftirqd entirely, and replace it with with per-cpu and per-softirq workqueues?

Quoted 6 lines

Quoted 3 lines

How would that be better than what RT used to do, and I still do for my RT kernels via boot option, namely split ksoftirqd into per-softirq threads.

Workqueue are probably more simple. Unless you need to set specific prios to your ksoftirqds? Not sure if that's tunable on workqueues.

No, you can't prioritize workqueues, and they spawn threads whenever they bloody well feel like.

I carry a hack to give users minimal control over kthread/workqueue priority.  Very handy thing to have, especially if you're doing high utilization stuff, and would prefer your box actually survive it.

Quoted 29 lines · Mike Galbraith

Quoted 27 lines · Frederic Weisbecker

Quoted 17 lines · Mike Galbraith

Quoted 16 lines · Linus Torvalds

Quoted 4 lines · Eric Dumazet

Note that when I implemented TCP Small queues, I did experiments between using a work queue or a tasklet, and workqueues added unacceptable P99 latencies, when many user threads are competing with kernel threads.

Yes.

So I think one solution might be to have a hybrid system, where we do the softirq's synchronously normally (which is what you really want for good latency).

But then fall down on a threaded model - but that fallback case should be per-softirq, not global. So if one softirq uses a lot of CPU time, that shouldn't affect the latency of other softirqs.

Quoted 6 lines

Quoted 2 lines

So maybe we could get rid of the per-cpu ksoftirqd entirely, and replace it with with per-cpu and per-softirq workqueues?

How would that be better than what RT used to do, and I still do for my RT kernels via boot option, namely split ksoftirqd into per-softirq threads.

Workqueue are probably more simple. Unless you need to set specific prios to your ksoftirqds? Not sure if that's tunable on workqueues.

Quoted 6 lines

No, you can't prioritize workqueues, and they spawn threads whenever they bloody well feel like.

I carry a hack to give users minimal control over kthread/workqueue priority.  Very handy thing to have, especially if you're doing high utilization stuff, and would prefer your box actually survive it.

How useful system_highpri_wq can be in this regard?

Quoted 3 lines · Linus Torvalds

So maybe we could get rid of the per-cpu ksoftirqd entirely, and replace it with with per-cpu and per-softirq workqueues?

How would that be better than what RT used to do, and I still do for my RT kernels via boot option, namely split ksoftirqd into per-softirq threads.