Softirq:avoid large sched delay from the pending softirqs
PATCH v4 1/1 Softirq:avoid large sched delay from the pending softirqs
From: jun qian <[email protected]>
When get the pending softirqs, it need to process all the pending softirqs in the while loop. If the processing time of each pending softirq is need more than 2 msec in this loop, or one of the softirq will running a long time, according to the original code logic, it will process all the pending softirqs without wakeuping ksoftirqd, which will cause a relatively large scheduling delay on the corresponding CPU, which we do not wish to see. The patch will check the total time to process pending softirq, if the time exceeds 2 ms we need to wakeup the ksofirqd to aviod large sched delay.
--- kernel/softirq.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-)
kernel/softirq.c +20 -8
@@ -25,6 +25,7 @@#include <linux/smpboot.h> #include <linux/tick.h> #include <linux/irq.h> +#include <linux/sched/clock.h> #define CREATE_TRACE_POINTS #include <trace/events/irq.h>
@@ -200,17 +201,15 @@ void __local_bh_enable_ip(unsigned long ip, unsigned int cnt)/* * We restart softirq processing for at most MAX_SOFTIRQ_RESTART times, * but break the loop if need_resched() is set or after 2 ms. -* The MAX_SOFTIRQ_TIME provides a nice upper bound in most cases, but in -* certain cases, such as stop_machine(), jiffies may cease to -* increment and so we need the MAX_SOFTIRQ_RESTART limit as -* well to make sure we eventually return from this method. +* In the loop, if the processing time of the softirq has exceeded 2 +* milliseconds, we also need to break the loop to wakeup the ksofirqd. * * These limits have been established via experimentation. * The two things to balance is latency against fairness - * we want to handle softirqs as soon as possible, but they * should not be able to lock up the box. */ -#define MAX_SOFTIRQ_TIME msecs_to_jiffies(2) +#define MAX_SOFTIRQ_TIME_NS 2000000#define MAX_SOFTIRQ_RESTART 10#ifdef CONFIG_TRACE_IRQFLAGS
@@ -248,7 +247,7 @@ static inline void lockdep_softirq_end(bool in_hardirq) { } asmlinkage __visible void __softirq_entry __do_softirq(void) { -unsigned long end = jiffies +MAX_SOFTIRQ_TIME ; +u64 end =sched_clock () +MAX_SOFTIRQ_TIME_NS ;unsigned long old_flags = current->flags;int max_restart =MAX_SOFTIRQ_RESTART ;struct softirq_action *h;
@@ -299,6 +298,19 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) } h++; pending >>= softirq_bit; + +/* +* the softirq's action has been running for too much time +* so it may need to wakeup the ksoftirqd +*/ +if (need_resched () &&sched_clock () > end) { +/* +* Ensure that the remaining pending bits are +* handled. +*/ +or_softirq_pending (pending << (vec_nr +1 )); +break ; + } }if (__this_cpu_read (ksoftirqd) == current)
@@ -307,8 +319,8 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) pending =local_softirq_pending ();if (pending) { -if (time_before (jiffies, end) && !need_resched () && - --max_restart) +if (!need_resched () && --max_restart && +sched_clock () <= end) goto restart;wakeup_softirqd ();
Signature · 1 line
1.8.3.1
Quoted 81 lines · On Fri, Jul 24, 2020 at 10:31:23AM -0400, [email protected] wrote:
From: jun qian <[email protected]>
When get the pending softirqs, it need to process all the pending softirqs in the while loop. If the processing time of each pending softirq is need more than 2 msec in this loop, or one of the softirq will running a long time, according to the original code logic, it will process all the pending softirqs without wakeuping ksoftirqd, which will cause a relatively large scheduling delay on the corresponding CPU, which we do not wish to see. The patch will check the total time to process pending softirq, if the time exceeds 2 ms we need to wakeup the ksofirqd to aviod large sched delay.
Signed-off-by jun qian <[email protected]>--- kernel/softirq.c | 28 ++++++++++++++++++++-------- 1 file changed, 20 insertions(+), 8 deletions(-)kernel/softirq.c +18
-6@@ -25,6 +25,7 @@#include <linux/smpboot.h> #include <linux/tick.h> #include <linux/irq.h> +#include <linux/sched/clock.h> #define CREATE_TRACE_POINTS #include <trace/events/irq.h> @@ -200,17 +201,15 @@ void __local_bh_enable_ip(unsigned long ip, unsigned int cnt)/* * We restart softirq processing for at most MAX_SOFTIRQ_RESTART times, * but break the loop if need_resched() is set or after 2 ms. -* The MAX_SOFTIRQ_TIME provides a nice upper bound in most cases, but in -* certain cases, such as stop_machine(), jiffies may cease to -* increment and so we need the MAX_SOFTIRQ_RESTART limit as -* well to make sure we eventually return from this method. +* In the loop, if the processing time of the softirq has exceeded 2 +* milliseconds, we also need to break the loop to wakeup the ksofirqd. * * These limits have been established via experimentation. * The two things to balance is latency against fairness - * we want to handle softirqs as soon as possible, but they * should not be able to lock up the box. */ -#define MAX_SOFTIRQ_TIME msecs_to_jiffies(2) +#define MAX_SOFTIRQ_TIME_NS 2000000#define MAX_SOFTIRQ_RESTART 10#ifdef CONFIG_TRACE_IRQFLAGS @@ -248,7 +247,7 @@ static inline void lockdep_softirq_end(bool in_hardirq) { } asmlinkage __visible void __softirq_entry __do_softirq(void) { -unsigned long end = jiffies +MAX_SOFTIRQ_TIME ; +u64 end =sched_clock () +MAX_SOFTIRQ_TIME_NS ;unsigned long old_flags = current->flags;int max_restart =MAX_SOFTIRQ_RESTART ;struct softirq_action *h;@@ -299,6 +298,19 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) } h++; pending >>= softirq_bit; + +/* +* the softirq's action has been running for too much time +* so it may need to wakeup the ksoftirqd +*/ +if (need_resched () &&sched_clock () > end) { +/* +* Ensure that the remaining pending bits are +* handled. +*/ +or_softirq_pending (pending << (vec_nr +1 )); +break ; + } }if (__this_cpu_read (ksoftirqd) == current)
kernel/softirq.c asmlinkage __visible void __softirq_entry __do_softirq(void) · line 326
Quoted 10 lines
kernel/softirq.c +2
-2@@ -307,8 +319,8 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) pending =local_softirq_pending ();if (pending) { -if (time_before (jiffies, end) && !need_resched () && - --max_restart) +if (!need_resched () && --max_restart && +sched_clock () <= end) goto restart;wakeup_softirqd ();
To me it looks OKr.
Thank you for fixing the "2 msec resolution case".
--
Vlad Rezki
Qian,
Quoted 10 lines · [email protected] writes:
/*
* We restart softirq processing for at mostMAX_SOFTIRQ_RESTARTtimes,
* but break the loop ifneed_resched()is set or after 2 ms.
- * TheMAX_SOFTIRQ_TIMEprovides a nice upper bound in most cases, but in
- * certain cases, such asstop_machine(), jiffies may cease to
- * increment and so we need theMAX_SOFTIRQ_RESTARTlimit as
- * well to make sure we eventually return from this method.
+ * In the loop, if the processing time of the softirq has exceeded 2
+ * milliseconds, we also need to break the loop to wakeup the ksofirqd.
You are removing the MAX_SOFTIRQ_RESTART limit explanation and I rather have MAX_SOFTIRQ_TIME_NS there than '2 milliseconds' in case the value gets adjusted later on. Also while sched_clock() is granular on many systems it still can be jiffies based and then the above problem persists.
Quoted 5 lines
kernel/softirq.c +1
-0@@ -299,6 +298,19 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) } h++; pending >>= softirq_bit; +
kernel/softirq.c asmlinkage __visible void __softirq_entry __do_softirq(void) · line 311
Quoted 10 lines
kernel/softirq.c +10
-0+/* +* the softirq's action has been running for too much time +* so it may need to wakeup the ksoftirqd +*/ +if (need_resched () &&sched_clock () > end) { +/* +* Ensure that the remaining pending bits are +* handled. +*/ +or_softirq_pending (pending << (vec_nr +1 ));
To or the value interrupts need to be disabled because otherwise you can lose a bit when an interrupt happens in the middle of the RMW operation and raises a softirq which is not in @pending and not in the per CPU local softirq pending storage.
There is another problem. Assume bit 0 and 1 are pending when the processing starts. Now it breaks out after bit 0 has been handled and stores back bit 1 as pending. Before ksoftirqd runs bit 0 gets raised again. ksoftirqd runs and handles bit 0, which takes more than the timeout. As a result the bit 0 processing can starve all other softirqs.
So this needs more thought.
Thanks,
tglx
Quoted 42 lines · Thomas Gleixner
Qian,
Quoted 10 lines · [email protected] writes:
/*
* We restart softirq processing for at mostMAX_SOFTIRQ_RESTARTtimes,
* but break the loop ifneed_resched()is set or after 2 ms.
- * TheMAX_SOFTIRQ_TIMEprovides a nice upper bound in most cases, but in
- * certain cases, such asstop_machine(), jiffies may cease to
- * increment and so we need theMAX_SOFTIRQ_RESTARTlimit as
- * well to make sure we eventually return from this method.
+ * In the loop, if the processing time of the softirq has exceeded 2
+ * milliseconds, we also need to break the loop to wakeup the ksofirqd.You are removing the
MAX_SOFTIRQ_RESTARTlimit explanation and I rather haveMAX_SOFTIRQ_TIME_NSthere than '2 milliseconds' in case the value gets adjusted later on. Also whilesched_clock()is granular on many systems it still can be jiffies based and then the above problem persists.Quoted 5 lines
kernel/softirq.c +1
-0@@ -299,6 +298,19 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) } h++; pending >>= softirq_bit; +kernel/softirq.c asmlinkage __visible void __softirq_entry __do_softirq(void) · line 311
Quoted 10 lines
kernel/softirq.c +10
-0+/* +* the softirq's action has been running for too much time +* so it may need to wakeup the ksoftirqd +*/ +if (need_resched () &&sched_clock () > end) { +/* +* Ensure that the remaining pending bits are +* handled. +*/ +or_softirq_pending (pending << (vec_nr +1 ));To or the value interrupts need to be disabled because otherwise you can lose a bit when an interrupt happens in the middle of the RMW operation and raises a softirq which is not in @pending and not in the per CPU local softirq pending storage.
Quoted 6 lines
There is another problem. Assume bit 0 and 1 are pending when the processing starts. Now it breaks out after bit 0 has been handled and stores back bit 1 as pending. Before ksoftirqd runs bit 0 gets raised again. ksoftirqd runs and handles bit 0, which takes more than the timeout. As a result the bit 0 processing can starve all other softirqs.
I got it. I will try to slove this problem. Thanks.
Quoted 5 lines
So this needs more thought.
Thanks,
tglx
Quoted 33 lines · Thomas Gleixner
Qian,
Quoted 10 lines · [email protected] writes:
/*
* We restart softirq processing for at mostMAX_SOFTIRQ_RESTARTtimes,
* but break the loop ifneed_resched()is set or after 2 ms.
- * TheMAX_SOFTIRQ_TIMEprovides a nice upper bound in most cases, but in
- * certain cases, such asstop_machine(), jiffies may cease to
- * increment and so we need theMAX_SOFTIRQ_RESTARTlimit as
- * well to make sure we eventually return from this method.
+ * In the loop, if the processing time of the softirq has exceeded 2
+ * milliseconds, we also need to break the loop to wakeup the ksofirqd.You are removing the
MAX_SOFTIRQ_RESTARTlimit explanation and I rather haveMAX_SOFTIRQ_TIME_NSthere than '2 milliseconds' in case the value gets adjusted later on. Also whilesched_clock()is granular on many systems it still can be jiffies based and then the above problem persists.Quoted 12 lines
kernel/softirq.c +8
-0@@ -299,6 +298,19 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) } h++; pending >>= softirq_bit; + +/* +* the softirq's action has been running for too much time +* so it may need to wakeup the ksoftirqd +*/ +if (need_resched () &&sched_clock () > end) { +/* +* Ensure that the remaining pending bits are
Quoted 9 lines
kernel/softirq.c asmlinkage __visible void __softirq_entry __do_softirq(void) · line 311
Quoted 3 lines
kernel/softirq.c +3
-0+* handled. +*/ +or_softirq_pending (pending << (vec_nr +1 ));To or the value interrupts need to be disabled because otherwise you can lose a bit when an interrupt happens in the middle of the RMW operation and raises a softirq which is not in @pending and not in the per CPU local softirq pending storage.
I can't understand the problem described above, could you explain in detail.
thanks.
Quoted 6 lines
There is another problem. Assume bit 0 and 1 are pending when the processing starts. Now it breaks out after bit 0 has been handled and stores back bit 1 as pending. Before ksoftirqd runs bit 0 gets raised again. ksoftirqd runs and handles bit 0, which takes more than the timeout. As a result the bit 0 processing can starve all other softirqs.
May I use a percpu val to record the order of processing softirq, by the order val, when it is in ksoftirqd we can process the pending softirq in order. In the scenario you described above, before ksoftirqd runs, bit 0 gets raised again, ksoftirqd runs and handles bit 1 by the percpu val. just like a ring.
Looking forward to your suggestions
Thanks
Quoted 5 lines
So this needs more thought.
Thanks,
tglx
Qian,
Quoted 10 lines · jun qian
Quoted 7 lines · Thomas Gleixner
Quoted 1 line
+
or_softirq_pending(pending << (vec_nr + 1));To or the value interrupts need to be disabled because otherwise you can lose a bit when an interrupt happens in the middle of the RMW operation and raises a softirq which is not in @pending and not in the per CPU local softirq pending storage.
I can't understand the problem described above, could you explain in detail.
Oring a value to a memory location is a Read Modify Write (RMW) operation.
x |= a;
translates to pseudo code:
R1 = x // Load content of memory location x into register R1
R1 |= a // Or value a on R1
x = R1 // Write back resultSo assume:
x = 0
a = 1
R1 = x --> R1 == 0
R1 |= a --> R1 == 1
interrupt sets bit 1 in x --> x == 0x02
x = R1 --> x == 1
So you lost the set bit 1, right? Not really what you want.
Quoted 10 lines
Quoted 6 lines
There is another problem. Assume bit 0 and 1 are pending when the processing starts. Now it breaks out after bit 0 has been handled and stores back bit 1 as pending. Before ksoftirqd runs bit 0 gets raised again. ksoftirqd runs and handles bit 0, which takes more than the timeout. As a result the bit 0 processing can starve all other softirqs.
May I use a percpu val to record the order of processing softirq, by the order val, when it is in ksoftirqd we can process the pending softirq in order. In the scenario you described above, before ksoftirqd runs, bit 0 gets raised again, ksoftirqd runs and handles bit 1 by the percpu val. just like a ring.
Yes, you need something to save information about the not-processed bits. Keeping track of which bit to process next works, but whether that's going to result in efficient and simple code is a different question.
Thanks,
tglx
Quoted 34 lines · Thomas Gleixner
Qian,
Quoted 10 lines · jun qian
Quoted 7 lines · Thomas Gleixner
Quoted 1 line
+
or_softirq_pending(pending << (vec_nr + 1));To or the value interrupts need to be disabled because otherwise you can lose a bit when an interrupt happens in the middle of the RMW operation and raises a softirq which is not in @pending and not in the per CPU local softirq pending storage.
I can't understand the problem described above, could you explain in detail.
Oring a value to a memory location is a Read Modify Write (RMW) operation.
x |= a;translates to pseudo code:
R1 = x // Load content of memory location x into register R1 R1 |= a // Or value a on R1 x = R1 // Write back resultSo assume:
x = 0 a = 1 R1 = x --> R1 == 0 R1 |= a --> R1 == 1
Quoted 6 lines
interrupt sets bit 1 in x --> x ==
0x02x = R1 --> x == 1So you lost the set bit 1, right? Not really what you want.
wow, thanks a lot, i got it.
Quoted 11 lines
Quoted 10 lines
Quoted 6 lines
There is another problem. Assume bit 0 and 1 are pending when the processing starts. Now it breaks out after bit 0 has been handled and stores back bit 1 as pending. Before ksoftirqd runs bit 0 gets raised again. ksoftirqd runs and handles bit 0, which takes more than the timeout. As a result the bit 0 processing can starve all other softirqs.
May I use a percpu val to record the order of processing softirq, by the order val, when it is in ksoftirqd we can process the pending softirq in order. In the scenario you described above, before ksoftirqd runs, bit 0 gets raised again, ksoftirqd runs and handles bit 1 by the percpu val. just like a ring.
Quoted 5 lines
Yes, you need something to save information about the not-processed bits. Keeping track of which bit to process next works, but whether that's going to result in efficient and simple code is a different question.
ok, i will modify it in the next version.
Quoted 4 lines
Thanks,
tglx
Quoted 51 lines · Thomas Gleixner <[email protected]> 于2020年7月29日周三 下午8:16写道:
Qian,
Quoted 10 lines · jun qian
Quoted 7 lines · Thomas Gleixner
Quoted 1 line
+
or_softirq_pending(pending << (vec_nr + 1));To or the value interrupts need to be disabled because otherwise you can lose a bit when an interrupt happens in the middle of the RMW operation and raises a softirq which is not in @pending and not in the per CPU local softirq pending storage.
I can't understand the problem described above, could you explain in detail.
Oring a value to a memory location is a Read Modify Write (RMW) operation.
x |= a;translates to pseudo code:
R1 = x // Load content of memory location x into register R1 R1 |= a // Or value a on R1 x = R1 // Write back resultSo assume:
x = 0 a = 1 R1 = x --> R1 == 0 R1 |= a --> R1 == 1interrupt sets bit 1 in x --> x ==
0x02x = R1 --> x == 1So you lost the set bit 1, right? Not really what you want.
Quoted 10 lines
Quoted 6 lines
There is another problem. Assume bit 0 and 1 are pending when the processing starts. Now it breaks out after bit 0 has been handled and stores back bit 1 as pending. Before ksoftirqd runs bit 0 gets raised again. ksoftirqd runs and handles bit 0, which takes more than the timeout. As a result the bit 0 processing can starve all other softirqs.
May I use a percpu val to record the order of processing softirq, by the order val, when it is in ksoftirqd we can process the pending softirq in order. In the scenario you described above, before ksoftirqd runs, bit 0 gets raised again, ksoftirqd runs and handles bit 1 by the percpu val. just like a ring.
Quoted 9 lines
Yes, you need something to save information about the not-processed bits. Keeping track of which bit to process next works, but whether that's going to result in efficient and simple code is a different question.
Thanks,
tglx
Hi Thomas, I am so sorry, For personal reasons, the modification of
this patch was delayed, I will submit the next modified version in these two days, sorry again
Quoted 42 lines · Thomas Gleixner <[email protected]> 于2020年7月27日周一 下午11:41写道:
Qian,
Quoted 10 lines · [email protected] writes:
/*
* We restart softirq processing for at mostMAX_SOFTIRQ_RESTARTtimes,
* but break the loop ifneed_resched()is set or after 2 ms.
- * TheMAX_SOFTIRQ_TIMEprovides a nice upper bound in most cases, but in
- * certain cases, such asstop_machine(), jiffies may cease to
- * increment and so we need theMAX_SOFTIRQ_RESTARTlimit as
- * well to make sure we eventually return from this method.
+ * In the loop, if the processing time of the softirq has exceeded 2
+ * milliseconds, we also need to break the loop to wakeup the ksofirqd.You are removing the
MAX_SOFTIRQ_RESTARTlimit explanation and I rather haveMAX_SOFTIRQ_TIME_NSthere than '2 milliseconds' in case the value gets adjusted later on. Also whilesched_clock()is granular on many systems it still can be jiffies based and then the above problem persists.Quoted 5 lines
kernel/softirq.c +1
-0@@ -299,6 +298,19 @@ asmlinkage __visible void __softirq_entry __do_softirq(void) } h++; pending >>= softirq_bit; +kernel/softirq.c asmlinkage __visible void __softirq_entry __do_softirq(void) · line 311
Quoted 10 lines
kernel/softirq.c +10
-0+/* +* the softirq's action has been running for too much time +* so it may need to wakeup the ksoftirqd +*/ +if (need_resched () &&sched_clock () > end) { +/* +* Ensure that the remaining pending bits are +* handled. +*/ +or_softirq_pending (pending << (vec_nr +1 ));To or the value interrupts need to be disabled because otherwise you can lose a bit when an interrupt happens in the middle of the RMW operation and raises a softirq which is not in @pending and not in the per CPU local softirq pending storage.
Quoted 7 lines
There is another problem. Assume bit 0 and 1 are pending when the processing starts. Now it breaks out after bit 0 has been handled and stores back bit 1 as pending. Before ksoftirqd runs bit 0 gets raised again. ksoftirqd runs and handles bit 0, which takes more than the timeout. As a result the bit 0 processing can starve all other softirqs.
So this needs more thought.
HI Thomas, The problem as you described, i am trying to solve it, but i found the
other problem,just like this,for example
the pending bits is 1010110110, when the bit4 was handled, and the loop processing
time more than 2ms, in my patch, the loop will break early. In the next time, the loop
will process the bit5, if before the soft action, the bit6,bit8,bit0,bit3 were set, the loop
will process the bit5,6,7,8,9,0,1,2,3, that will be the bit6 and bit8 wil be processed before
bit0 and bit3.
Do we need to consider this scenario. Do you have any good suggestions.
Thanks.
Quoted 4 lines
Thanks,
tglx