Skip to contentSkip to conversation

Softirq:avoid large sched delay from the pending softirqs

PATCH Superseded [email protected] 3 participants 9 messages Sep 9, 2020 – Sep 14, 2020 1 message not in the archive

v6 of 7 Superseded Latest is v7 →

lore
Share & tools
lore
#0 lore raw

PATCH v6 1/1 Softirq:avoid large sched delay from the pending softirqs

From: jun qian <[email protected]>

When get the pending softirqs, it need to process all the pending softirqs in the while loop. If the processing time of each pending softirq is need more than 2 msec in this loop, or one of the softirq will running a long time, according to the original code logic, it will process all the pending softirqs without wakeuping ksoftirqd, which will cause a relatively large scheduling delay on the corresponding CPU, which we do not wish to see. The patch will check the total time to process pending softirq, if the time exceeds 2 ms we need to wakeup the ksofirqd to aviod large sched delay.

---
 kernel/softirq.c | 83 ++++++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 69 insertions(+), 14 deletions(-)
kernel/softirq.c +69 -14
index c4201b7f..1f696c8 100644
@@ -25,6 +25,7 @@
 #include <linux/smpboot.h>
 #include <linux/tick.h>
 #include <linux/irq.h>
+#include <linux/sched/clock.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/irq.h>
@@ -199,18 +200,17 @@ void __local_bh_enable_ip(unsigned long ip, unsigned int cnt)
 
 /*
  * We restart softirq processing for at most MAX_SOFTIRQ_RESTART times,
- * but break the loop if need_resched() is set or after 2 ms.
- * The MAX_SOFTIRQ_TIME provides a nice upper bound in most cases, but in
- * certain cases, such as stop_machine(), jiffies may cease to
- * increment and so we need the MAX_SOFTIRQ_RESTART limit as
- * well to make sure we eventually return from this method.
+ * but break the loop if need_resched() is set or after MAX_SOFTIRQ_TIME_NS
+ * ns. In the loop, if the processing time of the softirq has exceeded
+ * MAX_SOFTIRQ_TIME_NS ns, we also need to break the loop to wakeup the
+ * ksofirqd.
  *
  * These limits have been established via experimentation.
  * The two things to balance is latency against fairness -
  * we want to handle softirqs as soon as possible, but they
  * should not be able to lock up the box.
  */
-#define MAX_SOFTIRQ_TIME  msecs_to_jiffies(2)
+#define MAX_SOFTIRQ_TIME_NS 2000000
 #define MAX_SOFTIRQ_RESTART 10
 
 #ifdef CONFIG_TRACE_IRQFLAGS
@@ -246,15 +246,20 @@ static inline void lockdep_softirq_end(bool in_hardirq)
 static inline void lockdep_softirq_end(bool in_hardirq) { }
 #endif
 
+DEFINE_PER_CPU(__u32, pending_new_flag);
+DEFINE_PER_CPU(__u32, pending_next_bit);
+#define SOFTIRQ_PENDING_MASK ((1UL << NR_SOFTIRQS) - 1)
+
 asmlinkage __visible void __softirq_entry __do_softirq(void)
 {
-	unsigned long end = jiffies + MAX_SOFTIRQ_TIME;
+	u64 end = sched_clock() + MAX_SOFTIRQ_TIME_NS;
 	unsigned long old_flags = current->flags;
 	int max_restart = MAX_SOFTIRQ_RESTART;
 	struct softirq_action *h;
 	bool in_hardirq;
-	__u32 pending;
-	int softirq_bit;
+	__u32 pending, pending_left, pending_new;
+	int softirq_bit, next_bit;
+	unsigned long flags;
 
 	/*
 	 * Mask out PF_MEMALLOC as the current task context is borrowed for the
@@ -277,10 +282,33 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)
 
 	h = softirq_vec;
 
-	while ((softirq_bit = ffs(pending))) {
-		unsigned int vec_nr;
+	next_bit = per_cpu(pending_next_bit, smp_processor_id());
+	per_cpu(pending_new_flag, smp_processor_id()) = 0;
+
+	pending_left = pending &
+		(SOFTIRQ_PENDING_MASK << next_bit);
+	pending_new = pending &
+		(SOFTIRQ_PENDING_MASK >> (NR_SOFTIRQS - next_bit));
+
+	/*
+	 * In order to be fair, we shold process the pengding bits by the
+	 * last processing order.
+	 */
+	while ((softirq_bit = ffs(pending_left)) ||
+		(softirq_bit = ffs(pending_new))) {
 		int prev_count;
+		unsigned int vec_nr = 0;
 
+		/*
+		 * when the left pengding bits have been handled, we should
+		 * to reset the h to softirq_vec.
+		 */
+		if (!ffs(pending_left)) {
+			if (per_cpu(pending_new_flag, smp_processor_id()) == 0) {
+				h = softirq_vec;
+				per_cpu(pending_new_flag, smp_processor_id()) = 1;
+			}
+		}
 		h += softirq_bit - 1;
 
 		vec_nr = h - softirq_vec;
@@ -298,17 +326,44 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)
 			preempt_count_set(prev_count);
 		}
 		h++;
-		pending >>= softirq_bit;
+
+		if (ffs(pending_left))
+			pending_left >>= softirq_bit;
+		else
+			pending_new >>= softirq_bit;
+
+		/*
+		 * the softirq's action has been run too much time,
+		 * so it may need to wakeup the ksoftirqd
+		 */
+		if (need_resched() && sched_clock() > end) {
+			/*
+			 * Ensure that the remaining pending bits will be
+			 * handled.
+			 */
+			local_irq_save(flags);
+			if (ffs(pending_left))
+				or_softirq_pending((pending_left << (vec_nr + 1)) |
+							pending_new);
+			else
+				or_softirq_pending(pending_new << (vec_nr + 1));
+			local_irq_restore(flags);
+			per_cpu(pending_next_bit, smp_processor_id()) = vec_nr + 1;
+			break;
+		}
 	}
 
+	/* reset the pending_next_bit */
+	per_cpu(pending_next_bit, smp_processor_id()) = 0;
+
 	if (__this_cpu_read(ksoftirqd) == current)
 		rcu_softirq_qs();
 	local_irq_disable();
 
 	pending = local_softirq_pending();
 	if (pending) {
-		if (time_before(jiffies, end) && !need_resched() &&
-		    --max_restart)
+		if (!need_resched() && --max_restart &&
+		    sched_clock() <= end)
 			goto restart;
 
 		wakeup_softirqd();
Signature · 1 line
1.8.3.1
#1 lore raw
Quoted 11 lines · On Wed, Sep 09, 2020 at 05:09:31PM +0800, [email protected] wrote:

From: jun qian <[email protected]>

When get the pending softirqs, it need to process all the pending softirqs in the while loop. If the processing time of each pending softirq is need more than 2 msec in this loop, or one of the softirq will running a long time, according to the original code logic, it will process all the pending softirqs without wakeuping ksoftirqd, which will cause a relatively large scheduling delay on the corresponding CPU, which we do not wish to see. The patch will check the total time to process pending softirq, if the time exceeds 2 ms we need to wakeup the ksofirqd to aviod large sched delay.

But what is all that unreadaable gibberish with pending_new_{flag,bit} ?

Random comments below..

Quoted 1 line

+#define MAX_SOFTIRQ_TIME_NS 2000000

	2*NSEC_PER_MSEC

Review

Quoted 2 lines
(quoted diff) +2 -0
+DEFINE_PER_CPU(__u32, pending_new_flag);
+DEFINE_PER_CPU(__u32, pending_next_bit);

__u32 is for userspace ABI, this is not it, use u32

Quoted 19 lines
(quoted diff) +6 -3
+#define SOFTIRQ_PENDING_MASK ((1UL << NR_SOFTIRQS) - 1)
+
 asmlinkage __visible void __softirq_entry __do_softirq(void)
 {
-	unsigned long end = jiffies + MAX_SOFTIRQ_TIME;
+	u64 end = sched_clock() + MAX_SOFTIRQ_TIME_NS;
 	unsigned long old_flags = current->flags;
 	int max_restart = MAX_SOFTIRQ_RESTART;
 	struct softirq_action *h;
 	bool in_hardirq;
-	__u32 pending;
-	int softirq_bit;
+	__u32 pending, pending_left, pending_new;
+	int softirq_bit, next_bit;
+	unsigned long flags;
 
 	/*
 	 * Mask out PF_MEMALLOC as the current task context is borrowed for the

Review asmlinkage __visible void __softirq_entry __do_softirq(void) · line 286

Quoted 7 lines
(quoted diff) +2 -2
@@ -277,10 +282,33 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)
 
 	h = softirq_vec;
 
-	while ((softirq_bit = ffs(pending))) {
-		unsigned int vec_nr;
+	next_bit = per_cpu(pending_next_bit, smp_processor_id());
+	per_cpu(pending_new_flag, smp_processor_id()) = 0;
	__this_cpu_read() / __this_cpu_write()

Review asmlinkage __visible void __softirq_entry __do_softirq(void) · line 291

Quoted 5 lines
(quoted diff) +5 -0
+
+	pending_left = pending &
+		(SOFTIRQ_PENDING_MASK << next_bit);
+	pending_new = pending &
+		(SOFTIRQ_PENDING_MASK >> (NR_SOFTIRQS - next_bit));

The second mask is the inverse of the first.

Quoted 23 lines
(quoted diff) +17 -0
+	/*
+	 * In order to be fair, we shold process the pengding bits by the
+	 * last processing order.
+	 */
+	while ((softirq_bit = ffs(pending_left)) ||
+		(softirq_bit = ffs(pending_new))) {
 		int prev_count;
+		unsigned int vec_nr = 0;
 
+		/*
+		 * when the left pengding bits have been handled, we should
+		 * to reset the h to softirq_vec.
+		 */
+		if (!ffs(pending_left)) {
+			if (per_cpu(pending_new_flag, smp_processor_id()) == 0) {
+				h = softirq_vec;
+				per_cpu(pending_new_flag, smp_processor_id()) = 1;
+			}
+		}
 		h += softirq_bit - 1;
 
 		vec_nr = h - softirq_vec;

Review asmlinkage __visible void __softirq_entry __do_softirq(void) · line 330

Quoted 6 lines
(quoted diff) +2 -1
@@ -298,17 +326,44 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)
 			preempt_count_set(prev_count);
 		}
 		h++;
-		pending >>= softirq_bit;
+
+		if (ffs(pending_left))

This is the _third_ ffs(pending_left), those things are _expensive_ (on some archs, see include/asm-generic/bitops/__ffs.h).

Quoted 5 lines
(quoted diff) +5 -0
+			pending_left >>= softirq_bit;
+		else
+			pending_new >>= softirq_bit;
+
+		/*

Review asmlinkage __visible void __softirq_entry __do_softirq(void) · line 345

Quoted 10 lines
(quoted diff) +10 -0
+		 * the softirq's action has been run too much time,
+		 * so it may need to wakeup the ksoftirqd
+		 */
+		if (need_resched() && sched_clock() > end) {
+			/*
+			 * Ensure that the remaining pending bits will be
+			 * handled.
+			 */
+			local_irq_save(flags);
+			if (ffs(pending_left))

*fourth*...

Quoted 16 lines
(quoted diff) +11 -0
+				or_softirq_pending((pending_left << (vec_nr + 1)) |
+							pending_new);
+			else
+				or_softirq_pending(pending_new << (vec_nr + 1));
+			local_irq_restore(flags);
+			per_cpu(pending_next_bit, smp_processor_id()) = vec_nr + 1;
+			break;
+		}
 	}
 
+	/* reset the pending_next_bit */
+	per_cpu(pending_next_bit, smp_processor_id()) = 0;
+
 	if (__this_cpu_read(ksoftirqd) == current)
 		rcu_softirq_qs();
 	local_irq_disable();

Review asmlinkage __visible void __softirq_entry __do_softirq(void) · line 369

Quoted 10 lines
(quoted diff) +2 -2
 
 	pending = local_softirq_pending();
 	if (pending) {
-		if (time_before(jiffies, end) && !need_resched() &&
-		    --max_restart)
+		if (!need_resched() && --max_restart &&
+		    sched_clock() <= end)
 			goto restart;
 
 		wakeup_softirqd();

This really wants to be a number of separate patches; and I quickly lost the plot in your code. Instead of cleaning things up, you're making an even bigger mess of things.

That said, I _think_ I've managed to decode what you want. See the completely untested patches attached.

Subject: softirq: Rewrite softirq processing loop
From: Peter Zijlstra <[email protected]>
Date: Fri Sep 11 17:00:03 CEST 2020

Simplify the softirq processing loop by using the bitmap APIs

---
 kernel/softirq.c |   16 ++++++----------
 1 file changed, 6 insertions(+), 10 deletions(-)
kernel/softirq.c +6 -10
@@ -258,9 +258,9 @@ asmlinkage __visible void __softirq_entr
 	unsigned long old_flags = current->flags;
 	int max_restart = MAX_SOFTIRQ_RESTART;
 	struct softirq_action *h;
+	unsigned long pending;
+	unsigned int vec_nr;
 	bool in_hardirq;
-	__u32 pending;
-	int softirq_bit;
 
 	/*
 	 * Mask out PF_MEMALLOC as the current task context is borrowed for the
@@ -281,15 +281,13 @@ asmlinkage __visible void __softirq_entr
 
 	local_irq_enable();
 
-	h = softirq_vec;
+	for_each_set_bit(vec_nr, &pending, NR_SOFTIRQS) {
+		unsigned int prev_count;
 
-	while ((softirq_bit = ffs(pending))) {
-		unsigned int vec_nr;
-		int prev_count;
+		__clear_bit(vec_nr, &pending);
 
-		h += softirq_bit - 1;
+		h = softirq_vec + vec_nr;
 
-		vec_nr = h - softirq_vec;
 		prev_count = preempt_count();
 
 		kstat_incr_softirqs_this_cpu(vec_nr);
@@ -303,8 +301,6 @@ asmlinkage __visible void __softirq_entr
 			       prev_count, preempt_count());
 			preempt_count_set(prev_count);
 		}
-		h++;
-		pending >>= softirq_bit;
 	}
 
 	if (__this_cpu_read(ksoftirqd) == current)

Subject: softirq: Use sched_clock() based timeout
From: Peter Zijlstra <[email protected]>
Date: Fri Sep 11 17:30:01 CEST 2020

Replace the jiffies based timeout with a sched_clock() based one.

---
 kernel/softirq.c |    7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)
kernel/softirq.c +4 -3
@@ -25,6 +25,7 @@
 #include <linux/smpboot.h>
 #include <linux/tick.h>
 #include <linux/irq.h>
+#include <linux/sched/clock.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/irq.h>
@@ -216,7 +217,7 @@ EXPORT_SYMBOL(__local_bh_enable_ip);
  * we want to handle softirqs as soon as possible, but they
  * should not be able to lock up the box.
  */
-#define MAX_SOFTIRQ_TIME  msecs_to_jiffies(2)
+#define MAX_SOFTIRQ_TIME	2*NSEC_PER_MSEC
 #define MAX_SOFTIRQ_RESTART 10
 
 #ifdef CONFIG_TRACE_IRQFLAGS
@@ -254,9 +255,9 @@ static inline void lockdep_softirq_end(b
 
 asmlinkage __visible void __softirq_entry __do_softirq(void)
 {
-	unsigned long end = jiffies + MAX_SOFTIRQ_TIME;
 	unsigned long old_flags = current->flags;
 	int max_restart = MAX_SOFTIRQ_RESTART;
+	u64 start = sched_clock();
 	struct softirq_action *h;
 	unsigned long pending;
 	unsigned int vec_nr;
@@ -309,7 +310,7 @@ asmlinkage __visible void __softirq_entr
 
 	pending = local_softirq_pending();
 	if (pending) {
-		if (time_before(jiffies, end) && !need_resched() &&
+		if (sched_clock() - start < MAX_SOFTIRQ_TIME && !need_resched() &&
 		    --max_restart)
 			goto restart;
 

Subject: softirq: Factor loop termination condition
From: Peter Zijlstra <[email protected]>
Date: Fri Sep 11 17:17:20 CEST 2020

---
 kernel/softirq.c |   44 +++++++++++++++++++++++++-------------------
 1 file changed, 25 insertions(+), 19 deletions(-)
kernel/softirq.c +25 -19

65 lines not on this page · raw message

Subject: softirq: Allow early break
From: Peter Zijlstra <[email protected]>
Date: Fri Sep 11 17:50:17 CEST 2020

Allow terminating the softirq processing loop without finishing the vectors.

---
 kernel/softirq.c |   16 ++++++++++------
 1 file changed, 10 insertions(+), 6 deletions(-)
kernel/softirq.c +10 -6
@@ -309,19 +309,23 @@ asmlinkage __visible void __softirq_entr
 			       prev_count, preempt_count());
 			preempt_count_set(prev_count);
 		}
+
+		if (pending && __softirq_needs_break(start))
+			break;
 	}
 
 	if (__this_cpu_read(ksoftirqd) == current)
 		rcu_softirq_qs();
 	local_irq_disable();
 
-	pending = local_softirq_pending();
-	if (pending) {
-		if (!__softirq_needs_break(start) && --max_restart)
-			goto restart;
+	if (pending)
+		or_softirq_pending(pending);
+	else if ((pending = local_softirq_pending()) &&
+		 !__softirq_needs_break(start) &&
+		 --max_restart)
+		goto restart;
 
-		wakeup_softirqd();
-	}
+	wakeup_softirqd();
 
 	lockdep_softirq_end(in_hardirq);
 	account_irq_exit_time(current);
author
#2 lore raw
Quoted 142 lines · <[email protected]> 于2020年9月11日周五 下午11:55写道:
Quoted 11 lines · On Wed, Sep 09, 2020 at 05:09:31PM +0800, [email protected] wrote:

From: jun qian <[email protected]>

When get the pending softirqs, it need to process all the pending softirqs in the while loop. If the processing time of each pending softirq is need more than 2 msec in this loop, or one of the softirq will running a long time, according to the original code logic, it will process all the pending softirqs without wakeuping ksoftirqd, which will cause a relatively large scheduling delay on the corresponding CPU, which we do not wish to see. The patch will check the total time to process pending softirq, if the time exceeds 2 ms we need to wakeup the ksofirqd to aviod large sched delay.

But what is all that unreadaable gibberish with pending_new_{flag,bit} ?

Random comments below..

Quoted 1 line

+#define MAX_SOFTIRQ_TIME_NS 2000000

        2*NSEC_PER_MSEC

Review

Quoted 2 lines
(quoted diff) +2 -0
+DEFINE_PER_CPU(__u32, pending_new_flag);
+DEFINE_PER_CPU(__u32, pending_next_bit);

__u32 is for userspace ABI, this is not it, use u32

Quoted 19 lines
kernel/softirq.c +6 -3
+#define SOFTIRQ_PENDING_MASK ((1UL << NR_SOFTIRQS) - 1)
+
 asmlinkage __visible void __softirq_entry __do_softirq(void)
 {
-     unsigned long end = jiffies + MAX_SOFTIRQ_TIME;
+     u64 end = sched_clock() + MAX_SOFTIRQ_TIME_NS;
      unsigned long old_flags = current->flags;
      int max_restart = MAX_SOFTIRQ_RESTART;
      struct softirq_action *h;
      bool in_hardirq;
-     __u32 pending;
-     int softirq_bit;
+     __u32 pending, pending_left, pending_new;
+     int softirq_bit, next_bit;
+     unsigned long flags;
 
      /*
       * Mask out PF_MEMALLOC as the current task context is borrowed for the

kernel/softirq.c asmlinkage __visible void __softirq_entry __do_softirq(void) · line 286

Quoted 7 lines
kernel/softirq.c +2 -2
@@ -277,10 +282,33 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)
 
      h = softirq_vec;
 
-     while ((softirq_bit = ffs(pending))) {
-             unsigned int vec_nr;
+     next_bit = per_cpu(pending_next_bit, smp_processor_id());
+     per_cpu(pending_new_flag, smp_processor_id()) = 0;
        __this_cpu_read() / __this_cpu_write()

Review asmlinkage __visible void __softirq_entry __do_softirq(void) · line 291

Quoted 5 lines
(quoted diff) +5 -0
+
+     pending_left = pending &
+             (SOFTIRQ_PENDING_MASK << next_bit);
+     pending_new = pending &
+             (SOFTIRQ_PENDING_MASK >> (NR_SOFTIRQS - next_bit));

The second mask is the inverse of the first.

Quoted 23 lines
kernel/softirq.c +17 -0
+     /*
+      * In order to be fair, we shold process the pengding bits by the
+      * last processing order.
+      */
+     while ((softirq_bit = ffs(pending_left)) ||
+             (softirq_bit = ffs(pending_new))) {
              int prev_count;
+             unsigned int vec_nr = 0;
 
+             /*
+              * when the left pengding bits have been handled, we should
+              * to reset the h to softirq_vec.
+              */
+             if (!ffs(pending_left)) {
+                     if (per_cpu(pending_new_flag, smp_processor_id()) == 0) {
+                             h = softirq_vec;
+                             per_cpu(pending_new_flag, smp_processor_id()) = 1;
+                     }
+             }
              h += softirq_bit - 1;
 
              vec_nr = h - softirq_vec;

kernel/softirq.c asmlinkage __visible void __softirq_entry __do_softirq(void) · line 330

Quoted 6 lines
kernel/softirq.c +2 -1
@@ -298,17 +326,44 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)
                      preempt_count_set(prev_count);
              }
              h++;
-             pending >>= softirq_bit;
+
+             if (ffs(pending_left))

This is the _third_ ffs(pending_left), those things are _expensive_ (on some archs, see include/asm-generic/bitops/__ffs.h).

Quoted 5 lines
(quoted diff) +5 -0
+                     pending_left >>= softirq_bit;
+             else
+                     pending_new >>= softirq_bit;
+
+             /*

Review asmlinkage __visible void __softirq_entry __do_softirq(void) · line 345

Quoted 10 lines
(quoted diff) +10 -0
+              * the softirq's action has been run too much time,
+              * so it may need to wakeup the ksoftirqd
+              */
+             if (need_resched() && sched_clock() > end) {
+                     /*
+                      * Ensure that the remaining pending bits will be
+                      * handled.
+                      */
+                     local_irq_save(flags);
+                     if (ffs(pending_left))

*fourth*...

Quoted 25 lines
(quoted diff) +13 -2
+                             or_softirq_pending((pending_left << (vec_nr + 1)) |
+                                                     pending_new);
+                     else
+                             or_softirq_pending(pending_new << (vec_nr + 1));
+                     local_irq_restore(flags);
+                     per_cpu(pending_next_bit, smp_processor_id()) = vec_nr + 1;
+                     break;
+             }
      }
 
+     /* reset the pending_next_bit */
+     per_cpu(pending_next_bit, smp_processor_id()) = 0;
+
      if (__this_cpu_read(ksoftirqd) == current)
              rcu_softirq_qs();
      local_irq_disable();
 
      pending = local_softirq_pending();
      if (pending) {
-             if (time_before(jiffies, end) && !need_resched() &&
-                 --max_restart)
+             if (!need_resched() && --max_restart &&
+                 sched_clock() <= end)
                      goto restart;
 
Quoted 10 lines

kernel/softirq.c asmlinkage __visible void __softirq_entr · line 323

Quoted 1 line
kernel/softirq.c +0 -0
              wakeup_softirqd();

This really wants to be a number of separate patches; and I quickly lost the plot in your code. Instead of cleaning things up, you're making an even bigger mess of things.

That said, I _think_ I've managed to decode what you want. See the completely untested patches attached.

thanks a lot thank for your suggestion. I will rewrite the patch by following you sugestion, Useing multiple patches to clarify ideas.

#3 lore raw
In reply to [email protected]
Quoted 12 lines · On 09/09/20 17:09, [email protected] wrote:

From: jun qian <[email protected]>

When get the pending softirqs, it need to process all the pending softirqs in the while loop. If the processing time of each pending softirq is need more than 2 msec in this loop, or one of the softirq will running a long time, according to the original code logic, it will process all the pending softirqs without wakeuping ksoftirqd, which will cause a relatively large scheduling delay on the corresponding CPU, which we do not wish to see. The patch will check the total time to process pending softirq, if the time exceeds 2 ms we need to wakeup the ksofirqd to aviod large sched delay.

Quoted 1 line

In Android there's a patch that tries to avoid schedling an RT task on a cpu that is running softirqs. I wonder if this patch helps with this case.

https://android.googlesource.com/kernel/msm/+/5c3f54c34acf4d9ed01530288d4a98acff815d79%5E%21/#F0

John, Wei, is this something of interest to you?

IIUC this patch will make sure the total softirq duration is 2ms rather than each call is 2ms.

I persume if there's a single handler that takes a lot of time then this won't help. But in that case, one can argue there's a potential bug with this handler.

Cheers

--
Qais Yousef

Quoted 151 lines
---
 kernel/softirq.c | 83 ++++++++++++++++++++++++++++++++++++++++++++++----------
 1 file changed, 69 insertions(+), 14 deletions(-)
kernel/softirq.c +69 -14
index c4201b7f..1f696c8 100644
@@ -25,6 +25,7 @@
 #include <linux/smpboot.h>
 #include <linux/tick.h>
 #include <linux/irq.h>
+#include <linux/sched/clock.h>
 
 #define CREATE_TRACE_POINTS
 #include <trace/events/irq.h>
@@ -199,18 +200,17 @@ void __local_bh_enable_ip(unsigned long ip, unsigned int cnt)
 
 /*
  * We restart softirq processing for at most MAX_SOFTIRQ_RESTART times,
- * but break the loop if need_resched() is set or after 2 ms.
- * The MAX_SOFTIRQ_TIME provides a nice upper bound in most cases, but in
- * certain cases, such as stop_machine(), jiffies may cease to
- * increment and so we need the MAX_SOFTIRQ_RESTART limit as
- * well to make sure we eventually return from this method.
+ * but break the loop if need_resched() is set or after MAX_SOFTIRQ_TIME_NS
+ * ns. In the loop, if the processing time of the softirq has exceeded
+ * MAX_SOFTIRQ_TIME_NS ns, we also need to break the loop to wakeup the
+ * ksofirqd.
  *
  * These limits have been established via experimentation.
  * The two things to balance is latency against fairness -
  * we want to handle softirqs as soon as possible, but they
  * should not be able to lock up the box.
  */
-#define MAX_SOFTIRQ_TIME  msecs_to_jiffies(2)
+#define MAX_SOFTIRQ_TIME_NS 2000000
 #define MAX_SOFTIRQ_RESTART 10
 
 #ifdef CONFIG_TRACE_IRQFLAGS
@@ -246,15 +246,20 @@ static inline void lockdep_softirq_end(bool in_hardirq)
 static inline void lockdep_softirq_end(bool in_hardirq) { }
 #endif
 
+DEFINE_PER_CPU(__u32, pending_new_flag);
+DEFINE_PER_CPU(__u32, pending_next_bit);
+#define SOFTIRQ_PENDING_MASK ((1UL << NR_SOFTIRQS) - 1)
+
 asmlinkage __visible void __softirq_entry __do_softirq(void)
 {
-	unsigned long end = jiffies + MAX_SOFTIRQ_TIME;
+	u64 end = sched_clock() + MAX_SOFTIRQ_TIME_NS;
 	unsigned long old_flags = current->flags;
 	int max_restart = MAX_SOFTIRQ_RESTART;
 	struct softirq_action *h;
 	bool in_hardirq;
-	__u32 pending;
-	int softirq_bit;
+	__u32 pending, pending_left, pending_new;
+	int softirq_bit, next_bit;
+	unsigned long flags;
 
 	/*
 	 * Mask out PF_MEMALLOC as the current task context is borrowed for the
@@ -277,10 +282,33 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)
 
 	h = softirq_vec;
 
-	while ((softirq_bit = ffs(pending))) {
-		unsigned int vec_nr;
+	next_bit = per_cpu(pending_next_bit, smp_processor_id());
+	per_cpu(pending_new_flag, smp_processor_id()) = 0;
+
+	pending_left = pending &
+		(SOFTIRQ_PENDING_MASK << next_bit);
+	pending_new = pending &
+		(SOFTIRQ_PENDING_MASK >> (NR_SOFTIRQS - next_bit));
+
+	/*
+	 * In order to be fair, we shold process the pengding bits by the
+	 * last processing order.
+	 */
+	while ((softirq_bit = ffs(pending_left)) ||
+		(softirq_bit = ffs(pending_new))) {
 		int prev_count;
+		unsigned int vec_nr = 0;
 
+		/*
+		 * when the left pengding bits have been handled, we should
+		 * to reset the h to softirq_vec.
+		 */
+		if (!ffs(pending_left)) {
+			if (per_cpu(pending_new_flag, smp_processor_id()) == 0) {
+				h = softirq_vec;
+				per_cpu(pending_new_flag, smp_processor_id()) = 1;
+			}
+		}
 		h += softirq_bit - 1;
 
 		vec_nr = h - softirq_vec;
@@ -298,17 +326,44 @@ asmlinkage __visible void __softirq_entry __do_softirq(void)
 			preempt_count_set(prev_count);
 		}
 		h++;
-		pending >>= softirq_bit;
+
+		if (ffs(pending_left))
+			pending_left >>= softirq_bit;
+		else
+			pending_new >>= softirq_bit;
+
+		/*
+		 * the softirq's action has been run too much time,
+		 * so it may need to wakeup the ksoftirqd
+		 */
+		if (need_resched() && sched_clock() > end) {
+			/*
+			 * Ensure that the remaining pending bits will be
+			 * handled.
+			 */
+			local_irq_save(flags);
+			if (ffs(pending_left))
+				or_softirq_pending((pending_left << (vec_nr + 1)) |
+							pending_new);
+			else
+				or_softirq_pending(pending_new << (vec_nr + 1));
+			local_irq_restore(flags);
+			per_cpu(pending_next_bit, smp_processor_id()) = vec_nr + 1;
+			break;
+		}
 	}
 
+	/* reset the pending_next_bit */
+	per_cpu(pending_next_bit, smp_processor_id()) = 0;
+
 	if (__this_cpu_read(ksoftirqd) == current)
 		rcu_softirq_qs();
 	local_irq_disable();
 
 	pending = local_softirq_pending();
 	if (pending) {
-		if (time_before(jiffies, end) && !need_resched() &&
-		    --max_restart)
+		if (!need_resched() && --max_restart &&
+		    sched_clock() <= end)
 			goto restart;
 
 		wakeup_softirqd();
Signature · 1 line
1.8.3.1
#4 lore raw
Quoted 13 lines · Qais Yousef
Quoted 12 lines · On 09/09/20 17:09, [email protected] wrote:

From: jun qian <[email protected]>

When get the pending softirqs, it need to process all the pending softirqs in the while loop. If the processing time of each pending softirq is need more than 2 msec in this loop, or one of the softirq will running a long time, according to the original code logic, it will process all the pending softirqs without wakeuping ksoftirqd, which will cause a relatively large scheduling delay on the corresponding CPU, which we do not wish to see. The patch will check the total time to process pending softirq, if the time exceeds 2 ms we need to wakeup the ksofirqd to aviod large sched delay.

Quoted 8 lines
Quoted 1 line

In Android there's a patch that tries to avoid schedling an RT task on a cpu that is running softirqs. I wonder if this patch helps with this case.

https://android.googlesource.com/kernel/msm/+/5c3f54c34acf4d9ed01530288d4a98acff815d79%5E%21/#F0

John, Wei, is this something of interest to you?

Urgh.. that's pretty gross. I think the sane approach is indeed getting softirqs to react to need_resched() better.

#5 lore raw
Quoted 19 lines · On 09/11/20 20:28, [email protected] wrote:
Quoted 18 lines · Qais Yousef
Quoted 12 lines · On 09/09/20 17:09, [email protected] wrote:

From: jun qian <[email protected]>

When get the pending softirqs, it need to process all the pending softirqs in the while loop. If the processing time of each pending softirq is need more than 2 msec in this loop, or one of the softirq will running a long time, according to the original code logic, it will process all the pending softirqs without wakeuping ksoftirqd, which will cause a relatively large scheduling delay on the corresponding CPU, which we do not wish to see. The patch will check the total time to process pending softirq, if the time exceeds 2 ms we need to wakeup the ksofirqd to aviod large sched delay.

Quoted 1 line

In Android there's a patch that tries to avoid schedling an RT task on a cpu that is running softirqs. I wonder if this patch helps with this case.

Quoted 6 lines
Quoted 3 lines

https://android.googlesource.com/kernel/msm/+/5c3f54c34acf4d9ed01530288d4a98acff815d79%5E%21/#F0

John, Wei, is this something of interest to you?

Urgh.. that's pretty gross. I think the sane approach is indeed getting softirqs to react to need_resched() better.

What does PREEMPT_RT do to deal with softirqs delays?

I have tried playing with enabling threadirqs, which AFAIU should make softirqs preemptible, right?

I realize this patch is still missing from mainline at least:

	https://gitlab.com/kalilinux/packages/linux/blob/a17bad0db9da44cd73f594794a58cc5646393b13/debian/patches-rt/softirq-Add-preemptible-softirq.patch

Would this be a heavy handed approach to make available for non PREEMPT_RT kernels?

I only worry about potential NET_RX throughput issues. Which by the way is protected with preempt_disable currently in mainline. See netif_rx_ni().

I am guessing here, but I suspect this NET_RX softirq is one source of big delays when network activity is high.

Thanks

--
Qais Yousef

#6 lore raw
Quoted 1 line · Qais Yousef

What does PREEMPT_RT do to deal with softirqs delays?

Makes the lot preemptible, you found the patch below.

Quoted 2 lines

I have tried playing with enabling threadirqs, which AFAIU should make softirqs preemptible, right?

Not yet,..

Quoted 6 lines

I realize this patch is still missing from mainline at least:

	https://gitlab.com/kalilinux/packages/linux/blob/a17bad0db9da44cd73f594794a58cc5646393b13/debian/patches-rt/softirq-Add-preemptible-softirq.patch

Would this be a heavy handed approach to make available for non PREEMPT_RT kernels?

Not sure, I suspect it relies on migrate_disable(), which is preempt_disable() on !RT and then we're back to square one.

Quoted 2 lines

I only worry about potential NET_RX throughput issues. Which by the way is protected with preempt_disable currently in mainline. See netif_rx_ni().

So preempt_disable() isn't necessairily a problem, you just want it to terminate soonish after need_resched() becomes true. Also, I'm having a wee problem getting from net_rx_action() to netif_rx_ni()

Quoted 2 lines

I am guessing here, but I suspect this NET_RX softirq is one source of big delays when network activity is high.

Well, one approach is to more agressively limit how long softirq processing can run. Current measures are very soft in that regard.

#7 lore raw
Quoted 12 lines · On 09/14/20 16:14, [email protected] wrote:
Quoted 1 line · Qais Yousef

What does PREEMPT_RT do to deal with softirqs delays?

Makes the lot preemptible, you found the patch below.

Quoted 2 lines

I have tried playing with enabling threadirqs, which AFAIU should make softirqs preemptible, right?

Not yet,..

Quoted 2 lines

I realize this patch is still missing from mainline at least:

Quoted 7 lines
Quoted 4 lines
	https://gitlab.com/kalilinux/packages/linux/blob/a17bad0db9da44cd73f594794a58cc5646393b13/debian/patches-rt/softirq-Add-preemptible-softirq.patch

Would this be a heavy handed approach to make available for non PREEMPT_RT kernels?

Not sure, I suspect it relies on migrate_disable(), which is preempt_disable() on !RT and then we're back to square one.

I think it will depend on local_bh_disable(). I didn't dig into the patch above, but I believe it's doing that for RT.

Or maybe there's another aspect I am not aware of that relies on migrate_disable() too..

Quoted 5 lines
Quoted 3 lines

I only worry about potential NET_RX throughput issues. Which by the way is protected with preempt_disable currently in mainline. See netif_rx_ni().

So preempt_disable() isn't necessairily a problem, you just want it to

Yes. But high network traffic will make this a busy softirq. And it won't work with the patch above. But I assume the above will have to fix that with it.

	https://lore.kernel.org/netdev/[email protected]/

For the time being, it's just another potential path that could introduce latencies.

I can't follow the whole thing too, but if 5G modems ends up there; I can see this a big source of noise when the user is downloading a big file. Assuming 5g lives up to its reputation of 400+ Mbps in practice.

So there might be a tangible trade off between better softirqs latencies vs better network throughput.

Quoted 2 lines

terminate soonish after need_resched() becomes true. Also, I'm having a wee problem getting from net_rx_action() to netif_rx_ni()

I can investigate this direction :)

Quoted 6 lines
Quoted 3 lines

I am guessing here, but I suspect this NET_RX softirq is one source of big delays when network activity is high.

Well, one approach is to more agressively limit how long softirq processing can run. Current measures are very soft in that regard.

Which is this patch. Although it doesn't take into account a single softirq exceeding the quota IIUC. But the need_resched() bit above should address that.

Thanks

--
Qais Yousef

A message that is not in the archive (sent off-list or not yet received): CA+njcd3HFV5Gqtt9qzTAzpnA4-4ngPBQ7T0gwgc0Fm9_VoJLcQ@mail.gmail.com

#9 lore raw
Quoted 6 lines · John Dias

I agree that the rt-softirq interaction patches are a gross hack (and I wrote the original versions, so it's my grossness). The problem is that even a 1ms delay in executing those low-latency audio threads is enough to cause obvious glitching in audio under very real circumstances, which is not an acceptable user experience -- and some softirq handlers run for >1ms on their own. (And 1ms is a high upper bound, not a median target.)

AFAIK professional audio apps have the toughest limit of sub 10ms. 120MHz screens impose a stricter limit on display pipeline too to finish its frame in 8ms.

1ms is too short and approaches PREEMPT_RT realm.

Is it possible to expand on the use case here? What application requires this constraint?

Thanks

--
Qais Yousef