Skip to contentSkip to conversation

softirq: wake up ktimer thread in softirq context

PATCH In review Junxiao Chang 3 participants 11 messages Dec 8, 2022 – Feb 20, 2023

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 softirq: wake up ktimer thread in softirq context

Occiaionally timer interrupt might be triggered in softirq context, ktimer thread should be woken up with RT kernel, or else ktimer thread might stay in sleep state although timer interrupt has been triggered.

This change fixes a latency issue that timer handler is delayed for more than 4ms in network related test.

Fixes 2165d27554e8 ("softirq: Use a dedicated thread for timer wakeups.")
---
 kernel/softirq.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
kernel/softirq.c +4 -7
index ab1fe34326bab..34ae39e4a3d10 100644
@@ -664,13 +664,10 @@ static inline void __irq_exit_rcu(void)
 #endif
 	account_hardirq_exit(current);
 	preempt_count_sub(HARDIRQ_OFFSET);
-	if (!in_interrupt()) {
-		if (local_softirq_pending())
-			invoke_softirq();
-
-		if (IS_ENABLED(CONFIG_PREEMPT_RT) && local_pending_timers())
-			wake_timersd();
-	}
+	if (!in_interrupt() && local_softirq_pending())
+		invoke_softirq();
+	if (!(in_nmi() || in_hardirq()) && local_pending_timers())
+		wake_timersd();
 
 	tick_irq_exit();
 }
Signature · 1 line
2.25.1
#1 lore raw

Any comment? This patch is for 6.1-rt, issue could be reproduced with 5.19-rt kernel as well.

This issue is easier to reproduced when there is heavy network workload which introduces a lot of softirq events. If hrtimer interrupt is triggered in softirq context, with current RT kernel, it will not wake up ktimers thread which handles hrtimer event because in function __irq_exit_rcu, "in_interrupt()" is true:

static inline void __irq_exit_rcu(void)
{
...

        preempt_count_sub(HARDIRQ_OFFSET);
        if (!in_interrupt()) {
                if (local_softirq_pending())
                        invoke_softirq();

                if (IS_ENABLED(CONFIG_PREEMPT_RT) && local_pending_timers())
                        wake_timersd();
        }

...
}

Then ktimers threads stays in sleep state, hrtimer function will not be called although hrtimer interrupt has been triggered. Ktimers thread might be woken up in next timer interrupt which introduces long delay.

Any comments are welcome.

Regards,
Junxiao

-----Original Message-----
From: Chang, Junxiao <[email protected]>
Sent: Thursday, December 8, 2022 3:56 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Chang, Junxiao <[email protected]>; Peh, Hock Zhang <[email protected]>
Subject: [PATCH] softirq: wake up ktimer thread in softirq context

Occiaionally timer interrupt might be triggered in softirq context, ktimer thread should be woken up with RT kernel, or else ktimer thread might stay in sleep state although timer interrupt has been triggered.

This change fixes a latency issue that timer handler is delayed for more than 4ms in network related test.

Fixes 2165d27554e8 ("softirq: Use a dedicated thread for timer wakeups.")
---
 kernel/softirq.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
kernel/softirq.c +4 -8
@@ -664,13 +664,10 @@ static inline void __irq_exit_rcu(void)  #endif
 	account_hardirq_exit(current);
 	preempt_count_sub(HARDIRQ_OFFSET);
-	if (!in_interrupt()) {
-		if (local_softirq_pending())
-			invoke_softirq();
-
-		if (IS_ENABLED(CONFIG_PREEMPT_RT) && local_pending_timers())
-			wake_timersd();
-	}
+	if (!in_interrupt() && local_softirq_pending())
+		invoke_softirq();
+	if (!(in_nmi() || in_hardirq()) && local_pending_timers())
+		wake_timersd();
 
 	tick_irq_exit();
 }
--

2.25.1

#2 lore raw
Quoted 1 line · Junxiao

Any comment? This patch is for 6.1-rt, issue could be reproduced with 5.19-rt kernel as well.

In

https://lore.kernel.org/linux-rt-users/CAFzL-7v-NSFKAsyhxReEES7bMomSTwYyrZscnjbkydEP3CTXmQ@mail.gmail.com/

we reported an occasional problem with an x86 system entering a deep C
state even though timers were pending. Perhaps your patch would
prevent this transition.

Quoted 13 lines

This issue is easier to reproduced when there is heavy network workload which introduces a lot of softirq events. If hrtimer interrupt is triggered in softirq context, with current RT kernel, it will not wake up ktimers thread which handles hrtimer event because in function __irq_exit_rcu, "in_interrupt()" is true:

static inline void __irq_exit_rcu(void)
{
...

        preempt_count_sub(HARDIRQ_OFFSET);
        if (!in_interrupt()) {
                if (local_softirq_pending())
                        invoke_softirq();

                if (IS_ENABLED(CONFIG_PREEMPT_RT) && local_pending_timers())
                        wake_timersd();
        }
Quoted 3 lines

...
}

Isn't removing the check for IS_ENABLED(CONFIG_PREEMPT_RT) inadvisable?

Quoted 6 lines

Then ktimers threads stays in sleep state, hrtimer function will not be called although hrtimer interrupt has been triggered. Ktimers thread might be woken up in next timer interrupt which introduces long delay.

Any comments are welcome.

Regards,
Junxiao

-- Alison Chaiken
Aurora Innovation

-----Original Message-----

Quoted 40 lines

From: Chang, Junxiao <[email protected]>
Sent: Thursday, December 8, 2022 3:56 PM
To: [email protected]
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Chang, Junxiao <[email protected]>; Peh, Hock Zhang <[email protected]>
Subject: [PATCH] softirq: wake up ktimer thread in softirq context

Occiaionally timer interrupt might be triggered in softirq context, ktimer thread should be woken up with RT kernel, or else ktimer thread might stay in sleep state although timer interrupt has been triggered.

This change fixes a latency issue that timer handler is delayed for more than 4ms in network related test.

Fixes 2165d27554e8 ("softirq: Use a dedicated thread for timer wakeups.")
---
 kernel/softirq.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)
kernel/softirq.c +4 -8
@@ -664,13 +664,10 @@ static inline void __irq_exit_rcu(void)  #endif
        account_hardirq_exit(current);
        preempt_count_sub(HARDIRQ_OFFSET);
-       if (!in_interrupt()) {
-               if (local_softirq_pending())
-                       invoke_softirq();
-
-               if (IS_ENABLED(CONFIG_PREEMPT_RT) && local_pending_timers())
-                       wake_timersd();
-       }
+       if (!in_interrupt() && local_softirq_pending())
+               invoke_softirq();
+       if (!(in_nmi() || in_hardirq()) && local_pending_timers())
+               wake_timersd();
 
        tick_irq_exit();
 }
--

2.25.1

#3 lore raw
In reply to Chang, Junxiao
Quoted 1 line · Junxiao

Any comment? This patch is for 6.1-rt, issue could be reproduced with 5.19-rt kernel as well.

Thanks for the ping. I did see the initial email and I didn't get to it yet. I need to re-test, confirm and then apply.
The ktimer patch is not in v5.15 and this is currently the latest one maintained by the stable team. I don't know which one will be the following LTS kernel but this one needs to have this addressed. The v5.19 is not receiving any updates.
Given the current timing, I will look into this in January.

Quoted 2 lines

Regards,
Junxiao

Sebastian

#4 lore raw
Quoted 1 line · Junxiao
Quoted 1 line

Any comment? This patch is for 6.1-rt, issue could be reproduced with 5.19-rt kernel as well.

Quoted 6 lines · On Tue, Dec 20, 2022 at 10:03 AM [email protected] <[email protected]>…

Thanks for the ping. I did see the initial email and I didn't get to it yet. I need to re-test, confirm and then apply.
The ktimer patch is not in v5.15 and this is currently the latest one maintained by the stable team. I don't know which one will be the following LTS kernel but this one needs to have this addressed. The v5.19 is not receiving any updates.

We backported the timersd patch to 5.15, where is where we have
observed the C-state transition despite pending timers. The
reported trace sequence does show that __do_softirq() starts just before the hrtimer_interrupt when the problem occurs:

When the timers are delayed, the trouble appears to begin when the hrtimer_interrupt results in execution of hrtimer_run_queues() instead of raise_hrtimer_softirq():
<userspace>-13534 [007] 16947.069504: enqueue_hrtimer

          <idle>-0       [007] 16947.069547:
hrtimer_next_event_without: 16947.067643167
          <idle>-0       [007] 16947.069553: enqueue_hrtimer
          <idle>-0       [007] 16947.069567: enqueue_hrtimer
          <idle>-0       [007] 16947.069575:
hrtimer_next_event_without: 16947.067643167
          <idle>-0       [007] 16947.069579: enqueue_hrtimer
          <idle>-0       [007] 16947.078270: hrtimer_interrupt
          <idle>-0       [007] 16947.078278: hrtimer_run_queues.
          <idle>-0       [007] 16947.078300: enqueue_hrtimer
       ktimers/7-80      [007] 16947.078308: __do_softirq
     ksoftirqd/7-81      [007] 16947.078338: ksoftirqd_should_run 0
          <idle>-0       [007] 16947.078361:
hrtimer_next_event_without: 16947.067643167
          <idle>-0       [007] 16947.079323: hrtimer_interrupt
          <idle>-0       [007] 16947.079328: hrtimer_run_queues.
          <idle>-0       [007] 16947.079334: enqueue_hrtimer
     ksoftirqd/7-81      [007] 16947.079359: ksoftirqd_should_run 128
     ksoftirqd/7-81      [007] 16947.079360: __do_softirq
     ksoftirqd/7-81      [007] 16947.079361: hrtimer_interrupt
     ksoftirqd/7-81      [007] 16947.079361: raise_hrtimer_softirq.
     ksoftirqd/7-81      [007] 16947.079364: ksoftirqd_should_run 0
          <idle>-0       [007] 16947.079375:
hrtimer_next_event_without: 9223372036854775807
          <idle>-0       [007] 16947.079376:
tick_nohz_get_sleep_length: 86.838 ms
          <idle>-0       [007] 16947.079378: enqueue_hrtimer

Note the bogus value printed by hrtimer_next_event_without().

Quoted 6 lines

Given the current timing, I will look into this in January.

Quoted 2 lines

Regards,
Junxiao

Sebastian

-- Alison Chaiken
Aurora Innovation

#5 lore raw
In reply to [email protected]
Quoted 12 lines · On Tue, Dec 20, 2022 at 10:03 AM [email protected] <[email protected]>…
Quoted 1 line · Junxiao

Any comment? This patch is for 6.1-rt, issue could be reproduced with 5.19-rt kernel as well.

Thanks for the ping. I did see the initial email and I didn't get to it yet. I need to re-test, confirm and then apply.
The ktimer patch is not in v5.15 and this is currently the latest one maintained by the stable team. I don't know which one will be the following LTS kernel but this one needs to have this addressed. The v5.19 is not receiving any updates.
Given the current timing, I will look into this in January.

Quoted 4 lines
Quoted 2 lines

Regards,
Junxiao

Sebastian

Any further thoughts about Junxiao Chang's patch? We backported
ktimer patches from Sebastian and Frederic to 5.15 and would like this fix it is sensible.

Thanks,
Alison Chaiken
Aurora Innovation

#6 lore raw
In reply to [email protected]
Quoted 12 lines · On Tue, Dec 20, 2022 at 10:03 AM [email protected] <[email protected]>…
Quoted 1 line · Junxiao

Any comment? This patch is for 6.1-rt, issue could be reproduced with 5.19-rt kernel as well.

Thanks for the ping. I did see the initial email and I didn't get to it yet. I need to re-test, confirm and then apply.
The ktimer patch is not in v5.15 and this is currently the latest one maintained by the stable team. I don't know which one will be the following LTS kernel but this one needs to have this addressed. The v5.19 is not receiving any updates.
Given the current timing, I will look into this in January.

Quoted 4 lines
Quoted 2 lines

Regards,
Junxiao

Sebastian

Should users of ktimers adopt Junxiao Chang's patch

https://lore.kernel.org/linux-rt-users/BN9PR11MB5370BA8A506EB8519DC879C1ECEA9@BN9PR11MB5370.namprd11.prod.outlook.com/

from 20 Dec? It appears to be a bug fix.

Thanks,
Alison Chaiken
Aurora Innovation

#7 lore raw
Quoted 6 lines · Alison Chaiken

Should users of ktimers adopt Junxiao Chang's patch

https://lore.kernel.org/linux-rt-users/BN9PR11MB5370BA8A506EB8519DC879C1ECEA9@BN9PR11MB5370.namprd11.prod.outlook.com/

from 20 Dec? It appears to be a bug fix.

I know. I have a backlog. I looked at this a while ago but got distracted. I hopefully get it done next week. Now that 6.1 is LTS I need to bring it in shape before I can hand it over.

Quoted 3 lines

Thanks,
Alison Chaiken
Aurora Innovation

Sebastian

#8 lore raw
In reply to Junxiao Chang
Quoted 7 lines · Junxiao Chang

Occiaionally timer interrupt might be triggered in softirq context, ktimer thread should be woken up with RT kernel, or else ktimer thread might stay in sleep state although timer interrupt has been triggered.

This change fixes a latency issue that timer handler is delayed for more than 4ms in network related test.

Sorry for keeping you waiting. Your observation and patch is correct. I'm going to apply a slightly modified version of the patch (see below) after I reworded the commit message on Monday.

kernel/softirq.c +5 -6
index ab1fe34326bab..82f3e68fbe220 100644
@@ -664,13 +664,12 @@ static inline void __irq_exit_rcu(void)
 #endif
 	account_hardirq_exit(current);
 	preempt_count_sub(HARDIRQ_OFFSET);
-	if (!in_interrupt()) {
-		if (local_softirq_pending())
-			invoke_softirq();
+	if (!in_interrupt() && local_softirq_pending())
+		invoke_softirq();
 
-		if (IS_ENABLED(CONFIG_PREEMPT_RT) && local_pending_timers())
-			wake_timersd();
-	}
+	if (IS_ENABLED(CONFIG_PREEMPT_RT) && local_pending_timers() &&
+	    !(in_nmi() | in_hardirq()))
+		wake_timersd();
 
 	tick_irq_exit();
 }

Sebastian

#9 lore raw

Thank you for handling this, modified patch looks good to me.

Regards,
Junxiao

-----Original Message-----
From: Sebastian Andrzej Siewior <[email protected]>
Sent: Saturday, February 18, 2023 1:30 AM
To: Chang, Junxiao <[email protected]>
Cc: [email protected]; [email protected]; [email protected]; [email protected]; Peh, Hock Zhang <[email protected]>
Subject: Re: [PATCH] softirq: wake up ktimer thread in softirq context

Quoted 6 lines · Junxiao Chang

Occiaionally timer interrupt might be triggered in softirq context, ktimer thread should be woken up with RT kernel, or else ktimer thread might stay in sleep state although timer interrupt has been triggered.

This change fixes a latency issue that timer handler is delayed for more than 4ms in network related test.

Sorry for keeping you waiting. Your observation and patch is correct. I'm going to apply a slightly modified version of the patch (see below) after I reworded the commit message on Monday.

kernel/softirq.c +5 -6
@@ -664,13 +664,12 @@ static inline void __irq_exit_rcu(void)  #endif
 	account_hardirq_exit(current);
 	preempt_count_sub(HARDIRQ_OFFSET);
-	if (!in_interrupt()) {
-		if (local_softirq_pending())
-			invoke_softirq();
+	if (!in_interrupt() && local_softirq_pending())
+		invoke_softirq();
 
-		if (IS_ENABLED(CONFIG_PREEMPT_RT) && local_pending_timers())
-			wake_timersd();
-	}
+	if (IS_ENABLED(CONFIG_PREEMPT_RT) && local_pending_timers() &&
+	    !(in_nmi() | in_hardirq()))
+		wake_timersd();
 
 	tick_irq_exit();
 }
 

Sebastian

#10 lore raw

------->8------

From: Junxiao Chang <[email protected]>
Date: Mon, 20 Feb 2023 09:12:20 +0100
Subject: [PATCH] softirq: Wake ktimers thread also in softirq.

If the hrtimer is raised while a softirq is processed then it does not wake the corresponding ktimers thread. This is due to the optimisation in the irq-exit path which is also used to wake the ktimers thread. For the other softirqs, this is okay because the additional softirq bits will be handled by the currently running softirq handler.
The timer related softirq bits are added to a different variable and rely on the ktimers thread.
As a consuequence the wake up of ktimersd is delayed until the next timer tick.

Always wake the ktimers thread if a timer related softirq is pending.

---
 kernel/softirq.c |   11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)
kernel/softirq.c +5 -6
@@ -664,13 +664,12 @@ static inline void __irq_exit_rcu(void)
 #endif
 	account_hardirq_exit(current);
 	preempt_count_sub(HARDIRQ_OFFSET);
-	if (!in_interrupt()) {
-		if (local_softirq_pending())
-			invoke_softirq();
+	if (!in_interrupt() && local_softirq_pending())
+		invoke_softirq();
 
-		if (IS_ENABLED(CONFIG_PREEMPT_RT) && local_pending_timers())
-			wake_timersd();
-	}
+	if (IS_ENABLED(CONFIG_PREEMPT_RT) && local_pending_timers() &&
+	    !(in_nmi() | in_hardirq()))
+		wake_timersd();
 
 	tick_irq_exit();
 }