Skip to contentSkip to conversation

Clarify "NOHZ: local_softirq_pending" warning

PATCH In review Paul E. McKenney 4 participants 7 messages Jun 26, 2020 – Jul 6, 2020 Nohz, dynticks support

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Clarify "NOHZ: local_softirq_pending" warning

Currently, can_stop_idle_tick() prints "NOHZ: local_softirq_pending HH" (where "HH" is the hexadecimal softirq vector number) when one or more non-RCU softirq handlers are still enablded when checking to stop the scheduler-tick interrupt. This message is not as enlightening as one might hope, so this commit changes it to "NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #HH.

---

 tick-sched.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/time/tick-sched.c +1 -1
index f0199a4..349a25a 100644
@@ -927,7 +927,7 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
 
 		if (ratelimit < 10 &&
 		    (local_softirq_pending() & SOFTIRQ_STOP_IDLE_MASK)) {
-			pr_warn("NOHZ: local_softirq_pending %02x\n",
+			pr_warn("NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #%02x\n",
 				(unsigned int) local_softirq_pending());
 			ratelimit++;
 		}
#1 lore raw
Quoted 7 lines · Paul E. McKenney

Currently, can_stop_idle_tick() prints "NOHZ: local_softirq_pending HH" (where "HH" is the hexadecimal softirq vector number) when one or more non-RCU softirq handlers are still enablded when checking to stop the scheduler-tick interrupt. This message is not as enlightening as one might hope, so this commit changes it to "NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #HH.

Thank you! It would be even better if it would explain *why* the problem happened, but I suppose this code doesn't actually know.

--Andy

Quoted 22 lines

---

 tick-sched.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/time/tick-sched.c +1 -1
index f0199a4..349a25a 100644
@@ -927,7 +927,7 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
 
                if (ratelimit < 10 &&
                    (local_softirq_pending() & SOFTIRQ_STOP_IDLE_MASK)) {
-                       pr_warn("NOHZ: local_softirq_pending %02x\n",
+                       pr_warn("NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #%02x\n",
                                (unsigned int) local_softirq_pending());
                        ratelimit++;
                }
#2 lore raw
Quoted 11 lines · Andy Lutomirski
Quoted 7 lines · Paul E. McKenney

Currently, can_stop_idle_tick() prints "NOHZ: local_softirq_pending HH" (where "HH" is the hexadecimal softirq vector number) when one or more non-RCU softirq handlers are still enablded when checking to stop the scheduler-tick interrupt. This message is not as enlightening as one might hope, so this commit changes it to "NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #HH.

Thank you! It would be even better if it would explain *why* the problem happened, but I suppose this code doesn't actually know.

Glad to help!

To your point, is it possible to bisect the appearance of this message, or is it as usual non-reproducible? (Hey, had to ask!)

Thanx, Paul

Quoted 24 lines

--Andy

Quoted 22 lines

---

 tick-sched.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/time/tick-sched.c +1 -1
index f0199a4..349a25a 100644
@@ -927,7 +927,7 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
 
                if (ratelimit < 10 &&
                    (local_softirq_pending() & SOFTIRQ_STOP_IDLE_MASK)) {
-                       pr_warn("NOHZ: local_softirq_pending %02x\n",
+                       pr_warn("NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #%02x\n",
                                (unsigned int) local_softirq_pending());
                        ratelimit++;
                }
#3 lore raw
Quoted 13 lines · Paul E. McKenney
Quoted 9 lines · Andy Lutomirski
Quoted 8 lines · Paul E. McKenney

Currently, can_stop_idle_tick() prints "NOHZ: local_softirq_pending HH" (where "HH" is the hexadecimal softirq vector number) when one or more non-RCU softirq handlers are still enablded when checking to stop the scheduler-tick interrupt. This message is not as enlightening as one might hope, so this commit changes it to "NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #HH.

Quoted 9 lines
Quoted 2 lines

Thank you! It would be even better if it would explain *why* the problem happened, but I suppose this code doesn't actually know.

Glad to help!

To your point, is it possible to bisect the appearance of this message, or is it as usual non-reproducible? (Hey, had to ask!)

In this particular case, I tracked it down by good old fashioned sleuthing for bugs, but it’s still unclear to me precisely how NOHZ gets involved. The bug is that we were entering the kernel from usermode, doing nmi_enter(), turning on interrupts, maybe getting a page fault, raising a signal, turning off interrupts, nmi_exit(), and back to usermode, with the signal still queued and undelivered. This is all kinds of bad, but I still don’t understand what softirqs or idle have to do with it.

But I have the bug fixed now!

#4 lore raw
Quoted 18 lines · Andy Lutomirski
Quoted 18 lines · Paul E. McKenney
Quoted 11 lines · Andy Lutomirski
Quoted 8 lines · Paul E. McKenney

Currently, can_stop_idle_tick() prints "NOHZ: local_softirq_pending HH" (where "HH" is the hexadecimal softirq vector number) when one or more non-RCU softirq handlers are still enablded when checking to stop the scheduler-tick interrupt. This message is not as enlightening as one might hope, so this commit changes it to "NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #HH.

Thank you! It would be even better if it would explain *why* the problem happened, but I suppose this code doesn't actually know.

Glad to help!

Quoted 8 lines
Quoted 4 lines

To your point, is it possible to bisect the appearance of this message, or is it as usual non-reproducible? (Hey, had to ask!)

In this particular case, I tracked it down by good old fashioned sleuthing for bugs, but it’s still unclear to me precisely how NOHZ gets involved. The bug is that we were entering the kernel from usermode, doing nmi_enter(), turning on interrupts, maybe getting a page fault, raising a signal, turning off interrupts, nmi_exit(), and back to usermode, with the signal still queued and undelivered. This is all kinds of bad, but I still don’t understand what softirqs or idle have to do with it.

But I have the bug fixed now!

Glad you found it!

Thanx, Paul

#5 lore raw
In reply to Paul E. McKenney
Quoted 9 lines · Paul E. McKenney

Currently, can_stop_idle_tick() prints "NOHZ: local_softirq_pending HH" (where "HH" is the hexadecimal softirq vector number) when one or more non-RCU softirq handlers are still enablded when checking to stop the scheduler-tick interrupt. This message is not as enlightening as one might hope, so this commit changes it to "NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #HH.

CPU hotplug will sometimes trigger this warning on linux-next from NUMA bare-metal which the commit makes it feel more like errors.

[  267.734981] smpboot: CPU 42 is now offline
[  267.991940] smpboot: Booting Node 2 Processor 42 APIC 0x29
[  267.998370] numa_add_cpu cpu 42 node 2: mask now 8-11,40-43
[  268.092380] ACPI: \_SB_.SCK0.C015: Found 2 idle states
[  268.181917] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #282
[  268.190585] numa_remove_cpu cpu 1 node 0: mask now 0,2-3,32-35
[  268.194075] smpboot: CPU 1 is now offline
[  268.415320] x86: Booting SMP configuration:
[  268.420245] smpboot: Booting Node 0 Processor 1 APIC 0x2
# git clone https://github.com/cailca/linux-mm
# cd linux-mm; make
# ./random 4 (it just soft offline and online each CPU)

The x86.config is also included there. This is only reproducible on x86 so far.

Quoted 19 lines

---

 tick-sched.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/time/tick-sched.c +1 -1
index f0199a4..349a25a 100644
@@ -927,7 +927,7 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
 
 		if (ratelimit < 10 &&
 		    (local_softirq_pending() & SOFTIRQ_STOP_IDLE_MASK)) {
-			pr_warn("NOHZ: local_softirq_pending %02x\n",
+			pr_warn("NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #%02x\n",
 				(unsigned int) local_softirq_pending());
 			ratelimit++;
 		}
#6 lore raw
Quoted 8 lines · Qian Cai
Quoted 7 lines · Paul E. McKenney

Currently, can_stop_idle_tick() prints "NOHZ: local_softirq_pending HH" (where "HH" is the hexadecimal softirq vector number) when one or more non-RCU softirq handlers are still enablded when checking to stop the scheduler-tick interrupt. This message is not as enlightening as one might hope, so this commit changes it to "NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #HH.

Quoted 5 lines
Quoted 2 lines

CPU hotplug will sometimes trigger this warning on linux-next from NUMA bare-metal which the commit makes it feel more like errors.

One point of the change was to make it clear that this is an error. So success! ;-)

Quoted 10 lines
[  267.734981] smpboot: CPU 42 is now offline
[  267.991940] smpboot: Booting Node 2 Processor 42 APIC 0x29
[  267.998370] numa_add_cpu cpu 42 node 2: mask now 8-11,40-43
[  268.092380] ACPI: \_SB_.SCK0.C015: Found 2 idle states
[  268.181917] NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #282
[  268.190585] numa_remove_cpu cpu 1 node 0: mask now 0,2-3,32-35
[  268.194075] smpboot: CPU 1 is now offline
[  268.415320] x86: Booting SMP configuration:
[  268.420245] smpboot: Booting Node 0 Processor 1 APIC 0x2
Quoted 6 lines
# git clone https://github.com/cailca/linux-mm
# cd linux-mm; make
# ./random 4 (it just soft offline and online each CPU)

The x86.config is also included there. This is only reproducible on x86 so far.

I must defer to Frederic and Andy on this one. But now that you mention it, I do see this on some of my runs, which are x86, but not bare metal. I clearly need to upgrade my console-log parsing.

Thanx, Paul

Quoted 18 lines
Quoted 18 lines

---

 tick-sched.c |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
kernel/time/tick-sched.c +1 -1
index f0199a4..349a25a 100644
@@ -927,7 +927,7 @@ static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
 
 		if (ratelimit < 10 &&
 		    (local_softirq_pending() & SOFTIRQ_STOP_IDLE_MASK)) {
-			pr_warn("NOHZ: local_softirq_pending %02x\n",
+			pr_warn("NOHZ tick-stop error: Non-RCU local softirq work is pending, handler #%02x\n",
 				(unsigned int) local_softirq_pending());
 			ratelimit++;
 		}