Skip to contentSkip to conversation

tick/sched: update full_nohz status after SCHED dep is cleared

RFC PATCH In review Juri Lelli 2 participants 14 messages May 20, 2020 – Jun 29, 2020 Nohz, dynticks support

lore
Share & tools
lore

Juri Lelli

author
#0 lore raw

RFC PATCH 1/1 tick/sched: update full_nohz status after SCHED dep is cleared

After tasks enter or leave a runqueue (wakeup/block) SCHED full_nohz dependency is checked (via sched_update_tick_dependency()). In case tick can be stopped on a CPU (see sched_can_stop_tick() for details), SCHED dependency for such CPU is cleared. However, this new information is not used right away to actually stop the tick.

In CONFIG_PREEMPT systems booted with threadirqs option, sched clock tick is serviced by an actual task (ksoftirqd corresponding to the CPU where tick timer fired). So, in case a CPU was running a single task, servicing the timer involves exiting full nozh mode. Problem at this point is that we might lose chances to enter back into full nozh mode, since info about ksoftirqd thread going back to sleep is not used (as mentioned above).

Fix it by calling tick_nozh_full_update_tick(_cpu)() right after SCHED dependency is cleared, so that tick can be promptly stopped.

---
Hi,

I noticed what seems to be the problem described in the changelog while running sysjitter [1] on a PREEMPT system setup for isolation and full nozh, i.e. ... skew_tick=1 nohz=on nohz_full=4-35 rcu_nocbs=4-35 threadirqs (36 CPUs box).

Starting sysjitter with something like the following

perf stat -C 4-35 -e irq_vectors:local_timer_entry taskset \
--cpu-list 4-35 ./sysjitter/sysjitter --runtime 30 200

with vanilla kernel was returning ~5k local_timer_entry events, while that number goes down to ~100 or so with the proposed patch applied.

The following trace snippet also highlight the problematic situation:

...

ksoftirqd/19-125   [019]   170.700754: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.700755: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.700756: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.700757: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.700759: sched_switch:         ksoftirqd/19:125 [120] S ==> sysjitter:2459 [120]
   sysjitter-2459  [019]   170.701740: local_timer_entry:    vector=236
   sysjitter-2459  [019]   170.701742: softirq_raise:        vec=1 [action=TIMER]
   sysjitter-2459  [019]   170.701743: softirq_raise:        vec=7 [action=SCHED]
   sysjitter-2459  [019]   170.701744: local_timer_exit:     vector=236
   sysjitter-2459  [019]   170.701747: sched_wakeup:         ksoftirqd/19:125 [120] success=1 CPU:019
   sysjitter-2459  [019]   170.701748: tick_stop:            success=0 dependency=SCHED
   sysjitter-2459  [019]   170.701749: irq_work_entry:       vector=246
   sysjitter-2459  [019]   170.701750: irq_work_exit:        vector=246
   sysjitter-2459  [019]   170.701751: tick_stop:            success=0 dependency=SCHED
   sysjitter-2459  [019]   170.701753: sched_switch:         sysjitter:2459 [120] R ==> ksoftirqd/19:125 [120]
ksoftirqd/19-125   [019]   170.701754: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.701756: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.701756: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.701758: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.701759: sched_switch:         ksoftirqd/19:125 [120] S ==> sysjitter:2459 [120]
   sysjitter-2459  [019]   170.702740: local_timer_entry:    vector=236
   sysjitter-2459  [019]   170.702742: softirq_raise:        vec=1 [action=TIMER]
   sysjitter-2459  [019]   170.702743: softirq_raise:        vec=7 [action=SCHED]
   sysjitter-2459  [019]   170.702744: local_timer_exit:     vector=236
   sysjitter-2459  [019]   170.702747: sched_wakeup:         ksoftirqd/19:125 [120] success=1 CPU:019
   sysjitter-2459  [019]   170.702748: tick_stop:            success=0 dependency=SCHED
   sysjitter-2459  [019]   170.702749: irq_work_entry:       vector=246
   sysjitter-2459  [019]   170.702750: irq_work_exit:        vector=246
   sysjitter-2459  [019]   170.702751: tick_stop:            success=0 dependency=SCHED
   sysjitter-2459  [019]   170.702753: sched_switch:         sysjitter:2459 [120] R ==> ksoftirqd/19:125 [120]
ksoftirqd/19-125   [019]   170.702755: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.702756: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.702757: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.702758: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.702760: sched_switch:         ksoftirqd/19:125 [120] S ==> sysjitter:2459 [120]
   sysjitter-2459  [019]   170.703740: local_timer_entry:    vector=236
   sysjitter-2459  [019]   170.703742: softirq_raise:        vec=1 [action=TIMER]
   sysjitter-2459  [019]   170.703743: softirq_raise:        vec=7 [action=SCHED]
   sysjitter-2459  [019]   170.703745: local_timer_exit:     vector=236
   sysjitter-2459  [019]   170.703747: sched_wakeup:         ksoftirqd/19:125 [120] success=1 CPU:019
   sysjitter-2459  [019]   170.703748: tick_stop:            success=0 dependency=SCHED
   sysjitter-2459  [019]   170.703749: irq_work_entry:       vector=246
   sysjitter-2459  [019]   170.703750: irq_work_exit:        vector=246
   sysjitter-2459  [019]   170.703751: tick_stop:            success=0 dependency=SCHED
   sysjitter-2459  [019]   170.703753: sched_switch:         sysjitter:2459 [120] R ==> ksoftirqd/19:125 [120]
ksoftirqd/19-125   [019]   170.703755: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.703756: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.703757: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.703758: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.703759: sched_switch:         ksoftirqd/19:125 [120] S ==> sysjitter:2459 [120]
   sysjitter-2459  [019]   170.704500: call_function_single_entry: vector=251
   sysjitter-2459  [019]   170.704502: call_function_single_exit: vector=251
   sysjitter-2459  [019]   170.704504: tick_stop:            success=1 dependency=NONE

...

Luckily sometimes it happens (like in the above) that an interrupt fires on a CPU while ksoftirqd is not executing, SCHED dep is seen as cleared and the CPU enters full nohz. I've however observed pathological cases where CPUs aren't able to enter back full nohz for a long time.

Hope my understanding is correct. Please find below a naive attempt to fix the problem. I'm pretty sure I'm missing details, e.g. I'm not sure it is safe to possibly stop the tick remotely. Consider the following as not more than another way of pointing the problem out. :-)

Best,

Juri

1 - https://github.com/alexeiz/sysjitter

---
 kernel/time/tick-sched.c | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
kernel/time/tick-sched.c +15 -3
index 3e2dc9b8858c..0d696e55d774 100644
@@ -342,11 +342,16 @@ void tick_nohz_dep_set_cpu(int cpu, enum tick_dep_bits bit)
 }
 EXPORT_SYMBOL_GPL(tick_nohz_dep_set_cpu);
 
+static void tick_nohz_full_update_tick_cpu(struct tick_sched *ts, int cpu);
+
 void tick_nohz_dep_clear_cpu(int cpu, enum tick_dep_bits bit)
 {
 	struct tick_sched *ts = per_cpu_ptr(&tick_cpu_sched, cpu);
 
 	atomic_andnot(BIT(bit), &ts->tick_dep_mask);
+
+	if (!in_irq())
+		tick_nohz_full_update_tick_cpu(ts, cpu);
 }
 EXPORT_SYMBOL_GPL(tick_nohz_dep_clear_cpu);
 
@@ -870,11 +875,9 @@ static void tick_nohz_restart_sched_tick(struct tick_sched *ts, ktime_t now)
 	tick_nohz_restart(ts, now);
 }
 
-static void tick_nohz_full_update_tick(struct tick_sched *ts)
+static void tick_nohz_full_update_tick_cpu(struct tick_sched *ts, int cpu)
 {
 #ifdef CONFIG_NO_HZ_FULL
-	int cpu = smp_processor_id();
-
 	if (!tick_nohz_full_cpu(cpu))
 		return;
 
@@ -888,6 +891,15 @@ static void tick_nohz_full_update_tick(struct tick_sched *ts)
 #endif
 }
 
+static void tick_nohz_full_update_tick(struct tick_sched *ts)
+{
+#ifdef CONFIG_NO_HZ_FULL
+	int cpu = smp_processor_id();
+
+	tick_nohz_full_update_tick_cpu(ts, cpu);
+#endif
+}
+
 static bool can_stop_idle_tick(int cpu, struct tick_sched *ts)
 {
 	/*
Signature · 1 line
2.25.3
#1 lore raw

Hi Juri,

Quoted 9 lines · Juri Lelli

After tasks enter or leave a runqueue (wakeup/block) SCHED full_nohz dependency is checked (via sched_update_tick_dependency()). In case tick can be stopped on a CPU (see sched_can_stop_tick() for details), SCHED dependency for such CPU is cleared. However, this new information is not used right away to actually stop the tick.

In CONFIG_PREEMPT systems booted with threadirqs option, sched clock tick is serviced by an actual task (ksoftirqd corresponding to the CPU where tick timer fired).

I must confess I haven't tested threaded IRQs but I was pretty sure that the timer tick is always serviced on hardirq.

Now the timer list callbacks are executed on softirqs. So if the tick itself, executed on hardirq, sees pending timers, it raise the softirq which wakes up ksoftirqd on forced irq thread mode while calling irq_exit(). Then tick_nohz_irq_exit() sees ksoftirqd and the current task on the runqueue, which together can indeed prevent from turning off the tick. But then the root cause is pending timer list callbacks.

Quoted 5 lines

So, in case a CPU was running a single task,
servicing the timer involves exiting full nozh mode. Problem at this point is that we might lose chances to enter back into full nozh mode, since info about ksoftirqd thread going back to sleep is not used (as mentioned above).

It should enter into nohz_full mode in the next tick, which is usually a reasonable delay. If you need guarantee that the tick is stopped before resuming userspace, you need a stronger machinery such as the task isolation patchset.

Let's have a look at the trace below:

Quoted 7 lines
ksoftirqd/19-125   [019]   170.700754: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.700755: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.700756: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.700757: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.700759: sched_switch:         ksoftirqd/19:125 [120] S ==> sysjitter:2459 [120]
   sysjitter-2459  [019]   170.701740: local_timer_entry:    vector=236
   sysjitter-2459  [019]   170.701742: softirq_raise:        vec=1 [action=TIMER]

See here the tick sees pending timer callbacks so it raises the timer softirq.

Quoted 1 line
   sysjitter-2459  [019]   170.701743: softirq_raise:        vec=7 [action=SCHED]

Oh and the scheduler tick activates the scheduler softirq as well.

Quoted 6 lines
   sysjitter-2459  [019]   170.701744: local_timer_exit:     vector=236
   sysjitter-2459  [019]   170.701747: sched_wakeup:         ksoftirqd/19:125 [120] success=1 CPU:019
   sysjitter-2459  [019]   170.701748: tick_stop:            success=0 dependency=SCHED
   sysjitter-2459  [019]   170.701749: irq_work_entry:       vector=246
   sysjitter-2459  [019]   170.701750: irq_work_exit:        vector=246
   sysjitter-2459  [019]   170.701751: tick_stop:            success=0 dependency=SCHED
Quoted 8 lines
   sysjitter-2459  [019]   170.701753: sched_switch:         sysjitter:2459 [120] R ==> ksoftirqd/19:125 [120]
ksoftirqd/19-125   [019]   170.701754: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.701756: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.701756: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.701758: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.701759: sched_switch:         ksoftirqd/19:125 [120] S ==> sysjitter:2459 [120]
   sysjitter-2459  [019]   170.702740: local_timer_entry:    vector=236
   sysjitter-2459  [019]   170.702742: softirq_raise:        vec=1 [action=TIMER]

A new tick but we still have pending timer callback so we'll need to wakeup ksoftirqd again.

I think you should trace timers and check which one is concerned here.

Thanks.

author
#2 lore raw
Quoted 66 lines · Frederic Weisbecker

Hi Juri,

Quoted 9 lines · Juri Lelli

After tasks enter or leave a runqueue (wakeup/block) SCHED full_nohz dependency is checked (via sched_update_tick_dependency()). In case tick can be stopped on a CPU (see sched_can_stop_tick() for details), SCHED dependency for such CPU is cleared. However, this new information is not used right away to actually stop the tick.

In CONFIG_PREEMPT systems booted with threadirqs option, sched clock tick is serviced by an actual task (ksoftirqd corresponding to the CPU where tick timer fired).

I must confess I haven't tested threaded IRQs but I was pretty sure that the timer tick is always serviced on hardirq.

Now the timer list callbacks are executed on softirqs. So if the tick itself, executed on hardirq, sees pending timers, it raise the softirq which wakes up ksoftirqd on forced irq thread mode while calling irq_exit(). Then tick_nohz_irq_exit() sees ksoftirqd and the current task on the runqueue, which together can indeed prevent from turning off the tick. But then the root cause is pending timer list callbacks.

Quoted 5 lines

So, in case a CPU was running a single task,
servicing the timer involves exiting full nozh mode. Problem at this point is that we might lose chances to enter back into full nozh mode, since info about ksoftirqd thread going back to sleep is not used (as mentioned above).

It should enter into nohz_full mode in the next tick, which is usually a reasonable delay. If you need guarantee that the tick is stopped before resuming userspace, you need a stronger machinery such as the task isolation patchset.

Let's have a look at the trace below:

Quoted 7 lines
ksoftirqd/19-125   [019]   170.700754: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.700755: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.700756: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.700757: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.700759: sched_switch:         ksoftirqd/19:125 [120] S ==> sysjitter:2459 [120]
   sysjitter-2459  [019]   170.701740: local_timer_entry:    vector=236
   sysjitter-2459  [019]   170.701742: softirq_raise:        vec=1 [action=TIMER]

See here the tick sees pending timer callbacks so it raises the timer softirq.

Quoted 1 line
   sysjitter-2459  [019]   170.701743: softirq_raise:        vec=7 [action=SCHED]

Oh and the scheduler tick activates the scheduler softirq as well.

Quoted 6 lines
   sysjitter-2459  [019]   170.701744: local_timer_exit:     vector=236
   sysjitter-2459  [019]   170.701747: sched_wakeup:         ksoftirqd/19:125 [120] success=1 CPU:019
   sysjitter-2459  [019]   170.701748: tick_stop:            success=0 dependency=SCHED
   sysjitter-2459  [019]   170.701749: irq_work_entry:       vector=246
   sysjitter-2459  [019]   170.701750: irq_work_exit:        vector=246
   sysjitter-2459  [019]   170.701751: tick_stop:            success=0 dependency=SCHED
Quoted 8 lines
   sysjitter-2459  [019]   170.701753: sched_switch:         sysjitter:2459 [120] R ==> ksoftirqd/19:125 [120]
ksoftirqd/19-125   [019]   170.701754: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.701756: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/19-125   [019]   170.701756: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.701758: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/19-125   [019]   170.701759: sched_switch:         ksoftirqd/19:125 [120] S ==> sysjitter:2459 [120]
   sysjitter-2459  [019]   170.702740: local_timer_entry:    vector=236
   sysjitter-2459  [019]   170.702742: softirq_raise:        vec=1 [action=TIMER]
Quoted 4 lines

A new tick but we still have pending timer callback so we'll need to wakeup ksoftirqd again.

I think you should trace timers and check which one is concerned here.

Hummm, so I enabled 'timer:*', anything else you think I should be looking at?

...

ksoftirqd/13-117   [013]   148.265945: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.265947: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.265948: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.265950: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.265952: sched_switch:         ksoftirqd/13:117 [120] S ==> sysjitter:2536 [120]
   sysjitter-2536  [013]   148.266912: local_timer_entry:    vector=236
   sysjitter-2536  [013]   148.266914: hrtimer_cancel:       hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.266916: hrtimer_expire_entry: hrtimer=0xffff9807df91bee0 now=148249107837 function=tick_sched_timer/0x0
   sysjitter-2536  [013]   148.266918: softirq_raise:        vec=1 [action=TIMER]
   sysjitter-2536  [013]   148.266920: softirq_raise:        vec=7 [action=SCHED]
   sysjitter-2536  [013]   148.266922: hrtimer_expire_exit:  hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.266924: hrtimer_start:        hrtimer=0xffff9807df91bee0 function=tick_sched_timer/0x0 expires=148250105422 softexpires=148250105422
   sysjitter-2536  [013]   148.266926: local_timer_exit:     vector=236
   sysjitter-2536  [013]   148.266930: sched_wakeup:         ksoftirqd/13:117 [120] success=1 CPU:013
   sysjitter-2536  [013]   148.266932: tick_stop:            success=0 dependency=SCHED
   sysjitter-2536  [013]   148.266934: irq_work_entry:       vector=246
   sysjitter-2536  [013]   148.266936: irq_work_exit:        vector=246
   sysjitter-2536  [013]   148.266937: tick_stop:            success=0 dependency=SCHED
   sysjitter-2536  [013]   148.266940: sched_switch:         sysjitter:2536 [120] R ==> ksoftirqd/13:117 [120]
ksoftirqd/13-117   [013]   148.266943: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.266946: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.266946: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.266948: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.266950: sched_switch:         ksoftirqd/13:117 [120] S ==> sysjitter:2536 [120]
   sysjitter-2536  [013]   148.267912: local_timer_entry:    vector=236
   sysjitter-2536  [013]   148.267914: hrtimer_cancel:       hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.267916: hrtimer_expire_entry: hrtimer=0xffff9807df91bee0 now=148250107750 function=tick_sched_timer/0x0
   sysjitter-2536  [013]   148.267918: softirq_raise:        vec=1 [action=TIMER]
   sysjitter-2536  [013]   148.267920: softirq_raise:        vec=7 [action=SCHED]
   sysjitter-2536  [013]   148.267922: hrtimer_expire_exit:  hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.267924: hrtimer_start:        hrtimer=0xffff9807df91bee0 function=tick_sched_timer/0x0 expires=148251105422 softexpires=148251105422
   sysjitter-2536  [013]   148.267926: local_timer_exit:     vector=236
   sysjitter-2536  [013]   148.267929: sched_wakeup:         ksoftirqd/13:117 [120] success=1 CPU:013
   sysjitter-2536  [013]   148.267931: tick_stop:            success=0 dependency=SCHED
   sysjitter-2536  [013]   148.267934: irq_work_entry:       vector=246
   sysjitter-2536  [013]   148.267936: irq_work_exit:        vector=246
   sysjitter-2536  [013]   148.267937: tick_stop:            success=0 dependency=SCHED
   sysjitter-2536  [013]   148.267940: sched_switch:         sysjitter:2536 [120] R ==> ksoftirqd/13:117 [120]
ksoftirqd/13-117   [013]   148.267943: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.267945: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.267946: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.267947: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.267949: sched_switch:         ksoftirqd/13:117 [120] S ==> sysjitter:2536 [120]
   sysjitter-2536  [013]   148.268865: call_function_single_entry: vector=251
   sysjitter-2536  [013]   148.268867: call_function_single_exit: vector=251
   sysjitter-2536  [013]   148.268870: tick_stop:            success=1 dependency=NONE
   sysjitter-2536  [013]   148.268872: hrtimer_cancel:       hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.268876: call_function_single_entry: vector=251
   sysjitter-2536  [013]   148.268878: call_function_single_exit: vector=251

...

it looks like tick sched timer is kept armed until tick_stop succedes (because of the check performed after _single_exit)?

Thanks for your quick reply!

#3 lore raw
Quoted 4 lines · Juri Lelli

On 20/05/20 18:24, Frederic Weisbecker wrote:

Hummm, so I enabled 'timer:*', anything else you think I should be looking at?

Are you sure you also enabled timer_expire_entry?
Because:

Quoted 3 lines

...
ksoftirqd/13-117 [013] 148.265945: softirq_entry: vec=1 [action=TIMER]

...the timer softirq keep triggering without any timer callback executed. If that's the case, we may be dealing with an interesting issue.

Thanks.

Quoted 55 lines
ksoftirqd/13-117   [013]   148.265947: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.265948: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.265950: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.265952: sched_switch:         ksoftirqd/13:117 [120] S ==> sysjitter:2536 [120]
   sysjitter-2536  [013]   148.266912: local_timer_entry:    vector=236
   sysjitter-2536  [013]   148.266914: hrtimer_cancel:       hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.266916: hrtimer_expire_entry: hrtimer=0xffff9807df91bee0 now=148249107837 function=tick_sched_timer/0x0
   sysjitter-2536  [013]   148.266918: softirq_raise:        vec=1 [action=TIMER]
   sysjitter-2536  [013]   148.266920: softirq_raise:        vec=7 [action=SCHED]
   sysjitter-2536  [013]   148.266922: hrtimer_expire_exit:  hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.266924: hrtimer_start:        hrtimer=0xffff9807df91bee0 function=tick_sched_timer/0x0 expires=148250105422 softexpires=148250105422
   sysjitter-2536  [013]   148.266926: local_timer_exit:     vector=236
   sysjitter-2536  [013]   148.266930: sched_wakeup:         ksoftirqd/13:117 [120] success=1 CPU:013
   sysjitter-2536  [013]   148.266932: tick_stop:            success=0 dependency=SCHED
   sysjitter-2536  [013]   148.266934: irq_work_entry:       vector=246
   sysjitter-2536  [013]   148.266936: irq_work_exit:        vector=246
   sysjitter-2536  [013]   148.266937: tick_stop:            success=0 dependency=SCHED
   sysjitter-2536  [013]   148.266940: sched_switch:         sysjitter:2536 [120] R ==> ksoftirqd/13:117 [120]
ksoftirqd/13-117   [013]   148.266943: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.266946: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.266946: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.266948: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.266950: sched_switch:         ksoftirqd/13:117 [120] S ==> sysjitter:2536 [120]
   sysjitter-2536  [013]   148.267912: local_timer_entry:    vector=236
   sysjitter-2536  [013]   148.267914: hrtimer_cancel:       hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.267916: hrtimer_expire_entry: hrtimer=0xffff9807df91bee0 now=148250107750 function=tick_sched_timer/0x0
   sysjitter-2536  [013]   148.267918: softirq_raise:        vec=1 [action=TIMER]
   sysjitter-2536  [013]   148.267920: softirq_raise:        vec=7 [action=SCHED]
   sysjitter-2536  [013]   148.267922: hrtimer_expire_exit:  hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.267924: hrtimer_start:        hrtimer=0xffff9807df91bee0 function=tick_sched_timer/0x0 expires=148251105422 softexpires=148251105422
   sysjitter-2536  [013]   148.267926: local_timer_exit:     vector=236
   sysjitter-2536  [013]   148.267929: sched_wakeup:         ksoftirqd/13:117 [120] success=1 CPU:013
   sysjitter-2536  [013]   148.267931: tick_stop:            success=0 dependency=SCHED
   sysjitter-2536  [013]   148.267934: irq_work_entry:       vector=246
   sysjitter-2536  [013]   148.267936: irq_work_exit:        vector=246
   sysjitter-2536  [013]   148.267937: tick_stop:            success=0 dependency=SCHED
   sysjitter-2536  [013]   148.267940: sched_switch:         sysjitter:2536 [120] R ==> ksoftirqd/13:117 [120]
ksoftirqd/13-117   [013]   148.267943: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.267945: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.267946: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.267947: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.267949: sched_switch:         ksoftirqd/13:117 [120] S ==> sysjitter:2536 [120]
   sysjitter-2536  [013]   148.268865: call_function_single_entry: vector=251
   sysjitter-2536  [013]   148.268867: call_function_single_exit: vector=251
   sysjitter-2536  [013]   148.268870: tick_stop:            success=1 dependency=NONE
   sysjitter-2536  [013]   148.268872: hrtimer_cancel:       hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.268876: call_function_single_entry: vector=251
   sysjitter-2536  [013]   148.268878: call_function_single_exit: vector=251

...

it looks like tick sched timer is kept armed until tick_stop succedes (because of the check performed after _single_exit)?

Thanks for your quick reply!

author
#4 lore raw
Quoted 8 lines · Frederic Weisbecker
Quoted 4 lines · Juri Lelli

On 20/05/20 18:24, Frederic Weisbecker wrote:

Hummm, so I enabled 'timer:*', anything else you think I should be looking at?

Are you sure you also enabled timer_expire_entry?
Because:

I run with

trace-cmd record -e sched_switch -e sched_wakeup -e sched_migrate_task \
-e 'timer:*' -e 'irq_vectors:*' -e 'irq:softirq*' \
-e 'irq:irq_handler*' taskset --cpu-list 4-35 ./sysjitter/sysjitter \
--runtime 10 200

And

$ grep ' timer_expire_entry:' trace.txt | wc -l
1173

seems to confirm that the event was indeed enabled, e.g.

ksoftirqd/11-100 [011] 159.270023: timer_expire_entry: timer=0xffff9807df8a5a60 function=delayed_work_timer_fn now=4294826418 baseclk=4294825984

Quoted 66 lines
Quoted 4 lines

...
ksoftirqd/13-117 [013] 148.265945: softirq_entry: vec=1 [action=TIMER]

...the timer softirq keep triggering without any timer callback executed. If that's the case, we may be dealing with an interesting issue.

Thanks.

Quoted 55 lines
ksoftirqd/13-117   [013]   148.265947: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.265948: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.265950: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.265952: sched_switch:         ksoftirqd/13:117 [120] S ==> sysjitter:2536 [120]
   sysjitter-2536  [013]   148.266912: local_timer_entry:    vector=236
   sysjitter-2536  [013]   148.266914: hrtimer_cancel:       hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.266916: hrtimer_expire_entry: hrtimer=0xffff9807df91bee0 now=148249107837 function=tick_sched_timer/0x0
   sysjitter-2536  [013]   148.266918: softirq_raise:        vec=1 [action=TIMER]
   sysjitter-2536  [013]   148.266920: softirq_raise:        vec=7 [action=SCHED]
   sysjitter-2536  [013]   148.266922: hrtimer_expire_exit:  hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.266924: hrtimer_start:        hrtimer=0xffff9807df91bee0 function=tick_sched_timer/0x0 expires=148250105422 softexpires=148250105422
   sysjitter-2536  [013]   148.266926: local_timer_exit:     vector=236
   sysjitter-2536  [013]   148.266930: sched_wakeup:         ksoftirqd/13:117 [120] success=1 CPU:013
   sysjitter-2536  [013]   148.266932: tick_stop:            success=0 dependency=SCHED
   sysjitter-2536  [013]   148.266934: irq_work_entry:       vector=246
   sysjitter-2536  [013]   148.266936: irq_work_exit:        vector=246
   sysjitter-2536  [013]   148.266937: tick_stop:            success=0 dependency=SCHED
   sysjitter-2536  [013]   148.266940: sched_switch:         sysjitter:2536 [120] R ==> ksoftirqd/13:117 [120]
ksoftirqd/13-117   [013]   148.266943: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.266946: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.266946: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.266948: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.266950: sched_switch:         ksoftirqd/13:117 [120] S ==> sysjitter:2536 [120]
   sysjitter-2536  [013]   148.267912: local_timer_entry:    vector=236
   sysjitter-2536  [013]   148.267914: hrtimer_cancel:       hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.267916: hrtimer_expire_entry: hrtimer=0xffff9807df91bee0 now=148250107750 function=tick_sched_timer/0x0
   sysjitter-2536  [013]   148.267918: softirq_raise:        vec=1 [action=TIMER]
   sysjitter-2536  [013]   148.267920: softirq_raise:        vec=7 [action=SCHED]
   sysjitter-2536  [013]   148.267922: hrtimer_expire_exit:  hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.267924: hrtimer_start:        hrtimer=0xffff9807df91bee0 function=tick_sched_timer/0x0 expires=148251105422 softexpires=148251105422
   sysjitter-2536  [013]   148.267926: local_timer_exit:     vector=236
   sysjitter-2536  [013]   148.267929: sched_wakeup:         ksoftirqd/13:117 [120] success=1 CPU:013
   sysjitter-2536  [013]   148.267931: tick_stop:            success=0 dependency=SCHED
   sysjitter-2536  [013]   148.267934: irq_work_entry:       vector=246
   sysjitter-2536  [013]   148.267936: irq_work_exit:        vector=246
   sysjitter-2536  [013]   148.267937: tick_stop:            success=0 dependency=SCHED
   sysjitter-2536  [013]   148.267940: sched_switch:         sysjitter:2536 [120] R ==> ksoftirqd/13:117 [120]
ksoftirqd/13-117   [013]   148.267943: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.267945: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/13-117   [013]   148.267946: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.267947: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/13-117   [013]   148.267949: sched_switch:         ksoftirqd/13:117 [120] S ==> sysjitter:2536 [120]
   sysjitter-2536  [013]   148.268865: call_function_single_entry: vector=251
   sysjitter-2536  [013]   148.268867: call_function_single_exit: vector=251
   sysjitter-2536  [013]   148.268870: tick_stop:            success=1 dependency=NONE
   sysjitter-2536  [013]   148.268872: hrtimer_cancel:       hrtimer=0xffff9807df91bee0
   sysjitter-2536  [013]   148.268876: call_function_single_entry: vector=251
   sysjitter-2536  [013]   148.268878: call_function_single_exit: vector=251

...

it looks like tick sched timer is kept armed until tick_stop succedes (because of the check performed after _single_exit)?

Thanks for your quick reply!

#5 lore raw
Quoted 17 lines · Juri Lelli
Quoted 8 lines · Frederic Weisbecker
Quoted 4 lines · Juri Lelli

On 20/05/20 18:24, Frederic Weisbecker wrote:

Hummm, so I enabled 'timer:*', anything else you think I should be looking at?

Are you sure you also enabled timer_expire_entry?
Because:

I run with

trace-cmd record -e sched_switch -e sched_wakeup -e sched_migrate_task \

(quoted diff) +0 -3
-e 'timer:*' -e 'irq_vectors:*' -e 'irq:softirq*' \
-e 'irq:irq_handler*' taskset --cpu-list 4-35 ./sysjitter/sysjitter \
--runtime 10 200
Quoted 8 lines

And

$ grep ' timer_expire_entry:' trace.txt | wc -l
1173

seems to confirm that the event was indeed enabled, e.g.

ksoftirqd/11-100 [011] 159.270023: timer_expire_entry: timer=0xffff9807df8a5a60 function=delayed_work_timer_fn now=4294826418 baseclk=4294825984

That's interesting! I'll see if I can reproduce that with threaded irqs. If not we'll have to investigate on your machine. We really don't want to trigger timer softirqs when it's not necessary.

author
#6 lore raw
Quoted 23 lines · Frederic Weisbecker
Quoted 22 lines · Juri Lelli
Quoted 8 lines · Frederic Weisbecker
Quoted 4 lines · Juri Lelli

On 20/05/20 18:24, Frederic Weisbecker wrote:

Hummm, so I enabled 'timer:*', anything else you think I should be looking at?

Are you sure you also enabled timer_expire_entry?
Because:

I run with

trace-cmd record -e sched_switch -e sched_wakeup -e sched_migrate_task \

(quoted diff) +0 -3
-e 'timer:*' -e 'irq_vectors:*' -e 'irq:softirq*' \
-e 'irq:irq_handler*' taskset --cpu-list 4-35 ./sysjitter/sysjitter \
--runtime 10 200

And

$ grep ' timer_expire_entry:' trace.txt | wc -l
1173
Quoted 8 lines
Quoted 3 lines

seems to confirm that the event was indeed enabled, e.g.

ksoftirqd/11-100 [011] 159.270023: timer_expire_entry: timer=0xffff9807df8a5a60 function=delayed_work_timer_fn now=4294826418 baseclk=4294825984

That's interesting! I'll see if I can reproduce that with threaded irqs. If not we'll have to investigate on your machine. We really don't want to trigger timer softirqs when it's not necessary.

So, I managed to get a bit more info on this, please check the following (which I edited a bit to hopefully make it clearer and apologies for long lines, but I believe trimming them would mess things up)

---

ksoftirqd/3-26    [003]    99.942485: timer_expire_entry:   timer=0xffffffffa55a9d20 function=clocksource_watchdog now=4294759328 baseclk=4294759328
ksoftirqd/3-26    [003]    99.942489: timer_start:          timer=0xffffffffa55a9d20 function=clocksource_watchdog expires=4294759822 [timeout=494] cpu=4 idx=114 flags=D|P|I
ksoftirqd/3-26    [003]    99.942491: timer_expire_exit:    timer=0xffffffffa55a9d20
  sysjitter-2377  [004]    99.942491: call_function_single_entry: vector=251
  sysjitter-2377  [004]    99.942492: call_function_single_exit: vector=251
ksoftirqd/3-26    [003]    99.942493: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/3-26    [003]    99.942494: softirq_entry:        vec=7 [action=SCHED]
  sysjitter-2377  [004]    99.942494: hrtimer_start:        hrtimer=0xffff92e9df91fd20 function=tick_sched_timer/0x0 expires=92630457385 softexpires=92630457385

---

So, my understanding is that clock gets reactivated on CPU 4 to run clocksource watchdog (indeed if I use tsc=nowatchdog this above doesn't happen, but the potential issue is still there I guess).

---

  sysjitter-2377  [004]   100.438455: local_timer_entry:    vector=236
  sysjitter-2377  [004]   100.438456: hrtimer_cancel:       hrtimer=0xffff92e9df91fd20
  sysjitter-2377  [004]   100.438458: hrtimer_expire_entry: hrtimer=0xffff92e9df91fd20 now=92630461337 function=tick_sched_timer/0x0
  sysjitter-2377  [004]   100.438460: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921)
=> __raise_softirq_irqoff (ffffffffa36daf50)
=> raise_softirq (ffffffffa36db0fe)
=> update_process_times (ffffffffa3768f9d)
=> tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0)
=> hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8)
=> apic_timer_interrupt (ffffffffa4001b7f)

  sysjitter-2377  [004]   100.438461: softirq_raise:        vec=1 [action=TIMER]
  sysjitter-2377  [004]   100.438464: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921)
=> __raise_softirq_irqoff (ffffffffa36daf50)
=> raise_softirq (ffffffffa36db0fe)
=> trigger_load_balance (ffffffffa371cb9c)
=> update_process_times (ffffffffa3768fc7)
=> tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0)
=> hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8)
=> apic_timer_interrupt (ffffffffa4001b7f)
sysjitter-2377 [004] 100.438465: softirq_raise: vec=7 [action=SCHED]
---

Tick fires, the two softirq_raise (which you wondered about) seems to come from

1. update_process_times::run_local_timers (TIMER_SOFTIRQ)
2. update_process_times::scheduler_tick::trigger_load_balance (SCHED_SOFTIRQ)

---

  sysjitter-2377  [004]   100.438468: hrtimer_expire_exit:  hrtimer=0xffff92e9df91fd20
  sysjitter-2377  [004]   100.438470: local_timer_exit:     vector=236
  sysjitter-2377  [004]   100.438473: sched_wakeup:         ksoftirqd/4:31 [120] success=1 CPU:004
  sysjitter-2377  [004]   100.438474: tick_stop:            success=0 dependency=SCHED
  sysjitter-2377  [004]   100.438476: hrtimer_start:        hrtimer=0xffff92e9df91fd20 function=tick_sched_timer/0x0 expires=92630512937 softexpires=92630512937
  sysjitter-2377  [004]   100.438484: irq_work_entry:       vector=246
  sysjitter-2377  [004]   100.438491: irq_work_exit:        vector=246
  sysjitter-2377  [004]   100.438492: tick_stop:            success=0 dependency=SCHED
  sysjitter-2377  [004]   100.438495: sched_switch:         sysjitter:2377 [120] R ==> ksoftirqd/4:31 [120]
ksoftirqd/4-31    [004]   100.438497: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/4-31    [004]   100.438499: timer_cancel:         timer=0xffffffffa55a9d20
ksoftirqd/4-31    [004]   100.438501: timer_expire_entry:   timer=0xffffffffa55a9d20 function=clocksource_watchdog now=4294759824 baseclk=4294759824
ksoftirqd/4-31    [004]   100.438504: timer_start:          timer=0xffffffffa55a9d20 function=clocksource_watchdog expires=4294760322 [timeout=498] cpu=5 idx=113 flags=D|P|I

---

Queueing clocksource watchdog on CPU5.

---

ksoftirqd/4-31    [004]   100.438506: timer_expire_exit:    timer=0xffffffffa55a9d20
ksoftirqd/4-31    [004]   100.438507: local_timer_entry:    vector=236

---

Back to back timer?

---

ksoftirqd/4-31    [004]   100.438509: hrtimer_cancel:       hrtimer=0xffff92e9df91fd20
ksoftirqd/4-31    [004]   100.438511: hrtimer_expire_entry: hrtimer=0xffff92e9df91fd20 now=92630515022 function=tick_sched_timer/0x0
ksoftirqd/4-31    [004]   100.438515: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921) => __raise_softirq_irqoff (ffffffffa36daf50) => raise_softirq (ffffffffa36db0fe)
=> rcu_sched_clock_irq (ffffffffa375af4a) => update_process_times (ffffffffa3768fa4) => tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0) => hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8) => apic_timer_interrupt (ffffffffa4001b7f) => filter_pred_32 (ffffffffa37f3357)
=> filter_match_preds (ffffffffa37f3510)
=> trace_event_buffer_commit (ffffffffa37dc7eb) => trace_event_raw_event_softirq (ffffffffa36dab77) => __do_softirq (ffffffffa420025a)
=> run_ksoftirqd (ffffffffa36dadc6)
=> smpboot_thread_fn (ffffffffa36ffdb8)
=> kthread (ffffffffa36f9fb7)
=> ret_from_fork (ffffffffa4000215)

ksoftirqd/4-31    [004]   100.438516: softirq_raise:        vec=9 [action=RCU]
ksoftirqd/4-31    [004]   100.438520: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921) => __raise_softirq_irqoff (ffffffffa36daf50) => raise_softirq (ffffffffa36db0fe)
=> trigger_load_balance (ffffffffa371cb9c) => update_process_times (ffffffffa3768fc7) => tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0) => hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8) => apic_timer_interrupt (ffffffffa4001b7f) => filter_pred_32 (ffffffffa37f3357)
=> filter_match_preds (ffffffffa37f3510)
=> trace_event_buffer_commit (ffffffffa37dc7eb) => trace_event_raw_event_softirq (ffffffffa36dab77) => __do_softirq (ffffffffa420025a)
=> run_ksoftirqd (ffffffffa36dadc6)
=> smpboot_thread_fn (ffffffffa36ffdb8)
=> kthread (ffffffffa36f9fb7)
=> ret_from_fork (ffffffffa4000215)

ksoftirqd/4-31    [004]   100.438520: softirq_raise:        vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438521: hrtimer_expire_exit:  hrtimer=0xffff92e9df91fd20
ksoftirqd/4-31    [004]   100.438523: hrtimer_start:        hrtimer=0xffff92e9df91fd20 function=tick_sched_timer/0x0 expires=92631512937 softexpires=92631512937
ksoftirqd/4-31    [004]   100.438525: local_timer_exit:     vector=236
ksoftirqd/4-31    [004]   100.438527: tick_stop:            success=0 dependency=SCHED

---

At this point (I think) ksoftirqd is already going to sleep, so only sysjitter will be eventually executing on CPU4, but tick_stop check still sees nr_running == 2.

---

ksoftirqd/4-31    [004]   100.438530: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/4-31    [004]   100.438531: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438533: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438534: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438535: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438536: softirq_entry:        vec=9 [action=RCU]
ksoftirqd/4-31    [004]   100.438537: softirq_exit:         vec=9 [action=RCU]
ksoftirqd/4-31    [004]   100.438540: sched_switch:         ksoftirqd/4:31 [120] S ==> sysjitter:2377 [120]

---

And tick again fires below.

---

  sysjitter-2377  [004]   100.439509: local_timer_entry:    vector=236
  sysjitter-2377  [004]   100.439511: hrtimer_cancel:       hrtimer=0xffff92e9df91fd20
  sysjitter-2377  [004]   100.439512: hrtimer_expire_entry: hrtimer=0xffff92e9df91fd20 now=92631515408 function=tick_sched_timer/0x0
  sysjitter-2377  [004]   100.439514: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921)
=> __raise_softirq_irqoff (ffffffffa36daf50)
=> raise_softirq (ffffffffa36db0fe)
=> update_process_times (ffffffffa3768f9d)
=> tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0)
=> hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8)
=> apic_timer_interrupt (ffffffffa4001b7f)

sysjitter-2377  [004]   100.439515: softirq_raise:        vec=1 [action=TIMER]
sysjitter-2377  [004]   100.439517: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921)
=> __raise_softirq_irqoff (ffffffffa36daf50)
=> raise_softirq (ffffffffa36db0fe)
=> trigger_load_balance (ffffffffa371cb9c)
=> update_process_times (ffffffffa3768fc7)
=> tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0)
=> hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8)
=> apic_timer_interrupt (ffffffffa4001b7f)

  sysjitter-2377  [004]   100.439518: softirq_raise:        vec=7 [action=SCHED]
  sysjitter-2377  [004]   100.439520: hrtimer_expire_exit:  hrtimer=0xffff92e9df91fd20
  sysjitter-2377  [004]   100.439521: hrtimer_start:        hrtimer=0xffff92e9df91fd20 function=tick_sched_timer/0x0 expires=92632512937 softexpires=92632512937
  sysjitter-2377  [004]   100.439522: local_timer_exit:     vector=236
  sysjitter-2377  [004]   100.439525: sched_wakeup:         ksoftirqd/4:31 [120] success=1 CPU:004
  sysjitter-2377  [004]   100.439527: tick_stop:            success=0 dependency=SCHED
  sysjitter-2377  [004]   100.439529: irq_work_entry:       vector=246
  sysjitter-2377  [004]   100.439531: irq_work_exit:        vector=246
  sysjitter-2377  [004]   100.439532: tick_stop:            success=0 dependency=SCHED
  sysjitter-2377  [004]   100.439534: sched_switch:         sysjitter:2377 [120] R ==> ksoftirqd/4:31 [120]
ksoftirqd/4-31    [004]   100.439536: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/4-31    [004]   100.439537: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/4-31    [004]   100.439538: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.439540: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.439541: sched_switch:         ksoftirqd/4:31 [120] S ==> sysjitter:2377 [120]

---

History seems to repeat itself until ~2s later (this is one of the pathological runs) when an ipi fires on CPU4 and SCHED dependecy is seen as cleared (as only sysjitter thread was is running at that point).

Does this make sense and help in any way? Suggestions for debugging this further? :-)

Thanks!

Juri

author
#7 lore raw

Hi,

Quoted 227 lines · Juri Lelli
Quoted 23 lines · Frederic Weisbecker
Quoted 22 lines · Juri Lelli
Quoted 8 lines · Frederic Weisbecker
Quoted 4 lines · Juri Lelli

On 20/05/20 18:24, Frederic Weisbecker wrote:

Hummm, so I enabled 'timer:*', anything else you think I should be looking at?

Are you sure you also enabled timer_expire_entry?
Because:

I run with

trace-cmd record -e sched_switch -e sched_wakeup -e sched_migrate_task \

(quoted diff) +0 -3
-e 'timer:*' -e 'irq_vectors:*' -e 'irq:softirq*' \
-e 'irq:irq_handler*' taskset --cpu-list 4-35 ./sysjitter/sysjitter \
--runtime 10 200

And

$ grep ' timer_expire_entry:' trace.txt | wc -l
1173
Quoted 8 lines
Quoted 3 lines

seems to confirm that the event was indeed enabled, e.g.

ksoftirqd/11-100 [011] 159.270023: timer_expire_entry: timer=0xffff9807df8a5a60 function=delayed_work_timer_fn now=4294826418 baseclk=4294825984

That's interesting! I'll see if I can reproduce that with threaded irqs. If not we'll have to investigate on your machine. We really don't want to trigger timer softirqs when it's not necessary.

So, I managed to get a bit more info on this, please check the following (which I edited a bit to hopefully make it clearer and apologies for long lines, but I believe trimming them would mess things up)

---

ksoftirqd/3-26    [003]    99.942485: timer_expire_entry:   timer=0xffffffffa55a9d20 function=clocksource_watchdog now=4294759328 baseclk=4294759328
ksoftirqd/3-26    [003]    99.942489: timer_start:          timer=0xffffffffa55a9d20 function=clocksource_watchdog expires=4294759822 [timeout=494] cpu=4 idx=114 flags=D|P|I
ksoftirqd/3-26    [003]    99.942491: timer_expire_exit:    timer=0xffffffffa55a9d20
  sysjitter-2377  [004]    99.942491: call_function_single_entry: vector=251
  sysjitter-2377  [004]    99.942492: call_function_single_exit: vector=251
ksoftirqd/3-26    [003]    99.942493: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/3-26    [003]    99.942494: softirq_entry:        vec=7 [action=SCHED]
  sysjitter-2377  [004]    99.942494: hrtimer_start:        hrtimer=0xffff92e9df91fd20 function=tick_sched_timer/0x0 expires=92630457385 softexpires=92630457385

---

So, my understanding is that clock gets reactivated on CPU 4 to run clocksource watchdog (indeed if I use tsc=nowatchdog this above doesn't happen, but the potential issue is still there I guess).

---

  sysjitter-2377  [004]   100.438455: local_timer_entry:    vector=236
  sysjitter-2377  [004]   100.438456: hrtimer_cancel:       hrtimer=0xffff92e9df91fd20
  sysjitter-2377  [004]   100.438458: hrtimer_expire_entry: hrtimer=0xffff92e9df91fd20 now=92630461337 function=tick_sched_timer/0x0
  sysjitter-2377  [004]   100.438460: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921)
=> __raise_softirq_irqoff (ffffffffa36daf50)
=> raise_softirq (ffffffffa36db0fe)
=> update_process_times (ffffffffa3768f9d)
=> tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0)
=> hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8)
=> apic_timer_interrupt (ffffffffa4001b7f)

  sysjitter-2377  [004]   100.438461: softirq_raise:        vec=1 [action=TIMER]
  sysjitter-2377  [004]   100.438464: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921)
=> __raise_softirq_irqoff (ffffffffa36daf50)
=> raise_softirq (ffffffffa36db0fe)
=> trigger_load_balance (ffffffffa371cb9c)
=> update_process_times (ffffffffa3768fc7)
=> tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0)
=> hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8)
=> apic_timer_interrupt (ffffffffa4001b7f)
sysjitter-2377 [004] 100.438465: softirq_raise: vec=7 [action=SCHED]
---

Tick fires, the two softirq_raise (which you wondered about) seems to come from

1. update_process_times::run_local_timers (TIMER_SOFTIRQ)
2. update_process_times::scheduler_tick::trigger_load_balance (SCHED_SOFTIRQ)

---

  sysjitter-2377  [004]   100.438468: hrtimer_expire_exit:  hrtimer=0xffff92e9df91fd20
  sysjitter-2377  [004]   100.438470: local_timer_exit:     vector=236
  sysjitter-2377  [004]   100.438473: sched_wakeup:         ksoftirqd/4:31 [120] success=1 CPU:004
  sysjitter-2377  [004]   100.438474: tick_stop:            success=0 dependency=SCHED
  sysjitter-2377  [004]   100.438476: hrtimer_start:        hrtimer=0xffff92e9df91fd20 function=tick_sched_timer/0x0 expires=92630512937 softexpires=92630512937
  sysjitter-2377  [004]   100.438484: irq_work_entry:       vector=246
  sysjitter-2377  [004]   100.438491: irq_work_exit:        vector=246
  sysjitter-2377  [004]   100.438492: tick_stop:            success=0 dependency=SCHED
  sysjitter-2377  [004]   100.438495: sched_switch:         sysjitter:2377 [120] R ==> ksoftirqd/4:31 [120]
ksoftirqd/4-31    [004]   100.438497: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/4-31    [004]   100.438499: timer_cancel:         timer=0xffffffffa55a9d20
ksoftirqd/4-31    [004]   100.438501: timer_expire_entry:   timer=0xffffffffa55a9d20 function=clocksource_watchdog now=4294759824 baseclk=4294759824
ksoftirqd/4-31    [004]   100.438504: timer_start:          timer=0xffffffffa55a9d20 function=clocksource_watchdog expires=4294760322 [timeout=498] cpu=5 idx=113 flags=D|P|I

---

Queueing clocksource watchdog on CPU5.

---

ksoftirqd/4-31    [004]   100.438506: timer_expire_exit:    timer=0xffffffffa55a9d20
ksoftirqd/4-31    [004]   100.438507: local_timer_entry:    vector=236

---

Back to back timer?

---

ksoftirqd/4-31    [004]   100.438509: hrtimer_cancel:       hrtimer=0xffff92e9df91fd20
ksoftirqd/4-31    [004]   100.438511: hrtimer_expire_entry: hrtimer=0xffff92e9df91fd20 now=92630515022 function=tick_sched_timer/0x0
ksoftirqd/4-31    [004]   100.438515: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921) => __raise_softirq_irqoff (ffffffffa36daf50) => raise_softirq (ffffffffa36db0fe)
=> rcu_sched_clock_irq (ffffffffa375af4a) => update_process_times (ffffffffa3768fa4) => tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0) => hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8) => apic_timer_interrupt (ffffffffa4001b7f) => filter_pred_32 (ffffffffa37f3357)
=> filter_match_preds (ffffffffa37f3510)
=> trace_event_buffer_commit (ffffffffa37dc7eb) => trace_event_raw_event_softirq (ffffffffa36dab77) => __do_softirq (ffffffffa420025a)
=> run_ksoftirqd (ffffffffa36dadc6)
=> smpboot_thread_fn (ffffffffa36ffdb8)
=> kthread (ffffffffa36f9fb7)
=> ret_from_fork (ffffffffa4000215)

ksoftirqd/4-31    [004]   100.438516: softirq_raise:        vec=9 [action=RCU]
ksoftirqd/4-31    [004]   100.438520: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921) => __raise_softirq_irqoff (ffffffffa36daf50) => raise_softirq (ffffffffa36db0fe)
=> trigger_load_balance (ffffffffa371cb9c) => update_process_times (ffffffffa3768fc7) => tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0) => hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8) => apic_timer_interrupt (ffffffffa4001b7f) => filter_pred_32 (ffffffffa37f3357)
=> filter_match_preds (ffffffffa37f3510)
=> trace_event_buffer_commit (ffffffffa37dc7eb) => trace_event_raw_event_softirq (ffffffffa36dab77) => __do_softirq (ffffffffa420025a)
=> run_ksoftirqd (ffffffffa36dadc6)
=> smpboot_thread_fn (ffffffffa36ffdb8)
=> kthread (ffffffffa36f9fb7)
=> ret_from_fork (ffffffffa4000215)

ksoftirqd/4-31    [004]   100.438520: softirq_raise:        vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438521: hrtimer_expire_exit:  hrtimer=0xffff92e9df91fd20
ksoftirqd/4-31    [004]   100.438523: hrtimer_start:        hrtimer=0xffff92e9df91fd20 function=tick_sched_timer/0x0 expires=92631512937 softexpires=92631512937
ksoftirqd/4-31    [004]   100.438525: local_timer_exit:     vector=236
ksoftirqd/4-31    [004]   100.438527: tick_stop:            success=0 dependency=SCHED

---

At this point (I think) ksoftirqd is already going to sleep, so only sysjitter will be eventually executing on CPU4, but tick_stop check still sees nr_running == 2.

---

ksoftirqd/4-31    [004]   100.438530: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/4-31    [004]   100.438531: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438533: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438534: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438535: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438536: softirq_entry:        vec=9 [action=RCU]
ksoftirqd/4-31    [004]   100.438537: softirq_exit:         vec=9 [action=RCU]
ksoftirqd/4-31    [004]   100.438540: sched_switch:         ksoftirqd/4:31 [120] S ==> sysjitter:2377 [120]

---

And tick again fires below.

---

  sysjitter-2377  [004]   100.439509: local_timer_entry:    vector=236
  sysjitter-2377  [004]   100.439511: hrtimer_cancel:       hrtimer=0xffff92e9df91fd20
  sysjitter-2377  [004]   100.439512: hrtimer_expire_entry: hrtimer=0xffff92e9df91fd20 now=92631515408 function=tick_sched_timer/0x0
  sysjitter-2377  [004]   100.439514: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921)
=> __raise_softirq_irqoff (ffffffffa36daf50)
=> raise_softirq (ffffffffa36db0fe)
=> update_process_times (ffffffffa3768f9d)
=> tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0)
=> hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8)
=> apic_timer_interrupt (ffffffffa4001b7f)

sysjitter-2377  [004]   100.439515: softirq_raise:        vec=1 [action=TIMER]
sysjitter-2377  [004]   100.439517: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921)
=> __raise_softirq_irqoff (ffffffffa36daf50)
=> raise_softirq (ffffffffa36db0fe)
=> trigger_load_balance (ffffffffa371cb9c)
=> update_process_times (ffffffffa3768fc7)
=> tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0)
=> hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8)
=> apic_timer_interrupt (ffffffffa4001b7f)

  sysjitter-2377  [004]   100.439518: softirq_raise:        vec=7 [action=SCHED]
  sysjitter-2377  [004]   100.439520: hrtimer_expire_exit:  hrtimer=0xffff92e9df91fd20
  sysjitter-2377  [004]   100.439521: hrtimer_start:        hrtimer=0xffff92e9df91fd20 function=tick_sched_timer/0x0 expires=92632512937 softexpires=92632512937
  sysjitter-2377  [004]   100.439522: local_timer_exit:     vector=236
  sysjitter-2377  [004]   100.439525: sched_wakeup:         ksoftirqd/4:31 [120] success=1 CPU:004
  sysjitter-2377  [004]   100.439527: tick_stop:            success=0 dependency=SCHED
  sysjitter-2377  [004]   100.439529: irq_work_entry:       vector=246
  sysjitter-2377  [004]   100.439531: irq_work_exit:        vector=246
  sysjitter-2377  [004]   100.439532: tick_stop:            success=0 dependency=SCHED
  sysjitter-2377  [004]   100.439534: sched_switch:         sysjitter:2377 [120] R ==> ksoftirqd/4:31 [120]
ksoftirqd/4-31    [004]   100.439536: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/4-31    [004]   100.439537: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/4-31    [004]   100.439538: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.439540: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.439541: sched_switch:         ksoftirqd/4:31 [120] S ==> sysjitter:2377 [120]
Quoted 8 lines

---

History seems to repeat itself until ~2s later (this is one of the pathological runs) when an ipi fires on CPU4 and SCHED dependecy is seen as cleared (as only sysjitter thread was is running at that point).

Does this make sense and help in any way? Suggestions for debugging this further? :-)

Gentle ping about this issue.

Thanks,

Juri

#8 lore raw
In reply to Juri Lelli
Quoted 8 lines · Juri Lelli
ksoftirqd/3-26    [003]    99.942485: timer_expire_entry:   timer=0xffffffffa55a9d20 function=clocksource_watchdog now=4294759328 baseclk=4294759328
ksoftirqd/3-26    [003]    99.942489: timer_start:          timer=0xffffffffa55a9d20 function=clocksource_watchdog expires=4294759822 [timeout=494] cpu=4 idx=114 flags=D|P|I
ksoftirqd/3-26    [003]    99.942491: timer_expire_exit:    timer=0xffffffffa55a9d20
  sysjitter-2377  [004]    99.942491: call_function_single_entry: vector=251
  sysjitter-2377  [004]    99.942492: call_function_single_exit: vector=251
ksoftirqd/3-26    [003]    99.942493: softirq_exit:         vec=1 [action=TIMER]
ksoftirqd/3-26    [003]    99.942494: softirq_entry:        vec=7 [action=SCHED]
  sysjitter-2377  [004]    99.942494: hrtimer_start:        hrtimer=0xffff92e9df91fd20 function=tick_sched_timer/0x0 expires=92630457385 softexpires=92630457385
Quoted 5 lines

---

So, my understanding is that clock gets reactivated on CPU 4 to run clocksource watchdog (indeed if I use tsc=nowatchdog this above doesn't happen, but the potential issue is still there I guess).

CPU 4 is being disturbed by an IPI. It's possibly caused by RCU, there
is a pending fix that I need to refresh and repost but you may
want to give it a try: https://lore.kernel.org/lkml/[email protected]/

Quoted 32 lines

---

  sysjitter-2377  [004]   100.438455: local_timer_entry:    vector=236
  sysjitter-2377  [004]   100.438456: hrtimer_cancel:       hrtimer=0xffff92e9df91fd20
  sysjitter-2377  [004]   100.438458: hrtimer_expire_entry: hrtimer=0xffff92e9df91fd20 now=92630461337 function=tick_sched_timer/0x0
  sysjitter-2377  [004]   100.438460: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921)
=> __raise_softirq_irqoff (ffffffffa36daf50)
=> raise_softirq (ffffffffa36db0fe)
=> update_process_times (ffffffffa3768f9d)
=> tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0)
=> hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8)
=> apic_timer_interrupt (ffffffffa4001b7f)

  sysjitter-2377  [004]   100.438461: softirq_raise:        vec=1 [action=TIMER]
  sysjitter-2377  [004]   100.438464: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921)
=> __raise_softirq_irqoff (ffffffffa36daf50)
=> raise_softirq (ffffffffa36db0fe)
=> trigger_load_balance (ffffffffa371cb9c)
=> update_process_times (ffffffffa3768fc7)
=> tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0)
=> hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8)
=> apic_timer_interrupt (ffffffffa4001b7f)
sysjitter-2377 [004] 100.438465: softirq_raise: vec=7 [action=SCHED]
---

Quoted 5 lines

Tick fires, the two softirq_raise (which you wondered about) seems to come from

1. update_process_times::run_local_timers (TIMER_SOFTIRQ)
2. update_process_times::scheduler_tick::trigger_load_balance (SCHED_SOFTIRQ)

Well, the places raising the softirqs are always the same. Now how the timer internals can decide to raise the TIMER_SOFTIRQ when there is no callback expired is more mysterious.

Quoted 17 lines

---

  sysjitter-2377  [004]   100.438468: hrtimer_expire_exit:  hrtimer=0xffff92e9df91fd20
  sysjitter-2377  [004]   100.438470: local_timer_exit:     vector=236
  sysjitter-2377  [004]   100.438473: sched_wakeup:         ksoftirqd/4:31 [120] success=1 CPU:004
  sysjitter-2377  [004]   100.438474: tick_stop:            success=0 dependency=SCHED
  sysjitter-2377  [004]   100.438476: hrtimer_start:        hrtimer=0xffff92e9df91fd20 function=tick_sched_timer/0x0 expires=92630512937 softexpires=92630512937
  sysjitter-2377  [004]   100.438484: irq_work_entry:       vector=246
  sysjitter-2377  [004]   100.438491: irq_work_exit:        vector=246
  sysjitter-2377  [004]   100.438492: tick_stop:            success=0 dependency=SCHED
  sysjitter-2377  [004]   100.438495: sched_switch:         sysjitter:2377 [120] R ==> ksoftirqd/4:31 [120]
ksoftirqd/4-31    [004]   100.438497: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/4-31    [004]   100.438499: timer_cancel:         timer=0xffffffffa55a9d20
ksoftirqd/4-31    [004]   100.438501: timer_expire_entry:   timer=0xffffffffa55a9d20 function=clocksource_watchdog now=4294759824 baseclk=4294759824
ksoftirqd/4-31    [004]   100.438504: timer_start:          timer=0xffffffffa55a9d20 function=clocksource_watchdog expires=4294760322 [timeout=498] cpu=5 idx=113 flags=D|P|I

---

Quoted 8 lines

Queueing clocksource watchdog on CPU5.

---

ksoftirqd/4-31    [004]   100.438506: timer_expire_exit:    timer=0xffffffffa55a9d20
ksoftirqd/4-31    [004]   100.438507: local_timer_entry:    vector=236

---

Back to back timer?

It looks like the tick has been reprogrammed very early indeed.

Quoted 52 lines

---

ksoftirqd/4-31    [004]   100.438509: hrtimer_cancel:       hrtimer=0xffff92e9df91fd20
ksoftirqd/4-31    [004]   100.438511: hrtimer_expire_entry: hrtimer=0xffff92e9df91fd20 now=92630515022 function=tick_sched_timer/0x0
ksoftirqd/4-31    [004]   100.438515: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921) => __raise_softirq_irqoff (ffffffffa36daf50) => raise_softirq (ffffffffa36db0fe)
=> rcu_sched_clock_irq (ffffffffa375af4a) => update_process_times (ffffffffa3768fa4) => tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0) => hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8) => apic_timer_interrupt (ffffffffa4001b7f) => filter_pred_32 (ffffffffa37f3357)
=> filter_match_preds (ffffffffa37f3510)
=> trace_event_buffer_commit (ffffffffa37dc7eb) => trace_event_raw_event_softirq (ffffffffa36dab77) => __do_softirq (ffffffffa420025a)
=> run_ksoftirqd (ffffffffa36dadc6)
=> smpboot_thread_fn (ffffffffa36ffdb8)
=> kthread (ffffffffa36f9fb7)
=> ret_from_fork (ffffffffa4000215)

ksoftirqd/4-31    [004]   100.438516: softirq_raise:        vec=9 [action=RCU]
ksoftirqd/4-31    [004]   100.438520: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921) => __raise_softirq_irqoff (ffffffffa36daf50) => raise_softirq (ffffffffa36db0fe)
=> trigger_load_balance (ffffffffa371cb9c) => update_process_times (ffffffffa3768fc7) => tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0) => hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8) => apic_timer_interrupt (ffffffffa4001b7f) => filter_pred_32 (ffffffffa37f3357)
=> filter_match_preds (ffffffffa37f3510)
=> trace_event_buffer_commit (ffffffffa37dc7eb) => trace_event_raw_event_softirq (ffffffffa36dab77) => __do_softirq (ffffffffa420025a)
=> run_ksoftirqd (ffffffffa36dadc6)
=> smpboot_thread_fn (ffffffffa36ffdb8)
=> kthread (ffffffffa36f9fb7)
=> ret_from_fork (ffffffffa4000215)

ksoftirqd/4-31    [004]   100.438520: softirq_raise:        vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438521: hrtimer_expire_exit:  hrtimer=0xffff92e9df91fd20
ksoftirqd/4-31    [004]   100.438523: hrtimer_start:        hrtimer=0xffff92e9df91fd20 function=tick_sched_timer/0x0 expires=92631512937 softexpires=92631512937
ksoftirqd/4-31    [004]   100.438525: local_timer_exit:     vector=236
ksoftirqd/4-31    [004]   100.438527: tick_stop:            success=0 dependency=SCHED
Quoted 5 lines

---

At this point (I think) ksoftirqd is already going to sleep, so only sysjitter will be eventually executing on CPU4, but tick_stop check still sees nr_running == 2.

As long at it hasn't been dequeued from the runqueue, tick_stop sees nr_running == 2.

Quoted 3 lines

---
ksoftirqd/4-31 [004] 100.438530: softirq_exit: vec=1 [action=TIMER]

Was there a corresponding softirq_enter? With timers expiring inside?

Quoted 7 lines
ksoftirqd/4-31    [004]   100.438531: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438533: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438534: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438535: softirq_exit:         vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438536: softirq_entry:        vec=9 [action=RCU]
ksoftirqd/4-31    [004]   100.438537: softirq_exit:         vec=9 [action=RCU]
ksoftirqd/4-31    [004]   100.438540: sched_switch:         ksoftirqd/4:31 [120] S ==> sysjitter:2377 [120]
Quoted 8 lines

---

And tick again fires below.

---

  sysjitter-2377  [004]   100.439509: local_timer_entry:    vector=236
  sysjitter-2377  [004]   100.439511: hrtimer_cancel:       hrtimer=0xffff92e9df91fd20
  sysjitter-2377  [004]   100.439512: hrtimer_expire_entry: hrtimer=0xffff92e9df91fd20 now=92631515408 function=tick_sched_timer/0x0

Right that's expected behaviour.

Quoted 4 lines
  sysjitter-2377  [004]   100.439518: softirq_raise:        vec=7 [action=SCHED]
  sysjitter-2377  [004]   100.439520: hrtimer_expire_exit:  hrtimer=0xffff92e9df91fd20
  sysjitter-2377  [004]   100.439521: hrtimer_start:        hrtimer=0xffff92e9df91fd20 function=tick_sched_timer/0x0 expires=92632512937 softexpires=92632512937
  sysjitter-2377  [004]   100.439522: local_timer_exit:     vector=236
Quoted 8 lines
  sysjitter-2377  [004]   100.439525: sched_wakeup:         ksoftirqd/4:31 [120] success=1 CPU:004
  sysjitter-2377  [004]   100.439527: tick_stop:            success=0 dependency=SCHED
  sysjitter-2377  [004]   100.439529: irq_work_entry:       vector=246
  sysjitter-2377  [004]   100.439531: irq_work_exit:        vector=246
  sysjitter-2377  [004]   100.439532: tick_stop:            success=0 dependency=SCHED
  sysjitter-2377  [004]   100.439534: sched_switch:         sysjitter:2377 [120] R ==> ksoftirqd/4:31 [120]
ksoftirqd/4-31    [004]   100.439536: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/4-31    [004]   100.439537: softirq_exit:         vec=1 [action=TIMER]

The empty timer softirq again.

Quoted 2 lines
ksoftirqd/4-31    [004]   100.439538: softirq_entry:        vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.439540: softirq_exit:         vec=7 [action=SCHED]

I'm also wondering if it's expected to see the sched softirq repeating so often.

Quoted 9 lines

ksoftirqd/4-31 [004] 100.439541: sched_switch: ksoftirqd/4:31 [120] S ==> sysjitter:2377 [120]
---

History seems to repeat itself until ~2s later (this is one of the pathological runs) when an ipi fires on CPU4 and SCHED dependecy is seen as cleared (as only sysjitter thread was is running at that point).

Does this make sense and help in any way? Suggestions for debugging this further? :-)

Let's see if I can reproduce it first. If not I'll need to bother you further :)

Thanks!

author
#9 lore raw
Quoted 1 line · Frederic Weisbecker

On Thu, May 21, 2020 at 07:00:20PM +0200, Juri Lelli wrote:

[...]

Quoted 5 lines
Quoted 5 lines
  sysjitter-2377  [004]   100.438495: sched_switch:         sysjitter:2377 [120] R ==> ksoftirqd/4:31 [120]
ksoftirqd/4-31    [004]   100.438497: softirq_entry:        vec=1 [action=TIMER]
ksoftirqd/4-31    [004]   100.438499: timer_cancel:         timer=0xffffffffa55a9d20
ksoftirqd/4-31    [004]   100.438501: timer_expire_entry:   timer=0xffffffffa55a9d20 function=clocksource_watchdog now=4294759824 baseclk=4294759824
ksoftirqd/4-31    [004]   100.438504: timer_start:          timer=0xffffffffa55a9d20 function=clocksource_watchdog expires=4294760322 [timeout=498] cpu=5 idx=113 flags=D|P|I

[...]

Quoted 50 lines
Quoted 50 lines
ksoftirqd/4-31    [004]   100.438509: hrtimer_cancel:       hrtimer=0xffff92e9df91fd20
ksoftirqd/4-31    [004]   100.438511: hrtimer_expire_entry: hrtimer=0xffff92e9df91fd20 now=92630515022 function=tick_sched_timer/0x0
ksoftirqd/4-31    [004]   100.438515: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921) => __raise_softirq_irqoff (ffffffffa36daf50) => raise_softirq (ffffffffa36db0fe)
=> rcu_sched_clock_irq (ffffffffa375af4a) => update_process_times (ffffffffa3768fa4) => tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0) => hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8) => apic_timer_interrupt (ffffffffa4001b7f) => filter_pred_32 (ffffffffa37f3357)
=> filter_match_preds (ffffffffa37f3510)
=> trace_event_buffer_commit (ffffffffa37dc7eb) => trace_event_raw_event_softirq (ffffffffa36dab77) => __do_softirq (ffffffffa420025a)
=> run_ksoftirqd (ffffffffa36dadc6)
=> smpboot_thread_fn (ffffffffa36ffdb8)
=> kthread (ffffffffa36f9fb7)
=> ret_from_fork (ffffffffa4000215)

ksoftirqd/4-31    [004]   100.438516: softirq_raise:        vec=9 [action=RCU]
ksoftirqd/4-31    [004]   100.438520: kernel_stack:         <stack trace>

=> __ftrace_trace_stack (ffffffffa37da921) => __raise_softirq_irqoff (ffffffffa36daf50) => raise_softirq (ffffffffa36db0fe)
=> trigger_load_balance (ffffffffa371cb9c) => update_process_times (ffffffffa3768fc7) => tick_sched_handle (ffffffffa377aaa2)
=> tick_sched_timer (ffffffffa377ad53)
=> __hrtimer_run_queues (ffffffffa3769cf0) => hrtimer_interrupt (ffffffffa376a58a)
=> smp_apic_timer_interrupt (ffffffffa40028f8) => apic_timer_interrupt (ffffffffa4001b7f) => filter_pred_32 (ffffffffa37f3357)
=> filter_match_preds (ffffffffa37f3510)
=> trace_event_buffer_commit (ffffffffa37dc7eb) => trace_event_raw_event_softirq (ffffffffa36dab77) => __do_softirq (ffffffffa420025a)
=> run_ksoftirqd (ffffffffa36dadc6)
=> smpboot_thread_fn (ffffffffa36ffdb8)
=> kthread (ffffffffa36f9fb7)
=> ret_from_fork (ffffffffa4000215)

ksoftirqd/4-31    [004]   100.438520: softirq_raise:        vec=7 [action=SCHED]
ksoftirqd/4-31    [004]   100.438521: hrtimer_expire_exit:  hrtimer=0xffff92e9df91fd20
ksoftirqd/4-31    [004]   100.438523: hrtimer_start:        hrtimer=0xffff92e9df91fd20 function=tick_sched_timer/0x0 expires=92631512937 softexpires=92631512937
ksoftirqd/4-31    [004]   100.438525: local_timer_exit:     vector=236
ksoftirqd/4-31    [004]   100.438527: tick_stop:            success=0 dependency=SCHED

[...]

Quoted 3 lines
Quoted 1 line

ksoftirqd/4-31 [004] 100.438530: softirq_exit: vec=1 [action=TIMER]

Was there a corresponding softirq_enter? With timers expiring inside?

The one up above, at 100.438497.

[...]

Quoted 5 lines
Quoted 3 lines

Does this make sense and help in any way? Suggestions for debugging this further? :-)

Let's see if I can reproduce it first. If not I'll need to bother you further :)

Sure. Let me know if you find anything.

Best,

Juri

#10 lore raw
Quoted 1 line · Juri Lelli

Sure. Let me know if you find anything.

I managed to reproduce. With "threadirqs" and without "tsc=reliable". I see tons of spurious TIMER softirqs. Investigation begins! I'll let you know.

Thanks!

author
#11 lore raw

Hi,

Quoted 6 lines · Frederic Weisbecker
Quoted 1 line · Juri Lelli

Sure. Let me know if you find anything.

I managed to reproduce. With "threadirqs" and without "tsc=reliable". I see tons of spurious TIMER softirqs. Investigation begins! I'll let you know.

Gentle ping. Any new findings about this issue?

Thanks,

Juri

#12 lore raw
Quoted 11 lines · Juri Lelli

Hi,

Quoted 6 lines · Frederic Weisbecker
Quoted 1 line · Juri Lelli

Sure. Let me know if you find anything.

I managed to reproduce. With "threadirqs" and without "tsc=reliable". I see tons of spurious TIMER softirqs. Investigation begins! I'll let you know.

Gentle ping. Any new findings about this issue?

Yep, I'm working on a series to reduce timer softirqs. I'm almost done, should post it soon.

I believe you guessed right:

* Timer tick raises timer softirq
* Ksoftirqd wakes up, restarts the tick
* Timer tick raises timer softirq
* ...

Thanks.

author
#13 lore raw
Quoted 7 lines · Frederic Weisbecker
Quoted 6 lines · Juri Lelli

Hi,

Quoted 3 lines · Frederic Weisbecker
Quoted 1 line · Juri Lelli

Sure. Let me know if you find anything.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I managed to reproduce. With "threadirqs" and without "tsc=reliable". I see tons of spurious TIMER softirqs. Investigation begins! I'll let you know.

Gentle ping. Any new findings about this issue?

Yep, I'm working on a series to reduce timer softirqs. I'm almost done, should post it soon.

Great!

Looking forward to it.

Best,

Juri