Skip to contentSkip to conversation

timers: Recalculate next timer interrupt only when necessary

PATCH Superseded Frederic Weisbecker 2 participants 3 messages Jul 21, 2020 – Jul 23, 2020 High-resolution timers, timer wheel, clockevents

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Frederic Weisbecker

author maintainer
#0 lore raw

PATCH 1/1 timers: Recalculate next timer interrupt only when necessary

The nohz tick code recalculates the timer wheel's next expiry on each idle loop iteration.

On the other hand, the base next expiry is now always cached and updated upon timer enqueue and execution. Only timer dequeue may leave base->next_expiry out of date (but then its stale value won't ever go past the actual next expiry to be recalculated).

Since recalculating the next_expiry isn't a free operation, especially when we must climb up the last wheel level to find out that no timer has been enqueued at all, lets reuse the next expiry cache when it is known to be reliable, which it is most of the time.

Cc Thomas Gleixner <[email protected]>
Cc Anna-Maria Behnsen <[email protected]>
---
 kernel/time/timer.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)
kernel/time/timer.c +18 -3
index 77e21e98ec32..e8002f86c5bc 100644
@@ -203,6 +203,7 @@ struct timer_base {
 #endif
 	unsigned long		clk;
 	unsigned long		next_expiry;
+	bool			next_expiry_recalc;
 	unsigned int		cpu;
 	bool			is_idle;
 	DECLARE_BITMAP(pending_map, WHEEL_SIZE);
@@ -593,6 +594,7 @@ static void enqueue_timer(struct timer_base *base, struct timer_list *timer,
 		 * can reevaluate the wheel:
 		 */
 		base->next_expiry = bucket_expiry;
+		base->next_expiry_recalc = false;
 		trigger_dyntick_cpu(base, timer);
 	}
 }
@@ -836,8 +838,10 @@ static int detach_if_pending(struct timer_list *timer, struct timer_base *base,
 	if (!timer_pending(timer))
 		return 0;
 
-	if (hlist_is_singular_node(&timer->entry, base->vectors + idx))
+	if (hlist_is_singular_node(&timer->entry, base->vectors + idx)) {
 		__clear_bit(idx, base->pending_map);
+		base->next_expiry_recalc = true;
+	}
 
 	detach_timer(timer, clear_pending);
 	return 1;
@@ -1571,6 +1575,9 @@ static unsigned long __next_timer_interrupt(struct timer_base *base)
 		clk >>= LVL_CLK_SHIFT;
 		clk += adj;
 	}
+
+	base->next_expiry_recalc = false;
+
 	return next;
 }
 
@@ -1631,9 +1638,11 @@ u64 get_next_timer_interrupt(unsigned long basej, u64 basem)
 		return expires;
 
 	raw_spin_lock(&base->lock);
-	nextevt = __next_timer_interrupt(base);
+	if (base->next_expiry_recalc)
+		base->next_expiry = __next_timer_interrupt(base);
+	nextevt = base->next_expiry;
 	is_max_delta = (nextevt == base->clk + NEXT_TIMER_MAX_DELTA);
-	base->next_expiry = nextevt;
+
 	/*
 	 * We have a fresh next event. Check whether we can forward the
 	 * base. We can only do that when @basej is past base->clk
@@ -1725,6 +1734,12 @@ static inline void __run_timers(struct timer_base *base)
 	while (time_after_eq(jiffies, base->clk) &&
 	       time_after_eq(jiffies, base->next_expiry)) {
 		levels = collect_expired_timers(base, heads);
+		/*
+		 * The only possible reason for not finding any expired
+		 * timer at this clk is that all matching timers have been
+		 * dequeued.
+		 */
+		WARN_ON_ONCE(!levels && !base->next_expiry_recalc);
 		base->clk++;
 		base->next_expiry = __next_timer_interrupt(base);
 
Signature · 1 line
2.26.2
#1 lore raw
Quoted 4 lines · Frederic Weisbecker

Since recalculating the next_expiry isn't a free operation, especially when we must climb up the last wheel level to find out that no timer has

I'm climbing stairs or mountains :)

Quoted 2 lines

been enqueued at all, lets reuse the next expiry cache when it is known

lets? Come on, the changelog is about facts not what we might do.

kernel/time/timer.c struct timer_base { · line 208

Quoted 5 lines
kernel/time/timer.c +1 -0
 	unsigned long		clk;
 	unsigned long		next_expiry;
+	bool			next_expiry_recalc;
 	unsigned int		cpu;
 	bool			is_idle;

Care to stare at the output of

pahole -C timer_base kernel/time/timer.o

before and after?

Thanks,

tglx

author maintainer
#2 lore raw
Quoted 7 lines · Thomas Gleixner
Quoted 4 lines · Frederic Weisbecker

Since recalculating the next_expiry isn't a free operation, especially when we must climb up the last wheel level to find out that no timer has

I'm climbing stairs or mountains :)

Arguably, climbing a wheel can be a never ending story :)

Quoted 5 lines
Quoted 3 lines

been enqueued at all, lets reuse the next expiry cache when it is known

lets? Come on, the changelog is about facts not what we might do.

Also since you applied the last patchset I have tried to quit using "we" in the changelog and use "the code" as a subject or even the passive form. But reading above, I did it again.

Too many habits :))

Quoted 9 lines

kernel/time/timer.c struct timer_base { · line 208

Quoted 6 lines
kernel/time/timer.c +1 -0
 	unsigned long		clk;
 	unsigned long		next_expiry;
+	bool			next_expiry_recalc;
 	unsigned int		cpu;
 	bool			is_idle;

Care to stare at the output of

Quoted 3 lines

pahole -C timer_base kernel/time/timer.o

before and after?

Ah right, I'll move the bool together.

Thanks.