Call get_time() only when necessary in run_hrtimer_queue
PATCH 1/1 Call get_time() only when necessary in run_hrtimer_queue
It seems that run_hrtimer_queue() is calling get_time() much more often than it needs to.
With this patch, it only calls get_time() if there's a pending timer.
Following is from a profile done without the patch:
kernel ticks: 30841 1.02 %
13572 44.01 44.01 time_interpolator_get_offset
155 0.50 96.91 hrtimer_run_queuesAnd with the patch:
kernel ticks: 18334 0.58 %
74 0.40 97.81 hrtimer_run_queues
43 0.23 98.63 time_interpolator_get_offset
kernel/hrtimer.c +9 -3
@@ -586,12 +586,17 @@ int hrtimer_get_res(const clockid_t whic*/ static inline void run_hrtimer_queue (struct hrtimer_base *base) { -ktime_t now = base->get_time (); -struct rb_node *node; +ktime_t now; +struct rb_node *node = base->first; + +if (!node) +return ; + + now = base->get_time ();spin_lock_irq (&base->lock); -while ((node = base->first)) { +while (node) {struct hrtimer *timer;int (*fn)(void *);int restart;
@@ -620,6 +625,7 @@ static inline void run_hrtimer_queue(strspin_lock_irq (&base->lock); + node = base->first;/* Another CPU has added back the timer */ if (timer->state !=HRTIMER_RUNNING )continue ;
Quoted 8 lines · Dimitri Sivanich
It seems that
run_hrtimer_queue()is callingget_time()much more often than it needs to.With this patch, it only calls
get_time()if there's a pending timer.Following is from a profile done without the patch:
kernel ticks: 30841 1.02 %
Quoted 9 lines
13572 44.01 44.01 time_interpolator_get_offset 155 0.50 96.91 hrtimer_run_queuesAnd with the patch:
kernel ticks: 18334 0.58 %
74 0.40 97.81hrtimer_run_queues
43 0.23 98.63time_interpolator_get_offset
This code has been extensively redone in -mm and I am planning on sending all that to Linus within a week.
The hrtimer rework in -mm might fix this performance problem, although from a quick peek, perhaps not.
So could you please verify that the problem still needs fixing in 2.6.16-mm1 and if so, raise a patch against that?
Thanks.
Quoted 8 lines · Andrew Morton
This code has been extensively redone in -mm and I am planning on sending all that to Linus within a week.
The hrtimer rework in -mm might fix this performance problem, although from a quick peek, perhaps not.
So could you please verify that the problem still needs fixing in 2.6.16-mm1 and if so, raise a patch against that?
I have a look into it.
tglx
Quoted 9 lines · Andrew Morton
This code has been extensively redone in -mm and I am planning on sending all that to Linus within a week.
The hrtimer rework in -mm might fix this performance problem, although from a quick peek, perhaps not.
So could you please verify that the problem still needs fixing in 2.6.16-mm1 and if so, raise a patch against that?
The hrtimer work in -mm does improve on the situation, although there appears to be some occasional cache line contention for xtime. The following patch (which is similiar to my previously submitted patch) is applicable to 2.6.16-mm1 and does take care of at least a good portion of that.
kernel/hrtimer.c +6 -2
@@ -604,14 +604,17 @@ int hrtimer_get_res(const clockid_t whic*/ static inline void run_hrtimer_queue (struct hrtimer_base *base) { -struct rb_node *node; +struct rb_node *node = base->first; + +if (!node) +return ;if (base->get_softirq_time) base->softirq_time = base->get_softirq_time ();spin_lock_irq (&base->lock); -while ((node = base->first)) { +while (node) {struct hrtimer *timer;int (*fn)(struct hrtimer *);int restart;
@@ -633,6 +636,7 @@ static inline void run_hrtimer_queue(strBUG_ON (hrtimer_active (timer));enqueue_hrtimer (timer, base); } + node = base->first; }set_curr_timer (base,NULL );spin_unlock_irq (&base->lock);
Quoted 6 lines · Dimitri Sivanich
The hrtimer work in -mm does improve on the situation, although there appears to be some occasional cache line contention for xtime. The following patch (which is similiar to my previously submitted patch) is applicable to 2.6.16-mm1 and does take care of at least a good portion of that.
I am not sure your patch is correct.
Quoted 12 lines
Index:
linux/kernel/hrtimer.c
===================================================================kernel/hrtimer.c +2
-1@@ -604,14 +604,17 @@ int hrtimer_get_res(const clockid_t whic*/ static inline void run_hrtimer_queue (struct hrtimer_base *base) { -struct rb_node *node; +struct rb_node *node = base->first; +
kernel/hrtimer.c int hrtimer_get_res(const clockid_t whic · line 617
Quoted 10 lines
kernel/hrtimer.c +3
-1+if (!node) +return ;if (base->get_softirq_time) base->softirq_time = base->get_softirq_time ();spin_lock_irq (&base->lock); -while ((node = base->first)) { +while (node) {
Are you sure of this change ?
base->first may have changed just before you locked the base->lock (spin_lock_irq(&base->lock);)
Eric
Quoted 20 lines · Eric Dumazet
Quoted 20 lines
Index:
linux/kernel/hrtimer.c
===================================================================kernel/hrtimer.c +4
-1@@ -604,14 +604,17 @@ int hrtimer_get_res(const clockid_t whic*/ static inline void run_hrtimer_queue (struct hrtimer_base *base) { -struct rb_node *node; +struct rb_node *node = base->first; + +if (!node) +return ;if (base->get_softirq_time) base->softirq_time = base->get_softirq_time ();spin_lock_irq (&base->lock);
Quoted 8 lines
kernel/hrtimer.c int hrtimer_get_res(const clockid_t whic · line 617
Quoted 2 lines
kernel/hrtimer.c +1
-1-while ((node = base->first)) { +while (node) {Are you sure of this change ?
base->firstmay have changed just before you locked thebase->lock(spin_lock_irq(&base->lock);)
We could the following simpler change instead:
kernel/hrtimer.c +3 -0
@@ -606,6 +606,9 @@ static inline void run_hrtimer_queue(str {struct rb_node *node; +if (!base->first) +return ; +if (base->get_softirq_time) base->softirq_time = base->get_softirq_time ();
Quoted 9 lines · Dimitri Sivanich
We could the following simpler change instead:
Signed-off-by Dimitri Sivanich <[email protected]>Index:
linux/kernel/hrtimer.c
===================================================================
kernel/hrtimer.c static inline void run_hrtimer_queue(str · line 614
Quoted 9 lines
kernel/hrtimer.c +3
-0@@ -606,6 +606,9 @@ static inline void run_hrtimer_queue(str {struct rb_node *node; +if (!base->first) +return ; +if (base->get_softirq_time) base->softirq_time = base->get_softirq_time ();
Looks much better. Can you please resubmit with a proper patch description ?
Please add: Signed-off-by: Thomas Gleixner <[email protected]>
tglx
PATCH 1/1 Call get_softirq_time() only when necessary in run_hrtimer_queue()
It seems that run_hrtimer_queue() is calling get_softirq_time() more often than it needs to.
With this patch, it only calls get_softirq_time() if there's a pending timer.
kernel/hrtimer.c +3 -0
@@ -606,6 +606,9 @@ static inline void run_hrtimer_queue(str {struct rb_node *node; +if (!base->first) +return ; +if (base->get_softirq_time) base->softirq_time = base->get_softirq_time ();