Skip to contentSkip to conversation

Call get_time() only when necessary in run_hrtimer_queue

PATCH In review Dimitri Sivanich 4 participants 8 messages Mar 24, 2006 – Mar 28, 2006

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Call get_time() only when necessary in run_hrtimer_queue

It seems that run_hrtimer_queue() is calling get_time() much more often than it needs to.

With this patch, it only calls get_time() if there's a pending timer.

Following is from a profile done without the patch:
kernel ticks: 30841 1.02 %

      13572       44.01    44.01      time_interpolator_get_offset
        155        0.50    96.91      hrtimer_run_queues

And with the patch:
kernel ticks: 18334 0.58 %
74 0.40 97.81 hrtimer_run_queues
43 0.23 98.63 time_interpolator_get_offset

kernel/hrtimer.c +9 -3
Index: linux-2.6.15/kernel/hrtimer.c
===================================================================
@@ -586,12 +586,17 @@ int hrtimer_get_res(const clockid_t whic
  */
 static inline void run_hrtimer_queue(struct hrtimer_base *base)
 {
-	ktime_t now = base->get_time();
-	struct rb_node *node;
+	ktime_t now;
+	struct rb_node *node = base->first;
+
+	if (!node)
+		return;
+
+	now = base->get_time();
 
 	spin_lock_irq(&base->lock);
 
-	while ((node = base->first)) {
+	while (node) {
 		struct hrtimer *timer;
 		int (*fn)(void *);
 		int restart;
@@ -620,6 +625,7 @@ static inline void run_hrtimer_queue(str
 
 		spin_lock_irq(&base->lock);
 
+		node = base->first;
 		/* Another CPU has added back the timer */
 		if (timer->state != HRTIMER_RUNNING)
 			continue;
#1 lore raw
Quoted 8 lines · Dimitri Sivanich

It seems that run_hrtimer_queue() is calling get_time() much more often than it needs to.

With this patch, it only calls get_time() if there's a pending timer.

Following is from a profile done without the patch:
kernel ticks: 30841 1.02 %

Quoted 9 lines
      13572       44.01    44.01      time_interpolator_get_offset
        155        0.50    96.91      hrtimer_run_queues

And with the patch:
kernel ticks: 18334 0.58 %
74 0.40 97.81 hrtimer_run_queues
43 0.23 98.63 time_interpolator_get_offset

This code has been extensively redone in -mm and I am planning on sending all that to Linus within a week.

The hrtimer rework in -mm might fix this performance problem, although from a quick peek, perhaps not.

So could you please verify that the problem still needs fixing in 2.6.16-mm1 and if so, raise a patch against that?

Thanks.

#2 lore raw
Quoted 8 lines · Andrew Morton

This code has been extensively redone in -mm and I am planning on sending all that to Linus within a week.

The hrtimer rework in -mm might fix this performance problem, although from a quick peek, perhaps not.

So could you please verify that the problem still needs fixing in 2.6.16-mm1 and if so, raise a patch against that?

I have a look into it.

tglx

#3 lore raw
In reply to Andrew Morton
Quoted 9 lines · Andrew Morton

This code has been extensively redone in -mm and I am planning on sending all that to Linus within a week.

The hrtimer rework in -mm might fix this performance problem, although from a quick peek, perhaps not.

So could you please verify that the problem still needs fixing in 2.6.16-mm1 and if so, raise a patch against that?

The hrtimer work in -mm does improve on the situation, although there appears to be some occasional cache line contention for xtime. The following patch (which is similiar to my previously submitted patch) is applicable to 2.6.16-mm1 and does take care of at least a good portion of that.

kernel/hrtimer.c +6 -2
Index: linux/kernel/hrtimer.c
===================================================================
@@ -604,14 +604,17 @@ int hrtimer_get_res(const clockid_t whic
  */
 static inline void run_hrtimer_queue(struct hrtimer_base *base)
 {
-	struct rb_node *node;
+	struct rb_node *node = base->first;
+
+	if (!node)
+		return;
 
 	if (base->get_softirq_time)
 		base->softirq_time = base->get_softirq_time();
 
 	spin_lock_irq(&base->lock);
 
-	while ((node = base->first)) {
+	while (node) {
 		struct hrtimer *timer;
 		int (*fn)(struct hrtimer *);
 		int restart;
@@ -633,6 +636,7 @@ static inline void run_hrtimer_queue(str
 			BUG_ON(hrtimer_active(timer));
 			enqueue_hrtimer(timer, base);
 		}
+		node = base->first;
 	}
 	set_curr_timer(base, NULL);
 	spin_unlock_irq(&base->lock);
#4 lore raw
Quoted 6 lines · Dimitri Sivanich

The hrtimer work in -mm does improve on the situation, although there appears to be some occasional cache line contention for xtime. The following patch (which is similiar to my previously submitted patch) is applicable to 2.6.16-mm1 and does take care of at least a good portion of that.

I am not sure your patch is correct.

Quoted 12 lines

Index: linux/kernel/hrtimer.c
===================================================================

kernel/hrtimer.c +2 -1
@@ -604,14 +604,17 @@ int hrtimer_get_res(const clockid_t whic
  */
 static inline void run_hrtimer_queue(struct hrtimer_base *base)
 {
-	struct rb_node *node;
+	struct rb_node *node = base->first;
+

kernel/hrtimer.c int hrtimer_get_res(const clockid_t whic · line 617

Quoted 10 lines
kernel/hrtimer.c +3 -1
+	if (!node)
+		return;
 
 	if (base->get_softirq_time)
 		base->softirq_time = base->get_softirq_time();
 
 	spin_lock_irq(&base->lock);
 
-	while ((node = base->first)) {
+	while (node) {

Are you sure of this change ?

base->first may have changed just before you locked the base->lock (spin_lock_irq(&base->lock);)

Eric

#5 lore raw
Quoted 20 lines · Eric Dumazet
Quoted 20 lines

Index: linux/kernel/hrtimer.c
===================================================================

kernel/hrtimer.c +4 -1
@@ -604,14 +604,17 @@ int hrtimer_get_res(const clockid_t whic
 */
 static inline void run_hrtimer_queue(struct hrtimer_base *base)
 {
-	struct rb_node *node;
+	struct rb_node *node = base->first;
+
+	if (!node)
+		return;
 
 	if (base->get_softirq_time)
 		base->softirq_time = base->get_softirq_time();
 
 	spin_lock_irq(&base->lock);
 
Quoted 8 lines

kernel/hrtimer.c int hrtimer_get_res(const clockid_t whic · line 617

Quoted 2 lines
kernel/hrtimer.c +1 -1
-	while ((node = base->first)) {
+	while (node) {

Are you sure of this change ?

base->first may have changed just before you locked the base->lock (spin_lock_irq(&base->lock);)

We could the following simpler change instead:

kernel/hrtimer.c +3 -0
Index: linux/kernel/hrtimer.c
===================================================================
@@ -606,6 +606,9 @@ static inline void run_hrtimer_queue(str
 {
 	struct rb_node *node;
 
+	if (!base->first)
+		return;
+
 	if (base->get_softirq_time)
 		base->softirq_time = base->get_softirq_time();
 
#6 lore raw
Quoted 9 lines · Dimitri Sivanich

We could the following simpler change instead:

Index: linux/kernel/hrtimer.c
===================================================================

kernel/hrtimer.c static inline void run_hrtimer_queue(str · line 614

Quoted 9 lines
kernel/hrtimer.c +3 -0
@@ -606,6 +606,9 @@ static inline void run_hrtimer_queue(str
 {
 	struct rb_node *node;
 
+	if (!base->first)
+		return;
+
 	if (base->get_softirq_time)
 		base->softirq_time = base->get_softirq_time();
 

Looks much better. Can you please resubmit with a proper patch description ?

Please add: Signed-off-by: Thomas Gleixner <[email protected]>

tglx

#7 lore raw

PATCH 1/1 Call get_softirq_time() only when necessary in run_hrtimer_queue()

It seems that run_hrtimer_queue() is calling get_softirq_time() more often than it needs to.

With this patch, it only calls get_softirq_time() if there's a pending timer.

kernel/hrtimer.c +3 -0
Index: linux/kernel/hrtimer.c
===================================================================
@@ -606,6 +606,9 @@ static inline void run_hrtimer_queue(str
 {
 	struct rb_node *node;
 
+	if (!base->first)
+		return;
+
 	if (base->get_softirq_time)
 		base->softirq_time = base->get_softirq_time();