Skip to contentSkip to conversation

Inline double_unlock_balance()

PATCH In review Sripathi Kodi 4 participants 5 messages Nov 5, 2008 – Nov 6, 2008

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 Inline double_unlock_balance()

Hi,

We have a test case which measures the variation in the amount of time needed to perform a fixed amount of work on the preempt_rt kernel. We started seeing deterioration in it's performance recently. The test should never take more than 10 microseconds, but we started 5-10% failure rate. Using elimination method, we traced the problem to commit 1b12bbc747560ea68bcc132c3d05699e52271da0 (lockdep: re-annotate scheduler runqueues). When LOCKDEP is disabled, this patch only adds an additional function call to double_unlock_balance(). Hence I inlined double_unlock_balance() and the problem went away. Here is a patch to make this change.

Thanks,
Sripathi.

lockdep: Inline double_unlock_balance()

Additional function call for double_unlock_balance() causes latency problems for some test cases on the preempt_rt kernel.

kernel/sched.c +1 -1
Index: linux-2.6.27.4/kernel/sched.c
===================================================================
@@ -2812,7 +2812,7 @@
 	return ret;
 }
 
-static void double_unlock_balance(struct rq *this_rq, struct rq *busiest)
+static inline void double_unlock_balance(struct rq *this_rq, struct rq *busiest)
 	__releases(busiest->lock)
 {
 	spin_unlock(&busiest->lock);
#1 lore raw
Quoted 16 lines · Sripathi Kodi

Hi,

We have a test case which measures the variation in the amount of time needed to perform a fixed amount of work on the preempt_rt kernel. We started seeing deterioration in it's performance recently. The test should never take more than 10 microseconds, but we started 5-10% failure rate. Using elimination method, we traced the problem to commit 1b12bbc747560ea68bcc132c3d05699e52271da0 (lockdep: re-annotate scheduler runqueues). When LOCKDEP is disabled, this patch only adds an additional function call to double_unlock_balance(). Hence I inlined double_unlock_balance() and the problem went away. Here is a patch to make this change.

Thanks,
Sripathi.

Quoted 6 lines

lockdep: Inline double_unlock_balance()

Additional function call for double_unlock_balance() causes latency problems for some test cases on the preempt_rt kernel.

Acked-by; Peter Zijlstra <[email protected]>

Quoted 13 lines

Index: linux-2.6.27.4/kernel/sched.c
===================================================================

kernel/sched.c +1 -1
@@ -2812,7 +2812,7 @@
 	return ret;
 }
 
-static void double_unlock_balance(struct rq *this_rq, struct rq *busiest)
+static inline void double_unlock_balance(struct rq *this_rq, struct rq *busiest)
 	__releases(busiest->lock)
 {
 	spin_unlock(&busiest->lock);
#2 lore raw
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Sripathi Kodi

Hi,

We have a test case which measures the variation in the amount of time needed to perform a fixed amount of work on the preempt_rt kernel. We started seeing deterioration in it's performance recently. The test should never take more than 10 microseconds, but we started 5-10% failure rate. Using elimination method, we traced the problem to commit 1b12bbc747560ea68bcc132c3d05699e52271da0 (lockdep: re-annotate scheduler runqueues). When LOCKDEP is disabled, this patch only adds an additional function call to double_unlock_balance(). Hence I inlined double_unlock_balance() and the problem went away. Here is a patch to make this change.

Thanks,
Sripathi.

Quoted 8 lines
Quoted 6 lines

lockdep: Inline double_unlock_balance()

Additional function call for double_unlock_balance() causes latency problems for some test cases on the preempt_rt kernel.

Acked-by; Peter Zijlstra <[email protected]>

hm, i'm not sure why it makes such a difference. Possibly cache alignment or code generation details pushing the critical path just beyond the L1 cache limit and causing thrashing?

Anyway, i've applied it to tip/sched/rt, as we generally want to inline such short locking ops.

Ingo

#3 lore raw
Quoted 26 lines · Ingo Molnar
Quoted 24 lines · Peter Zijlstra
Quoted 22 lines · Sripathi Kodi

Hi,

We have a test case which measures the variation in the amount of time needed to perform a fixed amount of work on the preempt_rt kernel. We started seeing deterioration in it's performance recently. The test should never take more than 10 microseconds, but we started 5-10% failure rate. Using elimination method, we traced the problem to commit 1b12bbc747560ea68bcc132c3d05699e52271da0 (lockdep: re-annotate scheduler runqueues). When LOCKDEP is disabled, this patch only adds an additional function call to double_unlock_balance(). Hence I inlined double_unlock_balance() and the problem went away. Here is a patch to make this change.

Thanks,
Sripathi.

lockdep: Inline double_unlock_balance()

Additional function call for double_unlock_balance() causes latency problems for some test cases on the preempt_rt kernel.

Quoted 8 lines
Quoted 1 line

Acked-by; Peter Zijlstra <[email protected]>

hm, i'm not sure why it makes such a difference. Possibly cache alignment or code generation details pushing the critical path just beyond the L1 cache limit and causing thrashing?

Anyway, i've applied it to tip/sched/rt, as we generally want to inline such short locking ops.

I'm thinking sripathi's gcc had a massive brainfart and did something funny, maybe the extra register pressure from the calling convention messed it up.

He failed to quantify the exact benefit, ie scheduling cost/latency before and after and what platform. But still the patch is simple enough.

#4 lore raw
In reply to Sripathi Kodi

Needs this to fix a warning as well

sched: fix double_unlock_balance compile warning

kernel/sched_rt.c:913: warning: ‘double_unlock_balance’ declared inline after being called kernel/sched_rt.c:913: warning: previous declaration of ‘double_unlock_balance’ was here

Correct the declaration.

---
 kernel/sched_rt.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
kernel/sched_rt.c +2 -1
Index: linux-2.6-tip/kernel/sched_rt.c
===================================================================
@@ -910,7 +910,8 @@ static void put_prev_task_rt(struct rq *
 #define RT_MAX_TRIES 3
 
 static int double_lock_balance(struct rq *this_rq, struct rq *busiest);
-static void double_unlock_balance(struct rq *this_rq, struct rq *busiest);
+static inline void double_unlock_balance(struct rq *this_rq,
+						struct rq *busiest);
 
 static void deactivate_task(struct rq *rq, struct task_struct *p, int sleep);
 
Signature · 2 lines
regards,
Dhaval