locking/rwsem: Add sched annotation to down_read_common()

RFC PATCH Superseded John Stultz 3 participants 10 messages Apr 12, 2023 – Apr 18, 2023

Subsystems (2)

Stable branch Locking primitives

lore

Share & tools

lore

John Stultz

author

2023-04-12 02:38 UTC #0 lore raw

RFC PATCH 1/1 locking/rwsem: Add __sched annotation to __down_read_common()

Apparently despite it being marked inline, the compiler may not inline __down_read_common() which makes it difficult to identify the cause of lock contention, as the blocked function will always be listed as __down_read_common().

So this patch adds __sched annotation to the function so the calling function will instead be listed.

Cc Minchan Kim <[email protected]>

Cc Tim Murray <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc Will Deacon <[email protected]>

Cc Waiman Long <[email protected]>

Cc Boqun Feng <[email protected]>

Cc [email protected]

Fixes c995e638ccbb ("locking/rwsem: Fold __down_{read,write}*()")

Reported-by Tim Murray <[email protected]>

Signed-off-by John Stultz <[email protected]>

---
 kernel/locking/rwsem.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/locking/rwsem.c +1 -1

index acb5a50309a1..cde2f22e65a8 100644
@@ -1240,7 +1240,7 @@ static struct rw_semaphore *rwsem_downgrade_wake(struct rw_semaphore *sem)
 /*
  * lock for reading
  */
-static inline int __down_read_common(struct rw_semaphore *sem, int state)
+static inline __sched int __down_read_common(struct rw_semaphore *sem, int state)
 {
 	int ret = 0;
 	long count;

Signature · 1 line

2.40.0.577.gac1e443424-goog

Waiman Long

reviewer

2023-04-12 02:59 UTC #1 lore raw

Quoted 29 lines · John Stultz

Apparently despite it being marked inline, the compiler may not inline __down_read_common() which makes it difficult to identify the cause of lock contention, as the blocked function will always be listed as __down_read_common().
So this patch adds __sched annotation to the function so the calling function will instead be listed.
Cc Minchan Kim <[email protected]>
Cc Tim Murray <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Will Deacon <[email protected]>
Cc Waiman Long <[email protected]>
Cc Boqun Feng <[email protected]>
Cc [email protected]
Cc [email protected]
Fixes c995e638ccbb ("locking/rwsem: Fold __down_{read,write}*()")
Reported-by Tim Murray <[email protected]>
Signed-off-by John Stultz <[email protected]>
---
  kernel/locking/rwsem.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
kernel/locking/rwsem.c +0 -0
index acb5a50309a1..cde2f22e65a8 100644

kernel/locking/rwsem.c static struct rw_semaphore *rwsem_downgrade_wake(struct rw_semaphore *sem) · line 1246

Quoted 8 lines

kernel/locking/rwsem.c +1 -1

@@ -1240,7 +1240,7 @@ static struct rw_semaphore *rwsem_downgrade_wake(struct rw_semaphore *sem)
  /*
   * lock for reading
   */
-static inline int __down_read_common(struct rw_semaphore *sem, int state)
+static inline __sched int __down_read_common(struct rw_semaphore *sem, int state)
  {
  	int ret = 0;
  	long count;

Change inline to __always_inline instead of adding __sched. __down_read_common() is not supposed to be a standalone function.

Cheers,
Longman

John Stultz

author

2023-04-12 03:03 UTC #2 lore raw

Quoted 10 lines · Waiman Long

Quoted 8 lines · John Stultz
Apparently despite it being marked inline, the compiler may not inline __down_read_common() which makes it difficult to identify the cause of lock contention, as the blocked function will always be listed as __down_read_common().
So this patch adds __sched annotation to the function so the calling function will instead be listed.

...

Quoted 8 lines

kernel/locking/rwsem.c static struct rw_semaphore *rwsem_downgrade_wake(struct rw_semaphore *sem) · line 1246
Quoted 5 lines
kernel/locking/rwsem.c +1 -1
-static inline int __down_read_common(struct rw_semaphore *sem, int state)
+static inline __sched int __down_read_common(struct rw_semaphore *sem, int state)
  {
      int ret = 0;
      long count;
Change inline to __always_inline instead of adding __sched. __down_read_common() is not supposed to be a standalone function.

Sounds good. I'll give that a go and will re-submit! Thanks for the review!
-john

John Stultz

author

2023-04-12 03:59 UTC #3 lore raw

PATCH v2 1/1 locking/rwsem: Add __always_inline annotation to __down_read_common()

In reply to John Stultz

So this patch adds __always_inline annotation to the function to force it to be inlines so the calling function will be listed.

Cc Minchan Kim <[email protected]>

Cc Tim Murray <[email protected]>

Cc Peter Zijlstra <[email protected]>

Cc Ingo Molnar <[email protected]>

Cc Will Deacon <[email protected]>

Cc Waiman Long <[email protected]>

Cc Boqun Feng <[email protected]>

Cc [email protected]

Fixes c995e638ccbb ("locking/rwsem: Fold __down_{read,write}*()")

Reported-by Tim Murray <[email protected]>

Signed-off-by John Stultz <[email protected]>

---
v2: Reworked to use __always_inline instead of __sched as
suggested by Waiman Long

---
 kernel/locking/rwsem.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

kernel/locking/rwsem.c +1 -1

index acb5a50309a1..e99eef8ea552 100644
@@ -1240,7 +1240,7 @@ static struct rw_semaphore *rwsem_downgrade_wake(struct rw_semaphore *sem)
 /*
  * lock for reading
  */
-static inline int __down_read_common(struct rw_semaphore *sem, int state)
+static __always_inline int __down_read_common(struct rw_semaphore *sem, int state)
 {
 	int ret = 0;
 	long count;

Signature · 1 line

2.40.0.577.gac1e443424-goog

Waiman Long

reviewer Reviewed-by on v2 1/1

2023-04-12 12:11 UTC #4 lore raw

Re: [PATCH v2] locking/rwsem: Add __always_inline annotation to __down_read_common()

Quoted 33 lines · John Stultz

Apparently despite it being marked inline, the compiler may not inline __down_read_common() which makes it difficult to identify the cause of lock contention, as the blocked function will always be listed as __down_read_common().
So this patch adds __always_inline annotation to the function to force it to be inlines so the calling function will be listed.
Cc Minchan Kim <[email protected]>
Cc Tim Murray <[email protected]>
Cc Peter Zijlstra <[email protected]>
Cc Ingo Molnar <[email protected]>
Cc Will Deacon <[email protected]>
Cc Waiman Long <[email protected]>
Cc Boqun Feng <[email protected]>
Cc [email protected]
Cc [email protected]
Fixes c995e638ccbb ("locking/rwsem: Fold __down_{read,write}*()")
Reported-by Tim Murray <[email protected]>
Signed-off-by John Stultz <[email protected]>
---
v2: Reworked to use __always_inline instead of __sched as
suggested by Waiman Long
---
  kernel/locking/rwsem.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
kernel/locking/rwsem.c +0 -0
index acb5a50309a1..e99eef8ea552 100644

kernel/locking/rwsem.c static struct rw_semaphore *rwsem_downgrade_wake(struct rw_semaphore *sem) · line 1246

Quoted 8 lines

kernel/locking/rwsem.c +1 -1

@@ -1240,7 +1240,7 @@ static struct rw_semaphore *rwsem_downgrade_wake(struct rw_semaphore *sem)
  /*
   * lock for reading
   */
-static inline int __down_read_common(struct rw_semaphore *sem, int state)
+static __always_inline int __down_read_common(struct rw_semaphore *sem, int state)
  {
  	int ret = 0;
  	long count;

Reviewed-by Waiman Long <[email protected]>

Peter Zijlstra

maintainer

2023-04-17 11:22 UTC #5 lore raw

Re: [PATCH v2] locking/rwsem: Add __always_inline annotation to __down_read_common()

In reply to John Stultz

Quoted 8 lines · John Stultz

Apparently despite it being marked inline, the compiler may not inline __down_read_common() which makes it difficult to identify the cause of lock contention, as the blocked function will always be listed as __down_read_common().
So this patch adds __always_inline annotation to the function to force it to be inlines so the calling function will be listed.

I'm a wee bit confused; what are you looking at? Wchan? What is stopping the compiler from now handing you __down_read{,_interruptible,_killable}() instead? Is that fine?

Waiman Long

reviewer

2023-04-17 14:10 UTC #6 lore raw

Re: [PATCH v2] locking/rwsem: Add __always_inline annotation to __down_read_common()

Quoted 6 lines · Peter Zijlstra

Quoted 5 lines · John Stultz
Apparently despite it being marked inline, the compiler may not inline __down_read_common() which makes it difficult to identify the cause of lock contention, as the blocked function will always be listed as __down_read_common().

Quoted 7 lines

Quoted 3 lines
So this patch adds __always_inline annotation to the function to force it to be inlines so the calling function will be listed.
I'm a wee bit confused; what are you looking at? Wchan? What is stopping the compiler from now handing you __down_read{,_interruptible,_killable}() instead? Is that fine?

My theory is that the compiler may refuse to inline __down_read_common() because it is called 3 times in order to reduce overall code size. The other __down_read*() functions you listed are only called once.

My 2 cents.

Cheers,
Longman

John Stultz

author

2023-04-17 16:22 UTC #7 lore raw

Re: [PATCH v2] locking/rwsem: Add __always_inline annotation to __down_read_common()

In reply to Peter Zijlstra

Quoted 7 lines · Peter Zijlstra

Quoted 5 lines · John Stultz
Apparently despite it being marked inline, the compiler may not inline __down_read_common() which makes it difficult to identify the cause of lock contention, as the blocked function will always be listed as __down_read_common().

Quoted 5 lines

Quoted 3 lines
So this patch adds __always_inline annotation to the function to force it to be inlines so the calling function will be listed.
I'm a wee bit confused; what are you looking at? Wchan?

Apologies! Yes, traceevent data via wchan, sorry I didn't make that clear.

Quoted 3 lines

What is stopping
the compiler from now handing you __down_read{,_interruptible,_killable}() instead? Is that fine?

No, we want to make the blocked calling function, rather than the locking functions, visible in the tracepoints captured. That said, the other __down_read* functions seem to be properly inlined in practice (Waiman's theory as to why sounds convincing to me).

If you'd like I can add those as well to be always_inline, as well so it's more consistent?

thanks
-john

Peter Zijlstra

maintainer

2023-04-18 10:30 UTC #8 lore raw

Re: [PATCH v2] locking/rwsem: Add __always_inline annotation to __down_read_common()

Quoted 8 lines · John Stultz

Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · John Stultz
Apparently despite it being marked inline, the compiler may not inline __down_read_common() which makes it difficult to identify the cause of lock contention, as the blocked function will always be listed as __down_read_common().

Quoted 7 lines

Quoted 5 lines
Quoted 3 lines
So this patch adds __always_inline annotation to the function to force it to be inlines so the calling function will be listed.
I'm a wee bit confused; what are you looking at? Wchan?
Apologies! Yes, traceevent data via wchan, sorry I didn't make that clear.

No worries; good addition to the v3 Changelog ;-)

Quoted 8 lines

Quoted 3 lines
What is stopping
the compiler from now handing you __down_read{,_interruptible,_killable}() instead? Is that fine?
No, we want to make the blocked calling function, rather than the locking functions, visible in the tracepoints captured. That said, the other __down_read* functions seem to be properly inlined in practice (Waiman's theory as to why sounds convincing to me).

Right, but we should not rely on the compiler heuristics for correctness :-)

Quoted 2 lines

If you'd like I can add those as well to be always_inline, as well so it's more consistent?

Yes please. I'm not sure I care much about the whole 'inline __sched' vs '__always_inline' thing, but I do feel it should all be consistently applied.

John Stultz

author

2023-04-18 12:07 UTC #9 lore raw

Re: [PATCH v2] locking/rwsem: Add __always_inline annotation to __down_read_common()

Quoted 32 lines · Peter Zijlstra

Quoted 8 lines · John Stultz
Quoted 7 lines · Peter Zijlstra
Quoted 5 lines · John Stultz
Apparently despite it being marked inline, the compiler may not inline __down_read_common() which makes it difficult to identify the cause of lock contention, as the blocked function will always be listed as __down_read_common().
Quoted 7 lines
Quoted 5 lines
Quoted 3 lines
So this patch adds __always_inline annotation to the function to force it to be inlines so the calling function will be listed.
I'm a wee bit confused; what are you looking at? Wchan?
Apologies! Yes, traceevent data via wchan, sorry I didn't make that clear.
No worries; good addition to the v3 Changelog ;-)
Quoted 8 lines
Quoted 3 lines
What is stopping
the compiler from now handing you __down_read{,_interruptible,_killable}() instead? Is that fine?
No, we want to make the blocked calling function, rather than the locking functions, visible in the tracepoints captured. That said, the other __down_read* functions seem to be properly inlined in practice (Waiman's theory as to why sounds convincing to me).
Right, but we should not rely on the compiler heuristics for correctness :-)

Quoted 6 lines

Quoted 2 lines
If you'd like I can add those as well to be always_inline, as well so it's more consistent?
Yes please. I'm not sure I care much about the whole 'inline __sched' vs '__always_inline' thing, but I do feel it should all be consistently applied.

Sounds good. I'll respin with this.

Thanks so much for the review!
-john