Skip to contentSkip to conversation

locking/mutex: Modifications to mutex

RFC PATCH Superseded Jason Low 5 participants 24 messages Jun 4, 2014 – Jun 12, 2014 Locking primitives

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Jason Low

author
#0 lore raw

cover locking/mutex: Modifications to mutex

This patchset contains modifications to mutex.

Patch 1 further reduces unnecessary atomic xchg() operations upon entering the mutex slowpath in mutex_lock().

Patch 2 corrects/updates the documentation on mutex optimistic spinning.

Patch 3 optimizes mutex trylock slowpath.

Jason Low (3):
mutex: Try to acquire mutex only if it is unlocked
mutex: Correct documentation on mutex optimistic spinning
mutex: Optimize mutex trylock slowpath

 kernel/locking/mutex.c |   47 ++++++++++++++++++++++++++---------------------
 1 files changed, 26 insertions(+), 21 deletions(-)
author
#1 lore raw

RFC PATCH 1/3 locking/mutex: Try to acquire mutex only if it is unlocked

Upon entering the slowpath in __mutex_lock_common(), we try once more to acquire the mutex. We only try to acquire it if MUTEX_SHOW_NO_WAITER (lock->count >= 0) is true in order to avoid using the atomic xchg() operation whenever it is not necessary. However, we really only need to try to acquire if the mutex is free (lock->count == 1).

This patch changes it so that we only try-acquire the mutex upon entering the slowpath if it is unlocked, rather than if there are no waiters. This helps further reduce unncessary atomic xchg() operations. Furthermore, this patch introduces and uses a new MUTEX_IS_UNLOCKED() macro to improve readbability.

---
 kernel/locking/mutex.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)
kernel/locking/mutex.c +6 -4
index bc73d33..0925968 100644
@@ -48,9 +48,10 @@
 
 /*
  * A negative mutex count indicates that waiters are sleeping waiting for the
- * mutex.
+ * mutex, and a count of one indicates the mutex is unlocked.
  */
 #define	MUTEX_SHOW_NO_WAITER(mutex)	(atomic_read(&(mutex)->count) >= 0)
+#define	MUTEX_IS_UNLOCKED(mutex)	(atomic_read(&(mutex)->count) == 1)
 
 void
 __mutex_init(struct mutex *lock, const char *name, struct lock_class_key *key)
@@ -440,7 +441,8 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass,
 		if (owner && !mutex_spin_on_owner(lock, owner))
 			break;
 
-		if ((atomic_read(&lock->count) == 1) &&
+		/* Try to acquire the lock if it is free. */
+		if (MUTEX_IS_UNLOCKED(lock) &&
 		    (atomic_cmpxchg(&lock->count, 1, 0) == 1)) {
 			lock_acquired(&lock->dep_map, ip);
 			if (use_ww_ctx) {
@@ -485,8 +487,8 @@ slowpath:
 #endif
 	spin_lock_mutex(&lock->wait_lock, flags);
 
-	/* once more, can we acquire the lock? */
-	if (MUTEX_SHOW_NO_WAITER(lock) && (atomic_xchg(&lock->count, 0) == 1))
+	/* once more, try to acquire the lock if it is free: */
+	if (MUTEX_IS_UNLOCKED(lock) && (atomic_xchg(&lock->count, 0) == 1))
 		goto skip_wait;
 
 	debug_mutex_lock_common(lock, &waiter);
Signature · 1 line
1.7.1
maintainer
#2 lore raw

Re: [RFC PATCH 1/3] locking/mutex: Try to acquire mutex only if it is unlocked

Quoted 22 lines · Jason Low

Upon entering the slowpath in __mutex_lock_common(), we try once more to acquire the mutex. We only try to acquire it if MUTEX_SHOW_NO_WAITER (lock->count >= 0) is true in order to avoid using the atomic xchg() operation whenever it is not necessary. However, we really only need to try to acquire if the mutex is free (lock->count == 1).

This patch changes it so that we only try-acquire the mutex upon entering the slowpath if it is unlocked, rather than if there are no waiters. This helps further reduce unncessary atomic xchg() operations. Furthermore, this patch introduces and uses a new MUTEX_IS_UNLOCKED() macro to improve readbability.

---
 kernel/locking/mutex.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)
kernel/locking/mutex.c +0 -0
index bc73d33..0925968 100644

kernel/locking/mutex.c line 54

Quoted 8 lines
kernel/locking/mutex.c +2 -1
@@ -48,9 +48,10 @@
 
 /*
  * A negative mutex count indicates that waiters are sleeping waiting for the
- * mutex.
+ * mutex, and a count of one indicates the mutex is unlocked.
  */
 #define	MUTEX_SHOW_NO_WAITER(mutex)	(atomic_read(&(mutex)->count) >= 0)
+#define	MUTEX_IS_UNLOCKED(mutex)	(atomic_read(&(mutex)->count) == 1)

So I recently saw that MUTEX_SHOW_NO_WAITER thing and cried a little; and now you're adding more of that same nonsense.

Please make them inline functions, also can we rename the SHOW_NO_WAITER thing, because its not at all clear to me wtf it does; should it be
called: mutex_no_waiters() or somesuch?

#3 lore raw

Re: [RFC PATCH 1/3] locking/mutex: Try to acquire mutex only if it is unlocked

Quoted 35 lines · Peter Zijlstra
Quoted 22 lines · Jason Low

Upon entering the slowpath in __mutex_lock_common(), we try once more to acquire the mutex. We only try to acquire it if MUTEX_SHOW_NO_WAITER (lock->count >= 0) is true in order to avoid using the atomic xchg() operation whenever it is not necessary. However, we really only need to try to acquire if the mutex is free (lock->count == 1).

This patch changes it so that we only try-acquire the mutex upon entering the slowpath if it is unlocked, rather than if there are no waiters. This helps further reduce unncessary atomic xchg() operations. Furthermore, this patch introduces and uses a new MUTEX_IS_UNLOCKED() macro to improve readbability.

---
 kernel/locking/mutex.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)
kernel/locking/mutex.c +0 -0
index bc73d33..0925968 100644

kernel/locking/mutex.c line 54

Quoted 8 lines
kernel/locking/mutex.c +2 -1
@@ -48,9 +48,10 @@
 
 /*
  * A negative mutex count indicates that waiters are sleeping waiting for the
- * mutex.
+ * mutex, and a count of one indicates the mutex is unlocked.
  */
 #define	MUTEX_SHOW_NO_WAITER(mutex)	(atomic_read(&(mutex)->count) >= 0)
+#define	MUTEX_IS_UNLOCKED(mutex)	(atomic_read(&(mutex)->count) == 1)

So I recently saw that MUTEX_SHOW_NO_WAITER thing and cried a little; and now you're adding more of that same nonsense.

Quoted 3 lines

Please make them inline functions, also can we rename the SHOW_NO_WAITER thing, because its not at all clear to me wtf it does; should it be
called: mutex_no_waiters() or somesuch?

Agreed.

In addition, how about the following helpers instead:
- mutex_is_unlocked() : count > 0
- mutex_has_waiters() : count < 0, or list_empty(->wait_list)

If combined with the negation, I think that pretty much covers the whole life-cycle of a mutex. The mutex.c file is full of places where we can replace crude atomic_reads with these.

Thanks,
Davidlohr

#4 lore raw

Re: [RFC PATCH 1/3] locking/mutex: Try to acquire mutex only if it is unlocked

Quoted 40 lines · Davidlohr Bueso
Quoted 35 lines · Peter Zijlstra
Quoted 22 lines · Jason Low

Upon entering the slowpath in __mutex_lock_common(), we try once more to acquire the mutex. We only try to acquire it if MUTEX_SHOW_NO_WAITER (lock->count >= 0) is true in order to avoid using the atomic xchg() operation whenever it is not necessary. However, we really only need to try to acquire if the mutex is free (lock->count == 1).

This patch changes it so that we only try-acquire the mutex upon entering the slowpath if it is unlocked, rather than if there are no waiters. This helps further reduce unncessary atomic xchg() operations. Furthermore, this patch introduces and uses a new MUTEX_IS_UNLOCKED() macro to improve readbability.

---
 kernel/locking/mutex.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)
kernel/locking/mutex.c +0 -0
index bc73d33..0925968 100644

kernel/locking/mutex.c line 54

Quoted 8 lines
kernel/locking/mutex.c +2 -1
@@ -48,9 +48,10 @@
 
 /*
  * A negative mutex count indicates that waiters are sleeping waiting for the
- * mutex.
+ * mutex, and a count of one indicates the mutex is unlocked.
  */
 #define	MUTEX_SHOW_NO_WAITER(mutex)	(atomic_read(&(mutex)->count) >= 0)
+#define	MUTEX_IS_UNLOCKED(mutex)	(atomic_read(&(mutex)->count) == 1)

So I recently saw that MUTEX_SHOW_NO_WAITER thing and cried a little; and now you're adding more of that same nonsense.

Quoted 3 lines

Please make them inline functions, also can we rename the SHOW_NO_WAITER thing, because its not at all clear to me wtf it does; should it be
called: mutex_no_waiters() or somesuch?

Quoted 5 lines

Agreed.

In addition, how about the following helpers instead:
- mutex_is_unlocked() : count > 0
- mutex_has_waiters() : count < 0, or list_empty(->wait_list)

                                      ^ err, that's !list_empty()
author
#5 lore raw

Re: [RFC PATCH 1/3] locking/mutex: Try to acquire mutex only if it is unlocked

Quoted 6 lines · Davidlohr Bueso
Quoted 3 lines · Davidlohr Bueso

In addition, how about the following helpers instead:
- mutex_is_unlocked() : count > 0
- mutex_has_waiters() : count < 0, or list_empty(->wait_list)

                                      ^ err, that's !list_empty()

Between checking for (count < 0) or checking for !list_empty(wait_list) for waiters:

Now that I think about it, I would expect a mutex_has_waiters() function to return !list_empty(wait_list) as that really tells whether or not there are waiters. For example, in highly contended cases, there can still be waiters on the mutex if count is 1.

Likewise, in places where we currently use "MUTEX_SHOW_NO_WAITER", we need to check for (count < 0) to ensure lock->count is a negative value before the thread sleeps on the mutex.

One option would be to still remove MUTEX_SHOW_NO_WAITER(), directly use atomic_read() in place of the macro, and just comment on why we have an extra atomic_read() that may "appear redundant". Another option could be to provide a function that checks for "potential waiters" on the mutex.

Any thoughts?

#6 lore raw

Re: [RFC PATCH 1/3] locking/mutex: Try to acquire mutex only if it is unlocked

Quoted 18 lines · Jason Low
Quoted 5 lines · Davidlohr Bueso
Quoted 3 lines · Davidlohr Bueso

In addition, how about the following helpers instead:
- mutex_is_unlocked() : count > 0
- mutex_has_waiters() : count < 0, or list_empty(->wait_list)

                                       ^ err, that's !list_empty()

Between checking for (count < 0) or checking for !list_empty(wait_list) for waiters:

Now that I think about it, I would expect a mutex_has_waiters() function to return !list_empty(wait_list) as that really tells whether or not there are waiters. For example, in highly contended cases, there can still be waiters on the mutex if count is 1.

Likewise, in places where we currently use "MUTEX_SHOW_NO_WAITER", we need to check for (count < 0) to ensure lock->count is a negative value before the thread sleeps on the mutex.

Quoted 7 lines

One option would be to still remove MUTEX_SHOW_NO_WAITER(), directly use atomic_read() in place of the macro, and just comment on why we have an extra atomic_read() that may "appear redundant". Another option could be to provide a function that checks for "potential waiters" on the mutex.

Any thoughts?

For the first MUTEX_SHOW_NO_WAITER() call site, you can replace it with a check for (count > 0). The second call site within the for loop, however, is a bit more tricky. It has to serve 2 purposes:

1. Opportunistically get the lock
2. Set the count value to -1 to indicate someone is waiting on the lock, that is why an xchg() operation has to be done even if its value is 0.

I do agree that the naming isn't that good. Maybe it can be changed to something like

static inline int mutex_value_has_waiters(mutex *lock) { return lock->count < 0; }

-Longman

author
#7 lore raw

Re: [RFC PATCH 1/3] locking/mutex: Try to acquire mutex only if it is unlocked

Quoted 24 lines · Wai Man
Quoted 23 lines · Jason Low
Quoted 5 lines · Davidlohr Bueso
Quoted 3 lines · Davidlohr Bueso

In addition, how about the following helpers instead:
- mutex_is_unlocked() : count > 0
- mutex_has_waiters() : count < 0, or list_empty(->wait_list)

                                       ^ err, that's !list_empty()

Between checking for (count < 0) or checking for !list_empty(wait_list) for waiters:

Now that I think about it, I would expect a mutex_has_waiters() function to return !list_empty(wait_list) as that really tells whether or not there are waiters. For example, in highly contended cases, there can still be waiters on the mutex if count is 1.

Likewise, in places where we currently use "MUTEX_SHOW_NO_WAITER", we need to check for (count < 0) to ensure lock->count is a negative value before the thread sleeps on the mutex.

One option would be to still remove MUTEX_SHOW_NO_WAITER(), directly use atomic_read() in place of the macro, and just comment on why we have an extra atomic_read() that may "appear redundant". Another option could be to provide a function that checks for "potential waiters" on the mutex.

Quoted 5 lines
Quoted 2 lines

Any thoughts?

For the first MUTEX_SHOW_NO_WAITER() call site, you can replace it with a check for (count > 0).

Yup, in my v2 patch, the first call site becomes !mutex_is_locked(lock) which is really a check for (count == 1).

Quoted 7 lines

The second call site within the for loop, however, is a bit more tricky. It has to serve 2 purposes:

1. Opportunistically get the lock
2. Set the count value to -1 to indicate someone is waiting on the lock, that is why an xchg() operation has to be done even if its value is 0.

Quoted 5 lines

I do agree that the naming isn't that good. Maybe it can be changed to something like

static inline int mutex_value_has_waiters(mutex *lock) { return
lock->count < 0; }

So I can imagine that a mutex_value_has_waiters() function might still not be a great name, since the mutex can have waiters in the case that the value lock->count >= 0.

In the second call site, do you think we should just do a direct atomic_read(lock->count) >= 0 and comment that we only do the xchg if the count is non-negative to avoid unnecessary xchg? That what I did in my v2 patch.

Thanks,
Jason

#8 lore raw

Re: [RFC PATCH 1/3] locking/mutex: Try to acquire mutex only if it is unlocked

Quoted 25 lines · Jason Low
Quoted 24 lines · Wai Man
Quoted 23 lines · Jason Low
Quoted 5 lines · Davidlohr Bueso
Quoted 3 lines · Davidlohr Bueso

In addition, how about the following helpers instead:
- mutex_is_unlocked() : count > 0
- mutex_has_waiters() : count < 0, or list_empty(->wait_list)

                                        ^ err, that's !list_empty()

Between checking for (count < 0) or checking for !list_empty(wait_list) for waiters:

Now that I think about it, I would expect a mutex_has_waiters() function to return !list_empty(wait_list) as that really tells whether or not there are waiters. For example, in highly contended cases, there can still be waiters on the mutex if count is 1.

Likewise, in places where we currently use "MUTEX_SHOW_NO_WAITER", we need to check for (count < 0) to ensure lock->count is a negative value before the thread sleeps on the mutex.

One option would be to still remove MUTEX_SHOW_NO_WAITER(), directly use atomic_read() in place of the macro, and just comment on why we have an extra atomic_read() that may "appear redundant". Another option could be to provide a function that checks for "potential waiters" on the mutex.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Any thoughts?

For the first MUTEX_SHOW_NO_WAITER() call site, you can replace it with a check for (count > 0).

Yup, in my v2 patch, the first call site becomes !mutex_is_locked(lock) which is really a check for (count == 1).

Yes, your v2 patch looks fine to me.

Quoted 12 lines
Quoted 7 lines

The second call site within the for loop, however, is a bit more tricky. It has to serve 2 purposes:

1. Opportunistically get the lock
2. Set the count value to -1 to indicate someone is waiting on the lock, that is why an xchg() operation has to be done even if its value is 0.

Quoted 5 lines

I do agree that the naming isn't that good. Maybe it can be changed to something like

static inline int mutex_value_has_waiters(mutex *lock) { return
lock->count < 0; }

Quoted 8 lines

So I can imagine that a mutex_value_has_waiters() function might still not be a great name, since the mutex can have waiters in the case that the value lock->count >= 0.

In the second call site, do you think we should just do a direct atomic_read(lock->count) >= 0 and comment that we only do the xchg if the count is non-negative to avoid unnecessary xchg? That what I did in my v2 patch.

I think that is a good idea to avoid any controversy in naming.

-Longman

author
#9 lore raw

Re: [RFC PATCH 1/3] locking/mutex: Try to acquire mutex only if it is unlocked

In reply to Davidlohr Bueso
Quoted 40 lines · Davidlohr Bueso
Quoted 35 lines · Peter Zijlstra
Quoted 22 lines · Jason Low

Upon entering the slowpath in __mutex_lock_common(), we try once more to acquire the mutex. We only try to acquire it if MUTEX_SHOW_NO_WAITER (lock->count >= 0) is true in order to avoid using the atomic xchg() operation whenever it is not necessary. However, we really only need to try to acquire if the mutex is free (lock->count == 1).

This patch changes it so that we only try-acquire the mutex upon entering the slowpath if it is unlocked, rather than if there are no waiters. This helps further reduce unncessary atomic xchg() operations. Furthermore, this patch introduces and uses a new MUTEX_IS_UNLOCKED() macro to improve readbability.

---
 kernel/locking/mutex.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)
kernel/locking/mutex.c +0 -0
index bc73d33..0925968 100644

kernel/locking/mutex.c line 54

Quoted 8 lines
kernel/locking/mutex.c +2 -1
@@ -48,9 +48,10 @@
 
 /*
  * A negative mutex count indicates that waiters are sleeping waiting for the
- * mutex.
+ * mutex, and a count of one indicates the mutex is unlocked.
  */
 #define	MUTEX_SHOW_NO_WAITER(mutex)	(atomic_read(&(mutex)->count) >= 0)
+#define	MUTEX_IS_UNLOCKED(mutex)	(atomic_read(&(mutex)->count) == 1)

So I recently saw that MUTEX_SHOW_NO_WAITER thing and cried a little; and now you're adding more of that same nonsense.

Quoted 3 lines

Please make them inline functions, also can we rename the SHOW_NO_WAITER thing, because its not at all clear to me wtf it does; should it be
called: mutex_no_waiters() or somesuch?

Quoted 5 lines

Agreed.

In addition, how about the following helpers instead:
- mutex_is_unlocked() : count > 0
- mutex_has_waiters() : count < 0, or list_empty(->wait_list)

Sounds good. Likewise, for "mutex_is_unlocked()" I've noticed that there is a mutex_is_locked() function provided in the linux/mutex.h file. Perhaps we can just reuse that function and use !mutex_is_locked() for places where we want to check if unlocked?

author
#10 lore raw

Re: [RFC PATCH 1/3] locking/mutex: Try to acquire mutex only if it is unlocked

In reply to Peter Zijlstra
Quoted 35 lines · Peter Zijlstra
Quoted 22 lines · Jason Low

Upon entering the slowpath in __mutex_lock_common(), we try once more to acquire the mutex. We only try to acquire it if MUTEX_SHOW_NO_WAITER (lock->count >= 0) is true in order to avoid using the atomic xchg() operation whenever it is not necessary. However, we really only need to try to acquire if the mutex is free (lock->count == 1).

This patch changes it so that we only try-acquire the mutex upon entering the slowpath if it is unlocked, rather than if there are no waiters. This helps further reduce unncessary atomic xchg() operations. Furthermore, this patch introduces and uses a new MUTEX_IS_UNLOCKED() macro to improve readbability.

---
 kernel/locking/mutex.c |   10 ++++++----
 1 files changed, 6 insertions(+), 4 deletions(-)
kernel/locking/mutex.c +0 -0
index bc73d33..0925968 100644

kernel/locking/mutex.c line 54

Quoted 8 lines
kernel/locking/mutex.c +2 -1
@@ -48,9 +48,10 @@
 
 /*
  * A negative mutex count indicates that waiters are sleeping waiting for the
- * mutex.
+ * mutex, and a count of one indicates the mutex is unlocked.
  */
 #define	MUTEX_SHOW_NO_WAITER(mutex)	(atomic_read(&(mutex)->count) >= 0)
+#define	MUTEX_IS_UNLOCKED(mutex)	(atomic_read(&(mutex)->count) == 1)

So I recently saw that MUTEX_SHOW_NO_WAITER thing and cried a little; and now you're adding more of that same nonsense.

Quoted 3 lines

Please make them inline functions, also can we rename the SHOW_NO_WAITER thing, because its not at all clear to me wtf it does; should it be
called: mutex_no_waiters() or somesuch?

Okay, I can make them inline functions. I mainly added the macro to keep it consistent with the MUTEX_SHOW_NO_WAITER() check, but we can surely make this more clear. mutex_no_waiters() sounds fine, or perhaps something like mutex_has_no_waiters()?

#11 lore raw

Re: [RFC PATCH 1/3] locking/mutex: Try to acquire mutex only if it is unlocked

Quoted 12 lines · Jason Low
Quoted 11 lines · Peter Zijlstra
Quoted 7 lines

/*
* A negative mutex count indicates that waiters are sleeping waiting for the
- * mutex.
+ * mutex, and a count of one indicates the mutex is unlocked.
*/
#define MUTEX_SHOW_NO_WAITER(mutex) (atomic_read(&(mutex)->count) >= 0)
+#define MUTEX_IS_UNLOCKED(mutex) (atomic_read(&(mutex)->count) == 1)

So I recently saw that MUTEX_SHOW_NO_WAITER thing and cried a little; and now you're adding more of that same nonsense.

Quoted 6 lines
Quoted 3 lines

Please make them inline functions, also can we rename the SHOW_NO_WAITER thing, because its not at all clear to me wtf it does; should it be
called: mutex_no_waiters() or somesuch?

Okay, I can make them inline functions. I mainly added the macro to keep it consistent with the MUTEX_SHOW_NO_WAITER() check, but we can surely

Consistency with a digusting and nonsensical macro is not really a good argument.

Quoted 2 lines

make this more clear. mutex_no_waiters() sounds fine, or perhaps something like mutex_has_no_waiters()?

Uuurg. So we end up with

       if (!mutex_has_no_waiters(m))

if we check for waiters?

Can we please go with the most intuitive thing:

mutex_has_waiters()

and have the callsites prepend the '!' in case they want to check there is no waiter?

For heavens sake, we do not name macros/inlines in a way which fits the intended use case. We name them so they make sense.

Your change log blurbs about readability. I have no idea what your understandig of readability is, but neither MUTEX_SHOW_NO_WAITERS nor mutex_has_no_waiters qualify for me. Ditto for MUTEX_IS_UNLOCKED.

Care to look at the other lock implementations:

     rt_mutex_has_waiters()
     spin_is_locked()
     ....

Why would it make sense to come up with reverse conventions for mutex?

Thanks,

tglx

author
#12 lore raw

Re: [RFC PATCH 1/3] locking/mutex: Try to acquire mutex only if it is unlocked

Quoted 17 lines · Thomas Gleixner
Quoted 16 lines · Jason Low
Quoted 14 lines · Peter Zijlstra
Quoted 7 lines

/*
* A negative mutex count indicates that waiters are sleeping waiting for the
- * mutex.
+ * mutex, and a count of one indicates the mutex is unlocked.
*/
#define MUTEX_SHOW_NO_WAITER(mutex) (atomic_read(&(mutex)->count) >= 0)
+#define MUTEX_IS_UNLOCKED(mutex) (atomic_read(&(mutex)->count) == 1)

So I recently saw that MUTEX_SHOW_NO_WAITER thing and cried a little; and now you're adding more of that same nonsense.

Please make them inline functions, also can we rename the SHOW_NO_WAITER thing, because its not at all clear to me wtf it does; should it be
called: mutex_no_waiters() or somesuch?

Quoted 5 lines
Quoted 2 lines

Okay, I can make them inline functions. I mainly added the macro to keep it consistent with the MUTEX_SHOW_NO_WAITER() check, but we can surely

Consistency with a digusting and nonsensical macro is not really a good argument.

I agree :)

Quoted 7 lines
Quoted 2 lines

make this more clear. mutex_no_waiters() sounds fine, or perhaps something like mutex_has_no_waiters()?

Uuurg. So we end up with

       if (!mutex_has_no_waiters(m))
Quoted 8 lines

if we check for waiters?

Can we please go with the most intuitive thing:

mutex_has_waiters()

and have the callsites prepend the '!' in case they want to check there is no waiter?

Yes, !mutex_has_waiters() sounds like the better option to check for no waiters. Same with using the already provided mutex_is_locked() function.

Quoted 13 lines

For heavens sake, we do not name macros/inlines in a way which fits the intended use case. We name them so they make sense.

Your change log blurbs about readability. I have no idea what your understandig of readability is, but neither MUTEX_SHOW_NO_WAITERS nor mutex_has_no_waiters qualify for me. Ditto for MUTEX_IS_UNLOCKED.

Care to look at the other lock implementations:

     rt_mutex_has_waiters()
     spin_is_locked()
     ....
Quoted 5 lines

Why would it make sense to come up with reverse conventions for mutex?

Thanks,

tglx

Thanks,
Jason

#13 lore raw

Re: [RFC PATCH 1/3] locking/mutex: Try to acquire mutex only if it is unlocked

In reply to Jason Low
Quoted 35 lines · Jason Low
Quoted 34 lines · Peter Zijlstra
Quoted 22 lines · Jason Low

Upon entering the slowpath in __mutex_lock_common(), we try once more to acquire the mutex. We only try to acquire it if MUTEX_SHOW_NO_WAITER (lock->count>= 0) is true in order to avoid using the atomic xchg() operation whenever it is not necessary. However, we really only need to try to acquire if the mutex is free (lock->count == 1).

This patch changes it so that we only try-acquire the mutex upon entering the slowpath if it is unlocked, rather than if there are no waiters. This helps further reduce unncessary atomic xchg() operations. Furthermore, this patch introduces and uses a new MUTEX_IS_UNLOCKED() macro to improve readbability.

---
  kernel/locking/mutex.c |   10 ++++++----
  1 files changed, 6 insertions(+), 4 deletions(-)
kernel/locking/mutex.c +0 -0
index bc73d33..0925968 100644

kernel/locking/mutex.c line 54

Quoted 8 lines
kernel/locking/mutex.c +2 -1
@@ -48,9 +48,10 @@
 
  /*
   * A negative mutex count indicates that waiters are sleeping waiting for the
- * mutex.
+ * mutex, and a count of one indicates the mutex is unlocked.
   */
  #define	MUTEX_SHOW_NO_WAITER(mutex)	(atomic_read(&(mutex)->count)>= 0)
+#define	MUTEX_IS_UNLOCKED(mutex)	(atomic_read(&(mutex)->count) == 1)

So I recently saw that MUTEX_SHOW_NO_WAITER thing and cried a little; and now you're adding more of that same nonsense.

Quoted 8 lines
Quoted 3 lines

Please make them inline functions, also can we rename the SHOW_NO_WAITER thing, because its not at all clear to me wtf it does; should it be
called: mutex_no_waiters() or somesuch?

Okay, I can make them inline functions. I mainly added the macro to keep it consistent with the MUTEX_SHOW_NO_WAITER() check, but we can surely make this more clear. mutex_no_waiters() sounds fine, or perhaps something like mutex_has_no_waiters()?

You can remove the MUTEX_SHOW_NO_WAITER macro as all the call sites are to be replaced. I didn't check directly for unlocked count because of fairness concern in my original patch, but I think checking directly for unlocked count should be fine too.

-Longman

author
#14 lore raw

Re: [RFC PATCH 1/3] locking/mutex: Try to acquire mutex only if it is unlocked

Quoted 5 lines · Waiman Long
Quoted 4 lines · Jason Low
Quoted 3 lines · Peter Zijlstra

Please make them inline functions, also can we rename the SHOW_NO_WAITER thing, because its not at all clear to me wtf it does; should it be
called: mutex_no_waiters() or somesuch?

Quoted 8 lines
Quoted 5 lines

Okay, I can make them inline functions. I mainly added the macro to keep it consistent with the MUTEX_SHOW_NO_WAITER() check, but we can surely make this more clear. mutex_no_waiters() sounds fine, or perhaps something like mutex_has_no_waiters()?

You can remove the MUTEX_SHOW_NO_WAITER macro as all the call sites are to be replaced.

Sure.

Quoted 3 lines

I didn't check directly for unlocked count because of fairness concern in my original patch, but I think checking directly for unlocked count should be fine too.

Can you elaborate on the "fairness concern"? In the current code, we're already directly checking for unlocked count in atomic_read(&lock->count) == 1 if that's what you're referring to.

Thanks,
Jason

author
#15 lore raw

RFC PATCH 2/3 locking/mutex: Correct documentation on mutex optimistic spinning

In reply to Jason Low

The mutex optimistic spinning documentation states that we spin for acquisition when we find that there are no pending waiters. However, in actuality, whether or not there are waiters for the mutex doesn't determine if we will spin for it.

This patch removes that statement and also adds a comment which mentions that we spin for the mutex while we don't need to reschedule.

---
 kernel/locking/mutex.c |   10 ++++------
 1 files changed, 4 insertions(+), 6 deletions(-)
kernel/locking/mutex.c +4 -6
index 0925968..fc55f72 100644
@@ -389,12 +389,10 @@ __mutex_lock_common(struct mutex *lock, long state, unsigned int subclass,
 	/*
 	 * Optimistic spinning.
 	 *
-	 * We try to spin for acquisition when we find that there are no
-	 * pending waiters and the lock owner is currently running on a
-	 * (different) CPU.
-	 *
-	 * The rationale is that if the lock owner is running, it is likely to
-	 * release the lock soon.
+	 * We try to spin for acquisition when we find that the lock owner
+	 * is currently running on a (different) CPU and while we don't
+	 * need to reschedule. The rationale is that if the lock owner is
+	 * running, it is likely to release the lock soon.
 	 *
 	 * Since this needs the lock owner, and this mutex implementation
 	 * doesn't track the owner atomically in the lock field, we need to
Signature · 1 line
1.7.1
Acked-by on v1 2/3
#16 lore raw

Re: [RFC PATCH 2/3] locking/mutex: Correct documentation on mutex optimistic spinning

Quoted 9 lines · Jason Low

The mutex optimistic spinning documentation states that we spin for acquisition when we find that there are no pending waiters. However, in actuality, whether or not there are waiters for the mutex doesn't determine if we will spin for it.

This patch removes that statement and also adds a comment which mentions that we spin for the mutex while we don't need to reschedule.

I think this could go in with v4 of the mutex doc rewrite patch... guessing for 3.17 now.

Thanks,
Davidlohr

author
#17 lore raw

Re: [RFC PATCH 2/3] locking/mutex: Correct documentation on mutex optimistic spinning

Quoted 9 lines · Davidlohr Bueso
Quoted 8 lines · Jason Low

The mutex optimistic spinning documentation states that we spin for acquisition when we find that there are no pending waiters. However, in actuality, whether or not there are waiters for the mutex doesn't determine if we will spin for it.

This patch removes that statement and also adds a comment which mentions that we spin for the mutex while we don't need to reschedule.

Quoted 6 lines
Quoted 1 line

I think this could go in with v4 of the mutex doc rewrite patch... guessing for 3.17 now.

Yeah, this patch could be for 3.17.

Thanks,
Jason

author
#18 lore raw

RFC PATCH 3/3 locking/mutex: Optimize mutex trylock slowpath

In reply to Jason Low

In __mutex_trylock_slowpath(), we acquire the wait_lock spinlock, xchg() lock->count with -1, then set lock->count back to 0 if there are no waiters, and return true if the prev lock count was 1.

However, if we the mutex is already locked, then there may not be much point in attempting the above operations. In this patch, we only attempt the above operations if the mutex is unlocked.

The new MUTEX_IS_UNLOCKED() macro is also used for this.

---
 kernel/locking/mutex.c |   27 ++++++++++++++++-----------
 1 files changed, 16 insertions(+), 11 deletions(-)
kernel/locking/mutex.c +16 -11
index fc55f72..c65680d 100644
@@ -821,21 +821,26 @@ static inline int __mutex_trylock_slowpath(atomic_t *lock_count)
 {
 	struct mutex *lock = container_of(lock_count, struct mutex, count);
 	unsigned long flags;
-	int prev;
+	int prev = 0;
 
-	spin_lock_mutex(&lock->wait_lock, flags);
+	/*
+	 * Only need to trylock the mutex if it is unlocked.
+	 */
+	if (MUTEX_IS_UNLOCKED(lock)) {
+		spin_lock_mutex(&lock->wait_lock, flags);
 
-	prev = atomic_xchg(&lock->count, -1);
-	if (likely(prev == 1)) {
-		mutex_set_owner(lock);
-		mutex_acquire(&lock->dep_map, 0, 1, _RET_IP_);
-	}
+		prev = atomic_xchg(&lock->count, -1);
+		if (likely(prev == 1)) {
+			mutex_set_owner(lock);
+			mutex_acquire(&lock->dep_map, 0, 1, _RET_IP_);
+		}
 
-	/* Set it back to 0 if there are no waiters: */
-	if (likely(list_empty(&lock->wait_list)))
-		atomic_set(&lock->count, 0);
+		/* Set it back to 0 if there are no waiters: */
+		if (likely(list_empty(&lock->wait_list)))
+			atomic_set(&lock->count, 0);
 
-	spin_unlock_mutex(&lock->wait_lock, flags);
+		spin_unlock_mutex(&lock->wait_lock, flags);
+	}
 
 	return prev == 1;
 }
Signature · 1 line
1.7.1
#19 lore raw

Re: [RFC PATCH 3/3] locking/mutex: Optimize mutex trylock slowpath

Quoted 5 lines · Jason Low

In __mutex_trylock_slowpath(), we acquire the wait_lock spinlock, xchg() lock->count with -1, then set lock->count back to 0 if there are no waiters, and return true if the prev lock count was 1.

However, if we the mutex is already locked, then there may not be

^^ leave that out.

Quoted 1 line

much point in attempting the above operations.

Isn't this redundant? I mean, if we enter the slowpath its because __mutex_fastpath_trylock() already failed so we already know that the lock is taken.

What kind of testing has this change been put through? Any advantages? (ie: how many cycles are we saving here?), the trylock mechanism is already pretty darn fast.

Thanks,
Davidlohr

author
#20 lore raw

Re: [RFC PATCH 3/3] locking/mutex: Optimize mutex trylock slowpath

Quoted 5 lines · Davidlohr Bueso
Quoted 4 lines · Jason Low

In __mutex_trylock_slowpath(), we acquire the wait_lock spinlock, xchg() lock->count with -1, then set lock->count back to 0 if there are no waiters, and return true if the prev lock count was 1.

Quoted 8 lines
Quoted 1 line

However, if we the mutex is already locked, then there may not be

^^ leave that out.

Quoted 1 line

much point in attempting the above operations.

Isn't this redundant? I mean, if we enter the slowpath its because __mutex_fastpath_trylock() already failed so we already know that the lock is taken.

This function is really just used as an alternative method of trylock for !__HAVE_ARCH_CMPXCHG. In that case, the fastpath can call directly into the slowpath function, without checking for if the lock is taken.

Quoted 3 lines

What kind of testing has this change been put through? Any advantages? (ie: how many cycles are we saving here?), the trylock mechanism is already pretty darn fast.

While I did run tests with this patch, this particular patch shouldn't show benefits on my machine as it should be using the more efficient atomic_cmpxchg. The advantage is in !__HAVE_ARCH_CMPXCHG, where we would avoid taking a spinlock and 2 atomic operations when the mutex is already taken.

Thanks.

#21 lore raw

Re: [RFC PATCH 3/3] locking/mutex: Optimize mutex trylock slowpath

Quoted 11 lines · Jason Low
Quoted 10 lines · Davidlohr Bueso
Quoted 5 lines · Jason Low

In __mutex_trylock_slowpath(), we acquire the wait_lock spinlock, xchg() lock->count with -1, then set lock->count back to 0 if there are no waiters, and return true if the prev lock count was 1.

However, if we the mutex is already locked, then there may not be

^^ leave that out.

Quoted 1 line

much point in attempting the above operations.

Quoted 7 lines
Quoted 3 lines

Isn't this redundant? I mean, if we enter the slowpath its because __mutex_fastpath_trylock() already failed so we already know that the lock is taken.

This function is really just used as an alternative method of trylock for !__HAVE_ARCH_CMPXCHG. In that case, the fastpath can call directly into the slowpath function, without checking for if the lock is taken.

Ah, ok I hadn't seen that we do this in 32bit x86 and was wondering why the heck we fallback on a slowpath for something like trylock, which should return right away no matter what. I'd suggest explicitly mentioning this in the changelog. Otherwise makes sense now.

author
#22 lore raw

Re: [RFC PATCH 3/3] locking/mutex: Optimize mutex trylock slowpath

Quoted 16 lines · Davidlohr Bueso
Quoted 15 lines · Jason Low
Quoted 13 lines · Davidlohr Bueso
Quoted 5 lines · Jason Low

In __mutex_trylock_slowpath(), we acquire the wait_lock spinlock, xchg() lock->count with -1, then set lock->count back to 0 if there are no waiters, and return true if the prev lock count was 1.

However, if we the mutex is already locked, then there may not be

^^ leave that out.

Quoted 1 line

much point in attempting the above operations.

Isn't this redundant? I mean, if we enter the slowpath its because __mutex_fastpath_trylock() already failed so we already know that the lock is taken.

Quoted 8 lines
Quoted 3 lines

This function is really just used as an alternative method of trylock for !__HAVE_ARCH_CMPXCHG. In that case, the fastpath can call directly into the slowpath function, without checking for if the lock is taken.

Ah, ok I hadn't seen that we do this in 32bit x86 and was wondering why the heck we fallback on a slowpath for something like trylock, which should return right away no matter what. I'd suggest explicitly mentioning this in the changelog. Otherwise makes sense now.

Yup, I was also initially wondering why we have a slowpath for trylock, which prompted me to take a look into this function. I'll add some more information about this to the changelog.

Thanks,
Jason