locking/rwsem: Avoid double checking before try acquiring write lock
PATCH v2 1/1 locking/rwsem: Avoid double checking before try acquiring write lock
Commit 9b0fc9c09f1b checks for if there are known active lockers in order to avoid write trylocking using expensive cmpxchg() when it likely wouldn't get the lock.
However, a subsequent patch was added such that we directly check for sem->count == RWSEM_WAITING_BIAS right before trying that cmpxchg(). Thus, commit 9b0fc9c09f1b now just adds extra overhead. This patch deletes it.
Also, add a comment on why we do an "extra check" of sem->count before the cmpxchg().
--- kernel/locking/rwsem-xadd.c | 24 +++++++++++++----------- 1 files changed, 13 insertions(+), 11 deletions(-)
kernel/locking/rwsem-xadd.c +13 -11
@@ -247,18 +247,20 @@ struct rw_semaphore __sched *rwsem_down_read_failed(struct rw_semaphore *sem)return sem; } -static inline bool rwsem_try_write_lock (long count,struct rw_semaphore *sem) +static inline bool rwsem_try_write_lock (struct rw_semaphore *sem) { -if (!(count &RWSEM_ACTIVE_MASK )) { -/* try acquiring the write lock */ -if (sem->count ==RWSEM_WAITING_BIAS && -cmpxchg (&sem->count,RWSEM_WAITING_BIAS , -RWSEM_ACTIVE_WRITE_BIAS ) ==RWSEM_WAITING_BIAS ) { -if (!list_is_singular (&sem->wait_list)) -rwsem_atomic_update (RWSEM_WAITING_BIAS , sem); -return true; - } +/* +* Try acquiring the write lock. Check sem->count first +* in order to reduce unnecessary expensive cmpxchg() operations. +*/ +if (sem->count ==RWSEM_WAITING_BIAS && +cmpxchg (&sem->count,RWSEM_WAITING_BIAS , +RWSEM_ACTIVE_WRITE_BIAS ) ==RWSEM_WAITING_BIAS ) { +if (!list_is_singular (&sem->wait_list)) +rwsem_atomic_update (RWSEM_WAITING_BIAS , sem); +return true; } +return false; }
@@ -446,7 +448,7 @@ struct rw_semaphore __sched *rwsem_down_write_failed(struct rw_semaphore *sem)/* wait until we successfully acquire the lock */ set_current_state (TASK_UNINTERRUPTIBLE );while (true) { -if (rwsem_try_write_lock (count, sem)) +if (rwsem_try_write_lock (sem))break ;raw_spin_unlock_irq (&sem->wait_lock);
Signature · 1 line
1.7.1
Hi Jason,
Quoted 8 lines · Jason Low
Commit
9b0fc9c09f1bchecks for if there are known active lockers in order to avoid write trylocking using expensivecmpxchg()when it likely wouldn't get the lock.However, a subsequent patch was added such that we directly check for
sem->count==RWSEM_WAITING_BIASright before trying thatcmpxchg(). Thus, commit9b0fc9c09f1bnow just adds extra overhead. This patch deletes it.
It would be better to just not reload sem->count, and check the parameter count == RWSEM_WAITING_BIAS instead. The count parameter is a very recent load of sem->count (one of which is the latest exclusive read from an atomic operation), so likely to be just as accurate as a reload of sem->count without causing more cache line contention.
Regards,
Peter Hurley
Quoted 53 lines
Also, add a comment on why we do an "extra check" of
sem->countbefore thecmpxchg().Signed-off-by Jason Low <[email protected]>--- kernel/locking/rwsem-xadd.c | 24 +++++++++++++----------- 1 files changed, 13 insertions(+), 11 deletions(-)kernel/locking/rwsem-xadd.c +13
-11@@ -247,18 +247,20 @@ struct rw_semaphore __sched *rwsem_down_read_failed(struct rw_semaphore *sem)return sem; } -static inline bool rwsem_try_write_lock (long count,struct rw_semaphore *sem) +static inline bool rwsem_try_write_lock (struct rw_semaphore *sem) { -if (!(count &RWSEM_ACTIVE_MASK )) { -/* try acquiring the write lock */ -if (sem->count ==RWSEM_WAITING_BIAS && -cmpxchg (&sem->count,RWSEM_WAITING_BIAS , -RWSEM_ACTIVE_WRITE_BIAS ) ==RWSEM_WAITING_BIAS ) { -if (!list_is_singular (&sem->wait_list)) -rwsem_atomic_update (RWSEM_WAITING_BIAS , sem); -return true; - } +/* +* Try acquiring the write lock. Check sem->count first +* in order to reduce unnecessary expensive cmpxchg() operations. +*/ +if (sem->count ==RWSEM_WAITING_BIAS && +cmpxchg (&sem->count,RWSEM_WAITING_BIAS , +RWSEM_ACTIVE_WRITE_BIAS ) ==RWSEM_WAITING_BIAS ) { +if (!list_is_singular (&sem->wait_list)) +rwsem_atomic_update (RWSEM_WAITING_BIAS , sem); +return true; } +return false; }@@ -446,7 +448,7 @@ struct rw_semaphore __sched *rwsem_down_write_failed(struct rw_semaphore *sem)/* wait until we successfully acquire the lock */ set_current_state (TASK_UNINTERRUPTIBLE );while (true) { -if (rwsem_try_write_lock (count, sem)) +if (rwsem_try_write_lock (sem))break ;raw_spin_unlock_irq (&sem->wait_lock);
Quoted 12 lines · Peter Hurley
Hi Jason,
Quoted 8 lines · Jason Low
Commit
9b0fc9c09f1bchecks for if there are known active lockers in order to avoid write trylocking using expensivecmpxchg()when it likely wouldn't get the lock.However, a subsequent patch was added such that we directly check for
sem->count==RWSEM_WAITING_BIASright before trying thatcmpxchg(). Thus, commit9b0fc9c09f1bnow just adds extra overhead. This patch deletes it.
Quoted 6 lines
It would be better to just not reload
sem->count, and check the parameter count ==RWSEM_WAITING_BIASinstead. The count parameter is a very recent load ofsem->count(one of which is the latest exclusive read from an atomic operation), so likely to be just as accurate as a reload ofsem->countwithout causing more cache line contention.
Agree with Peter.
I think the extra check in the original code was to try to avoid reloading sem->count.
So checking directly here (count == RWSEM_WAITING_BIAS) will accomplish that end. You'll need to modify your comment slightly to say
Try acquiring the write lock. Check count first ...
Thanks.
Tim
Quoted 57 lines
Regards,
Peter HurleyQuoted 53 lines
Also, add a comment on why we do an "extra check" of
sem->countbefore thecmpxchg().Signed-off-by Jason Low <[email protected]>--- kernel/locking/rwsem-xadd.c | 24 +++++++++++++----------- 1 files changed, 13 insertions(+), 11 deletions(-)kernel/locking/rwsem-xadd.c +13
-11@@ -247,18 +247,20 @@ struct rw_semaphore __sched *rwsem_down_read_failed(struct rw_semaphore *sem)return sem; } -static inline bool rwsem_try_write_lock (long count,struct rw_semaphore *sem) +static inline bool rwsem_try_write_lock (struct rw_semaphore *sem) { -if (!(count &RWSEM_ACTIVE_MASK )) { -/* try acquiring the write lock */ -if (sem->count ==RWSEM_WAITING_BIAS && -cmpxchg (&sem->count,RWSEM_WAITING_BIAS , -RWSEM_ACTIVE_WRITE_BIAS ) ==RWSEM_WAITING_BIAS ) { -if (!list_is_singular (&sem->wait_list)) -rwsem_atomic_update (RWSEM_WAITING_BIAS , sem); -return true; - } +/* +* Try acquiring the write lock. Check sem->count first +* in order to reduce unnecessary expensive cmpxchg() operations. +*/ +if (sem->count ==RWSEM_WAITING_BIAS && +cmpxchg (&sem->count,RWSEM_WAITING_BIAS , +RWSEM_ACTIVE_WRITE_BIAS ) ==RWSEM_WAITING_BIAS ) { +if (!list_is_singular (&sem->wait_list)) +rwsem_atomic_update (RWSEM_WAITING_BIAS , sem); +return true; } +return false; }@@ -446,7 +448,7 @@ struct rw_semaphore __sched *rwsem_down_write_failed(struct rw_semaphore *sem)/* wait until we successfully acquire the lock */ set_current_state (TASK_UNINTERRUPTIBLE );while (true) { -if (rwsem_try_write_lock (count, sem)) +if (rwsem_try_write_lock (sem))break ;raw_spin_unlock_irq (&sem->wait_lock);
Quoted 12 lines · Peter Hurley
Hi Jason,
Quoted 8 lines · Jason Low
Commit
9b0fc9c09f1bchecks for if there are known active lockers in order to avoid write trylocking using expensivecmpxchg()when it likely wouldn't get the lock.However, a subsequent patch was added such that we directly check for
sem->count==RWSEM_WAITING_BIASright before trying thatcmpxchg(). Thus, commit9b0fc9c09f1bnow just adds extra overhead. This patch deletes it.
Quoted 5 lines
It would be better to just not reload
sem->count, and check the parameter count ==RWSEM_WAITING_BIASinstead. The count parameter is a very recent load ofsem->count(one of which is the latest exclusive read from an atomic operation), so likely to be just as accurate as a reload ofsem->countwithout causing more cache line contention.
Hi Peter and Tim,
Yes, I also agree. I will send out a new patch with this update.
Thanks,
Jason