locking/rwsem: Optimize write lock by reducing operations in slowpath
PATCH v2 1/1 locking/rwsem: Optimize write lock by reducing operations in slowpath
When acquiring the rwsem write lock in the slowpath, we first try to set count to RWSEM_WAITING_BIAS. When that is successful, we then atomically add the RWSEM_WAITING_BIAS in cases where there are other tasks on the wait list. This causes write lock operations to often issue multiple atomic operations.
We can instead make the list_is_singular() check first, and then set the count accordingly, so that we issue at most 1 atomic operation when acquiring the write lock and reduce unnecessary cacheline contention.
Signed-off-by Jason Low <[email protected]>
Acked-by Waiman Long<[email protected]>
Acked-by Davidlohr Bueso <[email protected]>
--- kernel/locking/rwsem-xadd.c | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-)
kernel/locking/rwsem-xadd.c +18 -7
@@ -255,17 +255,28 @@ struct rw_semaphore __sched *rwsem_down_read_failed(struct rw_semaphore *sem) }EXPORT_SYMBOL (rwsem_down_read_failed); +/* +* This function must be called with the sem->wait_lock held to prevent +* race conditions between checking the rwsem wait list and setting the +* sem->count accordingly. +*/ static inline bool rwsem_try_write_lock (long count,struct rw_semaphore *sem) {/* -* Try acquiring the write lock. Check count first in order -* to reduce unnecessary expensive cmpxchg() operations. +* Avoid trying to acquire write lock if count isn't RWSEM_WAITING_BIAS. */ -if (count ==RWSEM_WAITING_BIAS && -cmpxchg_acquire (&sem->count,RWSEM_WAITING_BIAS , -RWSEM_ACTIVE_WRITE_BIAS ) ==RWSEM_WAITING_BIAS ) { -if (!list_is_singular (&sem->wait_list)) -rwsem_atomic_update (RWSEM_WAITING_BIAS , sem); +if (count !=RWSEM_WAITING_BIAS ) +return false; + +/* +* Acquire the lock by trying to set it to ACTIVE_WRITE_BIAS. If there +* are other tasks on the wait list, we need to add on WAITING_BIAS. +*/ + count =list_is_singular (&sem->wait_list) ? +RWSEM_ACTIVE_WRITE_BIAS : +RWSEM_ACTIVE_WRITE_BIAS +RWSEM_WAITING_BIAS ; + +if (cmpxchg_acquire (&sem->count,RWSEM_WAITING_BIAS , count) ==RWSEM_WAITING_BIAS ) {rwsem_set_owner (sem);return true; }
Signature · 1 line
2.1.4