locking/rwsem: Teach contention tracing about optimistic spinning
Subsystems (2)
PATCH 1/1 locking/rwsem: Teach contention tracing about optimistic spinning
Similar to the mutex counterpart, we can further distinguish the types of contention. Similarly this patch also introduces potentially various _begin() tracepoints with a single respective _end().
- The original _begin() for down_write() is moved further up, right after we know optimistic spinning is bust.
- For the scenario that spins after setting the hand-off bit and failing to grab the lock the code is change to duplicate the rwsem_try_write_lock() upon a OWNER_NULL, which minimizes the amounts of _begin() in the wait-loop - but also gets rid of a goto label and the logic is pretty encapsulated in the branch. In such cases the common case will be to acquire the lock, but if it is stolen in that window, this path won't see the signal_pending() now in the iteration and block. This should be benign as the check will come in after waking if the lock can still not be taken.
---
rtmutexes' top waiter will also do optimitic spinning, but I don't think it is worth adding tracepoints here as it is quite minimal, unlike the osq stuff.
include/trace/events/lock.h | 4 +++- kernel/locking/rwsem.c | 29 +++++++++++++++++++++-------- 2 files changed, 24 insertions(+), 9 deletions(-)
include/trace/events/lock.h +3 -1
@@ -15,6 +15,7 @@#define LCB_F_RT (1U << 3)#define LCB_F_PERCPU (1U << 4)#define LCB_F_MUTEX (1U << 5) +#define LCB_F_RWSEM (1U << 6)#ifdef CONFIG_LOCKDEP
@@ -115,7 +116,8 @@ TRACE_EVENT(contention_begin, { LCB_F_WRITE, "WRITE" }, { LCB_F_RT, "RT" }, { LCB_F_PERCPU, "PERCPU" }, - { LCB_F_MUTEX, "MUTEX" } + { LCB_F_MUTEX, "MUTEX" }, + { LCB_F_RWSEM, "RWSEM" } )) );
kernel/locking/rwsem.c +21 -8
@@ -1057,7 +1057,7 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int statif (!wake_q_empty (&wake_q))wake_up_q (&wake_q); -trace_contention_begin (sem,LCB_F_READ ); +trace_contention_begin (sem,LCB_F_RWSEM |LCB_F_READ );/* wait to be given the lock */ for (;;) {
@@ -1101,9 +1101,14 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)DEFINE_WAKE_Q (wake_q);/* do optimistic spinning and steal lock if possible */ -if (rwsem_can_spin_on_owner (sem) &&rwsem_optimistic_spin (sem)) { -/* rwsem_optimistic_spin() implies ACQUIRE on success */ -return sem; +if (rwsem_can_spin_on_owner (sem)) { +trace_contention_begin (sem, +LCB_F_RWSEM |LCB_F_WRITE |LCB_F_SPIN ); +if (rwsem_optimistic_spin (sem)) { +/* rwsem_optimistic_spin() implies ACQUIRE on success */ +trace_contention_end (sem,0 ); +return sem; + } }/*
@@ -1115,6 +1120,8 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) waiter.timeout = jiffies +RWSEM_WAIT_TIMEOUT ; waiter.handoff_set = false; +trace_contention_begin (sem,LCB_F_RWSEM |LCB_F_WRITE ); +raw_spin_lock_irq (&sem->wait_lock);rwsem_add_waiter (sem, &waiter);
@@ -1137,7 +1144,6 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)/* wait until we successfully acquire the lock */ set_current_state (state); -trace_contention_begin (sem,LCB_F_WRITE );for (;;) {if (rwsem_try_write_lock (sem, &waiter)) {
@@ -1161,18 +1167,25 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)if (waiter.handoff_set) {enum owner_state owner_state; +trace_contention_begin (sem,LCB_F_RWSEM | +LCB_F_WRITE |LCB_F_SPIN );preempt_disable (); owner_state =rwsem_spin_on_owner (sem);preempt_enable (); -if (owner_state ==OWNER_NULL ) - goto trylock_again; +if (owner_state ==OWNER_NULL ) { +raw_spin_lock_irq (&sem->wait_lock); +if (rwsem_try_write_lock (sem, &waiter)) +break ; +raw_spin_unlock_irq (&sem->wait_lock); + } + +trace_contention_begin (sem,LCB_F_RWSEM |LCB_F_WRITE ); }schedule ();lockevent_inc (rwsem_sleep_writer);set_current_state (state); -trylock_again:raw_spin_lock_irq (&sem->wait_lock); }__set_current_state (TASK_RUNNING );
Signature · 1 line
2.36.0
Hi Davidlohr,
Quoted 4 lines · Davidlohr Bueso
Similar to the mutex counterpart, we can further distinguish the types of contention. Similarly this patch also introduces potentially various
_begin()tracepoints with a single respective_end().
Thanks for doing this. I was thinking about it as well.
Quoted 38 lines
- The original
_begin()fordown_write()is moved further up, right after we know optimistic spinning is bust.- For the scenario that spins after setting the hand-off bit and failing to grab the lock the code is change to duplicate the
rwsem_try_write_lock()upon aOWNER_NULL, which minimizes the amounts of_begin()in the wait-loop - but also gets rid of a goto label and the logic is pretty encapsulated in the branch. In such cases the common case will be to acquire the lock, but if it is stolen in that window, this path won't see thesignal_pending()now in the iteration and block. This should be benign as the check will come in after waking if the lock can still not be taken.Signed-off-by Davidlohr Bueso <[email protected]>---
rtmutexes' top waiter will also do optimitic spinning, but I don't think it is worth adding tracepoints here as it is quite minimal, unlike the osq stuff.include/trace/events/lock.h | 4 +++- kernel/locking/rwsem.c | 29 +++++++++++++++++++++-------- 2 files changed, 24 insertions(+), 9 deletions(-)include/trace/events/lock.h +1
-0@@ -15,6 +15,7 @@#define LCB_F_RT (1U << 3)#define LCB_F_PERCPU (1U << 4)#define LCB_F_MUTEX (1U << 5) +#define LCB_F_RWSEM (1U << 6)#ifdef CONFIG_LOCKDEP
include/trace/events/lock.h TRACE_EVENT(contention_begin, · line 122
Quoted 8 lines
include/trace/events/lock.h +2
-1@@ -115,7 +116,8 @@ TRACE_EVENT(contention_begin, { LCB_F_WRITE, "WRITE" }, { LCB_F_RT, "RT" }, { LCB_F_PERCPU, "PERCPU" }, - { LCB_F_MUTEX, "MUTEX" } + { LCB_F_MUTEX, "MUTEX" }, + { LCB_F_RWSEM, "RWSEM" } )) );
Well I'm ok with this but it'd be better if we can do this without adding a new flag. Originally a mutex can be identified with 0, and a rwsem with either of READ or WRITE.
The MUTEX flag was added to note optimistic spins on mutex and now we need something similar for rwsem. Then can we change the MUTEX to OPTIMISTIC if it's not too late?
Quoted 59 lines
kernel/locking/rwsem.c +13
-5@@ -1057,7 +1057,7 @@ rwsem_down_read_slowpath(struct rw_semaphore *sem, long count, unsigned int statif (!wake_q_empty (&wake_q))wake_up_q (&wake_q); -trace_contention_begin (sem,LCB_F_READ ); +trace_contention_begin (sem,LCB_F_RWSEM |LCB_F_READ );/* wait to be given the lock */ for (;;) {@@ -1101,9 +1101,14 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)DEFINE_WAKE_Q (wake_q);/* do optimistic spinning and steal lock if possible */ -if (rwsem_can_spin_on_owner (sem) &&rwsem_optimistic_spin (sem)) { -/* rwsem_optimistic_spin() implies ACQUIRE on success */ -return sem; +if (rwsem_can_spin_on_owner (sem)) { +trace_contention_begin (sem, +LCB_F_RWSEM |LCB_F_WRITE |LCB_F_SPIN ); +if (rwsem_optimistic_spin (sem)) { +/* rwsem_optimistic_spin() implies ACQUIRE on success */ +trace_contention_end (sem,0 ); +return sem; + } }/* @@ -1115,6 +1120,8 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) waiter.timeout = jiffies +RWSEM_WAIT_TIMEOUT ; waiter.handoff_set = false; +trace_contention_begin (sem,LCB_F_RWSEM |LCB_F_WRITE ); +raw_spin_lock_irq (&sem->wait_lock);rwsem_add_waiter (sem, &waiter);@@ -1137,7 +1144,6 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)/* wait until we successfully acquire the lock */ set_current_state (state); -trace_contention_begin (sem,LCB_F_WRITE );for (;;) {if (rwsem_try_write_lock (sem, &waiter)) {@@ -1161,18 +1167,25 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)if (waiter.handoff_set) {enum owner_state owner_state; +trace_contention_begin (sem,LCB_F_RWSEM | +LCB_F_WRITE |LCB_F_SPIN );preempt_disable (); owner_state =rwsem_spin_on_owner (sem);preempt_enable ();
kernel/locking/rwsem.c rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) · line 1183
Quoted 10 lines
kernel/locking/rwsem.c +8
-2-if (owner_state ==OWNER_NULL ) - goto trylock_again; +if (owner_state ==OWNER_NULL ) { +raw_spin_lock_irq (&sem->wait_lock); +if (rwsem_try_write_lock (sem, &waiter)) +break ; +raw_spin_unlock_irq (&sem->wait_lock); + } + +trace_contention_begin (sem,LCB_F_RWSEM |LCB_F_WRITE );
I'm afraid that it'd generate many contention_begin trace events for a single lock acquisition.
Thanks,
Namhyung
Quoted 12 lines
kernel/locking/rwsem.c +0
-2}schedule ();lockevent_inc (rwsem_sleep_writer);set_current_state (state); -trylock_again:raw_spin_lock_irq (&sem->wait_lock); }__set_current_state (TASK_RUNNING ); --2.36.0
Sorry for the late reply.
Quoted 9 lines · Namhyung Kim
Hi Davidlohr,
Quoted 4 lines · Davidlohr Bueso
Similar to the mutex counterpart, we can further distinguish the types of contention. Similarly this patch also introduces potentially various
_begin()tracepoints with a single respective_end().Thanks for doing this. I was thinking about it as well.
I really like your work on this. I've always wanted a low overhead equivalent-ish of lock_stat, which could be used in production and look forward to see these tracepoints put to good use.
Quoted 14 lines
include/trace/events/lock.h TRACE_EVENT(contention_begin, · line 122
Quoted 9 lines
include/trace/events/lock.h +2
-1@@ -115,7 +116,8 @@ TRACE_EVENT(contention_begin, { LCB_F_WRITE, "WRITE" }, { LCB_F_RT, "RT" }, { LCB_F_PERCPU, "PERCPU" }, - { LCB_F_MUTEX, "MUTEX" } + { LCB_F_MUTEX, "MUTEX" }, + { LCB_F_RWSEM, "RWSEM" } )) );Well I'm ok with this but it'd be better if we can do this without adding a new flag. Originally a mutex can be identified with 0, and a rwsem with either of READ or WRITE.
Quoted 4 lines
The MUTEX flag was added to note optimistic spins on mutex and now we need something similar for rwsem. Then can we change the MUTEX to OPTIMISTIC if it's not too late?
Ok. Perhaps name it OSQ? I had thought of that but at the time was also thinking about potentially the rtmutex and rt spinlock spinning too - which don't use osq so the name would fit in that sense.
While not in Linus' tree, I wouldn't think it's too late.
Quoted 12 lines
Quoted 12 lines
(quoted diff) +0
-0for (;;) {if (rwsem_try_write_lock(sem, &waiter)) {kernel/locking/rwsem.c +2
-0@@ -1161,18 +1167,25 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)if (waiter.handoff_set) {enum owner_state owner_state; +trace_contention_begin (sem,LCB_F_RWSEM | +LCB_F_WRITE |LCB_F_SPIN );preempt_disable (); owner_state =rwsem_spin_on_owner (sem);preempt_enable ();
Quoted 13 lines
kernel/locking/rwsem.c rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) · line 1183
Quoted 10 lines
kernel/locking/rwsem.c +8
-2-if (owner_state ==OWNER_NULL ) - goto trylock_again; +if (owner_state ==OWNER_NULL ) { +raw_spin_lock_irq (&sem->wait_lock); +if (rwsem_try_write_lock (sem, &waiter)) +break ; +raw_spin_unlock_irq (&sem->wait_lock); + } + +trace_contention_begin (sem,LCB_F_RWSEM |LCB_F_WRITE );I'm afraid that it'd generate many
contention_begintrace events for a single lock acquisition.
You are right, lets just trace the "normal" optimistic spinning at the start of the write slowpath.
Thanks,
Davidlohr
Hi Davidlohr,
Quoted 38 lines · Davidlohr Bueso
Sorry for the late reply.
Quoted 9 lines · Namhyung Kim
Hi Davidlohr,
Quoted 4 lines · Davidlohr Bueso
Similar to the mutex counterpart, we can further distinguish the types of contention. Similarly this patch also introduces potentially various
_begin()tracepoints with a single respective_end().Thanks for doing this. I was thinking about it as well.
I really like your work on this. I've always wanted a low overhead equivalent-ish of
lock_stat, which could be used in production and look forward to see these tracepoints put to good use.Quoted 14 lines
include/trace/events/lock.h TRACE_EVENT(contention_begin, · line 122
Quoted 9 lines
include/trace/events/lock.h +2
-1@@ -115,7 +116,8 @@ TRACE_EVENT(contention_begin, { LCB_F_WRITE, "WRITE" }, { LCB_F_RT, "RT" }, { LCB_F_PERCPU, "PERCPU" }, - { LCB_F_MUTEX, "MUTEX" } + { LCB_F_MUTEX, "MUTEX" }, + { LCB_F_RWSEM, "RWSEM" } )) );Well I'm ok with this but it'd be better if we can do this without adding a new flag. Originally a mutex can be identified with 0, and a rwsem with either of READ or WRITE.
Quoted 4 lines
The MUTEX flag was added to note optimistic spins on mutex and now we need something similar for rwsem. Then can we change the MUTEX to OPTIMISTIC if it's not too late?
Quoted 6 lines
Ok. Perhaps name it OSQ? I had thought of that but at the time was also thinking about potentially the rtmutex and rt spinlock spinning too - which don't use osq so the name would fit in that sense.
While not in Linus' tree, I wouldn't think it's too late.
Any updates? It's now in Linus' tree so we should change this before the official release. Or we can keep the current flags and then add one like in your original code.
Quoted 27 lines
Quoted 13 lines
Quoted 13 lines
(quoted diff) +0
-0for (;;) {if (rwsem_try_write_lock(sem, &waiter)) {kernel/locking/rwsem.c +2
-0@@ -1161,18 +1167,25 @@ rwsem_down_write_slowpath(struct rw_semaphore *sem, int state)if (waiter.handoff_set) {enum owner_state owner_state; +trace_contention_begin (sem,LCB_F_RWSEM | +LCB_F_WRITE |LCB_F_SPIN );preempt_disable (); owner_state =rwsem_spin_on_owner (sem);preempt_enable ();Quoted 13 lines
kernel/locking/rwsem.c rwsem_down_write_slowpath(struct rw_semaphore *sem, int state) · line 1183
Quoted 10 lines
kernel/locking/rwsem.c +8
-2-if (owner_state ==OWNER_NULL ) - goto trylock_again; +if (owner_state ==OWNER_NULL ) { +raw_spin_lock_irq (&sem->wait_lock); +if (rwsem_try_write_lock (sem, &waiter)) +break ; +raw_spin_unlock_irq (&sem->wait_lock); + } + +trace_contention_begin (sem,LCB_F_RWSEM |LCB_F_WRITE );I'm afraid that it'd generate many
contention_begintrace events for a single lock acquisition.
Quoted 2 lines
You are right, lets just trace the "normal" optimistic spinning at the start of the write slowpath.
I have to admit that I overlooked the mutex code already has the same logic. I still prefer having less number of events but you might want to have the same with the mutex for the precise tracking of the spins.
Thanks,
Namhyung