locking/rwsem: reduce spinlock contention in wakeup after up_read/up_write
PATCH v2 1/1 locking/rwsem: reduce spinlock contention in wakeup after up_read/up_write
In up_read()/up_write(), rwsem_wake() will be called whenever it detects that some writers/readers are waiting. The rwsem_wake() function will take the wait_lock and call __rwsem_do_wake() to do the real wakeup. This can be a problem especially for up_read() where many readers can potentially call rwsem_wake() at more or less the same time even though a single call should be enough. This will cause contention in the wait_lock cacheline resulting in delay of the up_read/up_write operations.
This patch makes the wait_lock taking and the call to __rwsem_do_wake() optional if at least one spinning writer is present. The spinning writer will be able to take the rwsem and call rwsem_wake() later when it calls up_write(). With the presence of a spinning writer, rwsem_wake() will now try to acquire the lock using trylock. If that fails, it will just quit.
This patch also checks one more time to see if the rwsem was stolen just before doing the expensive wakeup operation which will be unnecessary if the lock was stolen.
On an 8-socket Westmere-EX server (80 cores, HT off), running AIM7's high_systime workload (1000 users) on a vanilla 4.0 kernel produced the following perf profile for spinlock contention:
9.23% reaim [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|--97.39%-- rwsem_wake
|--0.69%-- try_to_wake_up
|--0.52%-- release_pages
--1.40%-- [...] 1.70% reaim [kernel.kallsyms] [k] _raw_spin_lock_irq
|--96.61%-- rwsem_down_write_failed
|--2.03%-- __schedule
|--0.50%-- run_timer_softirq
--0.86%-- [...]With a patched 4.0 kernel, the perf profile became:
1.87% reaim [kernel.kallsyms] [k] _raw_spin_lock_irqsave
|--87.64%-- rwsem_wake
|--2.80%-- release_pages
|--2.56%-- try_to_wake_up
|--1.10%-- __wake_up
|--1.06%-- pagevec_lru_move_fn
|--0.93%-- prepare_to_wait_exclusive
|--0.71%-- free_pid
|--0.58%-- get_page_from_freelist
|--0.57%-- add_device_randomness
--2.04%-- [...] 0.80% reaim [kernel.kallsyms] [k] _raw_spin_lock_irq
|--92.49%-- rwsem_down_write_failed
|--4.24%-- __schedule
|--1.37%-- run_timer_softirq
--1.91%-- [...]The table below shows the % improvement in throughput (1100-2000 users) in the various AIM7's workloads:
Workload % increase in throughput
--- include/linux/osq_lock.h | 5 +++ kernel/locking/rwsem-xadd.c | 72 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 76 insertions(+), 1 deletions(-)
v1->v2:
- Add a memory barrier before calling spin_trylock for proper memory ordering.
include/linux/osq_lock.h +5 -0
@@ -32,4 +32,9 @@ static inline void osq_lock_init(struct optimistic_spin_queue *lock)extern bool osq_lock (struct optimistic_spin_queue *lock);extern void osq_unlock (struct optimistic_spin_queue *lock); +static inline bool osq_is_locked (struct optimistic_spin_queue *lock) +{ +return atomic_read (&lock->tail) !=OSQ_UNLOCKED_VAL ; +} +#endif
kernel/locking/rwsem-xadd.c +71 -1
90 lines not on this page · raw message
Signature · 1 line
1.7.1
Quoted 4 lines · Waiman Long
The table below shows the % improvement in throughput (1100-2000 users) in the various AIM7's workloads:
Workload % increase in throughput
Missing table here? :)
Quoted 8 lines
--- include/linux/osq_lock.h | 5 +++ kernel/locking/rwsem-xadd.c | 72 ++++++++++++++++++++++++++++++++++++++++++- 2 files changed, 76 insertions(+), 1 deletions(-)
v1->v2:
- Add a memory barrier before callingspin_trylockfor proper memory ordering.
Quoted 9 lines · Jason Low
Quoted 4 lines · Waiman Long
The table below shows the % improvement in throughput (1100-2000 users) in the various AIM7's workloads:
Workload % increase in throughput
Missing table here? :)
Oops, it was a cut-and-paste error. Will send out a new one to fix that.
Cheers,
Longman
It would be nice to not run into this by accident. Please CC all relevant parties ;)
Quoted 8 lines · Waiman Long
In up_read()/up_write(),
rwsem_wake()will be called whenever it detects that some writers/readers are waiting. Therwsem_wake()function will take thewait_lockand call__rwsem_do_wake()to do the real wakeup. This can be a problem especially forup_read()where many readers can potentially callrwsem_wake()at more or less the same time even though a single call should be enough. This will cause contention in thewait_lockcacheline resulting in delay of the up_read/up_write operations.
Ok.
Quoted 2 lines
This patch makes the
wait_locktaking and the call to__rwsem_do_wake()optional if at least one spinning writer is present.
But if the lock is taken by readers, like you suggest above, there cannot be any active spinners. We always block in these cases.
Thanks,
Davidlohr
Quoted 12 lines · Davidlohr Bueso
It would be nice to not run into this by accident. Please CC all relevant parties ;)
Quoted 8 lines · Waiman Long
In up_read()/up_write(),
rwsem_wake()will be called whenever it detects that some writers/readers are waiting. Therwsem_wake()function will take thewait_lockand call__rwsem_do_wake()to do the real wakeup. This can be a problem especially forup_read()where many readers can potentially callrwsem_wake()at more or less the same time even though a single call should be enough. This will cause contention in thewait_lockcacheline resulting in delay of the up_read/up_write operations.
Quoted 6 lines
Ok.
Quoted 2 lines
This patch makes the
wait_locktaking and the call to__rwsem_do_wake()optional if at least one spinning writer is present.But if the lock is taken by readers, like you suggest above, there cannot be any active spinners. We always block in these cases.
Yes, you are right. Will fix the log message.
Cheers,
Longman