locking/rwsem: Rwsem rearchitecture part 2 follow-up patches
Waiman Long
reviewercover locking/rwsem: Rwsem rearchitecture part 2 follow-up patches
This series contain 2 follow-up patches to alleviate the performance regression found in the page_fault1 test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.
The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.
Waiman Long (2):
locking/rwsem: Clarify usage of owner's nonspinaable bit locking/rwsem: Adaptive disabling of reader optimistic spinning
include/linux/rwsem.h | 2 +- kernel/locking/lock_events_list.h | 9 +- kernel/locking/rwsem.c | 178 +++++++++++++++++++++--------- 3 files changed, 131 insertions(+), 58 deletions(-)
Signature · 1 line
2.18.1
Waiman Long
reviewerPATCH 1/2 locking/rwsem: Clarify usage of owner's nonspinaable bit
Bit 1 of sem->owner was previously used to designate an anonymous owner - reader or anonymous writer. With the reader optimistic spinning patches, bit 1 is now used to indicate that optimistic spinning should be disabled. So change RWSEM_ANONYMOUSLY_OWNED to RWSEM_NONSPINNABLE to clarify its current meaning.
Since we leave sem->owner unchanged when the reader unlocks, the nonspinnable bit together with count's RWSEM_FLAG_WAITERS bit will act like the RWSEM_FLAG_HANDOFF bit. This is too draconian for its purpose.
For fairness, we want a writer to acquire the lock after the readers hold the lock for a relatively long time. In order to give preference to writers under such a circumstance, the single RWSEM_NONSPINNABLE bit is now split into two - one for reader and one for writer. When optimistic spinning is disabled, both bits will be set. When the reader count drop down to 0, the writer nonspinnable bit will be cleared to allow writers to spin on the lock, but not the readers. When a writer acquires the lock, it will write its own task structure pointer into sem->owner and clear the reader nonspinnable bit in the process.
--- include/linux/rwsem.h | 2 +- kernel/locking/rwsem.c | 123 ++++++++++++++++++++++++----------------- 2 files changed, 74 insertions(+), 51 deletions(-)
include/linux/rwsem.h +1 -1
@@ -50,7 +50,7 @@ struct rw_semaphore { };/* -* Setting bit 1 of the owner field but not bit 0 will indicate +* Setting all bits of the owner field except bit 0 will indicate * that the rwsem is writer-owned with an unknown owner. */ #define RWSEM_OWNER_UNKNOWN ((struct task_struct *)-2L)
kernel/locking/rwsem.c +73 -50
289 lines not on this page · raw message
Signature · 1 line
2.18.1
Waiman Long
reviewerPATCH 2/2 locking/rwsem: Adaptive disabling of reader optimistic spinning
In reply to Waiman LongReader optimistic spinning is helpful when the reader critical section is short and there aren't that many readers around. It makes readers relatively more preferred than writers. When a writer times out spinning on a reader-owned lock and set the nospinnable bits, there are two main reasons for that.
1) The reader critical section is long, perhaps the task sleeps after acquiring the read lock.
2) There are just too many readers contending the lock causing it to take a while to service all of them.
In the former case, long reader critical section will impede the progress of writers which is usually more important for system performance. In the later case, reader optimistic spinning tends to make the reader groups that contain readers that acquire the lock together smaller leading to more of them. That may hurt performance in some cases. In other words, the setting of nonspinnable bits indicates that reader optimistic spinning may not be helpful for those workloads that cause it.
Therefore, any writers that had observed the setting of the writer nonspinnable bit for a given rwsem after they fail to acquire the lock via optimistic spinning will set the reader nonspinnable bit once they acquire the write lock. This is to discourage reader optmistic spinning on that particular rwsem and make writers more preferred. This adaptive disabling of reader optimistic spinning will alleviate some of the negative side effect of this feature.
On a 2-socket 40-core 80-thread Skylake system, the page_fault1 test of the will-it-scale benchmark was run with various number of threads. The number of operations done before and after the patch were:
Threads Before patch After patch % change
------- ------------ ----------- --------
20 5409075 5436456 +0.5%
40 7174080 7903845 +10.2%
60 6749707 7009784 +3.9%
80 7071334 7353806 +4.0%This doesn't recover all the lost performance, but is close to half. Given the fact that reader optimistic spinning does benefit some workloads, this is a good compromise.
Using the rwsem locking microbenchmark with very short critical section, this patch also helps performance at high contention level as shown by the locking rates (kops/s) below with equal numbers of readers and writers before and after this patch:
# of Threads Pre-patch Post-patch
------------ --------- ----------
2 4,472 4,839
4 4,623 4,143
8 4,764 4,126
16 4,678 3,873
32 2,847 3,263
64 2,478 3,121
80 2,222 3,104--- kernel/locking/lock_events_list.h | 9 ++--- kernel/locking/rwsem.c | 55 +++++++++++++++++++++++++++++-- 2 files changed, 57 insertions(+), 7 deletions(-)
kernel/locking/lock_events_list.h +5 -4
@@ -56,10 +56,11 @@ LOCK_EVENT(rwsem_sleep_reader) /* # of reader sleeps */ LOCK_EVENT(rwsem_sleep_writer) /* # of writer sleeps */ LOCK_EVENT(rwsem_wake_reader) /* # of reader wakeups */ LOCK_EVENT(rwsem_wake_writer) /* # of writer wakeups */ -LOCK_EVENT(rwsem_opt_rlock) /* # of read locks opt-spin acquired */ -LOCK_EVENT(rwsem_opt_wlock) /* # of write locks opt-spin acquired */ -LOCK_EVENT(rwsem_opt_fail) /* # of failed opt-spinnings */ -LOCK_EVENT(rwsem_opt_nospin) /* # of disabled reader opt-spinnings */ +LOCK_EVENT(rwsem_opt_rlock) /* # of opt-acquired read locks */ +LOCK_EVENT(rwsem_opt_wlock) /* # of opt-acquired write locks */ +LOCK_EVENT(rwsem_opt_fail) /* # of failed optspins */ +LOCK_EVENT(rwsem_opt_nospin) /* # of disabled optspins */ +LOCK_EVENT(rwsem_opt_norspin) /* # of disabled reader-only optspins */ LOCK_EVENT(rwsem_rlock) /* # of read locks acquired */ LOCK_EVENT(rwsem_rlock_fast) /* # of fast read locks acquired */ LOCK_EVENT(rwsem_rlock_fail) /* # of failed read lock acquisitions */
kernel/locking/rwsem.c +52 -3
99 lines not on this page · raw message
Signature · 1 line
2.18.1
Peter Zijlstra
maintainerQuoted 9 lines · Waiman Long
This series contain 2 follow-up patches to alleviate the performance regression found in the
page_fault1test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.
Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.
Waiman Long
reviewerQuoted 6 lines · Peter Zijlstra
Quoted 5 lines · Waiman Long
This series contain 2 follow-up patches to alleviate the performance regression found in the
page_fault1test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.
Quoted 6 lines
Quoted 4 lines
The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.
Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.
I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.
Cheers,
Longman
Peter Zijlstra
maintainerQuoted 11 lines · Waiman Long
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long
This series contain 2 follow-up patches to alleviate the performance regression found in the
page_fault1test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.
Quoted 6 lines
Quoted 2 lines
Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.
I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.
Hmm, in that case I can fold it back too. So hold off on sending it.
I thought #2 was a fixup for an earlier patch as well.
Waiman Long
reviewerQuoted 12 lines · Peter Zijlstra
Quoted 11 lines · Waiman Long
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long
This series contain 2 follow-up patches to alleviate the performance regression found in the
page_fault1test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.
I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.
Hmm, in that case I can fold it back too. So hold off on sending it.
I thought #2 was a fixup for an earlier patch as well.
#2 is a performance fix.
Cheers,
Longman
Peter Zijlstra
maintainerQuoted 15 lines · Waiman Long
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Waiman Long
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long
This series contain 2 follow-up patches to alleviate the performance regression found in the
page_fault1test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.
Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.
Hmm, in that case I can fold it back too. So hold off on sending it.
I thought #2 was a fixup for an earlier patch as well.
#2 is a performance fix.
Of this patch?
206038 N T Apr 13 Waiman Long (7.5K) ├─>[PATCH v4 11/16] locking/rwsem: Enable readers spinning on writer
Fixes should have a Fixes: tag. And if the patch it fixes isn't a commit yet, the patch should be refreshed to not need a fix.
Waiman Long
reviewerQuoted 21 lines · Peter Zijlstra
Quoted 20 lines · Waiman Long
Quoted 19 lines · Peter Zijlstra
Quoted 11 lines · Waiman Long
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long
This series contain 2 follow-up patches to alleviate the performance regression found in the
page_fault1test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.
Quoted 5 lines
Quoted 2 lines
Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.
I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.
Hmm, in that case I can fold it back too. So hold off on sending it.
Quoted 8 lines
Quoted 2 lines
Quoted 1 line
I thought #2 was a fixup for an earlier patch as well.
#2 is a performance fix.
Of this patch?
206038 N T Apr 13 Waiman Long (7.5K) ├─>[PATCH v4 11/16] locking/rwsem: Enable readers spinning on writer
Fixes should have a Fixes: tag. And if the patch it fixes isn't a commit yet, the patch should be refreshed to not need a fix.
The original patch isn't wrong. This patch just introduce another idea to make it better. That is why I would still like to separate it as a distinct patch.
Cheers,
Longman
Quoted 26 lines · Waiman Long
Quoted 25 lines · Peter Zijlstra
Quoted 15 lines · Waiman Long
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Waiman Long
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long
This series contain 2 follow-up patches to alleviate the performance regression found in the
page_fault1test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.
Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.
Hmm, in that case I can fold it back too. So hold off on sending it.
I thought #2 was a fixup for an earlier patch as well.
#2 is a performance fix.
Of this patch?
Quoted 8 lines
Quoted 4 lines
206038 N T Apr 13 Waiman Long (7.5K) ├─>[PATCH v4 11/16] locking/rwsem: Enable readers spinning on writer
Fixes should have a Fixes: tag. And if the patch it fixes isn't a commit yet, the patch should be refreshed to not need a fix.
The original patch isn't wrong. This patch just introduce another idea to make it better. That is why I would still like to separate it as a distinct patch.
Yeah, I think it's better to have it in two separate patches. Basically patch #1 has a downside for certain workloads, which the heuristics in patch #2 improve. That's the only connection between the two patches.
If we find some other worst-case workload then the split of the commits would allow more finegrained examination of the effects of these performance tunings.
Thanks,
Ingo
Waiman Long
reviewerQuoted 39 lines · Ingo Molnar
Quoted 26 lines · Waiman Long
Quoted 25 lines · Peter Zijlstra
Quoted 15 lines · Waiman Long
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Waiman Long
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long
This series contain 2 follow-up patches to alleviate the performance regression found in the
page_fault1test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.
Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.
Quoted 7 lines
Quoted 6 lines
Quoted 3 lines
I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.
Hmm, in that case I can fold it back too. So hold off on sending it.
I thought #2 was a fixup for an earlier patch as well.
#2 is a performance fix.
Of this patch?
Quoted 7 lines
Quoted 4 lines
206038 N T Apr 13 Waiman Long (7.5K) ├─>[PATCH v4 11/16] locking/rwsem: Enable readers spinning on writer
Fixes should have a Fixes: tag. And if the patch it fixes isn't a commit yet, the patch should be refreshed to not need a fix.
The original patch isn't wrong. This patch just introduce another idea to make it better. That is why I would still like to separate it as a distinct patch.
Yeah, I think it's better to have it in two separate patches. Basically patch #1 has a downside for certain workloads, which the heuristics in patch #2 improve. That's the only connection between the two patches.
Quoted 7 lines
If we find some other worst-case workload then the split of the commits would allow more finegrained examination of the effects of these performance tunings.
Thanks,
Ingo
As the part2 patches are still being actively modified, it doesn't look like it will make the next merge window. I am fine with postponing it to 5.3. However, I would like to request the merging of just patch 1 of the part 2 patchset. It fixes a locking selftest problem that was introduced in the part 1 patches.
Thanks,
Longman
Quoted 5 lines · Waiman Long
As the part2 patches are still being actively modified, it doesn't look like it will make the next merge window. I am fine with postponing it to 5.3. However, I would like to request the merging of just patch 1 of the part 2 patchset. It fixes a locking selftest problem that was introduced in the part 1 patches.
Sure, I've promoted this commit:
26536e7c242e: locking/rwsem: Prevent unneeded warning during locking selftest
from tip:WIP.locking/core into tip:locking/core, for a v5.2 merge.
Thanks,
Ingo
Waiman Long
reviewerQuoted 9 lines · Ingo Molnar
Quoted 5 lines · Waiman Long
As the part2 patches are still being actively modified, it doesn't look like it will make the next merge window. I am fine with postponing it to 5.3. However, I would like to request the merging of just patch 1 of the part 2 patchset. It fixes a locking selftest problem that was introduced in the part 1 patches.
Sure, I've promoted this commit:
Quoted 7 lines
26536e7c242e: locking/rwsem: Prevent unneeded warning during locking selftestfrom tip:WIP.locking/core into tip:locking/core, for a v5.2 merge.
Thanks,
Ingo
Thanks for the help. I just don't want to get report of known problem during the next merge window.
Cheers,
Longman