Skip to contentSkip to conversation

locking/rwsem: Rwsem rearchitecture part 2 follow-up patches

PATCH In review Waiman Long 3 participants 13 messages Apr 15, 2019 – Apr 18, 2019 Locking primitives

lore
Share & tools
lore

Waiman Long

author reviewer
#0 lore raw

cover locking/rwsem: Rwsem rearchitecture part 2 follow-up patches

This series contain 2 follow-up patches to alleviate the performance regression found in the page_fault1 test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.

The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.

Waiman Long (2):
locking/rwsem: Clarify usage of owner's nonspinaable bit locking/rwsem: Adaptive disabling of reader optimistic spinning

 include/linux/rwsem.h             |   2 +-
 kernel/locking/lock_events_list.h |   9 +-
 kernel/locking/rwsem.c            | 178 +++++++++++++++++++++---------
 3 files changed, 131 insertions(+), 58 deletions(-)
Signature · 1 line
2.18.1
author reviewer
#1 lore raw

PATCH 1/2 locking/rwsem: Clarify usage of owner's nonspinaable bit

Bit 1 of sem->owner was previously used to designate an anonymous owner - reader or anonymous writer. With the reader optimistic spinning patches, bit 1 is now used to indicate that optimistic spinning should be disabled. So change RWSEM_ANONYMOUSLY_OWNED to RWSEM_NONSPINNABLE to clarify its current meaning.

Since we leave sem->owner unchanged when the reader unlocks, the nonspinnable bit together with count's RWSEM_FLAG_WAITERS bit will act like the RWSEM_FLAG_HANDOFF bit. This is too draconian for its purpose.

For fairness, we want a writer to acquire the lock after the readers hold the lock for a relatively long time. In order to give preference to writers under such a circumstance, the single RWSEM_NONSPINNABLE bit is now split into two - one for reader and one for writer. When optimistic spinning is disabled, both bits will be set. When the reader count drop down to 0, the writer nonspinnable bit will be cleared to allow writers to spin on the lock, but not the readers. When a writer acquires the lock, it will write its own task structure pointer into sem->owner and clear the reader nonspinnable bit in the process.

---
 include/linux/rwsem.h  |   2 +-
 kernel/locking/rwsem.c | 123 ++++++++++++++++++++++++-----------------
 2 files changed, 74 insertions(+), 51 deletions(-)
include/linux/rwsem.h +1 -1
index 148983e21d47..bb76e82398b2 100644
@@ -50,7 +50,7 @@ struct rw_semaphore {
 };
 
 /*
- * Setting bit 1 of the owner field but not bit 0 will indicate
+ * Setting all bits of the owner field except bit 0 will indicate
  * that the rwsem is writer-owned with an unknown owner.
  */
 #define RWSEM_OWNER_UNKNOWN	((struct task_struct *)-2L)
kernel/locking/rwsem.c +73 -50

289 lines not on this page · raw message

Signature · 1 line
2.18.1
author reviewer
#2 lore raw

PATCH 2/2 locking/rwsem: Adaptive disabling of reader optimistic spinning

In reply to Waiman Long

Reader optimistic spinning is helpful when the reader critical section is short and there aren't that many readers around. It makes readers relatively more preferred than writers. When a writer times out spinning on a reader-owned lock and set the nospinnable bits, there are two main reasons for that.

1) The reader critical section is long, perhaps the task sleeps after acquiring the read lock.
2) There are just too many readers contending the lock causing it to take a while to service all of them.

In the former case, long reader critical section will impede the progress of writers which is usually more important for system performance. In the later case, reader optimistic spinning tends to make the reader groups that contain readers that acquire the lock together smaller leading to more of them. That may hurt performance in some cases. In other words, the setting of nonspinnable bits indicates that reader optimistic spinning may not be helpful for those workloads that cause it.

Therefore, any writers that had observed the setting of the writer nonspinnable bit for a given rwsem after they fail to acquire the lock via optimistic spinning will set the reader nonspinnable bit once they acquire the write lock. This is to discourage reader optmistic spinning on that particular rwsem and make writers more preferred. This adaptive disabling of reader optimistic spinning will alleviate some of the negative side effect of this feature.

On a 2-socket 40-core 80-thread Skylake system, the page_fault1 test of the will-it-scale benchmark was run with various number of threads. The number of operations done before and after the patch were:

  Threads   Before patch   After patch   % change
  -------   ------------   -----------   --------
    20        5409075        5436456       +0.5%
    40        7174080        7903845      +10.2%
    60        6749707        7009784       +3.9%
    80        7071334        7353806       +4.0%

This doesn't recover all the lost performance, but is close to half. Given the fact that reader optimistic spinning does benefit some workloads, this is a good compromise.

Using the rwsem locking microbenchmark with very short critical section, this patch also helps performance at high contention level as shown by the locking rates (kops/s) below with equal numbers of readers and writers before and after this patch:

   # of Threads  Pre-patch    Post-patch
   ------------  ---------    ----------
        2          4,472        4,839
        4          4,623        4,143
        8          4,764        4,126
       16          4,678        3,873
       32          2,847        3,263
       64          2,478        3,121
       80          2,222        3,104
---
 kernel/locking/lock_events_list.h |  9 ++---
 kernel/locking/rwsem.c            | 55 +++++++++++++++++++++++++++++--
 2 files changed, 57 insertions(+), 7 deletions(-)
kernel/locking/lock_events_list.h +5 -4
index f3550aa5866a..b0eeb77070dd 100644
@@ -56,10 +56,11 @@ LOCK_EVENT(rwsem_sleep_reader)	/* # of reader sleeps			*/
 LOCK_EVENT(rwsem_sleep_writer)	/* # of writer sleeps			*/
 LOCK_EVENT(rwsem_wake_reader)	/* # of reader wakeups			*/
 LOCK_EVENT(rwsem_wake_writer)	/* # of writer wakeups			*/
-LOCK_EVENT(rwsem_opt_rlock)	/* # of read locks opt-spin acquired	*/
-LOCK_EVENT(rwsem_opt_wlock)	/* # of write locks opt-spin acquired	*/
-LOCK_EVENT(rwsem_opt_fail)	/* # of failed opt-spinnings		*/
-LOCK_EVENT(rwsem_opt_nospin)	/* # of disabled reader opt-spinnings	*/
+LOCK_EVENT(rwsem_opt_rlock)	/* # of opt-acquired read locks		*/
+LOCK_EVENT(rwsem_opt_wlock)	/* # of opt-acquired write locks	*/
+LOCK_EVENT(rwsem_opt_fail)	/* # of failed optspins			*/
+LOCK_EVENT(rwsem_opt_nospin)	/* # of disabled optspins		*/
+LOCK_EVENT(rwsem_opt_norspin)	/* # of disabled reader-only optspins	*/
 LOCK_EVENT(rwsem_rlock)		/* # of read locks acquired		*/
 LOCK_EVENT(rwsem_rlock_fast)	/* # of fast read locks acquired	*/
 LOCK_EVENT(rwsem_rlock_fail)	/* # of failed read lock acquisitions	*/
kernel/locking/rwsem.c +52 -3

99 lines not on this page · raw message

Signature · 1 line
2.18.1
maintainer
#3 lore raw
In reply to Waiman Long
Quoted 9 lines · Waiman Long

This series contain 2 follow-up patches to alleviate the performance regression found in the page_fault1 test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.

The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.

Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.

author reviewer
#4 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 5 lines · Waiman Long

This series contain 2 follow-up patches to alleviate the performance regression found in the page_fault1 test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.

Quoted 6 lines
Quoted 4 lines

The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.

Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.

I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.

Cheers,
Longman

maintainer
#5 lore raw
Quoted 11 lines · Waiman Long
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long

This series contain 2 follow-up patches to alleviate the performance regression found in the page_fault1 test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.

The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.

Quoted 6 lines
Quoted 2 lines

Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.

I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.

Hmm, in that case I can fold it back too. So hold off on sending it.

I thought #2 was a fixup for an earlier patch as well.

author reviewer
#6 lore raw
Quoted 12 lines · Peter Zijlstra
Quoted 11 lines · Waiman Long
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long

This series contain 2 follow-up patches to alleviate the performance regression found in the page_fault1 test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.

The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.

I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.

Hmm, in that case I can fold it back too. So hold off on sending it.

I thought #2 was a fixup for an earlier patch as well.

#2 is a performance fix.

Cheers,
Longman

maintainer
#7 lore raw
Quoted 15 lines · Waiman Long
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Waiman Long
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long

This series contain 2 follow-up patches to alleviate the performance regression found in the page_fault1 test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.

The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.

Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.

Hmm, in that case I can fold it back too. So hold off on sending it.

I thought #2 was a fixup for an earlier patch as well.

#2 is a performance fix.

Of this patch?

206038 N T Apr 13 Waiman Long (7.5K) ├─>[PATCH v4 11/16] locking/rwsem: Enable readers spinning on writer

Fixes should have a Fixes: tag. And if the patch it fixes isn't a commit yet, the patch should be refreshed to not need a fix.

author reviewer
#8 lore raw
Quoted 21 lines · Peter Zijlstra
Quoted 20 lines · Waiman Long
Quoted 19 lines · Peter Zijlstra
Quoted 11 lines · Waiman Long
Quoted 10 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long

This series contain 2 follow-up patches to alleviate the performance regression found in the page_fault1 test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.

The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.

Quoted 5 lines
Quoted 2 lines

Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.

I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.

Hmm, in that case I can fold it back too. So hold off on sending it.

Quoted 8 lines
Quoted 2 lines
Quoted 1 line

I thought #2 was a fixup for an earlier patch as well.

#2 is a performance fix.

Of this patch?

206038 N T Apr 13 Waiman Long (7.5K) ├─>[PATCH v4 11/16] locking/rwsem: Enable readers spinning on writer

Fixes should have a Fixes: tag. And if the patch it fixes isn't a commit yet, the patch should be refreshed to not need a fix.

The original patch isn't wrong. This patch just introduce another idea to make it better. That is why I would still like to separate it as a distinct patch.

Cheers,
Longman

#9 lore raw
Quoted 26 lines · Waiman Long
Quoted 25 lines · Peter Zijlstra
Quoted 15 lines · Waiman Long
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Waiman Long
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long

This series contain 2 follow-up patches to alleviate the performance regression found in the page_fault1 test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.

The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.

Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.

Hmm, in that case I can fold it back too. So hold off on sending it.

I thought #2 was a fixup for an earlier patch as well.

#2 is a performance fix.

Of this patch?

Quoted 8 lines
Quoted 4 lines

206038 N T Apr 13 Waiman Long (7.5K) ├─>[PATCH v4 11/16] locking/rwsem: Enable readers spinning on writer

Fixes should have a Fixes: tag. And if the patch it fixes isn't a commit yet, the patch should be refreshed to not need a fix.

The original patch isn't wrong. This patch just introduce another idea to make it better. That is why I would still like to separate it as a distinct patch.

Yeah, I think it's better to have it in two separate patches. Basically patch #1 has a downside for certain workloads, which the heuristics in patch #2 improve. That's the only connection between the two patches.

If we find some other worst-case workload then the split of the commits would allow more finegrained examination of the effects of these performance tunings.

Thanks,

Ingo

author reviewer
#10 lore raw
Quoted 39 lines · Ingo Molnar
Quoted 26 lines · Waiman Long
Quoted 25 lines · Peter Zijlstra
Quoted 15 lines · Waiman Long
Quoted 14 lines · Peter Zijlstra
Quoted 13 lines · Waiman Long
Quoted 12 lines · Peter Zijlstra
Quoted 9 lines · Waiman Long

This series contain 2 follow-up patches to alleviate the performance regression found in the page_fault1 test of the will-it-scale benchmark. This does not recover all the lost performance, but reclaim a sizeable portion of it.

The regression was found on an Intel system. I have run the test on an AMD system. The regression wasn't seen there. There are only minor variations in performance. Perhaps the page fault path is quite different between Intel and AMD systems.

Can you please just fold this back into the appropriate patches? Trying to review all the back and forth is painful.

Quoted 7 lines
Quoted 6 lines
Quoted 3 lines

I will send out an update part 2 patch with patch 1 of this series merged into the writer spinning on reader patch. Patch 2 of this series will be a standalone one.

Hmm, in that case I can fold it back too. So hold off on sending it.

I thought #2 was a fixup for an earlier patch as well.

#2 is a performance fix.

Of this patch?

Quoted 7 lines
Quoted 4 lines

206038 N T Apr 13 Waiman Long (7.5K) ├─>[PATCH v4 11/16] locking/rwsem: Enable readers spinning on writer

Fixes should have a Fixes: tag. And if the patch it fixes isn't a commit yet, the patch should be refreshed to not need a fix.

The original patch isn't wrong. This patch just introduce another idea to make it better. That is why I would still like to separate it as a distinct patch.

Yeah, I think it's better to have it in two separate patches. Basically patch #1 has a downside for certain workloads, which the heuristics in patch #2 improve. That's the only connection between the two patches.

Quoted 7 lines

If we find some other worst-case workload then the split of the commits would allow more finegrained examination of the effects of these performance tunings.

Thanks,

Ingo

As the part2 patches are still being actively modified, it doesn't look like it will make the next merge window. I am fine with postponing it to 5.3. However, I would like to request the merging of just patch 1 of the part 2 patchset. It fixes a locking selftest problem that was introduced in the part 1 patches.

Thanks,
Longman

#11 lore raw
Quoted 5 lines · Waiman Long

As the part2 patches are still being actively modified, it doesn't look like it will make the next merge window. I am fine with postponing it to 5.3. However, I would like to request the merging of just patch 1 of the part 2 patchset. It fixes a locking selftest problem that was introduced in the part 1 patches.

Sure, I've promoted this commit:

26536e7c242e: locking/rwsem: Prevent unneeded warning during locking selftest

from tip:WIP.locking/core into tip:locking/core, for a v5.2 merge.

Thanks,

Ingo

author reviewer
#12 lore raw
Quoted 9 lines · Ingo Molnar
Quoted 5 lines · Waiman Long

As the part2 patches are still being actively modified, it doesn't look like it will make the next merge window. I am fine with postponing it to 5.3. However, I would like to request the merging of just patch 1 of the part 2 patchset. It fixes a locking selftest problem that was introduced in the part 1 patches.

Sure, I've promoted this commit:

Quoted 7 lines

26536e7c242e: locking/rwsem: Prevent unneeded warning during locking selftest

from tip:WIP.locking/core into tip:locking/core, for a v5.2 merge.

Thanks,

Ingo

Thanks for the help. I just don't want to get report of known problem during the next merge window.

Cheers,
Longman