Skip to contentSkip to conversation

rwsem: optimistic spinning causing performance degradation

regression, 3.16-rc Davidlohr Bueso 5 participants 16 messages Jul 3, 2014 – Jul 16, 2014 3 messages not in the archive Locking primitives

lore
Share & tools
lore
#2 lore raw
In reply to a message not in the archive

Adding lkml.

Quoted 43 lines · Davidlohr Bueso
Quoted 12 lines · Jason Low

Review

Quoted 10 lines · Dave Chinner

This is what the kernel profile looks like on the strided run:

(quoted diff) +0 -2
-  83.06%  [kernel]  [k] osq_lock
   - osq_lock
      - 100.00% rwsem_down_write_failed
         - call_rwsem_down_write_failed
            - 99.55% sys_mprotect
                 tracesys
                 __GI___mprotect
-  12.02%  [kernel]  [k] rwsem_down_write_failed
Quoted 5 lines

Hi Dave,

So with no sign of rwsem_spin_on_owner(), yet with such heavy contention in osq_lock, this makes me wonder if it's spending most of its time spinning on !owner while a reader has the lock? (We don't set sem->owner for the readers.)

That would explain the long hold times with the memory allocation patterns between read and write locking described by Dave.

Quoted 4 lines

If that's an issue, maybe the below is worth a test, in which we'll just avoid spinning if rwsem_can_spin_on_owner() finds that there is no owner. If we just had to enter the slowpath yet there is no owner, we'll be conservative and assume readers have the lock.

I do worry a bit about the effects here when this is not an issue. Workloads that have smaller hold times could very well take a performance hit by blocking right away instead of wasting a few extra cycles just spinning.

Quoted 2 lines

(David, you've tested something like this in the original patch with AIM7 and still got the big performance boosts right?)

I have not, but will. I wouldn't mind sacrificing a bit of the great performance numbers we're getting on workloads that mostly take the lock for writing, if it means not being so devastating for when readers are in the picture. This is a major difference with mutexes wrt optimistic spinning.

Thanks,
Davidlohr

Tested-by
#3 lore raw
In reply to a message not in the archive

[re-added lkml]

Quoted 14 lines · Jason Low

Review

Quoted 10 lines · Dave Chinner

This is what the kernel profile looks like on the strided run:

(quoted diff) +0 -2
-  83.06%  [kernel]  [k] osq_lock
   - osq_lock
      - 100.00% rwsem_down_write_failed
         - call_rwsem_down_write_failed
            - 99.55% sys_mprotect
                 tracesys
                 __GI___mprotect
-  12.02%  [kernel]  [k] rwsem_down_write_failed

Hi Dave,

Quoted 8 lines

So with no sign of rwsem_spin_on_owner(), yet with such heavy contention in osq_lock, this makes me wonder if it's spending most of its time spinning on !owner while a reader has the lock? (We don't set sem->owner for the readers.)

If that's an issue, maybe the below is worth a test, in which we'll just avoid spinning if rwsem_can_spin_on_owner() finds that there is no owner. If we just had to enter the slowpath yet there is no owner, we'll be conservative and assume readers have the lock.

That makes it quite a bit faster:

        XFS_REPAIR Summary    Fri Jul  4 10:39:32 2014
Phase           Start           End             Duration
Phase 1:        07/04 10:38:04  07/04 10:38:05  1 second
Phase 2:        07/04 10:38:05  07/04 10:38:08  3 seconds
Phase 3:        07/04 10:38:08  07/04 10:39:12  1 minute, 4 seconds
Phase 4:        07/04 10:39:12  07/04 10:39:21  9 seconds
Phase 5:        07/04 10:39:21  07/04 10:39:22  1 second
Phase 6:        07/04 10:39:22  07/04 10:39:30  8 seconds
Phase 7:        07/04 10:39:30  07/04 10:39:30  

Total run time: 1 minute, 26 seconds
done

real    1m28.504s
user    1m23.990s
sys     3m20.132s

So system time goes down massively, and speed comes up to within 30% of the single AG run. But it's still 2-3000 IOPS down, but it's acceptible for the moment.

FWIW, the kernel profile ifor the multi-AG run now looks like:

29.64% [kernel] [k] _raw_spin_unlock_irq
- _raw_spin_unlock_irq
+ 35.34% __schedule
- 34.15% call_rwsem_down_write_failed
+ 99.27% sys_mprotect
- 30.02% call_rwsem_down_read_failed 95.59% __do_page_fault
- 24.65% [kernel] [k] _raw_spin_unlock_irqrestore
- _raw_spin_unlock_irqrestore
- 69.38% rwsem_wake
- call_rwsem_wake
- 83.32% sys_mprotect
+ 15.54% __do_page_fault
+ 22.55% try_to_wake_up

+   9.77%  [kernel]  [k] default_send_IPI_mask_sequence_phys
-   3.21%  [kernel]  [k] smp_call_function_many

- smp_call_function_many
- 99.22% flush_tlb_page
+ 2.51% [kernel] [k] rwsem_down_write_failed

It's much more like the 3.15 profile - it's only wasting half the CPU spinning on the internal spinlock and it's now going fast enough to be blowing another 10-12% of the CPU time sending tlb flushes to other CPUs....

One thing I did notice, even with the single-AG-at-a-time run, is that the system time is *significantly* reduced with this patch, even though it doesn't change performance.

ie unpatched:

unpatched patched

runtime   0m58s		  1m2s
systime   4m55s		  1m1s

So not spinning when there are read holders is a major win even when there are few threads contending on read/write.

FWIW, the rwsems in the struct xfs_inode are often heavily read/write contended, so there are lots of IO related workloads that are going to regress on XFS without this optimisation...

Anyway, consider the patch:

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#4 lore raw
Quoted 90 lines · Dave Chinner

[re-added lkml]

Quoted 14 lines · Jason Low

Review

Quoted 10 lines · Dave Chinner

This is what the kernel profile looks like on the strided run:

(quoted diff) +0 -2
-  83.06%  [kernel]  [k] osq_lock
   - osq_lock
      - 100.00% rwsem_down_write_failed
         - call_rwsem_down_write_failed
            - 99.55% sys_mprotect
                 tracesys
                 __GI___mprotect
-  12.02%  [kernel]  [k] rwsem_down_write_failed

Hi Dave,

Quoted 8 lines

So with no sign of rwsem_spin_on_owner(), yet with such heavy contention in osq_lock, this makes me wonder if it's spending most of its time spinning on !owner while a reader has the lock? (We don't set sem->owner for the readers.)

If that's an issue, maybe the below is worth a test, in which we'll just avoid spinning if rwsem_can_spin_on_owner() finds that there is no owner. If we just had to enter the slowpath yet there is no owner, we'll be conservative and assume readers have the lock.

That makes it quite a bit faster:

        XFS_REPAIR Summary    Fri Jul  4 10:39:32 2014
Phase           Start           End             Duration
Phase 1:        07/04 10:38:04  07/04 10:38:05  1 second
Phase 2:        07/04 10:38:05  07/04 10:38:08  3 seconds
Phase 3:        07/04 10:38:08  07/04 10:39:12  1 minute, 4 seconds
Phase 4:        07/04 10:39:12  07/04 10:39:21  9 seconds
Phase 5:        07/04 10:39:21  07/04 10:39:22  1 second
Phase 6:        07/04 10:39:22  07/04 10:39:30  8 seconds
Phase 7:        07/04 10:39:30  07/04 10:39:30  

Total run time: 1 minute, 26 seconds
done

real    1m28.504s
user    1m23.990s
sys     3m20.132s

So system time goes down massively, and speed comes up to within 30% of the single AG run. But it's still 2-3000 IOPS down, but it's acceptible for the moment.

FWIW, the kernel profile ifor the multi-AG run now looks like:

29.64% [kernel] [k] _raw_spin_unlock_irq
- _raw_spin_unlock_irq
+ 35.34% __schedule
- 34.15% call_rwsem_down_write_failed
+ 99.27% sys_mprotect
- 30.02% call_rwsem_down_read_failed
95.59% __do_page_fault

(quoted diff) +2 -2
-  24.65%  [kernel]  [k] _raw_spin_unlock_irqrestore
   - _raw_spin_unlock_irqrestore
      - 69.38% rwsem_wake
         - call_rwsem_wake
            - 83.32% sys_mprotect
            + 15.54% __do_page_fault
      + 22.55% try_to_wake_up
+   9.77%  [kernel]  [k] default_send_IPI_mask_sequence_phys
-   3.21%  [kernel]  [k] smp_call_function_many
   - smp_call_function_many
      - 99.22% flush_tlb_page
+   2.51%  [kernel]  [k] rwsem_down_write_failed

It's much more like the 3.15 profile - it's only wasting half the CPU spinning on the internal spinlock and it's now going fast enough to be blowing another 10-12% of the CPU time sending tlb flushes to other CPUs....

One thing I did notice, even with the single-AG-at-a-time run, is that the system time is *significantly* reduced with this patch, even though it doesn't change performance.

ie unpatched:

unpatched patched

runtime   0m58s		  1m2s
systime   4m55s		  1m1s

So not spinning when there are read holders is a major win even when there are few threads contending on read/write.

Quoted 7 lines

FWIW, the rwsems in the struct xfs_inode are often heavily read/write contended, so there are lots of IO related workloads that are going to regress on XFS without this optimisation...

Anyway, consider the patch:

Hi Dave,

Thanks for testing. I'll update the patch with an actual changelog.

Tested-by Dave Chinner
#5 lore raw
Quoted 6 lines · Jason Low
Quoted 4 lines · Dave Chinner

FWIW, the rwsems in the struct xfs_inode are often heavily read/write contended, so there are lots of IO related workloads that are going to regress on XFS without this optimisation...

Quoted 7 lines
Quoted 3 lines

Anyway, consider the patch:

Hi Dave,

Thanks for testing. I'll update the patch with an actual changelog.

---
Subject: [PATCH] rwsem: In rwsem_can_spin_on_owner(), return false if no owner

It was found that the rwsem optimistic spinning feature can potentially degrade performance when there are readers. Perf profiles indicate in some workloads that significant time can be spent spinning on !owner. This is because we don't set the lock owner when readers(s) obtain the rwsem.

In this patch, we'll modify rwsem_can_spin_on_owner() such that we'll return false if there is no lock owner. The rationale is that if we just entered the slowpath, yet there is no lock owner, then there is a possibility that a reader has the lock. To be conservative, we'll avoid spinning in these situations.

Dave Chinner found performance benefits with this patch in the xfs_repair workload, where the total run time went from approximately 4 minutes 24 seconds, down to approximately 1 minute 26 seconds with the patch.

---
 kernel/locking/rwsem-xadd.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/locking/rwsem-xadd.c +5 -5
index dacc321..c40c7d2 100644
@@ -285,10 +285,10 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)
 static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
 {
 	struct task_struct *owner;
-	bool on_cpu = true;
+	bool on_cpu = false;
 
 	if (need_resched())
-		return 0;
+		return false;
 
 	rcu_read_lock();
 	owner = ACCESS_ONCE(sem->owner);
@@ -297,9 +297,9 @@ static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
 	rcu_read_unlock();
 
 	/*
-	 * If sem->owner is not set, the rwsem owner may have
-	 * just acquired it and not set the owner yet or the rwsem
-	 * has been released.
+	 * If sem->owner is not set, yet we have just recently entered the
+	 * slowpath, then there is a possibility reader(s) may have the lock.
+	 * To be safe, avoid spinning in these situations.
 	 */
 	return on_cpu;
 }
Signature · 1 line
1.7.9.5
#6 lore raw
Quoted 15 lines · Jason Low
Quoted 6 lines · Jason Low
Quoted 4 lines · Dave Chinner

FWIW, the rwsems in the struct xfs_inode are often heavily read/write contended, so there are lots of IO related workloads that are going to regress on XFS without this optimisation...

Quoted 7 lines
Quoted 3 lines

Anyway, consider the patch:

Hi Dave,

Thanks for testing. I'll update the patch with an actual changelog.

Quoted 7 lines

---
Subject: [PATCH] rwsem: In rwsem_can_spin_on_owner(), return false if no owner

It was found that the rwsem optimistic spinning feature can potentially degrade performance when there are readers. Perf profiles indicate in some workloads that significant time can be spent spinning on !owner. This is because we don't set the lock owner when readers(s) obtain the rwsem.

I don't think you're being a little shifty with the truth here. There's no "potentially degrade performance" here - I reported a massive real world performance regression caused by optimistic spinning. That is:

"Commit 4fc828e ("locking/rwsem: Support optimistic spinning") introduced a major performance regression for workloads such as xfs_repair which mix read and write locking of the mmap_sem across many threads. The result was xfs_repair ran 5x slower on 3.16-rc2 than on 3.15 and using 20x more system CPU time."

"Perf profiles indicate....

Quoted 8 lines

In this patch, we'll modify rwsem_can_spin_on_owner() such that we'll return false if there is no lock owner. The rationale is that if we just entered the slowpath, yet there is no lock owner, then there is a possibility that a reader has the lock. To be conservative, we'll avoid spinning in these situations.

Dave Chinner found performance benefits with this patch in the xfs_repair workload, where the total run time went from approximately 4 minutes 24 seconds, down to approximately 1 minute 26 seconds with the patch.

Which brought it back to close to the same performance as on 3.15. This is not a new performance improvement patch - it's a regression fix and the commit message needs to reflect that.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
Tested-by Dave Chinner
#7 lore raw
Quoted 24 lines · Dave Chinner
Quoted 15 lines · Jason Low
Quoted 6 lines · Jason Low
Quoted 4 lines · Dave Chinner

FWIW, the rwsems in the struct xfs_inode are often heavily read/write contended, so there are lots of IO related workloads that are going to regress on XFS without this optimisation...

Quoted 7 lines
Quoted 3 lines

Anyway, consider the patch:

Hi Dave,

Thanks for testing. I'll update the patch with an actual changelog.

Quoted 7 lines

---
Subject: [PATCH] rwsem: In rwsem_can_spin_on_owner(), return false if no owner

It was found that the rwsem optimistic spinning feature can potentially degrade performance when there are readers. Perf profiles indicate in some workloads that significant time can be spent spinning on !owner. This is because we don't set the lock owner when readers(s) obtain the rwsem.

Quoted 4 lines

I don't think you're being a little shifty with the truth here. There's no "potentially degrade performance" here - I reported a massive real world performance regression caused by optimistic spinning.

Sure, though I mainly used the word "potentially" since there can be other workloads out there where spinning even when readers have the lock is a positive thing.

And agreed that the changelog can be modified to try reflecting more on it being a "regression fix" then a "new performance" addition.

So how about the following?

---
Subject: [PATCH] rwsem: In rwsem_can_spin_on_owner(), return false if no owner

Commit 4fc828e24cd9 ("locking/rwsem: Support optimistic spinning") introduced a major performance regression for workloads such as xfs_repair which mix read and write locking of the mmap_sem across many threads. The result was xfs_repair ran 5x slower on 3.16-rc2 than on 3.15 and using 20x more system CPU time.

Perf profiles indicate in some workloads that significant time can be spent spinning on !owner. This is because we don't set the lock owner when readers(s) obtain the rwsem.

In this patch, we'll modify rwsem_can_spin_on_owner() such that we'll return false if there is no lock owner. The rationale is that if we just entered the slowpath, yet there is no lock owner, then there is a possibility that a reader has the lock. To be conservative, we'll avoid spinning in these situations.

This patch reduced the total run time of the xfs_repair workload from about 4 minutes 24 seconds down to approximately 1 minute 26 seconds, back to close to the same performance as on 3.15.

---
 kernel/locking/rwsem-xadd.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/locking/rwsem-xadd.c +5 -5
index dacc321..c40c7d2 100644
@@ -285,10 +285,10 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)
 static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
 {
 	struct task_struct *owner;
-	bool on_cpu = true;
+	bool on_cpu = false;
 
 	if (need_resched())
-		return 0;
+		return false;
 
 	rcu_read_lock();
 	owner = ACCESS_ONCE(sem->owner);
@@ -297,9 +297,9 @@ static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
 	rcu_read_unlock();
 
 	/*
-	 * If sem->owner is not set, the rwsem owner may have
-	 * just acquired it and not set the owner yet or the rwsem
-	 * has been released.
+	 * If sem->owner is not set, yet we have just recently entered the
+	 * slowpath, then there is a possibility reader(s) may have the lock.
+	 * To be safe, avoid spinning in these situations.
 	 */
 	return on_cpu;
 }
Signature · 1 line
1.7.9.5
maintainer
#11 lore raw
In reply to Jason Low
Quoted 12 lines · Jason Low

Subject: [PATCH] rwsem: In rwsem_can_spin_on_owner(), return false if no owner

It was found that the rwsem optimistic spinning feature can potentially degrade performance when there are readers. Perf profiles indicate in some workloads that significant time can be spent spinning on !owner. This is because we don't set the lock owner when readers(s) obtain the rwsem.

In this patch, we'll modify rwsem_can_spin_on_owner() such that we'll return false if there is no lock owner. The rationale is that if we just entered the slowpath, yet there is no lock owner, then there is a possibility that a reader has the lock. To be conservative, we'll avoid spinning in these situations.

Quoted 6 lines

Dave Chinner found performance benefits with this patch in the xfs_repair workload, where the total run time went from approximately 4 minutes 24 seconds, down to approximately 1 minute 26 seconds with the patch.

Davidlohr, you'll be running this through your AIM and other benchmarks, I suppose?

#8 lore raw
In reply to Jason Low
Quoted 30 lines · Jason Low
Quoted 24 lines · Dave Chinner
Quoted 15 lines · Jason Low
Quoted 6 lines · Jason Low
Quoted 4 lines · Dave Chinner

FWIW, the rwsems in the struct xfs_inode are often heavily read/write contended, so there are lots of IO related workloads that are going to regress on XFS without this optimisation...

Quoted 7 lines
Quoted 3 lines

Anyway, consider the patch:

Hi Dave,

Thanks for testing. I'll update the patch with an actual changelog.

Quoted 7 lines

---
Subject: [PATCH] rwsem: In rwsem_can_spin_on_owner(), return false if no owner

It was found that the rwsem optimistic spinning feature can potentially degrade performance when there are readers. Perf profiles indicate in some workloads that significant time can be spent spinning on !owner. This is because we don't set the lock owner when readers(s) obtain the rwsem.

Quoted 4 lines

I don't think you're being a little shifty with the truth here. There's no "potentially degrade performance" here - I reported a massive real world performance regression caused by optimistic spinning.

Quoted 8 lines

Sure, though I mainly used the word "potentially" since there can be other workloads out there where spinning even when readers have the lock is a positive thing.

And agreed that the changelog can be modified to try reflecting more on it being a "regression fix" then a "new performance" addition.

So how about the following?

Looks good. Thanks!

-Dave.

Signature · 2 lines
Dave Chinner
[email protected]
author Acked-by
#12 lore raw
In reply to Peter Zijlstra
Quoted 17 lines · Peter Zijlstra
Quoted 16 lines · Jason Low

Subject: [PATCH] rwsem: In rwsem_can_spin_on_owner(), return false if no owner

It was found that the rwsem optimistic spinning feature can potentially degrade performance when there are readers. Perf profiles indicate in some workloads that significant time can be spent spinning on !owner. This is because we don't set the lock owner when readers(s) obtain the rwsem.

In this patch, we'll modify rwsem_can_spin_on_owner() such that we'll return false if there is no lock owner. The rationale is that if we just entered the slowpath, yet there is no lock owner, then there is a possibility that a reader has the lock. To be conservative, we'll avoid spinning in these situations.

Dave Chinner found performance benefits with this patch in the xfs_repair workload, where the total run time went from approximately 4 minutes 24 seconds, down to approximately 1 minute 26 seconds with the patch.

Quoted 5 lines
Quoted 2 lines

Davidlohr, you'll be running this through your AIM and other benchmarks, I suppose?

I ran it through aim7, and as I suspected we take a performance hit with 'custom' ~-14% throughput for > 300 users (which is still overall quite higher than rwsems without opt spinning, at around ~+45%), and we actually improve a bit (~+15%) in 'disk' with >1000 users -- which afaict resembles Dave's workload a bit. So all in all I'm quite happy with this patch.

#9 lore raw
In reply to Jason Low

On Fri, 2014-07-04 at 00:06 -0700, Jason Low

Quoted 1 line

Subject: [PATCH] rwsem: In rwsem_can_spin_on_owner(), return false if no owner

Could we change the subject to something more descriptive, ie:

rwsem: Allow conservative optimistic spinning for reader/writer paths

Thanks,
Davidlohr

Quoted 56 lines

Commit 4fc828e24cd9 ("locking/rwsem: Support optimistic spinning") introduced a major performance regression for workloads such as xfs_repair which mix read and write locking of the mmap_sem across many threads. The result was xfs_repair ran 5x slower on 3.16-rc2 than on 3.15 and using 20x more system CPU time.

Perf profiles indicate in some workloads that significant time can be spent spinning on !owner. This is because we don't set the lock owner when readers(s) obtain the rwsem.

In this patch, we'll modify rwsem_can_spin_on_owner() such that we'll return false if there is no lock owner. The rationale is that if we just entered the slowpath, yet there is no lock owner, then there is a possibility that a reader has the lock. To be conservative, we'll avoid spinning in these situations.

This patch reduced the total run time of the xfs_repair workload from about 4 minutes 24 seconds down to approximately 1 minute 26 seconds, back to close to the same performance as on 3.15.

---
 kernel/locking/rwsem-xadd.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/locking/rwsem-xadd.c +5 -5
index dacc321..c40c7d2 100644
@@ -285,10 +285,10 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)
 static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
 {
 	struct task_struct *owner;
-	bool on_cpu = true;
+	bool on_cpu = false;
 
 	if (need_resched())
-		return 0;
+		return false;
 
 	rcu_read_lock();
 	owner = ACCESS_ONCE(sem->owner);
@@ -297,9 +297,9 @@ static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
 	rcu_read_unlock();
 
 	/*
-	 * If sem->owner is not set, the rwsem owner may have
-	 * just acquired it and not set the owner yet or the rwsem
-	 * has been released.
+	 * If sem->owner is not set, yet we have just recently entered the
+	 * slowpath, then there is a possibility reader(s) may have the lock.
+	 * To be safe, avoid spinning in these situations.
 	 */
 	return on_cpu;
 }
#10 lore raw
Quoted 6 lines · Davidlohr Bueso

On Fri, 2014-07-04 at 00:06 -0700, Jason Low

Quoted 1 line

Subject: [PATCH] rwsem: In rwsem_can_spin_on_owner(), return false if no owner

Could we change the subject to something more descriptive, ie:

rwsem: Allow conservative optimistic spinning for reader/writer paths

Sure, a Subject like this would better explain what is the effect of the patch.

Tested-by Dave Chinner Acked-by Davidlohr Bueso
#13 lore raw
In reply to Davidlohr Bueso
Quoted 10 lines · Davidlohr Bueso
Quoted 2 lines · Peter Zijlstra

Davidlohr, you'll be running this through your AIM and other benchmarks, I suppose?

I ran it through aim7, and as I suspected we take a performance hit with 'custom' ~-14% throughput for > 300 users (which is still overall quite higher than rwsems without opt spinning, at around ~+45%), and we actually improve a bit (~+15%) in 'disk' with >1000 users -- which afaict resembles Dave's workload a bit. So all in all I'm quite happy with this patch.

Here is the patch with the updates to the changelog.

---
Subject: [PATCH] rwsem: Allow conservative optimistic spinning when readers have lock

Commit 4fc828e24cd9 ("locking/rwsem: Support optimistic spinning") introduced a major performance regression for workloads such as xfs_repair which mix read and write locking of the mmap_sem across many threads. The result was xfs_repair ran 5x slower on 3.16-rc2 than on 3.15 and using 20x more system CPU time.

Perf profiles indicate in some workloads that significant time can be spent spinning on !owner. This is because we don't set the lock owner when readers(s) obtain the rwsem.

In this patch, we'll modify rwsem_can_spin_on_owner() such that we'll return false if there is no lock owner. The rationale is that if we just entered the slowpath, yet there is no lock owner, then there is a possibility that a reader has the lock. To be conservative, we'll avoid spinning in these situations.

This patch reduced the total run time of the xfs_repair workload from about 4 minutes 24 seconds down to approximately 1 minute 26 seconds, back to close to the same performance as on 3.15.

Retesting of AIM7, which were some of the workloads used to test the original optimistic spinning code, confirmed that we still get big performance gains with optimistic spinning, even with this additional regression fix. Davidlohr found that while the 'custom' workload took a performance hit of ~-14% to throughput for >300 users with this additional patch, the overall gain with optimistic spinning is still ~+45%. The 'disk' workload even improved by ~+15% at >1000 users.

---
 kernel/locking/rwsem-xadd.c |   10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/locking/rwsem-xadd.c +5 -5
index dacc321..c40c7d2 100644
@@ -285,10 +285,10 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)
 static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
 {
 	struct task_struct *owner;
-	bool on_cpu = true;
+	bool on_cpu = false;
 
 	if (need_resched())
-		return 0;
+		return false;
 
 	rcu_read_lock();
 	owner = ACCESS_ONCE(sem->owner);
@@ -297,9 +297,9 @@ static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
 	rcu_read_unlock();
 
 	/*
-	 * If sem->owner is not set, the rwsem owner may have
-	 * just acquired it and not set the owner yet or the rwsem
-	 * has been released.
+	 * If sem->owner is not set, yet we have just recently entered the
+	 * slowpath, then there is a possibility reader(s) may have the lock.
+	 * To be safe, avoid spinning in these situations.
 	 */
 	return on_cpu;
 }
Signature · 1 line
1.7.9.5
#16 lore raw
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Xie Miao

Hi, Jason

Could you re-sent this patch? Because it seems it is ignored.

Its not ignored;

Right, I also noticed the patch in the git peterz/queue tree :)

#17 lore raw
In reply to Peter Zijlstra
Quoted 6 lines · Peter Zijlstra
Quoted 3 lines · Xie Miao

Hi, Jason

Could you re-sent this patch? Because it seems it is ignored.

Its not ignored; its not merged because of unrelated stability issues.

Is it going to be merged? If not, can you please revert the optimistic spinning patch that caused the regression?

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#18 lore raw

[tip:locking/urgent] locking/rwsem: Allow conservative optimistic spinning when readers have lock

In reply to Jason Low
locking/rwsem: Allow conservative optimistic spinning when readers have lock
Commit-ID:  37e9562453b813d2ea527bd9531fef2c3c592847
Gitweb:     http://git.kernel.org/tip/37e9562453b813d2ea527bd9531fef2c3c592847
Author:     Jason Low <[email protected]>
AuthorDate: Fri, 4 Jul 2014 20:49:32 -0700
Committer:  Ingo Molnar <[email protected]>
CommitDate: Wed, 16 Jul 2014 13:28:02 +0200

locking/rwsem: Allow conservative optimistic spinning when readers have lock

Commit 4fc828e24cd9 ("locking/rwsem: Support optimistic spinning") introduced a major performance regression for workloads such as xfs_repair which mix read and write locking of the mmap_sem across many threads. The result was xfs_repair ran 5x slower on 3.16-rc2 than on 3.15 and using 20x more system CPU time.

Perf profiles indicate in some workloads that significant time can be spent spinning on !owner. This is because we don't set the lock owner when readers(s) obtain the rwsem.

In this patch, we'll modify rwsem_can_spin_on_owner() such that we'll return false if there is no lock owner. The rationale is that if we just entered the slowpath, yet there is no lock owner, then there is a possibility that a reader has the lock. To be conservative, we'll avoid spinning in these situations.

This patch reduced the total run time of the xfs_repair workload from about 4 minutes 24 seconds down to approximately 1 minute 26 seconds, back to close to the same performance as on 3.15.

Retesting of AIM7, which were some of the workloads used to test the original optimistic spinning code, confirmed that we still get big performance gains with optimistic spinning, even with this additional regression fix. Davidlohr found that while the 'custom' workload took a performance hit of ~-14% to throughput for >300 users with this additional patch, the overall gain with optimistic spinning is still ~+45%. The 'disk' workload even improved by ~+15% at >1000 users.

---
 kernel/locking/rwsem-xadd.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)
kernel/locking/rwsem-xadd.c +5 -5
index dacc321..c40c7d2 100644
@@ -285,10 +285,10 @@ static inline bool rwsem_try_write_lock_unqueued(struct rw_semaphore *sem)
 static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
 {
 	struct task_struct *owner;
-	bool on_cpu = true;
+	bool on_cpu = false;
 
 	if (need_resched())
-		return 0;
+		return false;
 
 	rcu_read_lock();
 	owner = ACCESS_ONCE(sem->owner);
@@ -297,9 +297,9 @@ static inline bool rwsem_can_spin_on_owner(struct rw_semaphore *sem)
 	rcu_read_unlock();
 
 	/*
-	 * If sem->owner is not set, the rwsem owner may have
-	 * just acquired it and not set the owner yet or the rwsem
-	 * has been released.
+	 * If sem->owner is not set, yet we have just recently entered the
+	 * slowpath, then there is a possibility reader(s) may have the lock.
+	 * To be safe, avoid spinning in these situations.
 	 */
 	return on_cpu;
 }