Skip to contentSkip to conversation

rwsem: optimistic spinning causing performance degradation

regression, 3.16-rc Dave Chinner 3 participants 6 messages Jul 3, 2014

lore
Share & tools
lore
#0 lore raw

Hi folks,

I've got a workload that hammers the mmap_sem via multi-threads memory allocation and page faults: it's called xfs_repair. I've been debugging problems with the latest release, and in the process of tracking down some recent regressions I noticed that turning off all the cross-ag IO concurrency resulted in the repair running about 4-5x faster.

i.e.:

# time xfs_repair -f -vv fs.img
.....
        XFS_REPAIR Summary    Thu Jul  3 11:51:41 2014
Phase           Start           End             Duration
Phase 1:        07/03 11:47:15  07/03 11:47:15
Phase 2:        07/03 11:47:15  07/03 11:47:35  20 seconds
Phase 3:        07/03 11:47:35  07/03 11:51:26  3 minutes, 51 seconds
Phase 4:        07/03 11:51:26  07/03 11:51:31  5 seconds
Phase 5:        07/03 11:51:31  07/03 11:51:31
Phase 6:        07/03 11:51:31  07/03 11:51:39  8 seconds
Phase 7:        07/03 11:51:39  07/03 11:51:39

Total run time: 4 minutes, 24 seconds
done

real    4m26.399s
user    1m6.023s
sys     27m26.707s

$

And turning off AG striding:

# time xfs_repair -f -vv -o ag_stride=-1 fs.img
.....
        XFS_REPAIR Summary    Thu Jul  3 11:41:28 2014
Phase           Start           End             Duration
Phase 1:        07/03 11:40:27  07/03 11:40:27  
Phase 2:        07/03 11:40:27  07/03 11:40:36  9 seconds
Phase 3:        07/03 11:40:36  07/03 11:41:12  36 seconds
Phase 4:        07/03 11:41:12  07/03 11:41:17  5 seconds
Phase 5:        07/03 11:41:17  07/03 11:41:18  1 second
Phase 6:        07/03 11:41:18  07/03 11:41:25  7 seconds
Phase 7:        07/03 11:41:25  07/03 11:41:25  

Total run time: 58 seconds
done

real    0m59.893s
user    0m41.935s
sys     4m55.867s

$

The difference is in phase 2 and 3, which is where all the memory allocation and IO that populates the userspace buffer cache takes place. The remainder of the phases run from the cache. All IO is direct IO, so there is no kernel caching at all. The filesystem image has a lot of metadata in it - it has 336 AGs and the buffer cache grows to about 6GB in size during phase 3.

The difference in performance is in the system CPU time, and it results directly in lower IO dispatch rates - about 2,000 IOPS instead of ~12,000.

This is what the kernel profile looks like on the strided run:

- 83.06% [kernel] [k] osq_lock
- osq_lock
- 100.00% rwsem_down_write_failed
- call_rwsem_down_write_failed
- 99.55% sys_mprotect
tracesys
__GI___mprotect
- 12.02% [kernel] [k] rwsem_down_write_failed
rwsem_down_write_failed
call_rwsem_down_write_failed

+   1.09%  [kernel]  [k] _raw_spin_unlock_irqrestore
+   0.92%  [kernel]  [k] _raw_spin_unlock_irq
+   0.68%  [kernel]  [k] __do_softirq
+   0.33%  [kernel]  [k] default_send_IPI_mask_sequence_phys
+   0.10%  [kernel]  [k] __do_page_fault

Basically, all the kernel time is spent processing lock contention rather than doing real work.

I haven't tested back on 3.15 yet, but historically the strided AG repair for such filesystems (which I test all the time on 100+500TB filesystem images) is about 20-25% faster on this storage subsystem. Yes, the old code also burnt a lot of CPU due to lock contention, but it didn't go 5x slower than having no threading at all.

So this looks like a significant performance regression due to:

4fc828e locking/rwsem: Support optimistic spinning

which changed the rwsem behaviour in 3.16-rc1.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#1 lore raw
Quoted 4 lines · Dave Chinner

Hi folks,

I've got a workload that hammers the mmap_sem via multi-threads memory allocation and page faults: it's called xfs_repair.

Another reason for concurrent address space operations :/

Quoted 32 lines

I've been
debugging problems with the latest release, and in the process of tracking down some recent regressions I noticed that turning off all the cross-ag IO concurrency resulted in the repair running about 4-5x faster.

i.e.:

# time xfs_repair -f -vv fs.img
.....
        XFS_REPAIR Summary    Thu Jul  3 11:51:41 2014
Phase           Start           End             Duration
Phase 1:        07/03 11:47:15  07/03 11:47:15
Phase 2:        07/03 11:47:15  07/03 11:47:35  20 seconds
Phase 3:        07/03 11:47:35  07/03 11:51:26  3 minutes, 51 seconds
Phase 4:        07/03 11:51:26  07/03 11:51:31  5 seconds
Phase 5:        07/03 11:51:31  07/03 11:51:31
Phase 6:        07/03 11:51:31  07/03 11:51:39  8 seconds
Phase 7:        07/03 11:51:39  07/03 11:51:39

Total run time: 4 minutes, 24 seconds
done

real    4m26.399s
user    1m6.023s
sys     27m26.707s

$

And turning off AG striding:

Quoted 8 lines
# time xfs_repair -f -vv -o ag_stride=-1 fs.img
.....
        XFS_REPAIR Summary    Thu Jul  3 11:41:28 2014
Phase           Start           End             Duration
Phase 1:        07/03 11:40:27  07/03 11:40:27  
Phase 2:        07/03 11:40:27  07/03 11:40:36  9 seconds
Phase 3:        07/03 11:40:36  07/03 11:41:12  36 seconds

The *real* degradation is here then.

Quoted 42 lines
Phase 4:        07/03 11:41:12  07/03 11:41:17  5 seconds
Phase 5:        07/03 11:41:17  07/03 11:41:18  1 second
Phase 6:        07/03 11:41:18  07/03 11:41:25  7 seconds
Phase 7:        07/03 11:41:25  07/03 11:41:25  

Total run time: 58 seconds
done

real    0m59.893s
user    0m41.935s
sys     4m55.867s

$

The difference is in phase 2 and 3, which is where all the memory allocation and IO that populates the userspace buffer cache takes place. The remainder of the phases run from the cache. All IO is direct IO, so there is no kernel caching at all. The filesystem image has a lot of metadata in it - it has 336 AGs and the buffer cache grows to about 6GB in size during phase 3.

The difference in performance is in the system CPU time, and it results directly in lower IO dispatch rates - about 2,000 IOPS instead of ~12,000.

This is what the kernel profile looks like on the strided run:

(quoted diff) +5 -2
-  83.06%  [kernel]  [k] osq_lock
   - osq_lock
      - 100.00% rwsem_down_write_failed
         - call_rwsem_down_write_failed
            - 99.55% sys_mprotect
                 tracesys
                 __GI___mprotect
-  12.02%  [kernel]  [k] rwsem_down_write_failed
     rwsem_down_write_failed
     call_rwsem_down_write_failed
+   1.09%  [kernel]  [k] _raw_spin_unlock_irqrestore
+   0.92%  [kernel]  [k] _raw_spin_unlock_irq
+   0.68%  [kernel]  [k] __do_softirq
+   0.33%  [kernel]  [k] default_send_IPI_mask_sequence_phys
+   0.10%  [kernel]  [k] __do_page_fault
Quoted 2 lines

Basically, all the kernel time is spent processing lock contention rather than doing real work.

While before it just blocked.

Quoted 11 lines

I haven't tested back on 3.15 yet, but historically the strided AG repair for such filesystems (which I test all the time on 100+500TB filesystem images) is about 20-25% faster on this storage subsystem. Yes, the old code also burnt a lot of CPU due to lock contention, but it didn't go 5x slower than having no threading at all.

So this looks like a significant performance regression due to:

4fc828e locking/rwsem: Support optimistic spinning

which changed the rwsem behaviour in 3.16-rc1.

So the mmap_sem is held long enough in this workload that the cost of blocking is actually significantly smaller than just spinning -- particularly MCS. How many threads are you running when you see the issue?

Thanks,
Davidlohr

#2 lore raw
Quoted 7 lines · Davidlohr Bueso
Quoted 4 lines · Dave Chinner

Hi folks,

I've got a workload that hammers the mmap_sem via multi-threads memory allocation and page faults: it's called xfs_repair.

Another reason for concurrent address space operations :/

*nod*

Quoted 8 lines
Quoted 6 lines
        XFS_REPAIR Summary    Thu Jul  3 11:41:28 2014
Phase           Start           End             Duration
Phase 1:        07/03 11:40:27  07/03 11:40:27  
Phase 2:        07/03 11:40:27  07/03 11:40:36  9 seconds
Phase 3:        07/03 11:40:36  07/03 11:41:12  36 seconds

The *real* degradation is here then.

Yes, as I said, it's in phase 2 and 3 where all the IO and memory allocation is done.

Quoted 18 lines
Quoted 18 lines

This is what the kernel profile looks like on the strided run:

(quoted diff) +5 -2
-  83.06%  [kernel]  [k] osq_lock
   - osq_lock
      - 100.00% rwsem_down_write_failed
         - call_rwsem_down_write_failed
            - 99.55% sys_mprotect
                 tracesys
                 __GI___mprotect
-  12.02%  [kernel]  [k] rwsem_down_write_failed
     rwsem_down_write_failed
     call_rwsem_down_write_failed
+   1.09%  [kernel]  [k] _raw_spin_unlock_irqrestore
+   0.92%  [kernel]  [k] _raw_spin_unlock_irq
+   0.68%  [kernel]  [k] __do_softirq
+   0.33%  [kernel]  [k] default_send_IPI_mask_sequence_phys
+   0.10%  [kernel]  [k] __do_page_fault
Quoted 4 lines
Quoted 2 lines

Basically, all the kernel time is spent processing lock contention rather than doing real work.

While before it just blocked.

Yup, pretty much - there was contention on the rwsem internal spinlock, but nothing else burnt CPU time.

Quoted 6 lines
Quoted 6 lines

I haven't tested back on 3.15 yet, but historically the strided AG repair for such filesystems (which I test all the time on 100+500TB filesystem images) is about 20-25% faster on this storage subsystem. Yes, the old code also burnt a lot of CPU due to lock contention, but it didn't go 5x slower than having no threading at all.

Quoted 8 lines
Quoted 5 lines

So this looks like a significant performance regression due to:

4fc828e locking/rwsem: Support optimistic spinning

which changed the rwsem behaviour in 3.16-rc1.

So the mmap_sem is held long enough in this workload that the cost of blocking is actually significantly smaller than just spinning --

The issues is that the memory allocation pattern alternates between read and write locks. i.e. write lock on mprotect at allocation, read lock on page fault when processing the contents. Hence we have a pretty consistent pattern of allocation (and hence mprotect) in prefetch threads, while there page faults are in the processing threads.

Quoted 2 lines

particularly MCS. How many threads are you running when you see the issue?

Lots. In phase 3:

$ ps -eLF |grep [x]fs_repair | wc -l
140
$

We use 6 threads per AG being processed:

- 4 metadata prefetch threads (do allocation and IO),
- 1 prefetch control thread
- 1 metadata processing thread (do page faults)

That works out about right - default is to create a new processing group every 15 AGs, so with 336 AGs we should have roughly 22 AGs being processed concurrently...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#3 lore raw
Quoted 82 lines · Dave Chinner
Quoted 7 lines · Davidlohr Bueso
Quoted 4 lines · Dave Chinner

Hi folks,

I've got a workload that hammers the mmap_sem via multi-threads memory allocation and page faults: it's called xfs_repair.

Another reason for concurrent address space operations :/

*nod*

Quoted 8 lines
Quoted 6 lines
        XFS_REPAIR Summary    Thu Jul  3 11:41:28 2014
Phase           Start           End             Duration
Phase 1:        07/03 11:40:27  07/03 11:40:27  
Phase 2:        07/03 11:40:27  07/03 11:40:36  9 seconds
Phase 3:        07/03 11:40:36  07/03 11:41:12  36 seconds

The *real* degradation is here then.

Yes, as I said, it's in phase 2 and 3 where all the IO and memory allocation is done.

Quoted 18 lines
Quoted 18 lines

This is what the kernel profile looks like on the strided run:

(quoted diff) +5 -2
-  83.06%  [kernel]  [k] osq_lock
   - osq_lock
      - 100.00% rwsem_down_write_failed
         - call_rwsem_down_write_failed
            - 99.55% sys_mprotect
                 tracesys
                 __GI___mprotect
-  12.02%  [kernel]  [k] rwsem_down_write_failed
     rwsem_down_write_failed
     call_rwsem_down_write_failed
+   1.09%  [kernel]  [k] _raw_spin_unlock_irqrestore
+   0.92%  [kernel]  [k] _raw_spin_unlock_irq
+   0.68%  [kernel]  [k] __do_softirq
+   0.33%  [kernel]  [k] default_send_IPI_mask_sequence_phys
+   0.10%  [kernel]  [k] __do_page_fault
Quoted 4 lines
Quoted 2 lines

Basically, all the kernel time is spent processing lock contention rather than doing real work.

While before it just blocked.

Yup, pretty much - there was contention on the rwsem internal spinlock, but nothing else burnt CPU time.

Quoted 6 lines
Quoted 6 lines

I haven't tested back on 3.15 yet, but historically the strided AG repair for such filesystems (which I test all the time on 100+500TB filesystem images) is about 20-25% faster on this storage subsystem. Yes, the old code also burnt a lot of CPU due to lock contention, but it didn't go 5x slower than having no threading at all.

Quoted 8 lines
Quoted 5 lines

So this looks like a significant performance regression due to:

4fc828e locking/rwsem: Support optimistic spinning

which changed the rwsem behaviour in 3.16-rc1.

So the mmap_sem is held long enough in this workload that the cost of blocking is actually significantly smaller than just spinning --

The issues is that the memory allocation pattern alternates between read and write locks. i.e. write lock on mprotect at allocation, read lock on page fault when processing the contents. Hence we have a pretty consistent pattern of allocation (and hence mprotect) in prefetch threads, while there page faults are in the processing threads.

Quoted 2 lines

particularly MCS. How many threads are you running when you see the issue?

Lots. In phase 3:

$ ps -eLF |grep [x]fs_repair | wc -l
140
$

We use 6 threads per AG being processed:

Quoted 7 lines

- 4 metadata prefetch threads (do allocation and IO),
- 1 prefetch control thread
- 1 metadata processing thread (do page faults)

That works out about right - default is to create a new processing group every 15 AGs, so with 336 AGs we should have roughly 22 AGs being processed concurrently...

There's another regression with the optimisitic spinning in rwsems as well: it increases the size of the struct rw_semaphore by 16 bytes. That has increased the size of the struct xfs_inode by 32 bytes.

That's pretty damn significant - it's no uncommon to see machines with tens of millions of cached XFS inodes, so increasing the size of the inode by 4% is actually very significant. That's enough to go from having a well balanced workload to not being able to fit the working set of inodes in memory.

Filesystem developers will do almost anything to remove a few bytes from the struct inode because inode cache footprint is extremely important for performance. We also tend to get upset and unreasonable when other people undo that hard work by making changes that bloat the generic structures embedded in the inode structures....

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#4 lore raw
Quoted 11 lines · Dave Chinner

There's another regression with the optimisitic spinning in rwsems as well: it increases the size of the struct rw_semaphore by 16 bytes. That has increased the size of the struct xfs_inode by 32 bytes.

That's pretty damn significant - it's no uncommon to see machines with tens of millions of cached XFS inodes, so increasing the size of the inode by 4% is actually very significant. That's enough to go from having a well balanced workload to not being able to fit the working set of inodes in memory.

Quoted 6 lines

Filesystem developers will do almost anything to remove a few bytes from the struct inode because inode cache footprint is extremely important for performance. We also tend to get upset and unreasonable when other people undo that hard work by making changes that bloat the generic structures embedded in the inode structures....

Jason Low actually did a patch, yesterday, to shrink rwsem back to its old size (on 64bit).

#5 lore raw
Quoted 19 lines · Peter Zijlstra
Quoted 11 lines · Dave Chinner

There's another regression with the optimisitic spinning in rwsems as well: it increases the size of the struct rw_semaphore by 16 bytes. That has increased the size of the struct xfs_inode by 32 bytes.

That's pretty damn significant - it's no uncommon to see machines with tens of millions of cached XFS inodes, so increasing the size of the inode by 4% is actually very significant. That's enough to go from having a well balanced workload to not being able to fit the working set of inodes in memory.

Quoted 6 lines

Filesystem developers will do almost anything to remove a few bytes from the struct inode because inode cache footprint is extremely important for performance. We also tend to get upset and unreasonable when other people undo that hard work by making changes that bloat the generic structures embedded in the inode structures....

Quoted 2 lines

Jason Low actually did a patch, yesterday, to shrink rwsem back to its old size (on 64bit).

That's good to know. Thanks, Peter.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]