Skip to contentSkip to conversation

dax: hang on i_mmap_rwsem in generic/075

4.3-rc1, regression Dave Chinner 3 participants 4 messages Sep 22, 2015 – Sep 23, 2015 VFS

lore
Share & tools
lore
#0 lore raw

Hi folks,

I'm seeing hangs like this when using DAX on XFS on 4.3-rc1 running xfstests generic/075 (fsx test):

# echo w > /proc/sysrq-trigger
[71628.984872] sysrq: SysRq : Show Blocked State
[71628.985988]   task                        PC stack   pid father
[71628.987635] fsx             D ffff88043fd756c0 12824   520  32636 0x00000000
[71628.989251]  ffff88007f557ba8 0000000000000086 ffff88042eb40580 ffff8803c8bcc180
[71628.990645]  ffff88007f558000 ffff88041d748e80 ffff88041d748e68 ffffffff00000000
[71628.992068]  00000000fffffffe ffff88007f557bc0 ffffffff81d855ca ffff8803c8bcc180
[71628.993639] Call Trace:
[71628.994097]  [<ffffffff81d855ca>] schedule+0x3a/0x90
[71628.994997]  [<ffffffff81d88021>] rwsem_down_write_failed+0x141/0x340
[71628.996197]  [<ffffffff81792e13>] call_rwsem_down_write_failed+0x13/0x20
[71628.997548]  [<ffffffff81d87854>] ? down_write+0x24/0x40
[71628.998502]  [<ffffffff812110b6>] __dax_fault+0x546/0x6c0
[71628.999469]  [<ffffffff814b2900>] ? xfs_get_blocks+0x20/0x20
[71629.000515]  [<ffffffff814bd758>] xfs_filemap_fault+0xc8/0xf0
[71629.001668]  [<ffffffff811a0abd>] __do_fault+0x3d/0x80
[71629.002589]  [<ffffffff811a49da>] handle_mm_fault+0xb8a/0xfd0
[71629.003620]  [<ffffffff81094e3f>] __do_page_fault+0x15f/0x420
[71629.004680]  [<ffffffff810951c3>] trace_do_page_fault+0x43/0x110
[71629.005877]  [<ffffffff8108fd0a>] do_async_page_fault+0x1a/0xa0
[71629.006936]  [<ffffffff81d8b6c8>] async_page_fault+0x28/0x30

__dax_fault() gets stuck on this lock:

(gdb) l *(__dax_fault+0x546)
0xffffffff812110b6 is in __dax_fault (include/linux/fs.h:499). 494
495 int mapping_tagged(struct address_space *mapping, int tag);
496
497 static inline void i_mmap_lock_write(struct address_space *mapping)
498 {
499 down_write(&mapping->i_mmap_rwsem);
500 }
501
502 static inline void i_mmap_unlock_write(struct address_space *mapping)
503 {

This didn't happen on 4.2 + the XFS for-next code that was merged into 4.3-rc1, so it's come from changes somewhere else in the merge. I suspect either of these two commits:

46c043e mm: take i_mmap_lock in unmap_mapping_range() for DAX 8431729 dax: fix race between simultaneous faults

as they both modified the i_mmap_lock usage for DAX page faults.

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#1 lore raw
Quoted 42 lines · Dave Chinner

Hi folks,

I'm seeing hangs like this when using DAX on XFS on 4.3-rc1 running xfstests generic/075 (fsx test):

# echo w > /proc/sysrq-trigger
[71628.984872] sysrq: SysRq : Show Blocked State
[71628.985988]   task                        PC stack   pid father
[71628.987635] fsx             D ffff88043fd756c0 12824   520  32636 0x00000000
[71628.989251]  ffff88007f557ba8 0000000000000086 ffff88042eb40580 ffff8803c8bcc180
[71628.990645]  ffff88007f558000 ffff88041d748e80 ffff88041d748e68 ffffffff00000000
[71628.992068]  00000000fffffffe ffff88007f557bc0 ffffffff81d855ca ffff8803c8bcc180
[71628.993639] Call Trace:
[71628.994097]  [<ffffffff81d855ca>] schedule+0x3a/0x90
[71628.994997]  [<ffffffff81d88021>] rwsem_down_write_failed+0x141/0x340
[71628.996197]  [<ffffffff81792e13>] call_rwsem_down_write_failed+0x13/0x20
[71628.997548]  [<ffffffff81d87854>] ? down_write+0x24/0x40
[71628.998502]  [<ffffffff812110b6>] __dax_fault+0x546/0x6c0
[71628.999469]  [<ffffffff814b2900>] ? xfs_get_blocks+0x20/0x20
[71629.000515]  [<ffffffff814bd758>] xfs_filemap_fault+0xc8/0xf0
[71629.001668]  [<ffffffff811a0abd>] __do_fault+0x3d/0x80
[71629.002589]  [<ffffffff811a49da>] handle_mm_fault+0xb8a/0xfd0
[71629.003620]  [<ffffffff81094e3f>] __do_page_fault+0x15f/0x420
[71629.004680]  [<ffffffff810951c3>] trace_do_page_fault+0x43/0x110
[71629.005877]  [<ffffffff8108fd0a>] do_async_page_fault+0x1a/0xa0
[71629.006936]  [<ffffffff81d8b6c8>] async_page_fault+0x28/0x30

__dax_fault() gets stuck on this lock:

(gdb) l *(__dax_fault+0x546)
0xffffffff812110b6 is in __dax_fault (include/linux/fs.h:499). 494
495 int mapping_tagged(struct address_space *mapping, int tag);
496
497 static inline void i_mmap_lock_write(struct address_space *mapping)
498 {
499 down_write(&mapping->i_mmap_rwsem);
500 }
501
502 static inline void i_mmap_unlock_write(struct address_space *mapping)
503 {

Quoted 8 lines

This didn't happen on 4.2 + the XFS for-next code that was merged into 4.3-rc1, so it's come from changes somewhere else in the merge. I suspect either of these two commits:

46c043e mm: take i_mmap_lock in unmap_mapping_range() for DAX 8431729 dax: fix race between simultaneous faults

as they both modified the i_mmap_lock usage for DAX page faults.

It's likely we broke some locking ordering rules, but it's not obvious for me which one.

No lockdep complain? Or it's disabled?

Signature · 1 line
 Kirill A. Shutemov
#2 lore raw
Quoted 50 lines · Kirill A. Shutemov
Quoted 49 lines · Dave Chinner

Hi folks,

I'm seeing hangs like this when using DAX on XFS on 4.3-rc1 running xfstests generic/075 (fsx test):

# echo w > /proc/sysrq-trigger
[71628.984872] sysrq: SysRq : Show Blocked State
[71628.985988]   task                        PC stack   pid father
[71628.987635] fsx             D ffff88043fd756c0 12824   520  32636 0x00000000
[71628.989251]  ffff88007f557ba8 0000000000000086 ffff88042eb40580 ffff8803c8bcc180
[71628.990645]  ffff88007f558000 ffff88041d748e80 ffff88041d748e68 ffffffff00000000
[71628.992068]  00000000fffffffe ffff88007f557bc0 ffffffff81d855ca ffff8803c8bcc180
[71628.993639] Call Trace:
[71628.994097]  [<ffffffff81d855ca>] schedule+0x3a/0x90
[71628.994997]  [<ffffffff81d88021>] rwsem_down_write_failed+0x141/0x340
[71628.996197]  [<ffffffff81792e13>] call_rwsem_down_write_failed+0x13/0x20
[71628.997548]  [<ffffffff81d87854>] ? down_write+0x24/0x40
[71628.998502]  [<ffffffff812110b6>] __dax_fault+0x546/0x6c0
[71628.999469]  [<ffffffff814b2900>] ? xfs_get_blocks+0x20/0x20
[71629.000515]  [<ffffffff814bd758>] xfs_filemap_fault+0xc8/0xf0
[71629.001668]  [<ffffffff811a0abd>] __do_fault+0x3d/0x80
[71629.002589]  [<ffffffff811a49da>] handle_mm_fault+0xb8a/0xfd0
[71629.003620]  [<ffffffff81094e3f>] __do_page_fault+0x15f/0x420
[71629.004680]  [<ffffffff810951c3>] trace_do_page_fault+0x43/0x110
[71629.005877]  [<ffffffff8108fd0a>] do_async_page_fault+0x1a/0xa0
[71629.006936]  [<ffffffff81d8b6c8>] async_page_fault+0x28/0x30

__dax_fault() gets stuck on this lock:

(gdb) l *(__dax_fault+0x546)
0xffffffff812110b6 is in __dax_fault (include/linux/fs.h:499). 494
495 int mapping_tagged(struct address_space *mapping, int tag);
496
497 static inline void i_mmap_lock_write(struct address_space *mapping)
498 {
499 down_write(&mapping->i_mmap_rwsem);
500 }
501
502 static inline void i_mmap_unlock_write(struct address_space *mapping)
503 {

This didn't happen on 4.2 + the XFS for-next code that was merged into 4.3-rc1, so it's come from changes somewhere else in the merge. I suspect either of these two commits:

46c043e mm: take i_mmap_lock in unmap_mapping_range() for DAX 8431729 dax: fix race between simultaneous faults

Quoted 6 lines
Quoted 1 line

as they both modified the i_mmap_lock usage for DAX page faults.

It's likely we broke some locking ordering rules, but it's not obvious for me which one.

No lockdep complain? Or it's disabled?

Wasn't running lockdep, I don't always use it because of how slow it can make things. I'll turn it on and see what happens...

Cheers,

Dave.

Signature · 2 lines
Dave Chinner
[email protected]
#3 lore raw
Quoted 53 lines · Dave Chinner
Quoted 52 lines · Kirill A. Shutemov
Quoted 50 lines · Dave Chinner

Hi folks,

I'm seeing hangs like this when using DAX on XFS on 4.3-rc1 running xfstests generic/075 (fsx test):

# echo w > /proc/sysrq-trigger
[71628.984872] sysrq: SysRq : Show Blocked State
[71628.985988]   task                        PC stack   pid father
[71628.987635] fsx             D ffff88043fd756c0 12824   520  32636 0x00000000
[71628.989251]  ffff88007f557ba8 0000000000000086 ffff88042eb40580 ffff8803c8bcc180
[71628.990645]  ffff88007f558000 ffff88041d748e80 ffff88041d748e68 ffffffff00000000
[71628.992068]  00000000fffffffe ffff88007f557bc0 ffffffff81d855ca ffff8803c8bcc180
[71628.993639] Call Trace:
[71628.994097]  [<ffffffff81d855ca>] schedule+0x3a/0x90
[71628.994997]  [<ffffffff81d88021>] rwsem_down_write_failed+0x141/0x340
[71628.996197]  [<ffffffff81792e13>] call_rwsem_down_write_failed+0x13/0x20
[71628.997548]  [<ffffffff81d87854>] ? down_write+0x24/0x40
[71628.998502]  [<ffffffff812110b6>] __dax_fault+0x546/0x6c0
[71628.999469]  [<ffffffff814b2900>] ? xfs_get_blocks+0x20/0x20
[71629.000515]  [<ffffffff814bd758>] xfs_filemap_fault+0xc8/0xf0
[71629.001668]  [<ffffffff811a0abd>] __do_fault+0x3d/0x80
[71629.002589]  [<ffffffff811a49da>] handle_mm_fault+0xb8a/0xfd0
[71629.003620]  [<ffffffff81094e3f>] __do_page_fault+0x15f/0x420
[71629.004680]  [<ffffffff810951c3>] trace_do_page_fault+0x43/0x110
[71629.005877]  [<ffffffff8108fd0a>] do_async_page_fault+0x1a/0xa0
[71629.006936]  [<ffffffff81d8b6c8>] async_page_fault+0x28/0x30

__dax_fault() gets stuck on this lock:

(gdb) l *(__dax_fault+0x546)
0xffffffff812110b6 is in __dax_fault (include/linux/fs.h:499). 494
495 int mapping_tagged(struct address_space *mapping, int tag);
496
497 static inline void i_mmap_lock_write(struct address_space *mapping)
498 {
499 down_write(&mapping->i_mmap_rwsem);
500 }
501
502 static inline void i_mmap_unlock_write(struct address_space *mapping)
503 {

This didn't happen on 4.2 + the XFS for-next code that was merged into 4.3-rc1, so it's come from changes somewhere else in the merge. I suspect either of these two commits:

46c043e mm: take i_mmap_lock in unmap_mapping_range() for DAX 8431729 dax: fix race between simultaneous faults

as they both modified the i_mmap_lock usage for DAX page faults.

Quoted 7 lines
Quoted 4 lines

It's likely we broke some locking ordering rules, but it's not obvious for me which one.

No lockdep complain? Or it's disabled?

Wasn't running lockdep, I don't always use it because of how slow it can make things. I'll turn it on and see what happens...

I just wanted to let you know that I was able to reproduce this on my test setup and am planning on tracking it down tomorrow, unless someone gets to it first.