Badness in __mutex_unlock_slowpath with XFS stress tests
Hi all,
I was working on an issue with getting "Badness in __mutex_unlock_slowpath" and hence a stack trace, while running FS stress tests on XFS on 2.6.16-rc5 kernel.
The dmesg looks like :
Badness in __mutex_unlock_slowpath at kernel/mutex.c:207
[<c0103c0c>] show_trace+0x20/0x22 [<c0103d4b>] dump_stack+0x1e/0x20 [<c0473f1f>] __mutex_unlock_slowpath+0x12a/0x23b [<c0473938>] mutex_unlock+0xb/0xd [<c02a5720>] xfs_read+0x230/0x2d9 [<c02a1bed>] linvfs_aio_read+0x8d/0x98 [<c015f3df>] do_sync_read+0xb8/0x107 [<c015f4f7>] vfs_read+0xc9/0x19b [<c015f8b2>] sys_read+0x47/0x6e [<c0102db7>] sysenter_past_esp+0x54/0x75
This happens with XFS DIO reads. xfs_read holds the i_mutex and issues a __generic_file_aio_read(), which falls into __blockdev_direct_IO with DIO_OWN_LOCKING flag (since xfs uses own_locking ). Now __blockdev_direct_IO releases the i_mutex for READs with DIO_OWN_LOCKING.When it returns to xfs_read, it tries to unlock the i_mutex ( which is now already unlocked), causing the "Badness".
The possible solution which we can think of, is not to unlock thei_mutex for DIO_OWN_LOCKING. This will only affect the DIO_OWN_LOCKING users (as of now, only XFS ) with concurrent DIO sync read requests. AIO read requests would not suffer this problem since they would just return once the DIO is submitted.
Another work around for this can be adding a check "mutex_is_locked" before trying to unlock i_mutex in xfs_read. But this seems to be an ugly hack. :(
Comments ?
thanks,
Suzuki
Quoted 1 line · Suzuki
Hi all,
Hi there Suzuki,
Quoted 3 lines
I was working on an issue with getting "Badness in
__mutex_unlock_slowpath" and hence a stack trace, while running FS stress tests on XFS on 2.6.16-rc5 kernel.
Thanks for looking into this.
Quoted 13 lines
The dmesg looks like :
Badness in
__mutex_unlock_slowpathatkernel/mutex.c:207[<c0103c0c>] show_trace+0x20/0x22 [<c0103d4b>] dump_stack+0x1e/0x20 [<c0473f1f>] __mutex_unlock_slowpath+0x12a/0x23b [<c0473938>] mutex_unlock+0xb/0xd [<c02a5720>] xfs_read+0x230/0x2d9 [<c02a1bed>] linvfs_aio_read+0x8d/0x98 [<c015f3df>] do_sync_read+0xb8/0x107 [<c015f4f7>] vfs_read+0xc9/0x19b [<c015f8b2>] sys_read+0x47/0x6e [<c0102db7>] sysenter_past_esp+0x54/0x75
Yeah, test 008 from the xfstests suite was reliably hitting this for me, it'd just not percolated to the top of my todo list yet.
Quoted 6 lines
This happens with XFS DIO reads.
xfs_readholds thei_mutexand issues a__generic_file_aio_read(), which falls into__blockdev_direct_IOwithDIO_OWN_LOCKINGflag (since xfs usesown_locking). Now__blockdev_direct_IOreleases thei_mutexfor READs with DIO_OWN_LOCKING.When it returns toxfs_read, it tries to unlock thei_mutex( which is now already unlocked), causing the "Badness".
Indeed. And there's the problem - why is XFS releasing i_mutex for the direct read in xfs_read? Shouldn't be - fs/direct-io.c will always release i_mutex for a direct read in the own-locking case, so XFS shouldn't be doing it too (thats what the code does and thats what the comment preceding __blockdev_direct_IO says).
The only piece of the puzzle I don't understand is why we don't always get that badness message at the end of every direct read. Perhaps its some subtle fastpath/slowpath difference, or maybe "debug_mutex_on" gets switched off after the first occurance...
Anyway, with the above change (remove 2 lines near xfs_read end), I can no longer reproduce the problem in that previously-warning test case, and all the other XFS tests seem to be chugging along OK (which includes a healthy mix of dio testing).
Quoted 5 lines
The possible solution which we can think of, is not to unlock the
i_mutexforDIO_OWN_LOCKING. This will only affect theDIO_OWN_LOCKINGusers (as of now, only XFS ) with concurrent DIO sync read requests. AIO read requests would not suffer this problem since they would just return once the DIO is submitted.
I don't think that level of invasiveness is necessary at this stage, but perhaps you're seeing something that I've missed? Do you see any reason why removing the xfs_read unlock wont work?
Quoted 3 lines
Another work around for this can be adding a check "
mutex_is_locked" before trying to unlocki_mutexinxfs_read. But this seems to be an ugly hack. :(
Hmm, that just plain wouldn't work - what if the lock was released in generic direct IO code, and someone else had acquired it before we got to the end of xfs_read? Badness for sure.
cheers.
Signature · 1 line
Nathan
Hi Nathan,
Quoted 49 lines · Nathan Scott
Quoted 1 line · Suzuki
Hi all,
Hi there Suzuki,
Quoted 3 lines
I was working on an issue with getting "Badness in
__mutex_unlock_slowpath" and hence a stack trace, while running FS stress tests on XFS on 2.6.16-rc5 kernel.Thanks for looking into this.
Quoted 13 lines
The dmesg looks like :
Badness in
__mutex_unlock_slowpathatkernel/mutex.c:207[<c0103c0c>] show_trace+0x20/0x22 [<c0103d4b>] dump_stack+0x1e/0x20 [<c0473f1f>] __mutex_unlock_slowpath+0x12a/0x23b [<c0473938>] mutex_unlock+0xb/0xd [<c02a5720>] xfs_read+0x230/0x2d9 [<c02a1bed>] linvfs_aio_read+0x8d/0x98 [<c015f3df>] do_sync_read+0xb8/0x107 [<c015f4f7>] vfs_read+0xc9/0x19b [<c015f8b2>] sys_read+0x47/0x6e [<c0102db7>] sysenter_past_esp+0x54/0x75Yeah, test 008 from the xfstests suite was reliably hitting this for me, it'd just not percolated to the top of my todo list yet.
Quoted 6 lines
This happens with XFS DIO reads.
xfs_readholds thei_mutexand issues a__generic_file_aio_read(), which falls into__blockdev_direct_IOwithDIO_OWN_LOCKINGflag (since xfs usesown_locking). Now__blockdev_direct_IOreleases thei_mutexfor READs with DIO_OWN_LOCKING.When it returns toxfs_read, it tries to unlock thei_mutex( which is now already unlocked), causing the "Badness".Indeed. And there's the problem - why is XFS releasing
i_mutexfor the direct read inxfs_read? Shouldn't be -fs/direct-io.cwill always releasei_mutexfor a direct read in the own-locking case, so XFS shouldn't be doing it too (thats what the code does and thats what the comment preceding__blockdev_direct_IOsays).
Quoted 4 lines
The only piece of the puzzle I don't understand is why we don't always get that badness message at the end of every direct read. Perhaps its some subtle fastpath/slowpath difference, or maybe "
debug_mutex_on" gets switched off after the first occurance...
Yes, the debug_mutex_on gets switched off after the first occurence.
Quoted 7 lines
Anyway, with the above change (remove 2 lines near
xfs_readend), I can no longer reproduce the problem in that previously-warning test case, and all the other XFS tests seem to be chugging along OK (which includes a healthy mix of dio testing).
Quoted 8 lines
Quoted 2 lines
The possible solution which we can think of, is not to unlock the
i_mutexforDIO_OWN_LOCKING. This will only affect theDIO_OWN_LOCKINGusers (as of now, only XFS ) with concurrent DIO sync read requests. AIO read requests would not suffer this problem since they would just return once the DIO is submitted.I don't think that level of invasiveness is necessary at this stage, but perhaps you're seeing something that I've missed? Do you see any reason why removing the
xfs_readunlock wont work?
But, what happens if __generic_file_aio_read() hits some error before doing the aops->direct_IO ?
Quoted 6 lines
Quoted 4 lines
Another work around for this can be adding a check "
mutex_is_locked" before trying to unlocki_mutexinxfs_read. But this seems to be an ugly hack. :(
Quoted 6 lines
Hmm, that just plain wouldn't work - what if the lock was released in generic direct IO code, and someone else had acquired it before we got to the end of
xfs_read? Badness for sure.cheers.
Suzuki.