New: Hangs when checking torrent through libtorrent; kernel BUG at mm/iov_iter.c:219!

https://bugzilla.kernel.org/show_bug.cgi?id=89131

            Bug ID: 89131
           Summary: Hangs when checking torrent through libtorrent; kernel
                    BUG at mm/iov_iter.c:219!
           Product: File System
           Version: 2.5
    Kernel Version: 3.17.3
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: blocking
          Priority: P1
         Component: ext4
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

Created attachment 159351
--> https://bugzilla.kernel.org/attachment.cgi?id=159351&action=edit
Backtraces from both the systems

When i use qbittorrent (libtorrent) by explicitely disable the os cache and do the hash-check on a specific torrent data (it does not happen on all of the torrents), linux hangs leaving me no choice but to reboot somehow.

I managed to get two backtraces from completely different systems (an e7300 desktop and an ivybridge laptop), they're attached.

As a reference, i started to talk about the issue here:
https://github.com/qbittorrent/qBittorrent/issues/2200#issuecomment-64951598

...seems the issue is given by the use of the O_DIRECT flag

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=89131

Lubos Dolezel <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #1 from Lubos Dolezel <[email protected]> --- Created attachment 160921
--> https://bugzilla.kernel.org/attachment.cgi?id=160921&action=edit

Kernel BUG dump

I'm seeing the same kind of crash, 5 times just yesterday!

I'm using libtorrent to download a huge torrent (almost 50GB) with many files inside (BluRay dump). I haven't seen it with other torrents.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=89131

--- Comment #2 from [email protected] <[email protected]> --- Depending on the client you're using, you should be able to enable the kernel cache so that it doesn't crash anymore.
Just a workaround, of course.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=89131

--- Comment #3 from Lubos Dolezel <[email protected]> --- Yep, this workaround helps.

As a little side note, using direct I/O with ext4 has resulted in many other crashes on my system including softlockups and various other oopses. ext4's direct involvement could be seen in ~50% of them.

So this "kernel BUG" is not the only possible outcome.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=89131

--- Comment #4 from [email protected] <[email protected]> --- I wasn't able to reproduce the problem using dd with conv=direct How did you do that?
It may be handy for developers to have a quicker way to test the issue.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=89131

--- Comment #5 from Lubos Dolezel <[email protected]> ---
Sorry for misunderstanding, I was only able to reproduce it with libtorrent.

Torrent checking resulted in aforementioned BUGs, whereas torrent downloading (i.e. write operations) resulted in much more serious problems - e.g. lockups in kworker in kthread_data().

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=89131

Theodore Tso <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #6 from Theodore Tso <[email protected]> ---
Couple of questions. First of all, is there any chance you could give a try
with 3.18.0, and see if it fails there?

Also, can you try to get line numbers out of the stack dump?

Also, how long does it take before you trigger the failure? I'm in particular interested in seeing where this comes from:

[ 4244.680376]  [<ffffffff81203313>] __blockdev_direct_IO+0x25e3/0x3630

Part of the problem here is that fs/direct_io.c has a bunch of inline functions, so it becomes tricky to figure out where the call to advance_iovec
comes from. This looks like a bug in the core direct I/O code, but we do a
fair amount of regression testing and this is a new one; it's a bit surprising no once else found this when running xfstests.

To get the line numbers, compile the kernel with

CONFIG_DEBUG_INFO=y

(You can also set CONFIG_DEBUG_INFO_REDUCED to reduce the size of your kernel build.)

Then use the addr2line program to translate the address to a line number, i.e.:

addr2line -e /path/to/kernel/build/vmlinux -i -a ffffffff81203313

Finally, can you send me an output of dumpe2fs -h of your ext3/ext4 file system, just so I can understand what file system features are enabled with your file system.

Thanks!!

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=89131

--- Comment #7 from Lubos Dolezel <[email protected]> ---
I'd love to test it with 3.18.0, but I cannot due to #89881. But I'll try CONFIG_DEBUG_INFO=y.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=89131

--- Comment #8 from Lubos Dolezel <[email protected]> ---
The kernel BUG typically happened pretty quickly when doing torrent checking (reading).

Other symptoms (more fatal crashes) when downloading and uploading happened within minutes/hours, but that's understandable, because the I/O is lower in this case.

Note that I haven't had this issue for weeks until I started downloading a particular torrent.

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=89131

--- Comment #9 from Lubos Dolezel <[email protected]> --- Created attachment 161341
--> https://bugzilla.kernel.org/attachment.cgi?id=161341&action=edit
dumpe2fs

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=89131

--- Comment #10 from [email protected] <[email protected]> --- Unfortunately the crash still happens on 3.18.2

You are receiving this mail because:
You are watching the assignee of the bug.

https://bugzilla.kernel.org/show_bug.cgi?id=89131

--- Comment #11 from Theodore Tso <[email protected]> --- Created attachment 163831
--> https://bugzilla.kernel.org/attachment.cgi?id=163831&action=edit
debugging patch (should apply for 3.17 and 3.18)

This BUG_ON is happening from the mm part of the direct_io code, which is supposed to be file system independent, and so I don't know that half of the
DIO code as well. Can you try applying the attached patch to see if it gets
us any useful information when the bug triggers?

Also, at this point it would be *really* useful if we could get a reproduction case that doesn't require using rtorrent -- i.e., something that uses just dd,
or maybe a simple C program. Would it be possible to run the a tty-mode
torrent on a serial console under strace, or some such, so we can get figure out what the system calls were that apparently triggered the BUG_ON?

You are receiving this mail because:
You are watching the assignee of the bug.