kernel BUG at fs/ext4/inode.c:2982!
Just hit this on Linus' current tree while running my fuzz-tester. (No logs unfortunatly, so no idea what actually happened).
kernel BUG at fs/ext4/inode.c:2982!
invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: hidp rfcomm af_key llc2 can_bcm sctp libcrc32c can_raw nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 p8022 psnap llc ax25 nouveau cfg80211 rfkill kvm_intel kvm video backlight mxm_wmi wmi i2c_algo_bit drm_kms_helper ttm drm microcode tg3 serio_raw pcspkr ptp pps_core libphy i2c_core lpc_ich mfd_core rtc_cmos shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc raid0 floppy CPU: 3 PID: 24261 Comm: trinity-c10 Not tainted 3.17.0+ #5 Hardware name: Dell Inc. Precision WorkStation 490 /0DT031, BIOS A08 04/25/2008 task: ffff8802094ccb40 ti: ffff8800bc168000 task.ti: ffff8800bc168000 RIP: 0010:[<ffffffff9a27cf83>] [<ffffffff9a27cf83>] ext4_direct_IO+0x713/0x750 RSP: 0018:ffff8800bc16ba78 EFLAGS: 00010246 RAX: 0000000000020000 RBX: 0000000000000001 RCX: 000000000000000f RDX: 0000000000000008 RSI: ffff880033e368d0 RDI: ffff8802094cd3b8 RBP: ffff8800bc16baf8 R08: 0000000000000001 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000001 R12: ffff8800bc16bd40 R13: ffff880033e368d0 R14: ffff8800bc16bb30 R15: 000000000000001f FS: 00007f8cc4e8f740(0000) GS:ffff880226400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000001 CR3: 00000000b7747000 CR4: 00000000000007e0 DR0: 0000000001c16000 DR1: 000000000160a000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600 Stack: ffffea000560a600 ffffea00060dc480 ffffea000503d880 ffffea0005cbfc80 ffffea00056e6500 ffffea00049b1780 ffff880033e368d0 ffffea0005da7980 0000000000010000 0000000000010000 ffff8800bc16baf8 ffff880033e36ae0 Call Trace: [<ffffffff9a1838d9>] generic_file_direct_write+0xa9/0x170 [<ffffffff9a183c4c>] __generic_file_write_iter+0x2ac/0x350 [<ffffffff9a275df9>] ext4_file_write_iter+0x109/0x3f0 [<ffffffff9a1d8adc>] ? __kmalloc+0x39c/0x420 [<ffffffff9a0a89e8>] ? sched_clock_cpu+0xa8/0xd0 [<ffffffff9a227881>] ? iter_file_splice_write+0x91/0x450 [<ffffffff9a0a8a66>] ? local_clock+0x16/0x30 [<ffffffff9a227a53>] iter_file_splice_write+0x263/0x450 [<ffffffff9a226d06>] direct_splice_actor+0x36/0x40 [<ffffffff9a2272d3>] splice_direct_to_actor+0xc3/0x1f0 [<ffffffff9a226cd0>] ? generic_pipe_buf_nosteal+0x10/0x10 [<ffffffff9a229032>] do_splice_direct+0x82/0xb0 [<ffffffff9a1f454f>] do_sendfile+0x1af/0x3a0 [<ffffffff9a1f533a>] SyS_sendfile64+0x8a/0xa0 [<ffffffff9a6ea82a>] ? tracesys_phase2+0x75/0xd9 [<ffffffff9a6ea889>] tracesys_phase2+0xd4/0xd9 Code: e8 83 57 e4 ff 85 c0 0f 85 a0 fc ff ff e9 47 ff ff ff 48 c7 c7 e0 f4 c3 9a e8 6a 57 e4 ff 85 c0 0f 85 e7 fc ff ff e9 6c ff ff ff <0f> 0b be fe 0b 00 00 48 c7 c7 f9 4d a2 9a e8 7a 3b df ff e9 c8 RIP [<ffffffff9a27cf83>] ext4_direct_IO+0x713/0x750 RSP <ffff8800bc16ba78> ---[ end trace d80209ec68bf10b8 ]---
That BUG_ON is..
2982 BUG_ON(iocb->private == NULL);
I'll try and reproduce it in the morning.
Dave
Quoted 4 lines · Dave Jones
Just hit this on Linus' current tree while running my fuzz-tester. (No logs unfortunatly, so no idea what actually happened).
kernel BUG at
fs/ext4/inode.c:2982!
Looks familiar.http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-8086 Are you playing with fcntl?
Try this patch http://www.spinics.net/lists/linux-ext4/msg45683.html
Quoted 55 lines
invalid opcode: 0000 [#1] SMP
DEBUG_PAGEALLOCModules linked in: hidp rfcomm af_key llc2 can_bcm sctp libcrc32c can_raw nfc caif_socket caif af_802154 ieee802154 phonet af_rxrpc bluetooth can pppoe pppox ppp_generic slhc irda crc_ccitt rds rose x25 atm netrom appletalk ipx p8023 p8022 psnap llc ax25 nouveau cfg80211 rfkill kvm_intel kvm video backlight mxm_wmi wmi i2c_algo_bit drm_kms_helper ttm drm microcode tg3 serio_raw pcspkr ptp pps_core libphy i2c_core lpc_ich mfd_core rtc_cmos shpchp nfsd auth_rpcgss oid_registry nfs_acl lockd grace sunrpc raid0 floppy CPU: 3 PID: 24261 Comm: trinity-c10 Not tainted 3.17.0+ #5 Hardware name: Dell Inc. Precision WorkStation 490 /0DT031, BIOS A08 04/25/2008 task: ffff8802094ccb40 ti: ffff8800bc168000 task.ti: ffff8800bc168000 RIP: 0010:[<ffffffff9a27cf83>] [<ffffffff9a27cf83>] ext4_direct_IO+0x713/0x750 RSP: 0018:ffff8800bc16ba78 EFLAGS: 00010246 RAX: 0000000000020000 RBX: 0000000000000001 RCX: 000000000000000f RDX: 0000000000000008 RSI: ffff880033e368d0 RDI: ffff8802094cd3b8 RBP: ffff8800bc16baf8 R08: 0000000000000001 R09: 0000000000000000 R10: 0000000000000001 R11: 0000000000000001 R12: ffff8800bc16bd40 R13: ffff880033e368d0 R14: ffff8800bc16bb30 R15: 000000000000001f FS: 00007f8cc4e8f740(0000) GS:ffff880226400000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000001 CR3: 00000000b7747000 CR4: 00000000000007e0 DR0: 0000000001c16000 DR1: 000000000160a000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600 Stack: ffffea000560a600 ffffea00060dc480 ffffea000503d880 ffffea0005cbfc80 ffffea00056e6500 ffffea00049b1780 ffff880033e368d0 ffffea0005da7980 0000000000010000 0000000000010000 ffff8800bc16baf8 ffff880033e36ae0 Call Trace: [<ffffffff9a1838d9>] generic_file_direct_write+0xa9/0x170 [<ffffffff9a183c4c>] __generic_file_write_iter+0x2ac/0x350 [<ffffffff9a275df9>] ext4_file_write_iter+0x109/0x3f0 [<ffffffff9a1d8adc>] ? __kmalloc+0x39c/0x420 [<ffffffff9a0a89e8>] ? sched_clock_cpu+0xa8/0xd0 [<ffffffff9a227881>] ? iter_file_splice_write+0x91/0x450 [<ffffffff9a0a8a66>] ? local_clock+0x16/0x30 [<ffffffff9a227a53>] iter_file_splice_write+0x263/0x450 [<ffffffff9a226d06>] direct_splice_actor+0x36/0x40 [<ffffffff9a2272d3>] splice_direct_to_actor+0xc3/0x1f0 [<ffffffff9a226cd0>] ? generic_pipe_buf_nosteal+0x10/0x10 [<ffffffff9a229032>] do_splice_direct+0x82/0xb0 [<ffffffff9a1f454f>] do_sendfile+0x1af/0x3a0 [<ffffffff9a1f533a>] SyS_sendfile64+0x8a/0xa0 [<ffffffff9a6ea82a>] ? tracesys_phase2+0x75/0xd9 [<ffffffff9a6ea889>] tracesys_phase2+0xd4/0xd9 Code: e8 83 57 e4 ff 85 c0 0f 85 a0 fc ff ff e9 47 ff ff ff 48 c7 c7 e0 f4 c3 9a e8 6a 57 e4 ff 85 c0 0f 85 e7 fc ff ff e9 6c ff ff ff <0f> 0b be fe 0b 00 00 48 c7 c7 f9 4d a2 9a e8 7a 3b df ff e9 c8 RIP [<ffffffff9a27cf83>] ext4_direct_IO+0x713/0x750 RSP <ffff8800bc16ba78> ---[ end trace d80209ec68bf10b8 ]---That
BUG_ONis..2982
BUG_ON(iocb->private == NULL);I'll try and reproduce it in the morning.
Dave
--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 9 lines · Dmitry Monakhov
Quoted 4 lines · Dave Jones
Just hit this on Linus' current tree while running my fuzz-tester. (No logs unfortunatly, so no idea what actually happened).
kernel BUG at
fs/ext4/inode.c:2982!Looks familiar.http://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2014-8086 Are you playing with fcntl?
Actually, I was mistaken, there was a partial log.
Pid that oopsed was 24261. Last two calls of that pid was..
[child10:24261] [125] sendfile(out_fd=672, in_fd=672, offset=0x1, count=0xce89) = -1 (Bad address) [child10:24261] [126] sendfile(out_fd=672, in_fd=672, offset=0x0, count=0xa1f000)
The second call was the one that oopsed.
No idea what fd 672 was unfortunatly. I'll have to add better logging of fd types.
At the same time that was running, other pids were also doing stuff with that same fd.
[child6:24355] [124] writev(fd=672, vec=0x1d2ec60, vlen=206) [child8:24515] [1498] lseek(fd=672, offset=-1, whence=0x2)
Note that those calls also didn't complete, meaning they probably blocked on some lock that the oopsing pid was holding.
So no recent fcntl's unless they were done before the last 2 syscalls each process did.
Quoted 1 line
Try this patch http://www.spinics.net/lists/linux-ext4/msg45683.html
I'll give it a try if I can get it reproducing easily, otherwise I cna't say either way whether the patch is doing any good.
Dave
Quoted 4 lines · Dave Jones
Quoted 1 line
Try this patch http://www.spinics.net/lists/linux-ext4/msg45683.html
I'll give it a try if I can get it reproducing easily, otherwise I cna't say either way whether the patch is doing any good.
ok, I managed to reproduce it a few times, and then tried again with your patch. I'm not sure if it's related, but I'm now seeing lockups in ext4. The process that gets stuck looks like this..
trinity-c21 R running task 13232 9781 831 0x10000004
0000000000000000 0000000000000001 0000000000000000 0000000000000000 ffff8800c8d333d8 0000000000000000 0000000000000002 00000000cacaa650 ffff880217cf7db8 0000000000000000 0000000000000000 ffffffffa929ff8c
Call Trace: [<ffffffffa929ff8c>] ? ext4_map_blocks+0x31c/0x560 [<ffffffffa929fe75>] ? ext4_map_blocks+0x205/0x560 [<ffffffffa92e689c>] ? ext4_es_find_delayed_extent_range+0x48c/0x4e0 [<ffffffffa9298fe1>] ? ext4_llseek+0x261/0x3f0 [<ffffffffa922e6e9>] ? __fdget_pos+0x49/0x50 [<ffffffffa90e484e>] ? rcu_read_lock_held+0x6e/0x80 [<ffffffffa920ca44>] ? SyS_lseek+0x94/0xc0
I've got to run right now, but I'll look into tracing it some more when I get back.
Dave
Quoted 10 lines · Dave Jones
Quoted 4 lines · Dave Jones
Quoted 1 line
Try this patch http://www.spinics.net/lists/linux-ext4/msg45683.html
I'll give it a try if I can get it reproducing easily, otherwise I cna't say either way whether the patch is doing any good.
ok, I managed to reproduce it a few times, and then tried again with your patch. I'm not sure if it's related, but I'm now seeing lockups in ext4. The process that gets stuck looks like this..
99.99% this one is not related. This is just one more uncovered one by trinity magics. Looks like I've got what is wrong lseek SEEK_HOLE/SEEK_DATA try to find block allocated one by one instead of using ext4_ext_next_allocated_block() COMMIT c8c0df24 Plese look an my test below. I'm too tired for today (especially after Gone girl the movie). I'll send patch tomorrow.
Quoted 14 lines
trinity-c21 R running task 13232 9781 831
0x10000004
0000000000000000 0000000000000001 0000000000000000 0000000000000000ffff8800c8d333d80000000000000000 000000000000000200000000cacaa650ffff880217cf7db80000000000000000 0000000000000000ffffffffa929ff8cCall Trace: [<ffffffffa929ff8c>] ? ext4_map_blocks+0x31c/0x560 [<ffffffffa929fe75>] ? ext4_map_blocks+0x205/0x560 [<ffffffffa92e689c>] ? ext4_es_find_delayed_extent_range+0x48c/0x4e0 [<ffffffffa9298fe1>] ? ext4_llseek+0x261/0x3f0 [<ffffffffa922e6e9>] ? __fdget_pos+0x49/0x50 [<ffffffffa90e484e>] ? rcu_read_lock_held+0x6e/0x80 [<ffffffffa920ca44>] ? SyS_lseek+0x94/0xc0
Quoted 3 lines
I've got to run right now, but I'll look into tracing it some more when I get back.
Dave
/*
* regression test for bug in ext4_seek_data
* (C) Dmitry Monakhov
* original report https://lkml.org/lkml/2014/10/16/620
*/
#define _GNU_SOURCE
#define _LARGEFILE_SOURCE
#define _FILE_OFFSET_BITS 64
#include <sys/types.h>
#include <unistd.h>
#include <fcntl.h>
#ifndef SEEK_DATA
#define SEEK_DATA 3
#endif
int main(int argc, char **argv)
{
int fd;
loff_t size = 1ULL << 40;
fd = open("test", O_RDWR|O_CREAT, 0777);
ftruncate(fd, size);
size = 1ULL << 39;
/* Next line will likely spin forever :) */
llseek(fd, size, SEEK_DATA);
return 0;
}PATCH 1/1 ext4: fix suboptimal seek_{data,hole} extents traversial
It is rediculus practice to scan inode block by block, this technique applicable only for old indirect files. This takes signifficant amount of time for really large files. Let's reuse ext4_fiemap which already traverse inode-tree in most optimal meaner.
TESTCASE:ftruncate64(fd, 0);ftruncate64(fd, 1ULL << 40);
/* lseek will spin very long time */lseek64(fd, 0, SEEK_DATA);lseek64(fd, 0, SEEK_HOLE);
Original report: https://lkml.org/lkml/2014/10/16/620
##################################
BTW: Why do we need i_mutex here?
--- fs/ext4/extents.c | 4 +- fs/ext4/file.c | 220 +++++++++++++++++++++++++--------------------------- 2 files changed, 108 insertions(+), 116 deletions(-)
fs/ext4/extents.c +2 -2
@@ -5155,8 +5155,8 @@ int ext4_fiemap(struct inode *inode, struct fiemap_extent_info *fieinfo,/* fallback to generic here if not in extents fmt */ if (!(ext4_test_inode_flag (inode,EXT4_INODE_EXTENTS ))) -return generic_block_fiemap (inode, fieinfo, start, len, - ext4_get_block); +return __generic_block_fiemap (inode, fieinfo, start, len, + ext4_get_block);if (fiemap_check_flags (fieinfo,EXT4_FIEMAP_FLAGS ))return -EBADR ;
fs/ext4/file.c +106 -114
277 lines not on this page · raw message
Signature · 1 line
1.7.1
Quoted 13 lines · Dmitry Monakhov
Quoted 10 lines · Dave Jones
Quoted 4 lines · Dave Jones
Quoted 1 line
Try this patch http://www.spinics.net/lists/linux-ext4/msg45683.html
I'll give it a try if I can get it reproducing easily, otherwise I cna't say either way whether the patch is doing any good.
ok, I managed to reproduce it a few times, and then tried again with your patch. I'm not sure if it's related, but I'm now seeing lockups in ext4. The process that gets stuck looks like this..
Quoted 6 lines
99.99% this one is not related. This is just one more uncovered one by trinity magics. Looks like I've got what is wrong lseek SEEK_HOLE/SEEK_DATA try to find block allocated one by one instead of using
ext4_ext_next_allocated_block()COMMITc8c0df24Plese look an my test below. I'm too tired for today (especially after Gone girl the movie). I'll send patch tomorrow.
So the good news is that I'm only seeing this 2nd bug now, so it appears your patch referenced above does fix it, so it is a good band-aid until a more generic solution is found.
thanks,
Dave
Theodore Ts'o
maintainerRe: [PATCH] ext4: fix suboptimal seek_{data,hole} extents traversial
In reply to Dmitry MonakhovQuoted 13 lines · Dmitry Monakhov
It is rediculus practice to scan inode block by block, this technique applicable only for old indirect files. This takes signifficant amount of time for really large files. Let's reuse
ext4_fiemapwhich already traverse inode-tree in most optimal meaner.TESTCASE:
ftruncate64(fd, 0);ftruncate64(fd, 1ULL << 40);
/* lseek will spin very long time */lseek64(fd, 0, SEEK_DATA);lseek64(fd, 0, SEEK_HOLE);
Quoted 6 lines
Original report: https://lkml.org/lkml/2014/10/16/620
##################################
BTW: Why do we needi_mutexhere?Signed-off-by Dmitry Monakhov <[email protected]>
Note: this patch causes generic/285 to loop forever in inline-data mode. My guess is in the special case handling of inline data in ext4_fiemap not playing well with this change, but I haven't had a chance to look deeply into this yet.
- Ted
Re: [PATCH] ext4: fix suboptimal seek_{data,hole} extents traversial
Quoted 19 lines · Theodore Ts'o
Quoted 18 lines · Dmitry Monakhov
It is rediculus practice to scan inode block by block, this technique applicable only for old indirect files. This takes signifficant amount of time for really large files. Let's reuse
ext4_fiemapwhich already traverse inode-tree in most optimal meaner.TESTCASE:
ftruncate64(fd, 0);ftruncate64(fd, 1ULL << 40);
/* lseek will spin very long time */lseek64(fd, 0, SEEK_DATA);lseek64(fd, 0, SEEK_HOLE);Original report: https://lkml.org/lkml/2014/10/16/620
##################################
BTW: Why do we needi_mutexhere?
Quoted 6 lines
Quoted 1 line
Signed-off-by Dmitry Monakhov <[email protected]>Note: this patch causes generic/285 to loop forever in inline-data mode. My guess is in the special case handling of inline data in
ext4_fiemapnot playing well with this change, but I haven't had a chance to look deeply into this yet.
Correct, it is appeared I've missed inline case. Will fix.
BTW: this happens because xfstests-bld contains obsolete e2fsprogs-lib 1.41.14
Quoted 2 lines
- Ted
Re: [PATCH] ext4: fix suboptimal seek_{data,hole} extents traversial
In reply to Theodore Ts'oQuoted 19 lines · Theodore Ts'o
Quoted 18 lines · Dmitry Monakhov
It is rediculus practice to scan inode block by block, this technique applicable only for old indirect files. This takes signifficant amount of time for really large files. Let's reuse
ext4_fiemapwhich already traverse inode-tree in most optimal meaner.TESTCASE:
ftruncate64(fd, 0);ftruncate64(fd, 1ULL << 40);
/* lseek will spin very long time */lseek64(fd, 0, SEEK_DATA);lseek64(fd, 0, SEEK_HOLE);Original report: https://lkml.org/lkml/2014/10/16/620
##################################
BTW: Why do we needi_mutexhere?
Quoted 6 lines
Quoted 1 line
Signed-off-by Dmitry Monakhov <[email protected]>Note: this patch causes generic/285 to loop forever in inline-data mode. My guess is in the special case handling of inline data in
ext4_fiemapnot playing well with this change, but I haven't had a chance to look deeply into this yet.
FYI: inline_data feature is fatally broken
1) incorrect lock order (journal_start vs grab_cache_page_write_begin ) ext4_da_write_inline_data_begin
->start_journal
->grab_cache_page_write_begin
->pagecache_get_page -> FS_REENTRANCE->DEADLOCK
2) ext4_inline_data_fiemap(0 ignores start and len arguments from ext4_filemap() which obviously result in endless loop for anyone who want to use fiemap for inline files (which my patch try to do)
I'll send patches soon.
Quoted 2 lines
- Ted
Theodore Ts'o
maintainerRe: [PATCH] ext4: fix suboptimal seek_{data,hole} extents traversial
Quoted 1 line · Dmitry Monakhov
FYI:
inline_datafeature is fatally broken
Thanks for finding the bugs!
Quoted 3 lines
2) ext4_inline_data_fiemap(0 ignores start and len arguments from
ext4_filemap()which obviously result in endless loop for anyone who want to use fiemap for inline files (which my patch try to do)
Would this have been triggered using filefrag? I just checked and noticed we don't have a filefrag test in xfstests; of course it would be impossible to test other than the filefrag -v command successfully completed, but maybe it might be worth adding such a test.
- Ted
Re: [PATCH] ext4: fix suboptimal seek_{data,hole} extents traversial
Quoted 5 lines · Theodore Ts'o
Quoted 1 line · Dmitry Monakhov
FYI:
inline_datafeature is fatally brokenThanks for finding the bugs!
Quoted 8 lines
Quoted 3 lines
2) ext4_inline_data_fiemap(0 ignores start and len arguments from
ext4_filemap()which obviously result in endless loop for anyone who want to use fiemap for inline files (which my patch try to do)Would this have been triggered using filefrag? I just checked and noticed we don't have a filefrag test in xfstests; of course it would be impossible to test other than the filefrag -v command successfully completed, but maybe it might be worth adding such a test.
This works only because filefrag use big buffer. If one want to call fiemap with start != 0 it will loop. So we definitely need a regression test for fiemap. I'll do it.
BTW bug can be spotted even in case of filefrag -v:
#### Create small file with inlined data
# echo test > /mnt/f1
#### Now boost it's i_size
# truncate --size 4096000 /mnt/f1
# filefrag -v /mnt/f1
Filesystem cylinder groups approximately 39
File size of /mnt/f1 is 4096000 (1000 blocks of 4096 bytes)
ext: logical_offset: physical_offset: length: expected:
flags:
0: 0.. 4095999: 2762792.. 6858791: 4096000:last,not_aligned,inline,eof
/mnt/f1: 1 extent found
###According to fiemap we are inlined 4Mb data. :)At this moment I'm testing my patches, I'll send it today.
Quoted 2 lines
- Ted