Skip to contentSkip to conversation

f2fs: kernel BUG at fs/f2fs/segment.h:543

DISCUSSION Sasha Levin 3 participants 6 messages Apr 5, 2014 – Apr 9, 2014 F2FS file system

lore
Share & tools
lore
#0 lore raw

Hi all,

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel I've stumbled on the following:

[ 1789.220942] kernel BUG at fs/f2fs/segment.h:543!
[ 1789.220942] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 1789.220942] Dumping ftrace buffer:
[ 1789.220942]    (ftrace buffer empty)
[ 1789.220942] Modules linked in:
[ 1789.220942] CPU: 0 PID: 28161 Comm: trinity-c0 Not tainted 3.14.0-next-20140403-sasha-00019-g7474aa9-dirty #376
[ 1789.220942] task: ffff88032a598000 ti: ffff880329c3e000 task.ti: ffff880329c3e000
[ 1789.220942] RIP: f2fs_submit_page_mbio (fs/f2fs/segment.h:543 fs/f2fs/data.c:181)
[ 1789.220942] RSP: 0018:ffff880329c3fc08  EFLAGS: 00010287
[ 1789.220942] RAX: 000000000000ffff RBX: ffff88012ac66a00 RCX: 0000000000000009
[ 1789.220942] RDX: 0000000000000200 RSI: ffffea0001dd92c0 RDI: ffff88012ac667b0
[ 1789.220942] RBP: ffff880329c3fc58 R08: 00000000000ba28e R09: ffff88032a598d98
[ 1789.220942] R10: 0000000000000000 R11: 0000000000000000 R12: ffffea0001dd92c0
[ 1789.220942] R13: ffff88012ac667b0 R14: 0000000000000000 R15: ffff880329c3fc70
[ 1789.220942] FS:  00007f3599003700(0000) GS:ffff88007dc00000(0000) knlGS:0000000000000000
[ 1789.220942] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1789.220942] CR2: 00007faf4246a018 CR3: 0000000329b6f000 CR4: 00000000000006b0
[ 1789.220942] DR0: 0000000000696000 DR1: 0000000000696000 DR2: 0000000000000000
[ 1789.261947] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
[ 1789.261947] Stack:
[ 1789.261947]  ffff88032a598000 ffff880d2a478278 0000000000000006 0000000000000082
[ 1789.261947]  ffffffffaa289475 ffffea0001dd92c0 ffff88012ac667b0 ffff880329c3fd30
[ 1789.261947]  0000000000000000 ffffea0001dd92c0 ffff880329c3fc88 ffffffffaa9da033
[ 1789.278327] Call Trace:
[ 1789.278327] ? clear_page_dirty_for_io (arch/x86/include/asm/paravirt.h:809 include/linux/backing-dev.h:176 mm/page-writeback.c:2355)
[ 1789.278327] write_meta_page (fs/f2fs/segment.c:922)
[ 1789.278327] f2fs_write_meta_page (arch/x86/include/asm/atomic.h:103 fs/f2fs/f2fs.h:692 fs/f2fs/checkpoint.c:172)
[ 1789.288220] trinity-c32: vm86 mode not supported on 64 bit kernel
[ 1789.278327] sync_meta_pages (fs/f2fs/checkpoint.c:247)
[ 1789.278327] ? get_parent_ip (kernel/sched/core.c:2472)
[ 1789.278327] ? preempt_count_sub (kernel/sched/core.c:2527)
[ 1789.278327] ? _raw_spin_unlock_irqrestore (arch/x86/include/asm/preempt.h:98 include/linux/spinlock_api_smp.h:161 kernel/locking/spinlock.c:191)
[ 1789.278327] write_checkpoint (fs/f2fs/checkpoint.c:875 fs/f2fs/checkpoint.c:913)
[ 1789.310911] waiting module removal not supported: please upgrade
[ 1789.278327] ? SyS_tee (fs/sync.c:77)
[ 1789.278327] ? mutex_lock_nested (arch/x86/include/asm/paravirt.h:809 kernel/locking/mutex.c:569 kernel/locking/mutex.c:587)
[ 1789.278327] ? get_parent_ip (kernel/sched/core.c:2472)
[ 1789.278327] ? bit_waitqueue (kernel/sched/wait.c:291)
[ 1789.278327] ? SyS_tee (fs/sync.c:77)
[ 1789.278327] f2fs_sync_fs (fs/f2fs/super.c:456)
[ 1789.278327] sync_fs_one_sb (fs/sync.c:80)
[ 1789.278327] iterate_supers (fs/super.c:512)
[ 1789.278327] sys_sync (fs/sync.c:110)
[ 1789.278327] tracesys (arch/x86/kernel/entry_64.S:749)
[ 1789.278327] Code: 48 c1 e0 04 48 8d 9c 07 f0 00 00 00 49 8b 45 38 41 8b 8d d0 06 00 00 8b 50 78 8b 80 84 00 00 00 d3 e0 41 39 d6 8d 44 02 ff 73 09 <0f> 0b 0f 1f 80 00 00 00 00 41 39 c6 76 03 0f 0b 90 48 8d 43 20
[ 1789.278327] RIP f2fs_submit_page_mbio (fs/f2fs/segment.h:543 fs/f2fs/data.c:181)
[ 1789.278327]  RSP <ffff880329c3fc08>

Thanks,
Sasha

------------------------------------------------------------------------------

#1 lore raw

Hi,

Thank you for the report.

If possible, could you share fsck.f2fs or dump.f2fs on the corrupted partition?

Otherwise, how about this?

# fdisk /dev/sdx
Quoted 1 line

p

Thank you,

2014-04-05 (토), 11:12 -0400, Sasha Levin:

Quoted 56 lines

Hi all,

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel I've stumbled on the following:

[ 1789.220942] kernel BUG at fs/f2fs/segment.h:543!
[ 1789.220942] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 1789.220942] Dumping ftrace buffer:
[ 1789.220942]    (ftrace buffer empty)
[ 1789.220942] Modules linked in:
[ 1789.220942] CPU: 0 PID: 28161 Comm: trinity-c0 Not tainted 3.14.0-next-20140403-sasha-00019-g7474aa9-dirty #376
[ 1789.220942] task: ffff88032a598000 ti: ffff880329c3e000 task.ti: ffff880329c3e000
[ 1789.220942] RIP: f2fs_submit_page_mbio (fs/f2fs/segment.h:543 fs/f2fs/data.c:181)
[ 1789.220942] RSP: 0018:ffff880329c3fc08  EFLAGS: 00010287
[ 1789.220942] RAX: 000000000000ffff RBX: ffff88012ac66a00 RCX: 0000000000000009
[ 1789.220942] RDX: 0000000000000200 RSI: ffffea0001dd92c0 RDI: ffff88012ac667b0
[ 1789.220942] RBP: ffff880329c3fc58 R08: 00000000000ba28e R09: ffff88032a598d98
[ 1789.220942] R10: 0000000000000000 R11: 0000000000000000 R12: ffffea0001dd92c0
[ 1789.220942] R13: ffff88012ac667b0 R14: 0000000000000000 R15: ffff880329c3fc70
[ 1789.220942] FS:  00007f3599003700(0000) GS:ffff88007dc00000(0000) knlGS:0000000000000000
[ 1789.220942] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1789.220942] CR2: 00007faf4246a018 CR3: 0000000329b6f000 CR4: 00000000000006b0
[ 1789.220942] DR0: 0000000000696000 DR1: 0000000000696000 DR2: 0000000000000000
[ 1789.261947] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
[ 1789.261947] Stack:
[ 1789.261947]  ffff88032a598000 ffff880d2a478278 0000000000000006 0000000000000082
[ 1789.261947]  ffffffffaa289475 ffffea0001dd92c0 ffff88012ac667b0 ffff880329c3fd30
[ 1789.261947]  0000000000000000 ffffea0001dd92c0 ffff880329c3fc88 ffffffffaa9da033
[ 1789.278327] Call Trace:
[ 1789.278327] ? clear_page_dirty_for_io (arch/x86/include/asm/paravirt.h:809 include/linux/backing-dev.h:176 mm/page-writeback.c:2355)
[ 1789.278327] write_meta_page (fs/f2fs/segment.c:922)
[ 1789.278327] f2fs_write_meta_page (arch/x86/include/asm/atomic.h:103 fs/f2fs/f2fs.h:692 fs/f2fs/checkpoint.c:172)
[ 1789.288220] trinity-c32: vm86 mode not supported on 64 bit kernel
[ 1789.278327] sync_meta_pages (fs/f2fs/checkpoint.c:247)
[ 1789.278327] ? get_parent_ip (kernel/sched/core.c:2472)
[ 1789.278327] ? preempt_count_sub (kernel/sched/core.c:2527)
[ 1789.278327] ? _raw_spin_unlock_irqrestore (arch/x86/include/asm/preempt.h:98 include/linux/spinlock_api_smp.h:161 kernel/locking/spinlock.c:191)
[ 1789.278327] write_checkpoint (fs/f2fs/checkpoint.c:875 fs/f2fs/checkpoint.c:913)
[ 1789.310911] waiting module removal not supported: please upgrade
[ 1789.278327] ? SyS_tee (fs/sync.c:77)
[ 1789.278327] ? mutex_lock_nested (arch/x86/include/asm/paravirt.h:809 kernel/locking/mutex.c:569 kernel/locking/mutex.c:587)
[ 1789.278327] ? get_parent_ip (kernel/sched/core.c:2472)
[ 1789.278327] ? bit_waitqueue (kernel/sched/wait.c:291)
[ 1789.278327] ? SyS_tee (fs/sync.c:77)
[ 1789.278327] f2fs_sync_fs (fs/f2fs/super.c:456)
[ 1789.278327] sync_fs_one_sb (fs/sync.c:80)
[ 1789.278327] iterate_supers (fs/super.c:512)
[ 1789.278327] sys_sync (fs/sync.c:110)
[ 1789.278327] tracesys (arch/x86/kernel/entry_64.S:749)
[ 1789.278327] Code: 48 c1 e0 04 48 8d 9c 07 f0 00 00 00 49 8b 45 38 41 8b 8d d0 06 00 00 8b 50 78 8b 80 84 00 00 00 d3 e0 41 39 d6 8d 44 02 ff 73 09 <0f> 0b 0f 1f 80 00 00 00 00 41 39 c6 76 03 0f 0b 90 48 8d 43 20
[ 1789.278327] RIP f2fs_submit_page_mbio (fs/f2fs/segment.h:543 fs/f2fs/data.c:181)
[ 1789.278327]  RSP <ffff880329c3fc08>

Thanks,
Sasha

Signature · 14 lines
Jaegeuk Kim
Samsung


------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration
Continuously Automate Build, Test & Deployment 
Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees_APR
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel
#2 lore raw
Quoted 10 lines · Jaegeuk Kim

Hi,

Thank you for the report.

If possible, could you share fsck.f2fs or dump.f2fs on the corrupted partition?

Otherwise, how about this?

# fdisk /dev/sdx
Quoted 1 line
Quoted 1 line

p

Hey Jaegeuk,

Unfortunately it was a temporary filesystem inside the fuzzer that has gone away as soon as the kernel died, so I don't have access to it.

Thanks,
Sasha

------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test & Deployment Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees_APR

#3 lore raw

2014-04-06 (일), 21:55 -0400, Sasha Levin:

Quoted 8 lines · Jaegeuk Kim
Quoted 7 lines

Hi,

Thank you for the report.

If possible, could you share fsck.f2fs or dump.f2fs on the corrupted partition?

Quoted 8 lines
Quoted 3 lines

Otherwise, how about this?

# fdisk /dev/sdx
Quoted 1 line
Quoted 1 line

p

Hey Jaegeuk,

Unfortunately it was a temporary filesystem inside the fuzzer that has gone away as soon as the kernel died, so I don't have access to it.

Got it.
Then, just for sure, if it is reproducible, could you test f2fs with the following patch?
Thanks,

Quoted 1 line

From 84f80a126458eeeaa3c4ebcecfb6908f7d22b214 Mon Sep 17 00:00:00 2001

From: Jaegeuk Kim <[email protected]>
Date: Mon, 7 Apr 2014 12:37:39 +0900
Subject: [PATCH] test

---
 fs/f2fs/segment.h | 5 +++++
 1 file changed, 5 insertions(+)
fs/f2fs/segment.h +0 -0
index 7091204..d9cf06e 100644
@@ -540,6 +540,11 @@ static inline void verify_block_addr(struct

f2fs_sb_info *sbi, block_t blk_addr)
block_t total_blks = sm_info->segment_count <<
sbi->log_blocks_per_seg;
block_t start_addr = sm_info->seg0_blkaddr; block_t end_addr = start_addr + total_blks - 1;
+ if (blk_addr < start_addr) {
+ f2fs_msg(sbi->sb, KERN_ERR,
+ "blk_addr: %ld, start: %ld, end: %ld",
+ blk_addr, start_addr, end_addr);
+ }
BUG_ON(blk_addr < start_addr); BUG_ON(blk_addr > end_addr);
}

Signature · 6 lines
1.8.4.474.g128a96c


-- 
Jaegeuk Kim
Samsung
#4 lore raw
In reply to Sasha Levin

Hi Levin,

Could you share dump code info like following one which can be generated by 'objdump -Dl f2fs.ko > obj'.
It may help us to get a clue for this problem.

verify_block_addr()
segment.h:543 (discriminator 3)

    2f35:       39 55 ec                cmp    %edx,-0x14(%ebp)
segment.h:542 (discriminator 3)
    2f38:       8d 44 02 ff             lea    -0x1(%edx,%eax,1),%eax
segment.h:543 (discriminator 3)
    2f3c:       0f 82 dc 01 00 00       jb     311e <f2fs_submit_page_mbio+0x22e>
segment.h:544
    2f42:       39 45 ec                cmp    %eax,-0x14(%ebp)
    2f45:       0f 87 d1 01 00 00       ja     311c <f2fs_submit_page_mbio+0x22c>

Thank you

Quoted 69 lines

-----Original Message-----
From: Sasha Levin [mailto:[email protected]]
Sent: Saturday, April 05, 2014 11:12 PM
To: [email protected]; [email protected]
Cc: Dave Jones; LKML; [email protected]; Dave Jones; LKML; [email protected]
Subject: [f2fs-dev] f2fs: kernel BUG at fs/f2fs/segment.h:543

Hi all,

While fuzzing with trinity inside a KVM tools guest running the latest -next kernel I've stumbled on the following:

[ 1789.220942] kernel BUG at fs/f2fs/segment.h:543!
[ 1789.220942] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 1789.220942] Dumping ftrace buffer:
[ 1789.220942]    (ftrace buffer empty)
[ 1789.220942] Modules linked in:
[ 1789.220942] CPU: 0 PID: 28161 Comm: trinity-c0 Not tainted
3.14.0-next-20140403-sasha-00019-g7474aa9-dirty #376
[ 1789.220942] task: ffff88032a598000 ti: ffff880329c3e000 task.ti: ffff880329c3e000
[ 1789.220942] RIP: f2fs_submit_page_mbio (fs/f2fs/segment.h:543 fs/f2fs/data.c:181)
[ 1789.220942] RSP: 0018:ffff880329c3fc08  EFLAGS: 00010287
[ 1789.220942] RAX: 000000000000ffff RBX: ffff88012ac66a00 RCX: 0000000000000009
[ 1789.220942] RDX: 0000000000000200 RSI: ffffea0001dd92c0 RDI: ffff88012ac667b0
[ 1789.220942] RBP: ffff880329c3fc58 R08: 00000000000ba28e R09: ffff88032a598d98
[ 1789.220942] R10: 0000000000000000 R11: 0000000000000000 R12: ffffea0001dd92c0
[ 1789.220942] R13: ffff88012ac667b0 R14: 0000000000000000 R15: ffff880329c3fc70
[ 1789.220942] FS:  00007f3599003700(0000) GS:ffff88007dc00000(0000) knlGS:0000000000000000
[ 1789.220942] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1789.220942] CR2: 00007faf4246a018 CR3: 0000000329b6f000 CR4: 00000000000006b0
[ 1789.220942] DR0: 0000000000696000 DR1: 0000000000696000 DR2: 0000000000000000
[ 1789.261947] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
[ 1789.261947] Stack:
[ 1789.261947]  ffff88032a598000 ffff880d2a478278 0000000000000006 0000000000000082
[ 1789.261947]  ffffffffaa289475 ffffea0001dd92c0 ffff88012ac667b0 ffff880329c3fd30
[ 1789.261947]  0000000000000000 ffffea0001dd92c0 ffff880329c3fc88 ffffffffaa9da033
[ 1789.278327] Call Trace:
[ 1789.278327] ? clear_page_dirty_for_io (arch/x86/include/asm/paravirt.h:809
include/linux/backing-dev.h:176 mm/page-writeback.c:2355)
[ 1789.278327] write_meta_page (fs/f2fs/segment.c:922)
[ 1789.278327] f2fs_write_meta_page (arch/x86/include/asm/atomic.h:103 fs/f2fs/f2fs.h:692
fs/f2fs/checkpoint.c:172)
[ 1789.288220] trinity-c32: vm86 mode not supported on 64 bit kernel
[ 1789.278327] sync_meta_pages (fs/f2fs/checkpoint.c:247)
[ 1789.278327] ? get_parent_ip (kernel/sched/core.c:2472)
[ 1789.278327] ? preempt_count_sub (kernel/sched/core.c:2527)
[ 1789.278327] ? _raw_spin_unlock_irqrestore (arch/x86/include/asm/preempt.h:98
include/linux/spinlock_api_smp.h:161 kernel/locking/spinlock.c:191)
[ 1789.278327] write_checkpoint (fs/f2fs/checkpoint.c:875 fs/f2fs/checkpoint.c:913)
[ 1789.310911] waiting module removal not supported: please upgrade
[ 1789.278327] ? SyS_tee (fs/sync.c:77)
[ 1789.278327] ? mutex_lock_nested (arch/x86/include/asm/paravirt.h:809 kernel/locking/mutex.c:569
kernel/locking/mutex.c:587)
[ 1789.278327] ? get_parent_ip (kernel/sched/core.c:2472)
[ 1789.278327] ? bit_waitqueue (kernel/sched/wait.c:291)
[ 1789.278327] ? SyS_tee (fs/sync.c:77)
[ 1789.278327] f2fs_sync_fs (fs/f2fs/super.c:456)
[ 1789.278327] sync_fs_one_sb (fs/sync.c:80)
[ 1789.278327] iterate_supers (fs/super.c:512)
[ 1789.278327] sys_sync (fs/sync.c:110)
[ 1789.278327] tracesys (arch/x86/kernel/entry_64.S:749)
[ 1789.278327] Code: 48 c1 e0 04 48 8d 9c 07 f0 00 00 00 49 8b 45 38 41 8b 8d d0 06 00 00 8b 50 78 8b 80
84 00 00 00 d3 e0 41 39 d6 8d 44 02 ff 73 09 <0f> 0b 0f 1f 80 00 00 00 00 41 39 c6 76 03 0f 0b 90 48 8d
43 20
[ 1789.278327] RIP f2fs_submit_page_mbio (fs/f2fs/segment.h:543 fs/f2fs/data.c:181)
[ 1789.278327]  RSP <ffff880329c3fc08>
Quoted 8 lines

Thanks,
Sasha

------------------------------------------------------------------------------
_______________________________________________
Linux-f2fs-devel mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/linux-f2fs-devel

------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test & Deployment Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees

#5 lore raw
Quoted 8 lines · Chao Yu

Hi Levin,

Could you share dump code info like following one which can be generated by 'objdump -Dl f2fs.ko > obj'.
It may help us to get a clue for this problem.

verify_block_addr()
segment.h:543 (discriminator 3)

Quoted 8 lines
    2f35:       39 55 ec                cmp    %edx,-0x14(%ebp)
segment.h:542 (discriminator 3)
    2f38:       8d 44 02 ff             lea    -0x1(%edx,%eax,1),%eax
segment.h:543 (discriminator 3)
    2f3c:       0f 82 dc 01 00 00       jb     311e <f2fs_submit_page_mbio+0x22e>
segment.h:544
    2f42:       39 45 ec                cmp    %eax,-0x14(%ebp)
    2f45:       0f 87 d1 01 00 00       ja     311c <f2fs_submit_page_mbio+0x22c>

verify_block_addr():
/home/sasha/linux-next/fs/f2fs/data.c:1079 (discriminator 3)

    f710:       49 8b 45 38             mov    0x38(%r13),%rax
/home/sasha/linux-next/fs/f2fs/segment.h:540 (discriminator 3)
    f714:       41 8b 8d d0 06 00 00    mov    0x6d0(%r13),%ecx
/home/sasha/linux-next/fs/f2fs/segment.h:541 (discriminator 3)
    f71b:       44 8b 40 78             mov    0x78(%rax),%r8d
/home/sasha/linux-next/fs/f2fs/segment.h:540 (discriminator 3)
    f71f:       8b 80 84 00 00 00       mov    0x84(%rax),%eax
    f725:       d3 e0                   shl    %cl,%eax
/home/sasha/linux-next/fs/f2fs/segment.h:543 (discriminator 3)
    f727:       45 39 c6                cmp    %r8d,%r14d
/home/sasha/linux-next/fs/f2fs/segment.h:542 (discriminator 3)
    f72a:       45 8d 4c 00 ff          lea    -0x1(%r8,%rax,1),%r9d
/home/sasha/linux-next/fs/f2fs/segment.h:543 (discriminator 3)
    f72f:       73 1f                   jae    f750 <f2fs_submit_page_mbio+0xa0>
/home/sasha/linux-next/fs/f2fs/segment.h:544
    f731:       49 8b 7d 00             mov    0x0(%r13),%rdi
    f735:       44 89 f1                mov    %r14d,%ecx
    f738:       48 c7 c2 00 00 00 00    mov    $0x0,%rdx
    f73f:       48 c7 c6 00 00 00 00    mov    $0x0,%rsi
    f746:       31 c0                   xor    %eax,%eax
    f748:       e8 00 00 00 00          callq  f74d <f2fs_submit_page_mbio+0x9d>
/home/sasha/linux-next/fs/f2fs/segment.h:547
    f74d:       0f 0b                   ud2
    f74f:       90                      nop
/home/sasha/linux-next/fs/f2fs/segment.h:550
    f750:       45 39 ce                cmp    %r9d,%r14d
    f753:       76 0b                   jbe    f760 <f2fs_submit_page_mbio+0xb0>
    f755:       0f 0b                   ud2
    f757:       66 0f 1f 84 00 00 00    nopw   0x0(%rax,%rax,1)
    f75e:       00 00

Thanks,
Sasha

------------------------------------------------------------------------------
Put Bad Developers to Shame
Dominate Development with Jenkins Continuous Integration Continuously Automate Build, Test & Deployment Start a new project now. Try Jenkins in the cloud.
http://p.sf.net/sfu/13600_Cloudbees