Linux 3.14.18 : kernel BUG at fs/btrfs/file.c:890!
Hi,
I still have kernel BUG with recent btrfs fixes :
Sep 6 20:29:12 cm kernel: [ 6987.378592] ------------[ cut here ]------------ Sep 6 20:29:12 cm kernel: [ 6987.378613] kernel BUG at fs/btrfs/file.c:890! Sep 6 20:29:12 cm kernel: [ 6987.378630] invalid opcode: 0000 [#1] SMP Sep 6 20:29:12 cm kernel: [ 6987.378647] Modules linked in: ip6table_mangle nf_conntrack_ipv6 nf_defrag_ipv6 ip6table_filter ip6_tables xt_DSCP iptable_mangle xt_LOG xt_tcpudp nf_conntrack_ipv4 nf_defrag_ipv4 xt_state iptable_filter ip_tables x_tables nf_conntrack iTCO_wdt ehci_pci iTCO_vendor_support ehci_hcd i2c_i801 evdev i2c_core lpc_ich mfd_core battery video button btrfs xor raid6_pq dm_mod raid1 md_mod sg sd_mod crc_t10dif crct10dif_common ahci libahci e1000e libata ptp scsi_mod pps_core xhci_hcd fan thermal Sep 6 20:29:12 cm kernel: [ 6987.378806] CPU: 7 PID: 9462 Comm: btrfs-endio-wri Not tainted 3.14-dae-intel #1 Sep 6 20:29:12 cm kernel: [ 6987.378835] Hardware name: Digicube sas DediCube/DQ77MK, BIOS MKQ7710H.86A.0058.2013.0226.1541 02/26/2013 Sep 6 20:29:12 cm kernel: [ 6987.378868] task: ffff8800d09d9010 ti: ffff8801f5c38000 task.ti: ffff8801f5c38000 Sep 6 20:29:12 cm kernel: [ 6987.378898] RIP: 0010:[<ffffffffa017716f>] [<ffffffffa017716f>] __btrfs_drop_extents+0x685/0x953 [btrfs] Sep 6 20:29:12 cm kernel: [ 6987.378947] RSP: 0018:ffff8801f5c39c40 EFLAGS: 00010297 Sep 6 20:29:12 cm kernel: [ 6987.378964] RAX: 0000000020a4f000 RBX: 0000000020a6f000 RCX: 0000000020a4f000 Sep 6 20:29:12 cm kernel: [ 6987.378985] RDX: 0000000000000030 RSI: ffff8801edd8b5a6 RDI: ffff8801edd8b000 Sep 6 20:29:12 cm kernel: [ 6987.379005] RBP: ffff8801da452d60 R08: 0000000000001000 R09: ffff8801f5c39c00 Sep 6 20:29:12 cm kernel: [ 6987.379026] R10: 0000000000000000 R11: 0000000000000000 R12: ffff8801f1b42470 Sep 6 20:29:12 cm kernel: [ 6987.379047] R13: 0000000000000000 R14: 00000000000005a6 R15: ffff8800d5543000 Sep 6 20:29:12 cm kernel: [ 6987.379067] FS: 0000000000000000(0000) GS:ffff88021e3c0000(0000) knlGS:0000000000000000 Sep 6 20:29:12 cm kernel: [ 6987.379097] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Sep 6 20:29:12 cm kernel: [ 6987.379115] CR2: 00000000015e06d0 CR3: 000000000160b000 CR4: 00000000001407e0 Sep 6 20:29:12 cm kernel: [ 6987.379135] Stack: Sep 6 20:29:12 cm kernel: [ 6987.379148] 0000000020800000 0000000020ea4000 0000001200000020 0000000020a4f000 Sep 6 20:29:12 cm kernel: [ 6987.379182] ffff8800c0f6eba0 00000133f2f16000 ffff8800c11035d8 0000000000000000 Sep 6 20:29:12 cm kernel: [ 6987.379214] 0000001e00000000 0000000000000001 00000000000018f6 000000000000b000 Sep 6 20:29:12 cm kernel: [ 6987.379245] Call Trace: Sep 6 20:29:12 cm kernel: [ 6987.379262] [<ffffffff810cee61>] ? ____cache_alloc+0x38/0x271 Sep 6 20:29:12 cm kernel: [ 6987.379289] [<ffffffffa016acd0>] ? insert_reserved_file_extent.constprop.60+0x85/0x269 [btrfs] Sep 6 20:29:12 cm kernel: [ 6987.379329] [<ffffffffa016f1a7>] ? btrfs_finish_ordered_io+0x21d/0x3af [btrfs] Sep 6 20:29:12 cm kernel: [ 6987.379367] [<ffffffffa018b40b>] ? worker_loop+0x175/0x4ba [btrfs] Sep 6 20:29:12 cm kernel: [ 6987.379389] [<ffffffff81368197>] ? __schedule+0x310/0x47a Sep 6 20:29:12 cm kernel: [ 6987.379415] [<ffffffffa018b296>] ? btrfs_queue_worker+0x269/0x269 [btrfs] Sep 6 20:29:12 cm kernel: [ 6987.379438] [<ffffffff81046ad3>] ? kthread+0xa7/0xaf Sep 6 20:29:12 cm kernel: [ 6987.379458] [<ffffffff81040000>] ? flush_workqueue_prep_pwqs+0xff/0x14f Sep 6 20:29:12 cm kernel: [ 6987.379480] [<ffffffff81046a2c>] ? __kthread_parkme+0x5b/0x5b Sep 6 20:29:12 cm kernel: [ 6987.379500] [<ffffffff8136eebc>] ? ret_from_fork+0x7c/0xb0 Sep 6 20:29:12 cm kernel: [ 6987.379521] [<ffffffff81046a2c>] ? __kthread_parkme+0x5b/0x5b Sep 6 20:29:12 cm kernel: [ 6987.379539] Code: 0f 82 85 01 00 00 83 7c 24 14 00 75 11 8b 7d 40 c7 44 24 14 01 00 00 00 89 7c 24 44 eb 13 8b 54 24 44 03 54 24 14 3b 55 40 74 02 <0f> 0b ff 44 24 14 83 7c 24 48 00 75 2b 80 7c 24 63 00 74 24 48 Sep 6 20:29:12 cm kernel: [ 6987.379628] RIP [<ffffffffa017716f>] __btrfs_drop_extents+0x685/0x953 [btrfs] Sep 6 20:29:12 cm kernel: [ 6987.379666] RSP <ffff8801f5c39c40> Sep 6 20:29:12 cm kernel: [ 6987.379824] ---[ end trace 88011934e4908482 ]---
Olivier
PS : I already upload image of this system on 03 september, at
https://daevel.fr/img/btrfs-image.out (5.7GB)
(md5 hash ee5559ab31368aba60c259ce3b5b9504)
Quoted 13 lines · Olivier Bonvalet
I still have kernel BUG with recent btrfs fixes :
Sep 6 20:29:12 cm kernel: [ 6987.378592] ------------[ cut here ]------------ Sep 6 20:29:12 cm kernel: [ 6987.378613] kernel BUG at fs/btrfs/file.c:890! (..snip..) Sep 6 20:29:12 cm kernel: [ 6987.379245] Call Trace: Sep 6 20:29:12 cm kernel: [ 6987.379262] [<ffffffff810cee61>] ? ____cache_alloc+0x38/0x271 Sep 6 20:29:12 cm kernel: [ 6987.379289] [<ffffffffa016acd0>] ? insert_reserved_file_extent.constprop.60+0x85/0x269 [btrfs] Sep 6 20:29:12 cm kernel: [ 6987.379329] [<ffffffffa016f1a7>] ? btrfs_finish_ordered_io+0x21d/0x3af [btrfs] Sep 6 20:29:12 cm kernel: [ 6987.379367] [<ffffffffa018b40b>] ? worker_loop+0x175/0x4ba [btrfs] Sep 6 20:29:12 cm kernel: [ 6987.379389] [<ffffffff81368197>] ? __schedule+0x310/0x47a Sep 6 20:29:12 cm kernel: [ 6987.379415] [<ffffffffa018b296>] ? btrfs_queue_worker+0x269/0x269 [btrfs] Sep 6 20:29:12 cm kernel: [ 6987.379438] [<ffffffff81046ad3>] ? kthread+0xa7/0xaf
Quoted 8 lines
Sep 6 20:29:12 cm kernel: [ 6987.379458] [<ffffffff81040000>] ? flush_workqueue_prep_pwqs+0xff/0x14f Sep 6 20:29:12 cm kernel: [ 6987.379480] [<ffffffff81046a2c>] ? __kthread_parkme+0x5b/0x5b Sep 6 20:29:12 cm kernel: [ 6987.379500] [<ffffffff8136eebc>] ? ret_from_fork+0x7c/0xb0 Sep 6 20:29:12 cm kernel: [ 6987.379521] [<ffffffff81046a2c>] ? __kthread_parkme+0x5b/0x5b Sep 6 20:29:12 cm kernel: [ 6987.379539] Code: 0f 82 85 01 00 00 83 7c 24 14 00 75 11 8b 7d 40 c7 44 24 14 01 00 00 00 89 7c 24 44 eb 13 8b 54 24 44 03 54 24 14 3b 55 40 74 02 <0f> 0b ff 44 24 14 83 7c 24 48 00 75 2b 80 7c 24 63 00 74 24 48 Sep 6 20:29:12 cm kernel: [ 6987.379628] RIP [<ffffffffa017716f>] __btrfs_drop_extents+0x685/0x953 [btrfs] Sep 6 20:29:12 cm kernel: [ 6987.379666] RSP <ffff8801f5c39c40> Sep 6 20:29:12 cm kernel: [ 6987.379824] ---[ end trace 88011934e4908482 ]---
I cannot really offer much assistance, but that particular area of code looks suspiciously like something fixed later and *not* yet backported to 3.14.8:
"Btrfs: fix leaf corruption caused by ENOSPC while hole punching"
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/btrfs?id=fc19c5e73645f95d3eca12b4e91e7b56faf1e4a4
The symptoms described in the patch sound like they might be your OOPS. Unfortunately (I think) that would mean that just installing different kernels will not fix the problem, since the problem is on disk. :/
So to me it seems your best course of action is to backup/mkfs/restore the affected partition and stick with 3.17-rc3, since that has tons more fixes.
If you really need/want 3.14.x and are not afraid of building your own kernel, you can try applying my btrfs-* patches from:
https://github.com/hhoffstaette/kernel-patches/tree/master/3.14
which apply cleanly to the latest 3.14.18-stable.
However even that will not fix a bungled tree on disk.
-h
Quoted 22 lines · Holger Hoffstätte
On Sun, 07 Sep 2014 18:39:40 +0200, Olivier Bonvalet wrote:
I cannot really offer much assistance, but that particular area of code looks suspiciously like something fixed later and *not* yet backported to 3.14.8:
"Btrfs: fix leaf corruption caused by ENOSPC while hole punching"
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/fs/btrfs?id=fc19c5e73645f95d3eca12b4e91e7b56faf1e4a4The symptoms described in the patch sound like they might be your OOPS. Unfortunately (I think) that would mean that just installing different kernels will not fix the problem, since the problem is on disk. :/
So to me it seems your best course of action is to backup/mkfs/restore the affected partition and stick with 3.17-rc3, since that has tons more fixes.
If you really need/want 3.14.x and are not afraid of building your own kernel, you can try applying my btrfs-* patches from:
https://github.com/hhoffstaette/kernel-patches/tree/master/3.14
which apply cleanly to the latest 3.14.18-stable.
Quoted 5 lines
However even that will not fix a bungled tree on disk.
-h
--
Ok, so the cause is probably solved, this is a very good news, thanks. And yes, I can easily use 3.17-rc3 kernel.
So the problem is with current data : I don't know an easy way to export/restore subvolumes & snapshots thought network, and I don't have physical access to this rent servers.
Any hope that btrfsck be updated to fix that ?
Olivier
Quoted 1 line · Olivier Bonvalet
Ok, so the cause is probably solved, this is a very good news, thanks.
I don't *know* that this is your specific problem, it just looks like it might be judging by the description in the patch.
Quoted 1 line
And yes, I can easily use 3.17-rc3 kernel.
Good.
Quoted 3 lines
So the problem is with current data : I don't know an easy way to export/restore subvolumes & snapshots thought network, and I don't have physical access to this rent servers.
That's tricky, especially the snapshots. I don't remember seeing how large the partition in question is, but is there really no place where you can store a simple tar.gz dump, nuke & recreate the partition and restore the tarball? If so I'm afraid you are what's known as "up shit creek without a paddle"..
Quoted 1 line
Any hope that btrfsck be updated to fix that ?
By default btrfsck doesn't do anything. What happens when you run it without --repair, just to see what it finds?
-h
Quoted 2 lines · Holger Hoffstätte
By default btrfsck doesn't do anything. What happens when you run it without --repair, just to see what it finds?
In fact I haven't enough memory to run it (8GB of RAM only, with a 2*3TB FS in mirroring, btrfs mirroring).
Quoted 6 lines · Olivier Bonvalet
Quoted 2 lines · Holger Hoffstätte
By default btrfsck doesn't do anything. What happens when you run it without --repair, just to see what it finds?
In fact I haven't enough memory to run it (8GB of RAM only, with a 2*3TB FS in mirroring, btrfs mirroring).
Sigh..sorry to hear that. If you have another non-btrfs partition with plenty free space, you could try:
fallocate -l 32GiB /path/swapfile (more/less depending on disk space) mkswap /path/swapfile
swapon /path/swapfile
and try the btrfsck again. It really should not use that much RAM.. :(
If that's not an option either I'f afraid I don't know anything else to try.
-h