kernel BUG at fs/btrfs/ctree.c:1145

Hello,
Since 3.7.X kernel series I noticed kernel BUG in btrfs.
I use:
Linux demo 3.9.11-dlj #1 SMP Tue Jul 23 04:45:02 CEST 2013 x86_64 AMD FX(tm)-8150 Eight-Core Processor AuthenticAMD GNU/Linux.
I notice below BUG once per day.
After the BUG, the system stop running some processes, and can't be restarted.
I can run new ssh session and watch the dmesg.
I tested 3.10.2 with te same result.
I hope you can fix it. I will switch to longterm kernel. BTRFS mount options:
defaults,compress=no,autodefrag,discard,ssd

log from this BUG below:

[65534.453769] ------------[ cut here ]------------
[65534.455361] kernel BUG at fs/btrfs/ctree.c:1145!
[65534.456919] invalid opcode: 0000 [#1] SMP 
[65534.458491] Modules linked in: hwmon_vid ipv6 cpufreq_ondemand lp ppdev 

parport_pc parport i2c_dev acpi_cpufreq mperf freq_table sp5100_tco kvm crc32_pclmul crc32c_intel ghash_clmulni_intel microcode evdev radeon ttm drm_kms_helper drm fam15h_power agpgart 8139too i2c_algo_bit i2c_piix4 ohci_hcd mii ehci_pci k10temp i2c_core ehci_hcd firewire_ohci asus_atk0110 processor shpchp firewire_core thermal_sys wmi hwmon button loop

[65534.466324] CPU 6 
[65534.466358] Pid: 1181768, comm: btrfs-endio-wri Not tainted 3.9.11-dlj #1 
System manufacturer System Product Name/M5A88-V EVO
[65534.470602] RIP: 0010:[<ffffffff814a07db>]  [<ffffffff814a07db>] 
__tree_mod_log_rewind+0x20b/0x210
[65534.472912] RSP: 0018:ffff8802a903d8e0  EFLAGS: 00010293
[65534.475221] RAX: 0000000000000001 RBX: ffff880407a47e80 RCX: ffff8802a903d888
[65534.477596] RDX: ffff880407a47e81 RSI: ffff8802a474b0a7 RDI: ffff880407a47700
[65534.480032] RBP: ffff8802a903d908 R08: 0000000000001000 R09: ffff8802a903d898
[65534.482479] R10: 000000000008b72c R11: 0000000000000000 R12: ffff8803282522a0
[65534.484914] R13: 0000000000000003 R14: ffff880407a47900 R15: 00000000000000f3
[65534.487368] FS:  00007f47c00f0700(0000) GS:ffff88041ed80000(0000) 
knlGS:0000000000000000
[65534.489894] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[65534.492437] CR2: 00007f08be7eca08 CR3: 00000003f46b6000 CR4: 
00000000000407e0
[65534.495056] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[65534.497673] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[65534.500290] Process btrfs-endio-wri (pid: 1181768, threadinfo 
ffff8802a903c000, task ffff88040bbd5400)
[65534.503012] Stack:
[65534.505696]  ffff880328111e10 ffff880328111e18 0000160000000000 
0000000000000001
[65534.508536]  ffff8803281f3ce0 ffff8802a903d988 ffffffff814a83a2 ffff880407a47900
[65534.511396]  ffff88040b680000 00000000a903d948 00000000000000f3 
ffff8803282522a0
[65534.514293] Call Trace:
[65534.517195]  [<ffffffff814a83a2>] btrfs_search_old_slot+0x542/0x8c0
[65534.520178]  [<ffffffff814ab39f>] btrfs_next_old_leaf+0xff/0x4a0
[65534.523187]  [<ffffffff8152068f>] __resolve_indirect_refs+0x2df/0x5d0
[65534.526211]  [<ffffffff815212c4>] find_parent_nodes+0x944/0xfd0
[65534.529277]  [<ffffffff814c8115>] ? join_transaction.isra.12+0x25/0x380
[65534.532373]  [<ffffffff814cd060>] ? can_nocow_odirect+0x310/0x310
[65534.535448]  [<ffffffff8152237d>] iterate_extent_inodes+0xfd/0x2a0
[65534.538555]  [<ffffffff814cd060>] ? can_nocow_odirect+0x310/0x310
[65534.541683]  [<ffffffff815225b2>] iterate_inodes_from_logical+0x92/0xb0
[65534.544846]  [<ffffffff814cae77>] record_extent_backrefs+0x77/0xf0
[65534.548038]  [<ffffffff814d5df6>] btrfs_finish_ordered_io+0x156/0x9c0
[65534.551239]  [<ffffffff814d6825>] finish_ordered_fn+0x15/0x20
[65534.554446]  [<ffffffff814f5a0e>] worker_loop+0x13e/0x500
[65534.557656]  [<ffffffff814f58d0>] ? btrfs_queue_worker+0x300/0x300
[65534.560896]  [<ffffffff81079850>] kthread+0xc0/0xd0
[65534.564137]  [<ffffffff81050000>] ? syscall32_setup_pages+0x150/0x160
[65534.567429]  [<ffffffff81079790>] ? kthread_create_on_node+0x120/0x120
[65534.570784]  [<ffffffff81b68edc>] ret_from_fork+0x7c/0xb0
[65534.574131]  [<ffffffff81079790>] ? kthread_create_on_node+0x120/0x120
[65534.577557] Code: c1 48 63 43 58 48 89 c2 48 c1 e2 05 48 8d 54 10 65 48 63 
43 2c 48 89 c6 48 c1 e6 05 48 8d 74 30 65 e8 ba 99 04 00 e9 da fe ff ff <0f> 0b 
0f 0b 90 66 66 66 66 90 55 48 b9 00 00 00 00 00 16 00 00 
[65534.585040] RIP  [<ffffffff814a07db>] __tree_mod_log_rewind+0x20b/0x210
[65534.588670]  RSP <ffff8802a903d8e0>
[65534.605795] ---[ end trace c7a43fbc947f83d6 ]---

This should have been fixed in 3.10 RC1 with the commit "Btrfs: fix tree mod log regression on root split operations". If you still see it in 3.10, apparently it isn't.

Can you please post a log of the BUG when this happens with 3.10?

The recommendation at that time was to avoid balancing the filesystem:
http://marc.info/?l=linux-btrfs&m=136558416914156&w=2

from kernel 3.10.1:

[ 2911.005736] ------------[ cut here ]------------
[ 2911.008063] kernel BUG at fs/btrfs/relocation.c:4452!
[ 2911.010344] invalid opcode: 0000 [#1] SMP 
[ 2911.012610] Modules linked in: hwmon_vid ipv6 cpufreq_ondemand lp ppdev 

parport_pc parport i2c_dev acpi_cpufreq mperf freq_table kvm crc32_pclmul crc32c_intel ghash_clmulni_intel radeon ttm drm_kms_helper drm sp5100_tco 8139too ohci_hcd ehci_pci agpgart microcode ehci_hcd xhci_hcd i2c_piix4 i2c_algo_bit evdev mii firewire_ohci i2c_core firewire_core shpchp fam15h_power k10temp processor wmi asus_atk0110 thermal_sys button hwmon loop

[ 2911.023059] CPU: 2 PID: 19328 Comm: btrfs-endio-wri Not tainted 3.10.1 #1
[ 2911.025725] Hardware name: System manufacturer System Product Name/M5A88-V 
EVO, BIOS 1202    03/16/2012
[ 2911.028534] task: ffff8803a3698720 ti: ffff88039cd0a000 task.ti: ffff88039cd0a000
[ 2911.031337] RIP: 0010:[<ffffffff814ff4c6>]  [<ffffffff814ff4c6>] 
btrfs_reloc_cow_block+0x1c6/0x200
[ 2911.034204] RSP: 0018:ffff88039cd0b928  EFLAGS: 00010246
[ 2911.037085] RAX: ffff88040b55a000 RBX: ffff88035dbd0800 RCX: ffff8803392ebcc0
[ 2911.040000] RDX: ffff880331a712c0 RSI: ffff88040b6ee800 RDI: ffff88040655d0a0
[ 2911.042902] RBP: ffff88039cd0b978 R08: ffff88039cd0b9c0 R09: 0000000000001000
[ 2911.045878] R10: 0000000000000001 R11: ffffea0010116ac0 R12: ffff88040b6ee800
[ 2911.048832] R13: ffff88040655d0a0 R14: ffff88040655d0a0 R15: ffff8803392ebcc0
[ 2911.051777] FS:  00007f4386d91700(0000) GS:ffff88041ec80000(0000) 
knlGS:0000000000000000
[ 2911.054854] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2911.057886] CR2: 00007f77439426e8 CR3: 00000003f4232000 CR4: 
00000000000407e0
[ 2911.060946] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
0000000000000000
[ 2911.064049] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 2911.067114] Stack:
[ 2911.070115]  0000000000001000 ffff88039cd0b978 ffffffff814d42bd 0000000000000001
[ 2911.073269]  0000000000000000 ffff88040b6ee800 ffff880331a712c0 ffff88040655d0a0
[ 2911.076450]  0000000000000000 ffff8803392ebcc0 ffff88039cd0ba00 ffffffff8148e489
[ 2911.079577] Call Trace:
[ 2911.082705]  [<ffffffff814d42bd>] ? copy_extent_buffer+0xad/0x110
[ 2911.085900]  [<ffffffff8148e489>] __btrfs_cow_block+0x3a9/0x500
[ 2911.089078]  [<ffffffff8148e772>] btrfs_cow_block+0x112/0x1a0
[ 2911.092276]  [<ffffffff81492486>] btrfs_search_slot+0x426/0x970
[ 2911.095484]  [<ffffffff814a7a97>] btrfs_lookup_file_extent+0x37/0x40
[ 2911.098677]  [<ffffffff814c693e>] __btrfs_drop_extents+0x11e/0xae0
[ 2911.101881]  [<ffffffff8148fa8d>] ? generic_bin_search.constprop.43+0x8d/0x1a0
[ 2911.105146]  [<ffffffff8114b131>] ? kmem_cache_alloc+0xd1/0x120
[ 2911.108352]  [<ffffffff8148d2ca>] ? btrfs_alloc_path+0x1a/0x20
[ 2911.111590]  [<ffffffff814c7db0>] btrfs_drop_extents+0x60/0x90
[ 2911.114845]  [<ffffffff814c013a>] relink_extent_backref+0x30a/0x7a0
[ 2911.118073]  [<ffffffff814b7760>] ? can_nocow_odirect+0x2f0/0x2f0
[ 2911.121330]  [<ffffffff8150bd07>] ? iterate_inodes_from_logical+0x87/0xa0
[ 2911.124600]  [<ffffffff814b5617>] ? record_extent_backrefs+0x77/0xf0
[ 2911.127872]  [<ffffffff814c07c9>] btrfs_finish_ordered_io+0x1f9/0x9a0
[ 2911.131150]  [<ffffffff814c1155>] finish_ordered_fn+0x15/0x20
[ 2911.134418]  [<ffffffff814dfd4e>] worker_loop+0x13e/0x4f0
[ 2911.137678]  [<ffffffff814dfc10>] ? btrfs_queue_worker+0x330/0x330
[ 2911.140962]  [<ffffffff8107ae60>] kthread+0xc0/0xd0
[ 2911.144250]  [<ffffffff8107ada0>] ? kthread_create_on_node+0x120/0x120
[ 2911.147557]  [<ffffffff81b72d1c>] ret_from_fork+0x7c/0xb0
[ 2911.150825]  [<ffffffff8107ada0>] ? kthread_create_on_node+0x120/0x120
[ 2911.154117] Code: 80 00 00 00 00 4c 89 e2 48 89 de 4c 89 f7 e8 d2 b7 ff ff 85 
c0 0f 84 e3 fe ff ff 0f 0b 48 83 be d7 01 00 00 f7 0f 85 76 fe ff ff <0f> 0b 48 3b 
7e 20 0f 84 0b ff ff ff 0f 0b be 6d 11 00 00 48 c7 
[ 2911.161243] RIP  [<ffffffff814ff4c6>] btrfs_reloc_cow_block+0x1c6/0x200
[ 2911.164589]  RSP <ffff88039cd0b928>
[ 2911.180851] ---[ end trace b716b32a1783bdba ]---

You are right - I use autodefrag option.
To repoduce this bug faster I did filesystem balance. I can't start new ssh session so I can't give You more details. I hope, that the log above could help you find the bug.

Kind regards
Jaroslaw Siebert

Thanks for the 3.10 log.

That's a different issue that you see on 3.10. The tree mod log issue fix is apparently working well.

This relocation.c:4452 BUG has relink_extent_backref() in the call trace which is part of the snapshot-aware defrag code. But I cannot say whether this issue is fixed after 3.10.

You could try to run btrfs-next (see <https://btrfs.wiki.kernel.org/index.php/Btrfs_source_repositories>) or 3.11RC2. If it's not fixed, a bugzilla helps to remind people to fix the issue (see <https://btrfs.wiki.kernel.org/index.php/Problem_FAQ#How_do_I_report_bugs_and_issues.3F>).