Skip to contentSkip to conversation

kernel BUG at fs/btrfs/relocation.c:1062

DISCUSSION Tomasz Chmielewski 3 participants 12 messages Dec 19, 2013 – Dec 21, 2013 Btrfs file system

lore
Share & tools
lore
#0 lore raw

Got this while running balance. Some other operations were running in parallel (rsync etc.).

[152727.584641] ------------[ cut here ]------------
[152727.584723] kernel BUG at fs/btrfs/relocation.c:1062!
[152727.584802] invalid opcode: 0000 [#1] SMP 
[152727.584943] Modules linked in: veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop video i2c_i801 i2c_core button acpi_cpufreq pcspkr ehci_pci ehci_hcd lpc_ich mfd_core ext4 crc16 jbd2 mbcache raid1 sg sd_mod r8169 mii ahci libahci libata scsi_mod
[152727.586767] CPU: 3 PID: 1540 Comm: btrfs Tainted: G        W    3.13.0-rc4 #1
[152727.586888] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
[152727.587011] task: ffff8805bedf9710 ti: ffff8805a9d9a000 task.ti: ffff8805a9d9a000
[152727.587132] RIP: 0010:[<ffffffffa02ab31f>]  [<ffffffffa02ab31f>] build_backref_tree+0xb03/0xe7a [btrfs]
[152727.587300] RSP: 0018:ffff8805a9d9b898  EFLAGS: 00010246
[152727.587379] RAX: ffff8805a9d9b900 RBX: ffff8805a9d9b920 RCX: ffff88055a72fa10
[152727.587499] RDX: ffff88055a72ff50 RSI: ffff88055a72ff80 RDI: ffff8807e8878600
[152727.587619] RBP: ffff8805a9d9b998 R08: ffff8807e8878b80 R09: 0000000000001000
[152727.587739] R10: 0000000000001000 R11: 0000160000000000 R12: 0000000000000000
[152727.587859] R13: ffff88055a72ff90 R14: ffff8805a9d9b930 R15: ffff88028e648000
[152727.587980] FS:  00007fca66eca840(0000) GS:ffff88081fac0000(0000) knlGS:0000000000000000
[152727.588101] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[152727.588180] CR2: 00007fac0ba80000 CR3: 00000007006d9000 CR4: 00000000001407e0
[152727.588300] Stack:
[152727.588374]  ffff8800611992c0 0000000000000000 ffff8806c99dd500 ffff8807e8878600
[152727.588623]  ffff88055a72fde0 ffff8806c99dd240 ffff88028e648124 ffff88028e648124
[152727.588871]  ffff88028e648120 ffff88028e648020 ffff8807ece02c60 ffff8807ece02a20
[152727.589120] Call Trace:
[152727.589205]  [<ffffffffa02ac13a>] relocate_tree_blocks+0x1c0/0x544 [btrfs]
[152727.589292]  [<ffffffffa024f2a9>] ? btrfs_release_path+0x6b/0x8a [btrfs]
[152727.589382]  [<ffffffffa02ad399>] relocate_block_group+0x23f/0x4c7 [btrfs]
[152727.589470]  [<ffffffffa02ad76f>] btrfs_relocate_block_group+0x14e/0x28d [btrfs]
[152727.589598]  [<ffffffffa028c8d2>] btrfs_relocate_chunk.isra.65+0x58/0x60e [btrfs]
[152727.589727]  [<ffffffffa029a1ff>] ? btrfs_set_lock_blocking_rw+0x89/0xb2 [btrfs]
[152727.589853]  [<ffffffffa024f1b7>] ? btrfs_set_path_blocking+0x23/0x54 [btrfs]
[152727.589979]  [<ffffffffa0253a1b>] ? btrfs_search_slot+0x72f/0x789 [btrfs]
[152727.590068]  [<ffffffffa0288be1>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[152727.590157]  [<ffffffffa028f709>] btrfs_balance+0x9fe/0xbe0 [btrfs]
[152727.590245]  [<ffffffffa02954c9>] btrfs_ioctl_balance+0x220/0x29f [btrfs]
[152727.590334]  [<ffffffffa0298a66>] btrfs_ioctl+0xfce/0x2128 [btrfs]
[152727.590417]  [<ffffffff810c98c1>] ? handle_mm_fault+0x24f/0x965
[152727.590498]  [<ffffffff810ccc11>] ? __vm_enough_memory+0x26/0x13d
[152727.590581]  [<ffffffff8110219e>] do_vfs_ioctl+0x3f7/0x441
[152727.590661]  [<ffffffff8110223a>] SyS_ioctl+0x52/0x80
[152727.590741]  [<ffffffff8138f222>] system_call_fastpath+0x16/0x1b
[152727.590821] Code: 60 71 fd 49 89 40 50 49 89 40 58 49 8b 40 58 4d 89 68 58 49 83 c0 50 4d 89 45 00 49 89 45 08 4c 89 28 e9 aa 00 00 00 a8 10 75 02 <0f> 0b 83 e0 01 41 39 c4 74 02 0f 0b 45 85 e4 75 32 49 8b 70 18 
[152727.593073] RIP  [<ffffffffa02ab31f>] build_backref_tree+0xb03/0xe7a [btrfs]
[152727.593194]  RSP <ffff8805a9d9b898>
[152727.593290] ---[ end trace b67170e8ece9f591 ]---
Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#1 lore raw

Hello Tomasz,

This seems a known bug that has been fixed by josef, what is your kernel version?

Thanks,
Wang

Quoted 47 lines · Tomasz Chmielewski

Got this while running balance. Some other operations were running in parallel (rsync etc.).

[152727.584641] ------------[ cut here ]------------
[152727.584723] kernel BUG at fs/btrfs/relocation.c:1062!
[152727.584802] invalid opcode: 0000 [#1] SMP
[152727.584943] Modules linked in: veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop video i2c_i801 i2c_core button acpi_cpufreq pcspkr ehci_pci ehci_hcd lpc_ich mfd_core ext4 crc16 jbd2 mbcache raid1 sg sd_mod r8169 mii ahci libahci libata scsi_mod
[152727.586767] CPU: 3 PID: 1540 Comm: btrfs Tainted: G        W    3.13.0-rc4 #1
[152727.586888] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
[152727.587011] task: ffff8805bedf9710 ti: ffff8805a9d9a000 task.ti: ffff8805a9d9a000
[152727.587132] RIP: 0010:[<ffffffffa02ab31f>]  [<ffffffffa02ab31f>] build_backref_tree+0xb03/0xe7a [btrfs]
[152727.587300] RSP: 0018:ffff8805a9d9b898  EFLAGS: 00010246
[152727.587379] RAX: ffff8805a9d9b900 RBX: ffff8805a9d9b920 RCX: ffff88055a72fa10
[152727.587499] RDX: ffff88055a72ff50 RSI: ffff88055a72ff80 RDI: ffff8807e8878600
[152727.587619] RBP: ffff8805a9d9b998 R08: ffff8807e8878b80 R09: 0000000000001000
[152727.587739] R10: 0000000000001000 R11: 0000160000000000 R12: 0000000000000000
[152727.587859] R13: ffff88055a72ff90 R14: ffff8805a9d9b930 R15: ffff88028e648000
[152727.587980] FS:  00007fca66eca840(0000) GS:ffff88081fac0000(0000) knlGS:0000000000000000
[152727.588101] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[152727.588180] CR2: 00007fac0ba80000 CR3: 00000007006d9000 CR4: 00000000001407e0
[152727.588300] Stack:
[152727.588374]  ffff8800611992c0 0000000000000000 ffff8806c99dd500 ffff8807e8878600
[152727.588623]  ffff88055a72fde0 ffff8806c99dd240 ffff88028e648124 ffff88028e648124
[152727.588871]  ffff88028e648120 ffff88028e648020 ffff8807ece02c60 ffff8807ece02a20
[152727.589120] Call Trace:
[152727.589205]  [<ffffffffa02ac13a>] relocate_tree_blocks+0x1c0/0x544 [btrfs]
[152727.589292]  [<ffffffffa024f2a9>] ? btrfs_release_path+0x6b/0x8a [btrfs]
[152727.589382]  [<ffffffffa02ad399>] relocate_block_group+0x23f/0x4c7 [btrfs]
[152727.589470]  [<ffffffffa02ad76f>] btrfs_relocate_block_group+0x14e/0x28d [btrfs]
[152727.589598]  [<ffffffffa028c8d2>] btrfs_relocate_chunk.isra.65+0x58/0x60e [btrfs]
[152727.589727]  [<ffffffffa029a1ff>] ? btrfs_set_lock_blocking_rw+0x89/0xb2 [btrfs]
[152727.589853]  [<ffffffffa024f1b7>] ? btrfs_set_path_blocking+0x23/0x54 [btrfs]
[152727.589979]  [<ffffffffa0253a1b>] ? btrfs_search_slot+0x72f/0x789 [btrfs]
[152727.590068]  [<ffffffffa0288be1>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[152727.590157]  [<ffffffffa028f709>] btrfs_balance+0x9fe/0xbe0 [btrfs]
[152727.590245]  [<ffffffffa02954c9>] btrfs_ioctl_balance+0x220/0x29f [btrfs]
[152727.590334]  [<ffffffffa0298a66>] btrfs_ioctl+0xfce/0x2128 [btrfs]
[152727.590417]  [<ffffffff810c98c1>] ? handle_mm_fault+0x24f/0x965
[152727.590498]  [<ffffffff810ccc11>] ? __vm_enough_memory+0x26/0x13d
[152727.590581]  [<ffffffff8110219e>] do_vfs_ioctl+0x3f7/0x441
[152727.590661]  [<ffffffff8110223a>] SyS_ioctl+0x52/0x80
[152727.590741]  [<ffffffff8138f222>] system_call_fastpath+0x16/0x1b
[152727.590821] Code: 60 71 fd 49 89 40 50 49 89 40 58 49 8b 40 58 4d 89 68 58 49 83 c0 50 4d 89 45 00 49 89 45 08 4c 89 28 e9 aa 00 00 00 a8 10 75 02 <0f> 0b 83 e0 01 41 39 c4 74 02 0f 0b 45 85 e4 75 32 49 8b 70 18
[152727.593073] RIP  [<ffffffffa02ab31f>] build_backref_tree+0xb03/0xe7a [btrfs]
[152727.593194]  RSP <ffff8805a9d9b898>
[152727.593290] ---[ end trace b67170e8ece9f591 ]---
#2 lore raw

It was 3.13.0-rc4.

Quoted 77 lines · On Thu, 19 Dec 2013 20:14:17 +0800 Wang Shilong <[email protected]>…

Hello Tomasz,

This seems a known bug that has been fixed by josef, what is your kernel version?

Thanks,
Wang

Quoted 67 lines · Tomasz Chmielewski

Got this while running balance. Some other operations were running in parallel (rsync etc.).

[152727.584641] ------------[ cut here ]------------
[152727.584723] kernel BUG at fs/btrfs/relocation.c:1062!
[152727.584802] invalid opcode: 0000 [#1] SMP
[152727.584943] Modules linked in: veth ipt_MASQUERADE iptable_nat

nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop video i2c_i801 i2c_core button acpi_cpufreq pcspkr ehci_pci ehci_hcd lpc_ich mfd_core ext4 crc16 jbd2 mbcache raid1 sg sd_mod r8169 mii ahci libahci libata scsi_mod

[152727.586767] CPU: 3 PID: 1540 Comm: btrfs Tainted: G        W
3.13.0-rc4 #1 [152727.586888] Hardware name: System manufacturer
System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
[152727.587011] task: ffff8805bedf9710 ti: ffff8805a9d9a000

task.ti: ffff8805a9d9a000 [152727.587132] RIP:
0010:[<ffffffffa02ab31f>] [<ffffffffa02ab31f>] build_backref_tree+0xb03/0xe7a [btrfs] [152727.587300] RSP:
0018:ffff8805a9d9b898 EFLAGS: 00010246 [152727.587379] RAX:
ffff8805a9d9b900 RBX: ffff8805a9d9b920 RCX: ffff88055a72fa10

[152727.587499] RDX: ffff88055a72ff50 RSI: ffff88055a72ff80 RDI:
ffff8807e8878600 [152727.587619] RBP: ffff8805a9d9b998 R08:
ffff8807e8878b80 R09: 0000000000001000 [152727.587739] R10:
0000000000001000 R11: 0000160000000000 R12: 0000000000000000
[152727.587859] R13: ffff88055a72ff90 R14: ffff8805a9d9b930 R15:
ffff88028e648000 [152727.587980] FS:  00007fca66eca840(0000)
GS:ffff88081fac0000(0000) knlGS:0000000000000000 [152727.588101]
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [152727.588180]
CR2: 00007fac0ba80000 CR3: 00000007006d9000 CR4: 00000000001407e0
[152727.588300] Stack: [152727.588374]  ffff8800611992c0
0000000000000000 ffff8806c99dd500 ffff8807e8878600 [152727.588623]
ffff88055a72fde0 ffff8806c99dd240 ffff88028e648124 ffff88028e648124
[152727.588871]  ffff88028e648120 ffff88028e648020 ffff8807ece02c60
ffff8807ece02a20 [152727.589120] Call Trace: [152727.589205]
[<ffffffffa02ac13a>] relocate_tree_blocks+0x1c0/0x544 [btrfs]
[152727.589292]  [<ffffffffa024f2a9>] ?
btrfs_release_path+0x6b/0x8a [btrfs] [152727.589382]
[<ffffffffa02ad399>] relocate_block_group+0x23f/0x4c7 [btrfs]
[152727.589470]  [<ffffffffa02ad76f>]
btrfs_relocate_block_group+0x14e/0x28d [btrfs] [152727.589598]
[<ffffffffa028c8d2>] btrfs_relocate_chunk.isra.65+0x58/0x60e
[btrfs] [152727.589727]  [<ffffffffa029a1ff>] ?
btrfs_set_lock_blocking_rw+0x89/0xb2 [btrfs] [152727.589853]
[<ffffffffa024f1b7>] ? btrfs_set_path_blocking+0x23/0x54 [btrfs]
[152727.589979]  [<ffffffffa0253a1b>] ?
btrfs_search_slot+0x72f/0x789 [btrfs] [152727.590068]
[<ffffffffa0288be1>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[152727.590157]  [<ffffffffa028f709>] btrfs_balance+0x9fe/0xbe0
[btrfs] [152727.590245]  [<ffffffffa02954c9>]
btrfs_ioctl_balance+0x220/0x29f [btrfs] [152727.590334]
[<ffffffffa0298a66>] btrfs_ioctl+0xfce/0x2128 [btrfs]
[152727.590417]  [<ffffffff810c98c1>] ? handle_mm_fault+0x24f/0x965
[152727.590498]  [<ffffffff810ccc11>] ?
__vm_enough_memory+0x26/0x13d [152727.590581]  [<ffffffff8110219e>]
do_vfs_ioctl+0x3f7/0x441 [152727.590661]  [<ffffffff8110223a>]
SyS_ioctl+0x52/0x80 [152727.590741]  [<ffffffff8138f222>]
system_call_fastpath+0x16/0x1b [152727.590821] Code: 60 71 fd 49 89

40 50 49 89 40 58 49 8b 40 58 4d 89 68 58 49 83 c0 50 4d 89 45 00 49 89 45 08 4c 89 28 e9 aa 00 00 00 a8 10 75 02 <0f> 0b 83 e0 01 41 39 c4 74 02 0f 0b 45 85 e4 75 32 49 8b 70 18 [152727.593073] RIP [<ffffffffa02ab31f>] build_backref_tree+0xb03/0xe7a [btrfs]

[152727.593194]  RSP <ffff8805a9d9b898> [152727.593290] ---[ end

trace b67170e8ece9f591 ]---

#3 lore raw
Quoted 1 line · Tomasz Chmielewski

It was 3.13.0-rc4.

I take a look at line 1062, this should be a new bug!!!

Thanks,
Wang

Quoted 86 lines
Quoted 76 lines · On Thu, 19 Dec 2013 20:14:17 +0800 Wang Shilong <[email protected]>…

Hello Tomasz,

This seems a known bug that has been fixed by josef, what is your kernel version?

Thanks,
Wang

Quoted 67 lines · Tomasz Chmielewski

Got this while running balance. Some other operations were running in parallel (rsync etc.).

[152727.584641] ------------[ cut here ]------------
[152727.584723] kernel BUG at fs/btrfs/relocation.c:1062!
[152727.584802] invalid opcode: 0000 [#1] SMP
[152727.584943] Modules linked in: veth ipt_MASQUERADE iptable_nat

nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop video i2c_i801 i2c_core button acpi_cpufreq pcspkr ehci_pci ehci_hcd lpc_ich mfd_core ext4 crc16 jbd2 mbcache raid1 sg sd_mod r8169 mii ahci libahci libata scsi_mod

[152727.586767] CPU: 3 PID: 1540 Comm: btrfs Tainted: G        W
3.13.0-rc4 #1 [152727.586888] Hardware name: System manufacturer
System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
[152727.587011] task: ffff8805bedf9710 ti: ffff8805a9d9a000

task.ti: ffff8805a9d9a000 [152727.587132] RIP:
0010:[<ffffffffa02ab31f>] [<ffffffffa02ab31f>] build_backref_tree+0xb03/0xe7a [btrfs] [152727.587300] RSP:
0018:ffff8805a9d9b898 EFLAGS: 00010246 [152727.587379] RAX:
ffff8805a9d9b900 RBX: ffff8805a9d9b920 RCX: ffff88055a72fa10

[152727.587499] RDX: ffff88055a72ff50 RSI: ffff88055a72ff80 RDI:
ffff8807e8878600 [152727.587619] RBP: ffff8805a9d9b998 R08:
ffff8807e8878b80 R09: 0000000000001000 [152727.587739] R10:
0000000000001000 R11: 0000160000000000 R12: 0000000000000000
[152727.587859] R13: ffff88055a72ff90 R14: ffff8805a9d9b930 R15:
ffff88028e648000 [152727.587980] FS:  00007fca66eca840(0000)
GS:ffff88081fac0000(0000) knlGS:0000000000000000 [152727.588101]
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [152727.588180]
CR2: 00007fac0ba80000 CR3: 00000007006d9000 CR4: 00000000001407e0
[152727.588300] Stack: [152727.588374]  ffff8800611992c0
0000000000000000 ffff8806c99dd500 ffff8807e8878600 [152727.588623]
ffff88055a72fde0 ffff8806c99dd240 ffff88028e648124 ffff88028e648124
[152727.588871]  ffff88028e648120 ffff88028e648020 ffff8807ece02c60
ffff8807ece02a20 [152727.589120] Call Trace: [152727.589205]
[<ffffffffa02ac13a>] relocate_tree_blocks+0x1c0/0x544 [btrfs]
[152727.589292]  [<ffffffffa024f2a9>] ?
btrfs_release_path+0x6b/0x8a [btrfs] [152727.589382]
[<ffffffffa02ad399>] relocate_block_group+0x23f/0x4c7 [btrfs]
[152727.589470]  [<ffffffffa02ad76f>]
btrfs_relocate_block_group+0x14e/0x28d [btrfs] [152727.589598]
[<ffffffffa028c8d2>] btrfs_relocate_chunk.isra.65+0x58/0x60e
[btrfs] [152727.589727]  [<ffffffffa029a1ff>] ?
btrfs_set_lock_blocking_rw+0x89/0xb2 [btrfs] [152727.589853]
[<ffffffffa024f1b7>] ? btrfs_set_path_blocking+0x23/0x54 [btrfs]
[152727.589979]  [<ffffffffa0253a1b>] ?
btrfs_search_slot+0x72f/0x789 [btrfs] [152727.590068]
[<ffffffffa0288be1>] ? free_extent_buffer+0x6f/0x7c [btrfs]
[152727.590157]  [<ffffffffa028f709>] btrfs_balance+0x9fe/0xbe0
[btrfs] [152727.590245]  [<ffffffffa02954c9>]
btrfs_ioctl_balance+0x220/0x29f [btrfs] [152727.590334]
[<ffffffffa0298a66>] btrfs_ioctl+0xfce/0x2128 [btrfs]
[152727.590417]  [<ffffffff810c98c1>] ? handle_mm_fault+0x24f/0x965
[152727.590498]  [<ffffffff810ccc11>] ?
__vm_enough_memory+0x26/0x13d [152727.590581]  [<ffffffff8110219e>]
do_vfs_ioctl+0x3f7/0x441 [152727.590661]  [<ffffffff8110223a>]
SyS_ioctl+0x52/0x80 [152727.590741]  [<ffffffff8138f222>]
system_call_fastpath+0x16/0x1b [152727.590821] Code: 60 71 fd 49 89

40 50 49 89 40 58 49 8b 40 58 4d 89 68 58 49 83 c0 50 4d 89 45 00 49 89 45 08 4c 89 28 e9 aa 00 00 00 a8 10 75 02 <0f> 0b 83 e0 01 41 39 c4 74 02 0f 0b 45 85 e4 75 32 49 8b 70 18 [152727.593073] RIP [<ffffffffa02ab31f>] build_backref_tree+0xb03/0xe7a [btrfs]

[152727.593194]  RSP <ffff8805a9d9b898> [152727.593290] ---[ end

trace b67170e8ece9f591 ]---

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw

If it matters, I had to hard reboot after that bug; the balance continued after the system booted again and I got this a while later (filesystem was remounted read only):

[ 1781.321219] btrfs: found 232188 extents
[ 1781.994796] btrfs: relocating block group 3443586498560 flags 20
[ 2603.422490] btrfs: found 203955 extents
[ 2606.188826] btrfs: relocating block group 3051670732800 flags 20
[ 2806.720510] BTRFS debug (device sdb5): run_one_delayed_ref returned -28
[ 2806.720513] ------------[ cut here ]------------
[ 2806.720530] WARNING: CPU: 1 PID: 2359 at fs/btrfs/super.c:254 __btrfs_abort_transaction+0x4d/0xff [btrfs]()
[ 2806.720544] btrfs: Transaction aborted (error -28)
[ 2806.720544] Modules linked in: veth ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr lpc_ich mfd_core i2c_i801 i2c_core button acpi_cpufreq ehci_pci video ehci_hcd ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii
[ 2806.720626] CPU: 1 PID: 2359 Comm: btrfs-transacti Not tainted 3.13.0-rc4 #1
[ 2806.720636] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
[ 2806.720650]  0000000000000009 ffff8807ed73dc48 ffffffff8138a37d 0000000000000006
[ 2806.720705]  ffff8807ed73dc98 ffff8807ed73dc88 ffffffff810370a9 ffff8807ed73dd80
[ 2806.720797]  ffffffffa0227524 00000000ffffffe4 ffff8807f3389000 ffff8807efad4f00
[ 2806.720889] Call Trace:
[ 2806.720935]  [<ffffffff8138a37d>] dump_stack+0x46/0x58
[ 2806.720986]  [<ffffffff810370a9>] warn_slowpath_common+0x77/0x91
[ 2806.721005] BTRFS error (device sdb5) in __btrfs_free_extent:5783: errno=-28 No space left
[ 2806.721006] BTRFS info (device sdb5): forced readonly
[ 2806.721007] BTRFS debug (device sdb5): run_one_delayed_ref returned -28
[ 2806.721008] BTRFS error (device sdb5) in btrfs_run_delayed_refs:2730: errno=-28 No space left
[ 2806.721276]  [<ffffffffa0227524>] ? __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 2806.721372]  [<ffffffff81037157>] warn_slowpath_fmt+0x41/0x43
[ 2806.721426]  [<ffffffffa0227524>] __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 2806.721482]  [<ffffffffa023c6ed>] btrfs_run_delayed_refs+0x253/0x46f [btrfs]
[ 2806.721538]  [<ffffffffa0249aef>] btrfs_commit_transaction+0x70/0x7df [btrfs]
[ 2806.721593]  [<ffffffffa0248345>] transaction_kthread+0xef/0x1c2 [btrfs]
[ 2806.721646]  [<ffffffffa0248256>] ? open_ctree+0x1ac7/0x1ac7 [btrfs]
[ 2806.721697]  [<ffffffff8104ee9a>] kthread+0xcd/0xd5
[ 2806.721744]  [<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 2806.721794]  [<ffffffff8138f17c>] ret_from_fork+0x7c/0xb0
[ 2806.721844]  [<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 2806.721893] ---[ end trace 5d2cc0a807b9d02a ]---
[ 2806.721942] BTRFS error (device sdb5) in btrfs_run_delayed_refs:2730: errno=-28 No space left
Signature · 5 lines
Tomasz Chmielewski
http://wpkg.org


On Thu, 19 Dec 2013 20:49:04 +0800
Quoted 96 lines · Wang Shilong
Quoted 1 line · Tomasz Chmielewski

It was 3.13.0-rc4.

I take a look at line 1062, this should be a new bug!!!

Thanks,
Wang

Quoted 89 lines
Quoted 79 lines · On Thu, 19 Dec 2013 20:14:17 +0800 Wang Shilong <[email protected]>…

Hello Tomasz,

This seems a known bug that has been fixed by josef, what is your kernel version?

Thanks,
Wang

Quoted 70 lines · Tomasz Chmielewski

Got this while running balance. Some other operations were running in parallel (rsync etc.).

[152727.584641] ------------[ cut here ]------------
[152727.584723] kernel BUG at fs/btrfs/relocation.c:1062!
[152727.584802] invalid opcode: 0000 [#1] SMP
[152727.584943] Modules linked in: veth ipt_MASQUERADE iptable_nat

nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop video i2c_i801 i2c_core button acpi_cpufreq pcspkr ehci_pci ehci_hcd lpc_ich mfd_core ext4 crc16 jbd2 mbcache raid1 sg sd_mod r8169 mii ahci libahci libata scsi_mod [152727.586767] CPU: 3 PID: 1540 Comm: btrfs Tainted:
G W 3.13.0-rc4 #1 [152727.586888] Hardware name: System
manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013

[152727.587011] task: ffff8805bedf9710 ti: ffff8805a9d9a000

task.ti: ffff8805a9d9a000 [152727.587132] RIP:
0010:[<ffffffffa02ab31f>] [<ffffffffa02ab31f>] build_backref_tree+0xb03/0xe7a [btrfs] [152727.587300] RSP:
0018:ffff8805a9d9b898 EFLAGS: 00010246 [152727.587379] RAX:
ffff8805a9d9b900 RBX: ffff8805a9d9b920 RCX: ffff88055a72fa10

[152727.587499] RDX: ffff88055a72ff50 RSI: ffff88055a72ff80 RDI:
ffff8807e8878600 [152727.587619] RBP: ffff8805a9d9b998 R08:
ffff8807e8878b80 R09: 0000000000001000 [152727.587739] R10:
0000000000001000 R11: 0000160000000000 R12: 0000000000000000
[152727.587859] R13: ffff88055a72ff90 R14: ffff8805a9d9b930 R15:
ffff88028e648000 [152727.587980] FS:  00007fca66eca840(0000)
GS:ffff88081fac0000(0000) knlGS:0000000000000000 [152727.588101]
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [152727.588180]
CR2: 00007fac0ba80000 CR3: 00000007006d9000 CR4: 00000000001407e0
[152727.588300] Stack: [152727.588374]  ffff8800611992c0

0000000000000000 ffff8806c99dd500 ffff8807e8878600 [152727.588623] ffff88055a72fde0 ffff8806c99dd240 ffff88028e648124 ffff88028e648124 [152727.588871] ffff88028e648120 ffff88028e648020 ffff8807ece02c60 ffff8807ece02a20

[152727.589120] Call Trace: [152727.589205] [<ffffffffa02ac13a>]
relocate_tree_blocks+0x1c0/0x544 [btrfs] [152727.589292]
[<ffffffffa024f2a9>] ? btrfs_release_path+0x6b/0x8a [btrfs]
[152727.589382] [<ffffffffa02ad399>]
relocate_block_group+0x23f/0x4c7 [btrfs] [152727.589470]
[<ffffffffa02ad76f>] btrfs_relocate_block_group+0x14e/0x28d
[btrfs] [152727.589598] [<ffffffffa028c8d2>]
btrfs_relocate_chunk.isra.65+0x58/0x60e [btrfs] [152727.589727]
[<ffffffffa029a1ff>] ? btrfs_set_lock_blocking_rw+0x89/0xb2
[btrfs] [152727.589853] [<ffffffffa024f1b7>] ?
btrfs_set_path_blocking+0x23/0x54 [btrfs] [152727.589979]
[<ffffffffa0253a1b>] ? btrfs_search_slot+0x72f/0x789 [btrfs]
[152727.590068] [<ffffffffa0288be1>] ?
free_extent_buffer+0x6f/0x7c [btrfs] [152727.590157]
[<ffffffffa028f709>] btrfs_balance+0x9fe/0xbe0 [btrfs]
[152727.590245]  [<ffffffffa02954c9>]
btrfs_ioctl_balance+0x220/0x29f [btrfs] [152727.590334]
[<ffffffffa0298a66>] btrfs_ioctl+0xfce/0x2128 [btrfs]
[152727.590417]  [<ffffffff810c98c1>] ?
handle_mm_fault+0x24f/0x965 [152727.590498]
[<ffffffff810ccc11>] ? __vm_enough_memory+0x26/0x13d
[152727.590581]  [<ffffffff8110219e>] do_vfs_ioctl+0x3f7/0x441
[152727.590661]  [<ffffffff8110223a>] SyS_ioctl+0x52/0x80
[152727.590741]  [<ffffffff8138f222>]

system_call_fastpath+0x16/0x1b [152727.590821] Code: 60 71 fd 49 89 40 50 49 89 40 58 49 8b 40 58 4d 89 68 58 49 83 c0 50 4d 89 45 00 49 89 45 08 4c 89 28 e9 aa 00 00 00 a8 10 75 02 <0f> 0b 83 e0 01 41 39 c4 74 02 0f 0b 45 85 e4 75 32 49 8b 70 18

[152727.593073] RIP [<ffffffffa02ab31f>]

build_backref_tree+0xb03/0xe7a [btrfs] [152727.593194] RSP <ffff8805a9d9b898> [152727.593290] ---[ end trace b67170e8ece9f591 ]---

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#5 lore raw
Quoted 3 lines · On Thu, 19 Dec 2013 22:07:37 +0900 Tomasz Chmielewski <[email protected]> wrote:

If it matters, I had to hard reboot after that bug; the balance continued after the system booted again and I got this a while later (filesystem was remounted read only):

Actually, looks like the fs is quite hosed now :(

If I want to do an operation like removing a snapshot (btrfs sub del ...), the command returns, there is IO for a few minutes, but then, the kernel complains and fs is remounted readonly:

[ 5111.773900] BTRFS debug (device sdb5): run_one_delayed_ref returned -17
[ 5111.773902] ------------[ cut here ]------------
[ 5111.773957] WARNING: CPU: 0 PID: 15042 at fs/btrfs/super.c:254 __btrfs_abort_transaction+0x4d/0xff [btrfs]()
[ 5111.774047] btrfs: Transaction aborted (error -17)
[ 5111.774048] Modules linked in: ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr button video ehci_pci ehci_hcd acpi_cpufreq i2c_i801 i2c_core lpc_ich mfd_core ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii
[ 5111.774355] CPU: 0 PID: 15042 Comm: btrfs-transacti Tainted: G        W    3.13.0-rc4 #1
[ 5111.774443] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
[ 5111.774533]  0000000000000009 ffff8800374ddc48 ffffffff8138a37d 0000000000000006
[ 5111.774622]  ffff8800374ddc98 ffff8800374ddc88 ffffffff810370a9 ffff8800374ddd80
[ 5111.774711]  ffffffffa020d524 00000000ffffffef ffff8807ead7d800 ffff8807ff0cc8c0
[ 5111.774800] Call Trace:
[ 5111.774846]  [<ffffffff8138a37d>] dump_stack+0x46/0x58
[ 5111.774894]  [<ffffffff810370a9>] warn_slowpath_common+0x77/0x91
[ 5111.774944]  [<ffffffffa020d524>] ? __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 5111.775032]  [<ffffffff81037157>] warn_slowpath_fmt+0x41/0x43
[ 5111.775081]  [<ffffffffa020d524>] __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 5111.775135]  [<ffffffffa02226ed>] btrfs_run_delayed_refs+0x253/0x46f [btrfs]
[ 5111.775189]  [<ffffffffa022fdec>] btrfs_commit_transaction+0x36d/0x7df [btrfs]
[ 5111.775281]  [<ffffffffa022e345>] transaction_kthread+0xef/0x1c2 [btrfs]
[ 5111.775333]  [<ffffffffa022e256>] ? open_ctree+0x1ac7/0x1ac7 [btrfs]
[ 5111.775382]  [<ffffffff8104ee9a>] kthread+0xcd/0xd5
[ 5111.775428]  [<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 5111.775477]  [<ffffffff8138f17c>] ret_from_fork+0x7c/0xb0
[ 5111.775524]  [<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 5111.775572] ---[ end trace b552aca9a0cff3cb ]---
[ 5111.775618] BTRFS error (device sdb5) in btrfs_run_delayed_refs:2730: errno=-17 Object already exists
[ 5111.775707] BTRFS info (device sdb5): forced readonly
[ 5111.775754] BTRFS warning (device sdb5): Skipping commit of aborted transaction.
[ 5111.775841] BTRFS error (device sdb5) in cleanup_transaction:1553: errno=-17 Object already exists

If I run balance (after unmounting and mounting the filesystem), it ends in a similar way:

[ 5927.338989] btrfs: relocating block group 4647284637696 flags 17
[ 6184.333629] btrfs: found 983 extents
[ 6269.512577] BTRFS debug (device sdb5): run_one_delayed_ref returned -17
[ 6269.512579] ------------[ cut here ]------------
[ 6269.512636] WARNING: CPU: 7 PID: 17836 at fs/btrfs/super.c:254 __btrfs_abort_transaction+0x4d/0xff [btrfs]()
[ 6269.512727] btrfs: Transaction aborted (error -17)
[ 6269.512727] Modules linked in: ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr button video ehci_pci ehci_hcd acpi_cpufreq i2c_i801 i2c_core lpc_ich mfd_core ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii
[ 6269.513036] CPU: 7 PID: 17836 Comm: btrfs-transacti Tainted: G        W    3.13.0-rc4 #1
[ 6269.513124] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
[ 6269.513214]  0000000000000009 ffff8807c5b07c48 ffffffff8138a37d 0000000000000006
[ 6269.513304]  ffff8807c5b07c98 ffff8807c5b07c88 ffffffff810370a9 ffff8807c5b07d80
[ 6269.513393]  ffffffffa020d524 00000000ffffffef ffff8807ead7e800 ffff8800371eedc0
[ 6269.513483] Call Trace:
[ 6269.513529]  [<ffffffff8138a37d>] dump_stack+0x46/0x58
[ 6269.513577]  [<ffffffff810370a9>] warn_slowpath_common+0x77/0x91
[ 6269.513627]  [<ffffffffa020d524>] ? __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 6269.513716]  [<ffffffff81037157>] warn_slowpath_fmt+0x41/0x43
[ 6269.513766]  [<ffffffffa020d524>] __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 6269.513820]  [<ffffffffa02226ed>] btrfs_run_delayed_refs+0x253/0x46f [btrfs]
[ 6269.513874]  [<ffffffffa022faef>] btrfs_commit_transaction+0x70/0x7df [btrfs]
[ 6269.513927]  [<ffffffffa022e345>] transaction_kthread+0xef/0x1c2 [btrfs]
[ 6269.513980]  [<ffffffffa022e256>] ? open_ctree+0x1ac7/0x1ac7 [btrfs]
[ 6269.514029]  [<ffffffff8104ee9a>] kthread+0xcd/0xd5
[ 6269.514076]  [<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 6269.514125]  [<ffffffff8138f17c>] ret_from_fork+0x7c/0xb0
[ 6269.514172]  [<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 6269.514220] ---[ end trace b552aca9a0cff3cc ]---
[ 6269.514266] BTRFS error (device sdb5) in btrfs_run_delayed_refs:2730: errno=-17 Object already exists
[ 6269.514356] BTRFS info (device sdb5): forced readonly
[ 6269.514430] BTRFS warning (device sdb5): Skipping commit of aborted transaction.
[ 6269.514559] BTRFS error (device sdb5) in cleanup_transaction:1553: errno=-17 Object already exists
[ 6269.527034] btrfs: found 983 extents

Is there any fix I can try?

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#6 lore raw

Hello Tomasz,

There should be some bugs flying on, unitil now, you can try to remount:
something like:

# mount  /dev/sda /mnt -o  skip_balance

It will skip unfinshed balance, hopley it can help you before we give a bug fix.

Thanks,
Wang
2013/12/20 Tomasz Chmielewski <[email protected]>:

Quoted 94 lines
Quoted 3 lines · On Thu, 19 Dec 2013 22:07:37 +0900 Tomasz Chmielewski <[email protected]> wrote:

If it matters, I had to hard reboot after that bug; the balance continued after the system booted again and I got this a while later (filesystem was remounted read only):

Actually, looks like the fs is quite hosed now :(

If I want to do an operation like removing a snapshot (btrfs sub del ...), the command returns, there is IO for a few minutes, but then, the kernel complains and fs is remounted readonly:

[ 5111.773900] BTRFS debug (device sdb5): run_one_delayed_ref returned -17
[ 5111.773902] ------------[ cut here ]------------
[ 5111.773957] WARNING: CPU: 0 PID: 15042 at fs/btrfs/super.c:254 __btrfs_abort_transaction+0x4d/0xff [btrfs]()
[ 5111.774047] btrfs: Transaction aborted (error -17)
[ 5111.774048] Modules linked in: ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr button video ehci_pci ehci_hcd acpi_cpufreq i2c_i801 i2c_core lpc_ich mfd_core ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii
[ 5111.774355] CPU: 0 PID: 15042 Comm: btrfs-transacti Tainted: G        W    3.13.0-rc4 #1
[ 5111.774443] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
[ 5111.774533]  0000000000000009 ffff8800374ddc48 ffffffff8138a37d 0000000000000006
[ 5111.774622]  ffff8800374ddc98 ffff8800374ddc88 ffffffff810370a9 ffff8800374ddd80
[ 5111.774711]  ffffffffa020d524 00000000ffffffef ffff8807ead7d800 ffff8807ff0cc8c0
[ 5111.774800] Call Trace:
[ 5111.774846]  [<ffffffff8138a37d>] dump_stack+0x46/0x58
[ 5111.774894]  [<ffffffff810370a9>] warn_slowpath_common+0x77/0x91
[ 5111.774944]  [<ffffffffa020d524>] ? __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 5111.775032]  [<ffffffff81037157>] warn_slowpath_fmt+0x41/0x43
[ 5111.775081]  [<ffffffffa020d524>] __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 5111.775135]  [<ffffffffa02226ed>] btrfs_run_delayed_refs+0x253/0x46f [btrfs]
[ 5111.775189]  [<ffffffffa022fdec>] btrfs_commit_transaction+0x36d/0x7df [btrfs]
[ 5111.775281]  [<ffffffffa022e345>] transaction_kthread+0xef/0x1c2 [btrfs]
[ 5111.775333]  [<ffffffffa022e256>] ? open_ctree+0x1ac7/0x1ac7 [btrfs]
[ 5111.775382]  [<ffffffff8104ee9a>] kthread+0xcd/0xd5
[ 5111.775428]  [<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 5111.775477]  [<ffffffff8138f17c>] ret_from_fork+0x7c/0xb0
[ 5111.775524]  [<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 5111.775572] ---[ end trace b552aca9a0cff3cb ]---
[ 5111.775618] BTRFS error (device sdb5) in btrfs_run_delayed_refs:2730: errno=-17 Object already exists
[ 5111.775707] BTRFS info (device sdb5): forced readonly
[ 5111.775754] BTRFS warning (device sdb5): Skipping commit of aborted transaction.
[ 5111.775841] BTRFS error (device sdb5) in cleanup_transaction:1553: errno=-17 Object already exists

If I run balance (after unmounting and mounting the filesystem), it ends in a similar way:

[ 5927.338989] btrfs: relocating block group 4647284637696 flags 17
[ 6184.333629] btrfs: found 983 extents
[ 6269.512577] BTRFS debug (device sdb5): run_one_delayed_ref returned -17
[ 6269.512579] ------------[ cut here ]------------
[ 6269.512636] WARNING: CPU: 7 PID: 17836 at fs/btrfs/super.c:254 __btrfs_abort_transaction+0x4d/0xff [btrfs]()
[ 6269.512727] btrfs: Transaction aborted (error -17)
[ 6269.512727] Modules linked in: ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr button video ehci_pci ehci_hcd acpi_cpufreq i2c_i801 i2c_core lpc_ich mfd_core ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii
[ 6269.513036] CPU: 7 PID: 17836 Comm: btrfs-transacti Tainted: G        W    3.13.0-rc4 #1
[ 6269.513124] Hardware name: System manufacturer System Product Name/P8H77-M PRO, BIOS 1101 02/04/2013
[ 6269.513214]  0000000000000009 ffff8807c5b07c48 ffffffff8138a37d 0000000000000006
[ 6269.513304]  ffff8807c5b07c98 ffff8807c5b07c88 ffffffff810370a9 ffff8807c5b07d80
[ 6269.513393]  ffffffffa020d524 00000000ffffffef ffff8807ead7e800 ffff8800371eedc0
[ 6269.513483] Call Trace:
[ 6269.513529]  [<ffffffff8138a37d>] dump_stack+0x46/0x58
[ 6269.513577]  [<ffffffff810370a9>] warn_slowpath_common+0x77/0x91
[ 6269.513627]  [<ffffffffa020d524>] ? __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 6269.513716]  [<ffffffff81037157>] warn_slowpath_fmt+0x41/0x43
[ 6269.513766]  [<ffffffffa020d524>] __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 6269.513820]  [<ffffffffa02226ed>] btrfs_run_delayed_refs+0x253/0x46f [btrfs]
[ 6269.513874]  [<ffffffffa022faef>] btrfs_commit_transaction+0x70/0x7df [btrfs]
[ 6269.513927]  [<ffffffffa022e345>] transaction_kthread+0xef/0x1c2 [btrfs]
[ 6269.513980]  [<ffffffffa022e256>] ? open_ctree+0x1ac7/0x1ac7 [btrfs]
[ 6269.514029]  [<ffffffff8104ee9a>] kthread+0xcd/0xd5
[ 6269.514076]  [<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 6269.514125]  [<ffffffff8138f17c>] ret_from_fork+0x7c/0xb0
[ 6269.514172]  [<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 6269.514220] ---[ end trace b552aca9a0cff3cc ]---
[ 6269.514266] BTRFS error (device sdb5) in btrfs_run_delayed_refs:2730: errno=-17 Object already exists
[ 6269.514356] BTRFS info (device sdb5): forced readonly
[ 6269.514430] BTRFS warning (device sdb5): Skipping commit of aborted transaction.
[ 6269.514559] BTRFS error (device sdb5) in cleanup_transaction:1553: errno=-17 Object already exists
[ 6269.527034] btrfs: found 983 extents

Is there any fix I can try?

--
Tomasz Chmielewski
http://wpkg.org

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#7 lore raw

-o skip_balance - didn't know this.

Actually, I was able to "skip" the balance, sort of, with this:

mount /mnt/btrfs ; btrfs fi balance cancel /mnt/btrfs

However, the fs is extremely unstable (will remount read only quite fast, i.e. if I start removing snapshot).

I'm trying to run btrfsck without the --repair option to see if it shows anything interesting.

Signature · 5 lines
Tomasz Chmielewski
http://wpkg.org


On Fri, 20 Dec 2013 23:52:08 +0800
Quoted 133 lines · Shilong Wang

Hello Tomasz,

There should be some bugs flying on, unitil now, you can try to
remount: something like:

# mount  /dev/sda /mnt -o  skip_balance

It will skip unfinshed balance, hopley it can help you before we give a bug fix.

Thanks,
Wang
2013/12/20 Tomasz Chmielewski <[email protected]>:

Quoted 120 lines
Quoted 3 lines · On Thu, 19 Dec 2013 22:07:37 +0900 Tomasz Chmielewski <[email protected]> wrote:

If it matters, I had to hard reboot after that bug; the balance continued after the system booted again and I got this a while later (filesystem was remounted read only):

Actually, looks like the fs is quite hosed now :(

If I want to do an operation like removing a snapshot (btrfs sub del ...), the command returns, there is IO for a few minutes, but then, the kernel complains and fs is remounted readonly:

[ 5111.773900] BTRFS debug (device sdb5): run_one_delayed_ref
returned -17 [ 5111.773902] ------------[ cut here ]------------
[ 5111.773957] WARNING: CPU: 0 PID: 15042 at fs/btrfs/super.c:254

__btrfs_abort_transaction+0x4d/0xff [btrfs]() [ 5111.774047] btrfs:
Transaction aborted (error -17) [ 5111.774048] Modules linked in:
ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr button video ehci_pci ehci_hcd acpi_cpufreq i2c_i801 i2c_core lpc_ich mfd_core ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii [ 5111.774355] CPU: 0 PID: 15042 Comm:
btrfs-transacti Tainted: G W 3.13.0-rc4 #1 [ 5111.774443]

Hardware name: System manufacturer System Product Name/P8H77-M PRO,
BIOS 1101 02/04/2013 [ 5111.774533]  0000000000000009
ffff8800374ddc48 ffffffff8138a37d 0000000000000006 [ 5111.774622]
ffff8800374ddc98 ffff8800374ddc88 ffffffff810370a9 ffff8800374ddd80
[ 5111.774711]  ffffffffa020d524 00000000ffffffef ffff8807ead7d800
ffff8807ff0cc8c0 [ 5111.774800] Call Trace: [ 5111.774846]
[<ffffffff8138a37d>] dump_stack+0x46/0x58 [ 5111.774894]
[<ffffffff810370a9>] warn_slowpath_common+0x77/0x91 [ 5111.774944]
[<ffffffffa020d524>] ? __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 5111.775032]  [<ffffffff81037157>] warn_slowpath_fmt+0x41/0x43
[ 5111.775081]  [<ffffffffa020d524>]
__btrfs_abort_transaction+0x4d/0xff [btrfs] [ 5111.775135]
[<ffffffffa02226ed>] btrfs_run_delayed_refs+0x253/0x46f [btrfs]
[ 5111.775189]  [<ffffffffa022fdec>]
btrfs_commit_transaction+0x36d/0x7df [btrfs] [ 5111.775281]
[<ffffffffa022e345>] transaction_kthread+0xef/0x1c2 [btrfs]
[ 5111.775333]  [<ffffffffa022e256>] ? open_ctree+0x1ac7/0x1ac7
[btrfs] [ 5111.775382]  [<ffffffff8104ee9a>] kthread+0xcd/0xd5
[ 5111.775428]  [<ffffffff8104edcd>] ?
kthread_freezable_should_stop+0x43/0x43 [ 5111.775477]
[<ffffffff8138f17c>] ret_from_fork+0x7c/0xb0 [ 5111.775524]
[<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 5111.775572] ---[ end trace b552aca9a0cff3cb ]--- [ 5111.775618]

BTRFS error (device sdb5) in btrfs_run_delayed_refs:2730: errno=-17 Object already exists [ 5111.775707] BTRFS info (device sdb5):
forced readonly [ 5111.775754] BTRFS warning (device sdb5):
Skipping commit of aborted transaction. [ 5111.775841] BTRFS error (device sdb5) in cleanup_transaction:1553: errno=-17 Object already exists

If I run balance (after unmounting and mounting the filesystem), it ends in a similar way:

[ 5927.338989] btrfs: relocating block group 4647284637696 flags 17
[ 6184.333629] btrfs: found 983 extents
[ 6269.512577] BTRFS debug (device sdb5): run_one_delayed_ref
returned -17 [ 6269.512579] ------------[ cut here ]------------
[ 6269.512636] WARNING: CPU: 7 PID: 17836 at fs/btrfs/super.c:254

__btrfs_abort_transaction+0x4d/0xff [btrfs]() [ 6269.512727] btrfs:
Transaction aborted (error -17) [ 6269.512727] Modules linked in:
ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr button video ehci_pci ehci_hcd acpi_cpufreq i2c_i801 i2c_core lpc_ich mfd_core ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii [ 6269.513036] CPU: 7 PID: 17836 Comm:
btrfs-transacti Tainted: G W 3.13.0-rc4 #1 [ 6269.513124]

Hardware name: System manufacturer System Product Name/P8H77-M PRO,
BIOS 1101 02/04/2013 [ 6269.513214]  0000000000000009
ffff8807c5b07c48 ffffffff8138a37d 0000000000000006 [ 6269.513304]
ffff8807c5b07c98 ffff8807c5b07c88 ffffffff810370a9 ffff8807c5b07d80
[ 6269.513393]  ffffffffa020d524 00000000ffffffef ffff8807ead7e800
ffff8800371eedc0 [ 6269.513483] Call Trace: [ 6269.513529]
[<ffffffff8138a37d>] dump_stack+0x46/0x58 [ 6269.513577]
[<ffffffff810370a9>] warn_slowpath_common+0x77/0x91 [ 6269.513627]
[<ffffffffa020d524>] ? __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 6269.513716]  [<ffffffff81037157>] warn_slowpath_fmt+0x41/0x43
[ 6269.513766]  [<ffffffffa020d524>]
__btrfs_abort_transaction+0x4d/0xff [btrfs] [ 6269.513820]
[<ffffffffa02226ed>] btrfs_run_delayed_refs+0x253/0x46f [btrfs]
[ 6269.513874]  [<ffffffffa022faef>]
btrfs_commit_transaction+0x70/0x7df [btrfs] [ 6269.513927]
[<ffffffffa022e345>] transaction_kthread+0xef/0x1c2 [btrfs]
[ 6269.513980]  [<ffffffffa022e256>] ? open_ctree+0x1ac7/0x1ac7
[btrfs] [ 6269.514029]  [<ffffffff8104ee9a>] kthread+0xcd/0xd5
[ 6269.514076]  [<ffffffff8104edcd>] ?
kthread_freezable_should_stop+0x43/0x43 [ 6269.514125]
[<ffffffff8138f17c>] ret_from_fork+0x7c/0xb0 [ 6269.514172]
[<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 6269.514220] ---[ end trace b552aca9a0cff3cc ]--- [ 6269.514266]

BTRFS error (device sdb5) in btrfs_run_delayed_refs:2730: errno=-17 Object already exists [ 6269.514356] BTRFS info (device sdb5):
forced readonly [ 6269.514430] BTRFS warning (device sdb5):
Skipping commit of aborted transaction. [ 6269.514559] BTRFS error (device sdb5) in cleanup_transaction:1553: errno=-17 Object already exists [ 6269.527034] btrfs: found 983 extents

Is there any fix I can try?

--
Tomasz Chmielewski
http://wpkg.org

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#8 lore raw

2013/12/20 Tomasz Chmielewski <[email protected]>:

Quoted 7 lines

-o skip_balance - didn't know this.

Actually, I was able to "skip" the balance, sort of, with this:

mount /mnt/btrfs ; btrfs fi balance cancel /mnt/btrfs

From your previous email, i suspend your filesystem is nearly fully.

"mount -o skip_balance" will avoid balance continuing while use btrfs balance cancel still
can not avoid balance totally, so i recommend you use skip_balance option when remounting.

Quoted 9 lines

However, the fs is extremely unstable (will remount read only quite fast, i.e. if I start removing snapshot).

I'm trying to run btrfsck without the --repair option to see if it shows anything interesting.

I'd like you not to use btrfsck --repair, it is a little dangerous anyway. If you still want to try, just use btrfsck is ok to check if your filesystem is consistent.

Or you can remount with option degraded, recovery(maybe help you to make filesystem stable
at least).

Thanks,
Wang

Quoted 143 lines

--
Tomasz Chmielewski
http://wpkg.org

Quoted 133 lines · On Fri, 20 Dec 2013 23:52:08 +0800 Shilong Wang <[email protected]>…

Hello Tomasz,

There should be some bugs flying on, unitil now, you can try to
remount: something like:

# mount  /dev/sda /mnt -o  skip_balance

It will skip unfinshed balance, hopley it can help you before we give a bug fix.

Thanks,
Wang
2013/12/20 Tomasz Chmielewski <[email protected]>:

Quoted 120 lines
Quoted 3 lines · On Thu, 19 Dec 2013 22:07:37 +0900 Tomasz Chmielewski <[email protected]> wrote:

If it matters, I had to hard reboot after that bug; the balance continued after the system booted again and I got this a while later (filesystem was remounted read only):

Actually, looks like the fs is quite hosed now :(

If I want to do an operation like removing a snapshot (btrfs sub del ...), the command returns, there is IO for a few minutes, but then, the kernel complains and fs is remounted readonly:

[ 5111.773900] BTRFS debug (device sdb5): run_one_delayed_ref
returned -17 [ 5111.773902] ------------[ cut here ]------------
[ 5111.773957] WARNING: CPU: 0 PID: 15042 at fs/btrfs/super.c:254

__btrfs_abort_transaction+0x4d/0xff [btrfs]() [ 5111.774047] btrfs:
Transaction aborted (error -17) [ 5111.774048] Modules linked in:
ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr button video ehci_pci ehci_hcd acpi_cpufreq i2c_i801 i2c_core lpc_ich mfd_core ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii [ 5111.774355] CPU: 0 PID: 15042 Comm:
btrfs-transacti Tainted: G W 3.13.0-rc4 #1 [ 5111.774443]

Hardware name: System manufacturer System Product Name/P8H77-M PRO,
BIOS 1101 02/04/2013 [ 5111.774533]  0000000000000009
ffff8800374ddc48 ffffffff8138a37d 0000000000000006 [ 5111.774622]
ffff8800374ddc98 ffff8800374ddc88 ffffffff810370a9 ffff8800374ddd80
[ 5111.774711]  ffffffffa020d524 00000000ffffffef ffff8807ead7d800
ffff8807ff0cc8c0 [ 5111.774800] Call Trace: [ 5111.774846]
[<ffffffff8138a37d>] dump_stack+0x46/0x58 [ 5111.774894]
[<ffffffff810370a9>] warn_slowpath_common+0x77/0x91 [ 5111.774944]
[<ffffffffa020d524>] ? __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 5111.775032]  [<ffffffff81037157>] warn_slowpath_fmt+0x41/0x43
[ 5111.775081]  [<ffffffffa020d524>]
__btrfs_abort_transaction+0x4d/0xff [btrfs] [ 5111.775135]
[<ffffffffa02226ed>] btrfs_run_delayed_refs+0x253/0x46f [btrfs]
[ 5111.775189]  [<ffffffffa022fdec>]
btrfs_commit_transaction+0x36d/0x7df [btrfs] [ 5111.775281]
[<ffffffffa022e345>] transaction_kthread+0xef/0x1c2 [btrfs]
[ 5111.775333]  [<ffffffffa022e256>] ? open_ctree+0x1ac7/0x1ac7
[btrfs] [ 5111.775382]  [<ffffffff8104ee9a>] kthread+0xcd/0xd5
[ 5111.775428]  [<ffffffff8104edcd>] ?
kthread_freezable_should_stop+0x43/0x43 [ 5111.775477]
[<ffffffff8138f17c>] ret_from_fork+0x7c/0xb0 [ 5111.775524]
[<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 5111.775572] ---[ end trace b552aca9a0cff3cb ]--- [ 5111.775618]

BTRFS error (device sdb5) in btrfs_run_delayed_refs:2730: errno=-17 Object already exists [ 5111.775707] BTRFS info (device sdb5):
forced readonly [ 5111.775754] BTRFS warning (device sdb5):
Skipping commit of aborted transaction. [ 5111.775841] BTRFS error (device sdb5) in cleanup_transaction:1553: errno=-17 Object already exists

If I run balance (after unmounting and mounting the filesystem), it ends in a similar way:

[ 5927.338989] btrfs: relocating block group 4647284637696 flags 17
[ 6184.333629] btrfs: found 983 extents
[ 6269.512577] BTRFS debug (device sdb5): run_one_delayed_ref
returned -17 [ 6269.512579] ------------[ cut here ]------------
[ 6269.512636] WARNING: CPU: 7 PID: 17836 at fs/btrfs/super.c:254

__btrfs_abort_transaction+0x4d/0xff [btrfs]() [ 6269.512727] btrfs:
Transaction aborted (error -17) [ 6269.512727] Modules linked in:
ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack ip_tables x_tables cpufreq_ondemand cpufreq_conservative cpufreq_powersave cpufreq_stats bridge stp llc ipv6 btrfs xor raid6_pq zlib_deflate loop pcspkr button video ehci_pci ehci_hcd acpi_cpufreq i2c_i801 i2c_core lpc_ich mfd_core ext4 crc16 jbd2 mbcache raid1 sg sd_mod ahci libahci libata scsi_mod r8169 mii [ 6269.513036] CPU: 7 PID: 17836 Comm:
btrfs-transacti Tainted: G W 3.13.0-rc4 #1 [ 6269.513124]

Hardware name: System manufacturer System Product Name/P8H77-M PRO,
BIOS 1101 02/04/2013 [ 6269.513214]  0000000000000009
ffff8807c5b07c48 ffffffff8138a37d 0000000000000006 [ 6269.513304]
ffff8807c5b07c98 ffff8807c5b07c88 ffffffff810370a9 ffff8807c5b07d80
[ 6269.513393]  ffffffffa020d524 00000000ffffffef ffff8807ead7e800
ffff8800371eedc0 [ 6269.513483] Call Trace: [ 6269.513529]
[<ffffffff8138a37d>] dump_stack+0x46/0x58 [ 6269.513577]
[<ffffffff810370a9>] warn_slowpath_common+0x77/0x91 [ 6269.513627]
[<ffffffffa020d524>] ? __btrfs_abort_transaction+0x4d/0xff [btrfs]
[ 6269.513716]  [<ffffffff81037157>] warn_slowpath_fmt+0x41/0x43
[ 6269.513766]  [<ffffffffa020d524>]
__btrfs_abort_transaction+0x4d/0xff [btrfs] [ 6269.513820]
[<ffffffffa02226ed>] btrfs_run_delayed_refs+0x253/0x46f [btrfs]
[ 6269.513874]  [<ffffffffa022faef>]
btrfs_commit_transaction+0x70/0x7df [btrfs] [ 6269.513927]
[<ffffffffa022e345>] transaction_kthread+0xef/0x1c2 [btrfs]
[ 6269.513980]  [<ffffffffa022e256>] ? open_ctree+0x1ac7/0x1ac7
[btrfs] [ 6269.514029]  [<ffffffff8104ee9a>] kthread+0xcd/0xd5
[ 6269.514076]  [<ffffffff8104edcd>] ?
kthread_freezable_should_stop+0x43/0x43 [ 6269.514125]
[<ffffffff8138f17c>] ret_from_fork+0x7c/0xb0 [ 6269.514172]
[<ffffffff8104edcd>] ? kthread_freezable_should_stop+0x43/0x43
[ 6269.514220] ---[ end trace b552aca9a0cff3cc ]--- [ 6269.514266]

BTRFS error (device sdb5) in btrfs_run_delayed_refs:2730: errno=-17 Object already exists [ 6269.514356] BTRFS info (device sdb5):
forced readonly [ 6269.514430] BTRFS warning (device sdb5):
Skipping commit of aborted transaction. [ 6269.514559] BTRFS error (device sdb5) in cleanup_transaction:1553: errno=-17 Object already exists [ 6269.527034] btrfs: found 983 extents

Is there any fix I can try?

--
Tomasz Chmielewski
http://wpkg.org

--
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html

#9 lore raw
Quoted 5 lines · On Sat, 21 Dec 2013 00:07:19 +0800 Shilong Wang <[email protected]>…

2013/12/20 Tomasz Chmielewski <[email protected]>:

Quoted 4 lines

-o skip_balance - didn't know this.

Actually, I was able to "skip" the balance, sort of, with this:

Quoted 7 lines
Quoted 1 line

mount /mnt/btrfs ; btrfs fi balance cancel /mnt/btrfs

From your previous email, i suspend your filesystem is nearly fully. "mount -o skip_balance" will avoid balance continuing while use btrfs balance cancel still
can not avoid balance totally, so i recommend you use skip_balance option when remounting.

My other thread a few days ago ("no space left, metadata usage almost full?"), if that's what you're referring to, was about a different filesystem.

This one should have enough free space left:

# btrfs fi
show /mnt/lxc1 Label: lxc1  uuid: 8d08ad6d-4543-4fe5-8b1b-640dc1423d41
        Total devices 2 FS bytes used 2.02TiB
        devid    1 size 2.62TiB used 2.02TiB path /dev/sda5
        devid    2 size 2.62TiB used 2.02TiB path /dev/sdb5

Btrfs v3.12

# btrfs fi df /mnt/lxc1
Data, RAID1: total=1.97TiB, used=1.97TiB
System, RAID1: total=32.00MiB, used=300.00KiB
Metadata, RAID1: total=50.00GiB, used=49.21GiB

I've tried using skip_balance - it mounted, but breaks soon after I try to remove any snapshot.

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org
#10 lore raw

2013/12/21 Tomasz Chmielewski <[email protected]>:

Quoted 30 lines
Quoted 5 lines · On Sat, 21 Dec 2013 00:07:19 +0800 Shilong Wang <[email protected]>…

2013/12/20 Tomasz Chmielewski <[email protected]>:

Quoted 4 lines

-o skip_balance - didn't know this.

Actually, I was able to "skip" the balance, sort of, with this:

Quoted 7 lines
Quoted 1 line

mount /mnt/btrfs ; btrfs fi balance cancel /mnt/btrfs

From your previous email, i suspend your filesystem is nearly fully. "mount -o skip_balance" will avoid balance continuing while use btrfs balance cancel still
can not avoid balance totally, so i recommend you use skip_balance option when remounting.

My other thread a few days ago ("no space left, metadata usage almost full?"), if that's what you're referring to, was about a different filesystem.

This one should have enough free space left:

# btrfs fi
show /mnt/lxc1 Label: lxc1  uuid: 8d08ad6d-4543-4fe5-8b1b-640dc1423d41
        Total devices 2 FS bytes used 2.02TiB
        devid    1 size 2.62TiB used 2.02TiB path /dev/sda5
        devid    2 size 2.62TiB used 2.02TiB path /dev/sdb5

Btrfs v3.12

Quoted 9 lines
# btrfs fi df /mnt/lxc1
Data, RAID1: total=1.97TiB, used=1.97TiB
System, RAID1: total=32.00MiB, used=300.00KiB
Metadata, RAID1: total=50.00GiB, used=49.21GiB

I've tried using skip_balance - it mounted, but breaks soon after I try to remove any snapshot.

ok, It seems that your filesystem is not in consistency anymore. To confirm your filesystem status, you can try to run btrfsck to check if there is something wrong.

To get a stable fs now, you can try to mount btrfs with option skip_balance and recovery, recovery options try to get a previous good tree root which might help you
get a stable fs.

Quoted 4 lines

--
Tomasz Chmielewski
http://wpkg.org

#11 lore raw
Quoted 8 lines · On Sat, 21 Dec 2013 01:11:00 +0800 Shilong Wang <[email protected]>…

ok, It seems that your filesystem is not in consistency anymore. To confirm your filesystem status, you can try to run btrfsck to check if there is something wrong.

To get a stable fs now, you can try to mount btrfs with option skip_balance and recovery, recovery options try to get a previous good tree root which might help you
get a stable fs.

Unfortunately even after using these mount options it breaks as before, as soon as I remove snapshots, or even copy some data in with rsync.

Oh well, at least I have read only access...

Signature · 2 lines
Tomasz Chmielewski
http://wpkg.org