Skip to contentSkip to conversation

btrfs scrub resulting in readonly filesystem

DISCUSSION Lai Wei-Hwa 3 participants 3 messages Sep 16, 2019 – Sep 17, 2019 1 message not in the archive Btrfs file system

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

#1 lore raw

Hi There,

Found a host with readonly filesystem. This is the output from dmesg which occurred right after a cron job was executed to scrub.

[Sep14 20:02] ------------[ cut here ]------------
[ +0.000042] WARNING: CPU: 18 PID: 28882 at /build/linux-TSOhpZ/linux-4.4.0/fs/btrfs/extent-tree.c:10046 btrfs_create_pending_block_groups+0x144/0x1f0 [btrfs]()
[ +0.000002] BTRFS: Transaction aborted (error -27)
[ +0.000002] Modules linked in: ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common xt_LOG xt_limit xt_tcpudp xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs cpuid unix_diag veth bridge stp llc bonding binfmt_misc zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ipmi_devintf ipmi_ssif ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul dcdbas glue_helper input_leds ablk_helper ipmi_si gpio_ich serio_raw i7core_edac joydev 8250_fintek acpi_power_meter mac_hid ipmi_msghandler shpchp edac_core cryptd lpc_ich ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi [ +0.000047] autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure scsi_transport_sas uas hid_generic usbhid usb_storage wmi psmouse hid pata_acpi megaraid_sas bnx2 fjes
[ +0.000019] CPU: 18 PID: 28882 Comm: btrfs Tainted: P IO 4.4.0-157-generic #185-Ubuntu
[ +0.000002] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 6.4.0 07/23/2013
[ +0.000001] 0000000000000286 1faf5c66f8e16e90 ffff8802adbcfa18 ffffffff8140c9a1
[ +0.000003] ffff8802adbcfa60 ffffffffc024d588 ffff8802adbcfa50 ffffffff810864d2
[ +0.000002] ffff880697e29720 ffff880c11881da0 ffff880c0f6e6800 ffff880697e29640
[ +0.000002] Call Trace:
[ +0.000008] [<ffffffff8140c9a1>] dump_stack+0x63/0x82
[ +0.000007] [<ffffffff810864d2>] warn_slowpath_common+0x82/0xc0
[ +0.000002] [<ffffffff8108656c>] warn_slowpath_fmt+0x5c/0x80
[ +0.000014] [<ffffffffc01f31c4>] ? btrfs_finish_chunk_alloc+0x204/0x5a0 [btrfs]
[ +0.000011] [<ffffffffc01b1d24>] btrfs_create_pending_block_groups+0x144/0x1f0 [btrfs]
[ +0.000012] [<ffffffffc01c7ed3>] __btrfs_end_transaction+0x93/0x340 [btrfs]
[ +0.000013] [<ffffffffc01c8190>] btrfs_end_transaction+0x10/0x20 [btrfs]
[ +0.000010] [<ffffffffc01b5a4d>] btrfs_inc_block_group_ro+0xed/0x1b0 [btrfs]
[ +0.000014] [<ffffffffc02253bf>] scrub_enumerate_chunks+0x21f/0x580 [btrfs]
[ +0.000004] [<ffffffff810cb700>] ? wake_atomic_t_function+0x60/0x60
[ +0.000013] [<ffffffffc0226d0c>] btrfs_scrub_dev+0x1bc/0x530 [btrfs]
[ +0.000004] [<ffffffff8123f306>] ? __mnt_want_write+0x56/0x60
[ +0.000013] [<ffffffffc0202408>] btrfs_ioctl+0x1ac8/0x28c0 [btrfs]
[ +0.000003] [<ffffffff8119a3b9>] ? unlock_page+0x69/0x70
[ +0.000002] [<ffffffff8119a654>] ? filemap_map_pages+0x224/0x230
[ +0.000004] [<ffffffff811cdb77>] ? handle_mm_fault+0x10f7/0x1b80
[ +0.000002] [<ffffffff811fb77b>] ? kmem_cache_alloc_node+0xbb/0x210
[ +0.000003] [<ffffffff813e13e3>] ? create_task_io_context+0x23/0x100
[ +0.000003] [<ffffffff812318ef>] do_vfs_ioctl+0x2af/0x4b0
[ +0.000002] [<ffffffff813e1510>] ? get_task_io_context+0x50/0x90
[ +0.000003] [<ffffffff813f0936>] ? set_task_ioprio+0x86/0xa0
[ +0.000002] [<ffffffff81231b69>] SyS_ioctl+0x79/0x90
[ +0.000004] [<ffffffff81864f1b>] entry_SYSCALL_64_fastpath+0x22/0xcb
[ +0.000002] ---[ end trace 13fce4e84d9b6aed ]---
[ +0.000003] BTRFS: error (device sda1) in btrfs_create_pending_block_groups:10046: errno=-27 unknown
[ +0.003942] BTRFS info (device sda1): forced readonly
[ +0.000193] pending csums is 28672
[ +0.002295] BTRFS warning (device sda1): failed setting block group ro, ret=-30

[ +7.197271] BTRFS warning (device sda1): failed setting block group ro, ret=-30

What's the cause here?

Best Regards,

Lai Wei-Hwa

Thanks!
Lai

#2 lore raw
Quoted 6 lines · Lai Wei-Hwa

Hi There,

Found a host with readonly filesystem. This is the output from dmesg which occurred right after a cron job was executed to scrub.

Quoted 6 lines

[Sep14 20:02] ------------[ cut here ]------------
[ +0.000042] WARNING: CPU: 18 PID: 28882 at /build/linux-TSOhpZ/linux-4.4.0/fs/btrfs/extent-tree.c:10046 btrfs_create_pending_block_groups+0x144/0x1f0 [btrfs]()
[ +0.000002] BTRFS: Transaction aborted (error -27)
[ +0.000002] Modules linked in: ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common xt_LOG xt_limit xt_tcpudp xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs cpuid unix_diag veth bridge stp llc bonding binfmt_misc zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ipmi_devintf ipmi_ssif ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul dcdbas glue_helper input_leds ablk_helper ipmi_si gpio_ich serio_raw i7core_edac joydev 8250_fintek acpi_power_meter mac_hid ipmi_msghandler shpchp edac_core cryptd lpc_ich ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi [ +0.000047] autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure scsi_transport_sas uas hid_generic usbhid usb_storage wmi psmouse hid pata_acpi megaraid_sas bnx2 fjes
[ +0.000019] CPU: 18 PID: 28882 Comm: btrfs Tainted: P IO 4.4.0-157-generic #185-Ubuntu

Although your old kernel is not causing the problem of this case, it's still recommended to upgrade to a newer kernel for btrfs usage.

Quoted 22 lines

[ +0.000002] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 6.4.0 07/23/2013 [ +0.000001] 0000000000000286 1faf5c66f8e16e90 ffff8802adbcfa18 ffffffff8140c9a1 [ +0.000003] ffff8802adbcfa60 ffffffffc024d588 ffff8802adbcfa50 ffffffff810864d2 [ +0.000002] ffff880697e29720 ffff880c11881da0 ffff880c0f6e6800 ffff880697e29640 [ +0.000002] Call Trace:
[ +0.000008] [<ffffffff8140c9a1>] dump_stack+0x63/0x82
[ +0.000007] [<ffffffff810864d2>] warn_slowpath_common+0x82/0xc0
[ +0.000002] [<ffffffff8108656c>] warn_slowpath_fmt+0x5c/0x80
[ +0.000014] [<ffffffffc01f31c4>] ? btrfs_finish_chunk_alloc+0x204/0x5a0 [btrfs] [ +0.000011] [<ffffffffc01b1d24>] btrfs_create_pending_block_groups+0x144/0x1f0 [btrfs] [ +0.000012] [<ffffffffc01c7ed3>] __btrfs_end_transaction+0x93/0x340 [btrfs] [ +0.000013] [<ffffffffc01c8190>] btrfs_end_transaction+0x10/0x20 [btrfs] [ +0.000010] [<ffffffffc01b5a4d>] btrfs_inc_block_group_ro+0xed/0x1b0 [btrfs] [ +0.000014] [<ffffffffc02253bf>] scrub_enumerate_chunks+0x21f/0x580 [btrfs] [ +0.000004] [<ffffffff810cb700>] ? wake_atomic_t_function+0x60/0x60
[ +0.000013] [<ffffffffc0226d0c>] btrfs_scrub_dev+0x1bc/0x530 [btrfs]
[ +0.000004] [<ffffffff8123f306>] ? __mnt_want_write+0x56/0x60
[ +0.000013] [<ffffffffc0202408>] btrfs_ioctl+0x1ac8/0x28c0 [btrfs]
[ +0.000003] [<ffffffff8119a3b9>] ? unlock_page+0x69/0x70
[ +0.000002] [<ffffffff8119a654>] ? filemap_map_pages+0x224/0x230
[ +0.000004] [<ffffffff811cdb77>] ? handle_mm_fault+0x10f7/0x1b80
[ +0.000002] [<ffffffff811fb77b>] ? kmem_cache_alloc_node+0xbb/0x210

Quoted 8 lines

[ +0.000003] [<ffffffff813e13e3>] ? create_task_io_context+0x23/0x100
[ +0.000003] [<ffffffff812318ef>] do_vfs_ioctl+0x2af/0x4b0
[ +0.000002] [<ffffffff813e1510>] ? get_task_io_context+0x50/0x90
[ +0.000003] [<ffffffff813f0936>] ? set_task_ioprio+0x86/0xa0
[ +0.000002] [<ffffffff81231b69>] SyS_ioctl+0x79/0x90
[ +0.000004] [<ffffffff81864f1b>] entry_SYSCALL_64_fastpath+0x22/0xcb
[ +0.000002] ---[ end trace 13fce4e84d9b6aed ]---
[ +0.000003] BTRFS: error (device sda1) in btrfs_create_pending_block_groups:10046: errno=-27 unknown

-27 is -EFBIG.

There are only several users of that errno, and according to your backtrace, it's from btrfs_finish_chunk_alloc() which will try to add your system chunk to super block syschunk array.

While the syschunk array in super block has a fixed size, if super block can no longer contain the new chunk, it returns -EFBIG and we abort transaction, causing the problem.

To verify my analyse, please provide the following dump:

# btrfs ins dump-super -fFa /dev/sda1
# btrfs ins dump-tree -t 3 /dev/sda1

The latter dump needs to be done on an RO fs, or unmounted fs.

I don't have a good workaround right now, but it looks like you have too many system chunks.
Normally this means you have too many data/metadata chunks, but I'd like to see the dumps to make the final call on what's causing the problem.

Thanks,
Qu

Quoted 18 lines

[ +0.003942] BTRFS info (device sda1): forced readonly
[ +0.000193] pending csums is 28672
[ +0.002295] BTRFS warning (device sda1): failed setting block group ro, ret=-30

[ +7.197271] BTRFS warning (device sda1): failed setting block group ro, ret=-30

What's the cause here?

Best Regards,

Lai Wei-Hwa

Thanks!
Lai

#3 lore raw
Quoted 5 lines · Qu Wenruo
Quoted 1 line · Lai Wei-Hwa

[ +0.000019] CPU: 18 PID: 28882 Comm: btrfs Tainted: P IO 4.4.0-157-generic #185-Ubuntu

Although your old kernel is not causing the problem of this case, it's still recommended to upgrade to a newer kernel for btrfs usage.

https://wiki.ubuntu.com/Kernel/LTSEnablementStack