btrfs scrub resulting in readonly filesystem
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi There,
Found a host with readonly filesystem. This is the output from dmesg which occurred right after a cron job was executed to scrub.
[Sep14 20:02] ------------[ cut here ]------------
[ +0.000042] WARNING: CPU: 18 PID: 28882 at /build/linux-TSOhpZ/linux-4.4.0/fs/btrfs/extent-tree.c:10046 btrfs_create_pending_block_groups+0x144/0x1f0 [btrfs]()
[ +0.000002] BTRFS: Transaction aborted (error -27)
[ +0.000002] Modules linked in: ip6t_REJECT nf_reject_ipv6 nf_log_ipv6 nf_log_common xt_LOG xt_limit xt_tcpudp xt_hl ip6t_rt nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables ufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs cpuid unix_diag veth bridge stp llc bonding binfmt_misc zfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO) intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ipmi_devintf ipmi_ssif ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul dcdbas glue_helper input_leds ablk_helper ipmi_si gpio_ich serio_raw i7core_edac joydev 8250_fintek acpi_power_meter mac_hid ipmi_msghandler shpchp edac_core cryptd lpc_ich ib_iser rdma_cm iw_cm ib_cm ib_sa ib_mad ib_core ib_addr iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi [ +0.000047] autofs4 btrfs raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ses enclosure scsi_transport_sas uas hid_generic usbhid usb_storage wmi psmouse hid pata_acpi megaraid_sas bnx2 fjes
[ +0.000019] CPU: 18 PID: 28882 Comm: btrfs Tainted: P IO 4.4.0-157-generic #185-Ubuntu
[ +0.000002] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 6.4.0 07/23/2013
[ +0.000001] 0000000000000286 1faf5c66f8e16e90 ffff8802adbcfa18 ffffffff8140c9a1
[ +0.000003] ffff8802adbcfa60 ffffffffc024d588 ffff8802adbcfa50 ffffffff810864d2
[ +0.000002] ffff880697e29720 ffff880c11881da0 ffff880c0f6e6800 ffff880697e29640
[ +0.000002] Call Trace:
[ +0.000008] [<ffffffff8140c9a1>] dump_stack+0x63/0x82
[ +0.000007] [<ffffffff810864d2>] warn_slowpath_common+0x82/0xc0
[ +0.000002] [<ffffffff8108656c>] warn_slowpath_fmt+0x5c/0x80
[ +0.000014] [<ffffffffc01f31c4>] ? btrfs_finish_chunk_alloc+0x204/0x5a0 [btrfs]
[ +0.000011] [<ffffffffc01b1d24>] btrfs_create_pending_block_groups+0x144/0x1f0 [btrfs]
[ +0.000012] [<ffffffffc01c7ed3>] __btrfs_end_transaction+0x93/0x340 [btrfs]
[ +0.000013] [<ffffffffc01c8190>] btrfs_end_transaction+0x10/0x20 [btrfs]
[ +0.000010] [<ffffffffc01b5a4d>] btrfs_inc_block_group_ro+0xed/0x1b0 [btrfs]
[ +0.000014] [<ffffffffc02253bf>] scrub_enumerate_chunks+0x21f/0x580 [btrfs]
[ +0.000004] [<ffffffff810cb700>] ? wake_atomic_t_function+0x60/0x60
[ +0.000013] [<ffffffffc0226d0c>] btrfs_scrub_dev+0x1bc/0x530 [btrfs]
[ +0.000004] [<ffffffff8123f306>] ? __mnt_want_write+0x56/0x60
[ +0.000013] [<ffffffffc0202408>] btrfs_ioctl+0x1ac8/0x28c0 [btrfs]
[ +0.000003] [<ffffffff8119a3b9>] ? unlock_page+0x69/0x70
[ +0.000002] [<ffffffff8119a654>] ? filemap_map_pages+0x224/0x230
[ +0.000004] [<ffffffff811cdb77>] ? handle_mm_fault+0x10f7/0x1b80
[ +0.000002] [<ffffffff811fb77b>] ? kmem_cache_alloc_node+0xbb/0x210
[ +0.000003] [<ffffffff813e13e3>] ? create_task_io_context+0x23/0x100
[ +0.000003] [<ffffffff812318ef>] do_vfs_ioctl+0x2af/0x4b0
[ +0.000002] [<ffffffff813e1510>] ? get_task_io_context+0x50/0x90
[ +0.000003] [<ffffffff813f0936>] ? set_task_ioprio+0x86/0xa0
[ +0.000002] [<ffffffff81231b69>] SyS_ioctl+0x79/0x90
[ +0.000004] [<ffffffff81864f1b>] entry_SYSCALL_64_fastpath+0x22/0xcb
[ +0.000002] ---[ end trace 13fce4e84d9b6aed ]---
[ +0.000003] BTRFS: error (device sda1) in btrfs_create_pending_block_groups:10046: errno=-27 unknown
[ +0.003942] BTRFS info (device sda1): forced readonly
[ +0.000193] pending csums is 28672
[ +0.002295] BTRFS warning (device sda1): failed setting block group ro, ret=-30
[ +7.197271] BTRFS warning (device sda1): failed setting block group ro, ret=-30
What's the cause here?
Best Regards,
Lai Wei-Hwa
Thanks!
Lai
Quoted 6 lines · Lai Wei-Hwa
Hi There,
Found a host with readonly filesystem. This is the output from dmesg which occurred right after a cron job was executed to scrub.
Quoted 6 lines
[Sep14 20:02] ------------[ cut here ]------------
[ +0.000042] WARNING: CPU: 18 PID: 28882 at /build/linux-TSOhpZ/linux-4.4.0/fs/btrfs/extent-tree.c:10046btrfs_create_pending_block_groups+0x144/0x1f0[btrfs]()
[ +0.000002] BTRFS: Transaction aborted (error -27)
[ +0.000002] Modules linked in:ip6t_REJECTnf_reject_ipv6nf_log_ipv6nf_log_commonxt_LOGxt_limitxt_tcpudpxt_hlip6t_rtnf_conntrack_ipv6nf_defrag_ipv6xt_conntracknf_conntrackip6table_filterip6_tablesx_tablesufs qnx4 hfsplus hfs minix ntfs msdos jfs xfs cpuidunix_diagveth bridge stp llc bondingbinfmt_misczfs(PO) zunicode(PO) zcommon(PO) znvpair(PO) spl(O) zavl(PO)intel_powerclampcoretempkvm_intelkvm irqbypasscrct10dif_pclmulcrc32_pclmulipmi_devintfipmi_ssifghash_clmulni_intelaesni_intelaes_x86_64lrw gf128mul dcdbasglue_helperinput_ledsablk_helperipmi_sigpio_ichserio_rawi7core_edacjoydev 8250_fintekacpi_power_metermac_hidipmi_msghandlershpchpedac_corecryptdlpc_ichib_iserrdma_cmiw_cmib_cmib_saib_madib_coreib_addriscsi_tcplibiscsi_tcplibiscsiscsi_transport_iscsi[ +0.000047] autofs4 btrfs raid10 raid456async_raid6_recovasync_memcpyasync_pqasync_xorasync_txxorraid6_pqlibcrc32c raid1 raid0 multipath linear ses enclosurescsi_transport_sasuashid_genericusbhidusb_storagewmi psmouse hidpata_acpimegaraid_sasbnx2 fjes
[ +0.000019] CPU: 18 PID: 28882 Comm: btrfs Tainted: P IO 4.4.0-157-generic #185-Ubuntu
Although your old kernel is not causing the problem of this case, it's still recommended to upgrade to a newer kernel for btrfs usage.
Quoted 22 lines
[ +0.000002] Hardware name: Dell Inc. PowerEdge R610/0F0XJ6, BIOS 6.4.0 07/23/2013 [ +0.000001] 0000000000000286
1faf5c66f8e16e90ffff8802adbcfa18ffffffff8140c9a1[ +0.000003]ffff8802adbcfa60ffffffffc024d588ffff8802adbcfa50ffffffff810864d2[ +0.000002]ffff880697e29720ffff880c11881da0ffff880c0f6e6800ffff880697e29640[ +0.000002] Call Trace:
[ +0.000008] [<ffffffff8140c9a1>]dump_stack+0x63/0x82
[ +0.000007] [<ffffffff810864d2>]warn_slowpath_common+0x82/0xc0
[ +0.000002] [<ffffffff8108656c>]warn_slowpath_fmt+0x5c/0x80
[ +0.000014] [<ffffffffc01f31c4>] ?btrfs_finish_chunk_alloc+0x204/0x5a0[btrfs] [ +0.000011] [<ffffffffc01b1d24>]btrfs_create_pending_block_groups+0x144/0x1f0[btrfs] [ +0.000012] [<ffffffffc01c7ed3>]__btrfs_end_transaction+0x93/0x340[btrfs] [ +0.000013] [<ffffffffc01c8190>]btrfs_end_transaction+0x10/0x20[btrfs] [ +0.000010] [<ffffffffc01b5a4d>]btrfs_inc_block_group_ro+0xed/0x1b0[btrfs] [ +0.000014] [<ffffffffc02253bf>]scrub_enumerate_chunks+0x21f/0x580[btrfs] [ +0.000004] [<ffffffff810cb700>] ?wake_atomic_t_function+0x60/0x60
[ +0.000013] [<ffffffffc0226d0c>]btrfs_scrub_dev+0x1bc/0x530[btrfs]
[ +0.000004] [<ffffffff8123f306>] ?__mnt_want_write+0x56/0x60
[ +0.000013] [<ffffffffc0202408>]btrfs_ioctl+0x1ac8/0x28c0[btrfs]
[ +0.000003] [<ffffffff8119a3b9>] ?unlock_page+0x69/0x70
[ +0.000002] [<ffffffff8119a654>] ?filemap_map_pages+0x224/0x230
[ +0.000004] [<ffffffff811cdb77>] ?handle_mm_fault+0x10f7/0x1b80
[ +0.000002] [<ffffffff811fb77b>] ?kmem_cache_alloc_node+0xbb/0x210
Quoted 8 lines
[ +0.000003] [<
ffffffff813e13e3>] ?create_task_io_context+0x23/0x100
[ +0.000003] [<ffffffff812318ef>]do_vfs_ioctl+0x2af/0x4b0
[ +0.000002] [<ffffffff813e1510>] ?get_task_io_context+0x50/0x90
[ +0.000003] [<ffffffff813f0936>] ?set_task_ioprio+0x86/0xa0
[ +0.000002] [<ffffffff81231b69>]SyS_ioctl+0x79/0x90
[ +0.000004] [<ffffffff81864f1b>]entry_SYSCALL_64_fastpath+0x22/0xcb
[ +0.000002] ---[ end trace13fce4e84d9b6aed]---
[ +0.000003] BTRFS: error (device sda1) in btrfs_create_pending_block_groups:10046: errno=-27 unknown
-27 is -EFBIG.
There are only several users of that errno, and according to your backtrace, it's from btrfs_finish_chunk_alloc() which will try to add your system chunk to super block syschunk array.
While the syschunk array in super block has a fixed size, if super block can no longer contain the new chunk, it returns -EFBIG and we abort transaction, causing the problem.
To verify my analyse, please provide the following dump:
# btrfs ins dump-super -fFa /dev/sda1 # btrfs ins dump-tree -t 3 /dev/sda1
The latter dump needs to be done on an RO fs, or unmounted fs.
I don't have a good workaround right now, but it looks like you have too many system chunks.
Normally this means you have too many data/metadata chunks, but I'd like to see the dumps to make the final call on what's causing the problem.
Thanks,
Qu
Quoted 18 lines
[ +0.003942] BTRFS info (device sda1): forced readonly
[ +0.000193] pending csums is 28672
[ +0.002295] BTRFS warning (device sda1): failed setting block group ro, ret=-30[ +7.197271] BTRFS warning (device sda1): failed setting block group ro, ret=-30
What's the cause here?
Best Regards,
Lai Wei-Hwa
Thanks!
Lai
Quoted 5 lines · Qu Wenruo
Quoted 1 line · Lai Wei-Hwa
[ +0.000019] CPU: 18 PID: 28882 Comm: btrfs Tainted: P IO 4.4.0-157-generic #185-Ubuntu
Although your old kernel is not causing the problem of this case, it's still recommended to upgrade to a newer kernel for btrfs usage.