Any ideas what this warnings are about?
Hi all,
on one of my machines I'm seeing these warnings a lot lately:
------------[ cut here ]------------ WARNING: CPU: 1 PID: 314 at fs/fs-writeback.c:2472 __writeback_inodes_sb_nr+0xb8/0xd0 Modules linked in: iTCO_wdt wireguard curve25519_x86_64 libchacha20poly1305 intel_pmc_bxt iTCO_vendor_support chacha_x86_64 poly1305_x86_64 libblake2s blake2s_x86_64 ip6_udp_tunnel udp_tunnel libcurve25519_generic libchacha libblake2s_generic psmouse joydev mousedev pcspkr i2c_i801 i2c_smbus lpc_ich iptable_filter xt_nat xt_tcpudp intel_agp intel_gtt iptable_nat nf_nat qemu_fw_cfg nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 mac_hid vfat fat auth_rpcgss sunrpc fuse ip_tables x_tables btrfs blake2b_generic libcrc32c crc32c_generic xor raid6_pq dm_crypt cbc encrypted_keys trusted tpm usbhid dm_mod virtio_gpu virtio_dma_buf drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec drm virtio_scsi virtio_balloon virtio_net virtio_console net_failover agpgart failover crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper serio_raw sr_mod cdrom xhci_pci virtio_pci virtio_rng rng_core CPU: 1 PID: 314 Comm: btrfs-transacti Tainted: G W 5.10.26-1-MANJARO #1 Hardware name: Hetzner vServer, BIOS 20171111 11/11/2017 RIP: 0010:__writeback_inodes_sb_nr+0xb8/0xd0 Code: 0f b6 d1 48 8d 74 24 10 e8 35 fc ff ff 48 89 e7 e8 7d fb ff ff 48 8b 44 24 48 65 48 2b 04 25 28 00 00 00 75 09 48 83 c4 50 c3 <0f> 0b eb cf e8 df 8c 75 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f RSP: 0018:ffffb1f5448f7d98 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff961185aa5488 RCX: 0000000000000000 RDX: 0000000000000002 RSI: 0000000000004b45 RDI: ffff9611b7220000 RBP: ffff9611b64d3958 R08: ffff961184efc800 R09: 0000000000000140 R10: ffff9611859e6400 R11: ffff961192af3c10 R12: ffff9611838f7c00 R13: ffff961185aa5000 R14: ffff961185aa5460 R15: 0000000000011a0f FS: 0000000000000000(0000) GS:ffff9611fad00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f654bc76b58 CR3: 0000000002c60000 CR4: 00000000003506e0 Call Trace: btrfs_commit_transaction+0x448/0xbc0 [btrfs] ? start_transaction+0xcc/0x5b0 [btrfs] transaction_kthread+0x143/0x170 [btrfs] ? btrfs_cleanup_transaction.isra.0+0x560/0x560 [btrfs] kthread+0x133/0x150 ? __kthread_bind_mask+0x60/0x60 ret_from_fork+0x22/0x30 ---[ end trace 3cefecf5d9d20b50 ]--- ------------[ cut here ]------------ WARNING: CPU: 0 PID: 758 at fs/fs-writeback.c:2472 __writeback_inodes_sb_nr+0xb8/0xd0 Modules linked in: iTCO_wdt wireguard curve25519_x86_64 libchacha20poly1305 intel_pmc_bxt iTCO_vendor_support chacha_x86_64 poly1305_x86_64 libblake2s blake2s_x86_> CPU: 0 PID: 758 Comm: journal-offline Tainted: G W 5.10.26-1-MANJARO #1 Hardware name: Hetzner vServer, BIOS 20171111 11/11/2017 RIP: 0010:__writeback_inodes_sb_nr+0xb8/0xd0 Code: 0f b6 d1 48 8d 74 24 10 e8 35 fc ff ff 48 89 e7 e8 7d fb ff ff 48 8b 44 24 48 65 48 2b 04 25 28 00 00 00 75 09 48 83 c4 50 c3 <0f> 0b eb cf e8 df 8c 75 00 66> RSP: 0018:ffffb1f540d7fd40 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff961185aa5488 RCX: 0000000000000000 RDX: 0000000000000002 RSI: 0000000000004be5 RDI: ffff9611b7220000 RBP: ffff961182209208 R08: ffff961184efc800 R09: 0000000000000140 R10: ffff9611859e6400 R11: ffff961192af3c10 R12: ffff961183891200 R13: ffff961185aa5000 R14: ffff961185aa5460 R15: ffff9611b65d5e10 FS: 00007f654b80e640(0000) GS:ffff9611fac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f654cf39010 CR3: 0000000003884000 CR4: 00000000003506f0 Call Trace: btrfs_commit_transaction+0x448/0xbc0 [btrfs] ? btrfs_wait_ordered_range+0x1b8/0x210 [btrfs] ? btrfs_sync_file+0x2b8/0x4e0 [btrfs] btrfs_sync_file+0x343/0x4e0 [btrfs] __x64_sys_fsync+0x34/0x60 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7f654cf26deb Code: 4a 00 00 00 0f 05 48 3d 00 f0 ff ff 77 41 c3 48 83 ec 18 89 7c 24 0c e8 33 f7 ff ff 8b 7c 24 0c 41 89 c0 b8 4a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89> RSP: 002b:00007f654b80db10 EFLAGS: 00000293 ORIG_RAX: 000000000000004a RAX: ffffffffffffffda RBX: 00005636c966b410 RCX: 00007f654cf26deb RDX: 0000000000000002 RSI: 00007f654d169d43 RDI: 0000000000000016 RBP: 00007f654d16c6f0 R08: 0000000000000000 R09: 00007f654b80e640 R10: 0000000000000003 R11: 0000000000000293 R12: 0000000000000002 R13: 00007ffc8bd7461f R14: 0000000000000000 R15: 00007f654b80e640 ---[ end trace 3cefecf5d9d20b84 ]---
This is a KVM guest on an AMD host. A similar machine on an Intel host doesn't show this. There is a single BTRFS filesystem on LUKS. It has been created like:
mkfs.btrfs --csum blake2 -d single -m single /dev/mapper/rootfs
I've tested Manjaro's 5.11.10 kernel too. No change. btrfs-check (5.11.1) shows nothing. I've even recreated the filesystem. No change. Maybe the machine without warnings uses the default hash. How can I find out?
Thanks!
Markus
Quoted 24 lines · Markus Schaaf
Hi all,
on one of my machines I'm seeing these warnings a lot lately:
------------[ cut here ]------------ WARNING: CPU: 1 PID: 314 at fs/fs-writeback.c:2472 __writeback_inodes_sb_nr+0xb8/0xd0 Modules linked in: iTCO_wdt wireguard curve25519_x86_64 libchacha20poly1305 intel_pmc_bxt iTCO_vendor_support chacha_x86_64 poly1305_x86_64 libblake2s blake2s_x86_64 ip6_udp_tunnel udp_tunnel libcurve25519_generic libchacha libblake2s_generic psmouse joydev mousedev pcspkr i2c_i801 i2c_smbus lpc_ich iptable_filter xt_nat xt_tcpudp intel_agp intel_gtt iptable_nat nf_nat qemu_fw_cfg nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 mac_hid vfat fat auth_rpcgss sunrpc fuse ip_tables x_tables btrfs blake2b_generic libcrc32c crc32c_generic xor raid6_pq dm_crypt cbc encrypted_keys trusted tpm usbhid dm_mod virtio_gpu virtio_dma_buf drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec drm virtio_scsi virtio_balloon virtio_net virtio_console net_failover agpgart failover crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper serio_raw sr_mod cdrom xhci_pci virtio_pci virtio_rng rng_core CPU: 1 PID: 314 Comm: btrfs-transacti Tainted: G W 5.10.26-1-MANJARO #1 Hardware name: Hetzner vServer, BIOS 20171111 11/11/2017 RIP: 0010:__writeback_inodes_sb_nr+0xb8/0xd0 Code: 0f b6 d1 48 8d 74 24 10 e8 35 fc ff ff 48 89 e7 e8 7d fb ff ff 48 8b 44 24 48 65 48 2b 04 25 28 00 00 00 75 09 48 83 c4 50 c3 <0f> 0b eb cf e8 df 8c 75 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f RSP: 0018:ffffb1f5448f7d98 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff961185aa5488 RCX: 0000000000000000 RDX: 0000000000000002 RSI: 0000000000004b45 RDI: ffff9611b7220000 RBP: ffff9611b64d3958 R08: ffff961184efc800 R09: 0000000000000140 R10: ffff9611859e6400 R11: ffff961192af3c10 R12: ffff9611838f7c00 R13: ffff961185aa5000 R14: ffff961185aa5460 R15: 0000000000011a0f FS: 0000000000000000(0000) GS:ffff9611fad00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f654bc76b58 CR3: 0000000002c60000 CR4: 00000000003506e0 Call Trace: btrfs_commit_transaction+0x448/0xbc0 [btrfs] ? start_transaction+0xcc/0x5b0 [btrfs] transaction_kthread+0x143/0x170 [btrfs]
Quoted 8 lines
? btrfs_cleanup_transaction.isra.0+0x560/0x560 [btrfs] kthread+0x133/0x150 ? __kthread_bind_mask+0x60/0x60 ret_from_fork+0x22/0x30 ---[ end trace 3cefecf5d9d20b50 ]--- ------------[ cut here ]------------ WARNING: CPU: 0 PID: 758 at fs/fs-writeback.c:2472 __writeback_inodes_sb_nr+0xb8/0xd0
That bug was introduced in 4.15 as part of a fix for a deadlock bug. It's still there today. Not very high on anyone's TODO list as it's mostly harmless--btrfs can't be umounted during a transaction as the umount itself uses a transaction. The warning doesn't come from btrfs code, so we can't just turn it off.
Quoted 13 lines
Modules linked in: iTCO_wdt wireguard curve25519_x86_64 libchacha20poly1305 intel_pmc_bxt iTCO_vendor_support chacha_x86_64 poly1305_x86_64 libblake2s blake2s_x86_> CPU: 0 PID: 758 Comm: journal-offline Tainted: G W 5.10.26-1-MANJARO #1 Hardware name: Hetzner vServer, BIOS 20171111 11/11/2017 RIP: 0010:__writeback_inodes_sb_nr+0xb8/0xd0 Code: 0f b6 d1 48 8d 74 24 10 e8 35 fc ff ff 48 89 e7 e8 7d fb ff ff 48 8b 44 24 48 65 48 2b 04 25 28 00 00 00 75 09 48 83 c4 50 c3 <0f> 0b eb cf e8 df 8c 75 00 66> RSP: 0018:ffffb1f540d7fd40 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff961185aa5488 RCX: 0000000000000000 RDX: 0000000000000002 RSI: 0000000000004be5 RDI: ffff9611b7220000 RBP: ffff961182209208 R08: ffff961184efc800 R09: 0000000000000140 R10: ffff9611859e6400 R11: ffff961192af3c10 R12: ffff961183891200 R13: ffff961185aa5000 R14: ffff961185aa5460 R15: ffff9611b65d5e10 FS: 00007f654b80e640(0000) GS:ffff9611fac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Quoted 8 lines
CR2: 00007f654cf39010 CR3: 0000000003884000 CR4: 00000000003506f0 Call Trace: btrfs_commit_transaction+0x448/0xbc0 [btrfs] ? btrfs_wait_ordered_range+0x1b8/0x210 [btrfs] ? btrfs_sync_file+0x2b8/0x4e0 [btrfs] btrfs_sync_file+0x343/0x4e0 [btrfs] __x64_sys_fsync+0x34/0x60 do_syscall_64+0x33/0x40
Normally you need to mount -o flushoncommit to trigger this warning. Maybe sync is triggering it too?
Quoted 22 lines
entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7f654cf26deb Code: 4a 00 00 00 0f 05 48 3d 00 f0 ff ff 77 41 c3 48 83 ec 18 89 7c 24 0c e8 33 f7 ff ff 8b 7c 24 0c 41 89 c0 b8 4a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89> RSP: 002b:00007f654b80db10 EFLAGS: 00000293 ORIG_RAX: 000000000000004a RAX: ffffffffffffffda RBX: 00005636c966b410 RCX: 00007f654cf26deb RDX: 0000000000000002 RSI: 00007f654d169d43 RDI: 0000000000000016 RBP: 00007f654d16c6f0 R08: 0000000000000000 R09: 00007f654b80e640 R10: 0000000000000003 R11: 0000000000000293 R12: 0000000000000002 R13: 00007ffc8bd7461f R14: 0000000000000000 R15: 00007f654b80e640 ---[ end trace 3cefecf5d9d20b84 ]---This is a KVM guest on an AMD host. A similar machine on an Intel host doesn't show this. There is a single BTRFS filesystem on LUKS. It has been created like:
mkfs.btrfs --csum blake2 -d single -m single /dev/mapper/rootfsI've tested Manjaro's 5.11.10 kernel too. No change. btrfs-check (5.11.1) shows nothing. I've even recreated the filesystem. No change. Maybe the machine without warnings uses the default hash. How can I find out?
Thanks!
Markus
Am 31.03.21 um 03:58 schrieb Zygo Blaxell:
Quoted 26 lines
Quoted 26 lines
------------[ cut here ]------------ WARNING: CPU: 1 PID: 314 at fs/fs-writeback.c:2472 __writeback_inodes_sb_nr+0xb8/0xd0 Modules linked in: iTCO_wdt wireguard curve25519_x86_64 libchacha20poly1305 intel_pmc_bxt iTCO_vendor_support chacha_x86_64 poly1305_x86_64 libblake2s blake2s_x86_64 ip6_udp_tunnel udp_tunnel libcurve25519_generic libchacha libblake2s_generic psmouse joydev mousedev pcspkr i2c_i801 i2c_smbus lpc_ich iptable_filter xt_nat xt_tcpudp intel_agp intel_gtt iptable_nat nf_nat qemu_fw_cfg nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 mac_hid vfat fat auth_rpcgss sunrpc fuse ip_tables x_tables btrfs blake2b_generic libcrc32c crc32c_generic xor raid6_pq dm_crypt cbc encrypted_keys trusted tpm usbhid dm_mod virtio_gpu virtio_dma_buf drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec drm virtio_scsi virtio_balloon virtio_net virtio_console net_failover agpgart failover crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper serio_raw sr_mod cdrom xhci_pci virtio_pci virtio_rng rng_core CPU: 1 PID: 314 Comm: btrfs-transacti Tainted: G W 5.10.26-1-MANJARO #1 Hardware name: Hetzner vServer, BIOS 20171111 11/11/2017 RIP: 0010:__writeback_inodes_sb_nr+0xb8/0xd0 Code: 0f b6 d1 48 8d 74 24 10 e8 35 fc ff ff 48 89 e7 e8 7d fb ff ff 48 8b 44 24 48 65 48 2b 04 25 28 00 00 00 75 09 48 83 c4 50 c3 <0f> 0b eb cf e8 df 8c 75 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f RSP: 0018:ffffb1f5448f7d98 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff961185aa5488 RCX: 0000000000000000 RDX: 0000000000000002 RSI: 0000000000004b45 RDI: ffff9611b7220000 RBP: ffff9611b64d3958 R08: ffff961184efc800 R09: 0000000000000140 R10: ffff9611859e6400 R11: ffff961192af3c10 R12: ffff9611838f7c00 R13: ffff961185aa5000 R14: ffff961185aa5460 R15: 0000000000011a0f FS: 0000000000000000(0000) GS:ffff9611fad00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f654bc76b58 CR3: 0000000002c60000 CR4: 00000000003506e0 Call Trace: btrfs_commit_transaction+0x448/0xbc0 [btrfs] ? start_transaction+0xcc/0x5b0 [btrfs] transaction_kthread+0x143/0x170 [btrfs] ? btrfs_cleanup_transaction.isra.0+0x560/0x560 [btrfs] kthread+0x133/0x150 ? __kthread_bind_mask+0x60/0x60 ret_from_fork+0x22/0x30 ---[ end trace 3cefecf5d9d20b50 ]---
Quoted 8 lines
Quoted 2 lines
------------[ cut here ]------------ WARNING: CPU: 0 PID: 758 at fs/fs-writeback.c:2472 __writeback_inodes_sb_nr+0xb8/0xd0That bug was introduced in 4.15 as part of a fix for a deadlock bug. It's still there today. Not very high on anyone's TODO list as it's mostly harmless--btrfs can't be umounted during a transaction as the umount itself uses a transaction. The warning doesn't come from btrfs code, so we can't just turn it off.
That warning spams my logs every 1-5 minutes. There is no unmounting happening.
Quoted 23 lines
Quoted 14 lines
Modules linked in: iTCO_wdt wireguard curve25519_x86_64 libchacha20poly1305 intel_pmc_bxt iTCO_vendor_support chacha_x86_64 poly1305_x86_64 libblake2s blake2s_x86_> CPU: 0 PID: 758 Comm: journal-offline Tainted: G W 5.10.26-1-MANJARO #1 Hardware name: Hetzner vServer, BIOS 20171111 11/11/2017 RIP: 0010:__writeback_inodes_sb_nr+0xb8/0xd0 Code: 0f b6 d1 48 8d 74 24 10 e8 35 fc ff ff 48 89 e7 e8 7d fb ff ff 48 8b 44 24 48 65 48 2b 04 25 28 00 00 00 75 09 48 83 c4 50 c3 <0f> 0b eb cf e8 df 8c 75 00 66> RSP: 0018:ffffb1f540d7fd40 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff961185aa5488 RCX: 0000000000000000 RDX: 0000000000000002 RSI: 0000000000004be5 RDI: ffff9611b7220000 RBP: ffff961182209208 R08: ffff961184efc800 R09: 0000000000000140 R10: ffff9611859e6400 R11: ffff961192af3c10 R12: ffff961183891200 R13: ffff961185aa5000 R14: ffff961185aa5460 R15: ffff9611b65d5e10 FS: 00007f654b80e640(0000) GS:ffff9611fac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033Quoted 8 lines
CR2: 00007f654cf39010 CR3: 0000000003884000 CR4: 00000000003506f0 Call Trace: btrfs_commit_transaction+0x448/0xbc0 [btrfs] ? btrfs_wait_ordered_range+0x1b8/0x210 [btrfs] ? btrfs_sync_file+0x2b8/0x4e0 [btrfs] btrfs_sync_file+0x343/0x4e0 [btrfs] __x64_sys_fsync+0x34/0x60 do_syscall_64+0x33/0x40
Quoted 2 lines
Normally you need to mount -o flushoncommit to trigger this warning. Maybe sync is triggering it too?
I've looked again and yes, this "special" filesystem is mounted flushoncommit and discard=async. Would it be better to not set these options, for now?
BR
Quoted 64 lines · Markus Schaaf
Am 31.03.21 um 03:58 schrieb Zygo Blaxell:
Quoted 26 lines
Quoted 26 lines
------------[ cut here ]------------ WARNING: CPU: 1 PID: 314 at fs/fs-writeback.c:2472 __writeback_inodes_sb_nr+0xb8/0xd0 Modules linked in: iTCO_wdt wireguard curve25519_x86_64 libchacha20poly1305 intel_pmc_bxt iTCO_vendor_support chacha_x86_64 poly1305_x86_64 libblake2s blake2s_x86_64 ip6_udp_tunnel udp_tunnel libcurve25519_generic libchacha libblake2s_generic psmouse joydev mousedev pcspkr i2c_i801 i2c_smbus lpc_ich iptable_filter xt_nat xt_tcpudp intel_agp intel_gtt iptable_nat nf_nat qemu_fw_cfg nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 mac_hid vfat fat auth_rpcgss sunrpc fuse ip_tables x_tables btrfs blake2b_generic libcrc32c crc32c_generic xor raid6_pq dm_crypt cbc encrypted_keys trusted tpm usbhid dm_mod virtio_gpu virtio_dma_buf drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec drm virtio_scsi virtio_balloon virtio_net virtio_console net_failover agpgart failover crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper serio_raw sr_mod cdrom xhci_pci virtio_pci virtio_rng rng_core CPU: 1 PID: 314 Comm: btrfs-transacti Tainted: G W 5.10.26-1-MANJARO #1 Hardware name: Hetzner vServer, BIOS 20171111 11/11/2017 RIP: 0010:__writeback_inodes_sb_nr+0xb8/0xd0 Code: 0f b6 d1 48 8d 74 24 10 e8 35 fc ff ff 48 89 e7 e8 7d fb ff ff 48 8b 44 24 48 65 48 2b 04 25 28 00 00 00 75 09 48 83 c4 50 c3 <0f> 0b eb cf e8 df 8c 75 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f RSP: 0018:ffffb1f5448f7d98 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff961185aa5488 RCX: 0000000000000000 RDX: 0000000000000002 RSI: 0000000000004b45 RDI: ffff9611b7220000 RBP: ffff9611b64d3958 R08: ffff961184efc800 R09: 0000000000000140 R10: ffff9611859e6400 R11: ffff961192af3c10 R12: ffff9611838f7c00 R13: ffff961185aa5000 R14: ffff961185aa5460 R15: 0000000000011a0f FS: 0000000000000000(0000) GS:ffff9611fad00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f654bc76b58 CR3: 0000000002c60000 CR4: 00000000003506e0 Call Trace: btrfs_commit_transaction+0x448/0xbc0 [btrfs] ? start_transaction+0xcc/0x5b0 [btrfs] transaction_kthread+0x143/0x170 [btrfs] ? btrfs_cleanup_transaction.isra.0+0x560/0x560 [btrfs] kthread+0x133/0x150 ? __kthread_bind_mask+0x60/0x60 ret_from_fork+0x22/0x30 ---[ end trace 3cefecf5d9d20b50 ]---Quoted 8 lines
Quoted 2 lines
------------[ cut here ]------------ WARNING: CPU: 0 PID: 758 at fs/fs-writeback.c:2472 __writeback_inodes_sb_nr+0xb8/0xd0That bug was introduced in 4.15 as part of a fix for a deadlock bug. It's still there today. Not very high on anyone's TODO list as it's mostly harmless--btrfs can't be umounted during a transaction as the umount itself uses a transaction. The warning doesn't come from btrfs code, so we can't just turn it off.
That warning spams my logs every 1-5 minutes. There is no unmounting happening.
Quoted 23 lines
Quoted 22 lines
Modules linked in: iTCO_wdt wireguard curve25519_x86_64 libchacha20poly1305 intel_pmc_bxt iTCO_vendor_support chacha_x86_64 poly1305_x86_64 libblake2s blake2s_x86_> CPU: 0 PID: 758 Comm: journal-offline Tainted: G W 5.10.26-1-MANJARO #1 Hardware name: Hetzner vServer, BIOS 20171111 11/11/2017 RIP: 0010:__writeback_inodes_sb_nr+0xb8/0xd0 Code: 0f b6 d1 48 8d 74 24 10 e8 35 fc ff ff 48 89 e7 e8 7d fb ff ff 48 8b 44 24 48 65 48 2b 04 25 28 00 00 00 75 09 48 83 c4 50 c3 <0f> 0b eb cf e8 df 8c 75 00 66> RSP: 0018:ffffb1f540d7fd40 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff961185aa5488 RCX: 0000000000000000 RDX: 0000000000000002 RSI: 0000000000004be5 RDI: ffff9611b7220000 RBP: ffff961182209208 R08: ffff961184efc800 R09: 0000000000000140 R10: ffff9611859e6400 R11: ffff961192af3c10 R12: ffff961183891200 R13: ffff961185aa5000 R14: ffff961185aa5460 R15: ffff9611b65d5e10 FS: 00007f654b80e640(0000) GS:ffff9611fac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f654cf39010 CR3: 0000000003884000 CR4: 00000000003506f0 Call Trace: btrfs_commit_transaction+0x448/0xbc0 [btrfs] ? btrfs_wait_ordered_range+0x1b8/0x210 [btrfs] ? btrfs_sync_file+0x2b8/0x4e0 [btrfs] btrfs_sync_file+0x343/0x4e0 [btrfs] __x64_sys_fsync+0x34/0x60 do_syscall_64+0x33/0x40
Quoted 6 lines
Quoted 2 lines
Normally you need to mount -o flushoncommit to trigger this warning. Maybe sync is triggering it too?
I've looked again and yes, this "special" filesystem is mounted flushoncommit and discard=async. Would it be better to not set these options, for now?
Flushoncommit is safe but noisy in dmesg, and can make things slow it just depends on the workload. And discard=async is also considered safe, though relatively new. The only way to know for sure is disable it, and only it, run for some time period to establish "normative" behavior, and then enable only this option and see if behavior changes from the baseline.
If you don't have a heavy write and delete workload, you may not really need discard=async anyway, and a weekly fstrim is generally sufficient for the fast majority of workloads. Conversely a heavy write and delete workload translates into a backlog of trim that gets issued all at once, once a week, and can make an SSD bog down after it's issued. So you just have to test it with your particular workload to know.
Discard=async exists because a weekly fstrim, and discard=sync can supply way too much hinting all at once to the drive about what blocks are no longer needed and are ready for garbage collection. But again, it's workload specific, and even hardware specific. Some hardware is sufficiently overprovisioned that there's no benefit to issuing discards at all, and normal usage gives the drive firmware all it needs to know about what blocks are ready for garbage collection (and erasing blocks to prepare them for future writes).
Signature · 1 line
Chris Murphy
Am 31.03.21 um 22:20 schrieb Chris Murphy:
Quoted 7 lines
Flushoncommit is safe but noisy in dmesg, and can make things slow it just depends on the workload. And discard=async is also considered safe, though relatively new. The only way to know for sure is disable it, and only it, run for some time period to establish "normative" behavior, and then enable only this option and see if behavior changes from the baseline
[...]
Thank you for your detailed explanation. I was trying flushoncommit for better crash consistency. But this was based on theory, not experience.
BR
Quoted 10 lines · Markus Schaaf
Am 31.03.21 um 22:20 schrieb Chris Murphy:
Quoted 7 lines
Flushoncommit is safe but noisy in dmesg, and can make things slow it just depends on the workload. And discard=async is also considered safe, though relatively new. The only way to know for sure is disable it, and only it, run for some time period to establish "normative" behavior, and then enable only this option and see if behavior changes from the baseline
[...]
Quoted 2 lines
Thank you for your detailed explanation. I was trying flushoncommit for better crash consistency. But this was based on theory, not experience.
flushoncommit does give better crash consistency--I've been using it for years for that reason. I have to patch the kernel to silence the warning though.
Quoted 1 line
BR
Am 02.04.21 um 03:10 schrieb Zygo Blaxell:
Quoted 2 lines
flushoncommit does give better crash consistency--I've been using it for years for that reason.
Okay. This machine is the last I have setup, and it was my first try with flushoncommit. But I didn't remember, until you mentioned it. So I will leave it enabled and ignore these warnings. Thank you!
BR
Quoted 31 lines · Markus Schaaf
Am 31.03.21 um 03:58 schrieb Zygo Blaxell:
Quoted 29 lines
Quoted 28 lines
------------[ cut here ]------------ WARNING: CPU: 1 PID: 314 at fs/fs-writeback.c:2472 __writeback_inodes_sb_nr+0xb8/0xd0 Modules linked in: iTCO_wdt wireguard curve25519_x86_64 libchacha20poly1305 intel_pmc_bxt iTCO_vendor_support chacha_x86_64 poly1305_x86_64 libblake2s blake2s_x86_64 ip6_udp_tunnel udp_tunnel libcurve25519_generic libchacha libblake2s_generic psmouse joydev mousedev pcspkr i2c_i801 i2c_smbus lpc_ich iptable_filter xt_nat xt_tcpudp intel_agp intel_gtt iptable_nat nf_nat qemu_fw_cfg nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 mac_hid vfat fat auth_rpcgss sunrpc fuse ip_tables x_tables btrfs blake2b_generic libcrc32c crc32c_generic xor raid6_pq dm_crypt cbc encrypted_keys trusted tpm usbhid dm_mod virtio_gpu virtio_dma_buf drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops cec drm virtio_scsi virtio_balloon virtio_net virtio_console net_failover agpgart failover crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel crypto_simd cryptd glue_helper serio_raw sr_mod cdrom xhci_pci virtio_pci virtio_rng rng_core CPU: 1 PID: 314 Comm: btrfs-transacti Tainted: G W 5.10.26-1-MANJARO #1 Hardware name: Hetzner vServer, BIOS 20171111 11/11/2017 RIP: 0010:__writeback_inodes_sb_nr+0xb8/0xd0 Code: 0f b6 d1 48 8d 74 24 10 e8 35 fc ff ff 48 89 e7 e8 7d fb ff ff 48 8b 44 24 48 65 48 2b 04 25 28 00 00 00 75 09 48 83 c4 50 c3 <0f> 0b eb cf e8 df 8c 75 00 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f RSP: 0018:ffffb1f5448f7d98 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff961185aa5488 RCX: 0000000000000000 RDX: 0000000000000002 RSI: 0000000000004b45 RDI: ffff9611b7220000 RBP: ffff9611b64d3958 R08: ffff961184efc800 R09: 0000000000000140 R10: ffff9611859e6400 R11: ffff961192af3c10 R12: ffff9611838f7c00 R13: ffff961185aa5000 R14: ffff961185aa5460 R15: 0000000000011a0f FS: 0000000000000000(0000) GS:ffff9611fad00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f654bc76b58 CR3: 0000000002c60000 CR4: 00000000003506e0 Call Trace: btrfs_commit_transaction+0x448/0xbc0 [btrfs] ? start_transaction+0xcc/0x5b0 [btrfs] transaction_kthread+0x143/0x170 [btrfs] ? btrfs_cleanup_transaction.isra.0+0x560/0x560 [btrfs] kthread+0x133/0x150 ? __kthread_bind_mask+0x60/0x60 ret_from_fork+0x22/0x30 ---[ end trace 3cefecf5d9d20b50 ]--- ------------[ cut here ]------------ WARNING: CPU: 0 PID: 758 at fs/fs-writeback.c:2472 __writeback_inodes_sb_nr+0xb8/0xd0
Quoted 8 lines
Quoted 5 lines
That bug was introduced in 4.15 as part of a fix for a deadlock bug. It's still there today. Not very high on anyone's TODO list as it's mostly harmless--btrfs can't be umounted during a transaction as the umount itself uses a transaction. The warning doesn't come from btrfs code, so we can't just turn it off.
That warning spams my logs every 1-5 minutes. There is no unmounting happening.
The warning is complaining about a potential issue that could be triggered by umounting; however, the warning is generic to all filesystems, and btrfs has other mechanisms that prevent a problem.
flushoncommit is the known trigger for this. You had not mentioned it, so I was theorizing about other possible triggers.
Quoted 30 lines
Quoted 22 lines
Quoted 13 lines
Modules linked in: iTCO_wdt wireguard curve25519_x86_64 libchacha20poly1305 intel_pmc_bxt iTCO_vendor_support chacha_x86_64 poly1305_x86_64 libblake2s blake2s_x86_> CPU: 0 PID: 758 Comm: journal-offline Tainted: G W 5.10.26-1-MANJARO #1 Hardware name: Hetzner vServer, BIOS 20171111 11/11/2017 RIP: 0010:__writeback_inodes_sb_nr+0xb8/0xd0 Code: 0f b6 d1 48 8d 74 24 10 e8 35 fc ff ff 48 89 e7 e8 7d fb ff ff 48 8b 44 24 48 65 48 2b 04 25 28 00 00 00 75 09 48 83 c4 50 c3 <0f> 0b eb cf e8 df 8c 75 00 66> RSP: 0018:ffffb1f540d7fd40 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff961185aa5488 RCX: 0000000000000000 RDX: 0000000000000002 RSI: 0000000000004be5 RDI: ffff9611b7220000 RBP: ffff961182209208 R08: ffff961184efc800 R09: 0000000000000140 R10: ffff9611859e6400 R11: ffff961192af3c10 R12: ffff961183891200 R13: ffff961185aa5000 R14: ffff961185aa5460 R15: ffff9611b65d5e10 FS: 00007f654b80e640(0000) GS:ffff9611fac00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033Quoted 8 lines
CR2: 00007f654cf39010 CR3: 0000000003884000 CR4: 00000000003506f0 Call Trace: btrfs_commit_transaction+0x448/0xbc0 [btrfs] ? btrfs_wait_ordered_range+0x1b8/0x210 [btrfs] ? btrfs_sync_file+0x2b8/0x4e0 [btrfs] btrfs_sync_file+0x343/0x4e0 [btrfs] __x64_sys_fsync+0x34/0x60 do_syscall_64+0x33/0x40Quoted 2 lines
Normally you need to mount -o flushoncommit to trigger this warning. Maybe sync is triggering it too?
I've looked again and yes, this "special" filesystem is mounted flushoncommit and discard=async. Would it be better to not set these options, for now?
BR