fatal database corruption with btrfs "out of space" with ~50 GB left
Just FYI, how dangerous running btrfs can be - we had a fatal, unrecoverable MySQL corruption when btrfs decided to do one of these "I have ~50 GB left, so let's do out of space (and corrupt some files at the same time, ha ha!)".
Running btrfs RAID-1 with kernel 4.14.
Tomasz Chmielewski
https://lxadm.com
Tomasz Chmielewski posted on Wed, 14 Feb 2018 23:19:20 +0900 as excerpted:
Quoted 4 lines
Just FYI, how dangerous running btrfs can be - we had a fatal, unrecoverable MySQL corruption when btrfs decided to do one of these "I have ~50 GB left, so let's do out of space (and corrupt some files at the same time, ha ha!)".
Ouch!
Quoted 1 line
Running btrfs RAID-1 with kernel 4.14.
Kernel 4.14... quite current... good. But 4.14.0 first release, 4.14.x current stable, or somewhere (where?) in between?
And please post the output of btrfs fi usage for that filesystem. Without that (or fi sh and fi df, the pre-usage method of getting nearly the same info), it's hard to say where or what the problem was.
Meanwhile, FWIW there was a recent metadata over-reserve bug that should be fixed in 4.15 and the latest 4.14 stable, but IDR whether it affected 4.14.0 original or only the 4.13 series and early 4.14-rcs and was fixed by 4.14.0. The bug seemed to trigger most frequently when doing balances or other major writes to the filesystem, on middle to large sized filesystems. (My all under quarter-TB each btrfs didn't appear to be affected.)
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 4 lines · Tomasz Chmielewski
Just FYI, how dangerous running btrfs can be - we had a fatal, unrecoverable MySQL corruption when btrfs decided to do one of these "I have ~50 GB left, so let's do out of space (and corrupt some files at the same time, ha ha!)".
I'm recently looking into unexpected corruption problem of btrfs.
Would you please provide some extra info about how the corruption happened?
1) Is there any power reset?
Btrfs should be bullet proof, but in fact it's not, so I'm here to get some clue.
2) Are MySQL files set with nodatacow?
If so, data corruption is more or less expected, but should be handled by checkpoint of MySQL.
3) Is the filesystem metadata corrupted? (AKA, btrfs check report error) If so, that should be the problem I'm looking into.
4) Metadata/data ratio?
"btrfs fi usage" could have quite good result about it. And "btrfs fi df" also helps.
Thanks,
Qu
Quoted 11 lines
Running btrfs RAID-1 with kernel 4.14.
Tomasz Chmielewski
https://lxadm.comSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Qu Wenruo
Quoted 4 lines · Tomasz Chmielewski
Just FYI, how dangerous running btrfs can be - we had a fatal, unrecoverable MySQL corruption when btrfs decided to do one of these "I have ~50 GB left, so let's do out of space (and corrupt some files at the same time, ha ha!)".
Quoted 7 lines
I'm recently looking into unexpected corruption problem of btrfs.
Would you please provide some extra info about how the corruption happened?
1) Is there any power reset?
Btrfs should be bullet proof, but in fact it's not, so I'm here to get some clue.
No power reset.
Quoted 3 lines
2) Are MySQL files set with nodatacow?
If so, data corruption is more or less expected, but should be handled by checkpoint of MySQL.
Yes, MySQL files were using "nodatacow".
I've seen many cases of "filesystem full" with ext4, but none lead to database corruption (i.e. the database would always recover after releasing some space)
On the other hand, I've seen a handful of "out of space" with gigabytes of free space with btrfs, which lead to some light, heavy or unrecoverable MySQL or mongo corruption.
Can it be because of of how "predictable" out of space situations are with btrfs and other filesystems?
- in short, ext4 will report out of space when there is 0 bytes left (perhaps slightly faster for non-root users) - the application trying to write data will see "out of space" at some point, and it can stay like this for hours (i.e. until some data is removed manually)
- on the other hand, btrfs can report out of space when there is still 10, 50 or 100 GB left, meaning, any capacity planning is close to impossible; also, the application trying to write data can be seeing the fs as transitioning between "out of space" and "data written successfully" many times per minute/second?
Quoted 2 lines
3) Is the filesystem metadata corrupted? (AKA, btrfs check report error) If so, that should be the problem I'm looking into.
I don't think so, there are no scary things in dmesg. However, I didn't unmount the filesystem to run btrfs check.
Quoted 3 lines
4) Metadata/data ratio?
"btrfs fi usage" could have quite good result about it. And "btrfs fi df" also helps.
Here it is - however, that's after removing some 80 GB data, so most likely doesn't reflect when the failure happened.
# btrfs fi usage /var/lib/lxd
Overall:
Device size: 846.25GiB
Device allocated: 840.05GiB
Device unallocated: 6.20GiB
Device missing: 0.00B
Used: 498.26GiB
Free (estimated): 167.96GiB (min: 167.96GiB)
Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)Data,RAID1: Size:411.00GiB, Used:246.14GiB
/dev/sda3 411.00GiB
/dev/sdb3 411.00GiB
Metadata,RAID1: Size:9.00GiB, Used:2.99GiB
/dev/sda3 9.00GiB
/dev/sdb3 9.00GiB
System,RAID1: Size:32.00MiB, Used:80.00KiB
/dev/sda3 32.00MiB
/dev/sdb3 32.00MiB
Unallocated:
/dev/sda3 3.10GiB
/dev/sdb3 3.10GiB
# btrfs fi df /var/lib/lxd Data, RAID1: total=411.00GiB, used=246.15GiB System, RAID1: total=32.00MiB, used=80.00KiB Metadata, RAID1: total=9.00GiB, used=2.99GiB GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs fi show /var/lib/lxd
Label: 'btrfs' uuid: f5f30428-ec5b-4497-82de-6e20065e6f61
Total devices 2 FS bytes used 249.15GiB
devid 1 size 423.13GiB used 420.03GiB path /dev/sda3
devid 2 size 423.13GiB used 420.03GiB path /dev/sdb3Tomasz Chmielewski
https://lxadm.com
Quoted 25 lines · Tomasz Chmielewski
Quoted 6 lines · Qu Wenruo
Quoted 4 lines · Tomasz Chmielewski
Just FYI, how dangerous running btrfs can be - we had a fatal, unrecoverable MySQL corruption when btrfs decided to do one of these "I have ~50 GB left, so let's do out of space (and corrupt some files at the same time, ha ha!)".
Quoted 8 lines
I'm recently looking into unexpected corruption problem of btrfs.
Would you please provide some extra info about how the corruption happened?
1) Is there any power reset?
Btrfs should be bullet proof, but in fact it's not, so I'm here to get some clue.No power reset.
Quoted 3 lines
2) Are MySQL files set with nodatacow?
If so, data corruption is more or less expected, but should be handled by checkpoint of MySQL.Yes, MySQL files were using "nodatacow".
Quoted 7 lines
I've seen many cases of "filesystem full" with ext4, but none lead to database corruption (i.e. the database would always recover after releasing some space)
On the other hand, I've seen a handful of "out of space" with gigabytes of free space with btrfs, which lead to some light, heavy or unrecoverable MySQL or mongo corruption.
Is there any kernel message like kernel warning or backtrace?
Quoted 4 lines
Can it be because of of how "predictable" out of space situations are with btrfs and other filesystems?
It's possible.
As other filesystem doesn't really dynamically allocate its metadata space, (ext4/xfs just have limited inode numbers) and btrfs strictly split metadata and data usage, it's possible that we still have gigas of data space, but run out of meta space.
Quoted 13 lines
- in short, ext4 will report out of space when there is 0 bytes left (perhaps slightly faster for non-root users) - the application trying to write data will see "out of space" at some point, and it can stay like this for hours (i.e. until some data is removed manually)
- on the other hand, btrfs can report out of space when there is still 10, 50 or 100 GB left, meaning, any capacity planning is close to impossible; also, the application trying to write data can be seeing the fs as transitioning between "out of space" and "data written successfully" many times per minute/second?
Quoted 5 lines
Quoted 2 lines
3) Is the filesystem metadata corrupted? (AKA, btrfs check report error) If so, that should be the problem I'm looking into.
I don't think so, there are no scary things in dmesg. However, I didn't unmount the filesystem to run btrfs check.
If no scary kernel warning, then it may be less serious.
One of my assumption is, snapshots are used in your btrfs setup (well, more or less the only two selling points of btrfs), and even your DB is using nodatacow, due to snapshot, it still does data CoW.
And when the DB fails to write some critical data, maybe write ahead log or similiar things, due to ENOSPC, it causes inconsistency.
The problem here is, most DB assumes the filesystem is defaulted to do overwrite successfully, while it's not always true to btrfs.
If that's the case, would you please remove all snapshots of your DB subvolume? Or just put all DB into one subvolume and never snapshot it? Then nodatacow should work as expected, so it would be more or less more similar to ext4/xfs. (Although still slower than ext4/xfs)
Quoted 6 lines
Quoted 5 lines
4) Metadata/data ratio?
"btrfs fi usage" could have quite good result about it. And "btrfs fi df" also helps.
Quoted 8 lines
Here it is - however, that's after removing some 80 GB data, so most likely doesn't reflect when the failure happened.
# btrfs fi usage /var/lib/lxd Overall: Device size: 846.25GiB Device allocated: 840.05GiB Device unallocated: 6.20GiB
That's should prevent further ENOSPC, as long as this number is beyond 1G.
Quoted 25 lines
Device missing: 0.00B
Used: 498.26GiB
Free (estimated): 167.96GiB (min: 167.96GiB) Data ratio: 2.00
Metadata ratio: 2.00
Global reserve: 512.00MiB (used: 0.00B)Data,RAID1: Size:411.00GiB, Used:246.14GiB
/dev/sda3 411.00GiB
/dev/sdb3 411.00GiBMetadata,RAID1: Size:9.00GiB, Used:2.99GiB
/dev/sda3 9.00GiB
/dev/sdb3 9.00GiBSystem,RAID1: Size:32.00MiB, Used:80.00KiB
/dev/sda3 32.00MiB
/dev/sdb3 32.00MiBUnallocated:
/dev/sda3 3.10GiB
/dev/sdb3 3.10GiB
Quoted 4 lines
# btrfs fi df /var/lib/lxd Data, RAID1: total=411.00GiB, used=246.15GiB System, RAID1: total=32.00MiB, used=80.00KiB Metadata, RAID1: total=9.00GiB, used=2.99GiB
Not sure if the removal of 80G has anything to do with this, but this seems that your metadata (along with data) is quite scattered.
It's really recommended to keep some unallocated device space, and one of the method to do that is to use balance to free such scattered space from data/metadata usage.
And that's why balance routine is recommened for btrfs.
Thanks,
Qu
Quoted 14 lines
GlobalReserve, single: total=512.00MiB, used=0.00B
# btrfs fi show /var/lib/lxd Label: 'btrfs' uuid: f5f30428-ec5b-4497-82de-6e20065e6f61 Total devices 2 FS bytes used 249.15GiB devid 1 size 423.13GiB used 420.03GiB path /dev/sda3 devid 2 size 423.13GiB used 420.03GiB path /dev/sdb3Tomasz Chmielewski
https://lxadm.com
Quoted 1 line · Qu Wenruo
Is there any kernel message like kernel warning or backtrace?
I see there was this one:
Feb 13 13:53:32 lxd01 kernel: [9351710.878404] ------------[ cut here ]------------ Feb 13 13:53:32 lxd01 kernel: [9351710.878430] WARNING: CPU: 9 PID: 7780 at /home/kernel/COD/linux/fs/btrfs/tree-log.c:3361 log_dir_items+0x54b/0x560 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878431] Modules linked in: nfnetlink_queue bluetooth ecdh_generic xt_nat xt_REDIRECT nf_nat_redirect sunrpc cfg80211 tcp_diag inet_diag xt_NFLOG nfnetlink_log nfnetlink xt_conntrack ipt_REJECT nf_reject_ipv4 binfmt_misc veth ebtable_filter ebtables ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment nf_log_ipv4 nf_log_common xt_LOG ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_vs nf_conntrack ip6table_filter ip6_tables iptable_filter xt_CHECKSUM xt_tcpudp iptable_mangle ip_tables x_tables intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass btrfs bridge stp llc crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc zstd_compress aesni_intel aes_x86_64 Feb 13 13:53:32 lxd01 kernel: [9351710.878460] crypto_simd glue_helper cryptd input_leds intel_cstate ipmi_ssif intel_rapl_perf serio_raw lpc_ich shpchp ipmi_devintf ipmi_msghandler tpm_infineon acpi_pad mac_hid autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb drm dca ahci ptp pps_core libahci i2c_algo_bit wmi Feb 13 13:53:32 lxd01 kernel: [9351710.878484] CPU: 9 PID: 7780 Comm: TaskSchedulerBa Tainted: G W 4.14.0-041400rc6-generic #201710230731 Feb 13 13:53:32 lxd01 kernel: [9351710.878485] Hardware name: ASUSTeK COMPUTER INC. Z10PA-U8 Series/Z10PA-U8 Series, BIOS 0601 06/26/2015 Feb 13 13:53:32 lxd01 kernel: [9351710.878486] task: ffff9454227d1700 task.stack: ffffabc6a810c000 Feb 13 13:53:32 lxd01 kernel: [9351710.878502] RIP: 0010:log_dir_items+0x54b/0x560 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878502] RSP: 0018:ffffabc6a810f980 EFLAGS: 00010202 Feb 13 13:53:32 lxd01 kernel: [9351710.878503] RAX: 0000000000000001 RBX: 000000000008b771 RCX: 0000000000000000 Feb 13 13:53:32 lxd01 kernel: [9351710.878504] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 Feb 13 13:53:32 lxd01 kernel: [9351710.878505] RBP: ffffabc6a810fa28 R08: ffff9491a8f05540 R09: 0000000000000008 Feb 13 13:53:32 lxd01 kernel: [9351710.878506] R10: 0000000000000000 R11: ffffabc6a810f934 R12: ffffabc6a810fe50 Feb 13 13:53:32 lxd01 kernel: [9351710.878506] R13: ffff94666d426000 R14: ffff9491a8f05540 R15: 0000000000000054 Feb 13 13:53:32 lxd01 kernel: [9351710.878508] FS: 00007f9936e22700(0000) GS:ffff9491bf440000(0000) knlGS:0000000000000000 Feb 13 13:53:32 lxd01 kernel: [9351710.878508] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Feb 13 13:53:32 lxd01 kernel: [9351710.878509] CR2: 00007f6abef4d7b0 CR3: 00000023ecaf7006 CR4: 00000000001606e0 Feb 13 13:53:32 lxd01 kernel: [9351710.878510] Call Trace: Feb 13 13:53:32 lxd01 kernel: [9351710.878524] ? btrfs_search_slot+0x81b/0x9c0 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878538] log_directory_changes+0x83/0xd0 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878551] btrfs_log_inode+0xa24/0x11a0 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878563] ? generic_bin_search.constprop.37+0xe7/0x1f0 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878565] ? find_inode+0x59/0xb0 Feb 13 13:53:32 lxd01 kernel: [9351710.878567] ? iget5_locked+0x9e/0x1e0 Feb 13 13:53:32 lxd01 kernel: [9351710.878582] log_new_dir_dentries+0x203/0x4a7 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878595] btrfs_log_inode_parent+0x6c2/0xa10 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878598] ? pagevec_lookup_tag+0x21/0x30 Feb 13 13:53:32 lxd01 kernel: [9351710.878599] ? __filemap_fdatawait_range+0x9a/0x170 Feb 13 13:53:32 lxd01 kernel: [9351710.878614] ? wait_current_trans+0x33/0x110 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878627] ? join_transaction+0x27/0x420 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878639] btrfs_log_dentry_safe+0x60/0x80 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878658] btrfs_sync_file+0x2d1/0x410 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878661] vfs_fsync_range+0x4b/0xb0 Feb 13 13:53:32 lxd01 kernel: [9351710.878663] do_fsync+0x3d/0x70 Feb 13 13:53:32 lxd01 kernel: [9351710.878668] SyS_fdatasync+0x13/0x20 Feb 13 13:53:32 lxd01 kernel: [9351710.878670] do_syscall_64+0x61/0x120 Feb 13 13:53:32 lxd01 kernel: [9351710.878673] entry_SYSCALL64_slow_path+0x25/0x25 Feb 13 13:53:32 lxd01 kernel: [9351710.878674] RIP: 0033:0x7f99461437dd Feb 13 13:53:32 lxd01 kernel: [9351710.878675] RSP: 002b:00007f9936e20f10 EFLAGS: 00000293 ORIG_RAX: 000000000000004b Feb 13 13:53:32 lxd01 kernel: [9351710.878676] RAX: ffffffffffffffda RBX: 0000307d6f5d1070 RCX: 00007f99461437dd Feb 13 13:53:32 lxd01 kernel: [9351710.878677] RDX: 000000000000005c RSI: 0000000000080000 RDI: 000000000000005c Feb 13 13:53:32 lxd01 kernel: [9351710.878678] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000 Feb 13 13:53:32 lxd01 kernel: [9351710.878679] R10: 00000000ffffffff R11: 0000000000000293 R12: 0000000000001000 Feb 13 13:53:32 lxd01 kernel: [9351710.878679] R13: 0000307d6f550b00 R14: 0000000000000000 R15: 0000000000001000 Feb 13 13:53:32 lxd01 kernel: [9351710.878681] Code: 89 85 6c ff ff ff 4c 8b 95 70 ff ff ff 74 23 4c 89 f7 e8 a9 dc f8 ff 48 8b 7d 88 e8 a0 dc f8 ff 8b 85 6c ff ff ff e9 d8 fb ff ff <0f> ff e9 35 fe ff ff 4c 89 55 18 e9 56 fc ff ff e8 60 65 61 eb Feb 13 13:53:32 lxd01 kernel: [9351710.878707] ---[ end trace 81aeb3fb0c68ce00 ]---
BTW we've updated to the latest 4.15 kernel after that.
Quoted 8 lines
Not sure if the removal of 80G has anything to do with this, but this seems that your metadata (along with data) is quite scattered.
It's really recommended to keep some unallocated device space, and one of the method to do that is to use balance to free such scattered space from data/metadata usage.
And that's why balance routine is recommened for btrfs.
The balance might work on that server - it's less than 0.5 TB SSD disks.
However, on multi-terabyte servers with terabytes of data on HDD disks, running balance is not realistic. We have some servers where balance was taking 2 months or so, and was not even 50% done. And the IO load the balance was adding was slowing the things down a lot.
Tomasz Chmielewski
https://lxadm.com
Quoted 11 lines · Tomasz Chmielewski
Quoted 1 line · Qu Wenruo
Is there any kernel message like kernel warning or backtrace?
I see there was this one:
Feb 13 13:53:32 lxd01 kernel: [9351710.878404] ------------[ cut here ]------------ Feb 13 13:53:32 lxd01 kernel: [9351710.878430] WARNING: CPU: 9 PID:7780 at
/home/kernel/COD/linux/fs/btrfs/tree-log.c:3361log_dir_items+0x54b/0x560[btrfs]
(...)
Quoted 5 lines
Feb 13 13:53:32 lxd01 kernel: [9351710.878707] ---[ end trace
81aeb3fb0c68ce00]---BTW we've updated to the latest 4.15 kernel after that.
Also, we were just running a balance there and it printed this in dmesg.
Not sure if it's something serious or not. The filesystem still runs correctly.
[60082.349447] WARNING: CPU: 2 PID: 780 at /home/kernel/COD/linux/fs/btrfs/extent-tree.c:124 btrfs_put_block_group+0x4f/0x60 [btrfs]
[60082.349449] Modules linked in: xt_nat xt_REDIRECT nf_nat_redirect tcp_diag inet_diag sunrpc xt_NFLOG cfg80211 xt_conntrack nfnetlink_log nfnetlink ipt_REJECT nf_reject_ipv4 binfmt_misc veth ebtable_filter ebtables ip6t_MASQUERADE nf_nat_masquerade_ipv6 ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 xt_comment nf_log_ipv4 nf_log_common xt_LOG ipt_MASQUERADE nf_nat_masquerade_ipv4 iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat ip_vs nf_conntrack ip6table_filter ip6_tables iptable_filter xt_CHECKSUM xt_tcpudp iptable_mangle ip_tables x_tables intel_rapl sb_edac x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel btrfs bridge pcbc zstd_compress stp llc aesni_intel aes_x86_64 crypto_simd glue_helper input_leds
[60082.349471] cryptd intel_cstate intel_rapl_perf serio_raw shpchp lpc_ich acpi_pad mac_hid autofs4 raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 multipath linear ttm drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops igb drm dca ahci ptp libahci pps_core i2c_algo_bit wmi
[60082.349488] CPU: 2 PID: 780 Comm: btrfs-cleaner Tainted: G W
4.15.3-041503-generic #201802120730
[60082.349489] Hardware name: ASUSTeK COMPUTER INC. Z10PA-U8 Series/Z10PA-U8 Series, BIOS 0601 06/26/2015
[60082.349497] RIP: 0010:btrfs_put_block_group+0x4f/0x60 [btrfs]
[60082.349497] RSP: 0018:ffffc1468d3afe48 EFLAGS: 00010206
[60082.349498] RAX: 0000000000000000 RBX: ffff9d072f740000 RCX: 0000000000000000
[60082.349499] RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffff9d072bcb8c00
[60082.349499] RBP: ffffc1468d3afe50 R08: ffff9d072bcbfc00 R09: 0000000180200019
[60082.349500] R10: ffffc1468d3afe38 R11: 0000000000000100 R12: ffff9d072b87ce00
[60082.349500] R13: ffff9d072bcb8c00 R14: ffff9d072b87ced0 R15: ffff9d072bcb8d20
[60082.349501] FS: 0000000000000000(0000) GS:ffff9d073f280000(0000) knlGS:0000000000000000
[60082.349502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[60082.349502] CR2: 00007f9221375000 CR3: 00000019a660a005 CR4: 00000000001606e0
[60082.349503] Call Trace:
[60082.349513] btrfs_delete_unused_bgs+0x243/0x3c0 [btrfs]
[60082.349521] cleaner_kthread+0x159/0x170 [btrfs]
[60082.349524] kthread+0x121/0x140
[60082.349531] ? __btree_submit_bio_start+0x20/0x20 [btrfs]
[60082.349533] ? kthread_create_worker_on_cpu+0x70/0x70
[60082.349535] ret_from_fork+0x35/0x40
[60082.349536] Code: e8 01 00 00 48 85 c0 75 1a 48 89 fb 48 8b bf d8 00 00 00 e8 14 3a 66 d7 48 89 df e8 0c 3a 66 d7 5b 5d c3 0f ff eb e2 0f ff eb cb <0f> ff eb ce 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00
[60082.349554] ---[ end trace 9492ee1b902c858d ]---Tomasz Chmielewski
https://lxadm.com
Quoted 10 lines · Tomasz Chmielewski
Quoted 1 line · Qu Wenruo
Is there any kernel message like kernel warning or backtrace?
I see there was this one:
Feb 13 13:53:32 lxd01 kernel: [9351710.878404] ------------[ cut here ]------------ Feb 13 13:53:32 lxd01 kernel: [9351710.878430] WARNING: CPU: 9 PID: 7780 at
/home/kernel/COD/linux/fs/btrfs/tree-log.c:3361
Something with tree log (used by fsync) is not running as expected, and it seems to be a big problem.
As the code shows that, btrfs fails to search the key in log tree.
No wonder why MySQL is reporting error, as fsync is not executed correctly.
I strongly recommend to run offline btrfs check to ensure metadata is not corrupted before causing more problems.
Quoted 102 lines
log_dir_items+0x54b/0x560[btrfs] Feb 13 13:53:32 lxd01 kernel:[9351710.878431] Modules linked in: nfnetlink_queue bluetooth
ecdh_genericxt_natxt_REDIRECTnf_nat_redirectsunrpc cfg80211tcp_diaginet_diagxt_NFLOGnfnetlink_lognfnetlinkxt_conntrackipt_REJECTnf_reject_ipv4binfmt_miscvethebtable_filterebtablesip6t_MASQUERADEnf_nat_masquerade_ipv6ip6table_natnf_conntrack_ipv6nf_defrag_ipv6nf_nat_ipv6xt_commentnf_log_ipv4nf_log_commonxt_LOGipt_MASQUERADEnf_nat_masquerade_ipv4iptable_natnf_conntrack_ipv4nf_defrag_ipv4nf_nat_ipv4nf_natip_vsnf_conntrackip6table_filterip6_tablesiptable_filterxt_CHECKSUMxt_tcpudpiptable_mangleip_tablesx_tablesintel_raplx86_pkg_temp_thermalintel_powerclampcoretempkvm_intelkvm irqbypass btrfs bridge stp llccrct10dif_pclmulcrc32_pclmulghash_clmulni_intelpcbczstd_compressaesni_intelaes_x86_64Feb 13 13:53:32 lxd01 kernel: [9351710.878460]crypto_simdglue_helpercryptdinput_ledsintel_cstateipmi_ssifintel_rapl_perfserio_rawlpc_ichshpchpipmi_devintfipmi_msghandlertpm_infineonacpi_padmac_hidautofs4 raid10 raid456async_raid6_recovasync_memcpyasync_pqasync_xorasync_txxorraid6_pqlibcrc32c raid1 raid0 multipath linear ttmdrm_kms_helpersyscopyarea sysfillrect sysimgbltfb_sys_fopsigb drm dca ahci ptppps_corelibahcii2c_algo_bitwmiFeb 13 13:53:32 lxd01 kernel: [9351710.878484] CPU: 9 PID: 7780 Comm: TaskSchedulerBa Tainted: G W 4.14.0-041400rc6-generic #201710230731 Feb 13 13:53:32 lxd01 kernel: [9351710.878485] Hardware name: ASUSTeK COMPUTER INC. Z10PA-U8Series/Z10PA-U8 Series, BIOS 0601 06/26/2015 Feb 13 13:53:32 lxd01
kernel: [9351710.878486] task:ffff9454227d1700task.stack:ffffabc6a810c000Feb 13 13:53:32 lxd01 kernel: [9351710.878502] RIP:
0010:log_dir_items+0x54b/0x560 [btrfs] Feb 13 13:53:32 lxd01 kernel:[9351710.878502] RSP: 0018:ffffabc6a810f980 EFLAGS: 00010202 Feb 13 13:53:32 lxd01 kernel: [9351710.878503] RAX: 0000000000000001 RBX: 000000000008b771 RCX: 0000000000000000 Feb 13 13:53:32 lxd01 kernel: [9351710.878504] RDX: 0000000000000000 RSI: 0000000000000000 RDI:0000000000000000 Feb 13 13:53:32 lxd01 kernel: [9351710.878505] RBP:
ffffabc6a810fa28R08:ffff9491a8f05540R09: 0000000000000008 Feb 13 13:53:32 lxd01 kernel: [9351710.878506] R10: 0000000000000000 R11:ffffabc6a810f934R12:ffffabc6a810fe50Feb 13 13:53:32 lxd01 kernel:[9351710.878506] R13: ffff94666d426000 R14: ffff9491a8f05540 R15:0000000000000054 Feb 13 13:53:32 lxd01 kernel: [9351710.878508] FS:
00007f9936e22700(0000) GS:ffff9491bf440000(0000) knlGS:0000000000000000 Feb 13 13:53:32 lxd01 kernel: [9351710.878508]
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 Feb 13 13:53:32 lxd01 kernel: [9351710.878509] CR2:00007f6abef4d7b0CR3:00000023ecaf7006CR4:00000000001606e0Feb 13 13:53:32 lxd01 kernel:[9351710.878510] Call Trace: Feb 13 13:53:32 lxd01 kernel: [9351710.878524] ? btrfs_search_slot+0x81b/0x9c0 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878538] log_directory_changes+0x83/0xd0 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878551] btrfs_log_inode+0xa24/0x11a0 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878563] ? generic_bin_search.constprop.37+0xe7/0x1f0 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878565] ? find_inode+0x59/0xb0 Feb 13 13:53:32 lxd01 kernel: [9351710.878567] ? iget5_locked+0x9e/0x1e0 Feb 13 13:53:32 lxd01 kernel: [9351710.878582] log_new_dir_dentries+0x203/0x4a7 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878595] btrfs_log_inode_parent+0x6c2/0xa10 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878598] ? pagevec_lookup_tag+0x21/0x30 Feb 13 13:53:32 lxd01 kernel: [9351710.878599] ? __filemap_fdatawait_range+0x9a/0x170 Feb 13 13:53:32 lxd01 kernel: [9351710.878614] ? wait_current_trans+0x33/0x110 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878627] ? join_transaction+0x27/0x420 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878639] btrfs_log_dentry_safe+0x60/0x80 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878658] btrfs_sync_file+0x2d1/0x410 [btrfs] Feb 13 13:53:32 lxd01 kernel: [9351710.878661] vfs_fsync_range+0x4b/0xb0 Feb 13 13:53:32 lxd01 kernel: [9351710.878663] do_fsync+0x3d/0x70 Feb 13 13:53:32 lxd01 kernel: [9351710.878668] SyS_fdatasync+0x13/0x20 Feb 13 13:53:32 lxd01 kernel: [9351710.878670] do_syscall_64+0x61/0x120 Feb 13 13:53:32 lxd01 kernel: [9351710.878673] entry_SYSCALL64_slow_path+0x25/0x25 Feb 13 13:53:32 lxd01 kernel: [9351710.878674] RIP: 0033:0x7f99461437dd Feb 13 13:53:32 lxd01 kernel: [9351710.878675] RSP: 002b:00007f9936e20f10 EFLAGS: 00000293 ORIG_RAX: 000000000000004b Feb 13 13:53:32 lxd01 kernel: [9351710.878676] RAX: ffffffffffffffda RBX: 0000307d6f5d1070 RCX:
00007f99461437ddFeb 13 13:53:32 lxd01 kernel: [9351710.878677] RDX:000000000000005cRSI: 0000000000080000 RDI:000000000000005cFeb 13 13:53:32 lxd01 kernel: [9351710.878678] RBP: 0000000000000000 R08:
0000000000000000 R09: 0000000000000000 Feb 13 13:53:32 lxd01 kernel:[9351710.878679] R10: 00000000ffffffff R11: 0000000000000293 R12:0000000000001000 Feb 13 13:53:32 lxd01 kernel: [9351710.878679] R13:
0000307d6f550b00R14: 0000000000000000 R15: 0000000000001000 Feb 13 13:53:32 lxd01 kernel: [9351710.878681] Code: 89 85 6c ff ff ff 4c 8b 95 70 ff ff ff 74 23 4c 89 f7 e8 a9 dc f8 ff 48 8b 7d 88 e8 a0 dc f8 ff 8b 85 6c ff ff ff e9 d8 fb ff ff <0f> ff e9 35 fe ff ff 4c 89 55 18 e9 56 fc ff ff e8 60 65 61 eb Feb 13 13:53:32 lxd01 kernel:[9351710.878707] ---[ end trace 81aeb3fb0c68ce00 ]---BTW we've updated to the latest 4.15 kernel after that.
Quoted 9 lines
Not sure if the removal of 80G has anything to do with this, but this seems that your metadata (along with data) is quite scattered.
It's really recommended to keep some unallocated device space, and one of the method to do that is to use balance to free such scattered space from data/metadata usage.
And that's why balance routine is recommened for btrfs.
Quoted 7 lines
The balance might work on that server - it's less than 0.5 TB SSD disks.
However, on multi-terabyte servers with terabytes of data on HDD disks, running balance is not realistic> We have some servers where balance was taking 2 months or so, and was not even 50% done. And the IO load the balance was adding was slowing the things down a lot.
How did you do the balance?
Btrfs are completely OK to relocate certain chunks which meet certain condition.
For example, only to relocate chunk whose used space is lower than 15%.
If you're relocating the whole fs, no wonder it will takes a long long time.
And further more, btrfs is super fast in creating snapshots, but at the cost of dramatically slowing down balance and snapshot deletion.
So abusing snapshots is not a good idea for btrfs especially for balance.
Thanks,
Qu
Quoted 3 lines
Tomasz Chmielewski https://lxadm.com
Tomasz Chmielewski posted on Thu, 15 Feb 2018 16:02:59 +0900 as excerpted:
Quoted 7 lines
Quoted 7 lines
Not sure if the removal of 80G has anything to do with this, but this seems that your metadata (along with data) is quite scattered.
It's really recommended to keep some unallocated device space, and one of the method to do that is to use balance to free such scattered space from data/metadata usage.
Quoted 8 lines
Quoted 1 line
And that's why balance routine is recommened for btrfs.
The balance might work on that server - it's less than 0.5 TB SSD disks.
However, on multi-terabyte servers with terabytes of data on HDD disks, running balance is not realistic. We have some servers where balance was taking 2 months or so, and was not even 50% done. And the IO load the balance was adding was slowing the things down a lot.
Try a filtered balance. Something along the lines of:
btrfs balance start -dusage=10 <filesystem-path>
The -dusage number, a limit on the chunk usage percentage, can start small, even 0, and be increased as necessary, until btrfs fi usage reports data size (currently 411 GiB) closer to data usage (currently 246.14 GiB), with the freed space returning to unallocated.
I'd shoot for reducing data size to under 300 GiB, thus returning over 100 GiB to unallocated, while hopefully not requiring too high a -dusage percentage and thus too long a balance time. You could get it down under 250 gig size, but that would likely take a lot of rewriting for little additional gain, since with it under 300 gig size you should already have over 100 gig unallocated.
Balance time should be quite short for low percentages, with a big payback if there's quite a few chunks with little usage, because at 10%, the filesystem can get rid of 10 chunks while only rewriting the equivalent of a single full chunk.
Obviously as the chunk usage percentage goes up, the payback goes down, so at 50%, it can only clear two chunks while writing one, and at 66%, it has to write two chunks worth to clear three. Above that (tho I tend to round up to 70% here) is seldom worth it until the filesystem gets quite full and you're really fighting to keep a few gigs of unallocated space. (As Qu indicated, you always want at least a gig of unallocated space, on at least two devices if you're doing raid1.)
If you really wanted you could do the same with -musage for metadata, except that's not so bad, only 9 gig size, 3 gig used. But you could free 5 gigs or so, if desired.
That's assuming there's no problem. I see a followup indicating you're seeing problems in dmesg with a balance, however, and will let others deal with that.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 70 lines · Tomasz Chmielewski
Quoted 11 lines · Tomasz Chmielewski
Quoted 1 line · Qu Wenruo
Is there any kernel message like kernel warning or backtrace?
I see there was this one:
Feb 13 13:53:32 lxd01 kernel: [9351710.878404] ------------[ cut here ]------------ Feb 13 13:53:32 lxd01 kernel: [9351710.878430] WARNING: CPU: 9 PID:7780 at
/home/kernel/COD/linux/fs/btrfs/tree-log.c:3361log_dir_items+0x54b/0x560[btrfs](...)
Quoted 5 lines
Feb 13 13:53:32 lxd01 kernel: [9351710.878707] ---[ end trace
81aeb3fb0c68ce00]---BTW we've updated to the latest 4.15 kernel after that.
Also, we were just running a balance there and it printed this in dmesg.
Not sure if it's something serious or not. The filesystem still runs correctly.
[60082.349447] WARNING: CPU: 2 PID: 780 at /home/kernel/COD/linux/fs/btrfs/extent-tree.c:124 btrfs_put_block_group+0x4f/0x60 [btrfs] [60082.349449] Modules linked in: xt_nat xt_REDIRECT nf_nat_redirect
tcp_diaginet_diagsunrpcxt_NFLOGcfg80211xt_conntracknfnetlink_lognfnetlinkipt_REJECTnf_reject_ipv4binfmt_miscvethebtable_filterebtablesip6t_MASQUERADEnf_nat_masquerade_ipv6ip6table_natnf_conntrack_ipv6nf_defrag_ipv6nf_nat_ipv6xt_commentnf_log_ipv4nf_log_commonxt_LOGipt_MASQUERADEnf_nat_masquerade_ipv4iptable_natnf_conntrack_ipv4nf_defrag_ipv4nf_nat_ipv4nf_natip_vsnf_conntrackip6table_filterip6_tablesiptable_filterxt_CHECKSUMxt_tcpudpiptable_mangleip_tablesx_tablesintel_raplsb_edacx86_pkg_temp_thermalintel_powerclampcoretempkvm_intelkvm irqbypasscrct10dif_pclmulcrc32_pclmulghash_clmulni_intelbtrfs bridge pcbczstd_compressstp llcaesni_intelaes_x86_64crypto_simdglue_helperinput_leds[60082.349471] cryptd intel_cstate intel_rapl_perf serio_raw shpchp
lpc_ichacpi_padmac_hidautofs4 raid10 raid456async_raid6_recovasync_memcpyasync_pqasync_xorasync_txxorraid6_pqlibcrc32c raid1 raid0 multipath linear ttmdrm_kms_helpersyscopyarea sysfillrect sysimgbltfb_sys_fopsigb drm dca ahci ptp libahcipps_corei2c_algo_bitwmi[60082.349488] CPU: 2 PID: 780 Comm: btrfs-cleaner Tainted: G W 4.15.3-041503-generic #201802120730 [60082.349489] Hardware name: ASUSTeK COMPUTER INC. Z10PA-U8 Series/Z10PA-U8 Series, BIOS 0601 06/26/2015 [60082.349497] RIP: 0010:btrfs_put_block_group+0x4f/0x60 [btrfs] [60082.349497] RSP: 0018:ffffc1468d3afe48 EFLAGS: 00010206 [60082.349498] RAX: 0000000000000000 RBX: ffff9d072f740000 RCX: 0000000000000000 [60082.349499] RDX: 0000000000000001 RSI: 0000000000000246 RDI: ffff9d072bcb8c00 [60082.349499] RBP: ffffc1468d3afe50 R08: ffff9d072bcbfc00 R09: 0000000180200019 [60082.349500] R10: ffffc1468d3afe38 R11: 0000000000000100 R12: ffff9d072b87ce00 [60082.349500] R13: ffff9d072bcb8c00 R14: ffff9d072b87ced0 R15: ffff9d072bcb8d20 [60082.349501] FS: 0000000000000000(0000) GS:ffff9d073f280000(0000) knlGS:0000000000000000 [60082.349502] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [60082.349502] CR2: 00007f9221375000 CR3: 00000019a660a005 CR4: 00000000001606e0 [60082.349503] Call Trace: [60082.349513] btrfs_delete_unused_bgs+0x243/0x3c0 [btrfs] [60082.349521] cleaner_kthread+0x159/0x170 [btrfs]
Quoted 9 lines
[60082.349524] kthread+0x121/0x140 [60082.349531] ? __btree_submit_bio_start+0x20/0x20 [btrfs] [60082.349533] ? kthread_create_worker_on_cpu+0x70/0x70 [60082.349535] ret_from_fork+0x35/0x40 [60082.349536] Code: e8 01 00 00 48 85 c0 75 1a 48 89 fb 48 8b bf d8 00 00 00 e8 14 3a 66 d7 48 89 df e8 0c 3a 66 d7 5b 5d c3 0f ff eb e2 0f ff eb cb <0f> ff eb ce 0f 1f 00 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 [60082.349554] ---[ end trace 9492ee1b902c858d ]---
If you have the vmlinux for that particular kernel can you run the following command and provide the output:
./scripts/faddr2line <path-to-vmlinux> btrfs_delete_unused_bgs+0x243/0x3c0
./scripts/faddr2line is in the kernel source directory. This warn on indicates we are freeing an unused blockgroup (one which shouldn't have any space reserved in it) yet we trigger a warning due to having reserved space for that block group. Quite strange
Quoted 8 lines
Tomasz Chmielewski
https://lxadm.comSignature · 3 lines
To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line · Tomasz Chmielewski
Just FYI, how dangerous running btrfs can be - we had a fatal, unrecoverable MySQL corruption when btrfs decided to do one of these "I have ~50 GB left, so let's do out of space (and corrupt some files at the same time, ha ha!)".
Thanks for reporting.
Quoted 1 line
Running btrfs RAID-1 with kernel 4.14.
Can you pls let us know..
1. What tool cli/reported/identified that data is corrupted?
2. Disk error stat using.. btrfs dev stat <mnt> (dev stat is stored on disk)
3. Wheather the disk was mounted as degraded any time before?
Thanks, Anand
Quoted 9 lines · Anand Jain
Quoted 1 line · Tomasz Chmielewski
Just FYI, how dangerous running btrfs can be - we had a fatal, unrecoverable MySQL corruption when btrfs decided to do one of these "I have ~50 GB left, so let's do out of space (and corrupt some files at the same time, ha ha!)".
Thanks for reporting.
Quoted 1 line
Running btrfs RAID-1 with kernel 4.14.
Can you pls let us know..
1. What tool cli/reported/identified that data is corrupted?
mysqld log - mysqld would refuse to start because of database corruption.
And, the database wouldn't start even when "innodb_force_recovery = " was set to a high/max value.
In the past, with lower kernel versions, we had a similar issue with mongod - it wouldn't start anymore due to some corruption which happened when we hit "out of space" (again, with dozens of GBs free space).
Quoted 2 lines
2. Disk error stat using.. btrfs dev stat <mnt>
(dev stat is stored on disk)
# btrfs dev stat /var/lib/lxd [/dev/sda3].write_io_errs 0 [/dev/sda3].read_io_errs 0 [/dev/sda3].flush_io_errs 0 [/dev/sda3].corruption_errs 0 [/dev/sda3].generation_errs 0 [/dev/sdb3].write_io_errs 0 [/dev/sdb3].read_io_errs 0 [/dev/sdb3].flush_io_errs 0 [/dev/sdb3].corruption_errs 0 [/dev/sdb3].generation_errs 0
Quoted 1 line
3. Wheather the disk was mounted as degraded any time before?
No. Everything healthy with the disks.
Tomasz Chmielewski
https://lxadm.com