csum failed root raveled during balance
Hello devs,
I tested BTRFS in production for about a month:
21:08:17 up 34 days, 2:21, 3 users, load average: 0.06, 0.02, 0.00
Without power blackout, hardware failure, SSD's SMART is flawless etc. The tests ended with:
root@node0:~# dmesg | grep BTRFS | grep warn 185:980:[2927472.393557] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 186:981:[2927472.394158] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 191:986:[2928224.169814] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 192:987:[2928224.171433] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 206:1001:[2928298.039516] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 207:1002:[2928298.043103] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 208:1004:[2932213.513424] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 4564959232 csum 0xc616afb4 expected csum 0x5425e489 mirror 1 209:1005:[2932235.666368] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 16989835264 csum 0xd63ed5da expected csum 0x7429caa1 mirror 1 210:1072:[2936767.229277] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1 211:1073:[2936767.276229] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1
Above has been revealed during below command and quite high IO usage by few VMs (Linux on top Ext4 with firebird database, lots of random read/writes, two others with Windows 2016 and Windows Update in the background):
btrfs balance start -dusage=85 ./
Surprisingly without error counter increasing (how this is even possible?!):
root@node0:/var/lib/libvirt/images# backup_dir="/var/lib/libvirt"; btrfs
fi df $backup_dir; btrfs filesystem usage $backup_dir; btrfs dev stats
$backup_dir
Data, single: total=203.00GiB, used=188.28GiB
System, single: total=32.00MiB, used=48.00KiB
Metadata, single: total=3.00GiB, used=856.27MiB
GlobalReserve, single: total=512.00MiB, used=0.00B
Overall:
Device size: 300.00GiB
Device allocated: 206.03GiB
Device unallocated: 93.97GiB
Device missing: 0.00B
Used: 189.12GiB
Free (estimated): 108.68GiB (min: 108.68GiB)
Data ratio: 1.00
Metadata ratio: 1.00
Global reserve: 512.00MiB (used: 0.00B)Data,single: Size:203.00GiB, Used:188.28GiB
/dev/mapper/raid10-images 203.00GiB
Metadata,single: Size:3.00GiB, Used:856.27MiB
/dev/mapper/raid10-images 3.00GiB
System,single: Size:32.00MiB, Used:48.00KiB
/dev/mapper/raid10-images 32.00MiB
Unallocated:
/dev/mapper/raid10-images 93.97GiB
[/dev/mapper/raid10-images].write_io_errs 0 [/dev/mapper/raid10-images].read_io_errs 0 [/dev/mapper/raid10-images].flush_io_errs 0 [/dev/mapper/raid10-images].corruption_errs 0
Few words about the setup Hardware layout:
/dev/sda Model=Samsung SSD 850 PRO 256GB, FwRev=EXM02B6Q, SerialNo=S251NX0H822367V/dev/sdb Model=Samsung SSD 850 PRO 256GB, FwRev=EXM02B6Q, SerialNo=S251NX0H822370A/dev/sdc Model=INTEL SSDSC2BP240G4, FwRev=L2010420, SerialNo=BTJR6141002D240AGN/dev/sdd Model=INTEL SSDSC2BP240G4, FwRev=L2010420, SerialNo=BTJR6063000F240AGN
Linux Software RAID layout (mdraid):
md1 : active raid10 sdc1[2] sdb2[1] sdd1[3] sda2[0]
468596736 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
bitmap: 0/4 pages [0KB], 65536KB chunkmd0 : active raid1 sda1[0] sdb1[1]
15616000 blocks super 1.2 [2/2] [UU]
Logical layout:
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 238.5G 0 disk
├─sda1 8:1 0 14.9G 0 part
│ └─md0 9:0 0 14.9G 0 raid1
│ └─raid1-rootfs 253:2 0 14.9G 0 lvm /
└─sda2 8:2 0 223.6G 0 part
└─md1 9:1 0 446.9G 0 raid10
├─raid10-images 253:0 0 300G 0 lvm /var/lib/libvirt
└─raid10-swapfs 253:1 0 4G 0 lvm
sdb 8:16 0 238.5G 0 disk
├─sdb1 8:17 0 14.9G 0 part
│ └─md0 9:0 0 14.9G 0 raid1
│ └─raid1-rootfs 253:2 0 14.9G 0 lvm /
└─sdb2 8:18 0 223.6G 0 part
└─md1 9:1 0 446.9G 0 raid10
├─raid10-images 253:0 0 300G 0 lvm /var/lib/libvirt
└─raid10-swapfs 253:1 0 4G 0 lvm
sdc 8:32 0 223.6G 0 disk
└─sdc1 8:33 0 223.6G 0 part
└─md1 9:1 0 446.9G 0 raid10
├─raid10-images 253:0 0 300G 0 lvm /var/lib/libvirt
└─raid10-swapfs 253:1 0 4G 0 lvm
sdd 8:48 0 223.6G 0 disk
└─sdd1 8:49 0 223.6G 0 part
└─md1 9:1 0 446.9G 0 raid10
├─raid10-images 253:0 0 300G 0 lvm /var/lib/libvirt
└─raid10-swapfs 253:1 0 4G 0 lvm BTRFS is mounted with below options:
/dev/mapper/raid10-images on /var/lib/libvirt type btrfs (rw,noatime,nodiratime,compress=lzo:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/)
As you can see KVM VMs reside on BTRFS volume on top of LVM partition on top of software RAID10 on top of 4 SSD drives. Let's put away performance point of view which is at least 15-20 times worse (number of possible IO ops in VM and latency) comparing to the same setup but image residing on top of Ext4 volume. Do you also see so huge performance impact in your environment? The VMs lay on preallocated RAW images on top of BTRFS filesystem.
The filesystem was created and used with the following tools:
root@node0:/var/lib/libvirt/images# uname -a Linux node0 4.15.0-0.bpo.2-amd64 #1 SMP Debian 4.15.11-1~bpo9+1 (2018-04-07) x86_64 GNU/Linux
root@node0:/var/lib/libvirt/images# btrfs version btrfs-progs v4.13.3
root@node0:/var/lib/libvirt/images# cat /etc/debian_version 9.4
BTRFS progs and kernel was from backports. After I finished restoring from backup I ran again btrfs balance:
root@node0:/var/lib/libvirt/images# btrfs balance start ./ WARNING:
Full balance without filters requested. This operation is very intense and takes potentially very long. It is recommended to use the balance filters to narrow down the scope of balance. Use 'btrfs balance start --full-balance' option to skip this warning. The operation will start in 10 seconds. Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
May 22 21:41:28 node0 kernel: [2948094.226177] BTRFS info (device dm-0): relocating block group 437201666048 flags data May 22 21:41:28 node0 kernel: [2948094.818796] BTRFS info (device dm-0): found 1518 extents May 22 21:41:30 node0 kernel: [2948096.678234] BTRFS info (device dm-0): found 1518 extents May 22 21:41:30 node0 kernel: [2948096.736582] BTRFS info (device dm-0): relocating block group 436127924224 flags data May 22 21:41:32 node0 kernel: [2948098.811837] BTRFS info (device dm-0): found 3932 extents May 22 21:41:34 node0 kernel: [2948100.397485] BTRFS info (device dm-0): found 3932 extents May 22 21:41:34 node0 kernel: [2948100.473214] BTRFS info (device dm-0): relocating block group 428611731456 flags data May 22 21:41:37 node0 kernel: [2948103.862030] BTRFS info (device dm-0): found 3519 extents May 22 21:41:38 node0 kernel: [2948104.961367] BTRFS info (device dm-0): found 3519 extents May 22 21:41:39 node0 kernel: [2948105.026999] BTRFS info (device dm-0): relocating block group 427537989632 flags data May 22 21:41:40 node0 kernel: [2948106.155623] BTRFS warning (device dm-0): csum failed root -9 ino 336 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 May 22 21:41:40 node0 kernel: [2948106.156301] BTRFS warning (device
dm-0): csum failed root -9 ino 336 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1
ERROR: error during balancing './': Input/output error There may be more info in syslog - try dmesg | tail
root@node0:/var/lib/libvirt/images# backup_dir="/var/lib/libvirt"; btrfs dev stats $backup_dir [/dev/mapper/raid10-images].write_io_errs 0 [/dev/mapper/raid10-images].read_io_errs 0 [/dev/mapper/raid10-images].flush_io_errs 0 [/dev/mapper/raid10-images].corruption_errs 0 [/dev/mapper/raid10-images].generation_errs 0
Again no errors in btrfs stats. I am pretty sure that I corrupted the drive during remount:
mount -o remount,noatime,nodiratime,rw,ssd,space_cache,compress=lzo,autodefrag /dev/mapper/raid10-images /var/lib/libvirt
when I changed BTRFS compress parameters. Or during umount (can't recall now):
May 2 07:15:39 node0 kernel: [1168145.677431] WARNING: CPU: 6 PID: 3763 at /build/linux-8B5M4n/linux-4.15.11/fs/direct-io.c:293 dio_complete+0x1d6/0x220 May 2 07:15:39 node0 kernel: [1168145.678811] Modules linked in: fuse
ufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs vhost_net vhost tap tun ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter binfmt_misc bridge 8021q garp mrp stp llc snd_hda_codec_hdmi intel_rapl x86_pkg_temp_thermal intel_powerclamp coretemp snd_hda_codec_realtek kvm_intel snd_hda_codec_generic kvm i915 irqbypass crct10dif_pclmul snd_hda_intel crc32_pclmul ghash_clmulni_intel snd_hda_codec intel_cstate snd_hda_core iTCO_wdt iTCO_vendor_support
intel_uncore drm_kms_helper snd_hwdep wmi_bmof intel_rapl_perf joydev
evdev pcspkr snd_pcm snd_timer drm snd soundcore i2c_algo_bit sg mei_me lpc_ich shpchp mfd_core mei ie31200_e
dac wmi video button ib_iser rdma_cm iw_cm ib_cm ib_core configfs iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi ip_tables
May 2 07:15:39 node0 kernel: [1168145.685202] x_tables autofs4 ext4
crc16 mbcache jbd2 fscrypto ecb btrfs zstd_decompress zstd_compress xxhash raid456 async_raid6_recov async_mem
cpy async_pq async_xor async_tx xor raid6_pq libcrc32c crc32c_generic raid0 multipath linear hid_generic usbhid hid dm_mod raid10 raid1 md_mod sd_mod crc32c_intel ahci i2c_i801 lib
ahci aesni_intel xhci_pci aes_x86_64 ehci_pci libata crypto_simd xhci_hcd ehci_hcd cryptd glue_helper e1000e scsi_mod ptp usbcore pps_core usb_common fan thermal
May 2 07:15:39 node0 kernel: [1168145.689057] CPU: 6 PID: 3763 Comm: kworker/6:2 Not tainted 4.15.0-0.bpo.2-amd64 #1 Debian 4.15.11-1~bpo9+1 May 2 07:15:39 node0 kernel: [1168145.690347] Hardware name: LENOVO ThinkServer TS140/ThinkServer TS140, BIOS FBKTB3AUS 06/16/2015 May 2 07:15:39 node0 kernel: [1168145.691659] Workqueue: dio/dm-0 dio_aio_complete_work May 2 07:15:39 node0 kernel: [1168145.692935] RIP: 0010:dio_complete+0x1d6/0x220 May 2 07:15:39 node0 kernel: [1168145.694275] RSP: 0018:ffff9abc68447e50 EFLAGS: 00010286 May 2 07:15:39 node0 kernel: [1168145.695605] RAX: 00000000fffffff0 RBX: ffff8e33712e3480 RCX: ffff9abc68447c88 May 2 07:15:39 node0 kernel: [1168145.697024] RDX: fffff1dcc92e4c1f RSI: 0000000000000000 RDI: 0000000000000246 May 2 07:15:39 node0 kernel: [1168145.698389] RBP: 0000000000005000 R08: 0000000000000000 R09: ffffffffb7a075c0 May 2 07:15:39 node0 kernel: [1168145.699703] R10: ffff8e33bb4223c0 R11: 0000000000000195 R12: 0000000000005000 May 2 07:15:39 node0 kernel: [1168145.701044] R13: 0000000000000003 R14: 0000000403060000 R15: ffff8e33712e3500 May 2 07:15:39 node0 kernel: [1168145.702238] FS: 0000000000000000(0000) GS:ffff8e349eb80000(0000) knlGS:0000000000000000 May 2 07:15:39 node0 kernel: [1168145.703475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 2 07:15:39 node0 kernel: [1168145.704733] CR2: 00007ff89915b08e CR3: 00000005b2e0a005 CR4: 00000000001626e0 May 2 07:15:39 node0 kernel: [1168145.705955] Call Trace: May 2 07:15:39 node0 kernel: [1168145.707151] process_one_work+0x177/0x360 May 2 07:15:39 node0 kernel: [1168145.708373] worker_thread+0x4d/0x3c0 May 2 07:15:39 node0 kernel: [1168145.709501] kthread+0xf8/0x130 May 2 07:15:39 node0 kernel: [1168145.710603] ? process_one_work+0x360/0x360 May 2 07:15:39 node0 kernel: [1168145.711701] ? kthread_create_worker_on_cpu+0x70/0x70 May 2 07:15:39 node0 kernel: [1168145.712845] ? SyS_exit_group+0x10/0x10 May 2 07:15:39 node0 kernel: [1168145.713973] ret_from_fork+0x35/0x40 May 2 07:15:39 node0 kernel: [1168145.715072] Code: 8b 78 30 48 83 7f 58 00 0f 84 e5 fe ff ff 49 8d 54 2e ff 4c 89 f6 48 c1 fe 0c 48 c1 fa 0c e8 c2 e0 f3 ff 85 c0 0f 84 c8 fe ff f f <0f> 0b e9 c1 fe ff ff 8b 47 20 a8 10 0f 84 e2 fe ff ff 48 8b 77 May 2 07:15:39 node0 kernel: [1168145.717349] ---[ end trace
cfa707d6465e13d2 ]---
If someone is interested in investigating then please let me know. The data is not important. The lack of incrementing read_io_errs is particularly critical IMHO.
Signature · 3 lines
PGP Public Key (RSA/4096b): ID: 0xF2C6EA10 SHA-1: 51DA 40EE 832A 0572 5AD8 B3C0 7AFF 69E1 F2C6 EA10
Quoted 35 lines · ein
Hello devs,
I tested BTRFS in production for about a month:
21:08:17 up 34 days, 2:21, 3 users, load average: 0.06, 0.02, 0.00
Without power blackout, hardware failure, SSD's SMART is flawless etc. The tests ended with:
root@node0:~# dmesg | grep BTRFS | grep warn 185:980:[2927472.393557] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 186:981:[2927472.394158] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 191:986:[2928224.169814] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 192:987:[2928224.171433] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 206:1001:[2928298.039516] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 207:1002:[2928298.043103] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 208:1004:[2932213.513424] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 4564959232 csum 0xc616afb4 expected csum 0x5425e489 mirror 1 209:1005:[2932235.666368] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 16989835264 csum 0xd63ed5da expected csum 0x7429caa1 mirror 1 210:1072:[2936767.229277] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1 211:1073:[2936767.276229] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1
Quoted 4 lines
Above has been revealed during below command and quite high IO usage by few VMs (Linux on top Ext4 with firebird database, lots of random read/writes, two others with Windows 2016 and Windows Update in the background):
I believe you are hitting the issue described here:
https://www.mail-archive.com/[email protected]/msg25656.html
Essentially the way qemu operates on vm images atop btrfs is prone to producing such errors. As a matter of fact, other filesystems also suffer from this(i.e pages modified while being written, however due to lack of CRC on the data they don't detect it). Can you confirm that those inodes (312/314/319/219962/219915) belong to vm images files?
IMHO the best course of action would be to disable checksumming for you vm files.
For some background I suggest you read the following LWN articles:
https://lwn.net/Articles/486311/
https://lwn.net/Articles/442355/
Quoted 71 lines
when I changed BTRFS compress parameters. Or during umount (can't recall now):
May 2 07:15:39 node0 kernel: [1168145.677431] WARNING: CPU: 6 PID: 3763 at /build/linux-8B5M4n/linux-4.15.11/fs/direct-io.c:293 dio_complete+0x1d6/0x220 May 2 07:15:39 node0 kernel: [1168145.678811] Modules linked in: fuseufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
vhost_netvhost tap tunebtable_filterebtables ip6table_filterip6_tablesiptable_filterbinfmt_miscbridge 8021q garp mrp stp llcsnd_hda_codec_hdmiintel_raplx86_pkg_temp_thermalintel_powerclampcoretempsnd_hda_codec_realtekkvm_intelsnd_hda_codec_generickvm i915 irqbypasscrct10dif_pclmulsnd_hda_intelcrc32_pclmulghash_clmulni_intelsnd_hda_codecintel_cstatesnd_hda_coreiTCO_wdtiTCO_vendor_support
intel_uncoredrm_kms_helpersnd_hwdepwmi_bmofintel_rapl_perfjoydev
evdev pcspkrsnd_pcmsnd_timerdrm snd soundcorei2c_algo_bitsgmei_melpc_ichshpchpmfd_coremeiie31200_e
dac wmi video buttonib_iserrdma_cmiw_cmib_cmib_coreconfigfsiscsi_tcplibiscsi_tcplibiscsiscsi_transport_iscsiip_tablesMay 2 07:15:39 node0 kernel: [1168145.685202] x_tables autofs4 ext4crc16 mbcache jbd2 fscrypto ecb btrfs
zstd_decompresszstd_compressxxhash raid456async_raid6_recovasync_mem
cpyasync_pqasync_xorasync_txxorraid6_pqlibcrc32ccrc32c_genericraid0 multipath linearhid_genericusbhid hiddm_modraid10 raid1md_modsd_modcrc32c_intelahcii2c_i801lib
ahciaesni_intelxhci_pciaes_x86_64ehci_pcilibatacrypto_simdxhci_hcdehci_hcdcryptdglue_helpere1000escsi_modptp usbcorepps_coreusb_commonfan thermalMay 2 07:15:39 node0 kernel: [1168145.689057] CPU: 6 PID: 3763 Comm: kworker/6:2 Not tainted 4.15.0-0.bpo.2-amd64 #1 Debian 4.15.11-1~bpo9+1 May 2 07:15:39 node0 kernel: [1168145.690347] Hardware name: LENOVO ThinkServer TS140/ThinkServer TS140, BIOS FBKTB3AUS 06/16/2015 May 2 07:15:39 node0 kernel: [1168145.691659] Workqueue: dio/dm-0 dio_aio_complete_work May 2 07:15:39 node0 kernel: [1168145.692935] RIP: 0010:dio_complete+0x1d6/0x220 May 2 07:15:39 node0 kernel: [1168145.694275] RSP: 0018:ffff9abc68447e50 EFLAGS: 00010286 May 2 07:15:39 node0 kernel: [1168145.695605] RAX: 00000000fffffff0 RBX: ffff8e33712e3480 RCX: ffff9abc68447c88 May 2 07:15:39 node0 kernel: [1168145.697024] RDX: fffff1dcc92e4c1f RSI: 0000000000000000 RDI: 0000000000000246 May 2 07:15:39 node0 kernel: [1168145.698389] RBP: 0000000000005000 R08: 0000000000000000 R09: ffffffffb7a075c0 May 2 07:15:39 node0 kernel: [1168145.699703] R10: ffff8e33bb4223c0 R11: 0000000000000195 R12: 0000000000005000 May 2 07:15:39 node0 kernel: [1168145.701044] R13: 0000000000000003 R14: 0000000403060000 R15: ffff8e33712e3500 May 2 07:15:39 node0 kernel: [1168145.702238] FS: 0000000000000000(0000) GS:ffff8e349eb80000(0000) knlGS:0000000000000000 May 2 07:15:39 node0 kernel: [1168145.703475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 2 07:15:39 node0 kernel: [1168145.704733] CR2: 00007ff89915b08e CR3: 00000005b2e0a005 CR4: 00000000001626e0 May 2 07:15:39 node0 kernel: [1168145.705955] Call Trace: May 2 07:15:39 node0 kernel: [1168145.707151] process_one_work+0x177/0x360 May 2 07:15:39 node0 kernel: [1168145.708373] worker_thread+0x4d/0x3c0 May 2 07:15:39 node0 kernel: [1168145.709501] kthread+0xf8/0x130 May 2 07:15:39 node0 kernel: [1168145.710603] ? process_one_work+0x360/0x360 May 2 07:15:39 node0 kernel: [1168145.711701] ? kthread_create_worker_on_cpu+0x70/0x70 May 2 07:15:39 node0 kernel: [1168145.712845] ? SyS_exit_group+0x10/0x10 May 2 07:15:39 node0 kernel: [1168145.713973] ret_from_fork+0x35/0x40 May 2 07:15:39 node0 kernel: [1168145.715072] Code: 8b 78 30 48 83 7f 58 00 0f 84 e5 fe ff ff 49 8d 54 2e ff 4c 89 f6 48 c1 fe 0c 48 c1 fa 0c e8 c2 e0 f3 ff 85 c0 0f 84 c8 fe ff f f <0f> 0b e9 c1 fe ff ff 8b 47 20 a8 10 0f 84 e2 fe ff ff 48 8b 77 May 2 07:15:39 node0 kernel: [1168145.717349] ---[ end trace
Quoted 5 lines
cfa707d6465e13d2]---If someone is interested in investigating then please let me know. The data is not important. The lack of incrementing
read_io_errsis particularly critical IMHO.
This warning is due to mixing buffered/dio. For more info check the commit log of :
332391a9935d ("fs: Fix page cache inconsistency when mixing buffered and AIO DIO")
On 05/23/2018 08:32 AM, Nikolay Borisov wrote:
Nikolay, thank you for the answer.
Quoted 27 lines
Quoted 27 lines
[...]
root@node0:~# dmesg | grep BTRFS | grep warn 185:980:[2927472.393557] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 186:981:[2927472.394158] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 191:986:[2928224.169814] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 192:987:[2928224.171433] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 206:1001:[2928298.039516] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 207:1002:[2928298.043103] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 208:1004:[2932213.513424] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 4564959232 csum 0xc616afb4 expected csum 0x5425e489 mirror 1 209:1005:[2932235.666368] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 16989835264 csum 0xd63ed5da expected csum 0x7429caa1 mirror 1 210:1072:[2936767.229277] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1 211:1073:[2936767.276229] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1
Quoted 8 lines
Quoted 4 lines
Above has been revealed during below command and quite high IO usage by few VMs (Linux on top Ext4 with firebird database, lots of random read/writes, two others with Windows 2016 and Windows Update in the background):
I believe you are hitting the issue described here:
https://www.mail-archive.com/[email protected]/msg25656.html
It make sense, fsck.ext4, gbak - firebird integrity checking tool, chkdsk and sfc /scannow don't show any errors internally within VM. As far I can tell the data inside VMs is not corrupted somehow.
Quoted 5 lines
Essentially the way qemu operates on vm images atop btrfs is prone to producing such errors. As a matter of fact, other filesystems also suffer from this(i.e pages modified while being written, however due to lack of CRC on the data they don't detect it). Can you confirm that those inodes (312/314/319/219962/219915) belong to vm images files?
root@node0:/var/lib/libvirt# find ./ -inum 312 root@node0:/var/lib/libvirt# find ./ -inum 314 root@node0:/var/lib/libvirt# find ./ -inum 319 root@node0:/var/lib/libvirt# find ./ -inum 219962 ./images/rds.raw root@node0:/var/lib/libvirt# find ./ -inum 219915 ./images/database.raw
It seems so (219962, 219915):
- rds.raw - Windows 2016 server, Remote Desktop Server, raw preallocated image, NTFS
database.raw - Linux 3.8, Firebird DB server, raw preallocated image, Ext4
Quoted 3 lines
IMHO the best course of action would be to disable checksumming for you vm files.
Do you mean '-o nodatasum' mount flag? Is it possible to disable checksumming for singe file by setting some magical chattr? Google thinks it's not possible to disable csums for a single file.
Quoted 82 lines
For some background I suggest you read the following LWN articles:
https://lwn.net/Articles/486311/
https://lwn.net/Articles/442355/Quoted 71 lines
when I changed BTRFS compress parameters. Or during umount (can't recall now):
May 2 07:15:39 node0 kernel: [1168145.677431] WARNING: CPU: 6 PID: 3763 at /build/linux-8B5M4n/linux-4.15.11/fs/direct-io.c:293 dio_complete+0x1d6/0x220 May 2 07:15:39 node0 kernel: [1168145.678811] Modules linked in: fuseufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
vhost_netvhost tap tunebtable_filterebtables ip6table_filterip6_tablesiptable_filterbinfmt_miscbridge 8021q garp mrp stp llcsnd_hda_codec_hdmiintel_raplx86_pkg_temp_thermalintel_powerclampcoretempsnd_hda_codec_realtekkvm_intelsnd_hda_codec_generickvm i915 irqbypasscrct10dif_pclmulsnd_hda_intelcrc32_pclmulghash_clmulni_intelsnd_hda_codecintel_cstatesnd_hda_coreiTCO_wdtiTCO_vendor_support
intel_uncoredrm_kms_helpersnd_hwdepwmi_bmofintel_rapl_perfjoydev
evdev pcspkrsnd_pcmsnd_timerdrm snd soundcorei2c_algo_bitsgmei_melpc_ichshpchpmfd_coremeiie31200_e
dac wmi video buttonib_iserrdma_cmiw_cmib_cmib_coreconfigfsiscsi_tcplibiscsi_tcplibiscsiscsi_transport_iscsiip_tablesMay 2 07:15:39 node0 kernel: [1168145.685202] x_tables autofs4 ext4crc16 mbcache jbd2 fscrypto ecb btrfs
zstd_decompresszstd_compressxxhash raid456async_raid6_recovasync_mem
cpyasync_pqasync_xorasync_txxorraid6_pqlibcrc32ccrc32c_genericraid0 multipath linearhid_genericusbhid hiddm_modraid10 raid1md_modsd_modcrc32c_intelahcii2c_i801lib
ahciaesni_intelxhci_pciaes_x86_64ehci_pcilibatacrypto_simdxhci_hcdehci_hcdcryptdglue_helpere1000escsi_modptp usbcorepps_coreusb_commonfan thermalMay 2 07:15:39 node0 kernel: [1168145.689057] CPU: 6 PID: 3763 Comm: kworker/6:2 Not tainted 4.15.0-0.bpo.2-amd64 #1 Debian 4.15.11-1~bpo9+1 May 2 07:15:39 node0 kernel: [1168145.690347] Hardware name: LENOVO ThinkServer TS140/ThinkServer TS140, BIOS FBKTB3AUS 06/16/2015 May 2 07:15:39 node0 kernel: [1168145.691659] Workqueue: dio/dm-0 dio_aio_complete_work May 2 07:15:39 node0 kernel: [1168145.692935] RIP: 0010:dio_complete+0x1d6/0x220 May 2 07:15:39 node0 kernel: [1168145.694275] RSP: 0018:ffff9abc68447e50 EFLAGS: 00010286 May 2 07:15:39 node0 kernel: [1168145.695605] RAX: 00000000fffffff0 RBX: ffff8e33712e3480 RCX: ffff9abc68447c88 May 2 07:15:39 node0 kernel: [1168145.697024] RDX: fffff1dcc92e4c1f RSI: 0000000000000000 RDI: 0000000000000246 May 2 07:15:39 node0 kernel: [1168145.698389] RBP: 0000000000005000 R08: 0000000000000000 R09: ffffffffb7a075c0 May 2 07:15:39 node0 kernel: [1168145.699703] R10: ffff8e33bb4223c0 R11: 0000000000000195 R12: 0000000000005000 May 2 07:15:39 node0 kernel: [1168145.701044] R13: 0000000000000003 R14: 0000000403060000 R15: ffff8e33712e3500 May 2 07:15:39 node0 kernel: [1168145.702238] FS: 0000000000000000(0000) GS:ffff8e349eb80000(0000) knlGS:0000000000000000 May 2 07:15:39 node0 kernel: [1168145.703475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 2 07:15:39 node0 kernel: [1168145.704733] CR2: 00007ff89915b08e CR3: 00000005b2e0a005 CR4: 00000000001626e0 May 2 07:15:39 node0 kernel: [1168145.705955] Call Trace: May 2 07:15:39 node0 kernel: [1168145.707151] process_one_work+0x177/0x360 May 2 07:15:39 node0 kernel: [1168145.708373] worker_thread+0x4d/0x3c0 May 2 07:15:39 node0 kernel: [1168145.709501] kthread+0xf8/0x130 May 2 07:15:39 node0 kernel: [1168145.710603] ? process_one_work+0x360/0x360 May 2 07:15:39 node0 kernel: [1168145.711701] ? kthread_create_worker_on_cpu+0x70/0x70 May 2 07:15:39 node0 kernel: [1168145.712845] ? SyS_exit_group+0x10/0x10 May 2 07:15:39 node0 kernel: [1168145.713973] ret_from_fork+0x35/0x40 May 2 07:15:39 node0 kernel: [1168145.715072] Code: 8b 78 30 48 83 7f 58 00 0f 84 e5 fe ff ff 49 8d 54 2e ff 4c 89 f6 48 c1 fe 0c 48 c1 fa 0c e8 c2 e0 f3 ff 85 c0 0f 84 c8 fe ff f f <0f> 0b e9 c1 fe ff ff 8b 47 20 a8 10 0f 84 e2 fe ff ff 48 8b 77 May 2 07:15:39 node0 kernel: [1168145.717349] ---[ end traceQuoted 5 lines
cfa707d6465e13d2]---If someone is interested in investigating then please let me know. The data is not important. The lack of incrementing
read_io_errsis particularly critical IMHO.
Quoted 5 lines
This warning is due to mixing buffered/dio. For more info check the commit log of :
332391a9935d("fs: Fix page cache inconsistency when mixing buffered and AIO DIO")
Reading the BTRFS code is beyond my understanding. Have you thought about read_io_errs counter?
Balance reveals IO read error, copying VM file ends with IO read error, read_io_errors is unchanged - still shows "0".
Signature · 3 lines
PGP Public Key (RSA/4096b): ID: 0xF2C6EA10 SHA-1: 51DA 40EE 832A 0572 5AD8 B3C0 7AFF 69E1 F2C6 EA10
ein posted on Wed, 23 May 2018 10:03:52 +0200 as excerpted:
Quoted 7 lines
Quoted 4 lines
IMHO the best course of action would be to disable checksumming for you vm files.
Do you mean '-o nodatasum' mount flag? Is it possible to disable checksumming for singe file by setting some magical chattr? Google thinks it's not possible to disable csums for a single file.
You can use nocow (-C), but of course that has other restrictions (like setting it on the files when they're zero-length, easiest done for existing data by setting it on the containing dir and copying files (no reflink) in) as well as the nocow effects, and nocow becomes cow1 after a snapshot (which locks the existing copy in place so changes written to a block /must/ be written elsewhere, thus the cow1, aka cow the first time written after the snapshot but retain the nocow for repeated writes between snapshots).
But if you're disabling checksumming anyway, nocow's likely the way to go.
Signature · 3 lines
Duncan - List replies preferred. No HTML msgs. "Every nonfree program has a lord, a master -- and if you use the program, he is your master." Richard Stallman
Quoted 18 lines · Duncan
ein posted on Wed, 23 May 2018 10:03:52 +0200 as excerpted:
Quoted 6 lines
Quoted 2 lines
IMHO the best course of action would be to disable checksumming for you vm files.
Do you mean '-o nodatasum' mount flag? Is it possible to disable checksumming for singe file by setting some magical chattr? Google thinks it's not possible to disable csums for a single file.
You can use nocow (-C), but of course that has other restrictions (like setting it on the files when they're zero-length, easiest done for existing data by setting it on the containing dir and copying files (no reflink) in) as well as the nocow effects, and nocow becomes cow1 after a snapshot (which locks the existing copy in place so changes written to a block /must/ be written elsewhere, thus the cow1, aka cow the first time written after the snapshot but retain the nocow for repeated writes between snapshots).
Quoted 1 line
But if you're disabling checksumming anyway, nocow's likely the way to go.
Disabling checksumming only may be a way to go - we live without it every day. But nocow @ VM files defeats whole purpose of using BTRFS for me, even with huge performance penalty - backup reasons - I mean few snapshots (20-30), send & receive.
Signature · 3 lines
PGP Public Key (RSA/4096b): ID: 0xF2C6EA10 SHA-1: 51DA 40EE 832A 0572 5AD8 B3C0 7AFF 69E1 F2C6 EA10
Quoted 19 lines · ein
Quoted 18 lines · Duncan
ein posted on Wed, 23 May 2018 10:03:52 +0200 as excerpted:
Quoted 6 lines
Quoted 2 lines
IMHO the best course of action would be to disable checksumming for you vm files.
Do you mean '-o nodatasum' mount flag? Is it possible to disable checksumming for singe file by setting some magical chattr? Google thinks it's not possible to disable csums for a single file.
You can use nocow (-C), but of course that has other restrictions (like setting it on the files when they're zero-length, easiest done for existing data by setting it on the containing dir and copying files (no reflink) in) as well as the nocow effects, and nocow becomes cow1 after a snapshot (which locks the existing copy in place so changes written to a block /must/ be written elsewhere, thus the cow1, aka cow the first time written after the snapshot but retain the nocow for repeated writes between snapshots).
Quoted 7 lines
Quoted 1 line
But if you're disabling checksumming anyway, nocow's likely the way to go.
Disabling checksumming only may be a way to go - we live without it every day. But nocow @ VM files defeats whole purpose of using BTRFS for me, even with huge performance penalty - backup reasons - I mean few snapshots (20-30), send & receive.
Setting NOCOW on a file doesn't prevent it from being snapshotted, it just prevents COW operations from happening under most normal circumstances. In essence, it prevents COW from happening except for writing right after the snapshot. More specifically, the first write to a given block in a file set for NOCOW after taking a snapshot will trigger a _single_ COW operation for _only_ that block (unless you have autodefrag enabled too), after which that block will revert to not doing COW operations on write. This way, you still get consistent and working snapshots, but you also don't take the performance hits from COW except right after taking a snapshot.
Quoted 30 lines · Austin S. Hemmelgarn
Quoted 21 lines · ein
Quoted 20 lines · Duncan
ein posted on Wed, 23 May 2018 10:03:52 +0200 as excerpted:
Quoted 7 lines
Quoted 3 lines
IMHO the best course of action would be to disable checksumming for you
vm files.Do you mean '-o nodatasum' mount flag? Is it possible to disable checksumming for singe file by setting some magical chattr? Google thinks it's not possible to disable csums for a single file.
You can use nocow (-C), but of course that has other restrictions (like setting it on the files when they're zero-length, easiest done for existing data by setting it on the containing dir and copying files (no reflink) in) as well as the nocow effects, and nocow becomes cow1 after a
snapshot (which locks the existing copy in place so changes written to a block /must/ be written elsewhere, thus the cow1, aka cow the first time written after the snapshot but retain the nocow for repeated writes between snapshots).Quoted 8 lines
Quoted 2 lines
But if you're disabling checksumming anyway, nocow's likely the way to go.
Disabling checksumming only may be a way to go - we live without it every day. But nocow @ VM files defeats whole purpose of using BTRFS for me, even with huge performance penalty - backup reasons - I mean few snapshots (20-30), send & receive.
Quoted 10 lines
Setting NOCOW on a file doesn't prevent it from being snapshotted, it just prevents COW operations from happening under most normal circumstances. In essence, it prevents COW from happening except for writing right after the snapshot. More specifically, the first write to a given block in a file set for NOCOW after taking a snapshot will trigger a _single_ COW operation for _only_ that block (unless you have autodefrag enabled too), after which that block will revert to not doing COW operations on write. This way, you still get consistent and working snapshots, but you also don't take the performance hits from COW except right after taking a snapshot.
Yeah, just after I've post it, I've found some Duncan post from 2015, explaining it, thank you anyway.
Is there anything we can do better in random/write VM workload to speed the BTRFS up and why?
My settings:
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/libvirt/images/db.raw'/>
<target dev='vda' bus='virtio'/>
[...]</disk>
/dev/mapper/raid10-images on /var/lib/libvirt type btrfs (rw,noatime,nodiratime,compress=lzo:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/)
md1 : active raid10 sdc1[2] sdb2[1] sdd1[3] sda2[0]
468596736 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
bitmap: 0/4 pages [0KB], 65536KB chunkCPU: E3-1246 with: VT-x, VT-d, HT, EPT, TSX-NI, AES-NI on debian's kernel 4.15.0-0.bpo.2-amd64
As far as I understand compress and autodefrag are impacting negatively for performance (latency), especially autodefrag. I think also that nodatacow shall also speed things up and it's a must when using qemu and BTRFS. Is it better to use virtio or virt-scsi with TRIM support?
Signature · 3 lines
PGP Public Key (RSA/4096b): ID: 0xF2C6EA10 SHA-1: 51DA 40EE 832A 0572 5AD8 B3C0 7AFF 69E1 F2C6 EA10
Quoted 64 lines · ein
Quoted 30 lines · Austin S. Hemmelgarn
Quoted 21 lines · ein
Quoted 20 lines · Duncan
ein posted on Wed, 23 May 2018 10:03:52 +0200 as excerpted:
Quoted 7 lines
Quoted 3 lines
IMHO the best course of action would be to disable checksumming for you
vm files.Do you mean '-o nodatasum' mount flag? Is it possible to disable checksumming for singe file by setting some magical chattr? Google thinks it's not possible to disable csums for a single file.
You can use nocow (-C), but of course that has other restrictions (like setting it on the files when they're zero-length, easiest done for existing data by setting it on the containing dir and copying files (no reflink) in) as well as the nocow effects, and nocow becomes cow1 after a
snapshot (which locks the existing copy in place so changes written to a block /must/ be written elsewhere, thus the cow1, aka cow the first time written after the snapshot but retain the nocow for repeated writes between snapshots).Quoted 8 lines
Quoted 2 lines
But if you're disabling checksumming anyway, nocow's likely the way to go.
Disabling checksumming only may be a way to go - we live without it every day. But nocow @ VM files defeats whole purpose of using BTRFS for me, even with huge performance penalty - backup reasons - I mean few snapshots (20-30), send & receive.
Quoted 10 lines
Setting NOCOW on a file doesn't prevent it from being snapshotted, it just prevents COW operations from happening under most normal circumstances. In essence, it prevents COW from happening except for writing right after the snapshot. More specifically, the first write to a given block in a file set for NOCOW after taking a snapshot will trigger a _single_ COW operation for _only_ that block (unless you have autodefrag enabled too), after which that block will revert to not doing COW operations on write. This way, you still get consistent and working snapshots, but you also don't take the performance hits from COW except right after taking a snapshot.
Yeah, just after I've post it, I've found some Duncan post from 2015, explaining it, thank you anyway.
Is there anything we can do better in random/write VM workload to speed the BTRFS up and why?
My settings:
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/> <source file='/var/lib/libvirt/images/db.raw'/> <target dev='vda' bus='virtio'/> [...]</disk>
/dev/mapper/raid10-imageson/var/lib/libvirttype btrfs (rw,noatime,nodiratime,compress=lzo:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/)md1 : active raid10 sdc1[2] sdb2[1] sdd1[3] sda2[0]
468596736 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU] bitmap: 0/4 pages [0KB], 65536KB chunk
Quoted 8 lines
CPU: E3-1246 with: VT-x, VT-d, HT, EPT, TSX-NI, AES-NI on debian's kernel 4.15.0-0.bpo.2-amd64
As far as I understand compress and autodefrag are impacting negatively for performance (latency), especially autodefrag. I think also that nodatacow shall also speed things up and it's a must when using qemu and BTRFS. Is it better to use virtio or virt-scsi with TRIM support?
FWIW, I've been doing just fine without nodatacow, but I also use raw images contained in sparse files, and keep autodefrag off for the dedicated filesystem I put the images on.
Compression shouldn't have much in the way of negative impact unless you're also using transparent compression (or disk for file encryption) inside the VM (in fact, it may speed things up significantly depending on what filesystem is being used by the guest OS, the ext4 inode table in particular seems to compress very well). If you are using nodatacow though, you can just turn compression off, as it's not going to be used anyway. If you want to keep using compression, then I'd suggest using compress-force instead of compress, which makes BTRFS a bit more aggressive about trying to compress things, but makes the performance much more deterministic. You may also want to look int using zstd instead of lzo for the compression, it gets better ratios most of the time, and usually performs better than lzo does.
Autodefrag should probably be off. If you have nodatacow set (or just have all the files marked with the NOCOW attribute), then there's not really any point to having autodefrag on. If like me you aren't turning off COW for data, it's still a good idea to have it off and just do batch defragmentation at a regularly scheduled time.
For the VM settings, everything looks fine to me (though if you have somewhat slow storage and aren't giving the VM's lots of memory to work with, doing write-through caching might be helpful). I would probably be using virtio-scsi for the TRIM support, as with raw images you will get holes in the file where the TRIM command was issued, which can actually improve performance (and does improve storage utilization (though doing batch trims instead of using the discard mount option is better for performance if you have Linux guests).
You're using an MD RAID10 array. This is generally the fastest option in terms of performance, but it also means you can't take advantage of BTRFS' self repairing ability very well, and you may be wasting space and some performance (because you probably have the 'dup' profile set for metadata). If it's an option, I'd suggest converting this to a BTRFS raid1 volume on top of two MD RAID0 volumes, which should either get the same performance, or slightly better performance, will avoid wasting space storing metadata, and will also let you take advantage of the self-repair functionality in BTRFS.
You should probably switch the ssd mount option to nossd (and then run a full recursive defrag on the volume, as this option affects the allocation policy, so the changes only take effect for new allocations).
The SSD allocator can actually pretty significantly hurt performance
in many cases, and has at best very limited benefits for device lifetimes (you'll maybe get another few months out of a device that will last for ten years without issue). Make a point to test this though, because you're on a RAID array, this may actually be improving performance slightly.
Quoted 68 lines · Austin S. Hemmelgarn
Quoted 67 lines · ein
Quoted 30 lines · Austin S. Hemmelgarn
Quoted 21 lines · ein
Quoted 20 lines · Duncan
ein posted on Wed, 23 May 2018 10:03:52 +0200 as excerpted:
Quoted 7 lines
Quoted 3 lines
IMHO the best course of action would be to disable checksumming for you
vm files.Do you mean '-o nodatasum' mount flag? Is it possible to disable checksumming for singe file by setting some magical chattr? Google thinks it's not possible to disable csums for a single file.
You can use nocow (-C), but of course that has other restrictions (like setting it on the files when they're zero-length, easiest done for existing data by setting it on the containing dir and copying files (no reflink) in) as well as the nocow effects, and nocow becomes cow1 after a
snapshot (which locks the existing copy in place so changes written to a block /must/ be written elsewhere, thus the cow1, aka cow the first time written after the snapshot but retain the nocow for repeated writes between snapshots).Quoted 8 lines
Quoted 2 lines
But if you're disabling checksumming anyway, nocow's likely the way to go.
Disabling checksumming only may be a way to go - we live without it every day. But nocow @ VM files defeats whole purpose of using BTRFS for me, even with huge performance penalty - backup reasons - I mean few snapshots (20-30), send & receive.
Quoted 10 lines
Setting NOCOW on a file doesn't prevent it from being snapshotted, it just prevents COW operations from happening under most normal circumstances. In essence, it prevents COW from happening except for writing right after the snapshot. More specifically, the first write to a given block in a file set for NOCOW after taking a snapshot will trigger a _single_ COW operation for _only_ that block (unless you have autodefrag enabled too), after which that block will revert to not doing COW operations on write. This way, you still get consistent and working snapshots, but you also don't take the performance hits from COW except right after taking a snapshot.
Yeah, just after I've post it, I've found some Duncan post from 2015, explaining it, thank you anyway.
Is there anything we can do better in random/write VM workload to speed the BTRFS up and why?
My settings:
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/> <source file='/var/lib/libvirt/images/db.raw'/> <target dev='vda' bus='virtio'/>
[...]
</disk>
/dev/mapper/raid10-imageson/var/lib/libvirttype btrfs (rw,noatime,nodiratime,compress=lzo:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/)md1 : active raid10 sdc1[2] sdb2[1] sdd1[3] sda2[0] 468596736 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
bitmap: 0/4 pages [0KB], 65536KB chunkCPU: E3-1246 with: VT-x, VT-d, HT, EPT, TSX-NI, AES-NI on debian's kernel 4.15.0-0.bpo.2-amd64
Quoted 7 lines
Quoted 5 lines
As far as I understand compress and autodefrag are impacting negatively for performance (latency), especially autodefrag. I think also that nodatacow shall also speed things up and it's a must when using qemu and BTRFS. Is it better to use virtio or virt-scsi with TRIM support?
FWIW, I've been doing just fine without nodatacow, but I also use raw images contained in sparse files, and keep autodefrag off for the dedicated filesystem I put the images on.
So do I, RAW images created by qemu-img, but I am not sure if preallocation works as expected. The size of disks in filesystem looks fine though.
May I ask in what workloads? From my testing while having VM on BTRFS storage:
- file/web servers works perfect on BTRFS.
- Windows (2012/2016) file servers with AD, are perfect too, besides time required for Windows Update, but this service is... let's say not fine enough.
- database (firebird) impact is huuuge, guest filesystem is Ext4, the database performs slower in this conditions (4 SSDs in RAID10) than when it was on raid1 with 2 10krpm SASes. I am still thinking how to benchmark it properly. A lot of iowait in host's kernel.
Quoted 9 lines
Compression shouldn't have much in the way of negative impact unless you're also using transparent compression (or disk for file encryption) inside the VM (in fact, it may speed things up significantly depending on what filesystem is being used by the guest OS, the ext4 inode table in particular seems to compress very well). If you are using
nodatacowthough, you can just turn compression off, as it's not going to be used anyway. If you want to keep using compression, then I'd suggest usingcompress-forceinstead ofcompress, which makes BTRFS a bit more aggressive about trying to compress things, but makes the performance much more deterministic. You may also want to look int usingzstdinstead oflzofor the compression, it gets better ratios most of the time, and usually performs better thanlzodoes.
Yeah, I do know exact values from the post we both know for sure:
https://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git/commit/?h=next&id=5c1aab1dd5445ed8bdcdbb575abc1b0d7ee5b2e7
Quoted 4 lines
Autodefrag should probably be off. If you have nodatacow set (or just have all the files marked with the NOCOW attribute), then there's not really any point to having autodefrag on. If like me you aren't turning off COW for data, it's still a good idea to have it off and just do batch defragmentation at a regularly scheduled time.
Well, at least I need to try nodatacow to check the impact.
Quoted 6 lines
For the VM settings, everything looks fine to me (though if you have somewhat slow storage and aren't giving the VM's lots of memory to work with, doing write-through caching might be helpful). I would probably be using virtio-scsi for the TRIM support, as with raw images you will get holes in the file where the TRIM command was issued, which can actually improve performance (and does improve storage utilization (though doing batch trims instead of using the
discardmount option is better for performance if you have Linux guests).
I don't consider this... :
4: Model=Samsung SSD 850 PRO 256GB, FwRev=EXM02B6Q, SerialNo=S251NX0H822367V 21: Model=Samsung SSD 850 PRO 256GB, FwRev=EXM02B6Q, SerialNo=S251NX0H822370A 38: Model=INTEL SSDSC2BP240G4, FwRev=L2010420, SerialNo=BTJR6141002D240AGN 55: Model=INTEL SSDSC2BP240G4, FwRev=L2010420, SerialNo=BTJR6063000F240AGN as slow in Raid 10 mode, because:
root@node0:~# time dd if=/dev/md1 of=/dev/null bs=4096 count=10M 10485760+0 records in 10485760+0 records out 42949672960 bytes (43 GB, 40 GiB) copied, 31.6336 s, 1.4 GB/s
real 0m31.636s user 0m1.949s sys 0m12.222s
root@node0:~# iostat -x 5 /dev/md1 /dev/sda /dev/sdb /dev/sdc /dev/sdd
avg-cpu: %user %nice %system %iowait %steal %idle
0.63 0.00 4.85 6.61 0.00 87.91Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdc 306.80 0.00 672.00 0.00 333827.20 0.00 993.53 1.05 1.57 1.57 0.00 1.21 81.20 sdb 329.80 0.00 663.40 0.00 332640.00 0.00 1002.83 0.94 1.41 1.41 0.00 1.05 69.44 sdd 298.80 0.00 664.80 0.00 329110.40 0.00 990.10 1.00 1.50 1.50 0.00 1.22 80.96 sda 291.60 0.00 657.40 0.00 330297.60 0.00 1004.86 0.92 1.40 1.40 0.00 1.05 69.20
md1 0.00 0.00 3884.80 0.00 2693254.40 0.00 1386.56 0.00 0.00
0.00 0.00 0.00 0.00
It gives me much more than 100k IOPs to the BTRFS filesystem if I remember correctly while fio benchmark random workload (75% reads, 25% writes), 2 threads.
Quoted 7 lines
You're using an MD RAID10 array. This is generally the fastest option in terms of performance, but it also means you can't take advantage of BTRFS' self repairing ability very well, and you may be wasting space and some performance (because you probably have the 'dup' profile set for metadata). If it's an option, I'd suggest converting this to a BTRFS raid1 volume on top of two MD RAID0 volumes, which should either get the same performance, or slightly better performance, will avoid wasting space storing metadata, and will also let you take advantage of the self-repair functionality in BTRFS.
That's a good point.
Quoted 6 lines
You should probably switch the
ssdmount option tonossd(and then run a full recursive defrag on the volume, as this option affects the allocation policy, so the changes only take effect for new allocations). The SSD allocator can actually pretty significantly hurt performance in many cases, and has at best very limited benefits for device lifetimes (you'll maybe get another few months out of a device that will last for ten years without issue). Make a point to test this though, because you're on a RAID array, this may actually be improving performance slightly.
Good point either. I am going to test ssd parameter impact to, I think that recreating fs and copy the data may be good idea.
We should not care about the wearout:
20 users working on the database every day (work week), for about a year and:
Model Family: Intel 730 and DC S35x0/3610/3700 Series SSDs Device Model: INTEL SSDSC2BP240G4
233 Media_Wearout_Indicator 0x0032 098 098 000 Old_age Always - 0
and
177 Wear_Leveling_Count 0x0013 095 095 000 Pre-fail Always - 282
Model Family: Samsung based SSDs Device Model: Samsung SSD 850 PRO 256GB
Which means 50 more years @ Samsung Pro 850 and 100 years @ Intel 730, which is interesting. (btw. start date is exactly the same).
Thank you for sharing Austin.
Signature · 3 lines
PGP Public Key (RSA/4096b): ID: 0xF2C6EA10 SHA-1: 51DA 40EE 832A 0572 5AD8 B3C0 7AFF 69E1 F2C6 EA10
Quoted 74 lines · ein
Quoted 73 lines · Austin S. Hemmelgarn
Quoted 72 lines · ein
Quoted 30 lines · Austin S. Hemmelgarn
Quoted 21 lines · ein
Quoted 20 lines · Duncan
ein posted on Wed, 23 May 2018 10:03:52 +0200 as excerpted:
Quoted 7 lines
Quoted 3 lines
IMHO the best course of action would be to disable checksumming for you
vm files.Do you mean '-o nodatasum' mount flag? Is it possible to disable checksumming for singe file by setting some magical chattr? Google thinks it's not possible to disable csums for a single file.
You can use nocow (-C), but of course that has other restrictions (like setting it on the files when they're zero-length, easiest done for existing data by setting it on the containing dir and copying files (no reflink) in) as well as the nocow effects, and nocow becomes cow1 after a
snapshot (which locks the existing copy in place so changes written to a block /must/ be written elsewhere, thus the cow1, aka cow the first time written after the snapshot but retain the nocow for repeated writes between snapshots).Quoted 8 lines
Quoted 2 lines
But if you're disabling checksumming anyway, nocow's likely the way to go.
Disabling checksumming only may be a way to go - we live without it every day. But nocow @ VM files defeats whole purpose of using BTRFS for me, even with huge performance penalty - backup reasons - I mean few snapshots (20-30), send & receive.
Quoted 10 lines
Setting NOCOW on a file doesn't prevent it from being snapshotted, it just prevents COW operations from happening under most normal circumstances. In essence, it prevents COW from happening except for writing right after the snapshot. More specifically, the first write to a given block in a file set for NOCOW after taking a snapshot will trigger a _single_ COW operation for _only_ that block (unless you have autodefrag enabled too), after which that block will revert to not doing COW operations on write. This way, you still get consistent and working snapshots, but you also don't take the performance hits from COW except right after taking a snapshot.
Yeah, just after I've post it, I've found some Duncan post from 2015, explaining it, thank you anyway.
Is there anything we can do better in random/write VM workload to speed the BTRFS up and why?
My settings:
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/> <source file='/var/lib/libvirt/images/db.raw'/> <target dev='vda' bus='virtio'/>
[...]
</disk>
/dev/mapper/raid10-imageson/var/lib/libvirttype btrfs (rw,noatime,nodiratime,compress=lzo:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/)md1 : active raid10 sdc1[2] sdb2[1] sdd1[3] sda2[0]
468596736 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
bitmap: 0/4 pages [0KB], 65536KB chunkCPU: E3-1246 with: VT-x, VT-d, HT, EPT, TSX-NI, AES-NI on debian's kernel 4.15.0-0.bpo.2-amd64
As far as I understand compress and autodefrag are impacting negatively for performance (latency), especially autodefrag. I think also that nodatacow shall also speed things up and it's a must when using qemu and BTRFS. Is it better to use virtio or virt-scsi with TRIM support?
Quoted 5 lines
Quoted 2 lines
FWIW, I've been doing just fine without nodatacow, but I also use raw images contained in sparse files, and keep autodefrag off for the dedicated filesystem I put the images on.
So do I, RAW images created by qemu-img, but I am not sure if preallocation works as expected. The size of disks in filesystem looks fine though.
Unless I'm mistaken, qemu-img will fully pre-allocate the images.
You can easily check though with ls -ls, which will show the amount of space taken up by the file on-disk (before compression or deduplication) on the left. If that first column on the left doesn't match up with the apparent file size, then the file is sparse and not fully pre-allocated.
From a practical perspective, if you really want maximal performance, it's worth pre-allocating space, as that both avoids the non-determinism of allocating blocks on first-write, and avoids some degree of fragmentation.
If you would rather save the space and not pre-allocate, you can either use touch with the --size argument to quickly create an apropriately sized virtual disk image file.
Quoted 8 lines
May I ask in what workloads? From my testing while having VM on BTRFS storage:
- file/web servers works perfect on BTRFS.
- Windows (2012/2016) file servers with AD, are perfect too, besides time required for Windows Update, but this service is... let's say not fine enough.
- database (firebird) impact is huuuge, guest filesystem is Ext4, the database performs slower in this conditions (4 SSDs in RAID10) than when it was on raid1 with 2 10krpm SASes. I am still thinking how to benchmark it properly. A lot of iowait in host's kernel.
In my case, I've got a couple of different types of VM's, each with it's own type of workload:
- A total of 8 static VM's that are always running, each running a different distribution/version of Linux. These see very little activity most of the time (I keep them around as reference systems so i have something I can look at directly when doing development or providing support), use ext4 for the internal filesystems, and are not particularly big to begin with.
- A bunch of transient VM's used for testing kernel patches for BTRFS. These literally start up, run xfstests, copy the results out to a file share on the host, and shut down. The overall behavior for these shouldn't be too drastically different from most database workloads (the internals of BTRFS are very similar to many database systems).
- Less frequently, transient VM's for testing other software (mostly Netdata recently). These have varied workloads depending on what exactly I'm testing, but often don't touch the disk much.
So, overall, I don't have any systems quite comparable to what you're running, but still at least have a reasonable spread of workloads.
Quoted 14 lines
Quoted 10 lines
Compression shouldn't have much in the way of negative impact unless you're also using transparent compression (or disk for file encryption) inside the VM (in fact, it may speed things up significantly depending on what filesystem is being used by the guest OS, the ext4 inode table in particular seems to compress very well). If you are using
nodatacowthough, you can just turn compression off, as it's not going to be used anyway. If you want to keep using compression, then I'd suggest usingcompress-forceinstead ofcompress, which makes BTRFS a bit more aggressive about trying to compress things, but makes the performance much more deterministic. You may also want to look int usingzstdinstead oflzofor the compression, it gets better ratios most of the time, and usually performs better thanlzodoes.Yeah, I do know exact values from the post we both know for sure:
https://git.kernel.org/pub/scm/linux/kernel/git/mason/linux-btrfs.git/commit/?h=next&id=5c1aab1dd5445ed8bdcdbb575abc1b0d7ee5b2e7
Quoted 6 lines
Quoted 4 lines
Autodefrag should probably be off. If you have nodatacow set (or just have all the files marked with the NOCOW attribute), then there's not really any point to having autodefrag on. If like me you aren't turning off COW for data, it's still a good idea to have it off and just do batch defragmentation at a regularly scheduled time.
Well, at least I need to try nodatacow to check the impact.
Provided that the files aren't fragmented, you should see an increase in write performance, but probably not much improvement for reads.
Quoted 38 lines
Quoted 7 lines
For the VM settings, everything looks fine to me (though if you have somewhat slow storage and aren't giving the VM's lots of memory to work with, doing write-through caching might be helpful). I would probably be using virtio-scsi for the TRIM support, as with raw images you will get holes in the file where the TRIM command was issued, which can actually improve performance (and does improve storage utilization (though doing batch trims instead of using the
discardmount option is better for performance if you have Linux guests).I don't consider this... :
4: Model=Samsung SSD 850 PRO 256GB, FwRev=EXM02B6Q, SerialNo=S251NX0H822367V 21: Model=Samsung SSD 850 PRO 256GB, FwRev=EXM02B6Q, SerialNo=S251NX0H822370A 38: Model=INTEL SSDSC2BP240G4, FwRev=L2010420, SerialNo=BTJR6141002D240AGN 55: Model=INTEL SSDSC2BP240G4, FwRev=L2010420, SerialNo=BTJR6063000F240AGN as slow in Raid 10 mode, because:root@node0:~# time dd if=/dev/md1 of=/dev/null bs=4096 count=10M 10485760+0 records in 10485760+0 records out 42949672960 bytes (43 GB, 40 GiB) copied, 31.6336 s, 1.4 GB/sreal 0m31.636s user 0m1.949s sys 0m12.222sroot@node0:~# iostat -x 5 /dev/md1 /dev/sda /dev/sdb /dev/sdc /dev/sdd avg-cpu: %user %nice %system %iowait %steal %idle 0.63 0.00 4.85 6.61 0.00 87.91Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util sdc 306.80 0.00 672.00 0.00 333827.20 0.00 993.53 1.05 1.57 1.57 0.00 1.21 81.20 sdb 329.80 0.00 663.40 0.00 332640.00 0.00 1002.83 0.94 1.41 1.41 0.00 1.05 69.44 sdd 298.80 0.00 664.80 0.00 329110.40 0.00 990.10 1.00 1.50 1.50 0.00 1.22 80.96 sda 291.60 0.00 657.40 0.00 330297.60 0.00 1004.86 0.92 1.40 1.40 0.00 1.05 69.20
Quoted 5 lines
md1 0.00 0.00 3884.80 0.00 2693254.40 0.00 1386.56 0.00 0.00
0.00 0.00 0.00 0.00It gives me much more than 100k IOPs to the BTRFS filesystem if I remember correctly while fio benchmark random workload (75% reads, 25% writes), 2 threads.
Yeah, I wouldn't consider that 'slow' either. In my case, I'm running my VM's with the back-end storage being a BTRFS raid1 volume on top of two LVM thinp targets, which are in turn on top of a pair of consumer 7200RPM SATA3 HDD's (because I ran out of space on the two half-TB SSD's that I had everything in the system on, and happened to have some essentially new 1TB HDD's around still from before I converted to SSD's everywhere), and that I would definitely call slow, and it's probably worth noting that my definition of 'works just fine' is at least partly based on the fact that the storage is so slow.>
Quoted 17 lines
Quoted 7 lines
You're using an MD RAID10 array. This is generally the fastest option in terms of performance, but it also means you can't take advantage of BTRFS' self repairing ability very well, and you may be wasting space and some performance (because you probably have the 'dup' profile set for metadata). If it's an option, I'd suggest converting this to a BTRFS raid1 volume on top of two MD RAID0 volumes, which should either get the same performance, or slightly better performance, will avoid wasting space storing metadata, and will also let you take advantage of the self-repair functionality in BTRFS.
That's a good point.
Quoted 6 lines
You should probably switch the
ssdmount option tonossd(and then run a full recursive defrag on the volume, as this option affects the allocation policy, so the changes only take effect for new allocations). The SSD allocator can actually pretty significantly hurt performance in many cases, and has at best very limited benefits for device lifetimes (you'll maybe get another few months out of a device that will last for ten years without issue). Make a point to test this though, because you're on a RAID array, this may actually be improving performance slightly.
Quoted 2 lines
Good point either. I am going to test ssd parameter impact to, I think that recreating fs and copy the data may be good idea.
One quick point, do make sure you explicitly set 'nossd', as BTRFS tries to set the 'ssd' parameter automatically based on whether or not the underlying storage is rotational (and I don't remember if MD copies the rotational flag from the lower-level storage or not).
Quoted 11 lines
We should not care about the wearout:
20 users working on the database every day (work week), for about a year and:Model Family: Intel 730 and DC S35x0/3610/3700 Series SSDs Device Model: INTEL SSDSC2BP240G4233
Media_Wearout_Indicator0x0032098 098 000Old_ageAlways - 0and
Quoted 7 lines
177
Wear_Leveling_Count0x0013095 095 000 Pre-fail Always - 282Model Family: Samsung based SSDs Device Model: Samsung SSD 850 PRO 256GBWhich means 50 more years @ Samsung Pro 850 and 100 years @ Intel 730, which is interesting. (btw. start date is exactly the same).
For what it's worth, based on my own experience, the degradation isn't exactly linear, it's more of an exponential falloff (as more blocks go bad, there's less extra space for the FTL to work with for wear leveling, so it can't do as good a job wear-leveling, which in turn causes blocks to fail faster). Realistically though, you do still probably have a few decades worth of life in them at minimum.
Quoted 2 lines
Thank you for sharing Austin.
Glad I could help!
Quoted 63 lines · ein
On 05/23/2018 08:32 AM, Nikolay Borisov wrote:
Nikolay, thank you for the answer.
Quoted 27 lines
Quoted 27 lines
[...]
root@node0:~# dmesg | grep BTRFS | grep warn 185:980:[2927472.393557] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 186:981:[2927472.394158] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 191:986:[2928224.169814] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 192:987:[2928224.171433] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 206:1001:[2928298.039516] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 207:1002:[2928298.043103] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 208:1004:[2932213.513424] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 4564959232 csum 0xc616afb4 expected csum 0x5425e489 mirror 1 209:1005:[2932235.666368] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 16989835264 csum 0xd63ed5da expected csum 0x7429caa1 mirror 1 210:1072:[2936767.229277] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1 211:1073:[2936767.276229] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1Quoted 8 lines
Quoted 4 lines
Above has been revealed during below command and quite high IO usage by few VMs (Linux on top Ext4 with firebird database, lots of random read/writes, two others with Windows 2016 and Windows Update in the background):
I believe you are hitting the issue described here:
https://www.mail-archive.com/[email protected]/msg25656.html
It make sense, fsck.ext4, gbak - firebird integrity checking tool, chkdsk and sfc /scannow don't show any errors internally within VM. As far I can tell the data inside VMs is not corrupted somehow.
Quoted 5 lines
Essentially the way qemu operates on vm images atop btrfs is prone to producing such errors. As a matter of fact, other filesystems also suffer from this(i.e pages modified while being written, however due to lack of CRC on the data they don't detect it). Can you confirm that those inodes (312/314/319/219962/219915) belong to vm images files?
root@node0:/var/lib/libvirt# find ./ -inum 312 root@node0:/var/lib/libvirt# find ./ -inum 314 root@node0:/var/lib/libvirt# find ./ -inum 319 root@node0:/var/lib/libvirt# find ./ -inum 219962 ./images/rds.raw root@node0:/var/lib/libvirt# find ./ -inum 219915 ./images/database.rawIt seems so (219962, 219915):
- rds.raw - Windows 2016 server, Remote Desktop Server, raw preallocated image, NTFS
database.raw - Linux 3.8, Firebird DB server, raw preallocated image, Ext4
Quoted 7 lines
Quoted 3 lines
IMHO the best course of action would be to disable checksumming for you vm files.
Do you mean '-o nodatasum' mount flag? Is it possible to disable checksumming for singe file by setting some magical chattr? Google thinks it's not possible to disable csums for a single file.
You can't disable checksumming for a single file. However, what you could do is set a the "No CoW" flag via chattr +c /path/to/file since it also disables checksumming. Bear in mind you can't set this flag to a file which already has allocated blocks. So you'd have to create an empty file, set the +C flag and then copy the data with dd for example.
On a different note - for database workloads and generally random workloads it makes no sense to have CoW since you'd see very spikey io performance.
Quoted 83 lines
Quoted 83 lines
For some background I suggest you read the following LWN articles:
https://lwn.net/Articles/486311/
https://lwn.net/Articles/442355/Quoted 76 lines
when I changed BTRFS compress parameters. Or during umount (can't recall now):
May 2 07:15:39 node0 kernel: [1168145.677431] WARNING: CPU: 6 PID: 3763 at /build/linux-8B5M4n/linux-4.15.11/fs/direct-io.c:293 dio_complete+0x1d6/0x220 May 2 07:15:39 node0 kernel: [1168145.678811] Modules linked in: fuseufs qnx4 hfsplus hfs minix ntfs vfat msdos fat jfs xfs
vhost_netvhost tap tunebtable_filterebtables ip6table_filterip6_tablesiptable_filterbinfmt_miscbridge 8021q garp mrp stp llcsnd_hda_codec_hdmiintel_raplx86_pkg_temp_thermalintel_powerclampcoretempsnd_hda_codec_realtekkvm_intelsnd_hda_codec_generickvm i915 irqbypasscrct10dif_pclmulsnd_hda_intelcrc32_pclmulghash_clmulni_intelsnd_hda_codecintel_cstatesnd_hda_coreiTCO_wdtiTCO_vendor_support
intel_uncoredrm_kms_helpersnd_hwdepwmi_bmofintel_rapl_perfjoydev
evdev pcspkrsnd_pcmsnd_timerdrm snd soundcorei2c_algo_bitsgmei_melpc_ichshpchpmfd_coremeiie31200_e
dac wmi video buttonib_iserrdma_cmiw_cmib_cmib_coreconfigfsiscsi_tcplibiscsi_tcplibiscsiscsi_transport_iscsiip_tablesMay 2 07:15:39 node0 kernel: [1168145.685202] x_tables autofs4 ext4crc16 mbcache jbd2 fscrypto ecb btrfs
zstd_decompresszstd_compressxxhash raid456async_raid6_recovasync_mem
cpyasync_pqasync_xorasync_txxorraid6_pqlibcrc32ccrc32c_genericraid0 multipath linearhid_genericusbhid hiddm_modraid10 raid1md_modsd_modcrc32c_intelahcii2c_i801lib
ahciaesni_intelxhci_pciaes_x86_64ehci_pcilibatacrypto_simdxhci_hcdehci_hcdcryptdglue_helpere1000escsi_modptp usbcorepps_coreusb_commonfan thermalMay 2 07:15:39 node0 kernel: [1168145.689057] CPU: 6 PID: 3763 Comm: kworker/6:2 Not tainted 4.15.0-0.bpo.2-amd64 #1 Debian 4.15.11-1~bpo9+1 May 2 07:15:39 node0 kernel: [1168145.690347] Hardware name: LENOVO ThinkServer TS140/ThinkServer TS140, BIOS FBKTB3AUS 06/16/2015 May 2 07:15:39 node0 kernel: [1168145.691659] Workqueue: dio/dm-0 dio_aio_complete_work May 2 07:15:39 node0 kernel: [1168145.692935] RIP: 0010:dio_complete+0x1d6/0x220 May 2 07:15:39 node0 kernel: [1168145.694275] RSP: 0018:ffff9abc68447e50 EFLAGS: 00010286 May 2 07:15:39 node0 kernel: [1168145.695605] RAX: 00000000fffffff0 RBX: ffff8e33712e3480 RCX: ffff9abc68447c88 May 2 07:15:39 node0 kernel: [1168145.697024] RDX: fffff1dcc92e4c1f RSI: 0000000000000000 RDI: 0000000000000246 May 2 07:15:39 node0 kernel: [1168145.698389] RBP: 0000000000005000 R08: 0000000000000000 R09: ffffffffb7a075c0 May 2 07:15:39 node0 kernel: [1168145.699703] R10: ffff8e33bb4223c0 R11: 0000000000000195 R12: 0000000000005000 May 2 07:15:39 node0 kernel: [1168145.701044] R13: 0000000000000003 R14: 0000000403060000 R15: ffff8e33712e3500 May 2 07:15:39 node0 kernel: [1168145.702238] FS: 0000000000000000(0000) GS:ffff8e349eb80000(0000) knlGS:0000000000000000 May 2 07:15:39 node0 kernel: [1168145.703475] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 May 2 07:15:39 node0 kernel: [1168145.704733] CR2: 00007ff89915b08e CR3: 00000005b2e0a005 CR4: 00000000001626e0 May 2 07:15:39 node0 kernel: [1168145.705955] Call Trace: May 2 07:15:39 node0 kernel: [1168145.707151] process_one_work+0x177/0x360 May 2 07:15:39 node0 kernel: [1168145.708373] worker_thread+0x4d/0x3c0 May 2 07:15:39 node0 kernel: [1168145.709501] kthread+0xf8/0x130 May 2 07:15:39 node0 kernel: [1168145.710603] ? process_one_work+0x360/0x360 May 2 07:15:39 node0 kernel: [1168145.711701] ? kthread_create_worker_on_cpu+0x70/0x70 May 2 07:15:39 node0 kernel: [1168145.712845] ? SyS_exit_group+0x10/0x10 May 2 07:15:39 node0 kernel: [1168145.713973] ret_from_fork+0x35/0x40 May 2 07:15:39 node0 kernel: [1168145.715072] Code: 8b 78 30 48 83 7f 58 00 0f 84 e5 fe ff ff 49 8d 54 2e ff 4c 89 f6 48 c1 fe 0c 48 c1 fa 0c e8 c2 e0 f3 ff 85 c0 0f 84 c8 fe ff f f <0f> 0b e9 c1 fe ff ff 8b 47 20 a8 10 0f 84 e2 fe ff ff 48 8b 77 May 2 07:15:39 node0 kernel: [1168145.717349] ---[ end trace
cfa707d6465e13d2]---If someone is interested in investigating then please let me know. The data is not important. The lack of incrementing
read_io_errsis particularly critical IMHO.
Quoted 8 lines
Quoted 5 lines
This warning is due to mixing buffered/dio. For more info check the commit log of :
332391a9935d("fs: Fix page cache inconsistency when mixing buffered and AIO DIO")Reading the BTRFS code is beyond my understanding. Have you thought about
read_io_errscounter?
I didn't say read the btrfs code but rather read the commit messages.
Quoted 3 lines
Balance reveals IO read error, copying VM file ends with IO read error,
read_io_errorsis unchanged - still shows "0".
Will have to investigate and see whether the current behavior is intentional or not.
Quoted 2 lines
23.05.2018 09:32, Nikolay Borisov пишет:
Quoted 53 lines · ein
Quoted 35 lines
Hello devs,
I tested BTRFS in production for about a month:
21:08:17 up 34 days, 2:21, 3 users, load average: 0.06, 0.02, 0.00
Without power blackout, hardware failure, SSD's SMART is flawless etc. The tests ended with:
root@node0:~# dmesg | grep BTRFS | grep warn 185:980:[2927472.393557] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 186:981:[2927472.394158] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 191:986:[2928224.169814] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 192:987:[2928224.171433] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 206:1001:[2928298.039516] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 207:1002:[2928298.043103] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 208:1004:[2932213.513424] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 4564959232 csum 0xc616afb4 expected csum 0x5425e489 mirror 1 209:1005:[2932235.666368] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 16989835264 csum 0xd63ed5da expected csum 0x7429caa1 mirror 1 210:1072:[2936767.229277] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1 211:1073:[2936767.276229] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1Quoted 4 lines
Above has been revealed during below command and quite high IO usage by few VMs (Linux on top Ext4 with firebird database, lots of random read/writes, two others with Windows 2016 and Windows Update in the background):
I believe you are hitting the issue described here:
https://www.mail-archive.com/[email protected]/msg25656.html
Essentially the way qemu operates on vm images atop btrfs is prone to producing such errors. As a matter of fact, other filesystems also suffer from this(i.e pages modified while being written, however due to lack of CRC on the data they don't detect it). Can you confirm that those inodes (312/314/319/219962/219915) belong to vm images files?
Quoted 9 lines
IMHO the best course of action would be to disable checksumming for you vm files.
For some background I suggest you read the following LWN articles:
https://lwn.net/Articles/486311/
https://lwn.net/Articles/442355/
Hmm ... according to these articles, "pages under writeback are marked as not being writable; any process attempting to write to such a page will block until the writeback completes". And it says this feature is available since 3.0 and btrfs has it. So how comes it still happens? Were stable patches removed since then?
Quoted 64 lines · Andrei Borzenkov
23.05.2018 09:32, Nikolay Borisov пишет:
Quoted 53 lines · ein
Quoted 35 lines
Hello devs,
I tested BTRFS in production for about a month:
21:08:17 up 34 days, 2:21, 3 users, load average: 0.06, 0.02, 0.00
Without power blackout, hardware failure, SSD's SMART is flawless etc. The tests ended with:
root@node0:~# dmesg | grep BTRFS | grep warn 185:980:[2927472.393557] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 186:981:[2927472.394158] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 191:986:[2928224.169814] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 192:987:[2928224.171433] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 206:1001:[2928298.039516] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 207:1002:[2928298.043103] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 208:1004:[2932213.513424] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 4564959232 csum 0xc616afb4 expected csum 0x5425e489 mirror 1 209:1005:[2932235.666368] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 16989835264 csum 0xd63ed5da expected csum 0x7429caa1 mirror 1 210:1072:[2936767.229277] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1 211:1073:[2936767.276229] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1Quoted 4 lines
Above has been revealed during below command and quite high IO usage by few VMs (Linux on top Ext4 with firebird database, lots of random read/writes, two others with Windows 2016 and Windows Update in the background):
I believe you are hitting the issue described here:
https://www.mail-archive.com/[email protected]/msg25656.html
Essentially the way qemu operates on vm images atop btrfs is prone to producing such errors. As a matter of fact, other filesystems also suffer from this(i.e pages modified while being written, however due to lack of CRC on the data they don't detect it). Can you confirm that those inodes (312/314/319/219962/219915) belong to vm images files?
Quoted 9 lines
IMHO the best course of action would be to disable checksumming for you vm files.
For some background I suggest you read the following LWN articles:
https://lwn.net/Articles/486311/
https://lwn.net/Articles/442355/
Quoted 5 lines
Hmm ... according to these articles, "pages under writeback are marked as not being writable; any process attempting to write to such a page will block until the writeback completes". And it says this feature is available since 3.0 and btrfs has it. So how comes it still happens? Were stable patches removed since then?
If you are using buffered writes, then yes you won't have the problem. However qemu by default bypasses host's page cache and instead uses DIO:
https://btrfs.wiki.kernel.org/index.php/Gotchas#Direct_IO_and_CRCs
Quoted 1 line
Quoted 73 lines · Nikolay Borisov
Quoted 64 lines · Andrei Borzenkov
23.05.2018 09:32, Nikolay Borisov пишет:
Quoted 53 lines · ein
Quoted 35 lines
Hello devs,
I tested BTRFS in production for about a month:
21:08:17 up 34 days, 2:21, 3 users, load average: 0.06, 0.02, 0.00
Without power blackout, hardware failure, SSD's SMART is flawless etc. The tests ended with:
root@node0:~# dmesg | grep BTRFS | grep warn 185:980:[2927472.393557] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 186:981:[2927472.394158] BTRFS warning (device dm-0): csum failed root -9 ino 312 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 191:986:[2928224.169814] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 192:987:[2928224.171433] BTRFS warning (device dm-0): csum failed root -9 ino 314 off 608284672 csum 0x7da1b152 expected csum 0x3163a9b7 mirror 1 206:1001:[2928298.039516] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 207:1002:[2928298.043103] BTRFS warning (device dm-0): csum failed root -9 ino 319 off 608284672 csum 0x7d03a376 expected csum 0x3163a9b7 mirror 1 208:1004:[2932213.513424] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 4564959232 csum 0xc616afb4 expected csum 0x5425e489 mirror 1 209:1005:[2932235.666368] BTRFS warning (device dm-0): csum failed root 5 ino 219962 off 16989835264 csum 0xd63ed5da expected csum 0x7429caa1 mirror 1 210:1072:[2936767.229277] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1 211:1073:[2936767.276229] BTRFS warning (device dm-0): csum failed root 5 ino 219915 off 82318458880 csum 0x83614341 expected csum 0x0b8706f8 mirror 1Quoted 4 lines
Above has been revealed during below command and quite high IO usage by few VMs (Linux on top Ext4 with firebird database, lots of random read/writes, two others with Windows 2016 and Windows Update in the background):
I believe you are hitting the issue described here:
https://www.mail-archive.com/[email protected]/msg25656.html
Essentially the way qemu operates on vm images atop btrfs is prone to producing such errors. As a matter of fact, other filesystems also suffer from this(i.e pages modified while being written, however due to lack of CRC on the data they don't detect it). Can you confirm that those inodes (312/314/319/219962/219915) belong to vm images files?
Quoted 9 lines
IMHO the best course of action would be to disable checksumming for you vm files.
For some background I suggest you read the following LWN articles:
https://lwn.net/Articles/486311/
https://lwn.net/Articles/442355/Quoted 5 lines
Hmm ... according to these articles, "pages under writeback are marked as not being writable; any process attempting to write to such a page will block until the writeback completes". And it says this feature is available since 3.0 and btrfs has it. So how comes it still happens? Were stable patches removed since then?
Quoted 4 lines
If you are using buffered writes, then yes you won't have the problem. However qemu by default bypasses host's page cache and instead uses DIO:
https://btrfs.wiki.kernel.org/index.php/Gotchas#Direct_IO_and_CRCs
I can confirm that writing data to the filesystem on guest side is not buffered at host with config:
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source file='/var/lib/libvirt/images/db.raw'/>
<target dev='vda' bus='virtio'/>
[...]</disk>
Because buff/cache memory usage stays unchanged at host during high sequential writing and there's no kworker/flush process committing the data. How qemu can avoid dirty page buffering? There's nothing else than:ppoll, read, io_sumbit and write in strace:
read(52, "\1\0\0\0\0\0\0\0", 512) = 8
io_submit(0x7f35367f7000, 2, [{pwritev, fildes=19, iovec=[{iov_base="\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=368640}, {iov_base="\0\0\0\0\0\0 \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=368640}, {iov_base="\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=679
\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=368640}, {iov_base="\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=679936}, {iov_bas
\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=368640}, {iov_base="\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=679936}, {iov_base="\0\0\0\0\0\
1048576}, {iov_base="\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=1048576}, {iov_base="\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\read(38, "\3\0\0\0\0\0\0\0", 512) = 8
ppoll([{fd=52, events=POLLIN|POLLERR|POLLHUP}, {fd=38, events=POLLIN|POLLERR|POLLHUP}, {fd=10, events=POLLIN|POLLERR|POLLHUP}], 3, NULL, NULL, 8) = 1 ([{fd=52, revents=POLLIN}])read(52, "\1\0\0\0\0\0\0\0", 512) = 8
io_submit(0x7f35367f7000, 1, [{pwritev, fildes=19, iovec=[{iov_base="\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=368640}, {iov_base="\0\0\0\0\0\0
\0\0\0\0\0\0\0\0\0\0"..., iov_len=1048576}, {iov_base="\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., iov_len=368640}, {iov_base="\0\0\0\0\0\0\0\0\0\0\0\0\0
ppoll([{fd=52, events=POLLIN|POLLERR|POLLHUP}, {fd=38, events=POLLIN|POLLERR|POLLHUP}, {fd=10, events=POLLIN|POLLERR|POLLHUP}], 3, {tv_sec=0, tv_nsec=0}, NULL, 8) = 2 ([{fd=52, re
Signature · 3 lines
PGP Public Key (RSA/4096b): ID: 0xF2C6EA10 SHA-1: 51DA 40EE 832A 0572 5AD8 B3C0 7AFF 69E1 F2C6 EA10