Skip to contentSkip to conversation

compression silently skipped with O_DIRECT & checksum/corruption issue with qemu live migration ?

DISCUSSION Roland 3 participants 6 messages Mar 15, 2024 – Mar 19, 2024 Btrfs file system

lore
Share & tools
lore

Roland

author
#0 lore raw

Hello,

can someone explain why compression is skipped when writing with direct-i/o ?

is this to be expected ?

i wondered why i got uncompressed raw disk images in proxmox after disk migration to btrfs fs with compress-force=zstd, so i tried with dd if i can reproduce - and i can.

the problem is, that i cannot seem to disable direct I/O for different proxmox gui actions, will discuss in proxmox community what can be done, but i need more infos on this behaviour.

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd
if=disk.raw of=test.dat bs=1024k
20480+0 Datensätze ein
20480+0 Datensätze aus
21474836480 Bytes (21 GB, 20 GiB) kopiert, 46,411 s, 463 MB/s

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 161844 regular extents (161844 refs), 0 inline, 85412 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced TOTAL        7%      1.5G          20G          20G
none       100%      339M         339M         339M
zstd         6%      1.2G          19G          19G

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd
if=disk.raw of=test.dat oflag=direct bs=1024k
20480+0 Datensätze ein
20480+0 Datensätze aus
21474836480 Bytes (21 GB, 20 GiB) kopiert, 161,319 s, 133 MB/s

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 20480 regular extents (20480 refs), 0 inline, 1035 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced TOTAL      100%       20G          20G          20G
none       100%       20G          20G          20G

furthermore,  with qemu/proxmox virtual disk live migration, i'm getting reproducible checksum error with btrfs/compress-force=zstd when live migrating virtual disk within same harddisk , whereas copying the same disk/file with dd (cached and direct) does not trigger that.

happens on hdd and also on hdd, so this seems not to be disk related

# uname -a
Linux pve-test1 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1
(2024-02-05T13:50Z) x86_64 GNU/Linux

regards
Roland

[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0x5517115a expected csum 0x6b2af31e mirror 1 [Fr Mär 15 20:07:23 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0xd1d1c892 expected csum 0x0c24b5cd mirror 1 [Fr Mär 15 20:08:24 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0x809096c2 expected csum 0x56b58dc6 mirror 1 [Fr Mär 15 20:09:25 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0xc8a6977d expected csum 0xdfc1f678 mirror 1 [Fr Mär 15 20:29:10 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10

[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): csum failed root 260 ino 257 off 6655332352 csum 0x20ff0887 expected csum 0x08ed0bce mirror 1 [Fr Mär 15 20:33:41 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): direct IO failed ino 257 op 0x0 offset 0x18cb02000 len 16384 err no 10
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): csum failed root 256 ino 290 off 6655332352 csum 0xb023863a expected csum 0x4f8e355d mirror 1 [Fr Mär 15 20:35:18 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): direct IO failed ino 290 op 0x0 offset 0x18cb05000 len 4096 err no 10

create full clone of drive scsi0 (btrfs-ssd-zstd-qcow2:106/vm-106-disk-0.raw)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 34.2 MiB of 20.0 GiB (0.17%) in 0s
drive-scsi0: transferred 301.6 MiB of 20.0 GiB (1.47%) in 1s
drive-scsi0: transferred 489.1 MiB of 20.0 GiB (2.39%) in 2s
drive-scsi0: transferred 679.9 MiB of 20.0 GiB (3.32%) in 3s
drive-scsi0: transferred 1.1 GiB of 20.0 GiB (5.70%) in 4s
drive-scsi0: transferred 1.4 GiB of 20.0 GiB (6.89%) in 5s
drive-scsi0: transferred 1.7 GiB of 20.0 GiB (8.55%) in 6s
drive-scsi0: transferred 1.9 GiB of 20.0 GiB (9.53%) in 7s
drive-scsi0: transferred 2.2 GiB of 20.0 GiB (10.97%) in 8s
drive-scsi0: transferred 2.4 GiB of 20.0 GiB (11.84%) in 9s
drive-scsi0: transferred 2.8 GiB of 20.0 GiB (13.89%) in 10s
drive-scsi0: transferred 2.9 GiB of 20.0 GiB (14.74%) in 11s
drive-scsi0: transferred 6.2 GiB of 20.0 GiB (31.14%) in 12s
drive-scsi0: transferred 8.2 GiB of 20.0 GiB (41.17%) in 13s
drive-scsi0: transferred 10.3 GiB of 20.0 GiB (51.27%) in 14s
drive-scsi0: transferred 10.5 GiB of 20.0 GiB (52.75%) in 15s
drive-scsi0: transferred 19.2 GiB of 20.0 GiB (95.86%) in 16s
drive-scsi0: transferred 19.3 GiB of 20.0 GiB (96.66%) in 17s
drive-scsi0: transferred 19.5 GiB of 20.0 GiB (97.46%) in 18s
drive-scsi0: transferred 19.7 GiB of 20.0 GiB (98.27%) in 19s
drive-scsi0: transferred 19.8 GiB of 20.0 GiB (99.08%) in 20s
drive-scsi0: Cancelling block job
drive-scsi0: Done.
TASK ERROR: storage migration failed: block job (mirror) error:
drive-scsi0: 'mirror' has been cancelled

author
#1 lore raw
Quoted 1 line

happens on hdd and also on hdd, so this seems not to be disk related

sorry, i meant hdd + ssd , and it happens with compress + compress-force

# cat /etc/fstab |grep btrfs
UUID=2fb42259-2c95-4250-b49c-5f3067f2f991 /btrfs-ssd-zstd btrfs
compress=zstd,defaults 0 1
UUID=2578a21c-0ba1-408a-b2cc-b54653a65b33 /btrfs-hdd-zstd btrfs
compress-force=zstd,defaults 0 2

Am 15.03.24 um 20:39 schrieb Roland:

Quoted 133 lines

Hello,

can someone explain why compression is skipped when writing with direct-i/o ?

is this to be expected ?

i wondered why i got uncompressed raw disk images in proxmox after disk migration to btrfs fs with compress-force=zstd, so i tried with dd if i can reproduce - and i can.

the problem is, that i cannot seem to disable direct I/O for different proxmox gui actions, will discuss in proxmox community what can be done, but i need more infos on this behaviour.

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd
if=disk.raw of=test.dat bs=1024k
20480+0 Datensätze ein
20480+0 Datensätze aus
21474836480 Bytes (21 GB, 20 GiB) kopiert, 46,411 s, 463 MB/s

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 161844 regular extents (161844 refs), 0 inline, 85412 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced TOTAL        7%      1.5G          20G          20G none       100%      339M         339M         339M zstd         6%      1.2G          19G          19G

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd
if=disk.raw of=test.dat oflag=direct bs=1024k
20480+0 Datensätze ein
20480+0 Datensätze aus
21474836480 Bytes (21 GB, 20 GiB) kopiert, 161,319 s, 133 MB/s

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 20480 regular extents (20480 refs), 0 inline, 1035 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced TOTAL      100%       20G          20G          20G
none       100%       20G          20G          20G

furthermore,  with qemu/proxmox virtual disk live migration, i'm getting reproducible checksum error with btrfs/compress-force=zstd when live migrating virtual disk within same harddisk , whereas copying the same disk/file with dd (cached and direct) does not trigger that.

happens on hdd and also on hdd, so this seems not to be disk related

# uname -a
Linux pve-test1 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1
(2024-02-05T13:50Z) x86_64 GNU/Linux

regards
Roland

[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0x5517115a expected csum 0x6b2af31e mirror 1
[Fr Mär 15 20:07:23 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10 [Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0xd1d1c892 expected csum 0x0c24b5cd mirror 1
[Fr Mär 15 20:08:24 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10 [Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0x809096c2 expected csum 0x56b58dc6 mirror 1
[Fr Mär 15 20:09:25 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10 [Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0xc8a6977d expected csum 0xdfc1f678 mirror 1
[Fr Mär 15 20:29:10 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10

[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): csum failed root 260 ino 257 off 6655332352 csum 0x20ff0887 expected csum 0x08ed0bce mirror 1
[Fr Mär 15 20:33:41 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): direct IO failed ino 257 op 0x0 offset 0x18cb02000 len 16384 err no 10 [Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): csum failed root 256 ino 290 off 6655332352 csum 0xb023863a expected csum 0x4f8e355d mirror 1
[Fr Mär 15 20:35:18 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): direct IO failed ino 290 op 0x0 offset 0x18cb05000 len 4096 err no 10

create full clone of drive scsi0 (btrfs-ssd-zstd-qcow2:106/vm-106-disk-0.raw)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 34.2 MiB of 20.0 GiB (0.17%) in 0s
drive-scsi0: transferred 301.6 MiB of 20.0 GiB (1.47%) in 1s
drive-scsi0: transferred 489.1 MiB of 20.0 GiB (2.39%) in 2s
drive-scsi0: transferred 679.9 MiB of 20.0 GiB (3.32%) in 3s
drive-scsi0: transferred 1.1 GiB of 20.0 GiB (5.70%) in 4s
drive-scsi0: transferred 1.4 GiB of 20.0 GiB (6.89%) in 5s
drive-scsi0: transferred 1.7 GiB of 20.0 GiB (8.55%) in 6s
drive-scsi0: transferred 1.9 GiB of 20.0 GiB (9.53%) in 7s
drive-scsi0: transferred 2.2 GiB of 20.0 GiB (10.97%) in 8s
drive-scsi0: transferred 2.4 GiB of 20.0 GiB (11.84%) in 9s
drive-scsi0: transferred 2.8 GiB of 20.0 GiB (13.89%) in 10s
drive-scsi0: transferred 2.9 GiB of 20.0 GiB (14.74%) in 11s
drive-scsi0: transferred 6.2 GiB of 20.0 GiB (31.14%) in 12s
drive-scsi0: transferred 8.2 GiB of 20.0 GiB (41.17%) in 13s
drive-scsi0: transferred 10.3 GiB of 20.0 GiB (51.27%) in 14s
drive-scsi0: transferred 10.5 GiB of 20.0 GiB (52.75%) in 15s
drive-scsi0: transferred 19.2 GiB of 20.0 GiB (95.86%) in 16s
drive-scsi0: transferred 19.3 GiB of 20.0 GiB (96.66%) in 17s
drive-scsi0: transferred 19.5 GiB of 20.0 GiB (97.46%) in 18s
drive-scsi0: transferred 19.7 GiB of 20.0 GiB (98.27%) in 19s
drive-scsi0: transferred 19.8 GiB of 20.0 GiB (99.08%) in 20s
drive-scsi0: Cancelling block job
drive-scsi0: Done.
TASK ERROR: storage migration failed: block job (mirror) error:
drive-scsi0: 'mirror' has been cancelled

#2 lore raw
In reply to Roland
Quoted 4 lines · Roland

Hello,

can someone explain why compression is skipped when writing with direct-i/o ?

Because compression only happen with buffered write.

For direct IO we either go NOCOW or regular COW, no compression support.

The current compression code is always using page cache, just check the different algo implementations of btrfs_compress_pages(), which all fetch their data from page cache.

E.g. in zstd_compress_pages(), we go find_get_page() to grab the page from page cache as compression source.

This is incompatible with the direct IO scheme, which is designed to avoid page cache completely (unless fall back to buffered write).

Maybe it's possible to make all those *_compress_pages() to support direct IO, but that doesn't looks sane to me.

As the idea of direct IO is to fully avoid page cache, and allow the user space program to take full control of its own cache, doing compression would introduce extra latency and make the performance characteristic much complex to user space.

Thanks,
Qu

Quoted 129 lines

is this to be expected ?

i wondered why i got uncompressed raw disk images in proxmox after disk migration to btrfs fs with compress-force=zstd, so i tried with dd if i can reproduce - and i can.

the problem is, that i cannot seem to disable direct I/O for different proxmox gui actions, will discuss in proxmox community what can be done, but i need more infos on this behaviour.

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd
if=disk.raw of=test.dat bs=1024k
20480+0 Datensätze ein
20480+0 Datensätze aus
21474836480 Bytes (21 GB, 20 GiB) kopiert, 46,411 s, 463 MB/s

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 161844 regular extents (161844 refs), 0 inline, 85412 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced TOTAL        7%      1.5G          20G          20G
none       100%      339M         339M         339M
zstd         6%      1.2G          19G          19G

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd
if=disk.raw of=test.dat oflag=direct bs=1024k
20480+0 Datensätze ein
20480+0 Datensätze aus
21474836480 Bytes (21 GB, 20 GiB) kopiert, 161,319 s, 133 MB/s

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 20480 regular extents (20480 refs), 0 inline, 1035 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced TOTAL      100%       20G          20G          20G
none       100%       20G          20G          20G

furthermore,  with qemu/proxmox virtual disk live migration, i'm getting reproducible checksum error with btrfs/compress-force=zstd when live migrating virtual disk within same harddisk , whereas copying the same disk/file with dd (cached and direct) does not trigger that.

happens on hdd and also on hdd, so this seems not to be disk related

# uname -a
Linux pve-test1 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1
(2024-02-05T13:50Z) x86_64 GNU/Linux

regards
Roland

[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0x5517115a expected csum 0x6b2af31e mirror 1
[Fr Mär 15 20:07:23 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0xd1d1c892 expected csum 0x0c24b5cd mirror 1
[Fr Mär 15 20:08:24 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0x809096c2 expected csum 0x56b58dc6 mirror 1
[Fr Mär 15 20:09:25 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0xc8a6977d expected csum 0xdfc1f678 mirror 1
[Fr Mär 15 20:29:10 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10

[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): csum failed root 260 ino 257 off 6655332352 csum 0x20ff0887 expected csum 0x08ed0bce mirror 1
[Fr Mär 15 20:33:41 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): direct IO failed ino 257 op 0x0 offset 0x18cb02000 len 16384 err no 10
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): csum failed root 256 ino 290 off 6655332352 csum 0xb023863a expected csum 0x4f8e355d mirror 1
[Fr Mär 15 20:35:18 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): direct IO failed ino 290 op 0x0 offset 0x18cb05000 len 4096 err no 10

create full clone of drive scsi0 (btrfs-ssd-zstd-qcow2:106/vm-106-disk-0.raw)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 34.2 MiB of 20.0 GiB (0.17%) in 0s
drive-scsi0: transferred 301.6 MiB of 20.0 GiB (1.47%) in 1s
drive-scsi0: transferred 489.1 MiB of 20.0 GiB (2.39%) in 2s
drive-scsi0: transferred 679.9 MiB of 20.0 GiB (3.32%) in 3s
drive-scsi0: transferred 1.1 GiB of 20.0 GiB (5.70%) in 4s
drive-scsi0: transferred 1.4 GiB of 20.0 GiB (6.89%) in 5s
drive-scsi0: transferred 1.7 GiB of 20.0 GiB (8.55%) in 6s
drive-scsi0: transferred 1.9 GiB of 20.0 GiB (9.53%) in 7s
drive-scsi0: transferred 2.2 GiB of 20.0 GiB (10.97%) in 8s
drive-scsi0: transferred 2.4 GiB of 20.0 GiB (11.84%) in 9s
drive-scsi0: transferred 2.8 GiB of 20.0 GiB (13.89%) in 10s
drive-scsi0: transferred 2.9 GiB of 20.0 GiB (14.74%) in 11s
drive-scsi0: transferred 6.2 GiB of 20.0 GiB (31.14%) in 12s
drive-scsi0: transferred 8.2 GiB of 20.0 GiB (41.17%) in 13s
drive-scsi0: transferred 10.3 GiB of 20.0 GiB (51.27%) in 14s
drive-scsi0: transferred 10.5 GiB of 20.0 GiB (52.75%) in 15s
drive-scsi0: transferred 19.2 GiB of 20.0 GiB (95.86%) in 16s
drive-scsi0: transferred 19.3 GiB of 20.0 GiB (96.66%) in 17s
drive-scsi0: transferred 19.5 GiB of 20.0 GiB (97.46%) in 18s
drive-scsi0: transferred 19.7 GiB of 20.0 GiB (98.27%) in 19s
drive-scsi0: transferred 19.8 GiB of 20.0 GiB (99.08%) in 20s
drive-scsi0: Cancelling block job
drive-scsi0: Done.
TASK ERROR: storage migration failed: block job (mirror) error:
drive-scsi0: 'mirror' has been cancelled

#3 lore raw
Quoted 12 lines · Qu Wenruo
Quoted 4 lines · Roland

Hello,

can someone explain why compression is skipped when writing with direct-i/o ?

Because compression only happen with buffered write.

For direct IO we either go NOCOW or regular COW, no compression support.

Should we mention this (no compression in O_DIRECT) in doc? I haven't impression that doc said this.

Quoted 151 lines

The current compression code is always using page cache, just check the different algo implementations of btrfs_compress_pages(), which all fetch their data from page cache.

E.g. in zstd_compress_pages(), we go find_get_page() to grab the page from page cache as compression source.

This is incompatible with the direct IO scheme, which is designed to avoid page cache completely (unless fall back to buffered write).

Maybe it's possible to make all those *_compress_pages() to support direct IO, but that doesn't looks sane to me.

As the idea of direct IO is to fully avoid page cache, and allow the user space program to take full control of its own cache, doing compression would introduce extra latency and make the performance characteristic much complex to user space.

Thanks,
Qu

Quoted 129 lines

is this to be expected ?

i wondered why i got uncompressed raw disk images in proxmox after disk migration to btrfs fs with compress-force=zstd, so i tried with dd if i can reproduce - and i can.

the problem is, that i cannot seem to disable direct I/O for different proxmox gui actions, will discuss in proxmox community what can be done, but i need more infos on this behaviour.

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd
if=disk.raw of=test.dat bs=1024k
20480+0 Datensätze ein
20480+0 Datensätze aus
21474836480 Bytes (21 GB, 20 GiB) kopiert, 46,411 s, 463 MB/s

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 161844 regular extents (161844 refs), 0 inline, 85412 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced TOTAL        7%      1.5G          20G          20G
none       100%      339M         339M         339M
zstd         6%      1.2G          19G          19G

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd
if=disk.raw of=test.dat oflag=direct bs=1024k
20480+0 Datensätze ein
20480+0 Datensätze aus
21474836480 Bytes (21 GB, 20 GiB) kopiert, 161,319 s, 133 MB/s

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 20480 regular extents (20480 refs), 0 inline, 1035 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced TOTAL      100%       20G          20G          20G
none       100%       20G          20G          20G

furthermore,  with qemu/proxmox virtual disk live migration, i'm getting reproducible checksum error with btrfs/compress-force=zstd when live migrating virtual disk within same harddisk , whereas copying the same disk/file with dd (cached and direct) does not trigger that.

happens on hdd and also on hdd, so this seems not to be disk related

# uname -a
Linux pve-test1 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1
(2024-02-05T13:50Z) x86_64 GNU/Linux

regards
Roland

[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0x5517115a expected csum 0x6b2af31e mirror 1
[Fr Mär 15 20:07:23 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0xd1d1c892 expected csum 0x0c24b5cd mirror 1
[Fr Mär 15 20:08:24 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0x809096c2 expected csum 0x56b58dc6 mirror 1
[Fr Mär 15 20:09:25 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0xc8a6977d expected csum 0xdfc1f678 mirror 1
[Fr Mär 15 20:29:10 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10

[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): csum failed root 260 ino 257 off 6655332352 csum 0x20ff0887 expected csum 0x08ed0bce mirror 1
[Fr Mär 15 20:33:41 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): direct IO failed ino 257 op 0x0 offset 0x18cb02000 len 16384 err no 10
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): csum failed root 256 ino 290 off 6655332352 csum 0xb023863a expected csum 0x4f8e355d mirror 1
[Fr Mär 15 20:35:18 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): direct IO failed ino 290 op 0x0 offset 0x18cb05000 len 4096 err no 10

create full clone of drive scsi0 (btrfs-ssd-zstd-qcow2:106/vm-106-disk-0.raw)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 34.2 MiB of 20.0 GiB (0.17%) in 0s
drive-scsi0: transferred 301.6 MiB of 20.0 GiB (1.47%) in 1s
drive-scsi0: transferred 489.1 MiB of 20.0 GiB (2.39%) in 2s
drive-scsi0: transferred 679.9 MiB of 20.0 GiB (3.32%) in 3s
drive-scsi0: transferred 1.1 GiB of 20.0 GiB (5.70%) in 4s
drive-scsi0: transferred 1.4 GiB of 20.0 GiB (6.89%) in 5s
drive-scsi0: transferred 1.7 GiB of 20.0 GiB (8.55%) in 6s
drive-scsi0: transferred 1.9 GiB of 20.0 GiB (9.53%) in 7s
drive-scsi0: transferred 2.2 GiB of 20.0 GiB (10.97%) in 8s
drive-scsi0: transferred 2.4 GiB of 20.0 GiB (11.84%) in 9s
drive-scsi0: transferred 2.8 GiB of 20.0 GiB (13.89%) in 10s
drive-scsi0: transferred 2.9 GiB of 20.0 GiB (14.74%) in 11s
drive-scsi0: transferred 6.2 GiB of 20.0 GiB (31.14%) in 12s
drive-scsi0: transferred 8.2 GiB of 20.0 GiB (41.17%) in 13s
drive-scsi0: transferred 10.3 GiB of 20.0 GiB (51.27%) in 14s
drive-scsi0: transferred 10.5 GiB of 20.0 GiB (52.75%) in 15s
drive-scsi0: transferred 19.2 GiB of 20.0 GiB (95.86%) in 16s
drive-scsi0: transferred 19.3 GiB of 20.0 GiB (96.66%) in 17s
drive-scsi0: transferred 19.5 GiB of 20.0 GiB (97.46%) in 18s
drive-scsi0: transferred 19.7 GiB of 20.0 GiB (98.27%) in 19s
drive-scsi0: transferred 19.8 GiB of 20.0 GiB (99.08%) in 20s
drive-scsi0: Cancelling block job
drive-scsi0: Done.
TASK ERROR: storage migration failed: block job (mirror) error:
drive-scsi0: 'mirror' has been cancelled

#4 lore raw
Quoted 10 lines · HAN Yuwei
Quoted 8 lines · Qu Wenruo
Quoted 4 lines · Roland

Hello,

can someone explain why compression is skipped when writing with direct-i/o ?

Quoted 6 lines
Quoted 4 lines

Because compression only happen with buffered write.

For direct IO we either go NOCOW or regular COW, no compression support.

Should we mention this (no compression in O_DIRECT) in doc? I haven't impression that doc said this.

I'd say yes, since there is already some confusion, feel free to submit a patch for the doc.

Thanks,
Qu

Quoted 151 lines
Quoted 151 lines

The current compression code is always using page cache, just check the different algo implementations of btrfs_compress_pages(), which all fetch their data from page cache.

E.g. in zstd_compress_pages(), we go find_get_page() to grab the page from page cache as compression source.

This is incompatible with the direct IO scheme, which is designed to avoid page cache completely (unless fall back to buffered write).

Maybe it's possible to make all those *_compress_pages() to support direct IO, but that doesn't looks sane to me.

As the idea of direct IO is to fully avoid page cache, and allow the user space program to take full control of its own cache, doing compression would introduce extra latency and make the performance characteristic much complex to user space.

Thanks,
Qu

Quoted 129 lines

is this to be expected ?

i wondered why i got uncompressed raw disk images in proxmox after disk migration to btrfs fs with compress-force=zstd, so i tried with dd if i can reproduce - and i can.

the problem is, that i cannot seem to disable direct I/O for different proxmox gui actions, will discuss in proxmox community what can be done, but i need more infos on this behaviour.

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd
if=disk.raw of=test.dat bs=1024k
20480+0 Datensätze ein
20480+0 Datensätze aus
21474836480 Bytes (21 GB, 20 GiB) kopiert, 46,411 s, 463 MB/s

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 161844 regular extents (161844 refs), 0 inline, 85412 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced TOTAL        7%      1.5G          20G          20G
none       100%      339M         339M         339M
zstd         6%      1.2G          19G          19G

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd
if=disk.raw of=test.dat oflag=direct bs=1024k
20480+0 Datensätze ein
20480+0 Datensätze aus
21474836480 Bytes (21 GB, 20 GiB) kopiert, 161,319 s, 133 MB/s

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 20480 regular extents (20480 refs), 0 inline, 1035 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced TOTAL      100%       20G          20G          20G
none       100%       20G          20G          20G

furthermore,  with qemu/proxmox virtual disk live migration, i'm getting reproducible checksum error with btrfs/compress-force=zstd when live migrating virtual disk within same harddisk , whereas copying the same disk/file with dd (cached and direct) does not trigger that.

happens on hdd and also on hdd, so this seems not to be disk related

# uname -a
Linux pve-test1 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1
(2024-02-05T13:50Z) x86_64 GNU/Linux

regards
Roland

[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0x5517115a expected csum 0x6b2af31e mirror 1
[Fr Mär 15 20:07:23 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0xd1d1c892 expected csum 0x0c24b5cd mirror 1
[Fr Mär 15 20:08:24 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0x809096c2 expected csum 0x56b58dc6 mirror 1
[Fr Mär 15 20:09:25 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0xc8a6977d expected csum 0xdfc1f678 mirror 1
[Fr Mär 15 20:29:10 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10

[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): csum failed root 260 ino 257 off 6655332352 csum 0x20ff0887 expected csum 0x08ed0bce mirror 1
[Fr Mär 15 20:33:41 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): direct IO failed ino 257 op 0x0 offset 0x18cb02000 len 16384 err no 10
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): csum failed root 256 ino 290 off 6655332352 csum 0xb023863a expected csum 0x4f8e355d mirror 1
[Fr Mär 15 20:35:18 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): direct IO failed ino 290 op 0x0 offset 0x18cb05000 len 4096 err no 10

create full clone of drive scsi0 (btrfs-ssd-zstd-qcow2:106/vm-106-disk-0.raw)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 34.2 MiB of 20.0 GiB (0.17%) in 0s
drive-scsi0: transferred 301.6 MiB of 20.0 GiB (1.47%) in 1s
drive-scsi0: transferred 489.1 MiB of 20.0 GiB (2.39%) in 2s
drive-scsi0: transferred 679.9 MiB of 20.0 GiB (3.32%) in 3s
drive-scsi0: transferred 1.1 GiB of 20.0 GiB (5.70%) in 4s
drive-scsi0: transferred 1.4 GiB of 20.0 GiB (6.89%) in 5s
drive-scsi0: transferred 1.7 GiB of 20.0 GiB (8.55%) in 6s
drive-scsi0: transferred 1.9 GiB of 20.0 GiB (9.53%) in 7s
drive-scsi0: transferred 2.2 GiB of 20.0 GiB (10.97%) in 8s
drive-scsi0: transferred 2.4 GiB of 20.0 GiB (11.84%) in 9s
drive-scsi0: transferred 2.8 GiB of 20.0 GiB (13.89%) in 10s
drive-scsi0: transferred 2.9 GiB of 20.0 GiB (14.74%) in 11s
drive-scsi0: transferred 6.2 GiB of 20.0 GiB (31.14%) in 12s
drive-scsi0: transferred 8.2 GiB of 20.0 GiB (41.17%) in 13s
drive-scsi0: transferred 10.3 GiB of 20.0 GiB (51.27%) in 14s
drive-scsi0: transferred 10.5 GiB of 20.0 GiB (52.75%) in 15s
drive-scsi0: transferred 19.2 GiB of 20.0 GiB (95.86%) in 16s
drive-scsi0: transferred 19.3 GiB of 20.0 GiB (96.66%) in 17s
drive-scsi0: transferred 19.5 GiB of 20.0 GiB (97.46%) in 18s
drive-scsi0: transferred 19.7 GiB of 20.0 GiB (98.27%) in 19s
drive-scsi0: transferred 19.8 GiB of 20.0 GiB (99.08%) in 20s
drive-scsi0: Cancelling block job
drive-scsi0: Done.
TASK ERROR: storage migration failed: block job (mirror) error:
drive-scsi0: 'mirror' has been cancelled

author
#5 lore raw

thanks for all your feedback

Am 19.03.24 um 05:18 schrieb Qu Wenruo:

Quoted 195 lines · HAN Yuwei
Quoted 10 lines · Qu Wenruo
Quoted 8 lines · Roland
Quoted 4 lines

Hello,

can someone explain why compression is skipped when writing with direct-i/o ?

Quoted 7 lines
Quoted 5 lines

Because compression only happen with buffered write.

For direct IO we either go NOCOW or regular COW, no compression support.

Should we mention this (no compression in O_DIRECT) in doc? I haven't impression that doc said this.

I'd say yes, since there is already some confusion, feel free to submit a patch for the doc.

Thanks,
Qu

Quoted 168 lines
Quoted 168 lines

The current compression code is always using page cache, just check the different algo implementations of btrfs_compress_pages(), which all fetch their data from page cache.

E.g. in zstd_compress_pages(), we go find_get_page() to grab the page from page cache as compression source.

This is incompatible with the direct IO scheme, which is designed to avoid page cache completely (unless fall back to buffered write).

Maybe it's possible to make all those *_compress_pages() to support direct IO, but that doesn't looks sane to me.

As the idea of direct IO is to fully avoid page cache, and allow the user space program to take full control of its own cache, doing compression would introduce extra latency and make the performance characteristic much complex to user space.

Thanks,
Qu

Quoted 146 lines

is this to be expected ?

i wondered why i got uncompressed raw disk images in proxmox after disk
migration to btrfs fs with compress-force=zstd, so i tried with dd if i
can reproduce - and i can.

the problem is, that i cannot seem to disable direct I/O for different proxmox gui actions, will discuss in proxmox community what can be done,
but i need more infos on this behaviour.

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd
if=disk.raw of=test.dat bs=1024k
20480+0 Datensätze ein
20480+0 Datensätze aus
21474836480 Bytes (21 GB, 20 GiB) kopiert, 46,411 s, 463 MB/s

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 161844 regular extents (161844 refs), 0 inline, 85412
fragments.
Type       Perc     Disk Usage   Uncompressed Referenced TOTAL        7%      1.5G          20G          20G none       100%      339M         339M         339M zstd         6%      1.2G          19G          19G

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd
if=disk.raw of=test.dat oflag=direct bs=1024k
20480+0 Datensätze ein
20480+0 Datensätze aus
21474836480 Bytes (21 GB, 20 GiB) kopiert, 161,319 s, 133 MB/s

root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 20480 regular extents (20480 refs), 0 inline, 1035 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced TOTAL      100%       20G          20G          20G
none       100%       20G          20G          20G

furthermore,  with qemu/proxmox virtual disk live migration, i'm getting
reproducible checksum error with btrfs/compress-force=zstd when live migrating virtual disk within same harddisk , whereas copying the same disk/file with dd (cached and direct) does not trigger that.

happens on hdd and also on hdd, so this seems not to be disk related

# uname -a
Linux pve-test1 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1
(2024-02-05T13:50Z) x86_64 GNU/Linux

regards
Roland

[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): csum failed root
260 ino 257 off 6655332352 csum 0x5517115a expected csum 0x6b2af31e mirror 1
[Fr Mär 15 20:07:23 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): direct IO failed
ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): csum failed root
260 ino 257 off 6655332352 csum 0xd1d1c892 expected csum 0x0c24b5cd mirror 1
[Fr Mär 15 20:08:24 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): direct IO failed
ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): csum failed root
260 ino 257 off 6655332352 csum 0x809096c2 expected csum 0x56b58dc6 mirror 1
[Fr Mär 15 20:09:25 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): direct IO failed
ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): csum failed root
260 ino 257 off 6655332352 csum 0xc8a6977d expected csum 0xdfc1f678 mirror 1
[Fr Mär 15 20:29:10 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): direct IO failed
ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10

[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): csum failed root
260 ino 257 off 6655332352 csum 0x20ff0887 expected csum 0x08ed0bce mirror 1
[Fr Mär 15 20:33:41 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): direct IO failed
ino 257 op 0x0 offset 0x18cb02000 len 16384 err no 10 [Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): csum failed root
256 ino 290 off 6655332352 csum 0xb023863a expected csum 0x4f8e355d mirror 1
[Fr Mär 15 20:35:18 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): direct IO failed
ino 290 op 0x0 offset 0x18cb05000 len 4096 err no 10

create full clone of drive scsi0 (btrfs-ssd-zstd-qcow2:106/vm-106-disk-0.raw)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 34.2 MiB of 20.0 GiB (0.17%) in 0s
drive-scsi0: transferred 301.6 MiB of 20.0 GiB (1.47%) in 1s
drive-scsi0: transferred 489.1 MiB of 20.0 GiB (2.39%) in 2s
drive-scsi0: transferred 679.9 MiB of 20.0 GiB (3.32%) in 3s
drive-scsi0: transferred 1.1 GiB of 20.0 GiB (5.70%) in 4s
drive-scsi0: transferred 1.4 GiB of 20.0 GiB (6.89%) in 5s
drive-scsi0: transferred 1.7 GiB of 20.0 GiB (8.55%) in 6s
drive-scsi0: transferred 1.9 GiB of 20.0 GiB (9.53%) in 7s
drive-scsi0: transferred 2.2 GiB of 20.0 GiB (10.97%) in 8s
drive-scsi0: transferred 2.4 GiB of 20.0 GiB (11.84%) in 9s
drive-scsi0: transferred 2.8 GiB of 20.0 GiB (13.89%) in 10s
drive-scsi0: transferred 2.9 GiB of 20.0 GiB (14.74%) in 11s
drive-scsi0: transferred 6.2 GiB of 20.0 GiB (31.14%) in 12s
drive-scsi0: transferred 8.2 GiB of 20.0 GiB (41.17%) in 13s
drive-scsi0: transferred 10.3 GiB of 20.0 GiB (51.27%) in 14s
drive-scsi0: transferred 10.5 GiB of 20.0 GiB (52.75%) in 15s
drive-scsi0: transferred 19.2 GiB of 20.0 GiB (95.86%) in 16s
drive-scsi0: transferred 19.3 GiB of 20.0 GiB (96.66%) in 17s
drive-scsi0: transferred 19.5 GiB of 20.0 GiB (97.46%) in 18s
drive-scsi0: transferred 19.7 GiB of 20.0 GiB (98.27%) in 19s
drive-scsi0: transferred 19.8 GiB of 20.0 GiB (99.08%) in 20s
drive-scsi0: Cancelling block job
drive-scsi0: Done.
TASK ERROR: storage migration failed: block job (mirror) error:
drive-scsi0: 'mirror' has been cancelled