compression silently skipped with O_DIRECT & checksum/corruption issue with qemu live migration ?
Hello,
can someone explain why compression is skipped when writing with direct-i/o ?
is this to be expected ?
i wondered why i got uncompressed raw disk images in proxmox after disk migration to btrfs fs with compress-force=zstd, so i tried with dd if i can reproduce - and i can.
the problem is, that i cannot seem to disable direct I/O for different proxmox gui actions, will discuss in proxmox community what can be done, but i need more infos on this behaviour.
root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd if=disk.raw of=test.dat bs=1024k 20480+0 Datensätze ein 20480+0 Datensätze aus 21474836480 Bytes (21 GB, 20 GiB) kopiert, 46,411 s, 463 MB/s
root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 161844 regular extents (161844 refs), 0 inline, 85412 fragments.
Type Perc Disk Usage Uncompressed Referenced TOTAL 7% 1.5G 20G 20G
none 100% 339M 339M 339M
zstd 6% 1.2G 19G 19G
root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd if=disk.raw of=test.dat oflag=direct bs=1024k 20480+0 Datensätze ein 20480+0 Datensätze aus 21474836480 Bytes (21 GB, 20 GiB) kopiert, 161,319 s, 133 MB/s
root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 20480 regular extents (20480 refs), 0 inline, 1035 fragments.
Type Perc Disk Usage Uncompressed Referenced TOTAL 100% 20G 20G 20G
none 100% 20G 20G 20G
furthermore, with qemu/proxmox virtual disk live migration, i'm getting reproducible checksum error with btrfs/compress-force=zstd when live migrating virtual disk within same harddisk , whereas copying the same disk/file with dd (cached and direct) does not trigger that.
happens on hdd and also on hdd, so this seems not to be disk related
# uname -a Linux pve-test1 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1 (2024-02-05T13:50Z) x86_64 GNU/Linux
regards
Roland
[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0x5517115a expected csum 0x6b2af31e mirror 1 [Fr Mär 15 20:07:23 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0xd1d1c892 expected csum 0x0c24b5cd mirror 1 [Fr Mär 15 20:08:24 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0x809096c2 expected csum 0x56b58dc6 mirror 1 [Fr Mär 15 20:09:25 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum 0xc8a6977d expected csum 0xdfc1f678 mirror 1 [Fr Mär 15 20:29:10 2024] BTRFS error (device sda1): bdev /dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): direct IO failed ino 257 op 0x0 offset 0x18cb05000 len 4096 err no 10
[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): csum failed root 260 ino 257 off 6655332352 csum 0x20ff0887 expected csum 0x08ed0bce mirror 1 [Fr Mär 15 20:33:41 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): direct IO failed ino 257 op 0x0 offset 0x18cb02000 len 16384 err no 10
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): csum failed root 256 ino 290 off 6655332352 csum 0xb023863a expected csum 0x4f8e355d mirror 1 [Fr Mär 15 20:35:18 2024] BTRFS error (device sdc1): bdev /dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): direct IO failed ino 290 op 0x0 offset 0x18cb05000 len 4096 err no 10
create full clone of drive scsi0 (btrfs-ssd-zstd-qcow2:106/vm-106-disk-0.raw)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 34.2 MiB of 20.0 GiB (0.17%) in 0s
drive-scsi0: transferred 301.6 MiB of 20.0 GiB (1.47%) in 1s
drive-scsi0: transferred 489.1 MiB of 20.0 GiB (2.39%) in 2s
drive-scsi0: transferred 679.9 MiB of 20.0 GiB (3.32%) in 3s
drive-scsi0: transferred 1.1 GiB of 20.0 GiB (5.70%) in 4s
drive-scsi0: transferred 1.4 GiB of 20.0 GiB (6.89%) in 5s
drive-scsi0: transferred 1.7 GiB of 20.0 GiB (8.55%) in 6s
drive-scsi0: transferred 1.9 GiB of 20.0 GiB (9.53%) in 7s
drive-scsi0: transferred 2.2 GiB of 20.0 GiB (10.97%) in 8s
drive-scsi0: transferred 2.4 GiB of 20.0 GiB (11.84%) in 9s
drive-scsi0: transferred 2.8 GiB of 20.0 GiB (13.89%) in 10s
drive-scsi0: transferred 2.9 GiB of 20.0 GiB (14.74%) in 11s
drive-scsi0: transferred 6.2 GiB of 20.0 GiB (31.14%) in 12s
drive-scsi0: transferred 8.2 GiB of 20.0 GiB (41.17%) in 13s
drive-scsi0: transferred 10.3 GiB of 20.0 GiB (51.27%) in 14s
drive-scsi0: transferred 10.5 GiB of 20.0 GiB (52.75%) in 15s
drive-scsi0: transferred 19.2 GiB of 20.0 GiB (95.86%) in 16s
drive-scsi0: transferred 19.3 GiB of 20.0 GiB (96.66%) in 17s
drive-scsi0: transferred 19.5 GiB of 20.0 GiB (97.46%) in 18s
drive-scsi0: transferred 19.7 GiB of 20.0 GiB (98.27%) in 19s
drive-scsi0: transferred 19.8 GiB of 20.0 GiB (99.08%) in 20s
drive-scsi0: Cancelling block job
drive-scsi0: Done.
TASK ERROR: storage migration failed: block job (mirror) error:
drive-scsi0: 'mirror' has been cancelled
Quoted 1 line
happens on hdd and also on hdd, so this seems not to be disk related
sorry, i meant hdd + ssd , and it happens with compress + compress-force
# cat /etc/fstab |grep btrfs UUID=2fb42259-2c95-4250-b49c-5f3067f2f991 /btrfs-ssd-zstd btrfs compress=zstd,defaults 0 1 UUID=2578a21c-0ba1-408a-b2cc-b54653a65b33 /btrfs-hdd-zstd btrfs compress-force=zstd,defaults 0 2
Am 15.03.24 um 20:39 schrieb Roland:
Quoted 133 lines
Hello,
can someone explain why compression is skipped when writing with direct-i/o ?
is this to be expected ?
i wondered why i got uncompressed raw disk images in proxmox after disk migration to btrfs fs with compress-force=zstd, so i tried with dd if i can reproduce - and i can.
the problem is, that i cannot seem to disable direct I/O for different proxmox gui actions, will discuss in proxmox community what can be done, but i need more infos on this behaviour.
root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd if=disk.raw of=test.dat bs=1024k 20480+0 Datensätze ein 20480+0 Datensätze aus 21474836480 Bytes (21 GB, 20 GiB) kopiert, 46,411 s, 463 MB/sroot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 161844 regular extents (161844 refs), 0 inline, 85412 fragments.
Type Perc Disk Usage Uncompressed Referenced TOTAL 7% 1.5G 20G 20G none 100% 339M 339M 339M zstd 6% 1.2G 19G 19Groot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd if=disk.raw of=test.dat oflag=direct bs=1024k 20480+0 Datensätze ein 20480+0 Datensätze aus 21474836480 Bytes (21 GB, 20 GiB) kopiert, 161,319 s, 133 MB/sroot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 20480 regular extents (20480 refs), 0 inline, 1035 fragments.
Type Perc Disk Usage Uncompressed Referenced TOTAL 100% 20G 20G 20G
none 100% 20G 20G 20Gfurthermore, with qemu/proxmox virtual disk live migration, i'm getting reproducible checksum error with btrfs/compress-force=zstd when live migrating virtual disk within same harddisk , whereas copying the same disk/file with dd (cached and direct) does not trigger that.
happens on hdd and also on hdd, so this seems not to be disk related
# uname -a Linux pve-test1 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1 (2024-02-05T13:50Z) x86_64 GNU/Linuxregards
Roland[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum
0x5517115aexpected csum0x6b2af31emirror 1
[Fr Mär 15 20:07:23 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10 [Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum0xd1d1c892expected csum0x0c24b5cdmirror 1
[Fr Mär 15 20:08:24 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10 [Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum0x809096c2expected csum0x56b58dc6mirror 1
[Fr Mär 15 20:09:25 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10 [Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum0xc8a6977dexpected csum0xdfc1f678mirror 1
[Fr Mär 15 20:29:10 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): csum failed root 260 ino 257 off 6655332352 csum
0x20ff0887expected csum0x08ed0bcemirror 1
[Fr Mär 15 20:33:41 2024] BTRFS error (device sdc1): bdev/dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): direct IO failed ino 257 op0x0offset0x18cb02000len 16384 err no 10 [Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): csum failed root 256 ino 290 off 6655332352 csum0xb023863aexpected csum0x4f8e355dmirror 1
[Fr Mär 15 20:35:18 2024] BTRFS error (device sdc1): bdev/dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): direct IO failed ino 290 op0x0offset0x18cb05000len 4096 err no 10create full clone of drive scsi0 (btrfs-ssd-zstd-qcow2:106/vm-106-disk-0.raw)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 34.2 MiB of 20.0 GiB (0.17%) in 0s
drive-scsi0: transferred 301.6 MiB of 20.0 GiB (1.47%) in 1s
drive-scsi0: transferred 489.1 MiB of 20.0 GiB (2.39%) in 2s
drive-scsi0: transferred 679.9 MiB of 20.0 GiB (3.32%) in 3s
drive-scsi0: transferred 1.1 GiB of 20.0 GiB (5.70%) in 4s
drive-scsi0: transferred 1.4 GiB of 20.0 GiB (6.89%) in 5s
drive-scsi0: transferred 1.7 GiB of 20.0 GiB (8.55%) in 6s
drive-scsi0: transferred 1.9 GiB of 20.0 GiB (9.53%) in 7s
drive-scsi0: transferred 2.2 GiB of 20.0 GiB (10.97%) in 8s
drive-scsi0: transferred 2.4 GiB of 20.0 GiB (11.84%) in 9s
drive-scsi0: transferred 2.8 GiB of 20.0 GiB (13.89%) in 10s
drive-scsi0: transferred 2.9 GiB of 20.0 GiB (14.74%) in 11s
drive-scsi0: transferred 6.2 GiB of 20.0 GiB (31.14%) in 12s
drive-scsi0: transferred 8.2 GiB of 20.0 GiB (41.17%) in 13s
drive-scsi0: transferred 10.3 GiB of 20.0 GiB (51.27%) in 14s
drive-scsi0: transferred 10.5 GiB of 20.0 GiB (52.75%) in 15s
drive-scsi0: transferred 19.2 GiB of 20.0 GiB (95.86%) in 16s
drive-scsi0: transferred 19.3 GiB of 20.0 GiB (96.66%) in 17s
drive-scsi0: transferred 19.5 GiB of 20.0 GiB (97.46%) in 18s
drive-scsi0: transferred 19.7 GiB of 20.0 GiB (98.27%) in 19s
drive-scsi0: transferred 19.8 GiB of 20.0 GiB (99.08%) in 20s
drive-scsi0: Cancelling block job
drive-scsi0: Done.
TASK ERROR: storage migration failed: block job (mirror) error:
drive-scsi0: 'mirror' has been cancelled
Quoted 4 lines · Roland
Hello,
can someone explain why compression is skipped when writing with direct-i/o ?
Because compression only happen with buffered write.
For direct IO we either go NOCOW or regular COW, no compression support.
The current compression code is always using page cache, just check the different algo implementations of btrfs_compress_pages(), which all fetch their data from page cache.
E.g. in zstd_compress_pages(), we go find_get_page() to grab the page from page cache as compression source.
This is incompatible with the direct IO scheme, which is designed to avoid page cache completely (unless fall back to buffered write).
Maybe it's possible to make all those *_compress_pages() to support direct IO, but that doesn't looks sane to me.
As the idea of direct IO is to fully avoid page cache, and allow the user space program to take full control of its own cache, doing compression would introduce extra latency and make the performance characteristic much complex to user space.
Thanks,
Qu
Quoted 129 lines
is this to be expected ?
i wondered why i got uncompressed raw disk images in proxmox after disk migration to btrfs fs with compress-force=zstd, so i tried with dd if i can reproduce - and i can.
the problem is, that i cannot seem to disable direct I/O for different proxmox gui actions, will discuss in proxmox community what can be done, but i need more infos on this behaviour.
root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd if=disk.raw of=test.dat bs=1024k 20480+0 Datensätze ein 20480+0 Datensätze aus 21474836480 Bytes (21 GB, 20 GiB) kopiert, 46,411 s, 463 MB/sroot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 161844 regular extents (161844 refs), 0 inline, 85412 fragments.
Type Perc Disk Usage Uncompressed Referenced TOTAL 7% 1.5G 20G 20G
none 100% 339M 339M 339M
zstd 6% 1.2G 19G 19Groot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd if=disk.raw of=test.dat oflag=direct bs=1024k 20480+0 Datensätze ein 20480+0 Datensätze aus 21474836480 Bytes (21 GB, 20 GiB) kopiert, 161,319 s, 133 MB/sroot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 20480 regular extents (20480 refs), 0 inline, 1035 fragments.
Type Perc Disk Usage Uncompressed Referenced TOTAL 100% 20G 20G 20G
none 100% 20G 20G 20Gfurthermore, with qemu/proxmox virtual disk live migration, i'm getting reproducible checksum error with btrfs/compress-force=zstd when live migrating virtual disk within same harddisk , whereas copying the same disk/file with dd (cached and direct) does not trigger that.
happens on hdd and also on hdd, so this seems not to be disk related
# uname -a Linux pve-test1 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1 (2024-02-05T13:50Z) x86_64 GNU/Linuxregards
Roland[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum
0x5517115aexpected csum0x6b2af31emirror 1
[Fr Mär 15 20:07:23 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum0xd1d1c892expected csum0x0c24b5cdmirror 1
[Fr Mär 15 20:08:24 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum0x809096c2expected csum0x56b58dc6mirror 1
[Fr Mär 15 20:09:25 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum0xc8a6977dexpected csum0xdfc1f678mirror 1
[Fr Mär 15 20:29:10 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): csum failed root 260 ino 257 off 6655332352 csum
0x20ff0887expected csum0x08ed0bcemirror 1
[Fr Mär 15 20:33:41 2024] BTRFS error (device sdc1): bdev/dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): direct IO failed ino 257 op0x0offset0x18cb02000len 16384 err no 10
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): csum failed root 256 ino 290 off 6655332352 csum0xb023863aexpected csum0x4f8e355dmirror 1
[Fr Mär 15 20:35:18 2024] BTRFS error (device sdc1): bdev/dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): direct IO failed ino 290 op0x0offset0x18cb05000len 4096 err no 10create full clone of drive scsi0 (btrfs-ssd-zstd-qcow2:106/vm-106-disk-0.raw)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 34.2 MiB of 20.0 GiB (0.17%) in 0s
drive-scsi0: transferred 301.6 MiB of 20.0 GiB (1.47%) in 1s
drive-scsi0: transferred 489.1 MiB of 20.0 GiB (2.39%) in 2s
drive-scsi0: transferred 679.9 MiB of 20.0 GiB (3.32%) in 3s
drive-scsi0: transferred 1.1 GiB of 20.0 GiB (5.70%) in 4s
drive-scsi0: transferred 1.4 GiB of 20.0 GiB (6.89%) in 5s
drive-scsi0: transferred 1.7 GiB of 20.0 GiB (8.55%) in 6s
drive-scsi0: transferred 1.9 GiB of 20.0 GiB (9.53%) in 7s
drive-scsi0: transferred 2.2 GiB of 20.0 GiB (10.97%) in 8s
drive-scsi0: transferred 2.4 GiB of 20.0 GiB (11.84%) in 9s
drive-scsi0: transferred 2.8 GiB of 20.0 GiB (13.89%) in 10s
drive-scsi0: transferred 2.9 GiB of 20.0 GiB (14.74%) in 11s
drive-scsi0: transferred 6.2 GiB of 20.0 GiB (31.14%) in 12s
drive-scsi0: transferred 8.2 GiB of 20.0 GiB (41.17%) in 13s
drive-scsi0: transferred 10.3 GiB of 20.0 GiB (51.27%) in 14s
drive-scsi0: transferred 10.5 GiB of 20.0 GiB (52.75%) in 15s
drive-scsi0: transferred 19.2 GiB of 20.0 GiB (95.86%) in 16s
drive-scsi0: transferred 19.3 GiB of 20.0 GiB (96.66%) in 17s
drive-scsi0: transferred 19.5 GiB of 20.0 GiB (97.46%) in 18s
drive-scsi0: transferred 19.7 GiB of 20.0 GiB (98.27%) in 19s
drive-scsi0: transferred 19.8 GiB of 20.0 GiB (99.08%) in 20s
drive-scsi0: Cancelling block job
drive-scsi0: Done.
TASK ERROR: storage migration failed: block job (mirror) error:
drive-scsi0: 'mirror' has been cancelled
Quoted 12 lines · Qu Wenruo
Quoted 4 lines · Roland
Hello,
can someone explain why compression is skipped when writing with direct-i/o ?
Because compression only happen with buffered write.
For direct IO we either go NOCOW or regular COW, no compression support.
Should we mention this (no compression in O_DIRECT) in doc? I haven't impression that doc said this.
Quoted 151 lines
The current compression code is always using page cache, just check the different algo implementations of
btrfs_compress_pages(), which all fetch their data from page cache.E.g. in
zstd_compress_pages(), we gofind_get_page()to grab the page from page cache as compression source.This is incompatible with the direct IO scheme, which is designed to avoid page cache completely (unless fall back to buffered write).
Maybe it's possible to make all those
*_compress_pages()to support direct IO, but that doesn't looks sane to me.As the idea of direct IO is to fully avoid page cache, and allow the user space program to take full control of its own cache, doing compression would introduce extra latency and make the performance characteristic much complex to user space.
Thanks,
QuQuoted 129 lines
is this to be expected ?
i wondered why i got uncompressed raw disk images in proxmox after disk migration to btrfs fs with compress-force=zstd, so i tried with dd if i can reproduce - and i can.
the problem is, that i cannot seem to disable direct I/O for different proxmox gui actions, will discuss in proxmox community what can be done, but i need more infos on this behaviour.
root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd if=disk.raw of=test.dat bs=1024k 20480+0 Datensätze ein 20480+0 Datensätze aus 21474836480 Bytes (21 GB, 20 GiB) kopiert, 46,411 s, 463 MB/sroot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 161844 regular extents (161844 refs), 0 inline, 85412 fragments.
Type Perc Disk Usage Uncompressed Referenced TOTAL 7% 1.5G 20G 20G
none 100% 339M 339M 339M
zstd 6% 1.2G 19G 19Groot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd if=disk.raw of=test.dat oflag=direct bs=1024k 20480+0 Datensätze ein 20480+0 Datensätze aus 21474836480 Bytes (21 GB, 20 GiB) kopiert, 161,319 s, 133 MB/sroot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 20480 regular extents (20480 refs), 0 inline, 1035 fragments.
Type Perc Disk Usage Uncompressed Referenced TOTAL 100% 20G 20G 20G
none 100% 20G 20G 20Gfurthermore, with qemu/proxmox virtual disk live migration, i'm getting reproducible checksum error with btrfs/compress-force=zstd when live migrating virtual disk within same harddisk , whereas copying the same disk/file with dd (cached and direct) does not trigger that.
happens on hdd and also on hdd, so this seems not to be disk related
# uname -a Linux pve-test1 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1 (2024-02-05T13:50Z) x86_64 GNU/Linuxregards
Roland[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum
0x5517115aexpected csum0x6b2af31emirror 1
[Fr Mär 15 20:07:23 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum0xd1d1c892expected csum0x0c24b5cdmirror 1
[Fr Mär 15 20:08:24 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum0x809096c2expected csum0x56b58dc6mirror 1
[Fr Mär 15 20:09:25 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum0xc8a6977dexpected csum0xdfc1f678mirror 1
[Fr Mär 15 20:29:10 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): csum failed root 260 ino 257 off 6655332352 csum
0x20ff0887expected csum0x08ed0bcemirror 1
[Fr Mär 15 20:33:41 2024] BTRFS error (device sdc1): bdev/dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): direct IO failed ino 257 op0x0offset0x18cb02000len 16384 err no 10
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): csum failed root 256 ino 290 off 6655332352 csum0xb023863aexpected csum0x4f8e355dmirror 1
[Fr Mär 15 20:35:18 2024] BTRFS error (device sdc1): bdev/dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): direct IO failed ino 290 op0x0offset0x18cb05000len 4096 err no 10create full clone of drive scsi0 (btrfs-ssd-zstd-qcow2:106/vm-106-disk-0.raw)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 34.2 MiB of 20.0 GiB (0.17%) in 0s
drive-scsi0: transferred 301.6 MiB of 20.0 GiB (1.47%) in 1s
drive-scsi0: transferred 489.1 MiB of 20.0 GiB (2.39%) in 2s
drive-scsi0: transferred 679.9 MiB of 20.0 GiB (3.32%) in 3s
drive-scsi0: transferred 1.1 GiB of 20.0 GiB (5.70%) in 4s
drive-scsi0: transferred 1.4 GiB of 20.0 GiB (6.89%) in 5s
drive-scsi0: transferred 1.7 GiB of 20.0 GiB (8.55%) in 6s
drive-scsi0: transferred 1.9 GiB of 20.0 GiB (9.53%) in 7s
drive-scsi0: transferred 2.2 GiB of 20.0 GiB (10.97%) in 8s
drive-scsi0: transferred 2.4 GiB of 20.0 GiB (11.84%) in 9s
drive-scsi0: transferred 2.8 GiB of 20.0 GiB (13.89%) in 10s
drive-scsi0: transferred 2.9 GiB of 20.0 GiB (14.74%) in 11s
drive-scsi0: transferred 6.2 GiB of 20.0 GiB (31.14%) in 12s
drive-scsi0: transferred 8.2 GiB of 20.0 GiB (41.17%) in 13s
drive-scsi0: transferred 10.3 GiB of 20.0 GiB (51.27%) in 14s
drive-scsi0: transferred 10.5 GiB of 20.0 GiB (52.75%) in 15s
drive-scsi0: transferred 19.2 GiB of 20.0 GiB (95.86%) in 16s
drive-scsi0: transferred 19.3 GiB of 20.0 GiB (96.66%) in 17s
drive-scsi0: transferred 19.5 GiB of 20.0 GiB (97.46%) in 18s
drive-scsi0: transferred 19.7 GiB of 20.0 GiB (98.27%) in 19s
drive-scsi0: transferred 19.8 GiB of 20.0 GiB (99.08%) in 20s
drive-scsi0: Cancelling block job
drive-scsi0: Done.
TASK ERROR: storage migration failed: block job (mirror) error:
drive-scsi0: 'mirror' has been cancelled
Quoted 10 lines · HAN Yuwei
Quoted 8 lines · Qu Wenruo
Quoted 4 lines · Roland
Hello,
can someone explain why compression is skipped when writing with direct-i/o ?
Quoted 6 lines
Quoted 4 lines
Because compression only happen with buffered write.
For direct IO we either go NOCOW or regular COW, no compression support.
Should we mention this (no compression in
O_DIRECT) in doc? I haven't impression that doc said this.
I'd say yes, since there is already some confusion, feel free to submit a patch for the doc.
Thanks,
Qu
Quoted 151 lines
Quoted 151 lines
The current compression code is always using page cache, just check the different algo implementations of
btrfs_compress_pages(), which all fetch their data from page cache.E.g. in
zstd_compress_pages(), we gofind_get_page()to grab the page from page cache as compression source.This is incompatible with the direct IO scheme, which is designed to avoid page cache completely (unless fall back to buffered write).
Maybe it's possible to make all those
*_compress_pages()to support direct IO, but that doesn't looks sane to me.As the idea of direct IO is to fully avoid page cache, and allow the user space program to take full control of its own cache, doing compression would introduce extra latency and make the performance characteristic much complex to user space.
Thanks,
QuQuoted 129 lines
is this to be expected ?
i wondered why i got uncompressed raw disk images in proxmox after disk migration to btrfs fs with compress-force=zstd, so i tried with dd if i can reproduce - and i can.
the problem is, that i cannot seem to disable direct I/O for different proxmox gui actions, will discuss in proxmox community what can be done, but i need more infos on this behaviour.
root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd if=disk.raw of=test.dat bs=1024k 20480+0 Datensätze ein 20480+0 Datensätze aus 21474836480 Bytes (21 GB, 20 GiB) kopiert, 46,411 s, 463 MB/sroot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 161844 regular extents (161844 refs), 0 inline, 85412 fragments.
Type Perc Disk Usage Uncompressed Referenced TOTAL 7% 1.5G 20G 20G
none 100% 339M 339M 339M
zstd 6% 1.2G 19G 19Groot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd if=disk.raw of=test.dat oflag=direct bs=1024k 20480+0 Datensätze ein 20480+0 Datensätze aus 21474836480 Bytes (21 GB, 20 GiB) kopiert, 161,319 s, 133 MB/sroot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 20480 regular extents (20480 refs), 0 inline, 1035 fragments.
Type Perc Disk Usage Uncompressed Referenced TOTAL 100% 20G 20G 20G
none 100% 20G 20G 20Gfurthermore, with qemu/proxmox virtual disk live migration, i'm getting reproducible checksum error with btrfs/compress-force=zstd when live migrating virtual disk within same harddisk , whereas copying the same disk/file with dd (cached and direct) does not trigger that.
happens on hdd and also on hdd, so this seems not to be disk related
# uname -a Linux pve-test1 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1 (2024-02-05T13:50Z) x86_64 GNU/Linuxregards
Roland[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum
0x5517115aexpected csum0x6b2af31emirror 1
[Fr Mär 15 20:07:23 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum0xd1d1c892expected csum0x0c24b5cdmirror 1
[Fr Mär 15 20:08:24 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum0x809096c2expected csum0x56b58dc6mirror 1
[Fr Mär 15 20:09:25 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): csum failed root 260 ino 257 off 6655332352 csum0xc8a6977dexpected csum0xdfc1f678mirror 1
[Fr Mär 15 20:29:10 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): direct IO failed ino 257 op0x0offset0x18cb05000len 4096 err no 10[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): csum failed root 260 ino 257 off 6655332352 csum
0x20ff0887expected csum0x08ed0bcemirror 1
[Fr Mär 15 20:33:41 2024] BTRFS error (device sdc1): bdev/dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): direct IO failed ino 257 op0x0offset0x18cb02000len 16384 err no 10
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): csum failed root 256 ino 290 off 6655332352 csum0xb023863aexpected csum0x4f8e355dmirror 1
[Fr Mär 15 20:35:18 2024] BTRFS error (device sdc1): bdev/dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): direct IO failed ino 290 op0x0offset0x18cb05000len 4096 err no 10create full clone of drive scsi0 (btrfs-ssd-zstd-qcow2:106/vm-106-disk-0.raw)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 34.2 MiB of 20.0 GiB (0.17%) in 0s
drive-scsi0: transferred 301.6 MiB of 20.0 GiB (1.47%) in 1s
drive-scsi0: transferred 489.1 MiB of 20.0 GiB (2.39%) in 2s
drive-scsi0: transferred 679.9 MiB of 20.0 GiB (3.32%) in 3s
drive-scsi0: transferred 1.1 GiB of 20.0 GiB (5.70%) in 4s
drive-scsi0: transferred 1.4 GiB of 20.0 GiB (6.89%) in 5s
drive-scsi0: transferred 1.7 GiB of 20.0 GiB (8.55%) in 6s
drive-scsi0: transferred 1.9 GiB of 20.0 GiB (9.53%) in 7s
drive-scsi0: transferred 2.2 GiB of 20.0 GiB (10.97%) in 8s
drive-scsi0: transferred 2.4 GiB of 20.0 GiB (11.84%) in 9s
drive-scsi0: transferred 2.8 GiB of 20.0 GiB (13.89%) in 10s
drive-scsi0: transferred 2.9 GiB of 20.0 GiB (14.74%) in 11s
drive-scsi0: transferred 6.2 GiB of 20.0 GiB (31.14%) in 12s
drive-scsi0: transferred 8.2 GiB of 20.0 GiB (41.17%) in 13s
drive-scsi0: transferred 10.3 GiB of 20.0 GiB (51.27%) in 14s
drive-scsi0: transferred 10.5 GiB of 20.0 GiB (52.75%) in 15s
drive-scsi0: transferred 19.2 GiB of 20.0 GiB (95.86%) in 16s
drive-scsi0: transferred 19.3 GiB of 20.0 GiB (96.66%) in 17s
drive-scsi0: transferred 19.5 GiB of 20.0 GiB (97.46%) in 18s
drive-scsi0: transferred 19.7 GiB of 20.0 GiB (98.27%) in 19s
drive-scsi0: transferred 19.8 GiB of 20.0 GiB (99.08%) in 20s
drive-scsi0: Cancelling block job
drive-scsi0: Done.
TASK ERROR: storage migration failed: block job (mirror) error:
drive-scsi0: 'mirror' has been cancelled
thanks for all your feedback
Am 19.03.24 um 05:18 schrieb Qu Wenruo:
Quoted 195 lines · HAN Yuwei
Quoted 10 lines · Qu Wenruo
Quoted 8 lines · Roland
Quoted 4 lines
Hello,
can someone explain why compression is skipped when writing with direct-i/o ?
Quoted 7 lines
Quoted 5 lines
Because compression only happen with buffered write.
For direct IO we either go NOCOW or regular COW, no compression support.
Should we mention this (no compression in
O_DIRECT) in doc? I haven't impression that doc said this.I'd say yes, since there is already some confusion, feel free to submit a patch for the doc.
Thanks,
QuQuoted 168 lines
Quoted 168 lines
The current compression code is always using page cache, just check the different algo implementations of
btrfs_compress_pages(), which all fetch their data from page cache.E.g. in
zstd_compress_pages(), we gofind_get_page()to grab the page from page cache as compression source.This is incompatible with the direct IO scheme, which is designed to avoid page cache completely (unless fall back to buffered write).
Maybe it's possible to make all those
*_compress_pages()to support direct IO, but that doesn't looks sane to me.As the idea of direct IO is to fully avoid page cache, and allow the user space program to take full control of its own cache, doing compression would introduce extra latency and make the performance characteristic much complex to user space.
Thanks,
QuQuoted 146 lines
is this to be expected ?
i wondered why i got uncompressed raw disk images in proxmox after disk
migration to btrfs fs with compress-force=zstd, so i tried with dd if i
can reproduce - and i can.the problem is, that i cannot seem to disable direct I/O for different proxmox gui actions, will discuss in proxmox community what can be done,
but i need more infos on this behaviour.root@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd if=disk.raw of=test.dat bs=1024k 20480+0 Datensätze ein 20480+0 Datensätze aus 21474836480 Bytes (21 GB, 20 GiB) kopiert, 46,411 s, 463 MB/sroot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 161844 regular extents (161844 refs), 0 inline, 85412
fragments.
Type Perc Disk Usage Uncompressed Referenced TOTAL 7% 1.5G 20G 20G none 100% 339M 339M 339M zstd 6% 1.2G 19G 19Groot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# dd if=disk.raw of=test.dat oflag=direct bs=1024k 20480+0 Datensätze ein 20480+0 Datensätze aus 21474836480 Bytes (21 GB, 20 GiB) kopiert, 161,319 s, 133 MB/sroot@pve-test1:/btrfs-hdd-zstd/vms-raw/images/106/vm-106-disk-0# compsize test.dat
Processed 1 file, 20480 regular extents (20480 refs), 0 inline, 1035 fragments.
Type Perc Disk Usage Uncompressed Referenced TOTAL 100% 20G 20G 20G
none 100% 20G 20G 20Gfurthermore, with qemu/proxmox virtual disk live migration, i'm getting
reproducible checksum error with btrfs/compress-force=zstd when live migrating virtual disk within same harddisk , whereas copying the same disk/file with dd (cached and direct) does not trigger that.happens on hdd and also on hdd, so this seems not to be disk related
# uname -a Linux pve-test1 6.5.13-1-pve #1 SMP PREEMPT_DYNAMIC PMX 6.5.13-1 (2024-02-05T13:50Z) x86_64 GNU/Linuxregards
Roland[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): csum failed root
260 ino 257 off 6655332352 csum0x5517115aexpected csum0x6b2af31emirror 1
[Fr Mär 15 20:07:23 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:07:23 2024] BTRFS warning (device sda1): direct IO failed
ino 257 op0x0offset0x18cb05000len 4096 err no 10
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): csum failed root
260 ino 257 off 6655332352 csum0xd1d1c892expected csum0x0c24b5cdmirror 1
[Fr Mär 15 20:08:24 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:08:24 2024] BTRFS warning (device sda1): direct IO failed
ino 257 op0x0offset0x18cb05000len 4096 err no 10
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): csum failed root
260 ino 257 off 6655332352 csum0x809096c2expected csum0x56b58dc6mirror 1
[Fr Mär 15 20:09:25 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[Fr Mär 15 20:09:25 2024] BTRFS warning (device sda1): direct IO failed
ino 257 op0x0offset0x18cb05000len 4096 err no 10
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): csum failed root
260 ino 257 off 6655332352 csum0xc8a6977dexpected csum0xdfc1f678mirror 1
[Fr Mär 15 20:29:10 2024] BTRFS error (device sda1): bdev/dev/sda1
errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
[Fr Mär 15 20:29:10 2024] BTRFS warning (device sda1): direct IO failed
ino 257 op0x0offset0x18cb05000len 4096 err no 10[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): csum failed root
260 ino 257 off 6655332352 csum0x20ff0887expected csum0x08ed0bcemirror 1
[Fr Mär 15 20:33:41 2024] BTRFS error (device sdc1): bdev/dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[Fr Mär 15 20:33:41 2024] BTRFS warning (device sdc1): direct IO failed
ino 257 op0x0offset0x18cb02000len 16384 err no 10 [Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): csum failed root
256 ino 290 off 6655332352 csum0xb023863aexpected csum0x4f8e355dmirror 1
[Fr Mär 15 20:35:18 2024] BTRFS error (device sdc1): bdev/dev/sdc1
errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[Fr Mär 15 20:35:18 2024] BTRFS warning (device sdc1): direct IO failed
ino 290 op0x0offset0x18cb05000len 4096 err no 10create full clone of drive scsi0 (btrfs-ssd-zstd-qcow2:106/vm-106-disk-0.raw)
drive mirror is starting for drive-scsi0
drive-scsi0: transferred 34.2 MiB of 20.0 GiB (0.17%) in 0s
drive-scsi0: transferred 301.6 MiB of 20.0 GiB (1.47%) in 1s
drive-scsi0: transferred 489.1 MiB of 20.0 GiB (2.39%) in 2s
drive-scsi0: transferred 679.9 MiB of 20.0 GiB (3.32%) in 3s
drive-scsi0: transferred 1.1 GiB of 20.0 GiB (5.70%) in 4s
drive-scsi0: transferred 1.4 GiB of 20.0 GiB (6.89%) in 5s
drive-scsi0: transferred 1.7 GiB of 20.0 GiB (8.55%) in 6s
drive-scsi0: transferred 1.9 GiB of 20.0 GiB (9.53%) in 7s
drive-scsi0: transferred 2.2 GiB of 20.0 GiB (10.97%) in 8s
drive-scsi0: transferred 2.4 GiB of 20.0 GiB (11.84%) in 9s
drive-scsi0: transferred 2.8 GiB of 20.0 GiB (13.89%) in 10s
drive-scsi0: transferred 2.9 GiB of 20.0 GiB (14.74%) in 11s
drive-scsi0: transferred 6.2 GiB of 20.0 GiB (31.14%) in 12s
drive-scsi0: transferred 8.2 GiB of 20.0 GiB (41.17%) in 13s
drive-scsi0: transferred 10.3 GiB of 20.0 GiB (51.27%) in 14s
drive-scsi0: transferred 10.5 GiB of 20.0 GiB (52.75%) in 15s
drive-scsi0: transferred 19.2 GiB of 20.0 GiB (95.86%) in 16s
drive-scsi0: transferred 19.3 GiB of 20.0 GiB (96.66%) in 17s
drive-scsi0: transferred 19.5 GiB of 20.0 GiB (97.46%) in 18s
drive-scsi0: transferred 19.7 GiB of 20.0 GiB (98.27%) in 19s
drive-scsi0: transferred 19.8 GiB of 20.0 GiB (99.08%) in 20s
drive-scsi0: Cancelling block job
drive-scsi0: Done.
TASK ERROR: storage migration failed: block job (mirror) error:
drive-scsi0: 'mirror' has been cancelled