Filesystem corruption with LVM's pvmove onto a PV with a larger physical block size
Hi,
we just encountered an error when using LVM's pvmove command to move the data from an un-encrypted LVM physical volume onto an encrypted volume. After the pvmove has completed, the file system on the logical volume that resides on the moved physical volumes is corrupted and all data on this LV is lost.
As a special condition, we are using the sector-size option with cryptsetup (LUKS2) to use 4096 byte sectors to increase the crypt performance.
According to Ondrej Kozina in https://www.saout.de/pipermail/dm-crypt/2019-February/006078.html this is due to the fact that a sector size of 4096 also causes the physical block size to be (at least) 4096. Using the default sector size of 512 does not show the problem.
The source PV of the pvmove has a physical block size of 512 (SCSI disk in our case). So when moving this to the encrypted volume the physical block size becomes 4096. The file system does not seem to like this....
Please note that this problem can also happen in other cases, such as mixing disks with different block sizes (e.g. SCSI disks with 512 bytes and s390x-DASDs with 4096 block size).
In my opinion this is a general problem that needs to be addressed by LVM.
Whenever you pvmove data to a PV with a larger physical block size you will corrupt the file system and thus cause data loss. The same probably happens when using lvextend to enlarge an LV onto a PV with a larger physical block size.
Bottom line: LVM should reject extending a volume group with a PV that has a larger physical block size than the PVs that belong to the volume group already. There might be a force option to override this check in case the user really knows what he does. But the default behavior should be to protect the user from file system corruptions and thus from data loss.
Yes, having a backup before doing pvmove or similar is of course wise, but I still think that LVM should prevent a user for damaging the file system by using LVM's command line tools.
Here is how to reproduce this (note the error messages on the very last command):
# sudo dd if=/dev/zero of=loopbackfile1.img bs=500M count=1 1+0 records in 1+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 2.32777 s, 225 MB/s
# sudo dd if=/dev/zero of=loopbackfile2.img bs=500M count=1 1+0 records in 1+0 records out 524288000 bytes (524 MB, 500 MiB) copied, 1.89992 s, 276 MB/s
# losetup -fP loopbackfile1.img
# losetup -fP loopbackfile2.img
# pvcreate /dev/loop0 Physical volume "/dev/loop0" successfully created.
# vgcreate LOOP_VG /dev/loop0 Volume group "LOOP_VG" successfully created
# lvcreate -L 300MB LOOP_VG -n LV /dev/loop0 Logical volume "LV" created.
# mkfs.ext4 /dev/mapper/LOOP_VG-LV
mke2fs 1.44.1 (24-Mar-2018)
Discarding device blocks: done
Creating filesystem with 307200 1k blocks and 76912 inodes
Filesystem UUID: 344289a3-e251-4d88-b03d-a71a4be2a8ec
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729, 204801, 221185Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
# mount /dev/mapper/LOOP_VG-LV /mnt
# cryptsetup luksFormat --type luks2 --sector-size 4096 /dev/loop1
WARNING!
========
This will overwrite data on /dev/loop1 irrevocably.
Are you sure? (Type uppercase yes): YES
Enter passphrase for /dev/loop1: loop
Verify passphrase: loop
# cryptsetup luksOpen /dev/loop1 enc-loop Enter passphrase for /dev/loop1: loop
# pvcreate /dev/mapper/enc-loop Physical volume "/dev/mapper/enc-loop" successfully created.
# vgextend LOOP_VG /dev/mapper/enc-loop Volume group "LOOP_VG" successfully extended
# pvs PV VG Fmt Attr PSize PFree /dev/loop0 LOOP_VG lvm2 a-- 496.00m 196.00m /dev/mapper/enc-loop LOOP_VG lvm2 a-- 492.00m 492.00m
# pvmove /dev/loop0 /dev/mapper/enc-loop /dev/loop0: Moved: 30.67% /dev/loop0: Moved: 100.00%
# pvs /dev/LOOP_VG/LV: read failed after 0 of 1024 at 0: Invalid argument /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314507264: Invalid argument /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314564608: Invalid argument /dev/LOOP_VG/LV: read failed after 0 of 1024 at 4096: Invalid argument PV VG Fmt Attr PSize PFree /dev/loop0 LOOP_VG lvm2 a-- 496.00m 496.00m /dev/mapper/enc-loop LOOP_VG lvm2 a-- 492.00m 192.00m
In case the filesystem of the logical volume is not mounted at the time of pvmove, it gets corrupted anyway, but you only see errors when trying to mount it.
Signature · 10 lines
Ingo Franzki eMail: [email protected] Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/
Quoted 2 lines · Ingo Franzki
we just encountered an error when using LVM's pvmove command to move the data from an un-encrypted LVM physical volume onto an encrypted volume. After the pvmove has completed, the file system on the logical volume that resides on the moved physical volumes is corrupted and all data on this LV is lost.
Hello, your message is interesting. And also this thread:
https://www.redhat.com/archives/linux-lvm/2019-February/msg00002.html
But I'd like to know if I understood correctly.
Should I care about the physical disk size when I use LVM? Mixing disk with different sector size (512b and 4k) is dangerous?
Your message and others from the other thread, seems to say that LVM doesn't handle correctly that situation and that if I pvmove data between a 512b disk and a 4k disk (or viceversa), it will lead to a massive filesystem corruption. If I understood correctly, the problem that you described looks unrelated to encrypted volume and was only exacerbated by that. Right?
Cesare.
Quoted 9 lines · Cesare Leonardi
Quoted 2 lines · Ingo Franzki
we just encountered an error when using LVM's pvmove command to move the data from an un-encrypted LVM physical volume onto an encrypted volume. After the pvmove has completed, the file system on the logical volume that resides on the moved physical volumes is corrupted and all data on this LV is lost.
Hello, your message is interesting. And also this thread:
https://www.redhat.com/archives/linux-lvm/2019-February/msg00002.htmlBut I'd like to know if I understood correctly.
Should I care about the physical disk size when I use LVM? Mixing disk with different sector size (512b and 4k) is dangerous?
As far as I can tell: Yes if you pvmove data around or lvextend an LV onto another PV with a larger physical block size that is dangerous. Creating new LVs and thus new file systems on mixed configurations seem to be OK.
Quoted 2 lines
Your message and others from the other thread, seems to say that LVM doesn't handle correctly that situation and that if I pvmove data between a 512b disk and a 4k disk (or viceversa), it will lead to a massive filesystem corruption. If I understood correctly, the problem that you described looks unrelated to encrypted volume and was only exacerbated by that. Right?
Moving from 512 to 4096 seems to cause FS corruption, moving from 4096 to 512 does not. So I guess its only a problem when moving to larger physical block sizes.
And yes, its unrelated to encrypted volumes, it can happen with any block device of different physical block sizes that you use as PV. E.g. SCSI disks exist in 512 bytes block size and 4096 bytes block size. Or s390-DASDs which always have 4096 bytes blocks.
We just encountered it using encrypted disks with the sector-size option to increase encryption performance. So actually we want people to use larger sector sizes, but this seems to cause problems with LVM.
The good thing about the example with encrypted volumes on loopback devices is that you can reproduce the problem on any platform, without having certain hardware requirements.
Quoted 4 lines
Cesare.
Signature · 10 lines
Ingo Franzki eMail: [email protected] Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/
Quoted 3 lines · Ingo Franzki
The good thing about the example with encrypted volumes on loopback devices is that you can reproduce the problem on any platform, without having certain hardware requirements.
The losetup command has a --sector-size option that sets the logical sector size. I wonder if that is sufficient to reproduce the problem.
Signature · 3 lines
Stuart D. Gathman <[email protected]> "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 9 lines · Stuart D. Gathman
Quoted 3 lines · Ingo Franzki
The good thing about the example with encrypted volumes on loopback devices is that you can reproduce the problem on any platform, without having certain hardware requirements.
The losetup command has a --sector-size option that sets the logical sector size. I wonder if that is sufficient to reproduce the problem.
Yes that should work:
# losetup -fP loopbackfile.img --sector-size 4096 # blockdev --getpbsz /dev/loop0 4096
Signature · 10 lines
Ingo Franzki eMail: [email protected] Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/
Quoted 2 lines · Ingo Franzki
As far as I can tell: Yes if you pvmove data around or lvextend an LV onto another PV with a larger physical block size that is dangerous. Creating new LVs and thus new file systems on mixed configurations seem to be OK.
[...]
Quoted 1 line
And yes, its unrelated to encrypted volumes, it can happen with any block device of different physical block sizes that you use as PV.
Thank you Ingo for the precious informations you are giving here.
Not to be pedantic, but what do you mean with physical block? Because with modern disks the term is not always clear. Let's take a mechanical disk with 512e sectors, that is with 4k sectors but exposed as 512 byte sectors. Fdisk will refer to it with these terms:
Sector size (logical/physical): 512 bytes / 4096 bytes
What you are referring as physical size is actually the logical size reported by fdisk, right? And if it's correct, I guess that should be safe to add the above disk with 512e sectors to an LVM storage composed only by disks with real 512 byte sectors. I expect that from the LVM point of view this should not be even considered a mixed sector size setup, even if the real physical sector size of the added disk is 4096 byte.
Do you agree or do you think it would be better to test this specific setup?
Cesare.
Quoted 6 lines · Cesare Leonardi
Not to be pedantic, but what do you mean with physical block? Because with modern disks the term is not always clear. Let's take a mechanical disk with 512e sectors, that is with 4k sectors but exposed as 512 byte sectors. Fdisk will refer to it with these terms:
Sector size (logical/physical): 512 bytes / 4096 bytesWhat you are referring as physical size is actually the logical size reported by fdisk, right? And if it's correct, I guess that should be safe to add the above disk with 512e sectors to an LVM storage composed only by disks with real 512 byte sectors. I expect that from the LVM point of view this should not be even considered a mixed sector size setup, even if the real physical sector size of the added disk is 4096 byte.
Do you agree or do you think it would be better to test this specific setup?
I would definitely test it, using the same test script that reproduces the problem with loopback devices.
That said, I believe you are right - it should definitely work. Most of my drives are 512/4096 logical/phys. If you actually write a single 512 byte sector, however, the disk firmware will have to do a read/modify/write cycle - which can tank performance.
hdparm will report logical and physical sector size - but there doesn't seem to be an option to set logical sectory size. There really is no need once you already support a smaller logical sector size, as the performance hit can be avoided by aligned filesystems with 4k+ block size (most modern filesystems).
Once I encountered a bug in drive firmware where the R/M/W did not work correctly with certain read/write patterns (involving unaligned multi sector writes). I do not wish that on anyone. (don't worry, that drive model is long gone...).
Signature · 3 lines
Stuart D. Gathman <[email protected]> "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 8 lines · Cesare Leonardi
Quoted 2 lines · Ingo Franzki
As far as I can tell: Yes if you pvmove data around or lvextend an LV onto another PV with a larger physical block size that is dangerous. Creating new LVs and thus new file systems on mixed configurations seem to be OK.
[...]
Quoted 1 line
And yes, its unrelated to encrypted volumes, it can happen with any block device of different physical block sizes that you use as PV.
Quoted 8 lines
Thank you Ingo for the precious informations you are giving here.
Not to be pedantic, but what do you mean with physical block? Because with modern disks the term is not always clear. Let's take a mechanical disk with 512e sectors, that is with 4k sectors but exposed as 512 byte sectors. Fdisk will refer to it with these terms:
Sector size (logical/physical): 512 bytes / 4096 bytesWhat you are referring as physical size is actually the logical size reported by fdisk, right? And if it's correct, I guess that should be safe to add the above disk with 512e sectors to an LVM storage composed only by disks with real 512 byte sectors. I expect that from the LVM point of view this should not be even considered a mixed sector size setup, even if the real physical sector size of the added disk is 4096 byte.
Do you agree or do you think it would be better to test this specific setup?
Well, there are the following 2 commands:
Get physical block size:
blockdev --getpbsz <device>
Get logical block size:
blockdev --getbsz <device>
Filesystems seem to care about the physical block size only, not the logical block size.
So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful...
Quoted 10 lines
Cesare.
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Signature · 10 lines
Ingo Franzki eMail: [email protected] Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/
Quoted 11 lines
Well, there are the following 2 commands:
Get physical block size:
blockdev --getpbsz <device>
Get logical block size:
blockdev --getbsz <device>Filesystems seem to care about the physical block size only, not the logical block size.
So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful...
Hello everybody.
Maybe, I don’t understand what do you mean. What the logical block size mean? But on my machines(CentOS7), this utility get me the strange results (output reduced):
smartctl -i /dev/sda; blockdev --getbsz --getpbsz /dev/sda
Device Model: INTEL SSDSC2KB480G8 User Capacity: 480,103,981,056 bytes [480 GB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: Solid State Device
4096
4096
smartctl -i /dev/sdb; blockdev --getbsz --getpbsz /dev/sdb
Device Model: HGST HUS722T2TALA604 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches
4096
512
As you see “–getbsz” forever 4096.
But I think it must be forever 512.
What does it mean?
Thank you.
Ilia.
Quoted 31 lines · Ilia Zykov
Quoted 11 lines
Well, there are the following 2 commands:
Get physical block size:
blockdev --getpbsz <device>
Get logical block size:
blockdev --getbsz <device>Filesystems seem to care about the physical block size only, not the logical block size.
So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful...
Hello everybody.
Maybe, I don’t understand what do you mean. What the logical block size mean? But on my machines(CentOS7), this utility get me the strange results (output reduced):smartctl -i
/dev/sda; blockdev --getbsz --getpbsz/dev/sdaDevice Model: INTEL SSDSC2KB480G8 User Capacity: 480,103,981,056 bytes [480 GB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: Solid State Device4096
4096smartctl -i
/dev/sdb; blockdev --getbsz --getpbsz/dev/sdbDevice Model: HGST HUS722T2TALA604 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches
Quoted 4 lines
4096
512As you see “–getbsz” forever 4096.
I also see logical block size to be 4096 for all devices on my system.
Quoted 2 lines
But I think it must be forever 512.
What does it mean?
I have seen the following description about logical and physical block sizes somewhere in the internet:
"Logical block sizes are the units used by the 'kernel' for read/write operations. Physical block sizes are the units which 'disk controllers' use for read/write operations."
For the problem mentioned in this thread, the physical block size is what you are looking for.
Quoted 4 lines
Thank you.
Ilia.
Signature · 10 lines
Ingo Franzki eMail: [email protected] Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/
Quoted 10 lines
Quoted 10 lines
smartctl -i
/dev/sdb; blockdev --getbsz --getpbsz/dev/sdbDevice Model: HGST HUS722T2TALA604 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches4096
512
Quoted 6 lines
Quoted 1 line
As you see “–getbsz” forever 4096.
I also see logical block size to be 4096 for all devices on my system.
Quoted 2 lines
But I think it must be forever 512.
What does it mean?I have seen the following description about logical and physical block sizes somewhere in the internet:
"Logical block sizes are the units used by the 'kernel' for read/write operations.
Kernel can but usually does not want, because reduce performance.
Quoted 1 line
Physical block sizes are the units which 'disk controllers' use for read/write operations."
Not disk controller on the motherboard, but controller inside disk. We don't have access to it.
Quoted 3 lines
For the problem mentioned in this thread, the physical block size is what you are looking for.
Quoted 1 line
I think it is BUG in the "blockdev".
My question was:
Can this error(or similar) be related to a problem in pvmove?
Quoted 10 lines
Quoted 10 lines
smartctl -i
/dev/sdb; blockdev --getbsz --getpbsz/dev/sdbDevice Model: HGST HUS722T2TALA604 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: 7200 rpm Form Factor: 3.5 inches4096
512
Quoted 6 lines
Quoted 1 line
As you see “–getbsz” forever 4096.
I also see logical block size to be 4096 for all devices on my system.
Quoted 2 lines
But I think it must be forever 512.
What does it mean?I have seen the following description about logical and physical block sizes somewhere in the internet:
"Logical block sizes are the units used by the 'kernel' for read/write operations.
Kernel can but usually does not want, because reduce performance.
Quoted 1 line
Physical block sizes are the units which 'disk controllers' use for read/write operations."
Not disk controller on the motherboard, but controller inside disk. We don't have access to it.
Quoted 3 lines
For the problem mentioned in this thread, the physical block size is what you are looking for.
Quoted 1 line
I think it is BUG in the "blockdev(util-linux)".
My question was:
Can this error(or similar) be related to a problem in pvmove?
Quoted 6 lines
Quoted 4 lines
For the problem mentioned in this thread, the physical block size is what you are looking for.
Quoted 1 line
I think it is BUG in the "blockdev(util-linux)".
Tt's not a bug it's a feature :O
https://bugzilla.redhat.com/show_bug.cgi?id=1684078
Quoted 5 lines
My question was:
Can this error(or similar) be related to a problem in pvmove?
My question now:
Can this feature(or similar) be related to a problem in pvmove?
Thank you.
Quoted 12 lines
Discarding device blocks: done
Creating filesystem with 307200 1k blocks and 76912 inodes
......# pvs /dev/LOOP_VG/LV: read failed after 0 of 1024 at 0: Invalid argument /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314507264: Invalid argument /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314564608: Invalid argument /dev/LOOP_VG/LV: read failed after 0 of 1024 at 4096: Invalid argument PV VG Fmt Attr PSize PFree /dev/loop0 LOOP_VG lvm2 a-- 496.00m 496.00m /dev/mapper/enc-loop LOOP_VG lvm2 a-- 492.00m 192.00m
Quoted 2 lines
In case the filesystem of the logical volume is not mounted at the time of pvmove, it gets corrupted anyway, but you only see errors when trying to mount it.
It's because you FS had 1k blocks.
New device can't read with block 1k.
If you plan pvmove on device with 4k. Maybe you need make fs with:
"mkfs.ext4 -b 4096"
See comments:
https://bugzilla.redhat.com/show_bug.cgi?id=1684078
Quoted 12 lines · Ilia Zykov
Quoted 12 lines
Discarding device blocks: done
Creating filesystem with 307200 1k blocks and 76912 inodes
......# pvs /dev/LOOP_VG/LV: read failed after 0 of 1024 at 0: Invalid argument /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314507264: Invalid argument /dev/LOOP_VG/LV: read failed after 0 of 1024 at 314564608: Invalid argument /dev/LOOP_VG/LV: read failed after 0 of 1024 at 4096: Invalid argument PV VG Fmt Attr PSize PFree /dev/loop0 LOOP_VG lvm2 a-- 496.00m 496.00m /dev/mapper/enc-loop LOOP_VG lvm2 a-- 492.00m 192.00m
Quoted 7 lines
Quoted 2 lines
In case the filesystem of the logical volume is not mounted at the time of pvmove, it gets corrupted anyway, but you only see errors when trying to mount it.
It's because you FS had 1k blocks.
New device can't read with block 1k.
If you plan pvmove on device with 4k. Maybe you need make fs with:
"mkfs.ext4 -b 40
The beauty of pvmove is that it works on existing LVs with filesystems on it.
So I might have filesystems on LVs that used the default block size that I want to move via pvmove.
At the time the file system was created (possibly may years ago), I did not know that I would ever move it to a device with a larger block size. If I have to create a new filesystem after the move, then pvmove is useless.... I then could just create a new LV on the desired PV(s).
Quoted 4 lines
See comments:
https://bugzilla.redhat.com/show_bug.cgi?id=1684078
Signature · 10 lines
Ingo Franzki eMail: [email protected] Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/
Quoted 2 lines
At the time the file system was created (possibly may years ago), I did not know that I would ever move it to a device with a larger block size.
For this purpose all 4k disks have logical sector size 512. Don't look at "blockdev --getbsz" it's not property of physical(real) device.
Quoted 6 lines · Ingo Franzki
Well, there are the following 2 commands:
Get physical block size:
blockdev --getpbsz <device>
Get logical block size:
blockdev --getbsz <device>
I didn't know the blockdev command and, to recap, we have:
--getpbsz: physical sector size
--getss: logical sector size
--getbsz: blocksize used internally by the kernel
getpbsz/getss correspond to physical/logical sector size reported by fdisk, smartctl, etc.
Quoted 3 lines
Filesystems seem to care about the physical block size only, not the logical block size.
So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful...
I've done the test suggested by Stuart and it seems to contradict this.
I have pvmoved data from a 512/512 (logical/physical) disk to a newly added 512/4096 disk but I had no data corruption. Unfortunately I haven't any native 4k disk to repeat the same test.
Here is what I've done./dev/sdb: SSD with 512/512 sector size /dev/sdc: mechanical disk with 512/4096 sector size
# blockdev -v --getss --getpbsz --getbsz /dev/sdb get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096
# blockdev -v --getss --getpbsz --getbsz /dev/sdc get logical block (sector) size: 512 get physical block (sector) size: 4096 get blocksize: 4096
# pvcreate /dev/sdb4 # vgcreate vgtest /dev/sdb4 # lvcreate -L 1G vgtest /dev/sdb4 # mkfs.ext4 /dev/mapper/vgtest-lvol0 # mkdir /media/test # mount /dev/mapper/vgtest-lvol0 /media/test # cp -a SOMEDATA /media/test/ # umount /media/test # fsck.ext4 -f /dev/mapper/vgtest-lvol0
Filesystem created and no error on it. Now the disk with different physical size:
# pvcreate /dev/sdc1 # vgextend vgtest /dev/sdc1 # pvmove /dev/sdb4 /dev/sdc1 # fsck.ext4 -f /dev/mapper/vgtest-lvol0 # mount /dev/mapper/vgtest-lvol0 /media/test # ls /media/test/
The fsck command reports no errors, the filesystem is mountable and the data is there.
Looks like physical sector size didn't matter here. Or I'm missing something?
Cesare.
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PVwith a larger physical block size
Hello,
I think the filesystems address in Logical blocks, so this is the size which should match.
However the physical size might be relevant for alignment/sizing decisions of mkfs (but you would expect the to be encoded in the metadata of the filesystem so you can transport them (losing proper alignment which might affect Performance or robustness).
For ext3/4 I think the mkfs will use -b 4k by Default if your FS is at least 0,5GB.
BTW: some applications (like SQL Server) also care about the physical size to make sure they always write complete sectors in transactions and avoid read-modify-write scenarios.
Gruss
Bernd
Signature · 6 lines
http://bernd.eckenfels.net Von: Cesare Leonardi Gesendet: Freitag, 1. März 2019 02:24 An: Ingo Franzki; LVM general discussion and development Betreff: Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PVwith a larger physical block size
Quoted 6 lines · Ingo Franzki
Well, there are the following 2 commands:
Get physical block size:
blockdev --getpbsz <device>
Get logical block size:
blockdev --getbsz <device>
I didn't know the blockdev command and, to recap, we have:
--getpbsz: physical sector size
--getss: logical sector size
--getbsz: blocksize used internally by the kernel
getpbsz/getss correspond to physical/logical sector size reported by fdisk, smartctl, etc.
Quoted 3 lines
Filesystems seem to care about the physical block size only, not the logical block size.
So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful...
I've done the test suggested by Stuart and it seems to contradict this. I have pvmoved data from a 512/512 (logical/physical) disk to a newly added 512/4096 disk but I had no data corruption. Unfortunately I haven't any native 4k disk to repeat the same test.
Here is what I've done./dev/sdb: SSD with 512/512 sector size /dev/sdc: mechanical disk with 512/4096 sector size
# blockdev -v --getss --getpbsz --getbsz /dev/sdb get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096
# blockdev -v --getss --getpbsz --getbsz /dev/sdc get logical block (sector) size: 512 get physical block (sector) size: 4096 get blocksize: 4096
# pvcreate /dev/sdb4 # vgcreate vgtest /dev/sdb4 # lvcreate -L 1G vgtest /dev/sdb4 # mkfs.ext4 /dev/mapper/vgtest-lvol0 # mkdir /media/test # mount /dev/mapper/vgtest-lvol0 /media/test # cp -a SOMEDATA /media/test/ # umount /media/test # fsck.ext4 -f /dev/mapper/vgtest-lvol0
Filesystem created and no error on it. Now the disk with different physical size:
# pvcreate /dev/sdc1 # vgextend vgtest /dev/sdc1 # pvmove /dev/sdb4 /dev/sdc1 # fsck.ext4 -f /dev/mapper/vgtest-lvol0 # mount /dev/mapper/vgtest-lvol0 /media/test # ls /media/test/
The fsck command reports no errors, the filesystem is mountable and the data is there.
Looks like physical sector size didn't matter here. Or I'm missing something?
Cesare.
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 2 lines · Cesare Leonardi
I've done the test suggested by Stuart and it seems to contradict this.
I have pvmoved data from a 512/512 (logical/physical) disk to a newly added 512/4096 disk but I had no data corruption. Unfortunately I haven't any native 4k disk to repeat the same test.
Use a loopback device with logical block size set to 4096 to confirm that your test does detect corruption (using the same LV, filesystem, data).
I believe by "physical sector", the original reporter means logical, as he was using an encrypted block device that was virtual - there was no "physical" sector size. It was "physical" as far as the file system was concerned - where "physical" means "the next layer down".
Indeed, even the rotating disk drives make the physical sector size invisible except to performance tests. SSD drives have a "sector" size of 128k or 256k - the erase block, and performance improves when aligned to that.
Signature · 3 lines
Stuart D. Gathman <[email protected]> "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 9 lines · Stuart D. Gathman
Quoted 2 lines · Cesare Leonardi
I've done the test suggested by Stuart and it seems to contradict this.
I have pvmoved data from a 512/512 (logical/physical) disk to a newly added 512/4096 disk but I had no data corruption. Unfortunately I haven't any native 4k disk to repeat the same test.Use a loopback device with logical block size set to 4096 to confirm that your test does detect corruption (using the same LV, filesystem, data).
Quoted 5 lines
I believe by "physical sector", the original reporter means logical, as he was using an encrypted block device that was virtual - there was no "physical" sector size. It was "physical" as far as the file system was concerned - where "physical" means "the next layer down".
Well, let me cite from https://www.saout.de/pipermail/dm-crypt/2019-February/006078.html from Ondrej Kozina which is also referenced in my original post:
"dm-crypt advertise itself as a block device with physical sector size *at least* equal to encryption sector size. Traditionally it's been only 512B. So classical dm-crypt mapped over device with phys. sector size = 512B has no impact. If you mapped dm-crypt over block device with native physical sector size = 4096 you got dm-crypt exposing same limits as underlying block device. Again no problem. Just internally dm-crypt performed encryption on 512B blocks, but it had no impact on exposed limits.
But things get a bit different with encryption sector size > 512B.
If you map dm-crypt with encryption sector size set to 4096B over block device with physical sector size = 512B, dm-crypt must increase device limits to 4096. Because when it does encryption it must be aligned to 4096 bytes (and same wrt minimal i/o size)."
Quoted 6 lines
Indeed, even the rotating disk drives make the physical sector size invisible except to performance tests. SSD drives have a "sector" size of 128k or 256k - the erase block, and performance improves when aligned to that.
Signature · 10 lines
Ingo Franzki eMail: [email protected] Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/
Re: [linux-lvm] Filesystem corruption with LVM's pvmove onto a PVwith a larger physical block size
In reply to Bernd EckenfelsQuoted 7 lines · Bernd Eckenfels
Hello,
I think the filesystems address in Logical blocks, so this is the size which should match.
However the physical size might be relevant for alignment/sizing decisions of mkfs (but you would expect the to be encoded in the metadata of the filesystem so you can transport them (losing proper alignment which might affect Performance or robustness).
For ext3/4 I think the mkfs will use -b 4k by Default if your FS is at least 0,5GB.
That may make a difference. My tests were with relatively small volumes, so it might not be using -b 4k due to the size of the volume?
Quoted 6 lines
BTW: some applications (like SQL Server) also care about the physical size to make sure they always write complete sectors in transactions and avoid read-modify-write scenarios.
Gruss
Bernd
Signature · 10 lines
Ingo Franzki eMail: [email protected] Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/
Quoted 13 lines · Cesare Leonardi
Quoted 6 lines · Ingo Franzki
Well, there are the following 2 commands:
Get physical block size:
blockdev --getpbsz <device>
Get logical block size:
blockdev --getbsz <device>I didn't know the blockdev command and, to recap, we have:
--getpbsz: physical sector size
--getss: logical sector size
--getbsz: blocksize used internally by the kernel
Quoted 8 lines
getpbsz/getss correspond to physical/logical sector size reported by fdisk, smartctl, etc.
Quoted 3 lines
Filesystems seem to care about the physical block size only, not the logical block size.
So as soon as you have PVs with different physical block sizes (as reported by blockdev --getpbsz) I would be very careful...
I've done the test suggested by Stuart and it seems to contradict this.
I have pvmoved data from a 512/512 (logical/physical) disk to a newly added 512/4096 disk but I had no data corruption. Unfortunately I haven't any native 4k disk to repeat the same test.
Hmm, maybe the size of the volume plays a role as Bernd has pointed out. ext4 may use -b 4K by default on larger devices. Once the FS uses 4K block anyway you wont see the problem.
Use tune2fs -l <device> after you created the file system and check if it is using 4K blocks on your 512/512 device. If so, then you won't see the problem when moved to a 4K block size device.
Quoted 47 lines
Here is what I've done.
/dev/sdb: SSD with 512/512 sector size/dev/sdc: mechanical disk with 512/4096 sector size# blockdev -v --getss --getpbsz --getbsz /dev/sdb get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096# blockdev -v --getss --getpbsz --getbsz /dev/sdc get logical block (sector) size: 512 get physical block (sector) size: 4096 get blocksize: 4096# pvcreate /dev/sdb4 # vgcreate vgtest /dev/sdb4 # lvcreate -L 1G vgtest /dev/sdb4 # mkfs.ext4 /dev/mapper/vgtest-lvol0 # mkdir /media/test # mount /dev/mapper/vgtest-lvol0 /media/test # cp -a SOMEDATA /media/test/ # umount /media/test # fsck.ext4 -f /dev/mapper/vgtest-lvol0Filesystem created and no error on it. Now the disk with different physical size:
# pvcreate /dev/sdc1 # vgextend vgtest /dev/sdc1 # pvmove /dev/sdb4 /dev/sdc1 # fsck.ext4 -f /dev/mapper/vgtest-lvol0 # mount /dev/mapper/vgtest-lvol0 /media/test # ls /media/test/The fsck command reports no errors, the filesystem is mountable and the data is there.
Looks like physical sector size didn't matter here. Or I'm missing something?
Cesare.
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Signature · 10 lines
Ingo Franzki eMail: [email protected] Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/
Hello Ingo, I've made several tests but I was unable to trigger any filesystem corruption. Maybe the trouble you encountered are specific to encrypted device?
Yesterday and today I've used:
Debian unstable
kernel 4.19.20
lvm2 2.03.02
e2fsprogs 1.44.5
Quoted 4 lines · Ingo Franzki
Hmm, maybe the size of the volume plays a role as Bernd has pointed out. ext4 may use -b 4K by default on larger devices. Once the FS uses 4K block anyway you wont see the problem.
Use tune2fs -l <device> after you created the file system and check if it is using 4K blocks on your 512/512 device. If so, then you won't see the problem when moved to a 4K block size device.
I confirm that tune2fs reports 4096 block size for the 1 GB ext4 filesystem I've used.
I've also verified what Bernd said: mkfs.ext4 still use 4096 block size for a +512M partition, but use 1024 for +500M.
As suggested by Stuart, I also made a test using a 4k loop device and pvmoving the LV into it. As you expected, no data corruption. To do it I've recreated the same setup ad yesterday: /dev/mapper/vgtest-lvol0 on /dev/sdb4, a 512/512 disk, with some data on it. Then:
# fallocate -l 10G testdisk.img # losetup -f -L -P -b 4096 testdisk.img # pvcreate /dev/loop0 # vgextend vgtest /dev/loop0 # pvmove /dev/sdb4 /dev/loop0 # fsck.ext4 -f /dev/mapper/vgtest-lvol0
While I was there, out of curiosity, I've created an ext4 filesystem on a <500MB LV (block size = 1024) and I've tried pvmoving data from the 512/512 disk to 512/4096, then to the 4096/4096 loop device. New partitions and a new VG was used for that.
The setup:/dev/sdb5: 512/512/dev/sdc2: 512/4096/dev/loop0 4096/4096
# blockdev -v --getss --getpbsz --getbsz /dev/sdb get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096
# blockdev -v --getss --getpbsz --getbsz /dev/sdc get logical block (sector) size: 512 get physical block (sector) size: 4096 get blocksize: 4096
# blockdev -v --getss --getpbsz --getbsz /dev/loop0 get logical block (sector) size: 4096 get physical block (sector) size: 4096 get blocksize: 4096
# pvcreate /dev/sdb5 # vgcreate vgtest2 /dev/sdb5 # lvcreate -L 400M vgtest2 /dev/sdb5 # mkfs.ext4 /dev/mapper/vgtest2-lvol0
# tune2fs -l /dev/mapper/vgtest2-lvol0 [...] Block size: 1024 [...]
# mount /dev/mapper/vgtest2-lvol0 /media/test # cp -a SOMEDATA /media/test/ # umount /media/test # fsck.ext4 -f /dev/mapper/vgtest2-lvol0
Now I've moved data from the 512/512 to the 512/4096 disk:
# pvcreate /dev/sdc2 # vgextend vgtest2 /dev/sdc2 # pvmove /dev/sdb5 /dev/sdc2 # fsck.ext4 -f /dev/mapper/vgtest2-lvol0
No error reported.
Now I've moved data to the 4096/4096 loop device:
# pvcreate /dev/loop0 # vgextend vgtest2 /dev/loop0 # pvmove /dev/sdc2 /dev/loop0 # fsck.ext4 -f /dev/mapper/vgtest2-lvol0
Still no data corruption.
Cesare.
Quoted 5 lines · Ingo Franzki
Yes that should work:
# losetup -fP loopbackfile.img --sector-size 4096 # blockdev --getpbsz /dev/loop0 4096
-----
Something I noticed that is troublesome. When I first got my 4K sectory size disks, one of the numbers in the kernel listed it as 4K, now ~4-5 years later, that difference no longer appears in the kernel! To me, this seems a bit unwise -- as even though these disks are "512_e_" (a 512 byte sector size can be emulated but writes will usually or often take a minimum of 2 revolutions so that the rest of the 4k sector can be read, have 512-bytes
updated, then write the whole 4k sector size out. In otherwords -- hideous performance on writes.
Now the RAID controller that manages this disk also has info on the disks -- and
there it correctly shows the 4k sector size. So am wondering what in the kernel
"broke"(?) to disable detection & display of its native 4k sector size.
Anyway, I don't think it should cause problems unless I reformat or move the partitions, but it would be nice to know the various disk software knows not to try anything smaller than a 4k sector size.
Quoted 76 lines · Cesare Leonardi
Hello Ingo, I've made several tests but I was unable to trigger any filesystem corruption. Maybe the trouble you encountered are specific to encrypted device?
Yesterday and today I've used:
Debian unstable
kernel 4.19.20
lvm2 2.03.02
e2fsprogs 1.44.5Quoted 1 line · Ingo Franzki
Hmm, maybe the size of the volume plays a role as Bernd has pointed out.
ext4 may use -b 4K by default on larger devices.
Quoted 3 lines
Once the FS uses 4K block anyway you wont see the problem.
Use tune2fs -l <device> after you created the file system and check if
it is using 4K blocks on your 512/512 device. If so, then you won't see the problem when moved to a 4K block size device.
I confirm that tune2fs reports 4096 block size for the 1 GB ext4 filesystem I've used.
I've also verified what Bernd said: mkfs.ext4 still use 4096 block size for a +512M partition, but use 1024 for +500M.As suggested by Stuart, I also made a test using a 4k loop device and pvmoving the LV into it. As you expected, no data corruption. To do it I've recreated the same setup ad yesterday:
/dev/mapper/vgtest-lvol0on/dev/sdb4, a 512/512 disk, with some data on it. Then:# fallocate -l 10G testdisk.img # losetup -f -L -P -b 4096 testdisk.img # pvcreate /dev/loop0 # vgextend vgtest /dev/loop0 # pvmove /dev/sdb4 /dev/loop0 # fsck.ext4 -f /dev/mapper/vgtest-lvol0While I was there, out of curiosity, I've created an ext4 filesystem on a <500MB LV (block size = 1024) and I've tried pvmoving data from the 512/512 disk to 512/4096, then to the 4096/4096 loop device. New partitions and a new VG was used for that.
The setup:
/dev/sdb5: 512/512/dev/sdc2: 512/4096/dev/loop04096/4096# blockdev -v --getss --getpbsz --getbsz /dev/sdb get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096# blockdev -v --getss --getpbsz --getbsz /dev/sdc get logical block (sector) size: 512 get physical block (sector) size: 4096 get blocksize: 4096# blockdev -v --getss --getpbsz --getbsz /dev/loop0 get logical block (sector) size: 4096 get physical block (sector) size: 4096 get blocksize: 4096# pvcreate /dev/sdb5 # vgcreate vgtest2 /dev/sdb5 # lvcreate -L 400M vgtest2 /dev/sdb5 # mkfs.ext4 /dev/mapper/vgtest2-lvol0# tune2fs -l /dev/mapper/vgtest2-lvol0 [...] Block size: 1024 [...]# mount /dev/mapper/vgtest2-lvol0 /media/test # cp -a SOMEDATA /media/test/ # umount /media/test # fsck.ext4 -f /dev/mapper/vgtest2-lvol0
Quoted 8 lines
Now I've moved data from the 512/512 to the 512/4096 disk:
# pvcreate /dev/sdc2 # vgextend vgtest2 /dev/sdc2 # pvmove /dev/sdb5 /dev/sdc2 # fsck.ext4 -f /dev/mapper/vgtest2-lvol0No error reported.
Did you try to mount the lv after the pvmove?
Quoted 8 lines
Now I've moved data to the 4096/4096 loop device:
# pvcreate /dev/loop0 # vgextend vgtest2 /dev/loop0 # pvmove /dev/sdc2 /dev/loop0 # fsck.ext4 -f /dev/mapper/vgtest2-lvol0Still no data corruption.
I can reproduce this without moving data, just by extending vg with 4k device,
and then extending lv to use both devices.
Here is what I tested:
# truncate -s 500m disk1 # truncate -s 500m disk2 # losetup -f disk1 --sector-size 512 --show /dev/loop2 # losetup -f disk2 --sector-size 4096 --show /dev/loop3
# pvcreate /dev/loop2 Physical volume "/dev/loop2" successfully created. # pvcreate /dev/loop3 Physical volume "/dev/loop3" successfully created. # vgcreate test /dev/loop2 Volume group "test" successfully created # lvcreate -L400m -n lv1 test Logical volume "lv1" created.
# mkfs.xfs /dev/test/lv1
meta-data=/dev/test/lv1 isize=512 agcount=4, agsize=25600 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=0, rmapbt=0,
reflink=0
data = bsize=4096 blocks=102400, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=855, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0# mkdir /tmp/mnt # mount /dev/test/lv1 /tmp/mnt # vgextend test /dev/loop3 Volume group "test" successfully extended # lvextend -L+400m test/lv1 Size of logical volume test/lv1 changed from 400.00 MiB (100 extents) to 800.00 MiB (200 extents). Logical volume test/lv1 successfully resized. # umount /tmp/mnt
# mount /dev/test/lv1 /tmp/mnt mount: /tmp/mnt: mount(2) system call failed: Function not implemented.
From journalctl:
Mar 02 21:52:53 lean.local kernel: XFS (dm-7): Unmounting Filesystem Mar 02 21:53:01 lean.local kernel: XFS (dm-7): device supports 4096 byte
sectors (not 512)
I also tried the same with ext4:
(same disks/vg/lv setup as above)
# mkfs.ext4 /dev/test/lv1 mke2fs 1.44.2 (14-May-2018) Discarding device blocks: done Creating filesystem with 409600 1k blocks and 102400 inodes Filesystem UUID: 9283880e-ee89-4d79-9c29-41f4af98f894 Superblock backups stored on blocks: 8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409
Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
# vgextend test /dev/loop3 Volume group "test" successfully extended # lvextend -L+400 test/lv1 Size of logical volume test/lv1 changed from 400.00 MiB (100 extents) to 800.00 MiB (200 extents). Logical volume test/lv1 successfully resized.
# mount /dev/test/lv1 /tmp/mnt mount: /tmp/mnt: wrong fs type, bad option, bad superblock on /dev/mapper/test-lv1, missing codepage or helper program, or other error.
From journalctl:
Mar 02 22:06:09 lean.local kernel: EXT4-fs (dm-7): bad block size 1024
Now same with pvmove:
(same setup as above, using xfs)
# mount /dev/test/lv1 /tmp/mnt # dd if=/dev/urandom bs=8M count=1 of=/tmp/mnt/data # vgextend test /dev/loop3 Physical volume "/dev/loop3" successfully created. Volume group "test" successfully extended
# pvmove -v /dev/loop2 /dev/loop3
Cluster mirror log daemon is not running.
Wiping internal VG cache
Wiping cache of LVM-capable devices
Archiving volume group "test" metadata (seqno 3).
Creating logical volume pvmove0
Moving 100 extents of logical volume test/lv1.
activation/volume_list configuration setting not defined: Checking only
host tags for test/lv1.
Creating test-pvmove0
Loading table for test-pvmove0 (253:8).
Loading table for test-lv1 (253:7).
Suspending test-lv1 (253:7) with device flush
Resuming test-pvmove0 (253:8).
Resuming test-lv1 (253:7).
Creating volume group backup "/etc/lvm/backup/test" (seqno 4).
activation/volume_list configuration setting not defined: Checking only
host tags for test/pvmove0.
Checking progress before waiting every 15 seconds.
/dev/loop2: Moved: 15.00%
/dev/loop2: Moved: 100.00%
Polling finished successfully.
# umount /tmp/mnt
# mount /dev/test/lv1 /tmp/mnt
mount: /tmp/mnt: mount(2) system call failed: Function not implemented.From journalctl:
Mar 02 22:20:36 lean.local kernel: XFS (dm-7): device supports 4096 byte
sectors (not 512)
Tested on Fedora 28 with:
kernel-4.20.5-100.fc28.x86_64
lvm2-2.02.177-5.fc28.x86_64
Nir
Quoted 34 lines · Cesare Leonardi
Hello Ingo, I've made several tests but I was unable to trigger any filesystem corruption. Maybe the trouble you encountered are specific to encrypted device?
Yesterday and today I've used:
Debian unstable
kernel 4.19.20
lvm2 2.03.02
e2fsprogs 1.44.5Quoted 4 lines · Ingo Franzki
Hmm, maybe the size of the volume plays a role as Bernd has pointed out. ext4 may use -b 4K by default on larger devices. Once the FS uses 4K block anyway you wont see the problem.
Use tune2fs -l <device> after you created the file system and check if it is using 4K blocks on your 512/512 device. If so, then you won't see the problem when moved to a 4K block size device.
I confirm that tune2fs reports 4096 block size for the 1 GB ext4 filesystem I've used.
I've also verified what Bernd said: mkfs.ext4 still use 4096 block size for a +512M partition, but use 1024 for +500M.As suggested by Stuart, I also made a test using a 4k loop device and pvmoving the LV into it. As you expected, no data corruption. To do it I've recreated the same setup ad yesterday:
/dev/mapper/vgtest-lvol0on/dev/sdb4, a 512/512 disk, with some data on it. Then:# fallocate -l 10G testdisk.img # losetup -f -L -P -b 4096 testdisk.img # pvcreate /dev/loop0 # vgextend vgtest /dev/loop0 # pvmove /dev/sdb4 /dev/loop0 # fsck.ext4 -f /dev/mapper/vgtest-lvol0While I was there, out of curiosity, I've created an ext4 filesystem on a <500MB LV (block size = 1024) and I've tried pvmoving data from the 512/512 disk to 512/4096, then to the 4096/4096 loop device. New partitions and a new VG was used for that.
The setup:
/dev/sdb5: 512/512/dev/sdc2: 512/4096/dev/loop04096/4096
Quoted 4 lines
# blockdev -v --getss --getpbsz --getbsz /dev/sdb get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096
You display the physical block size of /dev/sdb here, but you use /dev/sdb5 later on. Not sure if this makes a difference ....
Quoted 5 lines
# blockdev -v --getss --getpbsz --getbsz /dev/sdc get logical block (sector) size: 512 get physical block (sector) size: 4096 get blocksize: 4096
Again, you display the physical block size of /dev/sdc here, but you use /dev/sdc2 later on.
Quoted 21 lines
# blockdev -v --getss --getpbsz --getbsz /dev/loop0 get logical block (sector) size: 4096 get physical block (sector) size: 4096 get blocksize: 4096# pvcreate /dev/sdb5 # vgcreate vgtest2 /dev/sdb5 # lvcreate -L 400M vgtest2 /dev/sdb5 # mkfs.ext4 /dev/mapper/vgtest2-lvol0# tune2fs -l /dev/mapper/vgtest2-lvol0 [...] Block size: 1024 [...]# mount /dev/mapper/vgtest2-lvol0 /media/test # cp -a SOMEDATA /media/test/ # umount /media/test # fsck.ext4 -f /dev/mapper/vgtest2-lvol0
Quoted 5 lines
Now I've moved data from the 512/512 to the 512/4096 disk:
# pvcreate /dev/sdc2 # vgextend vgtest2 /dev/sdc2 # pvmove /dev/sdb5 /dev/sdc2 # fsck.ext4 -f /dev/mapper/vgtest2-lvol0
Please note that fsck.ext4 does not seem to detect this kind of corruption.
In my case fsck.ext4 reported that the FS would be clean (!), but a mount count not mount it anymore...
Do a 'pvs' command here, this should show some error messages.
Quoted 7 lines
No error reported.
Now I've moved data to the 4096/4096 loop device:# pvcreate /dev/loop0 # vgextend vgtest2 /dev/loop0 # pvmove /dev/sdc2 /dev/loop0 # fsck.ext4 -f /dev/mapper/vgtest2-lvol0
Again, fsck did not show the corruption for me, but pvs did show error messages.
Quoted 12 lines
Still no data corruption.
Cesare.
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Signature · 10 lines
Ingo Franzki eMail: [email protected] Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/
Quoted 6 lines · Ingo Franzki
Quoted 4 lines
# blockdev -v --getss --getpbsz --getbsz /dev/sdb get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096You display the physical block size of
/dev/sdbhere, but you use/dev/sdb5later on. Not sure if this makes a difference ....
I thought that was the right thing to do, as they are disk parameters. At least the first two, for the last I'm not sure. However the output looks the same:
# blockdev -v --getss --getpbsz --getbsz /dev/sdb5 get logical block (sector) size: 512 get physical block (sector) size: 512 get blocksize: 4096
# blockdev -v --getss --getpbsz --getbsz /dev/sdc2 get logical block (sector) size: 512 get physical block (sector) size: 4096 get blocksize: 4096
Quoted 4 lines
Please note that fsck.ext4 does not seem to detect this kind of corruption.
In my case fsck.ext4 reported that the FS would be clean (!), but a mount count not mount it anymore...Do a 'pvs' command here, this should show some error messages.
Uh, I didn't really expect that such corruption could pass unoticed an fsck.ext4 check. During my tests, initially I surely tried to mount the filesystem and I did ls on it but it's possible that after some steps I only trusted fsck.
Today I repeated all the tests and indeed in one case the mount failed: after pvmoving from the 512/4096 disk to the 4096/4096 disk, with the LV ext4 using 1024 block size.
Here is what I've tested:/dev/sdb: SSD with 512/512 sector size /dev/sdc: mechanical disk with 512/4096 sector size /dev/loop0: emulated disk with 4096/4096 sector size
TEST 1
VG vgtest1: /dev/sdb4 /dev/sdc1 /dev/loop0p1
LV vgtest1-lvol0: filesystem ext4 with 4096 block size pvmove ext4-4096:
- from /dev/sdb4 (512/512) to /dev/sdc1 (512/4096): ok
- from /dev/sdc1 (512/4096) to /dev/loop0p1 (4096/4096): ok
TEST 2
VG vgtest2: /dev/sdb5 /dev/sdc2 /dev/loop0p2
LV vgtest2-lvol0: filesystem ext4 with 1024 block size pvmove ext4-1024:
- from /dev/sdb5 (512/512) to /dev/sdc2 (512/4096): ok
- from /dev/sdc2 (512/4096) to /dev/loop0p2 (4096/4096): fail
Here the outputs of the failed test:
=======================
# pvmove /dev/sdc2 /dev/loop0p2 /dev/sdc2: Moved: 9,00% /dev/sdc2: Moved: 100,00%
# mount /dev/mapper/vgtest2-lvol0 /media/test/ mount: /media/test: wrong fs type, bad option, bad superblock on /dev/mapper/vgtest2-lvol0, missing codepage or helper program, or other error.
# fsck.ext4 -f /dev/mapper/vgtest2-lvol0 e2fsck 1.44.5 (15-Dec-2018) Pass 1: Checking inodes, blocks, and sizes Pass 2: Checking directory structure Pass 3: Checking directory connectivity Pass 4: Checking reference counts Pass 5: Checking group summary information /dev/mapper/vgtest2-lvol0: 35/102400 files (17.1% non-contiguous), 304877/409600 blocks =======================
The error happened where you guys expected. And also for me fsck showed no errors.
But doesn't look like a filesystem corruption: if you pvmove back the data, it will become readable again:
# pvmove /dev/loop0p2 /dev/sdc2 /dev/loop0p2: Moved: 1,00% /dev/loop0p2: Moved: 100,00% # mount /dev/mapper/vgtest2-lvol0 /media/test/ # ls /media/test/ epson hp kerio lost+found
And also notice that the pvmove that generated the unreadable filesystem starts with an unusual high percentage (9%). In all other test it start from 0% or a small number near 1%. This happened in more that one case.
Cesare.
Quoted 11 lines · Nir Soffer
# mkfs.xfs /dev/test/lv1 meta-data=/dev/test/lv1 isize=512 agcount=4, agsize=25600 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 data = bsize=4096 blocks=102400, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=855, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0
Has the problem here the same root as for ext4? I guess sectsz should be >=4096 to avoid troubles, isn't it?
Just to draw some conlusion, could we say that currently, if we are going to move data around with LVM, it's better to check that the filesystem is using a block size >= than "blockdev --getbsz DESTINATIONDEVICE"? At least with ext4 and xfs.
Something that couldn't be true with really small devices (< 500 MB).
Is there already an open bug regarding the problem discussed in this thread?
Cesare.
Quoted 22 lines · Cesare Leonardi
Quoted 2 lines · Nir Soffer
# mkfs.xfs /dev/test/lv1 meta-data=/dev/test/lv1 isize=512 agcount=4, agsize=25600blks
Quoted 9 lines
= sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 data = bsize=4096 blocks=102400, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=855, version=2= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0Has the problem here the same root as for ext4? I guess sectsz should be
Quoted 1 line
=4096 to avoid troubles, isn't it?
Just to draw some conlusion, could we say that currently, if we are going to move data around with LVM, it's better to check that the filesystem is using a block size >= than "blockdev --getbsz DESTINATIONDEVICE"? At least with ext4 and xfs.
Quoted 5 lines
Something that couldn't be true with really small devices (< 500 MB).
Is there already an open bug regarding the problem discussed in this thread?
There is this bug about lvextend:
https://bugzilla.redhat.com/1669751
And this old bug from 2011, discussing mixing PVs with different block size. Comment 2 is very clear about this issue:
https://bugzilla.redhat.com/732980#c2
Nir
Quoted 1 line · Cesare Leonardi
Today I repeated all the tests and indeed in one case the mount failed: after pvmoving from the 512/4096 disk to the 4096/4096 disk, with the LV ext4 using 1024 block size.
...
Quoted 3 lines
The error happened where you guys expected. And also for me fsck showed no errors.
But doesn't look like a filesystem corruption: if you pvmove back the data, it will become readable again:
...
THAT is a crucial observation. It's not an LVM bug, but the filesystem trying to read 1024 bytes on a 4096 device. I suspect it could also happen with an unaligned filesystem on a 4096 device.
Signature · 3 lines
Stuart D. Gathman <[email protected]> "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.
Quoted 11 lines · Stuart D. Gathman
Quoted 1 line · Cesare Leonardi
Today I repeated all the tests and indeed in one case the mount failed: after pvmoving from the 512/4096 disk to the 4096/4096 disk, with the LV ext4 using 1024 block size.
...
Quoted 3 lines
The error happened where you guys expected. And also for me fsck showed no errors.
But doesn't look like a filesystem corruption: if you pvmove back the data, it will become readable again:
...
THAT is a crucial observation. It's not an LVM bug, but the filesystem trying to read 1024 bytes on a 4096 device.
Yes that's probably the reason. Nevertheless, its not really the FS's fault, since it was moved by LVM to a 4069 device. The FS does not know anything about the move, so it reads in the block size it was created with (1024 in this case).
I still think LVM should prevent one from mixing devices with different physical block sizes, or at least warn when pvmoving or lvextending onto a PV with a larger block size, since this can cause trouble.
Quoted 3 lines
I suspect it could also
happen with an unaligned filesystem on a 4096 device.
Signature · 10 lines
Ingo Franzki eMail: [email protected] Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/
Quoted 30 lines · Nir Soffer
Quoted 22 lines · Cesare Leonardi
Quoted 2 lines · Nir Soffer
# mkfs.xfs /dev/test/lv1 meta-data=/dev/test/lv1 isize=512 agcount=4, agsize=25600blks
Quoted 9 lines
= sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 data = bsize=4096 blocks=102400, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=855, version=2= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0Has the problem here the same root as for ext4? I guess sectsz should be
Quoted 1 line
=4096 to avoid troubles, isn't it?
Just to draw some conlusion, could we say that currently, if we are going to move data around with LVM, it's better to check that the filesystem is using a block size >= than "blockdev --getbsz DESTINATIONDEVICE"? At least with ext4 and xfs.
Quoted 5 lines
Something that couldn't be true with really small devices (< 500 MB).
Is there already an open bug regarding the problem discussed in this thread?
Quoted 6 lines
There is this bug about lvextend:
https://bugzilla.redhat.com/1669751And this old bug from 2011, discussing mixing PVs with different block size. Comment 2 is very clear about this issue:
https://bugzilla.redhat.com/732980#c2
I don't have access to that one, can you cite comment 2 ?
Quoted 3 lines
Nir
Signature · 10 lines
Ingo Franzki eMail: [email protected] Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/
Hello.
Quoted 7 lines
Quoted 2 lines
THAT is a crucial observation. It's not an LVM bug, but the filesystem trying to read 1024 bytes on a 4096 device.
Yes that's probably the reason. Nevertheless, its not really the FS's fault, since it was moved by LVM to a 4069 device. The FS does not know anything about the move, so it reads in the block size it was created with (1024 in this case).
I still think LVM should prevent one from mixing devices with different physical block sizes, or at least warn when pvmoving or lvextending onto a PV with a larger block size, since this can cause trouble.
In this case, "dd" tool and others should prevent too.
Because after:
dd if=/dev/DiskWith512block bs=4096 of=/dev/DiskWith4Kblock
You couldn't mount the "/dev/DiskWith4Kblock" with the same error ;) /dev/DiskWith512block has ext4 fs with 1k block.
P.S.
LVM,dd .. are low level tools and doesn't know about hi level anything. And in the your case and others cases can't know. You should test(if you need) the block size with other tools before moving or copying. Not a lvm bug.
Thank you.
Quoted 11 lines · Ilia Zykov
Hello.
Quoted 7 lines
Quoted 2 lines
THAT is a crucial observation. It's not an LVM bug, but the filesystem trying to read 1024 bytes on a 4096 device.
Yes that's probably the reason. Nevertheless, its not really the FS's fault, since it was moved by LVM to a 4069 device. The FS does not know anything about the move, so it reads in the block size it was created with (1024 in this case).
I still think LVM should prevent one from mixing devices with different physical block sizes, or at least warn when pvmoving or lvextending onto a PV with a larger block size, since this can cause trouble.
In this case, "dd" tool and others should prevent too.
Well, no, its LVM's pvmove who moves the data from a 512 block size to 4096 block size device. So its not dd that's causing the problem, but pvmove.
Quoted 5 lines
Because after:
dd if=/dev/DiskWith512block bs=4096 of=/dev/DiskWith4Kblock
Quoted 8 lines
You couldn't mount the "
/dev/DiskWith4Kblock" with the same error ;)/dev/DiskWith512blockhas ext4 fs with 1k block.
P.S.
LVM,dd .. are low level tools and doesn't know about hi level anything. And in the your case and others cases can't know. You should test(if you need) the block size with other tools before moving or copying. Not a lvm bug.
Well, maybe not a bug, but I would still like to see that LVM's pvmove/lvextend and/or vgextend issues at least a warning when you have mixed block sizes.
LVM knows the block sizes of the PVs it manages and it also knows when it changes the block size of an LV due to a pvmove or lvextend. So it can issue a warning (better a confirmation prompt) when this happens.
Quoted 2 lines
Thank you.
Signature · 10 lines
Ingo Franzki eMail: [email protected] Tel: ++49 (0)7031-16-4648 Fax: ++49 (0)7031-16-3456 Linux on IBM Z Development, Schoenaicher Str. 220, 71032 Boeblingen, Germany IBM Deutschland Research & Development GmbH / Vorsitzender des Aufsichtsrats: Matthias Hartmann Geschäftsführung: Dirk Wittkopp Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 IBM DATA Privacy Statement: https://www.ibm.com/privacy/us/en/
Quoted 20 lines · Ilia Zykov
Hello.
Quoted 3 lines
Quoted 2 lines
THAT is a crucial observation. It's not an LVM bug, but the filesystem trying to read 1024 bytes on a 4096 device.
Yes that's probably the reason. Nevertheless, its not really the FS's
fault, since it was moved by LVM to a 4069 device.
Quoted 1 line
The FS does not know anything about the move, so it reads in the block
size it was created with (1024 in this case).
Quoted 2 lines
I still think LVM should prevent one from mixing devices with different
physical block sizes, or at least warn when pvmoving or lvextending onto a PV with a larger block size, since this can cause trouble.
Quoted 1 line
In this case, "dd" tool and others should prevent too.
Because after:
dd if=/dev/DiskWith512block bs=4096 of=/dev/DiskWith4Kblock
Quoted 9 lines
You couldn't mount the "
/dev/DiskWith4Kblock" with the same error ;)/dev/DiskWith512blockhas ext4 fs with 1k block.
P.S.
LVM,dd .. are low level tools and doesn't know about hi level anything. And in the your case and others cases can't know. You should test(if you need) the block size with other tools before moving or copying. Not a lvm bug.
I don't this way of thinking is useful. If we go in this way, then write() should not
let you write data, and later maybe the disk controller should avoid this?
LVM is not a low level tool like dd. It is high level tool for managing device mapper,
and providing high level tools to create user level abstractions. We can expect it
to prevent system administrator from doing the wrong thing.
Maybe LVM should let you mix PVs with different logical block size, but it should
require --force.
David, what do you think?
Quoted 10 lines · Nir Soffer
I don't this way of thinking is useful. If we go in this way, then
write()should not
let you write data, and later maybe the disk controller should avoid this?LVM is not a low level tool like dd. It is high level tool for managing device mapper,
and providing high level tools to create user level abstractions. We can expect it
to prevent system administrator from doing the wrong thing.
Quoted 5 lines
Maybe LVM should let you mix PVs with different logical block size, but it should
require --force.David, what do you think?
LVM needs to fix this, your solution sounds like the right one.
Quoted 6 lines · David Teigland
Quoted 3 lines · Nir Soffer
Maybe LVM should let you mix PVs with different logical block size, but it should
require --force.LVM needs to fix this, your solution sounds like the right one.
Also, since nearly every modern device device has a physical block size of
4k or more, and even when the logical block size is (emulated) 512,
performance degradation occurs with smaller filesystem blocks, then the savvy admin should ensure that all filesystem have a min of 4k block size - except in special circustances.
Signature · 3 lines
Stuart D. Gathman <[email protected]> "Confutatis maledictis, flamis acribus addictis" - background song for a Microsoft sponsored "Where do you want to go from here?" commercial.