Skip to contentSkip to conversation

lvmcache with vdo - inconsistent block size

linux-lvm Gionatan Danti 2 participants 6 messages Sep 14, 2020 – Sep 17, 2020

lore
Share & tools
lore
#0 lore raw

Hi all,
I am testing lvmcache with VDO and I have issue with devices block size.

The big & slow VDO device is on top of a 4-disk MD RAID 10 device (itself on top of dm-integrity). Over the VDO device I created a thinpool and a thinvol [1]. When adding the cache device to the volume group via vgextend, I get an error stating "Devices have inconsistent logical block sizes (4096 and 512)." [2]

Now, I know why the error shows and what i means. However, I don't know how to force the cache device to act as a 4k sector device, and/if this is really required to cache a VDO device.

My current workaround is to set VDO with --emulate512=enabled, but this can be suboptimal and it is not recommended.

Any idea on what I am doing wrong?
Thanks.

[1] Output from lsblk -t:
[root@singularity dev]# lsblk -t | cut -c1-73
NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC
sda 0 512 0 512 512

├─sda1                                0     512       0     512     512
├─sda2                                0     512       0     512     512
│ ├─system-root                       0     512       0     512     512
│ └─system-swap                       0     512       0     512     512
├─sda3                                0     512       0     512     512
└─sda4                                0     512       0     512     512
sdb                                   0     512       0     512     512
└─sdb                                 0     512       0     512     512
  └─md200                             0  524288 1048576     512     512
    └─vdo1                            0    4096    4096    4096    4096
      ├─vg_kvm-thinpool_tmeta         0    4096    4096    4096    4096
      │ └─vg_kvm-thinpool-tpool       0 1048576 1048576    4096    4096
      │   ├─vg_kvm-thinpool           0 1048576 1048576    4096    4096
      │   └─vg_kvm-thinvol            0 1048576 1048576    4096    4096
      └─vg_kvm-thinpool_tdata         0    4096    4096    4096    4096
        └─vg_kvm-thinpool-tpool       0 1048576 1048576    4096    4096
          ├─vg_kvm-thinpool           0 1048576 1048576    4096    4096
          └─vg_kvm-thinvol            0 1048576 1048576    4096    4096
sdc                                   0     512       0     512     512
└─sdc                                 0     512       0     512     512
  └─md200                             0  524288 1048576     512     512
    └─vdo1                            0    4096    4096    4096    4096
      ├─vg_kvm-thinpool_tmeta         0    4096    4096    4096    4096
      │ └─vg_kvm-thinpool-tpool       0 1048576 1048576    4096    4096
      │   ├─vg_kvm-thinpool           0 1048576 1048576    4096    4096
      │   └─vg_kvm-thinvol            0 1048576 1048576    4096    4096
      └─vg_kvm-thinpool_tdata         0    4096    4096    4096    4096
        └─vg_kvm-thinpool-tpool       0 1048576 1048576    4096    4096
          ├─vg_kvm-thinpool           0 1048576 1048576    4096    4096
          └─vg_kvm-thinvol            0 1048576 1048576    4096    4096
sdd                                   0     512       0     512     512
└─sdd                                 0     512       0     512     512
  └─md200                             0  524288 1048576     512     512
    └─vdo1                            0    4096    4096    4096    4096
      ├─vg_kvm-thinpool_tmeta         0    4096    4096    4096    4096
      │ └─vg_kvm-thinpool-tpool       0 1048576 1048576    4096    4096
      │   ├─vg_kvm-thinpool           0 1048576 1048576    4096    4096
      │   └─vg_kvm-thinvol            0 1048576 1048576    4096    4096
      └─vg_kvm-thinpool_tdata         0    4096    4096    4096    4096
        └─vg_kvm-thinpool-tpool       0 1048576 1048576    4096    4096
          ├─vg_kvm-thinpool           0 1048576 1048576    4096    4096
          └─vg_kvm-thinvol            0 1048576 1048576    4096    4096
sde                                   0     512       0     512     512
└─sde                                 0     512       0     512     512
  └─md200                             0  524288 1048576     512     512
    └─vdo1                            0    4096    4096    4096    4096
      ├─vg_kvm-thinpool_tmeta         0    4096    4096    4096    4096
      │ └─vg_kvm-thinpool-tpool       0 1048576 1048576    4096    4096
      │   ├─vg_kvm-thinpool           0 1048576 1048576    4096    4096
      │   └─vg_kvm-thinvol            0 1048576 1048576    4096    4096
      └─vg_kvm-thinpool_tdata         0    4096    4096    4096    4096
        └─vg_kvm-thinpool-tpool       0 1048576 1048576    4096    4096
          ├─vg_kvm-thinpool           0 1048576 1048576    4096    4096
          └─vg_kvm-thinvol            0 1048576 1048576    4096    4096

[2] [root@singularity dev]# vgextend vg_kvm /dev/sda4 Devices have inconsistent logical block sizes (4096 and 512).

Signature · 5 lines
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: [email protected] - [email protected]
GPG public key ID: FF5F32A8
#1 lore raw

Dne 14. 09. 20 v 23:44 Gionatan Danti napsal(a):

Quoted 10 lines

Hi all,
I am testing lvmcache with VDO and I have issue with devices block size.

The big & slow VDO device is on top of a 4-disk MD RAID 10 device (itself on top of dm-integrity). Over the VDO device I created a thinpool and a thinvol [1]. When adding the cache device to the volume group via vgextend, I get an error stating "Devices have inconsistent logical block sizes (4096 and 512)." [2]

Now, I know why the error shows and what i means. However, I don't know how to force the cache device to act as a 4k sector device, and/if this is really required to cache a VDO device.

My current workaround is to set VDO with --emulate512=enabled, but this can be suboptimal and it is not recommended.

Any idea on what I am doing wrong?

Hi

LVM currently does not support mixing devices of different sector sizes within a single VG as it brings lot of troubles we have not yet clear vision what to do with all of them.

Also this combination of provisioned devices is not advised - since you are combining 2 kind of devices on top of each other and it can be a big problem to solve recovery case.

On lvm2 side we do not allow to use 'VDO LV' as backend for thin-pool device.

So ATM it's on a user to solve all the possible scenarios that may appear on such device stack.

Zdenek

#2 lore raw

Il 2020-09-15 20:34 Zdenek Kabelac ha scritto:

Quoted 10 lines

Dne 14. 09. 20 v 23:44 Gionatan Danti napsal(a):

Quoted 9 lines

Hi all,
I am testing lvmcache with VDO and I have issue with devices block size.

The big & slow VDO device is on top of a 4-disk MD RAID 10 device (itself on top of dm-integrity). Over the VDO device I created a thinpool and a thinvol [1]. When adding the cache device to the volume group via vgextend, I get an error stating "Devices have inconsistent logical block sizes (4096 and 512)." [2]

Now, I know why the error shows and what i means. However, I don't know how to force the cache device to act as a 4k sector device, and/if this is really required to cache a VDO device.

My current workaround is to set VDO with --emulate512=enabled, but this can be suboptimal and it is not recommended.

Quoted 7 lines
Quoted 1 line

Any idea on what I am doing wrong?

Hi

LVM currently does not support mixing devices of different sector sizes within a single VG as it brings lot of troubles we have not yet clear vision what to do with all of them.

Hi Zdenek, yes, I understand. What surprised me is that lvmvdo *can* be combined with caching, and it does not suffer from this issue. Can you elaborate on why it works in this case?

Quoted 4 lines

Also this combination of provisioned devices is not advised - since you are combining 2 kind of devices on top of each other and it can be a big problem
to solve recovery case.

True.

Quoted 1 line

On lvm2 side we do not allow to use 'VDO LV' as backend for thin-pool device.

I noticed it. However, from what I can read on RedHat docs, thinpool over VDO device should be perfectly fine (the other way around, not so much).

Quoted 4 lines

So ATM it's on a user to solve all the possible scenarios that may appear on such device stack.

Zdenek

Thanks.

Signature · 5 lines
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: [email protected] - [email protected]
GPG public key ID: FF5F32A8
#3 lore raw

Dne 16. 09. 20 v 0:32 Gionatan Danti napsal(a):

Quoted 26 lines

Il 2020-09-15 20:34 Zdenek Kabelac ha scritto:

Quoted 10 lines

Dne 14. 09. 20 v 23:44 Gionatan Danti napsal(a):

Quoted 9 lines

Hi all,
I am testing lvmcache with VDO and I have issue with devices block size.

The big & slow VDO device is on top of a 4-disk MD RAID 10 device (itself on top of dm-integrity). Over the VDO device I created a thinpool and a thinvol [1]. When adding the cache device to the volume group via vgextend, I get an error stating "Devices have inconsistent logical block sizes (4096 and 512)." [2]

Now, I know why the error shows and what i means. However, I don't know how to force the cache device to act as a 4k sector device, and/if this is really required to cache a VDO device.

My current workaround is to set VDO with --emulate512=enabled, but this can be suboptimal and it is not recommended.

Quoted 7 lines
Quoted 1 line

Any idea on what I am doing wrong?

Hi

LVM currently does not support mixing devices of different sector sizes within a single VG as it brings lot of troubles we have not yet clear vision what to do with all of them.

Hi Zdenek, yes, I understand. What surprised me is that lvmvdo *can* be combined with caching, and it does not suffer from this issue. Can you elaborate on why it works in this case?

Quoted 4 lines

Also this combination of provisioned devices is not advised - since you are combining 2 kind of devices on top of each other and it can be a big problem
to solve recovery case.

Quoted 6 lines

True.

Quoted 1 line

On lvm2 side we do not allow to use 'VDO LV' as backend for thin-pool device.

I noticed it. However, from what I can read on RedHat docs, thinpool over VDO device should be perfectly fine (the other way around, not so much).

You've most likely found the bug and this should be likely disable (and enabled only with some force option).

Problem is, when such device stack is used for XFS - where the 'geometry' changes, but for some other it's not a big issue (i.e. ext4).

So if you already hit some problem - feel free to open upstream BZ for this issue.

Zdenek

#4 lore raw

Il 2020-09-17 21:27 Zdenek Kabelac ha scritto:

Quoted 2 lines

You've most likely found the bug and this should be likely disable (and enabled only with some force option).

Hi Zdenek, I am not sure about what bug I found - can you be more explicit?

Quoted 3 lines

Problem is, when such device stack is used for XFS - where the 'geometry' changes, but for some other it's not a big issue (i.e. ext4).

Again, I am not sure about what we are speaking - generic thinlvm over VDO, or about the caching issue discussed before? Why this should be an issue for XFS vs EXT4? From RH docs, I can see VDO + thinlvm + XFS as being fully supported.

Quoted 2 lines

So if you already hit some problem - feel free to open upstream BZ for this issue.

I'll do, but I first need to understand the core issue better. Thanks.

Signature · 5 lines
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: [email protected] - [email protected]
GPG public key ID: FF5F32A8
#5 lore raw

Dne 17. 09. 20 v 23:41 Gionatan Danti napsal(a):

Quoted 9 lines

Il 2020-09-17 21:27 Zdenek Kabelac ha scritto:

Quoted 2 lines

You've most likely found the bug and this should be likely disable (and enabled only with some force option).

Hi Zdenek, I am not sure about what bug I found - can you be more explicit?

Quoted 3 lines

Problem is, when such device stack is used for XFS - where the 'geometry' changes, but for some other it's not a big issue (i.e. ext4).

XFS doesn't like when 4K device becomes 512b sector based.

Quoted 1 line

Again, I am not sure about what we are speaking - generic thinlvm over VDO, or about the caching issue discussed before? Why this should be an issue for XFS vs EXT4? From RH docs, I can see VDO + thinlvm + XFS as being fully supported.

Here you are using separate level of tooling - so you have separate dealing with errors on each device with its own tooling.

lvm2 is not yet capable to handle this stack on its own.

Quoted 2 lines

I'll do, but I first need to understand the core issue better.

Create XFS on 4K sector device - copy it on 512b and use it.

Zdenek