Skip to contentSkip to conversation

cache_check --clear-needs-check-flag does not clear needs_check flag?

linux-lvm Dennis Schridde 3 participants 9 messages Dec 6, 2020 – Dec 9, 2020

lore
Share & tools
lore
#0 lore raw

Hello!

A cached logical volume of mine cannot be activated anymore:

$ sudo vgchange -ay
   device-mapper: reload ioctl on   (253:6) failed: Invalid argument
   0 logical volume(s) in volume group "vg_ernie" now active

dmesg logs:

device-mapper: cache: 253:6: unable to switch cache to write mode until repaired.
device-mapper: cache: 253:6: switching cache to read-only mode
device-mapper: table: 253:6: cache: Unable to get write access to metadata, please check/repair metadata.
device-mapper: ioctl: error adding target to table

The code in question seems to be: https://github.com/torvalds/linux/blob/v5.8/
drivers/md/dm-cache-target.c#L957-L964

Hence I set out to check the cache and, if it is clean, clear the needs_check flag:

$ sudo lvchange -ay vg_ernie/lv_cache
Do you want to activate component LV in read-only mode? [y/n]: y
   Allowing activation of component LV.
$ sudo cache_check /dev/vg_ernie/lv_cache
examining superblock
examining mapping array
examining hint array
examining discard bitset
$ sudo cache_check --clear-needs-check-flag /dev/vg_ernie/lv_cache
examining superblock
examining mapping array
examining hint array
examining discard bitset
$ sudo lvchange -an vg_ernie/lv_cache

But the problem persists:

$ sudo vgchange -ay
   device-mapper: reload ioctl on   (253:6) failed: Invalid argument
   0 logical volume(s) in volume group "vg_ernie" now active

I tried again in read/write mode, in case that would make a difference / silently fail:

$ sudo lvchange -ay vg_ernie/lv_cache
Do you want to activate component LV in read-only mode? [y/n]: n
$ sudo cache_check --clear-needs-check-flag /dev/vg_ernie/lv_cache
examining superblock
examining mapping array
examining hint array
examining discard bitset
$ sudo lvchange -an vg_ernie/lv_cache

With the same results:

$ sudo vgchange -ay
   device-mapper: reload ioctl on   (253:6) failed: Invalid argument
   0 logical volume(s) in volume group "vg_ernie" now active

A bit puzzling is that the status of the needs_check flag appears to be "unknown":

$ sudo lvs -a -o +lv_check_needed
   LV                        VG          Attr          LSize    Pool
Origin                  Data%   Meta%   Move Log Cpy%Sync Convert CheckNeeded
   [lv_cache]            vg_ernie CRi-a-C--- 232.88g
unknown
   lv_system             vg_ernie Cwi---C---   <1.82t [lv_cache]
[lv_system_corig]
unknown
   [lv_system_corig] vg_ernie owi---C---   <1.82t
unknown

The live system I am running these commands from is a Fedora 33:

$ uname -a
Linux localhost-live 5.8.15-301.fc33.x86_64 #1 SMP Thu Oct 15 16:58:06 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
$ sudo lvm version
   LVM version:       2.03.10(2) (2020-08-09)
   Library version: 1.02.173 (2020-08-09)
   Driver version:   4.42.0
   Configuration:    ./configure --build=x86_64-redhat-linux-gnu --
host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking
--prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin --
sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/
lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib
--mandir=/usr/share/man --infodir=/usr/share/info --with-default-dm-run-dir=/
run --with-default-run-dir=/run/lvm --with-default-pid-dir=/run --with-
default-locking-dir=/run/lock/lvm --with-usrlibdir=/usr/lib64 --enable-fsadm
--enable-write_install --with-user= --with-group= --with-device-uid=0 --with-
device-gid=6 --with-device-mode=0660 --enable-pkgconfig --enable-cmdlib --
enable-dmeventd --enable-blkid_wiping --with-cluster=internal --enable-
cmirrord --with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync --with-
thin=internal --with-cache=internal --enable-lvmpolld --enable-lvmlockd-dlm --
enable-lvmlockd-dlmcontrol --enable-lvmlockd-sanlock --enable-dbus-service --
enable-notify-dbus --enable-dmfilemapd --with-writecache=internal --with-
vdo=internal --with-vdo-format=/usr/bin/vdoformat --disable-silent-rules

Is it possible that cache_check --clear-needs-check-flag does not clear needs_check flag does not actually clear the needs_check flag in my case?

Any help to get the system back online (without data loss, if possible) is appreciated!

Best regards,
Dennis

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#1 lore raw
Quoted 7 lines · On Sonntag, 6. Dezember 2020 22:01:37 CET Dennis Schridde wrote:

[...]

$ sudo lvchange -ay vg_ernie/lv_cache
Do you want to activate component LV in read-only mode? [y/n]: y
   Allowing activation of component LV.
[...]
Is it possible that `cache_check --clear-needs-check-flag does not clear
needs_check flag` does not actually clear the needs_check flag in my case?

I suspect that this has something to do with the read only status of the LV...

What if the superblock gets changed, but is then not written back to the LV, because it is read only, and failure to write it back was not treated as an error? I dd-ed lv_cache into a file, ran cache_check --clear-needs-check-flag on it, and then tried to dd it back to the LV. Sadly that results in a permission denied error. I was not able to figure out how to force writing it back.

--Dennis

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#2 lore raw
In reply to Dennis Schridde

Dne 06. 12. 20 v 22:01 Dennis Schridde napsal(a):

Quoted 23 lines

Hello!

A cached logical volume of mine cannot be activated anymore:

$ sudo vgchange -ay
    device-mapper: reload ioctl on   (253:6) failed: Invalid argument
    0 logical volume(s) in volume group "vg_ernie" now active

dmesg logs:

device-mapper: cache: 253:6: unable to switch cache to write mode until repaired.
device-mapper: cache: 253:6: switching cache to read-only mode
device-mapper: table: 253:6: cache: Unable to get write access to metadata, please check/repair metadata.
device-mapper: ioctl: error adding target to table

The code in question seems to be: https://github.com/torvalds/linux/blob/v5.8/
drivers/md/dm-cache-target.c#L957-L964

Quoted 6 lines

Hence I set out to check the cache and, if it is clean, clear the needs_check flag:

$ sudo lvchange -ay vg_ernie/lv_cache
Do you want to activate component LV in read-only mode? [y/n]: y
    Allowing activation of component LV.

As said - by component activation you will get 'read-only' volume thus you cannot do 'in-place' changes this way.

Quoted 18 lines

A bit puzzling is that the status of the needs_check flag appears to be "unknown":

$ sudo lvs -a -o +lv_check_needed
    LV                        VG          Attr          LSize    Pool
Origin                  Data%   Meta%   Move Log Cpy%Sync Convert CheckNeeded
    [lv_cache]            vg_ernie CRi-a-C--- 232.88g
unknown
    lv_system             vg_ernie Cwi---C---   <1.82t [lv_cache]
[lv_system_corig]
unknown
    [lv_system_corig] vg_ernie owi---C---   <1.82t
unknown

The live system I am running these commands from is a Fedora 33:

Quoted 8 lines
$ uname -a
Linux localhost-live 5.8.15-301.fc33.x86_64 #1 SMP Thu Oct 15 16:58:06 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
$ sudo lvm version
    LVM version:       2.03.10(2) (2020-08-09)
    Library version: 1.02.173 (2020-08-09)
    Driver version:   4.42.0

It seems you are using 'cvol' instead of 'cpool' solution which has been evolving and you will need a newer lvm2 version.

If you want to using caching with the version of lvm2 you have - you will need to use cpools (which are a bit faster anyway).

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#3 lore raw

Hello Zdenek!

Thank you for your reply!

Quoted 13 lines · On Mittwoch, 9. Dezember 2020 12:05:13 CET Zdenek Kabelac wrote:

Dne 06. 12. 20 v 22:01 Dennis Schridde napsal(a):

Quoted 11 lines

The live system I am running these commands from is a Fedora 33:

$ uname -a
Linux localhost-live 5.8.15-301.fc33.x86_64 #1 SMP Thu Oct 15 16:58:06 UTC
2020 x86_64 x86_64 x86_64 GNU/Linux
$ sudo lvm version
    LVM version:       2.03.10(2) (2020-08-09)
    Library version: 1.02.173 (2020-08-09)
    Driver version:   4.42.0
Quoted 5 lines

It seems you are using 'cvol' instead of 'cpool' solution which has been evolving and you will need a newer lvm2 version.

If you want to using caching with the version of lvm2 you have - you will need to use cpools (which are a bit faster anyway).

2.03.10 is the latest released version of lvm2. So I would have to use the Git main branch instead?

A bit confusing is that the lvmcache(7) manpage of lvm2 2.03.10 suggests to create a cachevol as the first option:

$ lvconvert --type cache --cachevol fast vg/main

Should this be adjusted, if cachevols are not yet fully supported in 2.03.10?

How would you proceed to repair this cache? I can recreate it as a cachepool, but first I need to gain back access to the data, i.e. get into a state where I can --uncache.

--Dennis

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#4 lore raw
Quoted 3 lines · Dennis Schridde

How would you proceed to repair this cache? I can recreate it as a cachepool, but first I need to gain back access to the data, i.e. get into a state where I can --uncache.

I know that cachevols do not yet work with lvconvert --repair, which uses cache_repair, but this must be another case where cache_repair is still needed. If this is writethough mode you should be able to simply drop the cache with lvconvert --splitcache and then recreate it.
Dave

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#5 lore raw
Quoted 9 lines · On Mittwoch, 9. Dezember 2020 20:30:51 CET David Teigland wrote:
Quoted 3 lines · Dennis Schridde

How would you proceed to repair this cache? I can recreate it as a cachepool, but first I need to gain back access to the data, i.e. get into a state where I can --uncache.

I know that cachevols do not yet work with lvconvert --repair, which uses cache_repair, but this must be another case where cache_repair is still needed. If this is writethough mode you should be able to simply drop the cache with lvconvert --splitcache and then recreate it.

Sadly the cache is in writeback mode.

Is my understanding correct that the cache itself is fine? It is just the inability to clear the needs_check flag that is preventing me from using the device? Or did I miss to run some checks that could reveal a problem / did I misinterpret the output of cache_check on lv_cache?

--Dennis

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#8 lore raw
In reply to David Teigland
Quoted 9 lines · On Mittwoch, 9. Dezember 2020 20:30:51 CET David Teigland wrote:
Quoted 3 lines · Dennis Schridde

How would you proceed to repair this cache? I can recreate it as a cachepool, but first I need to gain back access to the data, i.e. get into a state where I can --uncache.

I know that cachevols do not yet work with lvconvert --repair, which uses cache_repair, but this must be another case where cache_repair is still needed. If this is writethough mode you should be able to simply drop the cache with lvconvert --splitcache and then recreate it.

Would this be safe if cache_dump showed all blocks are clean (dirty=false), even if the cache was in writeback mode?

--Dennis

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#6 lore raw
In reply to Dennis Schridde
Quoted 11 lines · Dennis Schridde
Quoted 9 lines · On Mittwoch, 9. Dezember 2020 20:30:51 CET David Teigland wrote:
Quoted 3 lines · Dennis Schridde

How would you proceed to repair this cache? I can recreate it as a cachepool, but first I need to gain back access to the data, i.e. get into a state where I can --uncache.

I know that cachevols do not yet work with lvconvert --repair, which uses cache_repair, but this must be another case where cache_repair is still needed. If this is writethough mode you should be able to simply drop the cache with lvconvert --splitcache and then recreate it.

Quoted 5 lines

Sadly the cache is in writeback mode.

Is my understanding correct that the cache itself is fine? It is just the inability to clear the needs_check flag that is preventing me from using the device?

I suspect the cache itself is fine and we're just missing a way to clear that flag in the cache metadata. I'm looking for a workaround for that until we can sort out a complete fix.

Dave

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#7 lore raw
Quoted 3 lines

I suspect the cache itself is fine and we're just missing a way to clear that flag in the cache metadata. I'm looking for a workaround for that until we can sort out a complete fix.

The proper fix is here
https://sourceware.org/git/?p=lvm2.git;a=commit;h=9fe7aba251ff24c14277dfdf9be2d861a7699230

(A test for that using actual flagged metadata is still needed.)

Dave

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/