cache_check --clear-needs-check-flag does not clear needs_check flag?
Hello!
A cached logical volume of mine cannot be activated anymore:
$ sudo vgchange -ay device-mapper: reload ioctl on (253:6) failed: Invalid argument 0 logical volume(s) in volume group "vg_ernie" now active
dmesg logs:
device-mapper: cache: 253:6: unable to switch cache to write mode until repaired.
device-mapper: cache: 253:6: switching cache to read-only mode
device-mapper: table: 253:6: cache: Unable to get write access to metadata, please check/repair metadata.
device-mapper: ioctl: error adding target to table
The code in question seems to be: https://github.com/torvalds/linux/blob/v5.8/
drivers/md/dm-cache-target.c#L957-L964
Hence I set out to check the cache and, if it is clean, clear the needs_check flag:
$ sudo lvchange -ay vg_ernie/lv_cache Do you want to activate component LV in read-only mode? [y/n]: y Allowing activation of component LV.
$ sudo cache_check /dev/vg_ernie/lv_cache examining superblock examining mapping array examining hint array examining discard bitset
$ sudo cache_check --clear-needs-check-flag /dev/vg_ernie/lv_cache examining superblock examining mapping array examining hint array examining discard bitset
$ sudo lvchange -an vg_ernie/lv_cache
But the problem persists:
$ sudo vgchange -ay device-mapper: reload ioctl on (253:6) failed: Invalid argument 0 logical volume(s) in volume group "vg_ernie" now active
I tried again in read/write mode, in case that would make a difference / silently fail:
$ sudo lvchange -ay vg_ernie/lv_cache Do you want to activate component LV in read-only mode? [y/n]: n
$ sudo cache_check --clear-needs-check-flag /dev/vg_ernie/lv_cache examining superblock examining mapping array examining hint array examining discard bitset
$ sudo lvchange -an vg_ernie/lv_cache
With the same results:
$ sudo vgchange -ay device-mapper: reload ioctl on (253:6) failed: Invalid argument 0 logical volume(s) in volume group "vg_ernie" now active
A bit puzzling is that the status of the needs_check flag appears to be "unknown":
$ sudo lvs -a -o +lv_check_needed LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert CheckNeeded [lv_cache] vg_ernie CRi-a-C--- 232.88g unknown lv_system vg_ernie Cwi---C--- <1.82t [lv_cache] [lv_system_corig] unknown [lv_system_corig] vg_ernie owi---C--- <1.82t unknown
The live system I am running these commands from is a Fedora 33:
$ uname -a Linux localhost-live 5.8.15-301.fc33.x86_64 #1 SMP Thu Oct 15 16:58:06 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ sudo lvm version LVM version: 2.03.10(2) (2020-08-09) Library version: 1.02.173 (2020-08-09) Driver version: 4.42.0 Configuration: ./configure --build=x86_64-redhat-linux-gnu -- host=x86_64-redhat-linux-gnu --program-prefix= --disable-dependency-tracking --prefix=/usr --exec-prefix=/usr --bindir=/usr/bin --sbindir=/usr/sbin -- sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/ lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-default-dm-run-dir=/ run --with-default-run-dir=/run/lvm --with-default-pid-dir=/run --with- default-locking-dir=/run/lock/lvm --with-usrlibdir=/usr/lib64 --enable-fsadm --enable-write_install --with-user= --with-group= --with-device-uid=0 --with- device-gid=6 --with-device-mode=0660 --enable-pkgconfig --enable-cmdlib -- enable-dmeventd --enable-blkid_wiping --with-cluster=internal --enable- cmirrord --with-udevdir=/usr/lib/udev/rules.d --enable-udev_sync --with- thin=internal --with-cache=internal --enable-lvmpolld --enable-lvmlockd-dlm -- enable-lvmlockd-dlmcontrol --enable-lvmlockd-sanlock --enable-dbus-service -- enable-notify-dbus --enable-dmfilemapd --with-writecache=internal --with- vdo=internal --with-vdo-format=/usr/bin/vdoformat --disable-silent-rules
Is it possible that cache_check --clear-needs-check-flag does not clear needs_check flag does not actually clear the needs_check flag in my case?
Any help to get the system back online (without data loss, if possible) is appreciated!
Best regards,
Dennis
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 7 lines · On Sonntag, 6. Dezember 2020 22:01:37 CET Dennis Schridde wrote:
[...]
$ sudo lvchange -ay vg_ernie/lv_cache Do you want to activate component LV in read-only mode? [y/n]: y Allowing activation of component LV. [...] Is it possible that `cache_check --clear-needs-check-flag does not clear needs_check flag` does not actually clear the needs_check flag in my case?
I suspect that this has something to do with the read only status of the LV...
What if the superblock gets changed, but is then not written back to the LV, because it is read only, and failure to write it back was not treated as an error? I dd-ed lv_cache into a file, ran cache_check --clear-needs-check-flag on it, and then tried to dd it back to the LV. Sadly that results in a permission denied error. I was not able to figure out how to force writing it back.
--Dennis
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Dne 06. 12. 20 v 22:01 Dennis Schridde napsal(a):
Quoted 23 lines
Hello!
A cached logical volume of mine cannot be activated anymore:
$ sudo vgchange -ay device-mapper: reload ioctl on (253:6) failed: Invalid argument 0 logical volume(s) in volume group "vg_ernie" now activedmesg logs:
device-mapper: cache: 253:6: unable to switch cache to write mode until repaired.
device-mapper: cache: 253:6: switching cache to read-only mode
device-mapper: table: 253:6: cache: Unable to get write access to metadata, please check/repair metadata.
device-mapper: ioctl: error adding target to tableThe code in question seems to be: https://github.com/torvalds/linux/blob/v5.8/
drivers/md/dm-cache-target.c#L957-L964
Quoted 6 lines
Hence I set out to check the cache and, if it is clean, clear the
needs_checkflag:$ sudo lvchange -ay vg_ernie/lv_cache Do you want to activate component LV in read-only mode? [y/n]: y Allowing activation of component LV.
As said - by component activation you will get 'read-only' volume thus you cannot do 'in-place' changes this way.
Quoted 18 lines
A bit puzzling is that the status of the
needs_checkflag appears to be "unknown":$ sudo lvs -a -o +lv_check_needed LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert CheckNeeded [lv_cache] vg_ernie CRi-a-C--- 232.88g unknown lv_system vg_ernie Cwi---C--- <1.82t [lv_cache] [lv_system_corig] unknown [lv_system_corig] vg_ernie owi---C--- <1.82t unknownThe live system I am running these commands from is a Fedora 33:
Quoted 8 lines
$ uname -a Linux localhost-live 5.8.15-301.fc33.x86_64 #1 SMP Thu Oct 15 16:58:06 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux$ sudo lvm version LVM version: 2.03.10(2) (2020-08-09) Library version: 1.02.173 (2020-08-09) Driver version: 4.42.0
It seems you are using 'cvol' instead of 'cpool' solution which has been evolving and you will need a newer lvm2 version.
If you want to using caching with the version of lvm2 you have - you will need to use cpools (which are a bit faster anyway).
Regards
Zdenek
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Hello Zdenek!
Thank you for your reply!
Quoted 13 lines · On Mittwoch, 9. Dezember 2020 12:05:13 CET Zdenek Kabelac wrote:
Dne 06. 12. 20 v 22:01 Dennis Schridde napsal(a):
Quoted 11 lines
The live system I am running these commands from is a Fedora 33:
$ uname -a Linux localhost-live 5.8.15-301.fc33.x86_64 #1 SMP Thu Oct 15 16:58:06 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux$ sudo lvm versionLVM version: 2.03.10(2) (2020-08-09) Library version: 1.02.173 (2020-08-09) Driver version: 4.42.0
Quoted 5 lines
It seems you are using 'cvol' instead of 'cpool' solution which has been evolving and you will need a newer lvm2 version.
If you want to using caching with the version of lvm2 you have - you will need to use cpools (which are a bit faster anyway).
2.03.10 is the latest released version of lvm2. So I would have to use the Git main branch instead?
A bit confusing is that the lvmcache(7) manpage of lvm2 2.03.10 suggests to create a cachevol as the first option:
$ lvconvert --type cache --cachevol fast vg/main
Should this be adjusted, if cachevols are not yet fully supported in 2.03.10?
How would you proceed to repair this cache? I can recreate it as a cachepool, but first I need to gain back access to the data, i.e. get into a state where I can --uncache.
--Dennis
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 3 lines · Dennis Schridde
How would you proceed to repair this cache? I can recreate it as a cachepool, but first I need to gain back access to the data, i.e. get into a state where I can --uncache.
I know that cachevols do not yet work with lvconvert --repair, which uses cache_repair, but this must be another case where cache_repair is still needed. If this is writethough mode you should be able to simply drop the cache with lvconvert --splitcache and then recreate it.
Dave
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 9 lines · On Mittwoch, 9. Dezember 2020 20:30:51 CET David Teigland wrote:
Quoted 3 lines · Dennis Schridde
How would you proceed to repair this cache? I can recreate it as a cachepool, but first I need to gain back access to the data, i.e. get into a state where I can --uncache.
I know that cachevols do not yet work with lvconvert --repair, which uses
cache_repair, but this must be another case wherecache_repairis still needed. If this is writethough mode you should be able to simply drop the cache with lvconvert --splitcache and then recreate it.
Sadly the cache is in writeback mode.
Is my understanding correct that the cache itself is fine? It is just the inability to clear the needs_check flag that is preventing me from using the device? Or did I miss to run some checks that could reveal a problem / did I misinterpret the output of cache_check on lv_cache?
--Dennis
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 11 lines · Dennis Schridde
Quoted 9 lines · On Mittwoch, 9. Dezember 2020 20:30:51 CET David Teigland wrote:
Quoted 3 lines · Dennis Schridde
How would you proceed to repair this cache? I can recreate it as a cachepool, but first I need to gain back access to the data, i.e. get into a state where I can --uncache.
I know that cachevols do not yet work with lvconvert --repair, which uses
cache_repair, but this must be another case wherecache_repairis still needed. If this is writethough mode you should be able to simply drop the cache with lvconvert --splitcache and then recreate it.
Quoted 5 lines
Sadly the cache is in writeback mode.
Is my understanding correct that the cache itself is fine? It is just the inability to clear the
needs_checkflag that is preventing me from using the device?
I suspect the cache itself is fine and we're just missing a way to clear that flag in the cache metadata. I'm looking for a workaround for that until we can sort out a complete fix.
Dave
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 3 lines
I suspect the cache itself is fine and we're just missing a way to clear that flag in the cache metadata. I'm looking for a workaround for that until we can sort out a complete fix.
The proper fix is here
https://sourceware.org/git/?p=lvm2.git;a=commit;h=9fe7aba251ff24c14277dfdf9be2d861a7699230
(A test for that using actual flagged metadata is still needed.)
Dave
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
Quoted 9 lines · On Mittwoch, 9. Dezember 2020 20:30:51 CET David Teigland wrote:
Quoted 3 lines · Dennis Schridde
How would you proceed to repair this cache? I can recreate it as a cachepool, but first I need to gain back access to the data, i.e. get into a state where I can --uncache.
I know that cachevols do not yet work with lvconvert --repair, which uses
cache_repair, but this must be another case wherecache_repairis still needed. If this is writethough mode you should be able to simply drop the cache with lvconvert --splitcache and then recreate it.
Would this be safe if cache_dump showed all blocks are clean (dirty=false), even if the cache was in writeback mode?
--Dennis
_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/