Skip to contentSkip to conversation

Removing lvmcache fails

linux-lvm Roy Sigurd Karlsbakk 2 participants 5 messages Sep 21, 2020

lore
Share & tools
lore
#0 lore raw

Hi all

I have an ssd hooked up to my system to do some caching of the raid-6 array. Now, that SSD isn't very new and has started developing some smart errors, so I thought I'd just remove it and find a nice recycle bin for it. However, this fails.

This is Debian Buster (latest) with kernel 5.4 and lvm 2.03.02(2)

Any idea how to get rid of this?

# lvconvert --uncache data/data
  Unknown feature in status: 8 2488/262144 128 819192/819200 57263288 35277148 19955332 9916661 0 1 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 smq 0 rw -
  Flushing 1 blocks for cache data/data.
  Unknown feature in status: 8 2484/262144 128 819192/819200 56391295 33200598 19684834 6317925 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
  Flushing 1 blocks for cache data/data.
  Unknown feature in status: 8 2484/262144 128 819192/819200 56391295 33200598 19684835 6317925 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
  Flushing 1 blocks for cache data/data.
  Unknown feature in status: 8 2484/262144 128 819192/819200 56391295 33200598 19684835 6317925 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
  Flushing 1 blocks for cache data/data.
  Unknown feature in status: 8 2484/262144 128 819192/819200 56391295 33200598 19684835 6317925 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 
(and so on)
# lvs -o+cache_mode data/data
  Unknown feature in status: 8 2484/262144 128 819195/819200 57263401 35277148 19955449 9916671 0 3 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 smq 0 rw -
  LV   VG   Attr       LSize  Pool     Origin       Data%  Meta%  Move Log Cpy%Sync Convert CacheMode
  data data Cwi-aoC--- 13,67t [_cache] [data_corig] 99,99  0,95            0,01             writethrough

Vennlig hilsen

roy

Signature · 6 lines
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.
#1 lore raw

Dne 21. 09. 20 v 10:51 Roy Sigurd Karlsbakk napsal(a):

Quoted 3 lines

Hi all

I have an ssd hooked up to my system to do some caching of the raid-6 array. Now, that SSD isn't very new and has started developing some smart errors, so I thought I'd just remove it and find a nice recycle bin for it. However, this fails.

Hmm - cache isn't yet very easy to use once you start to have faulty devices in your device stack

Quoted 7 lines

This is Debian Buster (latest) with kernel 5.4 and lvm 2.03.02(2)

Any idea how to get rid of this?

# lvconvert --uncache data/data
   Unknown feature in status: 8 2488/262144 128 819192/819200 57263288 35277148 19955332 9916661 0 1 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 smq 0 rw -

However since you have 'writethrough' cache mode - it should be possible to use 'lvconvert --uncache --force'

But I'm somehow confused how you can have any dirty blocks in this case ??

lvm2 2.03.02 version was somewhat experimental release - so I've recommend something newer - which will also properly parse newer status output from newer kernel dm-cache module.

Regards

Zdenek

#2 lore raw
Quoted 10 lines

Dne 21. 09. 20 v 10:51 Roy Sigurd Karlsbakk napsal(a):

Quoted 6 lines

Hi all

I have an ssd hooked up to my system to do some caching of the raid-6 array. Now, that SSD isn't very new and has started developing some smart errors, so I thought I'd just remove it and find a nice recycle bin for it. However, this fails.

Hmm - cache isn't yet very easy to use once you start to have faulty devices in your device stack

Well, that's why I want to replace it ;)

Quoted 4 lines
Quoted 4 lines

This is Debian Buster (latest) with kernel 5.4 and lvm 2.03.02(2)

Any idea how to get rid of this?

Quoted 8 lines
Quoted 4 lines
# lvconvert --uncache data/data
   Unknown feature in status: 8 2488/262144 128 819192/819200 57263288 35277148
   19955332 9916661 0 1 1 3 metadata2 writethrough no_discard_passdown 2
   migration_threshold 2048 smq 0 rw -

However since you have 'writethrough' cache mode - it should be possible to use 'lvconvert --uncache --force'

# lvconvert --uncache --force data/data
  Unknown feature in status: 8 2484/262144 128 819198/819200 57470386 35323243 19987362 9940326 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 smq 0 rw -
  Flushing 1 blocks for cache data/data.
  Unknown feature in status: 8 2484/262144 128 819198/819200 57439248 35307947 19981793 9938992 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
  Flushing 1 blocks for cache data/data.
  Unknown feature in status: 8 2484/262144 128 819198/819200 57439248 35307947 19981793 9938992 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
(et cetera et cetera ad infinitum)
Quoted 1 line

But I'm somehow confused how you can have any dirty blocks in this case ??

So am I

Quoted 3 lines

lvm2 2.03.02 version was somewhat experimental release - so I've recommend something newer - which will also properly parse newer status output from newer kernel dm-cache module.

Well, that's weird - debian is usually rather conservative on its packages in a stable release. Are you sure?

I also updated to a backported kernel 5.7 without any change.

Vennlig hilsen

roy

Signature · 6 lines
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.
#3 lore raw

Dne 21. 09. 20 v 12:01 Roy Sigurd Karlsbakk napsal(a):

Quoted 19 lines
Quoted 6 lines

Dne 21. 09. 20 v 10:51 Roy Sigurd Karlsbakk napsal(a):

Quoted 2 lines

Hi all

However since you have 'writethrough' cache mode - it should be possible to use 'lvconvert --uncache --force'

# lvconvert --uncache --force data/data
   Unknown feature in status: 8 2484/262144 128 819198/819200 57470386 35323243 19987362 9940326 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 smq 0 rw -
   Flushing 1 blocks for cache data/data.
   Unknown feature in status: 8 2484/262144 128 819198/819200 57439248 35307947 19981793 9938992 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
   Flushing 1 blocks for cache data/data.
   Unknown feature in status: 8 2484/262144 128 819198/819200 57439248 35307947 19981793 9938992 0 0 1 3 metadata2 writethrough no_discard_passdown 2 migration_threshold 2048 cleaner 0 rw -
(et cetera et cetera ad infinitum)
Quoted 1 line

But I'm somehow confused how you can have any dirty blocks in this case ??

So am I

Quoted 7 lines
Quoted 3 lines

lvm2 2.03.02 version was somewhat experimental release - so I've recommend something newer - which will also properly parse newer status output from newer kernel dm-cache module.

Well, that's weird - debian is usually rather conservative on its packages in a stable release. Are you sure?

I also updated to a backported kernel 5.7 without any change.

Hi

Please can you open upstream BZ here:

https://bugzilla.redhat.com/enter_bug.cgi?product=LVM%20and%20device-mapper

List all the info - i.e. package versions (yep even .debs), kernel versions lvmdump archive.

I can probably easily 'hand-make' lvm2 metadata for you for 'vgcfgrestore' - but I'd like to track this case and create some reproducer for this issue so we can handle case were the cache cannot be cleared.

Regards

Zdenek

#4 lore raw
Quoted 10 lines

Please can you open upstream BZ here:

https://bugzilla.redhat.com/enter_bug.cgi?product=LVM%20and%20device-mapper

List all the info - i.e. package versions (yep even .debs), kernel versions lvmdump archive.

I can probably easily 'hand-make' lvm2 metadata for you for 'vgcfgrestore' - but I'd like to track this case and create some reproducer for this issue so we can handle case were the cache cannot be cleared.

https://bugzilla.redhat.com/show_bug.cgi?id=1881056

Here you go. Thanks for any help :)

PS: I love languages, I just wonder how you pronounce Zdenek ;)

Vennlig hilsen

roy

Signature · 6 lines
Roy Sigurd Karlsbakk
(+47) 98013356
http://blogg.karlsbakk.net/
GPG Public key: http://karlsbakk.net/roysigurdkarlsbakk.pubkey.txt
--
Hið góða skaltu í stein höggva, hið illa í snjó rita.