Skip to contentSkip to conversation

Running thin_trim before activating a thin pool

linux-lvm Demi Marie Obenour 3 participants 11 messages Jan 29, 2022 – Jan 30, 2022

lore
Share & tools
lore
#0 lore raw

Is it possible to configure LVM2 so that it runs thin_trim before it activates a thin pool? Qubes OS currently runs blkdiscard on every thin volume before deleting it, which is slow and unreliable. Would running thin_trim during system startup provide a better alternative?

Signature · 3 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#1 lore raw

Dne 29. 01. 22 v 19:52 Demi Marie Obenour napsal(a):

Quoted 4 lines

Is it possible to configure LVM2 so that it runs thin_trim before it activates a thin pool? Qubes OS currently runs blkdiscard on every thin volume before deleting it, which is slow and unreliable. Would running thin_trim during system startup provide a better alternative?

Hi

Nope there is currently no support from lvm2 side for this. Feel free to open RFE.

I guess this would possibly justify some form of support for 'writable' component activation.

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#2 lore raw
Quoted 11 lines · Zdenek Kabelac

Dne 29. 01. 22 v 19:52 Demi Marie Obenour napsal(a):

Quoted 4 lines

Is it possible to configure LVM2 so that it runs thin_trim before it activates a thin pool? Qubes OS currently runs blkdiscard on every thin volume before deleting it, which is slow and unreliable. Would running thin_trim during system startup provide a better alternative?

Hi

Nope there is currently no support from lvm2 side for this. Feel free to open RFE.

Done: https://bugzilla.redhat.com/show_bug.cgi?id=2048160

Signature · 3 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#3 lore raw

Dne 29. 01. 22 v 21:09 Demi Marie Obenour napsal(a):

Quoted 7 lines · Zdenek Kabelac
Quoted 6 lines

Dne 29. 01. 22 v 19:52 Demi Marie Obenour napsal(a):

Quoted 4 lines

Is it possible to configure LVM2 so that it runs thin_trim before it activates a thin pool? Qubes OS currently runs blkdiscard on every thin volume before deleting it, which is slow and unreliable. Would running thin_trim during system startup provide a better alternative?

Quoted 9 lines
Quoted 5 lines

Hi

Nope there is currently no support from lvm2 side for this. Feel free to open RFE.

Done: https://bugzilla.redhat.com/show_bug.cgi?id=2048160

Thanks

Although your use-case Thinpool on top of VDO is not really a good plan and there is a good reason behind why lvm2 does not support this device stack directly (aka thin-pool data LV as VDO LV). I'd say you are stepping on very very thin ice...

Also I assume you have already checked performance of discard on VDO, but I would not want to run this operation frequently on any larger volume...

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#4 lore raw
Quoted 18 lines · Zdenek Kabelac

Dne 29. 01. 22 v 21:09 Demi Marie Obenour napsal(a):

Quoted 7 lines · Zdenek Kabelac
Quoted 6 lines

Dne 29. 01. 22 v 19:52 Demi Marie Obenour napsal(a):

Quoted 4 lines

Is it possible to configure LVM2 so that it runs thin_trim before it activates a thin pool? Qubes OS currently runs blkdiscard on every thin volume before deleting it, which is slow and unreliable. Would running thin_trim during system startup provide a better alternative?

Quoted 9 lines
Quoted 5 lines

Hi

Nope there is currently no support from lvm2 side for this. Feel free to open RFE.

Done: https://bugzilla.redhat.com/show_bug.cgi?id=2048160

Quoted 6 lines

Thanks

Although your use-case Thinpool on top of VDO is not really a good plan and there is a good reason behind why lvm2 does not support this device stack directly (aka thin-pool data LV as VDO LV).
I'd say you are stepping on very very thin ice...

Thin pool on VDO is not my actual use-case. The actual reason for the ticket is slow discards of thin devices that are about to be deleted; you can find more details in the linked GitHub issue. That said, now I am curious why you state that dm-thin on top of dm-vdo (that is, userspace/filesystem/VM/etc ⇒ dm-thin data (*not* metadata) ⇒ dm-vdo ⇒ hardware/dm-crypt/etc) is a bad idea. It seems to be a decent way to add support for efficient snapshots of data stored on a VDO volume, and to have multiple volumes on top of a single VDO volume. Furthermore,
https://access.redhat.com/articles/2106521#vdo recommends exactly this use-case. Or am I misunderstanding you?

Quoted 2 lines

Also I assume you have already checked performance of discard on VDO, but I would not want to run this operation frequently on any larger volume...

I have never actually used VDO myself, although the documentation does warn about this.

Signature · 3 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#5 lore raw

Dne 30. 01. 22 v 2:20 Demi Marie Obenour napsal(a):

Quoted 21 lines · Zdenek Kabelac
Quoted 20 lines

Dne 29. 01. 22 v 21:09 Demi Marie Obenour napsal(a):

Quoted 7 lines · Zdenek Kabelac
Quoted 6 lines

Dne 29. 01. 22 v 19:52 Demi Marie Obenour napsal(a):

Quoted 4 lines

Is it possible to configure LVM2 so that it runs thin_trim before it activates a thin pool? Qubes OS currently runs blkdiscard on every thin volume before deleting it, which is slow and unreliable. Would running thin_trim during system startup provide a better alternative?

Quoted 9 lines
Quoted 5 lines

Hi

Nope there is currently no support from lvm2 side for this. Feel free to open RFE.

Done: https://bugzilla.redhat.com/show_bug.cgi?id=2048160

Thanks

Quoted 7 lines
Quoted 4 lines

Although your use-case Thinpool on top of VDO is not really a good plan and there is a good reason behind why lvm2 does not support this device stack directly (aka thin-pool data LV as VDO LV).
I'd say you are stepping on very very thin ice...

Thin pool on VDO is not my actual use-case. The actual reason for the ticket is slow discards of thin devices that are about to be deleted;

Hi

Discard of thins itself is AFAIC pretty fast - unless you have massively sized thin devices with many GiB of metadata - obviously you cannot process this amount of metadata in nanoseconds (and there are prepared kernel patches to make it even faster)

What is the problem is the speed of discard of physical devices. You could actually try to feel difference with:
lvchange --discards passdown|nopassdown thinpool

Also it's very important to keep metadata on fast storage device (SSD/NVMe)! Keeping metadata on same hdd spindle as data is always going to feel slow (in fact it's quite pointless to talk about performance and use hdd...)

Quoted 4 lines

you can find more details in the linked GitHub issue. That said, now I am curious why you state that dm-thin on top of dm-vdo (that is, userspace/filesystem/VM/etc ⇒ dm-thin data (*not* metadata) ⇒ dm-vdo ⇒ hardware/dm-crypt/etc) is a bad idea. It seems to be a decent way to

Out-of-space recoveries are ATM much harder then what we want.

So as long as user can maintain free space of your VDO and thin-pool it's ok. Once user runs out of space - recovery is pretty hard task (and there is reason we have support...)

Quoted 2 lines

add support for efficient snapshots of data stored on a VDO volume, and to have multiple volumes on top of a single VDO volume. Furthermore,

We hope we will add some direct 'snapshot' support to VDO so users will not need to combine both technologies together.

Thin is more oriented towards extreme speed.
VDO is more about 'compression & deduplication' - so space efficiency.

Combining both together is kind of harming their advantages.

Quoted 2 lines

https://access.redhat.com/articles/2106521#vdo recommends exactly this use-case. Or am I misunderstanding you?

There are many paths to Rome...
So as mentioned above - you need to pick performance/space effieciency.
And since you want to write your own thin volume managing software, I'm guessing you care about performance a lot (so we do - but with our given constrains that are limiting us to some level)...

Quoted 5 lines
Quoted 2 lines

Also I assume you have already checked performance of discard on VDO, but I would not want to run this operation frequently on any larger volume...

I have never actually used VDO myself, although the documentation does warn about this.

It's been purely related to the initial BZ description which cares a lot about thin discard performance and following comment adds VDO discard into same equation... :)

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#6 lore raw
Quoted 30 lines · Zdenek Kabelac

Dne 30. 01. 22 v 2:20 Demi Marie Obenour napsal(a):

Quoted 21 lines · Zdenek Kabelac
Quoted 20 lines

Dne 29. 01. 22 v 21:09 Demi Marie Obenour napsal(a):

Quoted 7 lines · Zdenek Kabelac
Quoted 6 lines

Dne 29. 01. 22 v 19:52 Demi Marie Obenour napsal(a):

Quoted 4 lines

Is it possible to configure LVM2 so that it runs thin_trim before it activates a thin pool? Qubes OS currently runs blkdiscard on every thin volume before deleting it, which is slow and unreliable. Would running thin_trim during system startup provide a better alternative?

Quoted 9 lines
Quoted 5 lines

Hi

Nope there is currently no support from lvm2 side for this. Feel free to open RFE.

Done: https://bugzilla.redhat.com/show_bug.cgi?id=2048160

Thanks

Quoted 7 lines
Quoted 4 lines

Although your use-case Thinpool on top of VDO is not really a good plan and there is a good reason behind why lvm2 does not support this device stack directly (aka thin-pool data LV as VDO LV).
I'd say you are stepping on very very thin ice...

Thin pool on VDO is not my actual use-case. The actual reason for the ticket is slow discards of thin devices that are about to be deleted;

Quoted 6 lines

Hi

Discard of thins itself is AFAIC pretty fast - unless you have massively sized thin devices with many GiB of metadata - obviously you cannot process this amount of metadata in nanoseconds (and there are prepared kernel patches to make it even faster)

Would you be willing and able to share those patches?

Quoted 3 lines

What is the problem is the speed of discard of physical devices. You could actually try to feel difference with:
lvchange --discards passdown|nopassdown thinpool

In Qubes OS I believe we do need the discards to be passed down eventually, but I doubt it needs to be synchronous. Being able to run the equivalent of fstrim -av periodically would be amazing. I’m CC’ing Marek Marczykowski-Górecki (Qubes OS project lead) in case he has something to say.

Quoted 3 lines

Also it's very important to keep metadata on fast storage device (SSD/NVMe)! Keeping metadata on same hdd spindle as data is always going to feel slow (in fact it's quite pointless to talk about performance and use hdd...)

That explains why I had such a horrible experience with my initial (split between NVMe and HDD) install. I would not be surprised if some or all of the metadata volume wound up on the spinning disk.

Quoted 6 lines
Quoted 4 lines

you can find more details in the linked GitHub issue. That said, now I am curious why you state that dm-thin on top of dm-vdo (that is, userspace/filesystem/VM/etc ⇒ dm-thin data (*not* metadata) ⇒ dm-vdo ⇒ hardware/dm-crypt/etc) is a bad idea. It seems to be a decent way to

Out-of-space recoveries are ATM much harder then what we want.

Okay, thanks! Will this be fixed in a future version?

Quoted 3 lines

So as long as user can maintain free space of your VDO and thin-pool it's ok. Once user runs out of space - recovery is pretty hard task (and there is reason we have support...)

Out of space is already a tricky issue in Qubes OS. I certainly would not want to make it worse.

Quoted 5 lines
Quoted 2 lines

add support for efficient snapshots of data stored on a VDO volume, and to have multiple volumes on top of a single VDO volume. Furthermore,

We hope we will add some direct 'snapshot' support to VDO so users will not need to combine both technologies together.

Does that include support for splitting a VDO volume into multiple, individually-snapshottable volumes, the way thin works?

Quoted 4 lines

Thin is more oriented towards extreme speed.
VDO is more about 'compression & deduplication' - so space efficiency.

Combining both together is kind of harming their advantages.

That makes sense.

Signature · 3 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#7 lore raw

Dne 30. 01. 22 v 18:30 Demi Marie Obenour napsal(a):

Quoted 5 lines · Zdenek Kabelac
Quoted 4 lines

Dne 30. 01. 22 v 2:20 Demi Marie Obenour napsal(a):

Quoted 3 lines · Zdenek Kabelac
Quoted 2 lines

Dne 29. 01. 22 v 21:09 Demi Marie Obenour napsal(a):

Quoted 1 line · Zdenek Kabelac
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

Dne 29. 01. 22 v 19:52 Demi Marie Obenour napsal(a):

Discard of thins itself is AFAIC pretty fast - unless you have massively sized thin devices with many GiB of metadata - obviously you cannot process this amount of metadata in nanoseconds (and there are prepared kernel patches to make it even faster)

Would you be willing and able to share those patches?

Then are always landing in upstream kernel once they are all validated & tested (recent kernel already has many speed enhancements).

Quoted 10 lines
Quoted 4 lines

What is the problem is the speed of discard of physical devices. You could actually try to feel difference with:
lvchange --discards passdown|nopassdown thinpool

In Qubes OS I believe we do need the discards to be passed down eventually, but I doubt it needs to be synchronous. Being able to run the equivalent of fstrim -av periodically would be amazing. I’m CC’ing Marek Marczykowski-Górecki (Qubes OS project lead) in case he has something to say.

You could easily run in parallel individual blkdiscards for your thin LVs.... For most modern drives thought it's somewhat waste of time...

Those trimming tools should be used when they are solving some real problems, running them just for fun is just energy & performance waste....

Quoted 8 lines
Quoted 4 lines

Also it's very important to keep metadata on fast storage device (SSD/NVMe)! Keeping metadata on same hdd spindle as data is always going to feel slow (in fact it's quite pointless to talk about performance and use hdd...)

That explains why I had such a horrible experience with my initial (split between NVMe and HDD) install. I would not be surprised if some or all of the metadata volume wound up on the spinning disk.

With lvm2 user can always 'pvmove' any LV to any desired PV. There is not yet any 'smart' logic to do it automatically.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

add support for efficient snapshots of data stored on a VDO volume, and to have multiple volumes on top of a single VDO volume. Furthermore,

We hope we will add some direct 'snapshot' support to VDO so users will not need to combine both technologies together.

Does that include support for splitting a VDO volume into multiple, individually-snapshottable volumes, the way thin works?

Yes - that's the plan - to have multiple VDO LV in a single VDOPool.

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#8 lore raw
Quoted 11 lines · Zdenek Kabelac

Dne 30. 01. 22 v 18:30 Demi Marie Obenour napsal(a):

Quoted 7 lines · Zdenek Kabelac
Quoted 4 lines

Discard of thins itself is AFAIC pretty fast - unless you have massively sized thin devices with many GiB of metadata - obviously you cannot process this amount of metadata in nanoseconds (and there are prepared kernel patches to make it even faster)

Would you be willing and able to share those patches?

Then are always landing in upstream kernel once they are all validated & tested (recent kernel already has many speed enhancements).

Thanks! Which mailing list should I be watching?

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

What is the problem is the speed of discard of physical devices. You could actually try to feel difference with:
lvchange --discards passdown|nopassdown thinpool

In Qubes OS I believe we do need the discards to be passed down eventually, but I doubt it needs to be synchronous. Being able to run the equivalent of fstrim -av periodically would be amazing. I’m CC’ing Marek Marczykowski-Górecki (Qubes OS project lead) in case he has something to say.

Quoted 5 lines

You could easily run in parallel individual blkdiscards for your thin LVs.... For most modern drives thought it's somewhat waste of time...

Those trimming tools should be used when they are solving some real problems, running them just for fun is just energy & performance waste....

My understanding (which could be wrong) is that periodic trim is necessary for SSDs.

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

Also it's very important to keep metadata on fast storage device (SSD/NVMe)! Keeping metadata on same hdd spindle as data is always going to feel slow (in fact it's quite pointless to talk about performance and use hdd...)

That explains why I had such a horrible experience with my initial (split between NVMe and HDD) install. I would not be surprised if some or all of the metadata volume wound up on the spinning disk.

With lvm2 user can always 'pvmove' any LV to any desired PV. There is not yet any 'smart' logic to do it automatically.

Good point. I was probably unware of that at the time.

Signature · 3 lines
Sincerely,
Demi Marie Obenour (she/her/hers)
Invisible Things Lab

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#9 lore raw

Dne 30. 01. 22 v 19:01 Demi Marie Obenour napsal(a):

Quoted 7 lines · Zdenek Kabelac
Quoted 4 lines

Dne 30. 01. 22 v 18:30 Demi Marie Obenour napsal(a):

Quoted 1 line · Zdenek Kabelac

Then are always landing in upstream kernel once they are all validated & tested (recent kernel already has many speed enhancements).

Thanks! Which mailing list should I be watching?

lkml

Quoted 8 lines
Quoted 5 lines

You could easily run in parallel individual blkdiscards for your thin LVs.... For most modern drives thought it's somewhat waste of time...

Those trimming tools should be used when they are solving some real problems, running them just for fun is just energy & performance waste....

My understanding (which could be wrong) is that periodic trim is necessary for SSDs.

This was useful for archaic SSDs. Modern SSD/NVMe drives are much smarter...

Regards

Zdenek

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#10 lore raw
In reply to Zdenek Kabelac

Il 2022-01-30 12:18 Zdenek Kabelac ha scritto:

Quoted 4 lines

Thin is more oriented towards extreme speed.
VDO is more about 'compression & deduplication' - so space efficiency.

Combining both together is kind of harming their advantages.

Unfortunately, it is the only (current) solution to have snapshotting with data compression/deduplication. Integrating snapshot capability into VDO would be awesome!
Thanks.

Signature · 11 lines
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: [email protected] - [email protected]
GPG public key ID: FF5F32A8

_______________________________________________
linux-lvm mailing list
[email protected]
https://listman.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/